Objective

As part of the OSIE project (Open Source Industrial Edge) we decided to provide a real-time linux image (with PREEMPT_RT kernel and isolated CPUs) suited for Time Sensitive Networking (TSN) - a set of standards to guarante packet transport with bounded latency, low packet delay variation and without packet loss - for an A20-OLinuXino-LIME2 board.

The flashable image can be found here. In the following post we will outline the steps to follow in order to create your such an image.

Sources

Install dependencies

sudo apt install bc bison curl flex gcc-arm-linux-gnueabihf git libncurses-dev libssl-dev make pigz u-boot-tools

Get a kernel: download or compile it

You can download the kernel here

or compile it yourself:

Patch

#Clone the official RT linux repository
git clone https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git

#Checkout v5.6.19-rt12 (the branch I worked on, more recent 5.6 branches will also probably work)
cd linux-rt-devel/
git checkout v5.6.19-rt12

#Apply small patch to enable XPS by default
curl https://softinst142593.host.vifib.net/0001-Unset-CONFIG_XPS-by-default.patch | git apply

Kernel configuration

export ARCH=arm; export CROSS_COMPILE=arm-linux-gnueabihf-;
make mrproper
curl -o .config https://softinst142593.host.vifib.net/5.6-custom
make olddefconfig

Compilation

nb_cpu=<nb_cpu_to_use_for_compilation>
make -j$nb_cpu LOADADDR=0x48000000 uImage
make -j$nb_cpu modules
make -j$nb_cpu INSTALL_MOD_PATH=output modules modules_install
make sun7i-a20-olinuxino-lime2.dtb

#Files required to install kernel
modules=$(readlink --canonicalize output/lib/modules)
uImage=$(readlink --canonicalize arch/arm/boot/uImage)
dtb=$(readlink --canonicalize arch/arm/boot/dts/sun7i-a20-olinuxino-lime2.dtb)

cd ..

Create rootfs archive

Download the rootfs archive here

Or make one yourself: linux-sunxi.org/Debootstrap

Create a bootable SD card

#Be very careful to choose correct device
device=/dev/<sd-card-device>

#Clear the partitions
sudo dd if=/dev/zero of=${device} bs=1M count=1
sync

sudo fdisk ${device}

o # clear the in memory partition table
n # new partition

  # default - primary partition
  # default - partition number 1
  # default - start at beginning of disk
+16M # 16 MB parttion
n # new partition
  # default - primary partition
  # default - partition number 2
  # default, start immediately after preceding partition
  # default, extend partition to end of disk
w # write the partition table

sudo mkfs.vfat ${device}1
sudo mkfs.ext4 ${device}2

curl -o u-boot-sunxi-with-spl.bin https://softinst142593.host.vifib.net/u-boot-sunxi-with-spl.bin
sudo dd if=u-boot-sunxi-with-spl.bin of=$device bs=1024 seek=8
sync

mkdir -p mnt_sd
sudo mount ${device}1 mnt_sd

sudo curl -o mnt_sd/boot.cmd https://softinst142593.host.vifib.net/boot.cmd
sudo mkimage -C none -A arm -T script -d mnt_sd/boot.cmd mnt_sd/boot.scr
sudo umount ${device}1

sudo mount ${device}2 mnt_sd
sudo tar -C mnt_sd/ -xjpf sunxi_rootfs.tar.bz2
sudo umount ${device}2
rm -d mnt_sd

Install the kernel

#If you downloaded the kernel :
mkdir -p kernel_output
tar -xvf v5.6.19-rt12_kernel.tar.gz -C kernel_output
modules=$(readlink --canonicalize kernel_output/modules)
uImage=$(readlink --canonicalize kernel_output/uImage)
dtb=$(readlink --canonicalize kernel_output/sun7i-a20-olinuxino-lime2.dtb)

#In any case do :
mkdir -p mnt_sd

sudo mount ${device}2 mnt_sd
rm -rf mnt_sd/lib/modules
sudo cp -R $modules mnt_sd/lib/
sudo umount mnt_sd

sudo mount ${device}1 mnt_sd

sudo cp $uImage mnt_sd/
sudo cp $dtb mnt_sd/
sudo umount mnt_sd
rm -rf mnt_sd

Boot on the SD card

login : olimex
password : olimex

Install PTP

sudo apt install git build-essential
git clone http://git.code.sf.net/p/linuxptp/code linuxptp
cd linuxptp
make
sudo make install

Pin processes and set priorities

Write the following shell script at /usr/local/bin/pin-processes.sh :

#!/bin/sh

# Pin ethernet IRQ bottom half to CPU1
echo 2 > /proc/irq/56/smp_affinity
# Get PID for ethernet IRQ
irq56_pid=$(pgrep "irq/56-eth0")
# Pin ethernet IRQ upper half to CPU1
taskset -p 2 $irq56_pid
# Get PID for ksoftirqd1
ksoftirq1_pid=$(pgrep "ksoftirqd/1")
# Set high priority for ksoftirqd/1
chrt -f -p 97 $ksoftirq1_pid

sudo chmod 755 /usr/local/bin/pin-processes.sh

Write the following service script at  /etc/systemd/system/pin-processes.service :

[Unit]
Description=Pin processes and set priorities
After=network.target

[Service]
ExecStart=/usr/local/bin/pin-processes.sh

[Install]
WantedBy=multi-user.target

sudo systemctl enable pin-processes
sudo systemctl start pin-processes

This is important especially for receiving packets quickly and for sending packets with ETF qdisc.

Check PREEMPT_RT is setup correctly

#Shows max wake-up latencies on both CPUs, should be around 50µs and 30µs respectively
sudo cyclictest --smp -p98

#Simulate load, latencies should be around 90µs and 60µs respectively
hackbench &> /dev/null & sudo cyclictest --smp -p98