Objective
As part of the OSIE project (Open Source Industrial Edge) we decided to provide a real-time linux image (with PREEMPT_RT kernel and isolated CPUs) suited for Time Sensitive Networking (TSN) - a set of standards to guarante packet transport with bounded latency, low packet delay variation and without packet loss - for an A20-OLinuXino-LIME2 board.
The flashable image can be found here. In the following post we will outline the steps to follow in order to create your such an image.
Sources
Downloads
This is the list of all files which can be useful in the following sections:
Steps
The following sections are:
Install dependencies
nexedi:~$ sudo apt-get install bc bison curl flex gcc-arm-linux-gnueabihf git libncurses-dev libssl-dev make pigz u-boot-tools
You can download the kernel here
or compile it yourself:
Clone the official RT linux repository:
nexedi:~$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git
Checkout v5.10.41-rt42 (the branch I worked on, more recent 5.10 branches will also probably work):
nexedi:~$ cd linux-stable-rt
nexedi:~/linux-stable-rt$ git checkout v5.10.41-rt42
Apply small patch to disable XPS by default:
nexedi:~/linux-stable-rt$ curl https://lab.nexedi.com/nexedi/rt-olimex-image/raw/master/0001-Unset-CONFIG_XPS-by-default.patch | git apply
nexedi:~/linux-stable-rt$ export ARCH=arm; export CROSS_COMPILE=arm-linux-gnueabihf-;
nexedi:~/linux-stable-rt$ make mrproper
nexedi:~/linux-stable-rt$ curl -o .config https://lab.nexedi.com/nexedi/rt-olimex-image/raw/master/5.10-custom
nexedi:~/linux-stable-rt$ make olddefconfig
nexedi:~/linux-stable-rt$ nb_cpu=<nb_cpu_to_use_for_compilation>
nexedi:~/linux-stable-rt$ make -j$nb_cpu LOADADDR=0x48000000 uImage
nexedi:~/linux-stable-rt$ make -j$nb_cpu modules
nexedi:~/linux-stable-rt$ make -j$nb_cpu INSTALL_MOD_PATH=output modules modules_install
nexedi:~/linux-stable-rt$ make sun7i-a20-olinuxino-lime2.dtb
Files required to install kernel:
nexedi:~/linux-stable-rt$ modules=$(readlink --canonicalize output/lib/modules)
nexedi:~/linux-stable-rt$ uImage=$(readlink --canonicalize arch/arm/boot/uImage)
nexedi:~/linux-stable-rt$ dtb=$(readlink --canonicalize arch/arm/boot/dts/sun7i-a20-olinuxino-lime2.dtb)
nexedi:~/linux-stable-rt$ cd ..
Download the rootfs archive here
Or make one yourself: linux-sunxi.org/Debootstrap
Be very careful to choose correct device:
nexedi:~$ device=/dev/<sd-card-device>
Clean SD card entry:
nexedi:~$ sudo dd if=/dev/zero of=${device} bs=1M count=1
nexedi:~$ sync
Create 2 partitions, one 16M bootstrap partition and another using the remaining space to hold the rootfs:
nexedi:~$ sudo fdisk ${device}
o # clear the in memory partition table
n # new partition
# default - primary partition
# default - partition number 1
# default - start at beginning of disk
+16M # 16 MB parttion
n # new partition
# default - primary partition
# default - partition number 2
# default, start immediately after preceding partition
# default, extend partition to end of disk
w # write the partition table
nexedi:~$ sudo mkfs.vfat ${device}1
nexedi:~$ sudo mkfs.ext4 ${device}2
If you want to build u-boot-sunxi-with-spl.bin there are explanations here, this fix was also applied to generate the provided binary
nexedi:~$ curl -o u-boot-sunxi-with-spl.bin https://lab.nexedi.com/nexedi/rt-olimex-image/raw/master/u-boot-sunxi-with-spl.bin
nexedi:~$ sudo dd if=u-boot-sunxi-with-spl.bin of=$device bs=1024 seek=8
nexedi:~$ sync
nexedi:~$ mkdir -p mnt_sd
nexedi:~$ sudo mount ${device}1 mnt_sd
nexedi:~$ sudo curl -o mnt_sd/boot.cmd https://lab.nexedi.com/nexedi/rt-olimex-image/raw/master/boot.cmd
nexedi:~$ sudo mkimage -C none -A arm -T script -d mnt_sd/boot.cmd mnt_sd/boot.scr
nexedi:~$ sudo umount ${device}1
nexedi:~$ sudo mount ${device}2 mnt_sd
nexedi:~$ sudo tar -C mnt_sd/ -xjpf sunxi_rootfs.tar.bz2
nexedi:~$ sudo umount ${device}2
nexedi:~$ rm -d mnt_sd
If you downloaded the kernel:
nexedi:~$ mkdir -p kernel_output
nexedi:~$ tar -xvf v5.10.41-rt42_kernel.tar.gz -C kernel_output
nexedi:~$ modules=$(readlink --canonicalize kernel_output/modules)
nexedi:~$ uImage=$(readlink --canonicalize kernel_output/uImage)
nexedi:~$ dtb=$(readlink --canonicalize kernel_output/sun7i-a20-olinuxino-lime2.dtb)
In any case do:
nexedi:~$ mkdir -p mnt_sd
nexedi:~$ sudo mount ${device}2 mnt_sd
nexedi:~$ rm -rf mnt_sd/lib/modules
nexedi:~$ sudo cp -R $modules mnt_sd/lib/
nexedi:~$ sudo umount mnt_sd
nexedi:~$ sudo mount ${device}1 mnt_sd
nexedi:~$ sudo cp $uImage mnt_sd/
nexedi:~$ sudo cp $dtb mnt_sd/
nexedi:~$ sudo umount mnt_sd
nexedi:~$ rm -rf mnt_sd
Boot on the SD card
login : olimex
password : olimex
olimex@debian:~$ sudo apt install git build-essential
olimex@debian:~$ git clone git://git.code.sf.net/p/linuxptp/code linuxptp
olimex@debian:~$ cd linuxptp
olimex@debian:~/linuxptp$ make
olimex@debian:~/linuxptp$ sudo make install
Write the following shell script at /usr/local/bin/pin-processes.sh :
#!/bin/sh
# Pin ethernet IRQ bottom half to CPU1
pin_number=$(cat /proc/interrupts | grep eth0 | grep -o '[0-9]\+' | head -1)
echo 2 > /proc/irq/$pin_number/smp_affinity
# Get PID for ethernet IRQ
irq_pid=$(pgrep "irq/$pin_number-eth0")
# Pin ethernet IRQ upper half to CPU1
taskset -p 2 $irq_pid
# Get PID for ksoftirqd1
ksoftirq1_pid=$(pgrep "ksoftirqd/1")
# Set high priority for ksoftirqd/1
chrt -f -p 97 $ksoftirq1_pid
olimex@debian:~$ sudo chmod 755 /usr/local/bin/pin-processes.sh
Write the following service script at /etc/systemd/system/pin-processes.service :
[Unit]
Description=Pin processes and set priorities
After=network.target
[Service]
ExecStart=/usr/local/bin/pin-processes.sh
[Install]
WantedBy=multi-user.target
olimex@debian:~$ sudo systemctl enable pin-processes
olimex@debian:~$ sudo systemctl start pin-processes
This is important especially for receiving packets quickly and for sending packets with ETF qdisc.
Shows max wake-up latencies on both CPUs, should be around 50µs and 30µs respectively:
olimex@debian:~$ sudo cyclictest --smp -p98
Simulate load, latencies should be around 90µs and 60µs respectively:
olimex@debian:~$ hackbench &> dev/null & sudo cyclictest --smp -p98
You can enable serial over usb otg following this article. However this is not enable by default because not using usb otg port when serial is enabled on it causes troubles (reboot hangs until a cable is plugged between usb otg and the terminal).