Objective

As part of the OSIE project (Open Source Industrial Edge) we wanted to demonstrate software defined TSN (time sensitive network) on generic open hardware boards running PREEMPT_RT Linux. For this purpose we used A20-OLinuXino-LIME2 boards setup as described in one of our previous blog post. The demonstration consists in having Olimex boards synchronously spin stepper motors, setup so that a slight error in synchronization causes the motors to collide.

Setup explanation

 

The server used is a Shuttle DH170, which has two gigabit ethernet interfaces with hardware PTP support, and 8 amd64 cores. The server is running PREEMPT_RT, has a wake-up latency of 14us maximum, and can generate PTP timestamps with nanosecond precision thanks to the hardware PTP support. We therefore considered the latencies and jitter induced by the server to be negligeable compared to the A20 boards.

As can be seen on the diagram, the time source for all devices is the realtime clock device of the server. As presented in one of our previous blog post TSN results on A20 Lime2, with this setup devices can synchronize with a precision of around 70us. Since the boards don't support hardware PTP, the path on which PTP measures time is between the server's NIC and the board's CPU, as is show on the diagram. The path between the board's NIC and the board's CPU is the main source of jitter for time synchronization.

Everytime a pulse is generated on the boards GPIO, the motor turns one step. The TSN data sent by the server to the boards contains a timestamp defining the time at which a GPIO pulse should be triggered. If the boards don't trigger the pulse at the correct time the motors collide.

Hardware

The stepper motor used is a 2-phase bipolar Nema 23 from Stepperonline, with a rated current of 2.8A, and 1.9Nm of torque. The driver used is a DM542 from ‎Yosoo Health Gear, with an input voltage of 18-48V and output current 1A to 4.2A. Since the GPIO's output voltage is 3.3V and the driver expects control signals to be at 5V, a logical level converter was used. The power supply used was a 360W regulated switching power supply. A breadboard was used to make the connections to the level converters, and since the GPIO on the boards are too thin an UEXT adaptor from Olimex was used.

List of all parts needed

Stepperonline Nema 23 Stepper Motor 1,9Nm 2,8A 4 wires Arbre 6,35 mm x 2 pieces

KeeYees 10pcs 4 Channels IIC I2C Module Bidirectionnal Logical Level Converter 3.3V-5V for Arduino x 2 pieces

Yosoo Health Gear Stepper Motor Driver 2 Phases DM542, 57/86 18-48VDC Peak 4.2A x 2 pieces

NEWSTYLE 110/220V 24V 15A Universal Regulated Switching Power Supply 360W x 1 pieces

Laser cut gear x 2 pieces

BREADBOARD-1 x 1 piece

Tightening ring x 2 pieces

IEC cord (or any cable with an electrical plug on one end) x 1 piece

Ethernet cables x 2 pieces

CABLE-40-40-10CM x 2 pieces

A20-OLinuXino-LIME2-UEXT x 2 pieces

A20-OLinuXino-LIME2 x 2 pieces

Shuttle DH170 x 1 piece

 

Setup the hardware

First the power supply needs to be plugged. For this an IEC cord can be used. Cut the wire, strip all three wires, twist and tin the stranded wires and connect them to the power supply. For safety measures you need to cut the tinned wires enough so that the metal part is not visible.

Then four cables to power the motor drivers are needed. You can cut the motor's cable since they are long enough. Again, strip both end of the wires and twist and tin the end of the stranded wires. Do the same thing for the end of the motor's wires. 

You can then connect the power supply and the driver, and the motors to the drivers (there is a table of all connections below for reference).

Next, you will need to solder the pins to the level shifters, and place them on the breadboard as shown on the following image.

Then, you need to put the GPIO adaptors on the Olimex boards. The GPIO on the boards are too small, which is why the adaptors are needed. The adaptor should be placed on the GPIO1 block.

Electrical wiring

This tables shows all wiring connections between (device A, connector A) to (device B, connector B)

Function Device A Connector A Connector B Device B
Driver pulse + Driver1 PUL+ +V Breadboard
Driver pulse - Driver1 PUL- HV4 Level Shifter
Driver direction + Driver1 DIR+ +V Breadboard
Driver direction - Driver1 DIR- HV3 Level Shifter
Driver ground Driver1 GND -V Power Supply
Driver power Driver1 +V +V Power Supply
Motor phase A + Driver1 A+ Black Wire Motor
Motor phase A - Driver1 A- Green Wire Motor
Motor phase B + Driver1 B+ Red Wire Motor
Motor phase B - Driver1 B- Blue Wire Motor
Level shifter 3.3V input Level Shifter LV 3.3V Olimex
Level shifter 5V input Level Shifter HV +V Breadboard
Level shifter ground Level Shifter High and Low GND GND Breadboard
Direction GPIO Level Shifter LV3 PG0 Olimex
Pulse GPIO Level Shifter LV4 PG1 Olimex
Breadboard ground Breadboard GND GND Olimex
Breadboard 5V Breadboard +V 5V Olimex

 

Function Device A Connector A Connector B Device B
Driver pulse + Driver2 PUL+ +V Breadboard
Driver pulse - Driver2 PUL- HV4 Level Shifter
Driver direction + Driver2 DIR+ HV3 Breadboard
Driver direction - Driver2 DIR- V+ Level Shifter
Driver ground Driver2 GND -V Power Supply
Driver power Driver2 +V +V Power Supply
Motor phase A + Driver2 A+ Black Wire Motor
Motor phase A - Driver2 A- Green Wire Motor
Motor phase B + Driver2 B+ Red Wire Motor
Motor phase B - Driver2 B- Blue Wire Motor
Level shifter 3.3V input Level Shifter LV 3.3V Olimex
Level shifter 5V input Level Shifter HV +V Breadboard
Level shifter ground Level Shifter High and Low GND GND Breadboard
Direction GPIO Level Shifter LV3 PG0 Olimex
Pulse GPIO Level Shifter LV4 PG1 Olimex
Breadboard ground Breadboard GND GND Olimex
Breadboard 5V Breadboard +V 5V Olimex

Since the stepper motor is a bipolar stepper motor, it needs for wires for the two phases (A and B).

Then, the driver needs two different signals: direction and pulse. Direction will control control the motor's spinning direction, you can invert DIR- and DIR+ input cables to invert the spinning direction. Each motor should have a different spinning direction. As for the pulse signal, every rise (0V to 5V) makes the motor spin one step (1 / 400th of a full 360° rotation).

Since the Olimex GPIO's signals are 3.3V, and we need 5V on the driver, we use a level shifter. The level shifter needs to be powered with a 5V signal and a 3.3V.

The two GPIOs, the 5V signal, the 3.3V signal, and the ground signal comes from the GPIO adaptor which is connected to GPIO1 block on Olimex. Here are schematics of the GPIO1 block:

Here is a close-up of the GPIO adaptor:

The colors of the close-up and the schematics corresponds.

The next step is to put the switches of the driver in the correct position, as shown in the following image:

The switch allows to control how much a motor should turn at each step. Here, we configure the motor to turn as little as possible at each step, which is 360° / 400

Motors

Place the gear between two ring tighteners on the motors and tighten the rings, be sure to put the gears at the same height.

You can then place the motors against each other, the magnets inside them will keep them attached to each other. A better option would be to use larger gears and mount them on a plate.

PREEMPT_RT kernel

Linux kernel 4.19 or higher (with PREEMPT_RT patch and isolated CPUs) is required to run the TSN demo.

Shuttle

On a debian based distribution you can install a kernel with PREEMPT_RT patch:

shuttle:~$ sudo apt install linux-image-rt-amd64

To isolate CPUs edit /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet isolcpus=1,2 rcu_nocbs=1,2 irqaffinity=1,2"

and then reboot.

Olimex boards

Images created in this blog post already match this requierements.

Network setup

To provide internet to the Shuttle and the Olimex boards we chose ip forwarding but any method achieving this purpose can be used.

Because both ethernet ports of the Shuttle are used to link it to Olimex boards we added an usb-ethernet adapter to provide an interface for internet. This interface will be refered as <primary-interface>.

Shuttle's network configuration

The Shuttle needs to be set as the gateway for Olimex boards. Shuttle's ethernet interfaces will be refered as <interface-to-Olimex1> and <interface-to-Olimex2>. <gateway-for-Olimex1> and <gateway-to-Olimex2> refer to the static addresses defined for the Shuttle.

Choose your IP addresses in different subnetworks. For example, if the IP address of your shuttle on the interface linked to internet is like 192.168.0.x then do not choose addresses 192.168.0.x for the gateways and prefere addresses like 192.168.1.1 and 192.168.2.1 . Also avoid IP addresses ending by 0 like 192.168.1.0 as they are reserved for broadcast.

The configuration I used :

Shuttle:
  enp1s0 : 192.168.101.1 <interface-to-Olimex1> : <gateway-for-Olimex1>
  enp2s0 : 192.168.102.1 <interface-to-Olimex2> : <gateway-to-Olimex2>
  enx00e04c88743d : 192.168.99.27 <primary-interface> :

Olimex1:
  eth0: 192.168.101.2 <Olimex1-ip>
  gateway: 192.168.101.1 <gateway-for-Olimex1>

Olimex2:
  eth0: 192.168.102.2 <Olimex2-ip>
  gateway: 192.168.102.2 <gateway-for-Olimex2>

Edit /etc/network/interfaces:

auto lo
iface lo inet loopback

auto <interface-to-Olimex1>
iface <interface-to-Olimex1> inet static
    address <gateway-for-Olimex1>
    netmask 255.255.255.0

auto <interface-to-Olimex2>
iface <interface-to-Olimex2> inet static
    address <gateway-to-Olimex2>
    netmask 255.255.255.0

auto <primary-interface>
allow-hotplug <primary-interface>
iface <primary-interface> inet dhcp

Olimex 1 network configuration

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address <Olimex1-ip>
    gateway <gateway-for-Olimex1>

It is recommanded to link Olimex boards to the shuttle one at a time to check the interface, becoming up on the shuttle when an ethernet cable is plugged, is the one expected to be linked to the Olimex (e.g. enp1s0 for Olimex1).

Olimex 2 network configuration

auto lo
iface lo inet loopback

auto eth0
allow-hotplug eth0
iface eth0 inet static
    address <Olimex2-ip>
    gateway <gateway-for-Olimex2>

For each device you can apply the new network configuration by running:

shuttle:~$ sudo ifdown -a && sudo ifup -a

Enable IP forwarding

From the Shuttle the following commands need to be run:

shuttle:~$ sudo sysctl -w net.ipv4.ip_forward=1
shuttle:~$ sudo iptables -t nat -A POSTROUTING -o <primary-interface> -j MASQUERADE
shuttle:~$ sudo iptables -I FORWARD -o <primary-interface> -s 192.168.0.0/16 -j ACCEPT
shuttle:~$ sudo iptables -I INPUT -s 192.168.0.0/16 -j ACCEPT

Motor control software

The software used to control motors is available in this repository: tsn-rt-measures, in the folder src/motor-control.

The motor control software contains a slave program and a master program. The slave programs run on the boards, and the master program runs on the server. The master sends packets to both boards containing timestamps, whenever a slave receives a packet it schedules a thread to wake-up before the specified timestamp, in order to trigger a pulse at the timestamp.

For this to work, the three devices need of course to have good time synchronization.

PTP Synchronization

On both server and boards make sure ntp is not running:

shuttle:~$ systemctl stop ntp

On both server and boards compile linuxptp from sources (versions from package managers are often outdated):

shuttle:~$ wget https://sourceforge.net/projects/linuxptp/files/v3.1/linuxptp-3.1.1.tgz
shuttle:~$ tar -xvf linuxptp-3.1.1.tgz
shuttle:~$ cd linuxptp-3.1.1
shuttle:~$ make
shuttle:~$ sudo make install

On server:

shuttle:~$ sudo su -
root:~# chrt -f 97 ptp4l -H -i <interface-to-Olimex1> --step_threshold=1 -m &> ptp4l_interface0.log &
root:~# pmc -u -b 0 -i <interface-to-Olimex1> "SET GRANDMASTER_SETTINGS_NP clockClass 248 clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37 leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1 frequencyTraceable 0 timeSource 0xa0"
root:~# chrt -f 95 phc2sys -m -c <interface-to-Olimex1> -s CLOCK_REALTIME --step_threshold=1 -w &> phc2sys_interface0.log

On Olimex1:

olimex1:~$ sudo su -
root:~# chrt -f 97 ptp4l -s -S -i eth0 --step_threshold=1 -m &> ptp4l_eth0.log &

On server:

root:~# chrt -f 97 ptp4l -H -i <interface-to-Olimex2> --step_threshold=1 -m &> ptp4l_interface1.log &
root:~# pmc -u -b 0 -i <interface-to-Olimex2> "SET GRANDMASTER_SETTINGS_NP clockClass 248 clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37 leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1 frequencyTraceable 0 timeSource 0xa0"
root:~# chrt -f 95 phc2sys -m -c <interface-to-Olimex2> -s CLOCK_REALTIME --step_threshold=1 -w &> phc2sys_interface1.log

On Olimex2:

olimex2:~$ sudo su -
root:~# chrt -f 97 ptp4l -s -S -i eth0 --step_threshold=1 -m &> ptp4l_eth0.log &

Motor control

On each board:

root:~# echo "192" > /sys/class/gpio/export
root:~# echo "out" > /sys/class/gpio/gpio192/direction
root:~# echo "0" > /sys/class/gpio/gpio192/value
root:~# echo "193" > /sys/class/gpio/export
root:~# echo "out" > /sys/class/gpio/gpio193/direction
root:~# echo "0" > /sys/class/gpio/gpio193/value
olimex1:~$ git clone https://lab.nexedi.com/nexedi/tsn-rt-measures.git
olimex1:~$ cd tsn-rt-measures/src/motor-control/build
olimex1:~/tsn-rt-measures/src/motor-control/build$ make slave
olimex1:~/tsn-rt-measures/src/motor-control/build$ sudo ./slave -f eth0 -a1 -p98

On shuttle:

shuttle:~$ git clone https://lab.nexedi.com/nexedi/tsn-rt-measures.git
shuttle:~$ cd tsn-rt-measures/src/motor-control/build
shuttle:~/tsn-rt-measures/src/motor-control/build$ make master
shuttle:~/tsn-rt-measures/src/motor-control/build$ ./master -a1 -p97 -t4000 <interface-to-Olimex1> <Olimex1-ip> <interface-to-Olimex2> <Olimex2-ip>

The master program is interactive, and has three commands:

  • <RPS> : Set target rotation per seconds to <RPS> 
  • t<T> : Set acceleration to be T ms / rotation per second 
  • r : Reverse motor direction

The motors need 400 steps to perform a full rotation, so the maximum speed is around 5 RPS (unless XDP is used).

If you don't use GPIO PG0 and PG1 you can modify the macros GPIO_DIRECTION_NUMBER and GPIO_PULSE_NUMBER at the beginning of the slave program (tsn-rt-measures/src/motor-control/src/slave.c). To know which number to put, you can use this formula:

gpioNumber = (Port Letter - 'A') * 32 + pinNumber

(taken from this Olimex Blog post)

Photos and videos

 

 

Gears slowly spinning at x 0.12 speed

 

Gears spinning at 6RPM then coming to a stop at x 0.12 speed