You are here: Home / Projects / 
2017-02-26 - 09:14

About the OSADL QA farm

Network monitoring - Real-time Ethernet test bench - Real-time host and kvm virtualization

During the development of a PREEMPT_RT patch series of a Linux kernel version, OSADL runs a number of tests to ensure such kernels are ready for production purposes. This QA activity includes monitored stress tests to detect

  • kernel crashes
  • memory leaks
  • driver misbehavior
  • unsatisfying worst-case latency
  • etc.

on a large number of different UP/SMP and 32/64-bit processors. A related PREEMPT_RT version is not released as "Latest Stable", unless all systems under test pass these quality checks. The systems are hosted in a QA farm on OSADL Test Racks in several OSADL testing labs. A part of this rack space is reserved for customized CPU boards that are tested continuously against every newly built kernel. This is a special service for OSADL member companies and also for non-members who may then "freeze" a particular kernel version in shipped systems but still follow the kernel development. Whenever a newly built kernel fails to run on such test systems or shows unsatisfying real-time behavior, a notification is sent out. Because of the close temporal relationship between the introduction of the bug and the notification, the underlying problem can be located easily. Should it become necessary to upgrade the "frozen" kernel of production systems, the required kernel version is already available and can be deployed without major upgrade hassles.

Test Racks #1 to #5 run, at least partly, popular systems the results of which may be interesting for a larger audience. They are, therefore, presented at the following pages. Some more systems have been added to create a wide variablility of different architectures (x86, ARM, PowerPC, MIPS), processor topologies (UP, single-socket SMP, multi-socket SMP), bit length (32, 64), memory size (64 MByte to 12 GByte), storage media (hard disk, SSD), storage arrays (raw, software RAID, hardware RAID) and distros (Arago, CentOS, ElinOS, Fedora, PTXdist, RAYS, Ubuntu). The data are obtained via a direct link to one of the OSADL testing labs and are updated continuously:

This testing activity is also intended to generate part of the data required for Safety Critical Linux.

Networking monitoring

The OSADL QA Farm uses the OpenNMS Networking Monitoring System. It was extended with a number of LLDP-based and other discovery scripts to allow for monitoring embedded systems that OpenNMS normally cannot handle. The main purpose of using OpenNMS is to provide an overall display of the topology of all systems. It allows for monitoring system availability in real-time as an important prerequisite to quickly react to any irregularity and ensure, whenever possible, that long-term stability measurements are not interrupted.

Click on the below button to display an example screenshot of the OpenNMS-provided topology overview that at the time of taking the picture included availability information of a total of 231 network peers.

Real-time Ethernet test bench

Generic UDP duplex link

Some of the test systems (rack #1, slot #2 and rack #3, slot #3) are equipped with a second Ethernet adapter to form a real-time network communication based on a peer-to-peer full duplex UDP link. The continuous recording of the round-trip time is available here, and the related latency plot here.

Powerlink kernel module master

Another system (rack #b, slot #5) is equipped with an additional stock Ethernet adapter and runs a Powerlink master stack on it. It is connected to a B&R Powerlink I/O node X20BC0083. In this, case we do not measure round-trip times, but the network traffic is sniffed continuously with a B&R Powerlink network analyzer X20ET8819, and the data are displayed graphically as in the other test setups (see Figure). The continuous recording is available here. The Open Source Powerlink master stack is developed, maintained and released by OSADL member Ethernet POWERLINK Standardization Group (EPSG). The source code as well as instructions to install and to use the stack can be obtained from the EPSG.

User-space software Ethercat master

Yet another system (rack #5, slot #1) is running a user-space software Ethercat master. It is connected to a Beckhoff EK1100 Ethercat coupler which, in turn, drives an EL2004 digital output terminal and listens to an EL1012 digital input terminal. The continuous recording is available here.

Hilscher netX master/slave link for various real-time Ethernet protocols

Some other test systems (rack #1, slot #1 and rack #1, slot #3) contain a Hilscher netX PCI (CIFX 50-RE) card that is loaded with various real-time Ethernet protocols such as Ethercat, Powerlink, Profinet and Sercos III.

Continuous rercording and latency plots

In some cases, round-trip times and cycle intervals are recorded and displayed continuously, e.g. Powerlink kernel module, user-space software Ethercat master and generic UDP link. In some other cases, regular 4-hour round-trip time measurements are made to determine the real-time Ethernet performance that can be achieved with Linux PREEMPT_RT systems. The resulting latency plots are also updated twice a day and uploaded to the Internet. Such round-trip time plots are available, for example, here.

Real-time host and kvm virtualization


There are a number of scenarios where is it highly desirable to run a virtualized guest system in a real-time compliant host computer. One scenario is the combination of an industrial control system with an integrated graphical user interface. In such situations, the host and the guest system have very different requirements with respect to real-time, safety and security. In addition, they often require different operating systems, since the graphical user interface is usually written for a proprietary tool chain which makes it impossible to port it to another environment - at least not in the short run. On the other hand, using a single combined computer system instead of two separate computers has a lot of advantages - less prone to hardware failures, easier system management, reduced purchase and service costs to just name a few.

The test setup

The most important prerequisite for such a combined control and graphics system is the continuous and unrestricted availability of real-time characteristics of the host system, irrespective of any action that may be taken by the guest system - including rebooting the guest system in case of a crash. One of the OSADL QA farm systems, therefore, regularly runs and benchmarks a virtualized Windows XP based guest system, while the host systems continues to execute monitored real-time tasks. Worst-case timer and wakeup latency of the host system, and load and uptime of the guest system are recorded in parallel and made available on-line.