You are here: Home / Projects / QA Farm Realtime / 
2018-02-23 - 19:48

What are latency plots?

Example of a latency plot
Example of a latency plot of a single-core system

Latency plots are used to graphically present the result of a latency determination of a real-time computer system. They are a special form of a frequency histogram with latency categories at the x scale and counts of samples that fall into a particular latency range at the y scale. A statistician would call it a distribution curve. It is often recommended to use linear scale divisions at the x scale and a logarithmic y scale. This makes it possible to graphically represent even very low frequencies and, particularly, the highest latency ever observed, since high latencies usually occur only rarely. The highest latency ever observed, the so-called "worst-case latency", represents the most important result of a latency plot.

The picture at the right shows an example of a latency plot gained on a single-core computer system. In this plot, the "worst-case latency" amounts to 55 ┬Ás. Click on the image to display it at the original resolution; this way, the stair-step-like pattern of the curve becomes clearly visible. The length of each step represents the size of every latency category and, thus, its resolution. The latency resolution of all OSADL latency plots is 1 microsecond.

How do we generate the latency plots?

The schedule of the OSADL QA Farm test procedures
The schedule of the OSADL QA Farm test procedures

The systems of the OSADL QA Farm undergo repeated periods of idle and load conditions as depicted in the figure at the left (click on the image or here to display it a full size) and as can be seen in the time course of continuous real-time monitoring. During the first 5 hours and 33 minutes of the two main load periods that start at 7:10 a.m. and 7:10 p.m., the cyclictest latency program is running. It creates a relatively high interrupt load but no other relevant load of the remaining system that may interfere with interrupt execution. At 7:20 a.m. and 7:20 p.m., the UnixBench CPU benchmark test is started which takes 30 to 60 minutes to complete depending on the numbers of cores of the CPU. The purpose of this CPU benchmark is twofold: First, the relatively high peak load that is generated during this benchmark is used to challenge the real-time capabilities of the systems. Second, the results are stored and made available to the public to facilitate the selection of an appropriate CPU for a particular project. By clicking on the header of a particular table column, the systems can be ordered according to the variable of this column (e.g. ordered by Dhrystones).

A second cyclictest run starts at 2:30 a.m. and 2:30 p.m., respectively, during which the gltestperf benchmark of the accelerated graphics processor (GPU) is executed. This benchmark was written by David Bucciarelli and is part of the mesa-demos package. Again, the purpose is to evaluate the influence of the graphics processor on the real-time performance of the entire system, if any, and to provide the benchmark results to assist system integrators when selecting CPU and GPU for a given project.

In the remaining time of the two main load periods, a standardized load generator is running in an attempt to represent an average machine control program. It generates about 10 Mbit/s network load, 1 MByte/s I/O and filesystem load and 1 MByte/s memory allocation load.

At every end of a cyclictest run, a latency plot is created, uploaded to the Internet and displayed here. Click on an item of the menu bar at the top of this page and select a system's rack and slot number to inspect its most recent latency distribution (e.g. the 2x2-core 64-bit x86 system in rack #1/slot #0). The latency plots are updated twice daily; the creation date is given at the bottom of every plot. Logged-in users with additional access rights can also display all data that have been stored since the beginning of the recording in May 2011. This allows to determine the worst-case latency with a much higher certainty than based on a single test run only. Access to the latency plots during accelerated graphics load as well as to a summary graph on the effect of accelerated graphics on the worst-case latency also requires additional user rights.

Below every latency plot, the original cyclictest command, the number of individual samples and the duration of the measurement are given.