You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 22:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Tue Dec 02, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1364059912720,1269cyclictest3056172sleep33
1364059912660,1262cyclictest1953202sleep33
1364059912450,1240cyclictest443437-21sleep3
136405992723,1cyclictest0-21swapper/33
136405991916,1cyclictest0-21swapper/33
13640599188,7cyclictest4116532sleep33
136405991815,1cyclictest0-21swapper/33
13640599176,7cyclictest337843-21turbostat3
136405991715,0cyclictest0-21swapper/33
13640599171,13cyclictest264188-21turbostat3
13640599166,7cyclictest3929302sleep33
13640599162,11cyclictest538693-21taskset3
136405991613,1cyclictest0-21swapper/33
13640599155,7cyclictest2899732sleep33
13640599155,7cyclictest1589742sleep33
13640599152,10cyclictest524232-21turbostat3
136405991513,1cyclictest0-21swapper/33
136405991513,0cyclictest0-21swapper/33
136405991513,0cyclictest0-21swapper/33
136405991513,0cyclictest0-21swapper/33
136405991512,1cyclictest0-21swapper/33
136405991512,1cyclictest0-21swapper/33
13640599151,11cyclictest556736-21taskset3
13640599151,11cyclictest188870-21taskset3
13640599151,10cyclictest207945-21taskset3
13640599150,11cyclictest612978-21taskset3
190898140,1rtkit-daemon0-21swapper/00
13640599147,4cyclictest621108-21taskset3
13640599145,7cyclictest457855-21taskset3
13640599144,8cyclictest378567-21turbostat3
13640599144,7cyclictest5978862sleep33
13640599141,9cyclictest2519002sleep33
136405991413,0cyclictest0-21swapper/33
136405991413,0cyclictest0-21swapper/33
136405991412,1cyclictest0-21swapper/33
136405991412,1cyclictest0-21swapper/33
136405991412,0cyclictest0-21swapper/33
136405991412,0cyclictest0-21swapper/33
136405991412,0cyclictest0-21swapper/33
136405991412,0cyclictest0-21swapper/33
136405991411,1cyclictest0-21swapper/33
136405991411,1cyclictest0-21swapper/33
13640599141,10cyclictest589517-21taskset3
13640599141,10cyclictest541108-21taskset3
13640599141,10cyclictest479582-21taskset3
13640599141,10cyclictest286047-21taskset3
13640599141,10cyclictest283126-21chrt3
13640599141,10cyclictest168802-21taskset3
6058742139,2sleep0523677-21kworker/0:2+kacpid0
2952452130,6sleep0295378-21sensors_temp0
190898130,1rtkit-daemon0-21swapper/00
13640599134,6cyclictest3765422sleep33
13640599131,9cyclictest557285-21taskset3
13640599131,9cyclictest508486-21taskset3
13640599131,9cyclictest409262-21taskset3
13640599131,9cyclictest338180-21taskset3
13640599131,9cyclictest338180-21taskset3
13640599131,9cyclictest322056-21taskset3
13640599131,9cyclictest268925-21taskset3
13640599131,9cyclictest220425-21taskset3
13640599131,9cyclictest1765522sleep33
13640599131,9cyclictest141582-21chrt3
136405991312,0cyclictest0-21swapper/33
136405991311,1cyclictest0-21swapper/33
13640599131,10cyclictest346209-21chrt3
136405991310,1cyclictest0-21swapper/33
136405991310,1cyclictest0-21swapper/33
3459262120,9sleep2346103-21grep2
3459262120,9sleep2346103-21grep2
2835712120,9sleep2283717-21df2
190898120,8rtkit-daemon546983-21ssh0
13640599125,5cyclictest3685672sleep33
13640599124,6cyclictest360074-21taskset3
13640599121,9cyclictest573045-21taskset3
13640599121,8cyclictest1890772sleep33
5218082110,8sleep1521910-21expr1
190898110,7rtkit-daemon586690-21strings1
190898110,7rtkit-daemon370413-21kworker/0:0+kacpid0
190898110,7rtkit-daemon310268-21cut0
190898110,6rtkit-daemon157518-21/usr/sbin/munin2
190898110,4rtkit-daemon667-21snmpd0
190898110,4rtkit-daemon337371-21ssh0
190898110,4rtkit-daemon142774-21munin-node0
190898110,1rtkit-daemon0-21swapper/00
1303242111,6sleep30-21swapper/33
6201932100,7sleep0620195-21ssh0
5967352101,6sleep0596743-21ssh0
5899592100,7sleep2590263-21cut2
5389282100,7sleep2538932-21ssh2
4846882101,6sleep1485663-21uname1
3952882102,4sleep1395325-21apt-get1
2886062100,7sleep1288714-21ntpq1
2644272101,5sleep0264460-21apt-get0
190898100,7rtkit-daemon667-21snmpd0
190898100,6rtkit-daemon385512-21ssh1
190898100,5rtkit-daemon475250-21ssh0
190898100,5rtkit-daemon450412-21ssh0
190898100,5rtkit-daemon257937-21munin-run1
1627932101,7sleep0162909-21/usr/sbin/munin0
587704290,6sleep0587711-21sh0
579740290,6sleep1579748-21ssh1
516871290,6sleep2517528-21cut2
435609291,4sleep0435648-21apt-get0
367999290,5sleep1368132-21python31
322260290,7sleep2322316-21cut2
322066291,4sleep0322103-21apt-get0
283279290,6sleep1283486-21latency_hist1
220432291,4sleep0220465-21apt-get0
210256290,7sleep0212362-21cut0
19089890,5rtkit-daemon667-21snmpd0
19089890,5rtkit-daemon432761-21/usr/sbin/munin0
19089890,5rtkit-daemon319-21systemd-udevd0
19089890,5rtkit-daemon289590-21diskmemload0
19089890,5rtkit-daemon0-21swapper/00
19089890,4rtkit-daemon503490-21kthreadcore0
19089890,4rtkit-daemon491642-21ssh0
19089890,4rtkit-daemon297899-21ssh0
19089890,3rtkit-daemon299-21systemd-journal0
628955280,5sleep2628963-21ssh2
622522280,5sleep1623457-21cut1
612808281,4sleep2612816-21ssh2
605941281,3sleep1605978-21apt-get1
571409280,6sleep1571417-21ssh1
564875280,1sleep10-21swapper/11
547722280,2sleep1547730-21ssh1
533522280,5sleep0534400-21cut0
524969281,4sleep2525209-21sed2
499284280,2sleep2499292-21ssh2
483801280,2sleep1483810-21ssh1
482403281,4sleep2482409-21ssh2
468580280,6sleep2469536-21/usr/sbin/munin2
458380281,4sleep2458388-21ssh2
411302280,1sleep10-21swapper/11
393519281,4sleep0393527-21ssh0
369965281,4sleep0369973-21ssh0
282454281,4sleep0282582-21grep0
260213281,4sleep00-21swapper/00
239299281,3sleep2239332-21apt-get2
231925280,4sleep1232050-21python31
19089880,5rtkit-daemon1928-21sendmail-mta2
19089880,4rtkit-daemon674-21lldpd1
19089880,4rtkit-daemon667-21snmpd2
19089880,4rtkit-daemon667-21snmpd1
19089880,4rtkit-daemon603147-21ntpq2
19089880,4rtkit-daemon438-21wpa_supplicant1
19089880,4rtkit-daemon421-21avahi-daemon1
19089880,4rtkit-daemon421-21avahi-daemon0
19089880,4rtkit-daemon239090-21/usr/sbin/munin1
19089880,4rtkit-daemon0-21swapper/00
19089880,1rtkit-daemon0-21swapper/11
19089880,1rtkit-daemon0-21swapper/11
19089880,1rtkit-daemon0-21swapper/00
19089880,1rtkit-daemon0-21swapper/00
170519281,4sleep20-21swapper/22
163833281,3sleep0163855-21apt-get0
142614281,4sleep0142774-21munin-node0
614153271,3sleep1614217-21fschecks_count1
565381271,3sleep0565533-21df0
499467271,3sleep1499475-21ssh1
475246270,5sleep1475254-21ssh1
449846270,1sleep1171rcu_preempt1
442626270,1sleep10-21swapper/11
427961270,4sleep2428699-21cut2
424562271,3sleep1424639-21ntp_states1
410760270,4sleep2410768-21ssh2
409506271,4sleep1299-21systemd-journal1
396772271,3sleep2398235-21kthreadcore2
393018270,1sleep20-21swapper/22
331913270,4sleep2333936-21tr2
331058270,4sleep1331911-21cut1
311192270,4sleep2311269-21ntp_states2
305643270,4sleep2305684-21awk2
233197270,4sleep2233396-21hddtemp_smartct2
216874270,5sleep1219014-21kthreadcore1
194269270,1sleep10-21swapper/11
19089870,4rtkit-daemon98210-21munin-node2
19089870,4rtkit-daemon674-21lldpd1
19089870,4rtkit-daemon674-21lldpd1
19089870,4rtkit-daemon674-21lldpd1
19089870,4rtkit-daemon671-21ntpd2
19089870,4rtkit-daemon671-21ntpd0
19089870,4rtkit-daemon667-21snmpd1
19089870,4rtkit-daemon473-21gmain2
19089870,4rtkit-daemon0-21swapper/00
19089870,3rtkit-daemon674-21lldpd1
19089870,3rtkit-daemon671-21ntpd1
19089870,3rtkit-daemon497518-21ntp_states0
19089870,3rtkit-daemon360804-40unattended-upgr0
19089870,3rtkit-daemon271387-21hddtemp_smartct0
19089870,3rtkit-daemon142774-21munin-node2
19089870,3rtkit-daemon142774-21munin-node1
19089870,3rtkit-daemon142774-21munin-node0
19089870,2rtkit-daemon169504-21users2
150121270,5sleep1144941-21/usr/sbin/munin1
598063260,3sleep1598271-21hddtemp_smartct1
595761260,3sleep10-21swapper/11
557670260,4sleep2557955-21irqcore2
555425260,1sleep20-21swapper/22
534401260,4sleep1536362-21cut1
501852260,3sleep1503351-21cut1
490480260,4sleep2490485-21sh2
459861260,2sleep2459896-21apt-get2
420856260,4sleep2422423-21kthreadcore2
419067261,3sleep20-21swapper/22
385105260,1sleep00-21swapper/00
377472261,3sleep2377480-21ssh2
354002260,1sleep20-21swapper/22
344519260,1sleep10-21swapper/11
344519260,1sleep10-21swapper/11
322040260,1sleep10-21swapper/11
321279260,1sleep20-21swapper/22
313316260,4sleep1313324-21ssh1
305039260,3sleep1305047-21ssh1
289931261,2sleep20-21swapper/22
253358261,2sleep00-21swapper/00
245994260,3sleep2246158-21irqcore2
245637260,3sleep0245793-21diskstats0
233980261,2sleep00-21swapper/00
227744260,3sleep0228920-21cut0
220618260,3sleep2220733-21perl2
214219260,3sleep0214365-21df0
207909261,2sleep20-21swapper/22
19089860,3rtkit-daemon671-21ntpd1
19089860,3rtkit-daemon671-21ntpd1
190475260,3sleep0191608-21cut0
184758260,3sleep0186793-21kthreadcore0
182537260,1sleep00-21swapper/00
582987250,2sleep20-21swapper/22
573820251,2sleep20-21swapper/22
555713250,3sleep1555721-21ssh1
546143251,1sleep20-21swapper/22
532153250,2sleep1532161-21ssh1
508442250,2sleep20-21swapper/22
508435250,3sleep10-21swapper/11
457789250,2sleep1457793-21ssh1
449504250,3sleep20-21swapper/22
435532250,1sleep20-21swapper/22
434118251,2sleep10-21swapper/11
384704250,3sleep20-21swapper/22
376052251,2sleep00-21swapper/00
368849250,3sleep2368857-21ssh2
346587250,2sleep10-21swapper/11
290256250,3sleep1289932-21/usr/sbin/munin1
282247250,3sleep2282387-21ntpq2
272185250,2sleep20-21swapper/22
241649250,3sleep0243743-21cut0
226926251,1sleep20-21swapper/22
220178251,2sleep20-21swapper/22
201303250,3sleep2201304-21sshd2
197031250,3sleep0199047-21kthreadcore0
182827251,1sleep20-21swapper/22
176546250,1sleep20-21swapper/22
157257250,1sleep00-21swapper/00
156908250,1sleep10-21swapper/11
138361250,2sleep1140394-21cut1
131782250,2sleep00-21swapper/00
620137240,2sleep20-21swapper/22
575309241,1sleep00-21swapper/00
500571241,1sleep20-21swapper/22
466485240,2sleep1466493-21ssh1
314054240,2sleep10-21swapper/11
271413240,2sleep10-21swapper/11
266828240,2sleep20-21swapper/22
258173240,1sleep20-21swapper/22
257691240,1sleep20-21swapper/22
207302240,3sleep289-1kworker/2:1H+kblockd2
203076240,2sleep0204183-21kthreadcore0
163211240,3sleep10-21swapper/11
146397240,2sleep0146433-21kthreadcore0
189319230,1sleep20-21swapper/22
170206230,1sleep00-21swapper/00
130516230,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional