You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 19:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Sat May 23, 2026 12:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
837021150,3sleep129-21ksoftirqd/108:57:391
837021150,3sleep129-21ksoftirqd/108:57:391
2541121080,2chrt25405-21gzip11:07:510
2541121080,2chrt25405-21gzip11:07:510
1542021070,2sleep10-21swapper/107:47:531
1542021070,2sleep10-21swapper/107:47:531
767821060,3sleep17681-21idleruntime11:47:481
767821060,3sleep17681-21idleruntime11:47:481
767821060,3sleep17681-21idleruntime11:47:481
290821060,2sleep12907-21gzip10:07:501
290821060,2sleep12907-21gzip10:07:501
3164921040,1sleep10-21swapper/109:57:531
3164921040,1sleep10-21swapper/109:57:531
3164921040,1sleep10-21swapper/109:57:531
264182990,4sleep11682-21cyclictest11:12:341
264182990,4sleep11682-21cyclictest11:12:341
226232980,3sleep30-21swapper/309:32:553
226232980,3sleep30-21swapper/309:32:553
226232980,3sleep30-21swapper/309:32:553
254312870,3sleep30-21swapper/308:14:403
254312870,3sleep30-21swapper/308:14:403
254312870,3sleep30-21swapper/308:14:403
149392750,3sleep30-21swapper/307:47:473
149392750,3sleep30-21swapper/307:47:473
108422710,4chrt10843-21cat11:57:331
108422710,4chrt10843-21cat11:57:331
108422710,4chrt10843-21cat11:57:331
59652680,3sleep30-21swapper/308:47:573
59652680,3sleep30-21swapper/308:47:573
154922610,4sleep115493-21needreboot12:07:501
154922610,4sleep115493-21needreboot12:07:501
123182510,1sleep112412-21fschecks_time09:07:451
123182510,1sleep112412-21fschecks_time09:07:451
123182510,1sleep112412-21fschecks_time09:07:451
1692995026,22cyclictest6352-21cron08:52:331
1692995026,22cyclictest6352-21cron08:52:331
211712450,2sleep30-21swapper/312:22:513
211712450,2sleep30-21swapper/312:22:513
44262420,5sleep1281ktimers/111:37:551
44262420,5sleep1281ktimers/111:37:551
92142410,3sleep30-21swapper/308:57:533
92142410,3sleep30-21swapper/308:57:533
1686994113,26cyclictest15649-21fschecks_time10:42:460
1686994113,26cyclictest15649-21fschecks_time10:42:460
1686994017,21cyclictest19857-21fschecks_count09:27:470
1686994012,6cyclictest15097-21seq10:41:420
260502390,1sleep125946-21/usr/sbin/munin12:37:461
260502390,1sleep125946-21/usr/sbin/munin12:37:461
260502390,1sleep125946-21/usr/sbin/munin12:37:461
1686993918,19cyclictest0-21swapper/012:17:480
1686993918,19cyclictest0-21swapper/012:17:480
1686993918,19cyclictest0-21swapper/008:02:330
1686993918,19cyclictest0-21swapper/008:02:330
1686993915,22cyclictest1682-21cyclictest09:07:540
1686993915,22cyclictest1682-21cyclictest09:07:540
1686993915,22cyclictest1682-21cyclictest09:07:540
1686993913,3cyclictest0-21swapper/011:42:480
1686993913,3cyclictest0-21swapper/011:42:480
1686993913,24cyclictest0-21swapper/009:12:580
1686993913,24cyclictest0-21swapper/009:12:580
1686993911,5cyclictest0-21swapper/010:02:570
1686993911,5cyclictest0-21swapper/010:02:570
1686993911,5cyclictest0-21swapper/010:02:570
168699391,12cyclictest8513-21sed10:22:510
168699391,12cyclictest8513-21sed10:22:510
1686993911,25cyclictest0-21swapper/007:32:380
1686993911,25cyclictest0-21swapper/007:32:380
1686993911,25cyclictest0-21swapper/007:32:380
1686993910,3cyclictest0-21swapper/012:02:490
1686993910,3cyclictest0-21swapper/012:02:490
1686993910,3cyclictest0-21swapper/012:02:490
168699389,4cyclictest0-21swapper/009:17:490
168699389,4cyclictest0-21swapper/009:17:490
168699389,3cyclictest0-21swapper/011:17:380
168699389,3cyclictest0-21swapper/011:17:380
1686993817,19cyclictest0-21swapper/011:37:460
1686993816,20cyclictest0-21swapper/011:02:390
1686993816,20cyclictest0-21swapper/011:02:390
1686993816,20cyclictest0-21swapper/008:12:490
1686993816,20cyclictest0-21swapper/008:12:490
1686993816,20cyclictest0-21swapper/008:12:490
1686993813,23cyclictest4784-21needreboot10:12:510
1686993813,23cyclictest4784-21needreboot10:12:510
1686993813,23cyclictest0-21swapper/011:27:460
168699381,11cyclictest22258-21seq12:27:430
168699381,11cyclictest22258-21seq12:27:430
74922370,2sleep07677-21/usr/sbin/munin11:47:480
74922370,2sleep07677-21/usr/sbin/munin11:47:480
74922370,2sleep07677-21/usr/sbin/munin11:47:480
1692993730,5cyclictest28374-21missed_timers08:22:481
1692993730,5cyclictest28374-21missed_timers08:22:481
1692993730,5cyclictest28374-21missed_timers08:22:481
1686993717,18cyclictest0-21swapper/008:22:470
1686993717,18cyclictest0-21swapper/008:22:470
1686993717,18cyclictest0-21swapper/008:22:470
1686993712,3cyclictest0-21swapper/008:02:500
1686993712,3cyclictest0-21swapper/008:02:500
1686993712,3cyclictest0-21swapper/008:02:500
1686993712,21cyclictest0-21swapper/012:32:440
1686993712,21cyclictest0-21swapper/012:32:440
1686993711,3cyclictest0-21swapper/008:32:560
1686993711,3cyclictest0-21swapper/008:32:560
168699371,12cyclictest32306-21/usr/sbin/munin11:27:450
168699371,11cyclictest29642-21memory09:52:500
168699371,11cyclictest29642-21memory09:52:500
1686993710,3cyclictest0-21swapper/012:12:440
1686993710,3cyclictest0-21swapper/012:12:440
64652360,3sleep30-21swapper/311:42:553
64652360,3sleep30-21swapper/311:42:553
225532360,3sleep30-21swapper/308:07:493
225532360,3sleep30-21swapper/308:07:493
225532360,3sleep30-21swapper/308:07:493
168699368,6cyclictest4134-21sshd08:43:210
168699368,6cyclictest4134-21sshd08:43:210
168699368,6cyclictest4134-21sshd08:43:210
1686993614,20cyclictest0-21swapper/010:47:500
1686993614,20cyclictest0-21swapper/010:47:500
1686993614,20cyclictest0-21swapper/010:47:500
1686993613,3cyclictest27167-21/usr/sbin/munin09:47:540
1686993613,3cyclictest27167-21/usr/sbin/munin09:47:540
1686993613,3cyclictest27167-21/usr/sbin/munin09:47:540
1686993613,3cyclictest1682-21cyclictest10:02:390
1686993613,3cyclictest1682-21cyclictest10:02:390
1686993613,3cyclictest1682-21cyclictest10:02:390
1686993612,22cyclictest0-21swapper/008:52:550
1686993612,22cyclictest0-21swapper/008:52:550
1686993611,5cyclictest11415-21gltestperf11:57:480
1686993611,5cyclictest11415-21gltestperf11:57:480
1686993611,5cyclictest11415-21gltestperf11:57:480
1686993611,3cyclictest0-21swapper/011:57:340
1686993611,3cyclictest0-21swapper/011:57:340
1686993611,3cyclictest0-21swapper/011:57:340
1686993611,3cyclictest0-21swapper/011:22:520
168699361,11cyclictest11689-21ntp_states07:38:070
168699361,11cyclictest11689-21ntp_states07:38:070
318612350,2sleep131866-21gltestperf08:32:471
318612350,2sleep131866-21gltestperf08:32:471
1692993529,4cyclictest9591-21latency07:32:481
1692993529,4cyclictest9591-21latency07:32:481
1692993529,4cyclictest545-21in:imuxsock11:22:341
1692993529,4cyclictest545-21in:imuxsock11:22:341
168699359,4cyclictest0-21swapper/010:17:570
168699359,4cyclictest0-21swapper/010:17:570
168699359,3cyclictest0-21swapper/011:17:460
168699359,3cyclictest0-21swapper/011:17:460
1686993513,20cyclictest0-21swapper/009:37:490
1686993513,20cyclictest0-21swapper/009:37:490
1686993511,22cyclictest0-21swapper/008:57:450
1686993511,22cyclictest0-21swapper/008:57:450
168699351,11cyclictest28992-21sed08:22:550
168699351,11cyclictest28992-21sed08:22:550
168699351,11cyclictest28992-21sed08:22:550
168699351,11cyclictest13396-21memory07:42:500
168699351,11cyclictest13396-21memory07:42:500
1686993510,3cyclictest0-21swapper/007:32:500
1686993510,3cyclictest0-21swapper/007:32:500
39762340,3chrt0-21swapper/108:42:561
39762340,3chrt0-21swapper/108:42:561
39762340,3chrt0-21swapper/108:42:561
184762340,3sleep30-21swapper/309:22:503
184762340,3sleep30-21swapper/309:22:503
184762340,3sleep30-21swapper/309:22:503
1686993412,20cyclictest0-21swapper/011:02:530
1686993412,20cyclictest0-21swapper/011:02:530
1686993412,20cyclictest0-21swapper/011:02:530
1686993411,21cyclictest0-21swapper/009:42:550
1686993411,21cyclictest0-21swapper/009:42:550
1686993411,21cyclictest0-21swapper/009:42:550
168699341,11cyclictest11100-21ntp_states09:02:520
168699341,11cyclictest11100-21ntp_states09:02:520
1686993410,4cyclictest0-21swapper/012:42:500
1686993410,4cyclictest0-21swapper/012:42:500
1686993410,3cyclictest0-21swapper/009:22:510
1686993410,3cyclictest0-21swapper/009:22:510
198832330,4sleep119882-21ntp_states10:52:531
198832330,4sleep119882-21ntp_states10:52:531
198832330,4sleep119882-21ntp_states10:52:531
1692993329,3cyclictest2595-21latency_hist08:42:331
1692993329,3cyclictest2595-21latency_hist08:42:331
1686993326,5cyclictest27667-21cron12:42:330
1686993326,5cyclictest27667-21cron12:42:330
1692993227,3cyclictest32451-21grep11:27:461
1692993227,3cyclictest32451-21grep11:27:461
1692993227,3cyclictest32451-21grep11:27:461
1692993225,5cyclictest17704-21threads12:12:531
1692993225,5cyclictest17704-21threads12:12:531
1686993227,3cyclictest20360-21unixbench_singl10:52:570
1686993227,3cyclictest20360-21unixbench_singl10:52:570
1686993227,3cyclictest20360-21unixbench_singl10:52:570
1686993211,2cyclictest0-21swapper/008:47:510
1686993211,2cyclictest0-21swapper/008:47:510
168699321,11cyclictest5794-21irqstats07:22:490
1686993210,3cyclictest0-21swapper/008:17:350
1686993210,3cyclictest0-21swapper/008:17:350
1686993210,3cyclictest0-21swapper/008:17:350
1692993125,5cyclictest16471-21kernelversion09:17:481
1692993125,5cyclictest16471-21kernelversion09:17:481
1692993125,4cyclictest28294-21latency_hist12:42:481
1692993125,4cyclictest28294-21latency_hist12:42:481
1692993125,4cyclictest28294-21latency_hist12:42:481
1692993125,4cyclictest28294-21latency_hist12:42:481
1692993125,4cyclictest1682-21cyclictest07:42:461
1692993125,4cyclictest1682-21cyclictest07:42:461
1692993124,5cyclictest27390-21cut09:47:471
1692993124,5cyclictest27390-21cut09:47:471
1692993124,5cyclictest27390-21cut09:47:471
168699317,3cyclictest0-21swapper/008:27:500
168699317,3cyclictest0-21swapper/008:27:500
168699317,3cyclictest0-21swapper/008:27:500
168699313,3cyclictest36350irq/48-eth010:07:540
168699313,3cyclictest36350irq/48-eth010:07:540
1686993110,3cyclictest0-21swapper/012:07:570
1686993110,3cyclictest0-21swapper/012:07:570
1692993026,3cyclictest29985-21gltestperf08:27:471
1692993026,3cyclictest29985-21gltestperf08:27:471
1692993026,3cyclictest29985-21gltestperf08:27:471
1692993023,6cyclictest25829-21kernelversion09:42:481
1692993023,6cyclictest25829-21kernelversion09:42:481
1692993023,6cyclictest25829-21kernelversion09:42:481
1686993025,3cyclictest0-21swapper/008:42:330
1686993025,3cyclictest0-21swapper/008:42:330
1692992925,3cyclictest1682-21cyclictest10:22:471
1692992925,3cyclictest1682-21cyclictest10:22:471
1692992923,5cyclictest6320-21threads07:22:561
1692992923,5cyclictest17049-21kernelversion07:52:481
1692992923,5cyclictest17049-21kernelversion07:52:481
1692992923,3cyclictest0-21swapper/109:02:521
1692992923,3cyclictest0-21swapper/109:02:521
1686992922,5cyclictest21876-21grep09:32:460
1686992922,5cyclictest21876-21grep09:32:460
1686992910,2cyclictest0-21swapper/010:33:380
1686992910,2cyclictest0-21swapper/010:33:380
1686992910,2cyclictest0-21swapper/010:33:380
1692992823,4cyclictest22581-21if_eth008:07:491
1692992823,4cyclictest22581-21if_eth008:07:491
1692992823,4cyclictest22581-21if_eth008:07:491
1692992822,4cyclictest25097-21ls09:42:331
1692992822,4cyclictest25097-21ls09:42:331
168699285,4cyclictest17237-21netstat07:52:510
168699285,4cyclictest17237-21netstat07:52:510
1692992723,3cyclictest11797-21ntp_offset11:57:531
1692992723,3cyclictest11797-21ntp_offset11:57:531
1692992723,3cyclictest11797-21ntp_offset11:57:531
1692992723,2cyclictest0-21swapper/109:12:491
1692992723,2cyclictest0-21swapper/109:12:491
1692992721,4cyclictest1346-21avahi-daemon08:07:561
1692992721,4cyclictest1346-21avahi-daemon08:07:561
1692992721,4cyclictest1346-21avahi-daemon08:07:561
1692992720,5cyclictest17557-21cut10:47:481
1692992720,5cyclictest17557-21cut10:47:481
1692992720,5cyclictest17557-21cut10:47:481
1692992719,6cyclictest19515-21ls12:17:551
1692992719,6cyclictest19515-21ls12:17:551
168699275,3cyclictest24790-21missed_timers12:32:500
168699275,3cyclictest24790-21missed_timers12:32:500
168699275,3cyclictest24790-21missed_timers12:32:500
1686992721,4cyclictest0-21swapper/011:27:560
1686992721,4cyclictest0-21swapper/011:27:560
1686992721,4cyclictest0-21swapper/011:27:560
1692992621,4cyclictest10885-21timerwakeupswit10:27:561
1692992621,4cyclictest10885-21timerwakeupswit10:27:561
1692992621,3cyclictest0-21swapper/109:31:351
1692992621,3cyclictest0-21swapper/109:31:351
1692992621,3cyclictest0-21swapper/109:31:351
1692992620,5cyclictest12138-21kernelversion10:32:501
1692992620,5cyclictest12138-21kernelversion10:32:501
1692992620,5cyclictest12138-21kernelversion10:32:501
1692992620,4cyclictest7763-21missed_timers07:27:501
1692992620,4cyclictest7763-21missed_timers07:27:501
1692992620,4cyclictest7763-21missed_timers07:27:501
1692992619,5cyclictest23010-21netstat12:27:501
1692992619,5cyclictest23010-21netstat12:27:501
1692992619,5cyclictest15168-21latency_hist10:42:461
1692992619,5cyclictest15168-21latency_hist10:42:461
1686992619,4cyclictest0-21swapper/010:47:330
1686992619,4cyclictest0-21swapper/010:47:330
1686992619,4cyclictest0-21swapper/010:47:330
1692992520,3cyclictest30602-21cut11:22:461
1692992520,3cyclictest30602-21cut11:22:461
1692992520,3cyclictest30602-21cut11:22:461
1692992519,5cyclictest596-21fschecks_time10:02:471
1692992519,5cyclictest596-21fschecks_time10:02:471
1692992519,5cyclictest596-21fschecks_time10:02:471
1692992519,5cyclictest29756-21netstat09:52:511
1692992519,5cyclictest29756-21netstat09:52:511
1692992519,5cyclictest22011-21threads10:57:551
1692992519,5cyclictest22011-21threads10:57:551
1692992519,5cyclictest14923-21sh12:07:451
1692992519,5cyclictest14923-21sh12:07:451
1692992519,4cyclictest3382-21latency_hist11:37:381
1692992519,4cyclictest3382-21latency_hist11:37:381
1692992518,5cyclictest25005-21switchtime08:12:571
1692992518,5cyclictest25005-21switchtime08:12:571
1692992518,5cyclictest25005-21switchtime08:12:571
1692992518,5cyclictest18568-21netstat09:22:511
1692992518,5cyclictest18568-21netstat09:22:511
1692992518,5cyclictest18568-21netstat09:22:511
1692992517,6cyclictest20473-21wc12:22:441
1692992517,6cyclictest20473-21wc12:22:441
1686992521,2cyclictest0-21swapper/007:47:470
1686992521,2cyclictest0-21swapper/007:47:470
1686992520,3cyclictest5614-21fschecks_count11:42:460
1686992520,3cyclictest5614-21fschecks_count11:42:460
1692992420,2cyclictest0-21swapper/111:17:341
1692992420,2cyclictest0-21swapper/111:17:341
1692992420,2cyclictest0-21swapper/111:07:441
1692992420,2cyclictest0-21swapper/111:07:441
1692992420,2cyclictest0-21swapper/111:07:441
1692992419,3cyclictest0-21swapper/109:32:511
1692992419,3cyclictest0-21swapper/109:32:511
1692992419,3cyclictest0-21swapper/109:32:511
1692992419,3cyclictest0-21swapper/107:57:541
1692992419,3cyclictest0-21swapper/107:57:541
1692992418,4cyclictest773-21apache210:13:371
1692992418,4cyclictest773-21apache210:13:371
1692992418,4cyclictest1346-21avahi-daemon08:22:531
1692992418,4cyclictest1346-21avahi-daemon08:22:531
1692992418,4cyclictest1346-21avahi-daemon08:22:531
1692992416,6cyclictest6061-21missed_timers11:42:491
1692992416,6cyclictest6061-21missed_timers11:42:491
1686992420,2cyclictest0-21swapper/009:32:450
170799231,17cyclictest18557-21chrt10:48:003
170799231,17cyclictest18557-21chrt10:48:003
170799231,17cyclictest18557-21chrt10:48:003
1692992318,3cyclictest0-21swapper/110:42:481
1692992318,3cyclictest0-21swapper/110:42:481
1692992318,3cyclictest0-21swapper/110:42:481
1692992317,5cyclictest6064-21cpuspeed10:17:461
1692992317,5cyclictest6064-21cpuspeed10:17:461
1686992318,3cyclictest0-21swapper/010:27:520
1686992318,3cyclictest0-21swapper/010:27:520
1692992217,3cyclictest0-21swapper/110:42:341
1692992217,3cyclictest0-21swapper/107:37:451
1692992217,3cyclictest0-21swapper/107:37:451
1686992215,4cyclictest0-21swapper/011:32:490
1699992110,8cyclictest17978-21chrt07:54:492
1699992110,8cyclictest17978-21chrt07:54:492
1692992117,2cyclictest0-21swapper/112:02:501
1686992116,3cyclictest0-21swapper/009:32:550
1686992116,3cyclictest0-21swapper/009:32:550
1686992116,3cyclictest0-21swapper/009:32:550
240192201,2sleep30-21swapper/311:02:573
240192201,2sleep30-21swapper/311:02:573
240192201,2sleep30-21swapper/311:02:573
170799201,14cyclictest1120-21chrt08:37:473
170799201,14cyclictest1120-21chrt08:37:473
169999209,8cyclictest18170-21chrt12:15:312
169999209,8cyclictest18170-21chrt12:15:312
1699992010,8cyclictest28657-21chrt09:50:222
1699992010,8cyclictest28657-21chrt09:50:222
1699992010,8cyclictest28657-21chrt09:50:222
1692992015,3cyclictest0-21swapper/108:57:551
1692992015,3cyclictest0-21swapper/108:57:551
169999199,8cyclictest13079-21chrt10:35:012
169999199,8cyclictest13079-21chrt10:35:012
169999199,8cyclictest13079-21chrt10:35:012
169999199,7cyclictest24266-21chrt11:04:192
169999199,7cyclictest24266-21chrt11:04:192
169999199,7cyclictest24266-21chrt11:04:192
169999199,7cyclictest22316-21chrt10:58:282
169999199,7cyclictest22316-21chrt10:58:282
169999199,7cyclictest12562-21chrt12:00:512
169999199,7cyclictest12562-21chrt12:00:512
169999199,7cyclictest12562-21chrt12:00:512
169999199,6cyclictest5577-21chrt10:15:032
169999199,6cyclictest5577-21chrt10:15:032
169999198,8cyclictest26208-21chrt11:10:222
169999198,8cyclictest26208-21chrt11:10:222
1699991914,4cyclictest351rcuc/208:21:142
1699991914,4cyclictest351rcuc/208:21:142
1699991914,4cyclictest351rcuc/208:21:142
1699991914,2cyclictest0-21swapper/207:33:022
1699991914,2cyclictest0-21swapper/207:33:022
124132190,3sleep20-21swapper/209:07:452
124132190,3sleep20-21swapper/209:07:452
124132190,3sleep20-21swapper/209:07:452
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional