You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-24 - 12:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Fri Jul 24, 2026 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1539321010,2sleep30-21swapper/319:28:523
1539321010,2sleep30-21swapper/319:28:523
210832980,2sleep30-21swapper/322:38:433
210832980,2sleep30-21swapper/322:38:433
210832980,2sleep30-21swapper/322:38:433
107212800,3sleep20-21swapper/223:38:402
107212800,3sleep20-21swapper/223:38:402
107212800,3sleep20-21swapper/223:38:402
292022690,1sleep30-21swapper/300:28:293
292022690,1sleep30-21swapper/300:28:293
292022690,1sleep30-21swapper/300:28:293
186532690,9sleep0201rcuc/023:58:430
186532690,9sleep0201rcuc/023:58:430
186532690,9sleep0201rcuc/023:58:430
224662680,2sleep122469-21sed21:13:541
224662680,2sleep122469-21sed21:13:541
224662680,2sleep122469-21sed21:13:541
302282650,3chrt0-21swapper/320:08:523
302282650,3chrt0-21swapper/320:08:523
231942560,2sleep123188-21tail22:43:471
231942560,2sleep123188-21tail22:43:471
202112550,3sleep20-21swapper/219:43:432
202112550,3sleep20-21swapper/219:43:432
19472550,2sleep20-21swapper/200:38:542
19472550,2sleep20-21swapper/200:38:542
19472550,2sleep20-21swapper/200:38:542
29752540,3sleep19134-21cyclictest21:48:521
29752540,3sleep19134-21cyclictest21:48:521
187602540,2chrt18763-21grep19:38:471
187602540,2chrt18763-21grep19:38:471
240022450,4chrt0-21swapper/219:53:442
240022450,4chrt0-21swapper/219:53:442
46052420,2sleep30-21swapper/320:28:423
46052420,2sleep30-21swapper/320:28:423
46052420,2sleep30-21swapper/320:28:423
46052420,2sleep30-21swapper/320:28:423
168102420,2chrt16780-21missed_timers19:33:451
168102420,2chrt16780-21missed_timers19:33:451
96292410,3chrt0-21swapper/222:08:432
96292410,3chrt0-21swapper/222:08:432
96292410,3chrt0-21swapper/222:08:432
23842410,2chrt0-21swapper/323:13:523
23842410,2chrt0-21swapper/323:13:523
23842410,2chrt0-21swapper/323:13:523
913799394,29cyclictest19826-21latency_hist00:03:451
913799394,29cyclictest19826-21latency_hist00:03:451
913799394,29cyclictest19826-21latency_hist00:03:451
113482390,3sleep30-21swapper/323:38:453
250732380,3sleep30-21swapper/300:13:543
250732380,3sleep30-21swapper/300:13:543
111902380,2sleep30-21swapper/320:43:533
111902380,2sleep30-21swapper/320:43:533
111902380,2sleep30-21swapper/320:43:533
53912370,1sleep05575-21/usr/sbin/munin20:28:520
53912370,1sleep05575-21/usr/sbin/munin20:28:520
53912370,1sleep05575-21/usr/sbin/munin20:28:520
53912370,1sleep05575-21/usr/sbin/munin20:28:520
9137993628,6cyclictest23882-21fschecks_time19:53:431
9137993628,6cyclictest23882-21fschecks_time19:53:431
913699361,29cyclictest21348-21users00:03:540
913699361,29cyclictest21348-21users00:03:540
913699361,29cyclictest21348-21users00:03:540
9136993527,6cyclictest9926-21kernelversion22:08:450
9136993527,6cyclictest9926-21kernelversion22:08:450
9136993527,6cyclictest9926-21kernelversion22:08:450
34542350,3sleep20-21swapper/223:18:422
34542350,3sleep20-21swapper/223:18:422
9137993425,7cyclictest4571-21gltestperf20:28:421
9137993425,7cyclictest4571-21gltestperf20:28:421
9137993425,7cyclictest4571-21gltestperf20:28:421
9137993425,7cyclictest4571-21gltestperf20:28:421
9136993327,5cyclictest24122-21kernelversion19:53:450
9136993327,5cyclictest24122-21kernelversion19:53:450
9136993326,5cyclictest22230-21threads21:13:530
9136993326,5cyclictest22230-21threads21:13:530
9136993326,5cyclictest22230-21threads21:13:530
159222330,2sleep116019-21/usr/sbin/munin20:58:441
159222330,2sleep116019-21/usr/sbin/munin20:58:441
114252330,1sleep111448-21/usr/sbin/munin23:38:481
114252330,1sleep111448-21/usr/sbin/munin23:38:481
9137993226,4cyclictest19879-40mandb00:03:431
9137993226,4cyclictest19879-40mandb00:03:431
9137993226,4cyclictest19879-40mandb00:03:431
913699329,4cyclictest171ktimers/022:56:560
913699329,4cyclictest171ktimers/022:56:560
913699329,4cyclictest171ktimers/022:56:560
9136993227,3cyclictest9134-21cyclictest20:12:210
9136993227,3cyclictest9134-21cyclictest20:12:210
9136993226,5cyclictest7267-21latency_hist22:03:290
9136993226,5cyclictest7267-21latency_hist22:03:290
9136993225,5cyclictest14911-21unixbench_multi20:53:530
9136993225,5cyclictest14911-21unixbench_multi20:53:530
9136993225,5cyclictest13537-21switchtime23:43:530
9136993225,5cyclictest13537-21switchtime23:43:530
9136993225,5cyclictest13537-21switchtime23:43:530
9136993124,5cyclictest28990-21sshd00:25:540
9136993124,5cyclictest28990-21sshd00:25:540
9136993124,5cyclictest28990-21sshd00:25:540
294072310,2sleep10-21swapper/120:08:421
294072310,2sleep10-21swapper/120:08:421
9137993026,3cyclictest27160-21sendmail_mailtr22:53:511
9137993026,3cyclictest27160-21sendmail_mailtr22:53:511
9137993026,3cyclictest27160-21sendmail_mailtr22:53:511
9137993024,4cyclictest9134-21cyclictest00:21:041
9137993024,4cyclictest9134-21cyclictest00:21:041
9137993024,4cyclictest0-21swapper/120:03:291
9137993024,4cyclictest0-21swapper/120:03:291
9137993022,6cyclictest15986-21latency_hist19:33:301
9137993022,6cyclictest15986-21latency_hist19:33:301
9137993021,7cyclictest22224-21cut00:08:451
9137993021,7cyclictest22224-21cut00:08:451
9137993021,7cyclictest22224-21cut00:08:451
913699308,3cyclictest201rcuc/022:30:290
913699308,3cyclictest201rcuc/022:30:290
9136993024,4cyclictest10468-21cron23:38:290
9136993024,4cyclictest10468-21cron23:38:290
9136993024,4cyclictest10468-21cron23:38:290
9136993024,3cyclictest0-21swapper/022:38:410
9136993024,3cyclictest0-21swapper/022:38:410
913699301,12cyclictest201rcuc/021:05:590
913699301,12cyclictest201rcuc/021:05:590
9137992923,4cyclictest13074-21gzip19:23:501
9137992919,8cyclictest19186-21latency_hist21:08:331
9137992919,8cyclictest19186-21latency_hist21:08:331
913699299,6cyclictest171ktimers/023:09:500
913699299,6cyclictest171ktimers/023:09:500
913699298,4cyclictest171ktimers/019:44:160
913699298,4cyclictest171ktimers/019:44:160
913699294,8cyclictest16-21ksoftirqd/020:08:390
913699294,8cyclictest16-21ksoftirqd/020:08:390
9136992923,5cyclictest24625-21fschecks_time22:48:420
9136992923,5cyclictest24625-21fschecks_time22:48:420
9136992923,5cyclictest24625-21fschecks_time22:48:420
9136992923,4cyclictest171ktimers/019:24:500
9136992922,6cyclictest31778-21kernelversion00:33:440
9136992922,6cyclictest31778-21kernelversion00:33:440
9136992922,6cyclictest31778-21kernelversion00:33:440
9136992922,5cyclictest23356-21sshd19:52:540
9136992922,5cyclictest23356-21sshd19:52:540
9136992922,5cyclictest16753-21latency_hist22:28:330
9136992922,5cyclictest16753-21latency_hist22:28:330
9136992921,6cyclictest27067-21cut21:28:430
9136992921,6cyclictest27067-21cut21:28:430
9136992921,6cyclictest27067-21cut21:28:430
9136992921,6cyclictest19271-21unixbench_singl19:38:520
9136992921,6cyclictest19271-21unixbench_singl19:38:520
9136992919,7cyclictest671-21needreboot21:43:480
9136992919,7cyclictest671-21needreboot21:43:480
9136992919,7cyclictest671-21needreboot21:43:480
9136992917,10cyclictest0-21swapper/020:43:410
9136992917,10cyclictest0-21swapper/020:43:410
9136992917,10cyclictest0-21swapper/020:43:410
9136992912,3cyclictest90-1pr/ttyS020:13:520
9136992912,3cyclictest90-1pr/ttyS020:13:520
112892290,2sleep10-21swapper/122:13:401
112892290,2sleep10-21swapper/122:13:401
112892290,2sleep10-21swapper/122:13:401
9137992823,3cyclictest1571-21timerandwakeup20:18:501
9137992823,3cyclictest1571-21timerandwakeup20:18:501
9137992822,5cyclictest3127-21netstat20:23:451
9137992822,5cyclictest3127-21netstat20:23:451
9137992821,5cyclictest749-21runrttasks22:13:451
9137992821,5cyclictest749-21runrttasks22:13:451
9137992821,5cyclictest749-21runrttasks22:13:451
9137992821,5cyclictest18964-21fschecks_count22:33:421
9137992821,5cyclictest15538-21latency_hist20:58:361
9137992821,5cyclictest15538-21latency_hist20:58:361
913699286,6cyclictest201rcuc/023:40:080
913699286,6cyclictest201rcuc/023:40:080
913699284,13cyclictest7609-21taskset20:35:500
913699284,13cyclictest7609-21taskset20:35:500
913699284,13cyclictest26946-21seq20:01:090
913699284,13cyclictest26946-21seq20:01:090
9136992823,3cyclictest0-21swapper/021:53:490
9136992823,3cyclictest0-21swapper/021:53:490
9136992821,5cyclictest1542-21timerandwakeup20:18:500
9136992821,5cyclictest1542-21timerandwakeup20:18:500
9136992820,6cyclictest22661-21fschecks_count22:43:430
9136992820,6cyclictest22661-21fschecks_count22:43:430
9136992816,2cyclictest181rcu_preempt00:19:280
9136992816,2cyclictest181rcu_preempt00:19:280
9136992814,8cyclictest749-21runrttasks19:36:030
9136992814,8cyclictest749-21runrttasks19:36:030
9136992813,3cyclictest201rcuc/022:22:240
9136992813,3cyclictest201rcuc/022:22:240
9136992813,3cyclictest201rcuc/022:22:240
322532280,3sleep20-21swapper/200:33:512
322532280,3sleep20-21swapper/200:33:512
322532280,3sleep20-21swapper/200:33:512
9137992722,4cyclictest32174-21timerwakeupswit20:13:511
9137992722,4cyclictest32174-21timerwakeupswit20:13:511
9137992722,3cyclictest9134-21cyclictest00:26:311
9137992722,3cyclictest9134-21cyclictest00:26:311
9137992722,3cyclictest9134-21cyclictest00:26:311
9137992720,5cyclictest9134-21cyclictest21:23:481
9137992720,5cyclictest9134-21cyclictest21:23:481
913699277,6cyclictest16-21ksoftirqd/023:28:520
913699277,6cyclictest16-21ksoftirqd/023:28:520
913699277,6cyclictest16-21ksoftirqd/023:28:520
913699275,11cyclictest21933-21taskset22:39:040
913699275,11cyclictest21933-21taskset22:39:040
913699275,11cyclictest21933-21taskset22:39:040
913699274,12cyclictest13261-21seq20:50:580
913699274,12cyclictest13261-21seq20:50:580
9136992722,3cyclictest0-21swapper/021:23:300
9136992722,3cyclictest0-21swapper/021:23:300
9136992722,3cyclictest0-21swapper/021:23:300
9136992722,3cyclictest0-21swapper/000:13:290
9136992722,3cyclictest0-21swapper/000:13:290
9136992722,3cyclictest0-21swapper/000:13:290
9136992722,2cyclictest201rcuc/020:26:100
9136992722,2cyclictest201rcuc/020:26:100
9136992722,2cyclictest171ktimers/019:19:120
9136992721,4cyclictest28992-21systemctl22:58:510
9136992721,4cyclictest28992-21systemctl22:58:510
9136992721,4cyclictest28992-21systemctl22:58:510
9136992720,2cyclictest181rcu_preempt23:14:240
9136992720,2cyclictest181rcu_preempt23:14:240
9136992720,2cyclictest181rcu_preempt23:14:240
9136992716,3cyclictest181rcu_preempt00:38:460
9136992716,3cyclictest181rcu_preempt00:38:460
9136992716,3cyclictest181rcu_preempt00:38:460
130612270,2chrt13062-21latency_hist23:43:451
130612270,2chrt13062-21latency_hist23:43:451
130612270,2chrt13062-21latency_hist23:43:451
97712260,2chrt9773-21date20:43:300
97712260,2chrt9773-21date20:43:300
9137992620,5cyclictest16819-21kernelversion23:53:451
9137992620,5cyclictest16819-21kernelversion23:53:451
9137992620,4cyclictest14355-21unixbench_multi22:18:541
9137992620,4cyclictest14355-21unixbench_multi22:18:541
9137992620,4cyclictest14355-21unixbench_multi22:18:541
9137992620,4cyclictest0-21swapper/123:08:541
9137992620,4cyclictest0-21swapper/123:08:541
9137992619,5cyclictest17404-21kernelversion22:28:441
9137992617,7cyclictest9081-21threads20:38:511
9137992617,7cyclictest9081-21threads20:38:511
9137992617,7cyclictest7190-21threads20:33:511
9137992617,7cyclictest7190-21threads20:33:511
913699268,3cyclictest171ktimers/021:08:490
913699268,3cyclictest171ktimers/021:08:490
913699267,3cyclictest201rcuc/022:13:550
913699267,3cyclictest201rcuc/022:13:550
913699267,3cyclictest201rcuc/022:13:550
913699266,3cyclictest0-21swapper/022:03:530
913699266,3cyclictest0-21swapper/022:03:530
913699266,3cyclictest0-21swapper/022:03:530
913699263,14cyclictest25300-21taskset00:16:410
913699263,14cyclictest25300-21taskset00:16:410
9136992620,5cyclictest16681-21timerwakeupswit20:58:520
9136992620,5cyclictest16681-21timerwakeupswit20:58:520
9136992620,4cyclictest5782-21ntp_kernel_err23:23:480
9136992620,4cyclictest5782-21ntp_kernel_err23:23:480
9136992620,4cyclictest5782-21ntp_kernel_err23:23:480
9136992618,2cyclictest181rcu_preempt19:30:520
9136992618,2cyclictest181rcu_preempt19:30:520
9137992521,2cyclictest0-21swapper/123:08:301
9137992521,2cyclictest0-21swapper/123:08:301
9137992521,2cyclictest0-21swapper/122:33:511
9137992521,2cyclictest0-21swapper/122:33:511
9137992520,4cyclictest545-21in:imuxsock21:43:311
9137992520,4cyclictest545-21in:imuxsock21:43:311
9137992520,4cyclictest545-21in:imuxsock21:43:311
9137992519,5cyclictest27350-21kernelversion21:28:451
9137992519,5cyclictest27350-21kernelversion21:28:451
9137992519,5cyclictest27350-21kernelversion21:28:451
9137992519,5cyclictest24481-21needreboot00:13:481
9137992519,5cyclictest24481-21needreboot00:13:481
9137992519,4cyclictest0-21swapper/121:43:541
9137992519,4cyclictest0-21swapper/121:43:541
9137992519,4cyclictest0-21swapper/121:43:541
9137992519,4cyclictest0-21swapper/119:53:311
9137992519,4cyclictest0-21swapper/119:53:311
9137992518,5cyclictest28600-21kernelversion22:58:451
9137992518,5cyclictest28600-21kernelversion22:58:451
9137992518,5cyclictest28600-21kernelversion22:58:451
9137992518,5cyclictest13503-21latency_hist20:53:301
9137992518,5cyclictest13503-21latency_hist20:53:301
913699252,14cyclictest31893-21taskset21:39:380
913699252,14cyclictest31893-21taskset21:39:380
913699252,14cyclictest31893-21taskset21:39:380
9136992519,4cyclictest31075-21timerwakeupswit23:03:540
9136992519,4cyclictest31075-21timerwakeupswit23:03:540
9136992519,4cyclictest15172-21ntp_kernel_err23:48:460
9136992519,4cyclictest15172-21ntp_kernel_err23:48:460
9136992517,2cyclictest181rcu_preempt23:55:100
9136992517,2cyclictest181rcu_preempt23:55:100
9136992516,3cyclictest181rcu_preempt21:27:040
9136992516,3cyclictest181rcu_preempt21:27:040
81682250,3chrt0-21swapper/322:03:453
81682250,3chrt0-21swapper/322:03:453
81682250,3chrt0-21swapper/322:03:453
307002250,1sleep0201rcuc/000:28:550
307002250,1sleep0201rcuc/000:28:550
124032250,3sleep20-21swapper/220:48:472
124032250,3sleep20-21swapper/220:48:472
9137992419,4cyclictest6116-21kernelversion21:58:441
9137992419,4cyclictest6116-21kernelversion21:58:441
9137992419,3cyclictest0-21swapper/123:38:421
9137992419,3cyclictest0-21swapper/123:38:421
9137992419,3cyclictest0-21swapper/123:38:421
9137992419,3cyclictest0-21swapper/100:38:451
9137992419,3cyclictest0-21swapper/100:38:451
9137992419,3cyclictest0-21swapper/100:38:451
913799241,8cyclictest297442gltestperf00:28:431
913799241,8cyclictest297442gltestperf00:28:431
9137992418,4cyclictest6254-21unixbench_multi23:23:531
9137992418,4cyclictest6254-21unixbench_multi23:23:531
9137992418,4cyclictest6254-21unixbench_multi23:23:531
9137992418,4cyclictest21197-21latency22:38:441
9137992418,4cyclictest21197-21latency22:38:441
9137992418,4cyclictest21197-21latency22:38:441
9137992418,3cyclictest0-21swapper/122:48:531
9137992418,3cyclictest0-21swapper/122:48:531
9137992418,3cyclictest0-21swapper/122:48:531
9137992417,5cyclictest15482-21timerandwakeup23:48:501
9137992417,5cyclictest15482-21timerandwakeup23:48:501
9137992416,6cyclictest29807-21ls21:33:511
9137992416,6cyclictest29807-21ls21:33:511
9137992415,7cyclictest4384-21unixbench_singl23:18:511
9137992415,7cyclictest4384-21unixbench_singl23:18:511
913699243,4cyclictest171ktimers/021:36:480
913699243,4cyclictest171ktimers/021:36:480
9136992415,7cyclictest4111-21sort23:18:500
9136992415,7cyclictest4111-21sort23:18:500
5372240,2sleep1538-21cut00:38:321
5372240,2sleep1538-21cut00:38:321
5372240,2sleep1538-21cut00:38:321
913999231,13cyclictest199302sleep300:03:343
913999231,13cyclictest199302sleep300:03:343
913999231,13cyclictest199302sleep300:03:343
9137992318,4cyclictest9134-21cyclictest20:43:431
9137992318,4cyclictest9134-21cyclictest20:43:431
9137992318,4cyclictest9134-21cyclictest20:43:431
9137992318,3cyclictest0-21swapper/123:13:431
9137992318,3cyclictest0-21swapper/123:13:431
9137992318,3cyclictest0-21swapper/123:13:431
9137992317,4cyclictest7886-21switchtime23:28:511
9137992317,4cyclictest7886-21switchtime23:28:511
9137992317,4cyclictest7886-21switchtime23:28:511
9137992317,3cyclictest0-21swapper/122:28:391
9137992317,3cyclictest0-21swapper/122:28:391
9137992316,5cyclictest0-21swapper/121:08:521
9137992316,5cyclictest0-21swapper/121:08:521
9136992316,6cyclictest3153-21vmstat21:48:550
9136992316,6cyclictest3153-21vmstat21:48:550
9139992216,3cyclictest318392sleep300:33:443
9139992216,3cyclictest318392sleep300:33:443
9139992216,3cyclictest318392sleep300:33:443
9137992216,4cyclictest0-21swapper/119:23:381
9137992215,5cyclictest23574-21kernelversion21:18:461
9137992215,5cyclictest23574-21kernelversion21:18:461
9137992215,5cyclictest23574-21kernelversion21:18:461
913899214,11cyclictest0-21swapper/200:03:452
913899214,11cyclictest0-21swapper/200:03:452
913899214,11cyclictest0-21swapper/200:03:452
9137992116,4cyclictest9134-21cyclictest21:57:361
9137992116,4cyclictest9134-21cyclictest21:57:361
9137992116,3cyclictest0-21swapper/122:03:511
9137992116,3cyclictest0-21swapper/122:03:511
9137992116,3cyclictest0-21swapper/122:03:511
9137992116,3cyclictest0-21swapper/119:43:501
9137992116,3cyclictest0-21swapper/119:43:501
9137992115,4cyclictest0-21swapper/120:08:321
9137992115,4cyclictest0-21swapper/120:08:321
9139992014,4cyclictest42342sleep323:18:513
9139992014,4cyclictest42342sleep323:18:513
913999193,9cyclictest0-21swapper/320:18:543
913999193,9cyclictest0-21swapper/320:18:543
913899193,9cyclictest0-21swapper/223:58:542
913899193,9cyclictest0-21swapper/223:58:542
913899193,9cyclictest0-21swapper/223:58:542
913999181,12cyclictest31191-21taskset23:03:553
913999181,12cyclictest31191-21taskset23:03:553
913999181,11cyclictest13594-21chrt22:18:453
913999181,11cyclictest13594-21chrt22:18:453
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional