You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-18 - 08:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Thu Jun 18, 2026 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3258899280,23cyclictest26218-21diskmemload21:50:552
3258899270,26cyclictest26218-21diskmemload23:18:442
3258899270,20cyclictest0-21swapper/223:10:012
32582992716,9cyclictest18556-21latency_hist23:05:000
32588992623,3cyclictest0-21swapper/200:27:572
32588992622,4cyclictest26218-21diskmemload23:54:172
32588992619,6cyclictest23853-21apt-get23:09:592
3258899260,25cyclictest19960-21ntp_states00:05:072
3258499260,12cyclictest0-21swapper/121:05:021
32592992523,1cyclictest0-21swapper/300:27:363
32592992522,2cyclictest24199-21sendmail00:40:003
3259299250,1cyclictest0-21swapper/321:10:543
3259299250,1cyclictest0-21swapper/321:10:543
32588992520,4cyclictest26218-21diskmemload21:35:442
32588992520,4cyclictest0-21swapper/223:36:132
32588992520,4cyclictest0-21swapper/222:06:532
32588992519,5cyclictest26218-21diskmemload00:15:502
32588992519,5cyclictest0-21swapper/223:57:002
32588992519,5cyclictest0-21swapper/200:24:592
3258899250,24cyclictest0-21swapper/223:02:012
3258899250,24cyclictest0-21swapper/222:42:342
3258899250,24cyclictest0-21swapper/221:40:102
3258899250,24cyclictest0-21swapper/200:35:412
3258899250,24cyclictest0-21swapper/200:04:282
3258899250,0cyclictest26218-21diskmemload21:23:372
3258899250,0cyclictest0-21swapper/223:25:022
3258499250,24cyclictest26218-21diskmemload23:33:351
3258299252,22cyclictest783-21snmpd22:22:530
32582992522,2cyclictest13-21ksoftirqd/023:45:000
3258299250,24cyclictest26218-21diskmemload23:33:250
3258299250,24cyclictest26218-21diskmemload21:17:530
3258299250,24cyclictest24740-21ssh21:39:280
3258299250,1cyclictest8315-21apt-get23:25:020
32592992422,1cyclictest0-21swapper/323:05:293
32592992421,2cyclictest8669-21munin-plugin-st22:55:013
32592992421,2cyclictest6782-21ssh23:23:353
32592992421,2cyclictest18574-21idleruntime-cro00:05:003
3259299240,24cyclictest0-21swapper/322:12:263
3259299240,23cyclictest0-21swapper/323:32:453
32588992420,4cyclictest0-21swapper/221:45:062
32588992420,3cyclictest26218-21diskmemload00:13:152
3258899240,24cyclictest0-21swapper/222:25:132
3258899240,23cyclictest3845-21ssh21:19:582
3258899240,23cyclictest18613-21perf20:55:002
3258899240,23cyclictest0-21swapper/222:03:302
3258899240,1cyclictest30828-21apt-get20:14:572
3258899240,1cyclictest26218-21diskmemload23:41:302
3258899240,0cyclictest0-21swapper/223:31:192
3258899240,0cyclictest0-21swapper/221:11:262
3258899240,0cyclictest0-21swapper/221:11:262
3258899240,0cyclictest0-21swapper/200:34:572
32584992422,1cyclictest1488-21ssh22:48:291
32584992422,1cyclictest0-21swapper/123:05:101
3258499240,23cyclictest6396-21sh22:22:091
3258499240,23cyclictest606-21dbus-daemon00:00:001
3258499240,23cyclictest30156-21grep23:45:001
3258499240,23cyclictest26218-21diskmemload22:14:311
3258499240,0cyclictest26218-21diskmemload21:10:431
3258499240,0cyclictest26218-21diskmemload21:10:431
32582992423,1cyclictest6106-21ssh21:20:190
3258299242,1cyclictest62550irq/127-eno122:56:220
3258299240,23cyclictest30779-21idleruntime22:15:010
3258299240,23cyclictest22626-21ssh22:37:270
3258299240,23cyclictest0-21swapper/023:52:460
32592992323,0cyclictest26218-21diskmemload00:32:283
32592992323,0cyclictest0-21swapper/323:35:473
32592992323,0cyclictest0-21swapper/323:10:433
32592992323,0cyclictest0-21swapper/323:01:203
32592992322,1cyclictest26218-21diskmemload22:41:173
32592992321,1cyclictest26218-21diskmemload23:55:003
32592992321,1cyclictest0-21swapper/321:48:263
32592992321,1cyclictest0-21swapper/320:00:003
32592992320,2cyclictest9878-21turbostat.cron21:25:013
32592992320,2cyclictest6083-21ssh21:51:053
3259299231,21cyclictest21559-21ssh22:05:243
3259299230,2cyclictest31738-21rm22:45:093
3259299230,2cyclictest31691-21ssh00:15:103
3259299230,23cyclictest0-21swapper/322:31:193
3259299230,23cyclictest0-21swapper/322:17:023
3259299230,22cyclictest31720-21munin-run20:15:003
3259299230,22cyclictest26854-21sh00:10:483
3259299230,22cyclictest21351-21sh22:35:393
3259299230,22cyclictest13500-21ssh22:29:473
3259299230,22cyclictest10141-21ssh22:55:323
3259299230,22cyclictest0-21swapper/300:05:333
3259299230,0cyclictest0-21swapper/321:37:153
32588992321,1cyclictest0-21swapper/222:59:052
32588992321,0cyclictest0-21swapper/222:35:342
32588992320,3cyclictest0-21swapper/222:20:182
32588992320,3cyclictest0-21swapper/222:16:292
32588992320,1cyclictest5605-21ssh22:51:452
32588992318,4cyclictest0-21swapper/222:31:122
3258899230,23cyclictest0-21swapper/222:45:032
3258899230,23cyclictest0-21swapper/221:27:352
3258899230,0cyclictest0-21swapper/223:45:542
32584992323,0cyclictest0-21swapper/123:22:341
32584992321,1cyclictest4593-21ssh23:50:171
32584992321,1cyclictest26218-21diskmemload21:41:231
32584992321,1cyclictest0-21swapper/122:15:401
32584992321,1cyclictest0-21swapper/121:27:091
3258499231,1cyclictest21431-21ssh22:35:461
3258499230,23cyclictest0-21swapper/121:52:141
3258499230,22cyclictest622-21gdbus20:20:001
3258499230,22cyclictest32306-21ntp_states21:15:051
3258499230,22cyclictest27480-21sh23:11:541
3258499230,22cyclictest26218-21diskmemload23:40:001
3258499230,22cyclictest26218-21diskmemload21:35:001
3258499230,22cyclictest18203-21ssh22:33:581
3258499230,22cyclictest0-21swapper/122:55:331
3258499230,22cyclictest0-21swapper/100:22:411
3258499230,1cyclictest23451-21tune2fs21:05:001
3258499230,1cyclictest0-21swapper/121:20:091
3258499230,0cyclictest0-21swapper/121:55:281
3258499230,0cyclictest0-21swapper/100:38:051
3258499230,0cyclictest0-21swapper/100:31:031
3258499230,0cyclictest0-21swapper/100:13:161
32582992323,0cyclictest0-21swapper/023:09:030
32582992323,0cyclictest0-21swapper/021:44:180
32582992323,0cyclictest0-21swapper/000:06:210
32582992322,1cyclictest62550irq/127-eno123:17:390
32582992321,1cyclictest26951-21sh00:10:550
32582992321,1cyclictest16426-21sh22:00:320
32582992321,1cyclictest14001-21apt-get21:59:580
32582992321,1cyclictest0-21swapper/023:20:050
32582992320,2cyclictest15857-21ssh00:00:550
3258299230,23cyclictest0-21swapper/021:48:260
3258299230,22cyclictest3789-21awk20:25:010
3258299230,22cyclictest26218-21diskmemload21:50:570
3258299230,22cyclictest24335-21sh22:09:320
3258299230,22cyclictest14493-21apt-get20:45:030
3258299230,22cyclictest0-21swapper/021:12:170
3258299230,22cyclictest0-21swapper/021:12:170
3258299230,21cyclictest20975-21ssh23:35:130
3258299230,0cyclictest26218-21diskmemload00:20:370
3258299230,0cyclictest0-21swapper/023:45:200
3258299230,0cyclictest0-21swapper/023:10:070
32592992222,0cyclictest26218-21diskmemload23:25:163
32592992220,1cyclictest27306-21ssh21:40:113
32592992220,1cyclictest13630-21idleruntime-cro20:45:013
32592992220,1cyclictest0-21swapper/322:21:273
32592992220,1cyclictest0-21swapper/321:25:073
3259299220,2cyclictest26489-21ssh23:40:323
3259299220,21cyclictest26717-21perf20:05:003
3259299220,21cyclictest26218-21diskmemload23:19:373
3259299220,21cyclictest26218-21diskmemload21:16:263
3259299220,21cyclictest15859-21awk20:50:003
3259299220,21cyclictest0-21swapper/322:02:193
3259299220,21cyclictest0-21swapper/319:54:593
3259299220,1cyclictest13970-21apt-get19:40:003
3259299220,1cyclictest11669-21ssh21:56:303
3259299220,1cyclictest0-21swapper/321:30:083
3259299220,1cyclictest0-21swapper/300:21:553
32588992220,1cyclictest8580-21perf20:35:012
32588992220,1cyclictest11298-21ssh21:56:012
3258899220,21cyclictest25402-21ssh22:10:012
3258899220,21cyclictest20425-21apt-get20:59:582
3258899220,1cyclictest0-21swapper/219:35:002
3258499224,17cyclictest24572-21/usr/sbin/munin20:00:001
32584992222,0cyclictest0-21swapper/123:28:051
32584992222,0cyclictest0-21swapper/122:00:311
32584992222,0cyclictest0-21swapper/100:26:371
32584992222,0cyclictest0-21swapper/100:05:431
32584992221,1cyclictest26218-21diskmemload21:38:391
32584992220,1cyclictest32178-21sh23:15:551
32584992220,1cyclictest20711-21sh22:05:051
32584992220,1cyclictest14487-21strings20:45:031
32584992220,1cyclictest0-21swapper/100:15:031
3258499220,22cyclictest26684-21ssh22:40:291
3258499220,22cyclictest0-21swapper/123:02:291
3258499220,21cyclictest20990-21perf21:00:001
3258499220,21cyclictest10391-21ssh22:25:111
3258499220,1cyclictest28435-21apt-get20:09:581
3258499220,1cyclictest17193-21perf19:45:011
3258499220,1cyclictest0-21swapper/121:45:551
32582992222,0cyclictest26218-21diskmemload21:30:460
32582992221,1cyclictest21177-21apt-get19:54:590
32582992220,1cyclictest31409-21ntp_states22:15:040
32582992220,1cyclictest16251-21ssh22:30:570
32582992220,1cyclictest14316-21expr19:40:010
3258299220,22cyclictest4124-21sh22:50:060
3258299220,22cyclictest0-21swapper/000:36:090
3258299220,21cyclictest26039-21apt-get20:04:570
3258299220,21cyclictest0-21swapper/000:27:510
3258299220,1cyclictest28263-21ssh22:42:480
3258299220,1cyclictest0-21swapper/023:55:010
3258299220,1cyclictest0-21swapper/000:15:090
3258299220,0cyclictest984-21rm22:47:410
32592992120,0cyclictest6780-21apt-get19:24:583
32592992119,1cyclictest23417-21fschecks_time21:05:003
3259299210,21cyclictest0-21swapper/320:09:583
3259299210,20cyclictest0-21swapper/319:19:593
32588992119,1cyclictest0-21swapper/219:40:012
3258899210,20cyclictest18780-21apt-get19:49:592
3258899210,1cyclictest15633-21apt-get20:50:002
32584992120,1cyclictest0-21swapper/100:00:091
3258499210,20cyclictest0-21swapper/119:14:581
3258299210,0cyclictest0-21swapper/000:30:140
32592992020,0cyclictest5631-21apt-get20:29:563
32592992019,1cyclictest0-21swapper/319:45:013
32588992020,0cyclictest21177-21apt-get19:54:592
32588992020,0cyclictest0-21swapper/221:04:582
32588992019,1cyclictest1968-21apt-get19:14:582
32588992019,0cyclictest16385-21apt-get19:44:572
32588992019,0cyclictest0-21swapper/220:19:592
32588992018,1cyclictest4508-21turbostat.cron19:20:002
32584992020,0cyclictest0-21swapper/120:29:561
32584992019,1cyclictest18031-21apt-get20:54:581
32584992018,1cyclictest0-21swapper/120:15:001
3258499200,20cyclictest0-21swapper/120:24:591
3258499200,0cyclictest0-21swapper/120:34:591
32582992020,0cyclictest0-21swapper/021:04:580
32582992020,0cyclictest0-21swapper/020:59:580
32582992018,1cyclictest0-21swapper/020:45:010
3258299200,20cyclictest0-21swapper/020:39:590
3258299200,19cyclictest0-21swapper/020:54:580
3258299200,0cyclictest0-21swapper/019:34:580
3259299190,0cyclictest0-21swapper/319:14:583
3258899190,0cyclictest0-21swapper/220:09:582
32584991919,0cyclictest0-21swapper/119:59:581
32584991919,0cyclictest0-21swapper/119:24:581
3258499190,1cyclictest1-21systemd19:20:001
32582991919,0cyclictest0-21swapper/019:14:580
32582991917,1cyclictest31721-21grep20:15:000
32592991818,0cyclictest0-21swapper/320:39:593
32592991816,1cyclictest606-21dbus-daemon21:10:003
32588991818,0cyclictest0-21swapper/220:24:592
32588991818,0cyclictest0-21swapper/219:24:582
32588991816,1cyclictest10892-21plymouth20:40:012
32582991816,1cyclictest29015-21perf20:10:000
32582991816,1cyclictest24323-21expr20:00:000
32592991717,0cyclictest0-21swapper/320:24:593
32592991717,0cyclictest0-21swapper/320:19:593
32592991716,0cyclictest8033-21apt-get20:34:593
32592991716,0cyclictest0-21swapper/319:49:593
32592991715,1cyclictest0-21swapper/319:35:003
32588991717,0cyclictest0-21swapper/220:04:572
32584991715,1cyclictest0-21swapper/119:40:011
32588991616,0cyclictest0-21swapper/219:59:582
32588991614,1cyclictest0-21swapper/221:05:022
32584991614,1cyclictest9988-21munin-run19:30:001
3258499160,15cyclictest10427-21apt-get20:39:591
32592991515,0cyclictest0-21swapper/320:54:583
32588991515,0cyclictest0-21swapper/220:29:562
3258899150,14cyclictest0-21swapper/220:40:182
32584991515,0cyclictest0-21swapper/120:44:581
32582991515,0cyclictest4366-21apt-get19:19:590
32582991515,0cyclictest0-21swapper/020:29:560
32582991515,0cyclictest0-21swapper/019:24:580
32582991513,1cyclictest8078-21ntp_states19:25:030
3259299140,14cyclictest0-21swapper/320:57:223
3259299140,14cyclictest0-21swapper/319:25:583
3258899140,14cyclictest0-21swapper/219:25:212
3258499140,14cyclictest0-21swapper/119:45:071
3258499140,0cyclictest0-21swapper/119:50:421
3258299140,14cyclictest0-21swapper/021:05:450
3258299140,14cyclictest0-21swapper/020:31:130
3258299140,14cyclictest0-21swapper/020:15:480
3258299140,14cyclictest0-21swapper/019:45:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional