You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-13 - 05:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Thu Aug 13, 2026 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2636199280,23cyclictest27588-21apt-get23:19:352
26361992722,4cyclictest20403-21diskmemload21:42:352
26361992721,5cyclictest29339-21apt-get19:19:342
2636199270,26cyclictest12941-21ssh22:00:002
2636199260,1cyclictest0-21swapper/222:24:352
2636099260,1cyclictest2791-21awk20:35:011
26361992521,4cyclictest20403-21diskmemload21:36:302
26361992520,4cyclictest0-21swapper/221:10:492
2636199250,3cyclictest31458-21grep22:19:392
2636199250,24cyclictest0-21swapper/221:04:342
2636099252,22cyclictest6589-21sendmail-msp23:40:011
2636099250,24cyclictest25610-21ssh21:43:501
2635999250,24cyclictest19377-21ls23:00:010
2636399240,23cyclictest21173-21ssh21:10:383
2636399240,1cyclictest28574-21ssh21:17:403
2636399240,1cyclictest20403-21diskmemload21:25:173
2636399240,1cyclictest19391-21sendmail-msp23:00:013
26361992420,3cyclictest0-21swapper/222:05:562
2636199240,23cyclictest20403-21diskmemload21:54:072
2636199240,23cyclictest0-21swapper/222:01:182
2636199240,0cyclictest0-21swapper/222:14:062
2636099240,23cyclictest0-21swapper/122:10:501
2635999240,24cyclictest0-21swapper/021:25:130
2635999240,23cyclictest17677-21ssh21:35:460
2635999240,23cyclictest0-21swapper/022:10:400
2635999240,0cyclictest20403-21diskmemload21:10:260
2635999240,0cyclictest0-21swapper/021:32:190
2635999240,0cyclictest0-21swapper/021:22:460
26363992323,0cyclictest20403-21diskmemload22:11:133
2636399231,21cyclictest0-21swapper/320:50:003
2636399230,22cyclictest3313-21sh21:24:143
2636399230,22cyclictest21390-21apt-get20:09:343
2636399230,22cyclictest20796-21munin-run20:05:003
2636399230,22cyclictest20403-21diskmemload22:06:193
2636399230,22cyclictest11176-21latency_hist19:45:013
2636399230,1cyclictest23278-21ssh21:40:403
2636399230,0cyclictest20403-21diskmemload21:37:163
2636399230,0cyclictest0-21swapper/322:04:433
2636399230,0cyclictest0-21swapper/321:58:003
2636399230,0cyclictest0-21swapper/321:32:003
26361992322,1cyclictest0-21swapper/221:25:082
26361992321,2cyclictest0-21swapper/221:15:062
2636199230,22cyclictest0-21swapper/221:25:012
2636199230,1cyclictest5136-21apt-get22:34:352
26360992323,0cyclictest0-21swapper/121:37:041
26360992323,0cyclictest0-21swapper/121:15:431
26360992322,1cyclictest20955-21sh21:10:221
26360992322,1cyclictest20403-21diskmemload22:02:531
26360992321,1cyclictest0-21swapper/121:27:321
26360992321,1cyclictest0-21swapper/120:05:001
2636099230,2cyclictest11196-21perf19:45:011
2636099230,22cyclictest0-21swapper/122:07:541
2636099230,1cyclictest0-21swapper/121:31:031
26359992321,1cyclictest4539-21munin-run22:30:000
26359992321,1cyclictest19651-21sh22:06:160
26359992320,2cyclictest3197-21ssh21:51:390
26359992320,2cyclictest13607-21ssh22:00:300
2635999230,2cyclictest29795-21sh21:46:150
2635999230,1cyclictest0-21swapper/021:43:330
26363992220,1cyclictest31127-21perf19:20:003
26363992219,2cyclictest15959-21apt19:55:003
2636399220,21cyclictest2781-21ls20:35:013
2636399220,21cyclictest2181-21ssh21:50:143
2636399220,21cyclictest12044-21apt-get23:54:343
2636399220,21cyclictest0-21swapper/321:46:573
2636399220,1cyclictest9384-21apt-get22:40:003
2636399220,1cyclictest0-21swapper/323:45:013
26361992221,0cyclictest0-21swapper/223:09:352
26361992220,1cyclictest0-21swapper/223:54:342
2636199220,21cyclictest31340-21apt-get00:34:372
2636199220,21cyclictest0-21swapper/220:59:362
2636199220,21cyclictest0-21swapper/200:09:402
2636199220,1cyclictest4523-21idleruntime-cro22:30:002
2636199220,1cyclictest0-21swapper/221:46:142
2636199220,1cyclictest0-21swapper/221:31:512
26360992222,0cyclictest0-21swapper/121:57:071
26360992220,1cyclictest0-21swapper/121:24:041
2636099220,22cyclictest0-21swapper/121:50:041
2636099220,21cyclictest313-21systemd-journal23:20:001
2636099220,21cyclictest28269-21apt-get21:45:021
2636099220,21cyclictest0-21swapper/122:44:351
2636099220,1cyclictest0-21swapper/120:00:001
26359992222,0cyclictest0-21swapper/022:19:330
26359992222,0cyclictest0-21swapper/021:56:050
26359992221,1cyclictest19999-21apt-get23:04:340
26359992220,1cyclictest24126-21apt-get00:19:360
2635999221,20cyclictest13435-21apt-get20:59:360
2635999220,21cyclictest29339-21apt-get19:19:340
2635999220,21cyclictest16256-21munin-plugin-st00:00:000
2635999220,21cyclictest15191-21apt-get22:54:340
2635999220,21cyclictest13585-21latency_hist19:50:010
2635999220,1cyclictest6983-21turbostat22:35:000
2635999220,0cyclictest0-21swapper/021:18:250
26363992120,1cyclictest31004-21apt-get20:29:353
26363992120,1cyclictest14595-21idleruntime-cro22:50:013
26363992120,0cyclictest0-21swapper/300:34:373
26363992119,1cyclictest1112-21dump-pmu-power19:25:003
2636399210,21cyclictest0-21swapper/320:14:343
2636399210,20cyclictest9651-21apt-get23:49:353
2636399210,20cyclictest28610-21apt-get20:24:353
2636399210,20cyclictest2386-21apt-get23:34:363
2636399210,20cyclictest0-21swapper/323:29:363
2636399210,0cyclictest0-21swapper/300:19:363
26361992120,1cyclictest0-21swapper/200:39:362
26361992119,1cyclictest9059-21awk23:45:012
26361992119,1cyclictest7587-21apt-get22:39:342
2636199210,20cyclictest26940-21apt-get19:14:342
2636199210,20cyclictest18996-21apt-get20:04:342
2636199210,1cyclictest868-21cat19:24:422
2636199210,0cyclictest0-21swapper/223:10:022
2636199210,0cyclictest0-21swapper/219:25:022
26360992121,0cyclictest0-21swapper/122:54:341
26360992120,0cyclictest0-21swapper/100:24:361
2636099210,20cyclictest0-21swapper/122:49:351
2636099210,20cyclictest0-21swapper/120:54:351
2636099210,20cyclictest0-21swapper/100:39:361
2636099210,20cyclictest0-21swapper/100:34:371
2636099210,0cyclictest0-21swapper/123:04:341
2636099210,0cyclictest0-21swapper/120:24:351
26359992120,0cyclictest0-21swapper/019:29:350
26363992020,0cyclictest22394-21apt-get23:09:353
26363992020,0cyclictest0-21swapper/300:29:353
26363992019,1cyclictest0-21swapper/322:55:013
26363992019,0cyclictest26184-21apt-get20:19:353
26363992018,1cyclictest23512-21perf00:15:003
26363992017,2cyclictest8788-21perf19:40:013
2636399200,0cyclictest0-21swapper/321:04:333
2636399200,0cyclictest0-21swapper/320:39:353
2636399200,0cyclictest0-21swapper/319:29:343
26361992020,0cyclictest16948-21apt-get00:04:362
26361992020,0cyclictest0-21swapper/220:45:012
2636199200,4cyclictest0-21swapper/223:24:412
2636199200,19cyclictest0-21swapper/220:44:362
2636199200,19cyclictest0-21swapper/220:29:442
2636199200,0cyclictest32401-21apt-get23:29:362
2636199200,0cyclictest0-21swapper/223:34:362
26360992020,0cyclictest31004-21apt-get20:29:351
26360992020,0cyclictest0-21swapper/123:29:361
26360992020,0cyclictest0-21swapper/123:14:341
26360992020,0cyclictest0-21swapper/120:49:341
26360992019,1cyclictest0-21swapper/120:19:351
26360992019,1cyclictest0-21swapper/119:39:341
26360992018,1cyclictest24195-21munin-run23:10:001
26360992017,2cyclictest23189-21munin-run20:10:001
2636099200,0cyclictest0-21swapper/119:24:331
26359992020,0cyclictest7587-21apt-get22:39:340
26359992020,0cyclictest0-21swapper/020:45:010
26359992020,0cyclictest0-21swapper/020:24:350
26359992020,0cyclictest0-21swapper/019:59:340
26359992020,0cyclictest0-21swapper/019:24:330
26359992019,1cyclictest0-21swapper/020:54:350
26359992018,1cyclictest14574-21sh22:50:010
2635999200,20cyclictest0-21swapper/023:54:340
2635999200,19cyclictest27986-21sendmail-msp20:20:000
2635999200,0cyclictest0-21swapper/020:09:340
2635999200,0cyclictest0-21swapper/019:44:350
26363991919,0cyclictest2738-21apt-get22:29:353
26363991919,0cyclictest0-21swapper/323:59:363
26363991919,0cyclictest0-21swapper/322:24:353
26363991919,0cyclictest0-21swapper/319:14:343
26363991918,1cyclictest0-21swapper/319:35:013
26361991919,0cyclictest0-21swapper/222:40:022
26361991919,0cyclictest0-21swapper/200:19:362
26361991912,6cyclictest28004-21munin-run20:20:002
2636199190,4cyclictest13058-21expr22:49:362
2636199190,1cyclictest0-21swapper/223:40:012
26360991919,0cyclictest0-21swapper/123:49:351
26360991919,0cyclictest0-21swapper/100:29:361
26360991917,1cyclictest14443-21apt-get23:59:361
26360991917,1cyclictest0-21swapper/123:34:361
26359991919,0cyclictest982-21apt-get20:34:380
26359991919,0cyclictest0-21swapper/023:49:350
26359991919,0cyclictest0-21swapper/000:09:400
26363991818,0cyclictest0-21swapper/320:54:343
2636399180,0cyclictest24174-21apt-get23:10:023
26361991818,0cyclictest0-21swapper/222:54:342
26361991818,0cyclictest0-21swapper/219:34:342
26360991817,0cyclictest4534-21apt-get19:34:341
26360991816,1cyclictest0-21swapper/123:00:011
26359991818,0cyclictest1320-21apt-get00:39:360
26359991818,0cyclictest0-21swapper/022:44:350
26359991818,0cyclictest0-21swapper/019:54:340
26359991817,1cyclictest18255-21apt-get21:05:020
26359991817,0cyclictest0-21swapper/020:14:340
26359991816,1cyclictest0-21swapper/019:40:010
26363991717,0cyclictest0-21swapper/323:04:343
26363991717,0cyclictest0-21swapper/300:24:363
2636399170,1cyclictest313-21systemd-journal20:45:003
26361991717,0cyclictest0-21swapper/220:09:342
26361991715,1cyclictest2756-21idleruntime-cro20:35:012
26361991715,1cyclictest17583-21apt-get22:59:342
2636199170,1cyclictest0-21swapper/223:49:402
26360991717,0cyclictest0-21swapper/123:54:341
26360991717,0cyclictest0-21swapper/123:24:361
26360991717,0cyclictest0-21swapper/122:39:341
26360991717,0cyclictest0-21swapper/120:59:351
2636099170,15cyclictest21850irq/141-nvme0q220:14:381
26359991717,0cyclictest31340-21apt-get00:34:370
26359991717,0cyclictest0-21swapper/023:44:350
26359991717,0cyclictest0-21swapper/023:29:360
26359991716,1cyclictest32754-21apt-get22:24:350
26359991716,1cyclictest30009-21apt-get23:24:360
26359991715,1cyclictest4193-21sleep23:35:010
26359991715,1cyclictest320-21perf20:30:010
26359991714,2cyclictest27966-21cstates23:19:360
2635999171,15cyclictest7635-21apt-get20:45:000
26363991616,0cyclictest0-21swapper/323:19:353
26363991616,0cyclictest0-21swapper/319:59:343
2636399160,1cyclictest6190-21ntp_states22:34:403
2636199164,12cyclictest0-21swapper/219:54:342
26361991616,0cyclictest0-21swapper/219:44:342
26361991615,0cyclictest21728-21apt-get00:14:352
26361991614,1cyclictest1-21systemd00:00:002
2636199160,14cyclictest0-21swapper/221:07:062
2636199160,0cyclictest0-21swapper/200:28:402
26360991615,0cyclictest0-21swapper/100:04:351
26360991614,1cyclictest15846-21apt-get21:04:331
26360991614,1cyclictest0-21swapper/120:40:011
26360991614,1cyclictest0-21swapper/119:54:351
26360991614,1cyclictest0-21swapper/100:19:401
26360991613,2cyclictest31078-21mii-tool22:19:361
26360991613,2cyclictest17628-21apt-get21:05:021
2636099160,15cyclictest0-21swapper/122:35:001
26359991616,0cyclictest0-21swapper/023:09:350
26359991616,0cyclictest0-21swapper/000:14:350
26359991615,1cyclictest0-21swapper/000:24:360
2635999160,15cyclictest28830-21munin-run19:15:000
26363991515,0cyclictest9384-21apt-get22:40:023
26363991515,0cyclictest18634-21apt-get21:09:343
26363991514,1cyclictest21129-21perf00:10:003
2636399150,14cyclictest0-21swapper/300:36:373
2636199152,6cyclictest28578-21/usr/sbin/munin20:24:362
26361991513,1cyclictest5189-21turbostat.cron20:40:012
26361991513,1cyclictest25570-21latency_hist20:15:012
2636199151,2cyclictest17964-21/usr/sbin/munin19:59:422
2636199150,14cyclictest27599-21ntp_states00:24:422
2636099155,9cyclictest600-21systemd-logind22:29:431
26360991514,1cyclictest0-21swapper/119:29:351
26360991513,1cyclictest7653-21munin-run20:45:001
26360991513,1cyclictest0-21swapper/122:24:351
26359991513,1cyclictest0-21swapper/000:04:370
2635999150,1cyclictest5687-21memory23:39:390
2635999150,1cyclictest0-21swapper/019:31:570
2635999150,15cyclictest0-21swapper/023:10:020
2636399140,14cyclictest0-21swapper/323:21:303
2636399140,14cyclictest0-21swapper/322:16:103
2636399140,14cyclictest0-21swapper/319:45:033
2636399140,14cyclictest0-21swapper/300:00:563
2636399140,0cyclictest0-21swapper/323:38:453
2636399140,0cyclictest0-21swapper/320:59:523
26361991413,1cyclictest0-21swapper/219:45:282
2636199140,14cyclictest0-21swapper/220:52:142
2636199140,14cyclictest0-21swapper/219:36:272
26360991414,0cyclictest0-21swapper/100:10:491
2636099140,14cyclictest0-21swapper/100:05:091
2636099140,0cyclictest0-21swapper/119:48:581
26359991414,0cyclictest0-21swapper/021:00:100
26359991414,0cyclictest0-21swapper/020:36:570
2635999140,14cyclictest0-21swapper/020:00:030
2635999140,14cyclictest0-21swapper/000:25:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional