You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-13 - 10:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sat Dec 13, 2025 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
835399240,23cyclictest7709-21apt20:19:132
835399240,23cyclictest32305-21sendmail-msp20:00:012
8349992422,1cyclictest28787-21munin-run00:20:011
8345992423,1cyclictest6568-21apt-get22:29:100
835399230,22cyclictest13975-21apt21:35:002
834999230,2cyclictest622-21gdbus20:10:011
834999230,22cyclictest25703-21cron00:10:011
834599230,22cyclictest32313-21awk20:00:010
834599230,22cyclictest0-21swapper/021:09:150
834599230,22cyclictest0-21swapper/020:34:100
834599230,1cyclictest0-21swapper/020:19:130
835699220,21cyclictest606-21dbus-daemon20:10:013
835699220,21cyclictest29750-21apt-get00:24:123
835699220,21cyclictest24513-21apt-get20:54:133
835699220,21cyclictest11943-21latency_hist20:25:013
835699220,21cyclictest0-21swapper/321:35:003
8353992222,0cyclictest0-21swapper/223:49:112
8353992222,0cyclictest0-21swapper/222:29:102
8353992220,1cyclictest8755-21idleruntime-cro21:25:002
835399220,22cyclictest11232-21munin-plugin-st23:40:012
835399220,21cyclictest28764-21cat22:05:012
835399220,21cyclictest22131-21apt-get21:54:092
835399220,21cyclictest2130-21apt-get00:34:132
835399220,21cyclictest16592-21apt-get23:54:112
835399220,21cyclictest15497-21sh19:25:012
835399220,21cyclictest1145-21apt-get23:20:022
835399220,21cyclictest0-21swapper/220:34:102
835399220,1cyclictest20779-21cron00:00:002
8349992220,1cyclictest4579-21apt-get23:29:091
8349992220,1cyclictest28768-21munin-run22:05:011
8349992220,1cyclictest20841-21munin-plugin-st00:00:001
8349992220,1cyclictest18488-21apt-get19:34:121
8349992220,1cyclictest13597-21perf22:40:011
8349992220,1cyclictest13597-21perf22:40:011
834999220,21cyclictest421-21apt-get20:04:121
834999220,21cyclictest2193-21apt-get23:24:151
834999220,21cyclictest16611-21apt-get22:45:021
834999220,21cyclictest0-21swapper/123:49:111
834999220,21cyclictest0-21swapper/121:14:131
834999220,1cyclictest0-21swapper/121:25:001
8345992220,1cyclictest0-21swapper/022:04:130
8345992220,1cyclictest0-21swapper/021:25:000
834599220,22cyclictest0-21swapper/023:49:110
834599220,21cyclictest3609-21awk22:20:000
834599220,21cyclictest15503-21latency_hist19:25:010
834599220,21cyclictest11230-21munin-run23:40:010
834599220,21cyclictest0-21swapper/023:44:100
834599220,21cyclictest0-21swapper/023:20:020
834599220,21cyclictest0-21swapper/023:15:020
834599220,1cyclictest13972-21perf21:35:000
834599220,1cyclictest1-21systemd00:00:000
8356992121,0cyclictest0-21swapper/323:59:103
8356992120,1cyclictest25701-21sh00:10:013
8356992120,1cyclictest21779-21apt-get21:50:023
8356992119,2cyclictest28796-21sendmail00:20:013
8356992118,2cyclictest8779-21latency_hist21:25:003
835699210,21cyclictest0-21swapper/323:14:103
835699210,20cyclictest21747-21apt-get22:59:093
835699210,20cyclictest0-21swapper/323:45:013
835699210,20cyclictest0-21swapper/322:09:103
835699210,20cyclictest0-21swapper/321:14:133
835699210,1cyclictest14534-21apt-get21:39:103
835699210,1cyclictest0-21swapper/321:05:013
835699210,0cyclictest0-21swapper/323:04:143
835699210,0cyclictest0-21swapper/319:19:133
835699210,0cyclictest0-21swapper/319:19:133
8353992120,1cyclictest20879-21apt-get19:39:112
8353992120,0cyclictest0-21swapper/221:19:132
8353992119,1cyclictest28384-21awk23:10:002
835399210,20cyclictest3604-21sendmail-msp22:20:002
835399210,20cyclictest1165-21munin-run23:20:012
835399210,20cyclictest0-21swapper/221:10:012
835399210,20cyclictest0-21swapper/219:14:132
8349992121,0cyclictest0-21swapper/122:44:131
8349992121,0cyclictest0-21swapper/120:44:131
8349992120,1cyclictest32109-21apt-get21:09:141
8349992120,1cyclictest0-21swapper/121:39:101
8349992120,0cyclictest0-21swapper/119:24:121
8349992119,1cyclictest0-21swapper/119:50:001
834999210,20cyclictest19342-21apt-get21:49:111
834999210,20cyclictest0-21swapper/121:19:131
834999210,20cyclictest0-21swapper/100:20:021
834999210,1cyclictest13628-21awk23:45:011
834999210,0cyclictest0-21swapper/121:04:111
834999210,0cyclictest0-21swapper/120:34:111
8345992121,0cyclictest0-21swapper/020:44:130
8345992120,1cyclictest9367-21apt-get22:34:090
8345992120,1cyclictest16592-21apt-get23:54:100
8345992120,1cyclictest0-21swapper/021:39:100
8345992120,0cyclictest19358-21apt-get22:54:100
834599210,1cyclictest29328-21apt-get22:09:100
834599210,0cyclictest0-21swapper/021:54:090
834599210,0cyclictest0-21swapper/000:34:130
8356992020,0cyclictest15998-21apt-get22:45:023
8356992020,0cyclictest14916-21apt-get20:34:113
8356992020,0cyclictest14197-21apt-get23:49:113
8356992020,0cyclictest0-21swapper/323:24:153
8356992020,0cyclictest0-21swapper/322:29:103
8356992020,0cyclictest0-21swapper/320:59:123
8356992020,0cyclictest0-21swapper/320:04:123
8356992020,0cyclictest0-21swapper/319:49:123
8356992020,0cyclictest0-21swapper/319:34:123
8356992019,1cyclictest0-21swapper/322:44:133
8356992019,1cyclictest0-21swapper/321:19:133
8356992018,1cyclictest4521-21apt-get00:39:103
8356992018,1cyclictest13696-21apt-get19:24:123
8356992018,1cyclictest13611-21awk22:40:013
8356992018,1cyclictest13611-21awk22:40:013
835699200,20cyclictest0-21swapper/320:49:133
835699200,0cyclictest0-21swapper/321:59:103
835699200,0cyclictest0-21swapper/319:39:123
8353992020,0cyclictest0-21swapper/222:59:092
8353992020,0cyclictest0-21swapper/220:44:132
8353992019,1cyclictest23899-21apt-get00:09:092
8353992019,1cyclictest0-21swapper/222:54:102
8353992019,0cyclictest0-21swapper/220:09:122
835399200,20cyclictest0-21swapper/220:04:122
835399200,0cyclictest0-21swapper/222:09:102
835399200,0cyclictest0-21swapper/221:39:102
8349992020,0cyclictest0-21swapper/123:34:101
8349992020,0cyclictest0-21swapper/122:29:101
8349992020,0cyclictest0-21swapper/122:24:091
8349992020,0cyclictest0-21swapper/120:14:121
8349992020,0cyclictest0-21swapper/119:54:131
8349992020,0cyclictest0-21swapper/119:39:111
8349992020,0cyclictest0-21swapper/100:39:101
8349992020,0cyclictest0-21swapper/100:34:121
8349992020,0cyclictest0-21swapper/100:29:101
8349992019,1cyclictest28947-21apt-get23:14:101
8349992019,1cyclictest24169-21apt-get23:04:141
8349992019,0cyclictest0-21swapper/121:59:101
8349992018,1cyclictest28710-21turbostat.cron21:00:011
8349992018,1cyclictest26315-21apt20:55:011
834999200,20cyclictest0-21swapper/121:44:101
834999200,20cyclictest0-21swapper/121:30:021
834999200,0cyclictest0-21swapper/121:54:091
8345992020,0cyclictest151rcu_preempt23:04:140
8345992020,0cyclictest0-21swapper/023:09:140
8345992020,0cyclictest0-21swapper/020:04:120
8345992020,0cyclictest0-21swapper/019:39:110
8345992020,0cyclictest0-21swapper/019:34:120
8345992020,0cyclictest0-21swapper/000:24:120
8345992020,0cyclictest0-21swapper/000:09:090
8345992019,1cyclictest10132-21apt-get20:24:140
834599200,20cyclictest0-21swapper/022:59:090
834599200,20cyclictest0-21swapper/000:39:100
8356991919,0cyclictest0-21swapper/321:49:113
8356991919,0cyclictest0-21swapper/320:19:133
8356991918,1cyclictest32142-21apt-get00:29:103
8356991918,1cyclictest19726-21apt-get20:44:133
8356991917,1cyclictest28755-21perf22:05:013
8353991919,0cyclictest0-21swapper/222:34:092
8353991919,0cyclictest0-21swapper/220:49:132
8353991919,0cyclictest0-21swapper/219:34:122
8353991919,0cyclictest0-21swapper/200:39:102
8353991919,0cyclictest0-21swapper/200:24:122
8353991918,1cyclictest11800-21apt-get23:44:102
8353991918,0cyclictest21505-21apt-get00:04:122
8353991918,0cyclictest0-21swapper/223:04:142
835399190,18cyclictest0-21swapper/219:44:132
8349991919,0cyclictest0-21swapper/122:19:131
8349991919,0cyclictest0-21swapper/100:14:101
8345991919,0cyclictest0-21swapper/022:14:130
8345991919,0cyclictest0-21swapper/000:29:100
8345991919,0cyclictest0-21swapper/000:04:120
8345991918,0cyclictest0-21swapper/021:00:020
8345991917,2cyclictest3970-21latency_hist21:15:010
834599190,18cyclictest0-21swapper/019:15:010
834599190,0cyclictest0-21swapper/019:44:130
8356991818,0cyclictest0-21swapper/321:44:103
8356991817,1cyclictest1707-21apt-get22:19:133
8356991817,0cyclictest0-21swapper/319:54:133
8353991818,0cyclictest0-21swapper/222:39:102
8353991818,0cyclictest0-21swapper/222:39:102
8353991818,0cyclictest0-21swapper/222:14:132
8353991818,0cyclictest0-21swapper/220:24:142
8353991818,0cyclictest0-21swapper/200:29:102
8353991817,1cyclictest4175-21apt-get22:24:102
8353991817,1cyclictest29358-21apt-get21:00:022
8353991816,1cyclictest3969-21turbostat.cron21:15:012
835399180,17cyclictest0-21swapper/220:59:122
8349991818,0cyclictest0-21swapper/122:09:091
8349991817,0cyclictest0-21swapper/123:39:091
8345991818,0cyclictest12517-21apt-get20:29:110
8345991818,0cyclictest0-21swapper/000:14:100
8345991817,1cyclictest14179-21apt-get22:44:130
8345991817,1cyclictest0-21swapper/019:29:130
8345991816,1cyclictest313-21systemd-journal23:30:010
8345991815,2cyclictest13582-21turbostat.cron22:40:010
8345991815,2cyclictest13582-21turbostat.cron22:40:010
834599180,1cyclictest26903-21apt-get20:59:120
834599180,17cyclictest0-21swapper/022:24:100
8356991717,0cyclictest0-21swapper/323:34:103
8356991717,0cyclictest0-21swapper/323:29:093
8356991717,0cyclictest0-21swapper/322:24:093
8356991717,0cyclictest0-21swapper/321:09:143
8356991717,0cyclictest0-21swapper/320:14:123
8356991717,0cyclictest0-21swapper/319:44:133
8356991717,0cyclictest0-21swapper/300:34:123
8356991716,1cyclictest26293-21apt-get00:14:103
8353991717,0cyclictest0-21swapper/220:29:112
8353991717,0cyclictest0-21swapper/200:14:102
8353991716,0cyclictest0-21swapper/220:54:132
8353991715,1cyclictest6385-21sh23:30:012
835399170,0cyclictest151rcu_preempt19:55:002
8349991717,0cyclictest31724-21apt-get22:14:131
8349991717,0cyclictest21747-21apt-get22:59:101
8349991716,1cyclictest0-21swapper/100:04:121
8345991717,0cyclictest0-21swapper/020:54:130
8345991717,0cyclictest0-21swapper/020:39:130
8345991716,1cyclictest5315-21apt-get20:14:120
8345991715,1cyclictest0-21swapper/019:49:110
8356991614,1cyclictest0-21swapper/323:39:093
835699160,15cyclictest19780-21diskstats22:54:123
8353991616,0cyclictest16091-21apt-get19:29:132
8353991616,0cyclictest0-21swapper/222:44:132
8353991616,0cyclictest0-21swapper/220:39:132
8353991614,1cyclictest25684-21apt-get19:49:112
8349991616,0cyclictest0-21swapper/123:54:101
8349991616,0cyclictest0-21swapper/123:09:131
8349991616,0cyclictest0-21swapper/120:24:131
8345991616,0cyclictest0-21swapper/021:19:130
8345991615,0cyclictest0-21swapper/021:49:110
8345991614,1cyclictest2566-21date20:05:010
834599160,2cyclictest30850-21cut23:15:010
8356991514,1cyclictest9015-21apt-get22:30:023
8356991514,1cyclictest0-21swapper/300:04:123
8356991513,1cyclictest8889-21apt-get19:14:133
835699150,14cyclictest0-21swapper/323:06:593
835699150,14cyclictest0-21swapper/320:25:233
8353991514,0cyclictest9355-21apt-get21:29:132
8349991515,0cyclictest7710-21apt-get20:19:121
8349991515,0cyclictest0-21swapper/122:30:021
8349991515,0cyclictest0-21swapper/120:29:111
8349991515,0cyclictest0-21swapper/119:59:121
8349991513,2cyclictest25089-21idleruntime-cro19:45:001
834999150,1cyclictest0-21swapper/122:54:121
834999150,0cyclictest0-21swapper/123:15:021
8345991515,0cyclictest0-21swapper/021:29:130
8356991414,0cyclictest0-21swapper/323:50:073
835699140,14cyclictest0-21swapper/322:10:093
835699140,14cyclictest0-21swapper/321:25:433
835699140,14cyclictest0-21swapper/319:55:143
835699140,0cyclictest0-21swapper/323:15:583
835699140,0cyclictest0-21swapper/319:27:163
8353991414,0cyclictest0-21swapper/222:45:132
8353991414,0cyclictest0-21swapper/219:15:302
8353991414,0cyclictest0-21swapper/219:15:302
835399140,14cyclictest0-21swapper/223:30:252
835399140,14cyclictest0-21swapper/223:12:412
835399140,14cyclictest0-21swapper/221:41:282
835399140,14cyclictest0-21swapper/220:10:182
835399140,14cyclictest0-21swapper/200:15:052
835399140,0cyclictest0-21swapper/221:55:012
835399140,0cyclictest0-21swapper/221:47:202
8349991414,0cyclictest0-21swapper/120:35:171
834999140,14cyclictest0-21swapper/120:45:221
834999140,14cyclictest0-21swapper/119:25:161
834999140,0cyclictest0-21swapper/119:15:141
834999140,0cyclictest0-21swapper/119:15:141
8345991414,0cyclictest0-21swapper/021:40:360
8345991414,0cyclictest0-21swapper/019:15:130
8345991414,0cyclictest0-21swapper/019:15:130
834599140,14cyclictest0-21swapper/023:30:250
834599140,14cyclictest0-21swapper/000:15:040
834599140,0cyclictest0-21swapper/022:45:100
834599140,0cyclictest0-21swapper/021:55:040
834599140,0cyclictest0-21swapper/019:50:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional