You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 13:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Mon Nov 24, 2025 00:44:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
688799240,1cyclictest0-21swapper/220:10:002
6880992422,1cyclictest31320-21perf20:00:000
6880992421,2cyclictest19191-21wc00:00:000
6892992323,0cyclictest0-21swapper/319:52:183
6892992322,0cyclictest0-21swapper/300:07:183
689299230,22cyclictest18967-21apt-get20:42:203
6887992323,0cyclictest0-21swapper/220:02:192
688799230,2cyclictest21640-21apt19:39:592
688799230,22cyclictest0-21swapper/200:22:202
6885992321,1cyclictest12054-21munin-plugin-st19:20:011
6885992321,1cyclictest10210-21munin-run21:30:001
6885992320,2cyclictest6083-21apt-get20:15:001
688599230,1cyclictest1281-21cron20:05:011
6880992323,0cyclictest26156-21apt-get20:57:190
6880992320,2cyclictest542-21munin-run21:10:000
6880992320,2cyclictest23322-21ls20:50:010
688099230,1cyclictest32394-21apt-get23:22:190
6892992222,0cyclictest0-21swapper/320:22:193
6892992220,1cyclictest19196-21perf00:00:003
689299220,21cyclictest1956-21perf23:25:013
689299220,21cyclictest10054-21apt-get19:17:183
689299220,21cyclictest0-21swapper/300:37:223
689299220,1cyclictest0-21swapper/319:30:003
6887992220,1cyclictest766-21snmpd23:52:232
6887992220,1cyclictest0-21swapper/223:45:002
688799220,21cyclictest27035-21munin-plugin-st22:05:002
688799220,21cyclictest0-21swapper/223:22:182
688799220,1cyclictest12270-21apt22:40:002
6885992220,1cyclictest26878-21apt-get19:52:191
6885992220,1cyclictest19235-21turbostat.cron19:35:001
6885992219,2cyclictest31875-21perf22:15:001
688599220,21cyclictest31299-21idleruntime-cro20:00:001
688599220,21cyclictest22647-21apt-get21:57:201
688599220,21cyclictest1863-21apt-get22:20:021
688599220,21cyclictest17846-21apt-get21:47:181
688599220,21cyclictest1-21systemd00:00:001
688599220,21cyclictest103082sleep122:37:171
688599220,1cyclictest9630-21idleruntime-cro19:15:011
688599220,1cyclictest31940-21sh23:20:011
688599220,1cyclictest27458-21apt-get22:07:191
688599220,1cyclictest21893-21apt-get23:00:021
688599220,1cyclictest0-21swapper/100:25:001
6880992220,1cyclictest9577-21sendmail-msp23:40:000
6880992220,1cyclictest6104-21perf20:14:590
688099220,21cyclictest2383-21apt-get23:27:200
688099220,21cyclictest14797-21apt-get23:52:220
688099220,21cyclictest0-21swapper/022:57:210
688099220,21cyclictest0-21swapper/020:27:190
688099220,21cyclictest0-21swapper/019:20:000
688099220,1cyclictest6083-21apt-get20:15:010
688099220,1cyclictest1306-21munin-run20:05:010
6892992121,0cyclictest28626-21apt-get21:02:163
6892992121,0cyclictest0-21swapper/320:15:013
6892992120,1cyclictest0-21swapper/321:32:213
6892992119,1cyclictest1893-21sendmail-msp22:20:003
689299210,20cyclictest0-21swapper/322:57:213
689299210,20cyclictest0-21swapper/321:47:183
689299210,0cyclictest0-21swapper/320:57:183
689299210,0cyclictest0-21swapper/320:32:193
6887992121,0cyclictest0-21swapper/223:02:172
6887992120,1cyclictest22090-21apt-get19:42:192
6887992120,1cyclictest11750-21apt-get20:27:192
688799210,21cyclictest0-21swapper/221:47:172
688799210,20cyclictest9350-21apt-get20:22:192
688799210,20cyclictest17521-21apt-get22:52:192
688799210,20cyclictest0-21swapper/223:27:202
688799210,1cyclictest1881-21perf22:20:002
688799210,0cyclictest29905-21apt-get22:12:182
688799210,0cyclictest19921-21apt-get22:57:212
688799210,0cyclictest17191-21apt-get23:57:202
688599213,2cyclictest23753-21unixbench_multi19:42:291
6885992121,0cyclictest10000-21apt-get23:42:201
6885992121,0cyclictest0-21swapper/123:27:201
6885992119,1cyclictest1863-21apt-get22:20:001
6885992119,1cyclictest16106-21turbostat20:35:001
688599210,20cyclictest0-21swapper/122:32:201
688599210,20cyclictest0-21swapper/122:02:181
6880992121,0cyclictest0-21swapper/020:22:190
6880992121,0cyclictest0-21swapper/000:07:180
6880992120,0cyclictest25059-21apt-get22:02:180
6880992119,1cyclictest14680-21cron22:45:000
6880992119,1cyclictest14353-21cron23:49:590
6880992119,1cyclictest12604-21grep21:35:000
6880992119,1cyclictest12266-21cron22:40:000
688099210,21cyclictest0-21swapper/022:12:170
688099210,20cyclictest16545-21apt-get20:37:180
688099210,20cyclictest16545-21apt-get20:37:180
688099210,20cyclictest0-21swapper/021:27:200
688099210,20cyclictest0-21swapper/019:42:200
688099210,20cyclictest0-21swapper/019:27:190
688099210,0cyclictest0-21swapper/019:12:190
6892992020,0cyclictest0-21swapper/320:47:203
6892992019,1cyclictest7513-21apt-get22:32:203
6892992019,1cyclictest19668-21apt-get19:37:183
6892992019,1cyclictest0-21swapper/322:02:163
6892992019,0cyclictest0-21swapper/321:07:193
6892992018,1cyclictest21660-21latency_hist00:05:013
6892992018,1cyclictest14431-21latency_hist19:25:003
689299200,20cyclictest0-21swapper/319:32:183
689299200,1cyclictest0-21swapper/322:52:193
689299200,19cyclictest0-21swapper/320:52:193
689299200,0cyclictest0-21swapper/322:47:193
689299200,0cyclictest0-21swapper/322:07:183
689299200,0cyclictest0-21swapper/319:42:203
6887992020,0cyclictest23760-21apt-get20:52:202
6887992020,0cyclictest0-21swapper/222:47:192
6887992020,0cyclictest0-21swapper/219:55:002
6887992019,1cyclictest19699-21apt-get00:02:192
6887992019,1cyclictest0-21swapper/222:42:202
688799200,20cyclictest0-21swapper/223:12:172
688799200,20cyclictest0-21swapper/221:27:182
688799200,0cyclictest0-21swapper/220:42:192
6885992020,0cyclictest7580-21apt-get23:37:191
6885992020,0cyclictest0-21swapper/123:22:191
6885992020,0cyclictest0-21swapper/121:42:171
6885992020,0cyclictest0-21swapper/121:12:191
688599200,20cyclictest0-21swapper/120:07:181
688599200,0cyclictest0-21swapper/120:57:191
688599200,0cyclictest0-21swapper/120:52:201
688599200,0cyclictest0-21swapper/119:22:181
6880992020,0cyclictest0-21swapper/023:17:180
6880992020,0cyclictest0-21swapper/022:17:180
6880992020,0cyclictest0-21swapper/021:42:180
6880992020,0cyclictest0-21swapper/020:52:190
6880992020,0cyclictest0-21swapper/019:22:190
6880992020,0cyclictest0-21swapper/000:37:220
688099200,0cyclictest0-21swapper/000:02:200
6892991919,0cyclictest24479-21apt-get19:47:173
6892991919,0cyclictest0-21swapper/320:37:183
6892991919,0cyclictest0-21swapper/320:37:183
6892991919,0cyclictest0-21swapper/319:57:193
6892991919,0cyclictest0-21swapper/300:32:183
6892991918,1cyclictest598-21rs:main3
6887991919,0cyclictest0-21swapper/222:27:192
6887991919,0cyclictest0-21swapper/221:17:182
6887991919,0cyclictest0-21swapper/220:37:172
6887991919,0cyclictest0-21swapper/220:37:172
6887991917,1cyclictest999-21apt-get21:12:192
6887991917,1cyclictest4136-21apt-get20:12:202
6887991917,1cyclictest26437-21apt19:49:592
6887991917,1cyclictest0-21swapper/200:19:592
6885991919,0cyclictest13035-21apt-get21:37:191
6885991919,0cyclictest0-21swapper/123:47:181
6885991919,0cyclictest0-21swapper/123:12:181
6885991919,0cyclictest0-21swapper/123:07:201
6885991919,0cyclictest0-21swapper/121:02:171
6885991919,0cyclictest0-21swapper/120:47:201
6885991919,0cyclictest0-21swapper/120:27:181
6885991919,0cyclictest0-21swapper/120:17:191
6885991919,0cyclictest0-21swapper/119:27:191
6885991919,0cyclictest0-21swapper/100:27:201
6885991919,0cyclictest0-21swapper/100:07:171
6885991917,2cyclictest7778-21cron21:25:001
6880991919,0cyclictest0-21swapper/019:47:170
6880991919,0cyclictest0-21swapper/000:32:190
6880991919,0cyclictest0-21swapper/000:27:190
6892991818,0cyclictest0-21swapper/323:52:223
6892991818,0cyclictest0-21swapper/323:37:183
6892991818,0cyclictest0-21swapper/323:17:183
6892991818,0cyclictest0-21swapper/321:42:183
6892991818,0cyclictest0-21swapper/321:37:193
6892991818,0cyclictest0-21swapper/321:12:193
6892991818,0cyclictest0-21swapper/319:10:023
6892991816,1cyclictest5403-21munin-run21:20:003
6887991818,0cyclictest22094-21apt-get00:07:182
6887991818,0cyclictest0-21swapper/223:47:182
6887991818,0cyclictest0-21swapper/223:37:192
6887991818,0cyclictest0-21swapper/223:32:182
6887991818,0cyclictest0-21swapper/221:57:212
6887991818,0cyclictest0-21swapper/200:27:192
6887991815,2cyclictest28117-21sh21:00:002
6885991818,0cyclictest0-21swapper/122:52:181
6885991818,0cyclictest0-21swapper/121:17:181
6885991818,0cyclictest0-21swapper/120:42:201
6885991818,0cyclictest0-21swapper/120:22:191
6885991817,0cyclictest20244-21apt-get21:52:191
6880991818,0cyclictest0-21swapper/023:02:180
6880991818,0cyclictest0-21swapper/022:52:200
6880991818,0cyclictest0-21swapper/022:47:190
6880991818,0cyclictest0-21swapper/021:52:190
6880991818,0cyclictest0-21swapper/021:47:180
6880991818,0cyclictest0-21swapper/019:52:180
6880991817,1cyclictest2503-21apt-get22:20:010
6892991717,0cyclictest0-21swapper/323:32:193
6892991717,0cyclictest0-21swapper/300:12:193
689299170,16cyclictest612-21gdbus22:14:593
689299170,16cyclictest0-21swapper/323:47:183
6887991717,0cyclictest12473-21apt-get19:22:192
6887991717,0cyclictest0-21swapper/222:32:202
6887991717,0cyclictest0-21swapper/221:37:202
6887991717,0cyclictest0-21swapper/221:05:002
6887991717,0cyclictest0-21swapper/200:37:222
6887991715,1cyclictest15450-21apt-get21:42:182
688799170,1cyclictest0-21swapper/223:07:192
688799170,16cyclictest21365-21apt-get20:47:192
6885991717,0cyclictest0-21swapper/100:37:221
6885991717,0cyclictest0-21swapper/100:12:181
6885991716,1cyclictest5110-21apt-get22:27:191
6885991715,1cyclictest583-21avahi-daemon19:40:001
6880991717,0cyclictest0-21swapper/021:02:170
6880991717,0cyclictest0-21swapper/020:07:180
6880991715,1cyclictest19240-21latency_hist19:35:000
688099170,16cyclictest5597-21grep22:27:200
6892991616,0cyclictest0-21swapper/323:07:203
6892991616,0cyclictest0-21swapper/322:42:203
6892991616,0cyclictest0-21swapper/321:52:193
6892991616,0cyclictest0-21swapper/320:27:193
6892991616,0cyclictest0-21swapper/320:12:203
6892991616,0cyclictest0-21swapper/320:02:193
689299160,15cyclictest28627-21chrt00:17:323
689299160,15cyclictest11166-21pmu-power23:42:273
689299160,15cyclictest0-21swapper/323:02:183
6887991616,0cyclictest2083-21apt-get00:32:192
6887991616,0cyclictest17269-21apt-get19:32:172
6887991614,1cyclictest26460-21munin-run00:15:002
6885991616,0cyclictest0-21swapper/100:02:181
688599160,1cyclictest0-21swapper/121:07:201
6880991616,0cyclictest0-21swapper/022:07:190
6880991616,0cyclictest0-21swapper/020:35:000
6880991614,1cyclictest0-21swapper/020:45:000
6880991613,2cyclictest7795-21wc21:25:000
688099160,15cyclictest25015-21perl00:12:210
6892991515,0cyclictest0-21swapper/322:37:173
6892991515,0cyclictest0-21swapper/321:22:193
6892991515,0cyclictest0-21swapper/300:22:203
689299150,1cyclictest1730-21apt-get20:07:183
689299150,1cyclictest0-21swapper/323:27:263
689299150,1cyclictest0-21swapper/321:57:213
6887991515,0cyclictest0-21swapper/219:57:192
6887991515,0cyclictest0-21swapper/219:17:182
688799150,0cyclictest0-21swapper/223:17:172
6885991515,0cyclictest0-21swapper/119:47:161
6885991514,1cyclictest19889-21/usr/sbin/munin22:57:271
6885991513,1cyclictest5824-21ntp_states23:32:231
688599150,5cyclictest0-21swapper/123:52:231
688599150,14cyclictest0-21swapper/122:45:101
6880991515,0cyclictest0-21swapper/023:12:180
6880991515,0cyclictest0-21swapper/000:17:180
6880991512,2cyclictest578-21irqbalance23:32:350
688099150,14cyclictest0-21swapper/021:15:000
688099150,0cyclictest0-21swapper/023:42:200
6892991414,0cyclictest0-21swapper/322:26:023
689299140,14cyclictest0-21swapper/323:10:013
689299140,14cyclictest0-21swapper/300:26:193
689299140,0cyclictest0-21swapper/321:25:163
6887991414,0cyclictest0-21swapper/221:07:192
6887991414,0cyclictest0-21swapper/220:32:002
6887991414,0cyclictest0-21swapper/219:26:012
688799140,14cyclictest0-21swapper/221:51:032
688799140,14cyclictest0-21swapper/221:30:472
688799140,14cyclictest0-21swapper/220:15:502
688799140,0cyclictest6775-21apt-get19:10:012
6885991414,0cyclictest0-21swapper/120:35:511
6885991414,0cyclictest0-21swapper/120:35:511
6885991414,0cyclictest0-21swapper/100:32:021
688599140,14cyclictest0-21swapper/121:30:521
6880991414,0cyclictest0-21swapper/023:05:090
6880991414,0cyclictest0-21swapper/021:35:050
6880991414,0cyclictest0-21swapper/019:36:320
688099140,14cyclictest0-21swapper/022:31:270
688099140,14cyclictest0-21swapper/021:58:310
688099140,14cyclictest0-21swapper/021:15:430
688099140,0cyclictest0-21swapper/000:20:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional