You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 11:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Fri Dec 05, 2025 00:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28999992624,1cyclictest12654-21awk20:50:001
2900099250,24cyclictest10784-21apt-get20:48:162
28999992420,3cyclictest19344-21apt-get23:18:181
28994992422,1cyclictest141rcuc/023:10:000
2899499240,23cyclictest0-21swapper/020:35:000
2900199230,2cyclictest28249-21apt00:39:593
29000992322,1cyclictest27664-21apt-get21:23:192
28999992321,1cyclictest616-21dump-pmu-power20:25:001
2899999230,22cyclictest0-21swapper/122:20:001
28994992321,1cyclictest621-21perf20:25:000
2899499230,22cyclictest583-21avahi-daemon23:05:000
29001992221,1cyclictest9339-21apt-get22:58:183
29001992220,1cyclictest13406-21munin-plugin-st19:45:003
2900199220,21cyclictest25846-21dump-pmu-power00:35:003
2900199220,21cyclictest16654-21apt-get23:10:023
2900199220,21cyclictest15578-21apt-get20:58:163
2900199220,21cyclictest0-21swapper/323:33:163
2900199220,1cyclictest0-21swapper/322:38:183
29000992220,1cyclictest31624-21turbostat22:35:002
29000992220,1cyclictest31624-21turbostat22:35:002
2900099220,21cyclictest4293-21apt-get19:28:172
2900099220,21cyclictest28219-21cron20:15:002
2900099220,21cyclictest0-21swapper/222:48:152
2900099220,1cyclictest6086-21cron23:55:002
2899999220,21cyclictest5970-21apt-get20:38:171
2899999220,21cyclictest24553-21apt-get23:28:191
2899999220,21cyclictest0-21swapper/120:58:161
2899999220,1cyclictest313-21systemd-journal19:40:001
2899999220,1cyclictest29547-21latency_hist21:25:001
2899999220,1cyclictest23909-21apt-get20:08:161
2899999220,0cyclictest0-21swapper/122:48:141
28994992222,0cyclictest0-21swapper/023:10:010
28994992220,1cyclictest9119-21apt-get00:03:180
28994992220,1cyclictest27119-21idleruntime-cro21:20:000
28994992220,1cyclictest1286-21sh23:45:000
28994992219,2cyclictest24368-21idleruntime-cro22:20:000
2899499220,21cyclictest0-21swapper/023:35:010
2899499220,21cyclictest0-21swapper/022:03:210
2899499220,1cyclictest3061-21awk20:30:000
2899499220,1cyclictest0-21swapper/021:55:020
29001992120,1cyclictest31882-21apt-get19:18:143
29001992119,1cyclictest11940-21apt-get21:55:003
29001992119,1cyclictest0-21swapper/321:05:003
2900199210,20cyclictest20577-21sh20:00:003
2900199210,20cyclictest0-21swapper/323:38:193
2900199210,20cyclictest0-21swapper/321:50:003
2900199210,20cyclictest0-21swapper/321:30:013
2900199210,20cyclictest0-21swapper/320:43:183
2900199210,1cyclictest3779-21grep19:25:003
2900199210,1cyclictest18667-21munin-run00:20:003
2900199210,0cyclictest14304-21apt-get00:13:163
29000992121,0cyclictest25243-21apt-get21:18:162
29000992121,0cyclictest0-21swapper/221:53:162
29000992119,1cyclictest22292-21munin-run21:10:012
29000992119,1cyclictest18662-21apt00:20:002
2900099210,20cyclictest0-21swapper/223:19:592
2900099210,20cyclictest0-21swapper/200:13:172
2900099210,1cyclictest21228-21apt-get20:00:022
28999992120,0cyclictest0-21swapper/119:53:131
28999992119,1cyclictest31323-21grep23:40:001
28999992119,1cyclictest313-21systemd-journal20:05:001
28999992119,1cyclictest0-21swapper/122:45:001
2899999210,1cyclictest31945-21apt-get21:30:011
2899999210,1cyclictest22288-21perf21:10:001
2899999210,0cyclictest31833-21apt-get23:43:211
2899999210,0cyclictest0-21swapper/123:13:181
2899999210,0cyclictest0-21swapper/119:18:141
2899999210,0cyclictest0-21swapper/100:08:161
2899999210,0cyclictest0-21swapper/100:03:181
28994992121,0cyclictest0-21swapper/022:08:140
28994992120,1cyclictest21248-21sendmail-msp23:20:000
28994992120,0cyclictest0-21swapper/020:48:150
28994992119,1cyclictest24677-21perf21:15:000
2899499210,21cyclictest0-21swapper/019:35:000
2899499210,0cyclictest18002-21apt-get21:03:220
2899499210,0cyclictest0-21swapper/021:28:170
2899499210,0cyclictest0-21swapper/020:43:180
2899499210,0cyclictest0-21swapper/020:08:160
29001992020,0cyclictest26356-21apt-get20:13:153
29001992020,0cyclictest0-21swapper/322:53:173
29001992020,0cyclictest0-21swapper/322:33:163
29001992020,0cyclictest0-21swapper/322:33:163
29001992020,0cyclictest0-21swapper/322:03:223
29001992018,1cyclictest0-21swapper/300:05:023
2900199200,0cyclictest0-21swapper/323:18:183
2900199200,0cyclictest0-21swapper/319:48:153
29000992020,0cyclictest2853-21apt-get21:38:192
29000992020,0cyclictest1818-21apt-get23:48:172
29000992020,0cyclictest0-21swapper/222:58:182
29000992020,0cyclictest0-21swapper/222:08:142
29000992020,0cyclictest0-21swapper/220:38:172
29000992020,0cyclictest0-21swapper/220:33:182
29000992020,0cyclictest0-21swapper/200:23:152
29000992018,1cyclictest16043-21apt-get23:10:012
29000992017,2cyclictest17502-21awk21:00:002
2900099200,0cyclictest5267-21apt-get21:43:152
2900099200,0cyclictest403-21apt-get21:33:212
2900099200,0cyclictest0-21swapper/223:28:182
2900099200,0cyclictest0-21swapper/222:03:222
2900099200,0cyclictest0-21swapper/221:58:192
2900099200,0cyclictest0-21swapper/200:28:192
28999992020,0cyclictest0-21swapper/122:28:201
28999992020,0cyclictest0-21swapper/121:18:161
28999992020,0cyclictest0-21swapper/119:33:161
28999992019,1cyclictest17666-21apt-get22:08:151
28999992018,1cyclictest593-21dbus-daemon19:30:011
2899999200,20cyclictest0-21swapper/122:23:151
2899999200,1cyclictest30661-21grep20:20:011
2899999200,19cyclictest4214-21apt-get23:53:171
2899999200,0cyclictest0-21swapper/122:58:181
28994992020,0cyclictest0-21swapper/022:58:180
28994992020,0cyclictest0-21swapper/022:38:170
28994992020,0cyclictest0-21swapper/000:33:190
28994992020,0cyclictest0-21swapper/000:18:180
28994992018,1cyclictest612-21gdbus20:05:000
2899499200,19cyclictest0-21swapper/019:48:150
29001991919,0cyclictest0-21swapper/300:23:163
29001991918,0cyclictest0-21swapper/321:38:193
29000991919,0cyclictest9089-21apt-get19:38:142
29000991919,0cyclictest6693-21apt-get19:33:162
29000991919,0cyclictest0-21swapper/223:58:172
29000991919,0cyclictest0-21swapper/223:33:172
29000991919,0cyclictest0-21swapper/221:48:182
29000991919,0cyclictest0-21swapper/219:58:152
29000991919,0cyclictest0-21swapper/219:13:172
29000991919,0cyclictest0-21swapper/200:03:182
29000991918,0cyclictest24935-21apt-get22:23:162
29000991918,0cyclictest22792-21apt-get21:13:162
29000991918,0cyclictest20065-21apt-get22:13:152
29000991917,1cyclictest0-21swapper/219:55:002
28999991919,0cyclictest0-21swapper/123:03:201
28999991919,0cyclictest0-21swapper/122:53:161
28999991919,0cyclictest0-21swapper/122:33:161
28999991919,0cyclictest0-21swapper/122:33:161
28999991919,0cyclictest0-21swapper/121:58:171
28999991919,0cyclictest0-21swapper/121:28:181
28999991919,0cyclictest0-21swapper/119:43:161
28999991918,1cyclictest18699-21apt-get19:58:141
28994991919,0cyclictest0-21swapper/022:33:160
28994991919,0cyclictest0-21swapper/022:33:150
28994991919,0cyclictest0-21swapper/021:53:170
28994991917,1cyclictest28247-21awk20:15:000
28994991917,1cyclictest20594-21perf20:00:000
2899499190,1cyclictest0-21swapper/000:38:170
29001991818,0cyclictest0-21swapper/323:43:213
29001991818,0cyclictest0-21swapper/323:25:023
29001991818,0cyclictest0-21swapper/322:23:143
29001991818,0cyclictest0-21swapper/320:53:173
29001991818,0cyclictest0-21swapper/319:33:153
29001991818,0cyclictest0-21swapper/300:03:183
29001991816,1cyclictest0-21swapper/321:20:003
29000991818,0cyclictest0-21swapper/222:38:182
29000991818,0cyclictest0-21swapper/221:03:222
29000991818,0cyclictest0-21swapper/200:05:022
29000991817,0cyclictest0-21swapper/219:48:152
28999991818,0cyclictest0-21swapper/121:53:161
28999991818,0cyclictest0-21swapper/121:43:161
28999991818,0cyclictest0-21swapper/120:13:151
28999991818,0cyclictest0-21swapper/100:23:161
28999991817,1cyclictest1893-21apt-get19:23:161
28999991817,0cyclictest0-21swapper/123:25:001
28994991818,0cyclictest0-21swapper/021:43:160
28994991818,0cyclictest0-21swapper/020:58:170
28994991818,0cyclictest0-21swapper/000:13:170
28994991816,1cyclictest8488-21ls00:00:000
28994991816,1cyclictest13806-21perf00:10:000
29001991717,0cyclictest3571-21apt-get20:33:173
29001991717,0cyclictest0-21swapper/321:13:173
29001991717,0cyclictest0-21swapper/320:23:163
29001991716,1cyclictest28759-21apt-get20:18:173
29001991716,0cyclictest0-21swapper/322:18:203
29000991717,0cyclictest14152-21apt-get23:08:172
29000991717,0cyclictest0-21swapper/219:43:162
29000991715,1cyclictest31182-21apt-get20:23:162
29000991715,1cyclictest0-21swapper/223:05:002
29000991715,1cyclictest0-21swapper/221:30:002
28999991717,0cyclictest0-21swapper/121:03:221
28999991717,0cyclictest0-21swapper/120:28:171
28999991717,0cyclictest0-21swapper/119:13:151
28999991716,1cyclictest27003-21apt-get23:33:171
28999991715,1cyclictest32138-21apt-get22:38:181
28999991715,1cyclictest23450-21perf00:30:001
28994991717,0cyclictest0-21swapper/023:23:170
28994991717,0cyclictest0-21swapper/019:38:130
28994991717,0cyclictest0-21swapper/019:23:150
28994991717,0cyclictest0-21swapper/019:18:150
28994991717,0cyclictest0-21swapper/019:13:170
28994991716,1cyclictest13-21ksoftirqd/021:25:000
28994991716,1cyclictest0-21swapper/023:48:170
28994991715,1cyclictest0-21swapper/022:15:000
29001991616,0cyclictest0-21swapper/322:48:153
29001991616,0cyclictest0-21swapper/321:28:163
29001991616,0cyclictest0-21swapper/320:28:173
29001991616,0cyclictest0-21swapper/320:03:153
29001991616,0cyclictest0-21swapper/319:13:163
29001991615,0cyclictest0-21swapper/323:48:173
2900099165,10cyclictest6166-21munin-node23:41:562
29000991616,0cyclictest1159-21apt-get20:28:182
29000991616,0cyclictest0-21swapper/220:53:172
29000991616,0cyclictest0-21swapper/220:18:172
2900099160,15cyclictest0-21swapper/223:38:192
28999991614,1cyclictest18628-21tr00:20:001
28999991614,1cyclictest0-21swapper/123:58:251
28999991614,1cyclictest0-21swapper/120:45:001
28999991613,2cyclictest17144-21cron22:05:001
2899999160,15cyclictest0-21swapper/100:13:251
28994991616,0cyclictest0-21swapper/022:53:170
28994991616,0cyclictest0-21swapper/022:28:190
28994991616,0cyclictest0-21swapper/020:38:170
28994991616,0cyclictest0-21swapper/020:18:170
28994991615,1cyclictest0-21swapper/023:53:150
2899499160,15cyclictest6173-21perf19:30:010
29001991515,0cyclictest20393-21apt-get21:08:173
29001991515,0cyclictest0-21swapper/323:58:173
29001991515,0cyclictest0-21swapper/319:38:143
29001991514,0cyclictest0-21swapper/321:43:163
29000991514,1cyclictest27334-21apt-get22:28:202
29000991514,0cyclictest0-21swapper/220:08:152
29000991513,1cyclictest6943-21apt-get22:53:172
2900099150,15cyclictest0-21swapper/219:18:142
28999991514,0cyclictest0-21swapper/123:08:181
28994991515,0cyclictest29411-21apt-get23:38:190
28994991514,1cyclictest13-21ksoftirqd/019:55:000
2899499150,9cyclictest0-21swapper/023:25:010
29001991414,0cyclictest0-21swapper/323:50:083
2900199140,14cyclictest0-21swapper/323:22:023
2900199140,14cyclictest0-21swapper/323:06:343
2900199140,14cyclictest0-21swapper/323:00:283
2900199140,14cyclictest0-21swapper/322:07:023
2900199140,14cyclictest0-21swapper/321:20:033
2900199140,14cyclictest0-21swapper/320:46:303
2900199140,14cyclictest0-21swapper/320:36:573
2900199140,14cyclictest0-21swapper/320:06:503
2900199140,14cyclictest0-21swapper/319:50:083
2900199140,14cyclictest0-21swapper/300:25:393
2900199140,0cyclictest0-21swapper/322:40:393
2900199140,0cyclictest0-21swapper/322:25:393
2900199140,0cyclictest0-21swapper/322:12:263
2900199140,0cyclictest0-21swapper/321:57:293
2900199140,0cyclictest0-21swapper/319:25:563
29000991414,0cyclictest0-21swapper/222:41:422
29000991414,0cyclictest0-21swapper/200:30:312
2900099140,14cyclictest0-21swapper/220:41:402
2900099140,14cyclictest0-21swapper/219:24:032
2900099140,14cyclictest0-21swapper/200:35:352
2900099140,0cyclictest0-21swapper/223:22:492
2900099140,0cyclictest0-21swapper/222:16:072
28999991414,0cyclictest0-21swapper/123:45:121
28999991414,0cyclictest0-21swapper/122:13:161
28999991414,0cyclictest0-21swapper/121:36:021
28999991414,0cyclictest0-21swapper/121:10:091
28999991414,0cyclictest0-21swapper/120:50:181
2899999140,14cyclictest0-21swapper/121:45:131
2899999140,14cyclictest0-21swapper/100:30:431
28994991414,0cyclictest0-21swapper/021:05:380
28994991414,0cyclictest0-21swapper/020:50:520
28994991414,0cyclictest0-21swapper/019:40:000
2899499140,14cyclictest0-21swapper/022:45:480
2899499140,14cyclictest0-21swapper/022:22:110
2899499140,14cyclictest0-21swapper/021:46:550
2899499140,14cyclictest0-21swapper/000:25:060
2899499140,0cyclictest0-21swapper/021:37:270
2899499140,0cyclictest0-21swapper/000:20:580
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional