You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 05:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Sep 16, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2444235137,3494sleep01233-21kworker/0:023:23:480
30382272415,693sleep130386-21diskstats19:47:521
30382272415,693sleep130386-21diskstats19:47:521
327261917,584sleep1384-21/usr/sbin/munin23:18:031
2765625718,551sleep027667-21ntpq19:38:060
29249254512,517sleep229307-21/usr/sbin/munin19:43:212
29249254512,517sleep229307-21/usr/sbin/munin19:43:212
31910249027,445sleep231922-21/usr/sbin/munin21:32:572
21042248524,443sleep221088-21/usr/sbin/munin00:22:572
18099241324,373sleep318116-21/usr/sbin/munin20:52:353
3021723757,357sleep0362-21snmpd23:08:540
15106232749,206sleep015228-21timerwakeupswit00:03:050
26023230611,281sleep326062-21ntpq19:33:043
26023230611,281sleep326062-21ntpq19:33:043
15464230614,17sleep00-21swapper/022:23:060
31923228410,258sleep331966-21ntpq21:32:583
11345225955,187sleep011257-21/usr/sbin/munin23:53:320
24147224471,114sleep00-21swapper/000:33:010
24147224471,114sleep00-21swapper/000:33:010
17535223811,204sleep017597-21grep19:07:530
252482232109,25sleep00-21swapper/022:53:190
15637221610,189sleep315678-21ntpq20:43:023
14394221615,174sleep014541-21unixbench_singl20:38:070
23276221110,181sleep023363-21apt-config21:07:350
23276221110,181sleep023363-21apt-config21:07:350
6426220649,49sleep00-21swapper/023:37:570
7588220318,135sleep07732-21unixbench_multi21:58:110
22476220140,54sleep022505-21munin-run22:47:330
1815219814,144sleep01894-21threads19:58:050
2285221909,165sleep022866-21grep22:47:540
2775218533,116sleep0362-21snmpd20:02:490
2775218533,116sleep0362-21snmpd20:02:490
1137218214,150sleep01177-21proc_pri21:38:040
10412218110,159sleep010459-21ntpq22:08:000
12545218016,143sleep1225-21systemd-journal23:57:331
25707217841,22sleep00-21swapper/019:32:530
25707217841,22sleep00-21swapper/019:32:530
5835217421,35sleep05851-21modprobe23:37:330
2667021708,71sleep00-21swapper/021:17:360
1864921666,148sleep018733-21expr22:33:080
14691216019,73sleep314709-21kernelversion00:02:563
32091215610,106sleep031520-21/usr/sbin/munin21:33:030
22636215516,122sleep222788-21awk21:03:042
22636215516,122sleep222788-21awk21:03:042
540821496,42sleep00-21swapper/021:52:580
1181621497,128sleep011851-21awk20:32:330
2043421489,126sleep020494-21latency_hist00:22:350
18630214710,119sleep218567-21apt-get20:53:272
2969721466,30sleep00-21swapper/021:27:250
1263921465,29sleep00-21swapper/022:15:110
10739214621,98sleep310878-21unixbench_singl22:08:073
17749213959,62sleep217784-21idleruntime-cro22:32:332
17539213917,19sleep00-21swapper/000:12:500
19726213817,96sleep20-21swapper/200:18:052
13327213710,66sleep00-21swapper/022:17:530
17242213210,76sleep00-21swapper/020:48:040
28952213112,104sleep028957-21modprobe23:07:320
2522321316,33sleep00-21swapper/019:31:110
12683213115,72sleep00-21swapper/020:33:060
9463212719,92sleep28981-21apt-get23:47:442
29194212527,78sleep229341-21unixbench_singl21:23:072
394212416,71sleep017215-21cyclictest23:18:060
30558212322,23sleep30-21swapper/323:12:363
21185212310,41sleep00-21swapper/020:58:420
2463121226,54sleep00-21swapper/021:08:190
29961212160,49sleep06510-1kworker/0:2H21:27:380
16909212115,68sleep00-21swapper/000:08:220
16909212115,68sleep00-21swapper/000:08:220
872621197,13sleep00-21swapper/022:02:550
5623211967,16sleep05633-21munin-run20:12:350
3951211916,69sleep00-21swapper/023:28:160
26771211729,65sleep00-21swapper/022:58:070
26771211729,65sleep00-21swapper/022:58:070
16542211710,73sleep016614-21latency_hist20:47:350
28206211512,87sleep228321-21sed21:22:352
28206211512,87sleep228321-21sed21:22:352
26871211531,36sleep10-21swapper/121:17:451
26871211531,36sleep10-21swapper/121:17:451
16490211527,68sleep2225-21systemd-journal20:47:332
472121146,71sleep00-21swapper/021:48:380
29381211416,40sleep00-21swapper/019:43:230
29381211416,40sleep00-21swapper/019:43:230
2143121149,94sleep021465-21ps19:18:030
1767321149,68sleep00-21swapper/022:31:340
3143421127,50sleep0362-21snmpd19:50:540
3143421127,50sleep0362-21snmpd19:50:540
750211013,39sleep1646-21/usr/sbin/munin21:37:521
2098521109,64sleep021137-21date22:42:370
608021099,45sleep06136-21interrupts20:12:550
3079621099,58sleep00-21swapper/023:12:490
2261621089,38sleep00-21swapper/019:22:560
9759210714,72sleep30-21swapper/322:07:343
30460210714,15sleep10-21swapper/121:28:031
2804210723,62sleep00-21swapper/021:43:040
944021067,37sleep00-21swapper/020:23:010
10232210618,71sleep010380-21grep23:48:470
9530210410,75sleep19482-21apt-get23:48:161
2065121047,41sleep01-21systemd19:17:350
2065121047,41sleep01-21systemd19:17:350
20366210418,66sleep319874-21apt-get20:57:443
10638210419,69sleep110670-21grep20:27:501
22266210313,76sleep322271-21systemd-run19:22:343
978210118,60sleep00-21swapper/019:57:370
5083210016,68sleep25230-21latency_hist21:52:352
2244121007,33sleep00-21swapper/000:27:560
2168429813,73sleep121736-21processes22:43:021
1445129812,60sleep10-21swapper/122:22:351
896529731,44sleep38997-21munin-run23:47:333
1800029612,70sleep118022-21pluginstate00:13:031
2537029515,67sleep325385-21latency_hist19:32:363
1875329413,61sleep10-21swapper/119:12:351
1678829319,61sleep116781-21apt-get20:47:531
1276329318,57sleep112878-21timerwakeupswit20:33:071
90042928,16sleep00-21swapper/023:47:340
722129113,64sleep17304-21wc20:17:361
2402628927,41sleep324042-21apt-get22:52:333
1648328816,37sleep30-21swapper/322:27:503
1729728710,16sleep20-21swapper/200:12:362
1729728710,16sleep20-21swapper/200:12:362
2094328626,37sleep00-21swapper/000:22:550
1784128619,50sleep117989-21sed22:32:361
500128513,57sleep25121-21cat20:08:082
2423728516,56sleep324250-21open_files00:33:043
2423728516,56sleep324250-21open_files00:33:043
1182828525,34sleep10-21swapper/122:12:581
2076028416,54sleep120840-21fschecks_time00:22:521
863928331,26sleep20-21swapper/222:02:532
3089328320,30sleep20-21swapper/221:28:082
2985128327,33sleep10-21swapper/123:08:061
287028329,31sleep30-21swapper/320:02:573
287028329,31sleep30-21swapper/320:02:573
1771728323,41sleep30-21swapper/319:07:573
1613328222,38sleep016192-21latency19:02:580
858728119,28sleep00-21swapper/020:20:500
2788528119,29sleep00-21swapper/021:19:060
2788528119,29sleep00-21swapper/021:19:060
2278928117,29sleep30-21swapper/321:03:053
2278928117,29sleep30-21swapper/321:03:053
168328116,30sleep20-21swapper/219:58:032
1595028133,26sleep10-21swapper/119:02:541
1260228114,17sleep30-21swapper/320:33:053
2900528028,29sleep30-21swapper/323:07:353
2611428011,18sleep30-21swapper/321:13:153
1533028027,30sleep30-21swapper/322:23:053
1397128019,28sleep00-21swapper/000:00:260
615927918,43sleep26229-21df23:37:512
3250627920,45sleep332517-21grep23:17:573
3216027913,50sleep132280-21timerandwakeup21:33:041
2580927916,17sleep20-21swapper/219:32:592
2580927916,17sleep20-21swapper/219:32:592
1477627918,30sleep20-21swapper/222:22:522
858327817,45sleep38585-21sshd20:20:493
579827816,43sleep35836-21/usr/sbin/munin21:53:093
432627816,31sleep10-21swapper/120:07:511
1912127828,34sleep119114-21apt-get00:17:471
1862827818,29sleep00-21swapper/020:53:250
1348427821,26sleep20-21swapper/223:58:102
972427719,37sleep28985-21/usr/sbin/munin20:23:072
3207927712,46sleep131574-21apt-get19:52:571
2255827718,25sleep10-21swapper/121:03:031
2255827718,25sleep10-21swapper/121:03:031
2003527719,40sleep319874-21apt-get20:57:363
646727626,34sleep16525-21memory23:37:581
497227614,30sleep30-21swapper/323:32:593
3147527622,29sleep10-21swapper/123:13:061
2040227619,30sleep20-21swapper/220:57:512
1873427617,36sleep118889-21grep22:33:091
909627526,25sleep10-21swapper/120:22:531
213692757,48sleep10-21swapper/119:18:001
305452740,8sleep30-21swapper/321:28:043
2851227411,49sleep228572-21latency_hist19:42:362
2372627420,34sleep223234-21apt-get21:07:452
2093427420,28sleep10-21swapper/120:58:391
737427320,26sleep10-21swapper/121:58:081
2923527320,29sleep10-21swapper/119:43:201
2923527320,29sleep10-21swapper/119:43:201
2581827322,36sleep30-21swapper/322:57:343
2269527321,27sleep10-21swapper/100:28:051
2097827322,24sleep30-21swapper/322:42:363
1009627321,36sleep210150-21fschecks_count22:07:542
370127229,24sleep30-21swapper/323:28:033
289927226,21sleep10-21swapper/121:43:051
1648327216,32sleep115957-21/usr/sbin/munin00:08:061
1648327216,32sleep115957-21/usr/sbin/munin00:08:061
3214427113,42sleep232256-21latency_hist23:17:362
210502717,14sleep20-21swapper/219:17:532
2008127125,28sleep20-21swapper/222:38:022
1856927116,38sleep118567-21apt-get20:52:451
1809127119,23sleep30-21swapper/300:13:053
119027114,42sleep11207-21apt19:57:461
1027927114,42sleep110313-21wc22:07:571
3080427018,28sleep20-21swapper/219:48:182
3080427018,28sleep20-21swapper/219:48:182
278427029,20sleep30-21swapper/323:27:343
2513227019,26sleep20-21swapper/200:37:352
2513227019,26sleep20-21swapper/200:37:352
1386827016,23sleep10-21swapper/120:37:551
66642690,8sleep30-21swapper/320:13:053
2389326915,39sleep123896-21/usr/sbin/munin21:08:041
1623326920,20sleep20-21swapper/219:03:002
172226821,21sleep30-21swapper/319:58:043
262212677,41sleep20-21swapper/222:57:532
262212677,41sleep20-21swapper/222:57:532
169126716,23sleep10-21swapper/123:22:561
2908326612,24sleep10-21swapper/121:23:061
1164526610,39sleep211730-21fschecks_time22:12:542
3206326516,33sleep332086-21modprobe23:17:333
3161926516,31sleep331714-21latency_hist19:52:353
3161926516,31sleep331714-21latency_hist19:52:353
2845926520,22sleep30-21swapper/319:42:333
2740626521,24sleep10-21swapper/123:02:321
2740626521,24sleep10-21swapper/123:02:321
1937626517,31sleep319451-21latency_hist22:37:353
143742655,10sleep20-21swapper/222:22:112
1105926520,24sleep30-21swapper/320:28:003
1078126516,24sleep30-21swapper/323:52:343
352926421,27sleep20-21swapper/221:47:352
1634826416,27sleep30-21swapper/300:08:003
1634826416,27sleep30-21swapper/300:08:003
1573926419,33sleep215718-21latency_hist00:07:352
828826316,27sleep18380-21/usr/sbin/munin23:43:031
823726321,27sleep28377-21latency_hist22:02:372
2777026321,19sleep10-21swapper/123:02:531
2301326314,12sleep10-21swapper/119:23:051
1921826316,33sleep319257-21fschecks_count00:17:533
871826216,29sleep20-21swapper/220:22:342
478826213,25sleep30-21swapper/320:08:053
3209826216,25sleep20-21swapper/219:52:582
2222726218,20sleep30-21swapper/300:27:513
2173726213,32sleep221191-21/usr/sbin/munin22:43:032
1387926221,27sleep314006-21unixbench_singl22:18:043
128792626,12sleep20-21swapper/220:33:082
845626119,22sleep10-21swapper/122:02:481
3269326118,26sleep30-21swapper/319:53:443
30982616,13sleep10-21swapper/123:27:481
304826118,24sleep20-21swapper/220:03:002
304826118,24sleep20-21swapper/220:03:002
2509626120,24sleep125131-21sed00:37:341
2509626120,24sleep125131-21sed00:37:341
1376926117,20sleep30-21swapper/320:37:533
560026014,25sleep10-21swapper/121:53:031
2625126019,18sleep10-21swapper/119:33:231
2625126019,18sleep10-21swapper/119:33:231
224692606,11sleep30-21swapper/322:47:283
2007326015,20sleep10-21swapper/122:38:011
1588926023,18sleep10-21swapper/120:43:081
1356126017,19sleep30-21swapper/323:58:113
526025914,26sleep25416-21grep23:33:062
179725919,21sleep20-21swapper/221:40:522
1793925915,29sleep217987-21grep00:13:002
1182525921,18sleep30-21swapper/323:53:553
782225820,21sleep10-21swapper/120:17:571
2993625819,21sleep20-21swapper/223:08:072
636725717,15sleep30-21swapper/323:37:563
569625717,26sleep1225-21systemd-journal23:35:481
2831425710,32sleep128315-21sshd19:40:411
2413325718,25sleep124265-21apt-key22:52:351
1261325714,26sleep212612-40mandb23:57:342
1055525717,25sleep210618-21cpuspeed20:27:472
70562566,11sleep20-21swapper/220:16:282
2887525617,23sleep328878-21fschecks_count19:42:583
2887525617,23sleep328878-21fschecks_count19:42:583
251152567,14sleep20-21swapper/219:29:522
2449225612,23sleep324629-21/usr/sbin/munin21:08:183
2426825619,20sleep20-21swapper/222:52:362
2318225620,17sleep30-21swapper/319:23:073
1798125611,28sleep218050-21sensors_temp19:08:042
690525514,22sleep30-21swapper/321:57:533
357625517,15sleep30-21swapper/321:47:363
302425514,24sleep23018-21apt-get23:27:452
2592325516,16sleep10-21swapper/122:57:361
203532556,41sleep10-21swapper/119:13:521
203532556,41sleep10-21swapper/119:13:521
140262556,35sleep10-21swapper/100:01:081
1155325514,23sleep311551-21apt-get22:12:473
263582544,12sleep10-21swapper/121:14:461
2455525415,24sleep224545-21apt-get22:52:482
1945925411,26sleep219468-21diskstats19:12:522
1945925411,26sleep219468-21diskstats19:12:522
1454525412,23sleep114637-21perl00:02:541
767725316,20sleep20-21swapper/223:42:432
2796725318,18sleep20-21swapper/223:02:582
2614125312,24sleep326132-21apt-get22:57:503
2614125312,24sleep326132-21apt-get22:57:503
219142537,34sleep30-21swapper/300:27:183
1761325313,23sleep217695-21users20:48:102
1586025312,24sleep315857-21apt-get19:02:483
424025216,17sleep30-21swapper/321:48:043
3265625214,18sleep20-21swapper/223:18:002
319425214,22sleep13231-21ntp_states20:03:031
319425214,22sleep13231-21ntp_states20:03:031
2713825214,22sleep327167-21irqstats21:17:553
2713825214,22sleep327167-21irqstats21:17:553
2350925214,23sleep223531-21systemd-run00:32:342
2277625116,21sleep222789-21irqstats19:22:582
205825114,17sleep20-21swapper/223:23:052
1031725115,18sleep30-21swapper/320:27:363
1491825013,21sleep114961-21irqstats22:22:551
541724917,15sleep20-21swapper/221:53:002
377424911,18sleep10-21swapper/121:47:501
2529024910,19sleep20-21swapper/221:12:502
1841724912,16sleep20-21swapper/222:33:022
1478824910,24sleep214791-21sshd20:40:392
2970524813,18sleep30-21swapper/321:27:323
2527124713,17sleep10-21swapper/119:31:491
198324518,15sleep30-21swapper/321:42:353
193822448,18sleep30-21swapper/319:12:493
193822448,18sleep30-21swapper/319:12:493
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional