You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-17 - 01:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Sep 16, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6612131413,1285sleep1680-21/usr/sbin/munin08:42:521
812276312,733sleep3828-21/usr/sbin/munin12:08:113
812276312,733sleep3828-21/usr/sbin/munin12:08:113
31893274219,705sleep031905-21/usr/sbin/munin08:38:110
30605253312,507sleep330649-21ntpq08:33:173
6313252243,462sleep26380-21latency_hist10:42:522
21016229529,142sleep221051-21grep11:28:202
147022957,256sleep2367-21snmpd12:08:302
147022957,256sleep2367-21snmpd12:08:302
112802279105,136sleep211363-21mailstats07:33:232
24158227310,249sleep024225-21systemctl11:38:190
2342322659,240sleep123433-21/usr/sbin/munin11:37:521
97342258109,55sleep20-21swapper/210:53:102
97342258109,55sleep20-21swapper/210:53:102
27766225454,20sleep20-21swapper/208:24:202
20890224162,164sleep220404-21/usr/sbin/munin08:03:192
4111222412,186sleep24241-21unixbench_multi10:33:232
2729222253,124sleep22310-21/usr/sbin/munin12:13:272
27140221210,167sleep227185-21ntpq11:48:152
1013221213,184sleep11053-21processes12:08:171
1013221213,184sleep11053-21processes12:08:171
397522098,162sleep24119-21cat07:12:552
397522098,162sleep24119-21cat07:12:552
15260220830,63sleep20-21swapper/207:47:532
2467220226,138sleep22600-21latency_hist08:47:542
8335220159,125sleep28421-21latency_hist12:32:532
177272201114,30sleep217791-21cat11:18:142
10505219915,142sleep210651-21latency_hist09:12:542
28766219624,57sleep20-21swapper/208:28:072
14086219616,137sleep214154-21latency_hist11:07:522
23525219539,49sleep20-21swapper/208:12:532
2013921919,167sleep220142-21cron08:02:512
5240218713,162sleep35263-21sed12:22:533
30107218710,162sleep230141-21proc_pri10:13:202
2284221779,153sleep222983-21awk11:33:252
5831217223,68sleep15873-21sendmail07:17:511
23640216610,59sleep223679-21proc_pri09:53:212
956216516,126sleep20-21swapper/210:23:222
17217216518,129sleep117326-21latency_hist11:17:531
26110216332,116sleep226260-21latency_hist10:02:542
26905216211,137sleep026984-21find10:03:200
32005215431,86sleep232063-21if_eth008:38:132
185921530,34sleep31863-21unixbench_multi07:03:273
3962213922,101sleep20-21swapper/208:52:512
1798521385,12sleep20-21swapper/209:34:552
30164213610,60sleep20-21swapper/211:58:122
2425213664,17sleep20-21swapper/207:07:542
2132321349,114sleep2367-21snmpd09:47:392
1959921340,14sleep00-21swapper/011:23:200
12408213430,91sleep212410-21apt-get07:38:032
940021327,15sleep00-21swapper/010:52:520
940021327,15sleep00-21swapper/010:52:520
704021327,15sleep27108-21readlink10:43:212
5510213027,64sleep25587-21apt-config08:57:522
1937321298,48sleep20-21swapper/211:23:152
15956212928,62sleep20-21swapper/211:13:092
8079212849,40sleep28092-21apt-get10:48:012
8079212849,40sleep28092-21apt-get10:48:012
30323212739,73sleep20-21swapper/208:33:122
15782212547,41sleep215737-21apt-get07:48:292
9156212122,26sleep30-21swapper/309:08:023
3924212123,77sleep13972-21latency_hist07:12:541
3924212123,77sleep13972-21latency_hist07:12:541
32536212126,60sleep232509-21latency_hist10:22:532
11681212128,51sleep20-21swapper/210:58:192
17722212022,14sleep20-21swapper/207:53:182
11382212013,48sleep211505-21timerwakeupswit09:13:232
24255211926,48sleep20-21swapper/208:13:212
509521185,56sleep20-21swapper/212:21:392
3640211817,73sleep00-21swapper/010:33:130
8437211725,70sleep18578-21grep07:23:241
14255211713,87sleep314293-21/usr/sbin/munin09:23:173
14255211713,87sleep314293-21/usr/sbin/munin09:23:173
25218211618,56sleep20-21swapper/211:43:062
25218211618,56sleep20-21swapper/211:43:062
10554211313,84sleep110656-21latency_hist07:32:531
27698211212,82sleep127765-21systemctl08:24:191
16057211215,16sleep00-21swapper/011:13:110
214221098,65sleep22220-21sed10:28:142
1501421070,7sleep00-21swapper/007:45:110
9393210612,79sleep19451-21irqstats09:08:131
1359421068,63sleep213602-21munin-run09:22:522
1359421068,63sleep213602-21munin-run09:22:522
9004210517,65sleep18072-21/usr/sbin/munin10:48:251
9004210517,65sleep18072-21/usr/sbin/munin10:48:251
16767210533,55sleep216911-21latency_hist09:32:532
28289210312,75sleep328308-21apt-get11:53:003
28289210312,75sleep328308-21apt-get11:53:003
29575210229,54sleep129613-21modprobe11:57:511
29575210229,54sleep129613-21modprobe11:57:511
14410210218,65sleep014479-21grep11:08:100
1664221019,77sleep316770-21/usr/sbin/munin11:13:223
1624210019,54sleep11708-21find07:03:261
2738229810,72sleep027498-21sed11:48:210
2386129815,65sleep123949-21/usr/sbin/munin08:13:121
1532129814,68sleep315471-21cat07:47:543
248029619,62sleep10-21swapper/112:13:221
675229512,66sleep36845-21cat07:18:213
59802956,72sleep20-21swapper/208:57:562
2519329415,29sleep30-21swapper/311:43:033
2519329415,29sleep30-21swapper/311:43:033
2367129419,14sleep20-21swapper/211:38:052
122929417,54sleep01376-21grep12:08:200
122929417,54sleep01376-21grep12:08:200
2806329320,14sleep20-21swapper/210:07:592
651529217,60sleep06561-21ntpq08:59:110
158562926,48sleep20-21swapper/209:28:212
1445029116,60sleep114532-21mailstats07:43:191
537629018,20sleep30-21swapper/310:38:173
3171629018,11sleep20-21swapper/212:03:132
859028925,12sleep20-21swapper/209:04:102
1876528914,59sleep018837-21latency_hist11:22:520
574128839,24sleep20-21swapper/212:23:122
319128718,52sleep13343-21sed10:32:541
2935228715,52sleep129386-21awk10:12:501
3184028614,55sleep331953-21cat10:18:223
1511928625,32sleep00-21swapper/009:27:500
1511928625,32sleep00-21swapper/009:27:500
1286228617,49sleep012754-21/usr/sbin/munin11:03:100
901828510,14sleep20-21swapper/212:33:182
2191228515,45sleep122118-21latency_hist08:07:551
1517128510,41sleep215252-21latency_hist09:27:522
1517128510,41sleep215252-21latency_hist09:27:522
745328410,62sleep27495-21latency_hist07:22:532
626228315,54sleep16312-21unin-run10:42:511
401528323,36sleep00-21swapper/008:52:530
304412830,7sleep10-21swapper/111:58:181
3098928220,47sleep331051-21sed10:17:533
1791828211,58sleep317934-21open_inodes11:18:193
183872815,11sleep20-21swapper/209:37:542
772528023,38sleep07875-21sed12:28:230
2806628017,44sleep027579-21apt-get10:08:010
2570728019,40sleep025791-21grep11:43:200
2570728019,40sleep025791-21grep11:43:200
57582795,21sleep20-21swapper/207:16:552
2999527919,38sleep330095-21latency_hist08:32:533
468827812,51sleep14751-21sendmail-msp10:37:501
2326227810,54sleep123285-21fschecks_count09:53:121
2048127816,43sleep020513-21ntp_kernel_pll_09:43:170
2579327714,47sleep125194-21/usr/sbin/munin11:43:211
2579327714,47sleep125194-21/usr/sbin/munin11:43:211
2515127719,39sleep025193-21awk09:58:160
166622776,17sleep00-21swapper/009:32:470
166622776,17sleep00-21swapper/009:32:470
154112776,12sleep20-21swapper/211:10:392
1130327613,45sleep111309-21diskstats10:58:101
1025527619,15sleep10-21swapper/110:53:211
1025527619,15sleep10-21swapper/110:53:211
612227522,33sleep06141-21apt-get07:18:020
605127516,44sleep15528-21apt-get08:58:031
2905927516,38sleep029202-21unixbench_singl11:53:220
2905927516,38sleep029202-21unixbench_singl11:53:220
246627512,15sleep10-21swapper/108:47:531
2269827518,29sleep30-21swapper/308:08:333
3063327413,36sleep330778-21grep11:58:213
569027318,25sleep00-21swapper/008:57:540
2549827319,34sleep10-21swapper/108:18:131
2252427311,13sleep00-21swapper/011:33:190
2105327323,23sleep30-21swapper/311:28:213
2091327317,29sleep30-21swapper/308:03:203
1901227315,38sleep119074-21wc09:38:221
1171227314,45sleep011786-21tr10:58:210
496227215,42sleep35008-21cat07:13:173
2639127217,29sleep10-21swapper/110:03:091
232682726,18sleep00-21swapper/008:10:120
1253927217,36sleep312509-21latency_hist11:02:523
205272718,27sleep10-21swapper/111:28:021
1671727120,32sleep10-21swapper/109:32:521
1671727120,32sleep10-21swapper/109:32:521
135282718,51sleep10-21swapper/109:22:141
135282718,51sleep10-21swapper/109:22:141
665227011,42sleep16744-21fschecks_time10:43:121
2715727010,31sleep00-21swapper/008:23:250
2182427014,40sleep021911-21apt-config08:07:540
878326913,41sleep10-21swapper/112:33:131
2756626910,47sleep227589-21munin-run10:07:512
2683926920,29sleep30-21swapper/311:48:083
2237126922,27sleep00-21swapper/009:48:220
216012699,31sleep30-21swapper/309:48:013
1727326914,38sleep317274-21/usr/sbin/munin07:53:033
1413726915,24sleep10-21swapper/109:23:151
1413726915,24sleep10-21swapper/109:23:151
1362626911,38sleep0367-21snmpd09:22:540
1362626911,38sleep0367-21snmpd09:22:540
1130526923,27sleep10-21swapper/109:13:221
1098626917,23sleep00-21swapper/010:57:530
1098626917,23sleep00-21swapper/010:57:530
89372685,30sleep20-21swapper/207:27:372
78726818,28sleep30-21swapper/310:23:193
1908726819,23sleep10-21swapper/111:23:101
1862326816,23sleep00-21swapper/009:38:120
3195826722,23sleep00-21swapper/010:18:230
2809926710,28sleep30-21swapper/310:08:083
153426722,26sleep00-21swapper/007:03:250
3022226614,24sleep00-21swapper/010:13:220
2681226615,32sleep326902-21grep10:03:193
141926614,35sleep31508-21/usr/sbin/munin08:43:153
652726511,41sleep36528-21sshd12:25:113
432126516,21sleep30-21swapper/308:53:083
316426514,31sleep33232-21/usr/sbin/munin07:08:243
316426514,31sleep33232-21/usr/sbin/munin07:08:243
3030926512,38sleep030315-21df_inode08:33:100
292492655,27sleep20-21swapper/211:53:402
292492655,27sleep20-21swapper/211:53:402
262982655,29sleep20-21swapper/208:18:542
1705326516,18sleep00-21swapper/009:33:120
904226415,31sleep09080-21/usr/sbin/munin12:33:200
36526413,37sleep0393-21sh10:23:070
3050626421,27sleep130565-21/usr/sbin/munin08:33:151
226602646,44sleep20-21swapper/209:49:582
1914726418,20sleep30-21swapper/309:38:243
1722926425,21sleep30-21swapper/309:33:153
1574126421,27sleep315237-21apt-get07:48:023
1352226423,23sleep10-21swapper/111:03:231
104422645,10sleep20-21swapper/207:32:052
1001826424,21sleep30-21swapper/310:53:153
1001826424,21sleep30-21swapper/310:53:153
521826315,27sleep00-21swapper/010:38:140
470726316,23sleep10-21swapper/108:53:181
2367226319,25sleep30-21swapper/311:38:063
1571826318,27sleep115737-21apt-get07:48:001
2443426221,17sleep00-21swapper/009:57:510
2258626217,19sleep30-21swapper/311:33:223
1547826213,31sleep015651-21apt-config07:47:550
1278826216,23sleep00-21swapper/007:38:170
119312620,7sleep10-21swapper/107:36:111
87926120,22sleep10-21swapper/110:23:211
652726111,25sleep06528-21/usr/sbin/munin10:43:020
3212326111,22sleep30-21swapper/312:03:223
2714926112,30sleep327124-21apt-get08:23:203
2422626117,27sleep124288-21/usr/sbin/munin11:38:201
2347626118,27sleep1307-21systemd-logind08:12:521
2066526113,19sleep10-21swapper/109:43:221
1744326117,23sleep10-21swapper/109:33:211
1280226120,20sleep30-21swapper/309:18:203
1280226120,20sleep30-21swapper/309:18:203
342926012,32sleep33497-21sh10:33:083
3180726016,27sleep331803-21apt-get08:38:063
903225921,20sleep30-21swapper/312:33:193
75952596,17sleep30-21swapper/309:03:023
451125922,23sleep04578-21switchtime12:18:210
2916925916,20sleep00-21swapper/008:28:170
234325917,24sleep00-21swapper/012:13:080
831025810,31sleep00-21swapper/007:23:230
456125815,26sleep04555-21apt-get07:13:050
317825816,24sleep00-21swapper/010:32:530
2664525814,28sleep026674-21munin-run08:22:520
2403825817,19sleep00-21swapper/008:13:150
1112825816,21sleep00-21swapper/009:13:160
1050225819,24sleep310539-21modprobe07:32:513
270525715,25sleep32700-21apt-get08:48:073
270425720,20sleep00-21swapper/007:08:130
270425720,20sleep00-21swapper/007:08:130
2501325718,25sleep124726-21/usr/sbin/munin09:58:121
1989425711,35sleep00-21swapper/007:59:420
185322578,40sleep10-21swapper/111:20:311
922125613,24sleep09239-21apt-get07:28:040
82322568,12sleep10-21swapper/109:03:241
284725615,18sleep30-21swapper/312:13:323
2802625618,21sleep128049-21modprobe11:52:501
187225614,20sleep30-21swapper/310:28:073
1730125613,22sleep00-21swapper/007:53:080
1568825617,23sleep315780-21memory09:28:183
1568825617,23sleep315780-21memory09:28:183
1333025618,19sleep30-21swapper/311:03:213
1119325616,20sleep30-21swapper/309:13:203
964525516,20sleep00-21swapper/009:08:200
85952556,35sleep00-21swapper/009:04:120
769225514,24sleep37688-21apt-get07:23:063
711125514,19sleep30-21swapper/312:28:103
67332554,14sleep10-21swapper/112:27:471
187725519,18sleep00-21swapper/008:43:240
1191825516,20sleep00-21swapper/007:36:110
1105525519,23sleep30-21swapper/307:33:163
393325413,24sleep13927-21apt-get12:18:061
2973025414,26sleep129766-21fschecks_count10:13:111
1940525413,17sleep30-21swapper/311:23:163
1863625411,27sleep118753-21latency_hist07:57:531
1469625417,22sleep114735-21ntpq11:08:161
1223625412,18sleep10-21swapper/107:37:541
321125316,23sleep13286-21run-parts08:48:221
2868125314,23sleep128676-21apt-get08:28:021
1887425314,22sleep318867-21apt-get07:58:073
740625213,22sleep11-21systemd07:22:521
610125212,23sleep36053-21apt-get08:58:333
594425211,22sleep00-21swapper/012:23:180
3027025217,20sleep00-21swapper/011:58:140
2511925215,17sleep30-21swapper/309:58:153
1241825212,24sleep312410-21apt-get07:38:083
2024725111,18sleep10-21swapper/108:02:541
810125012,22sleep38092-21apt-get10:48:063
810125012,22sleep38092-21apt-get10:48:063
3144725013,22sleep131444-21apt-get12:03:041
3210224915,21sleep132130-21users10:18:241
2419524912,22sleep324218-21ntp_states08:13:183
233862495,12sleep30-21swapper/309:53:143
655124810,21sleep36547-21apt-get10:43:053
2169724711,19sleep121707-21df_inode09:48:071
2138124714,18sleep30-21swapper/309:47:533
2055224710,18sleep00-21swapper/011:28:060
1459624711,15sleep30-21swapper/311:08:143
254302467,16sleep30-21swapper/308:18:113
70982425,13sleep10-21swapper/112:28:081
3191323910,14sleep00-21swapper/012:03:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional