You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-12 - 07:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Aug 12, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
126892297414,2946sleep012727-21kworker/u8:120:15:150
18003270215,670sleep018054-21/usr/sbin/munin22:15:200
1316625027,483sleep2367-21snmpd22:00:082
1316625027,483sleep2367-21snmpd22:00:082
9203246711,438sleep29345-21grep21:45:302
13160240413,376sleep113166-21ntpq20:15:371
28018238310,334sleep228088-21unin-run21:05:002
237802269120,130sleep223463-21/usr/sbin/munin00:15:242
26914225553,183sleep226945-21munin-plugin-st22:44:582
6739225158,147sleep26867-21latency_hist21:40:012
15434223726,182sleep00-21swapper/023:50:100
1630522349,206sleep216378-21mailstats20:25:312
1208622248,200sleep212146-21processes21:55:282
25368221318,137sleep225447-21latency_hist19:15:012
9019220514,167sleep19178-21latency_hist20:05:031
32677219518,62sleep20-21swapper/219:35:282
2765221947,45sleep20-21swapper/219:20:182
32592219254,79sleep232738-21grep23:00:292
30230218817,154sleep030300-21mailstats00:35:290
1666321888,52sleep20-21swapper/222:10:222
23669217223,14sleep20-21swapper/220:50:182
29392217045,107sleep229096-21/usr/sbin/munin19:25:222
10904216531,119sleep210955-21sort20:10:172
11809215935,105sleep20-21swapper/223:39:582
11809215935,105sleep20-21swapper/223:39:582
983021559,127sleep29337-21/usr/sbin/munin20:05:362
7240215233,58sleep27274-21idleruntime-cro19:59:592
27938215213,59sleep027946-21/usr/sbin/munin19:20:240
22438215214,122sleep221962-21apt-get19:05:082
22438215214,122sleep221962-21apt-get19:05:082
9632215021,105sleep09751-21cat23:30:310
17011214949,72sleep317008-21apt-get23:55:133
22398214731,100sleep20-21swapper/200:10:272
1548214435,67sleep20-21swapper/221:20:352
13859214415,94sleep00-21swapper/023:45:210
14543214340,65sleep214628-21latency20:20:222
3155213813,110sleep23218-21latency19:45:212
32087213664,54sleep20-21swapper/221:15:302
32087213664,54sleep20-21swapper/221:15:302
19501213543,78sleep218573-21/usr/sbin/munin00:00:382
2648621335,11sleep20-21swapper/219:15:292
31633213125,60sleep20-21swapper/223:00:002
15323213137,50sleep20-21swapper/222:05:292
15323213137,50sleep20-21swapper/222:05:292
6172212748,23sleep20-21swapper/223:20:202
22498212550,18sleep20-21swapper/222:30:122
22498212550,18sleep20-21swapper/222:30:122
1961821249,79sleep219679-21if_eth022:20:192
28453212315,95sleep128501-21latency00:30:211
31038212017,90sleep331081-21tr19:30:243
1701621207,76sleep217063-21cpu23:55:162
2649211927,63sleep30-21swapper/319:45:023
1545821199,96sleep215454-21apt-get23:50:122
25826211824,68sleep125976-21unixbench_singl00:20:311
25826211824,68sleep125976-21unixbench_singl00:20:311
494221176,80sleep20-21swapper/221:32:462
2200921175,28sleep20-21swapper/220:45:102
10158211720,80sleep00-21swapper/021:50:170
25914211516,21sleep00-21swapper/022:40:200
6585211415,71sleep36748-21unixbench_multi23:20:293
22435211423,75sleep322511-21sendmail_mailtr00:10:283
28500211213,82sleep328509-21munin-run22:50:003
2972211116,78sleep33041-21fschecks_time19:45:183
26947211113,22sleep326963-21munin-run22:44:593
2851621108,66sleep228544-21cpu21:05:172
30806210924,65sleep130951-21unixbench_singl21:10:331
28153210921,70sleep328229-21wc19:20:313
458021088,67sleep24645-21ls19:50:182
2684621081,91sleep20-21swapper/200:25:212
2684621081,91sleep20-21swapper/200:25:212
2439210840,20sleep20-21swapper/223:10:022
17285210714,78sleep117358-21sed23:55:241
20970210556,33sleep20-21swapper/222:25:122
29657210212,74sleep129728-21sensors_temp19:25:301
19514210152,34sleep20-21swapper/220:35:332
19514210152,34sleep20-21swapper/220:35:332
18446210118,34sleep20-21swapper/220:32:422
1829621017,58sleep218298-21ntp_states22:15:272
12655210132,21sleep20-21swapper/220:15:092
22542210034,50sleep022458-21apt-get19:06:030
22542210034,50sleep022458-21apt-get19:06:030
273829925,55sleep32249-21apt-get23:10:113
2981729734,41sleep20-21swapper/200:35:182
2102029715,37sleep20-21swapper/200:05:422
184342975,81sleep00-21swapper/020:32:320
1367529723,14sleep20-21swapper/223:45:152
881229627,50sleep28957-21sed23:30:012
2631829618,31sleep20-21swapper/200:24:402
2631829618,31sleep20-21swapper/200:24:402
153229613,70sleep01604-21sort23:05:320
1395429624,53sleep313960-21modprobe20:19:593
1239829610,23sleep20-21swapper/223:40:222
1239829610,23sleep20-21swapper/223:40:222
633929511,66sleep06354-21/usr/sbin/munin19:55:250
48532947,12sleep00-21swapper/023:15:290
440529213,57sleep20-21swapper/223:15:192
1938329211,64sleep019421-21/usr/sbin/munin20:35:310
1938329211,64sleep019421-21/usr/sbin/munin20:35:310
230729136,37sleep20-21swapper/221:25:152
2048829120,14sleep20-21swapper/220:40:132
1060629143,34sleep20-21swapper/221:50:272
479129014,58sleep30-21swapper/323:15:283
2845929029,45sleep228492-21munin-run22:49:592
2313729019,50sleep122499-21/usr/sbin/munin22:30:321
2313729019,50sleep122499-21/usr/sbin/munin22:30:321
921028936,34sleep20-21swapper/223:30:212
2426128919,51sleep124054-21/usr/sbin/munin22:35:201
2426128919,51sleep124054-21/usr/sbin/munin22:35:201
295928813,56sleep10-21swapper/119:45:161
1559928821,45sleep115744-21latency_hist20:25:021
78982875,35sleep20-21swapper/221:40:422
3029528713,60sleep030296-21/usr/sbin/munin22:55:140
2563828721,45sleep325708-21ls00:20:293
2563828721,45sleep325708-21ls00:20:293
142728721,44sleep01443-21fschecks_count19:40:180
1230128716,54sleep112458-21sed21:55:311
3107828614,17sleep20-21swapper/221:11:432
273228636,12sleep20-21swapper/223:10:062
2531828616,54sleep025336-21idleruntime-cro19:14:590
1565228621,44sleep315667-21fschecks_time23:50:333
219372856,15sleep10-21swapper/100:10:141
174152850,7sleep30-21swapper/320:30:163
51962840,7sleep30-21swapper/321:35:003
983428316,52sleep026715-1kworker/0:1H21:50:010
2930428312,50sleep20-21swapper/222:50:322
2915728319,47sleep129260-21cat21:05:521
2524528330,13sleep20-21swapper/222:38:482
2524528330,13sleep20-21swapper/222:38:482
2469428318,41sleep024817-21timerandwakeup22:35:310
2469428318,41sleep024817-21timerandwakeup22:35:310
2988828124,31sleep30-21swapper/321:10:023
443028014,51sleep04523-21switchtime21:30:300
2725828018,46sleep027322-21modprobe19:19:590
2380628020,35sleep323761-21latency_hist19:10:013
2380628020,35sleep323761-21latency_hist19:10:013
206728016,48sleep02075-21grep21:25:010
116012804,12sleep30-21swapper/323:37:163
116012804,12sleep30-21swapper/323:37:163
3274127910,55sleep3302-21users23:00:303
3195227940,25sleep131844-21/usr/sbin/munin23:00:161
223527925,35sleep32249-21apt-get23:09:593
2863827822,14sleep20-21swapper/200:30:262
2723327814,44sleep327310-21threads00:25:303
2723327814,44sleep327310-21threads00:25:303
2545027820,29sleep30-21swapper/319:15:023
2514027821,42sleep223822-21cyclictest20:55:152
2855227720,27sleep30-21swapper/300:30:233
1072027713,47sleep310863-21modprobe20:10:013
3190627630,11sleep20-21swapper/219:34:432
290127624,29sleep00-21swapper/023:10:530
2650427611,42sleep00-21swapper/021:00:020
869327529,11sleep20-21swapper/220:03:182
85127525,30sleep10-21swapper/121:20:221
56842750,12sleep00-21swapper/021:35:220
173527527,11sleep20-21swapper/219:40:252
3106527420,24sleep30-21swapper/300:40:003
1227627413,46sleep012343-21sort23:40:200
1227627413,46sleep012343-21sort23:40:200
1793027323,29sleep30-21swapper/322:15:193
823427216,41sleep38250-21munin-plugin-st21:44:593
3068827214,19sleep00-21swapper/021:10:310
2570827214,39sleep125817-21munin-node19:15:041
248002720,7sleep10-21swapper/119:10:301
1952227225,23sleep10-21swapper/122:20:181
1638127219,33sleep116399-21fschecks_count22:10:171
43427115,33sleep3462-21munin-run21:19:593
43427115,33sleep3462-21munin-run21:19:593
2923627118,28sleep10-21swapper/122:50:311
2571327125,29sleep10-21swapper/120:55:301
2327727112,43sleep023421-21cat00:15:010
1462927115,25sleep30-21swapper/320:20:233
955527020,31sleep39628-21ls23:30:303
67522708,15sleep00-21swapper/023:20:300
2949227022,33sleep029528-21ntpq19:25:240
267327022,30sleep30-21swapper/321:25:243
2484127016,37sleep324949-21sed00:20:013
452426922,26sleep10-21swapper/121:30:311
250832698,17sleep30-21swapper/322:36:433
250832698,17sleep30-21swapper/322:36:433
1887126921,27sleep30-21swapper/300:00:263
944626818,22sleep00-21swapper/020:05:260
782126818,27sleep10-21swapper/123:25:231
325526811,39sleep03300-21ntp_kernel_err19:45:230
3097026819,28sleep10-21swapper/122:55:311
276192686,15sleep00-21swapper/000:26:280
276192686,15sleep00-21swapper/000:26:280
1900526819,26sleep30-21swapper/320:35:213
1900526819,26sleep30-21swapper/320:35:213
154422688,14sleep10-21swapper/123:50:111
558926718,22sleep10-21swapper/123:19:581
356226716,30sleep13592-21munin-plugin-st21:29:591
2276026713,27sleep00-21swapper/022:30:230
2276026713,27sleep00-21swapper/022:30:230
2112926723,24sleep10-21swapper/120:40:281
635526614,34sleep16400-21/usr/sbin/munin19:55:261
2133626615,30sleep321363-21netstat22:25:213
2108826617,30sleep00-21swapper/000:05:440
2095726611,30sleep10-21swapper/100:05:411
102226619,29sleep11051-21munin-run19:39:591
742726514,26sleep00-21swapper/023:25:100
566926520,21sleep30-21swapper/321:35:223
1521326515,19sleep00-21swapper/022:05:270
1521326515,19sleep00-21swapper/022:05:270
813926422,22sleep00-21swapper/020:00:330
2283926424,20sleep10-21swapper/119:06:361
2283926424,20sleep10-21swapper/119:06:361
2268926415,36sleep022810-21timerwakeupswit20:45:320
1827226416,30sleep018339-21sendmail-msp23:59:590
1365026417,27sleep313669-21apt-get23:45:123
3005726320,20sleep30-21swapper/322:55:013
2819226324,21sleep00-21swapper/021:05:020
2720726316,32sleep127225-21cpu22:45:181
2230326318,24sleep30-21swapper/320:45:233
2223326318,29sleep122282-21unin-run22:29:591
2571226219,29sleep00-21swapper/000:20:300
2571226219,29sleep00-21swapper/000:20:300
2400226219,26sleep00-21swapper/000:15:300
1222326214,29sleep012300-21/usr/sbin/munin21:55:300
1154526222,20sleep10-21swapper/120:10:321
3221426120,25sleep332209-21apt-get19:35:133
2798326115,31sleep127996-21apt-get21:04:591
2476126117,27sleep124790-21munin-run00:19:591
1514826117,27sleep115187-21ntp_kernel_pll_22:05:241
1514826117,27sleep115187-21ntp_kernel_pll_22:05:241
144526119,23sleep11548-21perl19:40:191
1346626113,23sleep00-21swapper/022:00:220
1346626113,23sleep00-21swapper/022:00:220
1343426117,21sleep30-21swapper/323:45:003
1343426117,21sleep30-21swapper/323:45:003
1263426119,20sleep10-21swapper/123:40:301
1263426119,20sleep10-21swapper/123:40:301
81152605,10sleep00-21swapper/021:43:260
53852606,12sleep00-21swapper/019:51:010
2858126015,26sleep028620-21grep00:30:240
1650326011,34sleep316554-21if_eth022:10:193
2888425917,20sleep10-21swapper/119:25:011
249925913,20sleep00-21swapper/021:25:210
1999825916,17sleep30-21swapper/322:20:283
1714425914,19sleep10-21swapper/120:30:021
121162595,12sleep00-21swapper/020:14:270
1061625923,21sleep10-21swapper/120:09:591
879225815,31sleep18838-21sed21:45:211
51925817,20sleep10-21swapper/121:20:011
51925817,20sleep10-21swapper/121:20:011
3066925816,26sleep030663-21apt-get19:30:130
2700225813,20sleep10-21swapper/100:25:241
2700225813,20sleep10-21swapper/100:25:241
2590925818,25sleep326037-21/usr/sbin/munin20:55:323
231625822,20sleep10-21swapper/123:10:011
1606825813,19sleep30-21swapper/320:25:243
1557325815,23sleep00-21swapper/020:25:010
3265025710,19sleep00-21swapper/019:35:260
3049225717,25sleep330528-21ntp_states21:10:263
214392575,12sleep30-21swapper/300:07:173
2138125713,14sleep30-21swapper/320:40:303
1913525715,24sleep119184-21irqstats20:35:231
1913525715,24sleep119184-21irqstats20:35:231
112042577,10sleep10-21swapper/121:53:211
1031525715,18sleep10-21swapper/123:35:001
451925615,27sleep14579-21/usr/sbin/munin19:50:171
2681725619,20sleep10-21swapper/122:43:441
2433425620,20sleep30-21swapper/320:50:323
2414025617,18sleep10-21swapper/120:50:301
1924325616,22sleep00-21swapper/000:00:360
851825513,26sleep38515-21apt-get21:45:113
50525515,25sleep0367-21snmpd21:20:000
50525515,25sleep0367-21snmpd21:20:000
497625511,17sleep30-21swapper/319:50:273
3079925514,25sleep30-21swapper/322:55:293
2887125514,22sleep028957-21perl22:50:210
2234325517,16sleep30-21swapper/322:30:013
2234325517,16sleep30-21swapper/322:30:013
1942525513,25sleep019419-21apt-get22:20:130
1791725516,19sleep10-21swapper/122:15:171
75925413,25sleep3755-21apt-get21:20:163
625425415,18sleep30-21swapper/319:55:243
2896425414,25sleep028998-21ntp_states21:05:470
198412548,11sleep10-21swapper/100:04:441
936025314,23sleep39356-21apt-get20:05:213
68825314,21sleep0718-21munin-run23:04:590
1637425314,23sleep016380-21diskstats22:10:160
1493525315,21sleep10-21swapper/123:49:581
1188925313,17sleep00-21swapper/023:40:000
1188925313,17sleep00-21swapper/023:40:000
756125213,23sleep37557-21apt-get20:00:163
620125212,23sleep16301-21users21:35:321
2747825215,16sleep00-21swapper/022:45:250
2625325216,20sleep30-21swapper/319:15:223
2489925214,22sleep00-21swapper/020:55:010
2048625215,19sleep00-21swapper/020:40:110
1520525214,22sleep315211-21ntp_states22:05:263
1520525214,22sleep315211-21ntp_states22:05:263
1453725212,18sleep30-21swapper/322:05:023
1453725212,18sleep30-21swapper/322:05:023
3112325110,17sleep10-21swapper/119:30:261
2145525119,17sleep00-21swapper/022:25:260
171302509,17sleep00-21swapper/020:30:010
1114825012,23sleep111226-21find23:35:291
1114825012,23sleep111226-21find23:35:291
2981124913,21sleep129816-21diskstats00:35:171
2921024914,17sleep30-21swapper/319:25:193
1295724913,17sleep30-21swapper/322:00:013
716424818,18sleep11-21systemd23:24:591
753024611,19sleep37537-21diskstats23:25:173
28172458,13sleep10-21swapper/123:10:501
1862524413,17sleep118644-21munin-run20:34:591
131962447,20sleep10-21swapper/122:00:151
131962447,20sleep10-21swapper/122:00:151
1034024416,15sleep30-21swapper/321:50:203
2691924311,12sleep30-21swapper/321:00:213
2203124010,15sleep10-21swapper/120:45:131
73962366,14sleep10-21swapper/121:40:251
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional