You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 17:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Mar 17, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3101729528,933sleep07255-1kworker/0:0H07:48:220
2352294136,859sleep32501-21tail07:58:313
12806279510,750sleep312842-21ntpq11:58:233
14879269018,653sleep115016-21/usr/sbin/munin08:38:311
1771124239,397sleep317682-21latency_hist10:33:023
1771124239,397sleep317682-21latency_hist10:33:023
20328239916,368sleep220334-21ntpq07:13:272
16488238222,341sleep016578-21grep10:28:180
16488238222,341sleep016578-21grep10:28:180
11085234518,216sleep311122-21fschecks_count08:28:193
11085234518,216sleep311122-21fschecks_count08:28:193
30245232654,238sleep330325-21sed09:28:283
20129227511,248sleep020203-21find10:38:280
20317224966,67sleep320338-21fschecks_count12:23:183
20317224966,67sleep320338-21fschecks_count12:23:183
25546223627,193sleep125615-21/usr/sbin/munin09:13:281
25546223627,193sleep125615-21/usr/sbin/munin09:13:281
79882226102,20sleep30-21swapper/308:18:173
15335222146,55sleep315399-21mailstats10:23:333
28040221795,67sleep327647-21/usr/sbin/munin07:38:233
30546220616,130sleep30-21swapper/307:48:003
213921947,13sleep00-21swapper/007:58:270
14675219166,106sleep30-21swapper/308:38:293
6816218911,167sleep36828-21cpu09:58:123
250962189105,32sleep325163-21fschecks_time09:13:183
250962189105,32sleep325163-21fschecks_time09:13:183
23938217010,122sleep324031-21timerandwakeup12:33:323
22835216643,103sleep30-21swapper/310:48:203
25910216522,19sleep30-21swapper/307:33:023
22411216451,97sleep322488-21systemctl09:03:283
26673215510,132sleep326765-21cat07:33:313
26673215510,132sleep326765-21cat07:33:313
2166021538,133sleep321680-21cat09:03:003
9052215244,25sleep30-21swapper/310:03:303
3687215211,125sleep33729-21proc_pri08:03:273
18093215110,37sleep30-21swapper/310:33:183
6070214823,36sleep36103-21fschecks_count11:38:203
27103214811,122sleep327251-21latency_hist11:03:023
19452214730,101sleep319581-21awk12:18:303
19452214730,101sleep319581-21awk12:18:303
14462214668,29sleep30-21swapper/312:03:283
23464214530,104sleep323501-21ps07:23:293
20335214414,116sleep320378-21processes07:13:283
19144214415,114sleep219173-21nfs4_client08:53:262
19144214415,114sleep219173-21nfs4_client08:53:262
31812214055,25sleep30-21swapper/309:33:293
28608213948,79sleep328647-21wc09:23:283
30825213513,69sleep330889-21if_eth011:13:203
10001213527,52sleep30-21swapper/308:23:303
9736213413,16sleep30-21swapper/310:08:023
4783213311,110sleep34832-21ntpq11:33:233
3235521338,110sleep332504-21date07:53:013
28558213214,18sleep30-21swapper/311:07:593
77521307,68sleep3901-21cat07:53:343
1399221307,13sleep00-21swapper/012:03:120
25939212822,79sleep07255-1kworker/0:0H10:58:190
20206212422,77sleep120344-21grep10:38:291
25859212316,91sleep225908-21latency_hist07:33:012
10863212320,85sleep110930-21df11:53:161
2193921208,38sleep30-21swapper/307:18:283
26280211815,84sleep026341-21sendmail-msp09:17:590
26280211815,84sleep026341-21sendmail-msp09:17:590
20034211811,93sleep320038-21cron08:57:593
20034211811,93sleep320038-21cron08:57:593
810421168,96sleep38187-21sed11:43:293
16244211651,23sleep316323-21switchtime08:43:293
16244211651,23sleep316323-21switchtime08:43:293
3263211550,26sleep33309-21ntp_offset11:28:263
3263211550,26sleep33309-21ntp_offset11:28:263
29680211416,82sleep229712-21proc_pri07:43:272
17666211457,40sleep30-21swapper/312:13:293
26466211317,80sleep126493-21awk07:33:251
26466211317,80sleep126493-21awk07:33:251
30397211120,78sleep030495-21latency_hist11:13:010
14499211118,78sleep214543-21ntpq08:38:242
12815211016,79sleep012885-21sort08:33:210
17023210829,60sleep217180-21grep07:03:302
4991210711,81sleep15196-21modprobe09:53:021
23036210615,77sleep223085-21unin-run12:32:592
23036210615,77sleep223085-21unin-run12:32:592
19593210620,69sleep30-21swapper/310:38:153
18364210518,68sleep018370-21modprobe12:17:580
7239210420,65sleep17247-21modprobe11:42:581
28604210413,76sleep228607-21ntpq09:23:262
2158021048,50sleep30-21swapper/310:43:283
16851210318,65sleep016974-21latency_hist12:13:020
10402210315,62sleep210472-21munin-run11:53:012
14074210210,23sleep10-21swapper/112:03:171
27905210118,66sleep128009-21cat09:23:011
2186821009,74sleep30-21swapper/312:28:193
2186821009,74sleep30-21swapper/312:28:193
16062210017,68sleep216100-21/usr/sbin/munin08:43:242
16062210017,68sleep216100-21/usr/sbin/munin08:43:242
57892995,61sleep30-21swapper/308:09:533
100329812,48sleep30-21swapper/309:38:303
535129712,72sleep15449-21cat08:08:301
2574129712,57sleep10-21swapper/110:58:111
253232977,10sleep30-21swapper/310:56:403
2425729716,64sleep124316-21grep10:53:191
23172978,45sleep30-21swapper/309:43:263
23172978,45sleep30-21swapper/309:43:263
108302969,23sleep30-21swapper/311:53:103
812829513,54sleep20-21swapper/210:03:002
275732950,7sleep20-21swapper/207:38:012
275732950,7sleep20-21swapper/207:38:012
2069729516,64sleep020720-21pluginstate12:23:280
2069729516,64sleep020720-21pluginstate12:23:280
3124029412,65sleep131214-21/usr/sbin/munin09:33:151
1431729417,58sleep214334-21munin-run10:22:592
2979429311,66sleep029885-21sed07:43:290
1050329326,17sleep30-21swapper/311:53:023
3100429216,61sleep231019-21netstat11:13:232
1918829215,18sleep30-21swapper/307:12:593
1844529217,61sleep218503-21date12:18:002
2996529112,17sleep00-21swapper/009:28:210
1040929016,51sleep010488-21mailstats10:08:270
2911828915,13sleep30-21swapper/311:08:353
1696728917,52sleep217036-21/usr/sbin/munin10:28:282
1696728917,52sleep217036-21/usr/sbin/munin10:28:282
1680528920,47sleep316837-21/usr/sbin/munin07:03:273
1563728915,60sleep115742-21fschecks_time12:08:181
1327528925,40sleep113420-21unixbench_singl08:33:301
1848228811,17sleep30-21swapper/308:52:583
125432887,33sleep30-21swapper/308:33:123
2356128723,44sleep023652-21cut09:08:180
2340028712,59sleep023442-21tr07:23:260
1230328721,12sleep30-21swapper/310:13:313
69312866,68sleep00-21swapper/011:39:010
1230728622,47sleep012397-21latency_hist08:33:000
1230728622,47sleep012397-21latency_hist08:33:000
321852855,44sleep30-21swapper/311:18:043
3129528514,52sleep230974-21/usr/sbin/munin07:49:412
322528419,24sleep20-21swapper/208:03:142
2769328416,50sleep027731-21/usr/sbin/munin11:03:230
669028312,54sleep26724-21ntp_kernel_err08:13:242
391728316,48sleep23430-21/usr/sbin/munin09:48:262
25182835,18sleep30-21swapper/311:27:213
25182835,18sleep30-21swapper/311:27:213
2165828310,60sleep021750-21/usr/sbin/munin10:43:290
1481428321,40sleep114344-21apt-get10:23:101
2824828215,51sleep128344-21switchtime07:38:291
2121628210,55sleep021338-21wc07:18:010
1796528223,37sleep118109-21unixbench_singl08:48:281
294828113,54sleep12993-21fschecks_count11:28:201
294828113,54sleep12993-21fschecks_count11:28:201
2785928115,49sleep127939-21sed11:03:281
2696728116,30sleep00-21swapper/009:18:240
981528011,39sleep10-21swapper/108:23:241
2346628010,15sleep30-21swapper/309:08:133
1254727919,28sleep20-21swapper/208:33:152
206442788,47sleep20-21swapper/208:58:232
1765327724,33sleep117675-21munin-run07:08:001
761427629,31sleep07631-21uniq11:43:170
1745627612,39sleep10-21swapper/112:13:231
119202766,41sleep10-21swapper/110:13:251
112827620,22sleep00-21swapper/009:38:310
882827512,46sleep28837-21munin-run11:48:002
766527516,42sleep30-21swapper/308:18:013
224022746,12sleep00-21swapper/007:19:020
332927326,24sleep20-21swapper/211:28:292
332927326,24sleep20-21swapper/211:28:292
3047627322,32sleep10-21swapper/107:47:581
2969327322,28sleep10-21swapper/109:28:141
2430827326,26sleep20-21swapper/207:28:002
1123127319,27sleep10-21swapper/110:13:011
373127214,42sleep03808-21readlink08:03:280
324372727,13sleep20-21swapper/211:18:202
3034627220,31sleep230374-21munin-run11:12:582
2582627217,31sleep20-21swapper/210:58:162
1941827216,25sleep10-21swapper/108:53:331
1941827216,25sleep10-21swapper/108:53:331
1098427227,25sleep20-21swapper/211:53:212
3144227116,16sleep20-21swapper/209:33:202
2271127120,34sleep122736-21basename07:22:591
2056027117,25sleep00-21swapper/008:58:210
1928527121,26sleep10-21swapper/110:38:001
1760427119,29sleep017632-21munin-run07:07:590
992127023,26sleep20-21swapper/210:08:132
63202707,11sleep30-21swapper/309:55:423
539227014,38sleep05456-21/usr/sbin/munin09:53:210
484827018,35sleep04989-21latency_hist09:53:010
3230327022,24sleep00-21swapper/011:18:180
2238927018,29sleep10-21swapper/110:48:021
1830027013,18sleep10-21swapper/107:08:251
181592709,12sleep00-21swapper/010:33:190
3262626915,29sleep10-21swapper/107:53:201
313226925,26sleep00-21swapper/011:28:230
313226925,26sleep00-21swapper/011:28:230
2659726930,19sleep30-21swapper/309:18:143
2145426918,26sleep20-21swapper/207:18:152
165882696,11sleep30-21swapper/312:10:183
148082697,45sleep00-21swapper/010:23:050
140152696,11sleep30-21swapper/310:19:083
697426814,41sleep07038-21basename09:58:180
692626818,25sleep00-21swapper/008:13:300
298726810,19sleep10-21swapper/108:03:011
732126726,23sleep10-21swapper/109:58:271
608426720,22sleep00-21swapper/008:13:010
3007026715,27sleep20-21swapper/209:28:232
1476426718,23sleep00-21swapper/008:38:300
143326722,23sleep20-21swapper/207:58:012
246342669,13sleep00-21swapper/007:28:190
1999726619,25sleep20-21swapper/212:23:012
1999726619,25sleep20-21swapper/212:23:012
1458126621,27sleep114785-21gpgv10:23:021
1378426616,25sleep20-21swapper/212:03:022
968226515,24sleep20-21swapper/210:08:012
962626515,28sleep20-21swapper/208:23:212
262126512,38sleep22685-21cat11:28:012
262126512,38sleep22685-21cat11:28:012
2199026515,35sleep122011-21fschecks_count09:03:181
967326410,39sleep19681-21systemd-run10:08:001
516126415,28sleep20-21swapper/211:33:302
2945526412,37sleep029526-21cat09:28:010
240362649,14sleep00-21swapper/012:33:330
1913326421,27sleep119135-21ntp_states12:18:241
1913326421,27sleep119135-21ntp_states12:18:241
1860726421,17sleep00-21swapper/007:08:320
167102647,11sleep00-21swapper/008:45:120
167102647,11sleep00-21swapper/008:45:120
1521326422,25sleep215222-21/usr/sbin/munin10:23:282
1352426417,21sleep00-21swapper/010:18:250
496026324,19sleep00-21swapper/011:33:280
353926314,34sleep13544-21/usr/sbin/munin09:48:171
3147826322,23sleep10-21swapper/107:49:441
3123926315,28sleep031232-21apt-get09:33:140
91392626,15sleep10-21swapper/111:48:151
545026223,24sleep20-21swapper/208:08:312
2975726214,26sleep129854-21timerandwakeup11:09:181
236372626,12sleep00-21swapper/010:49:510
184272628,13sleep20-21swapper/207:08:302
161332625,11sleep10-21swapper/108:43:271
161332625,11sleep10-21swapper/108:43:271
44542615,11sleep30-21swapper/309:49:403
2686726117,22sleep20-21swapper/209:18:222
2465426110,34sleep224634-21grep12:37:592
2034826113,29sleep220426-21users10:38:302
139682615,12sleep20-21swapper/212:03:102
107112616,13sleep00-21swapper/008:28:000
2388026020,22sleep10-21swapper/110:52:581
2072126019,21sleep10-21swapper/112:23:291
2072126019,21sleep10-21swapper/112:23:291
131326016,28sleep11314-21cpu11:23:151
131326016,28sleep11314-21cpu11:23:151
1187426015,28sleep011919-21/usr/sbin/munin10:13:240
437725910,12sleep10-21swapper/111:33:091
349525920,25sleep10-21swapper/108:03:211
2216825918,27sleep022180-21/usr/sbin/munin09:03:210
1108025919,25sleep211084-21diskstats08:28:182
1108025919,25sleep211084-21diskstats08:28:182
256925822,22sleep02611-21systemd-run11:27:590
256925822,22sleep02611-21systemd-run11:27:590
2528125817,21sleep20-21swapper/209:13:212
2528125817,21sleep20-21swapper/209:13:212
2120925817,23sleep20-21swapper/210:43:182
1866825813,27sleep218812-21grep10:33:292
1128025813,18sleep00-21swapper/011:53:270
982025712,21sleep00-21swapper/011:48:300
84725713,23sleep10-21swapper/109:38:281
632125714,26sleep16314-21apt-get08:13:151
30552576,10sleep20-21swapper/209:46:362
30552576,10sleep20-21swapper/209:46:362
839125617,22sleep10-21swapper/108:18:271
2618225614,23sleep00-21swapper/007:33:190
2618225614,23sleep00-21swapper/007:33:190
210262566,10sleep10-21swapper/107:16:371
2042825616,24sleep120435-21diskstats08:58:171
1853525614,16sleep20-21swapper/208:53:012
166825621,20sleep20-21swapper/209:43:012
1284525616,24sleep012890-21ntp_states11:58:240
835125514,25sleep28358-21ntp_states08:18:242
827425513,21sleep00-21swapper/008:18:220
2464125511,15sleep10-21swapper/107:28:201
2416925514,24sleep024163-21apt-get10:53:140
190825514,24sleep01902-21apt-get09:43:150
190825514,24sleep01902-21apt-get09:43:150
1140025514,25sleep111446-21ntp_offset08:28:251
1140025514,25sleep111446-21ntp_offset08:28:251
920525414,16sleep00-21swapper/010:03:310
606425416,22sleep20-21swapper/211:38:192
2454125412,25sleep224575-21/usr/sbin/munin10:53:242
2346525415,23sleep223461-21apt-get09:08:122
2178025415,22sleep121774-21apt-get12:28:131
2178025415,22sleep121774-21apt-get12:28:131
2142425411,30sleep213441-21snmpd12:27:092
2142425411,30sleep213441-21snmpd12:27:092
1934625420,17sleep00-21swapper/008:53:320
1934625420,17sleep00-21swapper/008:53:320
1375725418,20sleep10-21swapper/112:03:011
520625317,21sleep24730-21apt-get09:53:072
3269125320,18sleep10-21swapper/111:18:271
3203325312,17sleep00-21swapper/011:18:010
2822625320,20sleep10-21swapper/109:23:171
2812725313,24sleep228133-21ntp_states07:38:262
2271025312,16sleep20-21swapper/209:03:322
1729425315,23sleep017298-21diskstats08:48:150
2767025215,20sleep027665-21apt-get07:38:120
2757325212,22sleep227668-21/usr/sbin/munin11:03:212
1889925212,16sleep00-21swapper/012:18:190
1889925212,16sleep00-21swapper/012:18:190
763225110,16sleep10-21swapper/111:43:181
3127125114,16sleep10-21swapper/111:13:301
223825113,22sleep12277-21memory09:43:241
223825113,22sleep12277-21memory09:43:241
1593725015,18sleep20-21swapper/212:08:232
1193125013,23sleep211932-21ntp_states10:13:262
326802499,15sleep00-21swapper/007:53:230
2215224816,16sleep00-21swapper/012:28:240
2215224816,16sleep00-21swapper/012:28:240
16432486,32sleep20-21swapper/207:58:092
230822479,14sleep10-21swapper/107:23:191
2475624614,20sleep11-21systemd09:12:591
2458424615,16sleep10-21swapper/112:37:581
234222468,14sleep20-21swapper/210:48:312
1646224611,20sleep116469-21df_inode10:28:151
1646224611,20sleep116469-21df_inode10:28:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional