You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-02 - 03:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Jun 02, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32221236926,321sleep032258-21fschecks_count23:58:380
21020234617,312sleep121051-21/usr/sbin/munin21:38:421
13178229814,268sleep113181-21ntpq21:13:511
31757228611,263sleep331882-21cat20:28:523
21248228630,191sleep321270-21apt-get19:58:203
251722282102,98sleep30-21swapper/320:08:413
954722596,218sleep3363-21snmpd22:45:103
954722596,218sleep3363-21snmpd22:45:103
10985225227,214sleep310998-21awk22:48:533
9807223315,19sleep30-21swapper/319:18:513
6837222955,108sleep3194-21jbd2/mmcblk2p2-20:53:473
1321222212,194sleep31332-21fschecks_count00:03:383
4118221810,190sleep34195-21latency22:28:433
212192212100,68sleep321257-21processes21:38:483
5496220416,101sleep35546-21fschecks_count22:33:393
16491219721,82sleep30-21swapper/321:23:463
8615219417,151sleep30-21swapper/320:58:513
8615219417,151sleep30-21swapper/320:58:513
2305421837,164sleep323100-21sed23:28:423
2305421837,164sleep323100-21sed23:28:423
16353218280,26sleep30-21swapper/323:08:233
16042218222,133sleep116196-21sed19:38:521
2871621788,150sleep30-21swapper/323:48:203
9028217420,136sleep19022-21apt-get00:28:341
8905217252,104sleep39048-21latency_hist19:18:233
19957217270,82sleep30-21swapper/323:18:413
26373216610,94sleep326414-21grep23:38:473
22300216513,90sleep30-21swapper/321:43:373
22300216513,90sleep30-21swapper/321:43:373
1414021629,139sleep314178-21ntpq19:33:453
13485215452,86sleep313531-21unin-run19:33:213
8888215311,118sleep28903-21latency_hist19:18:222
1139721420,30sleep211401-21unixbench_multi00:33:512
30272214150,56sleep330311-21unin-run23:53:203
25589214011,112sleep325603-21modprobe23:38:193
6024213927,94sleep30-21swapper/300:18:393
6024213927,94sleep30-21swapper/300:18:393
2466521398,94sleep324683-21grep20:08:213
13126213628,85sleep313127-21munin-run22:58:223
5330213437,75sleep35356-21latency_hist19:08:213
10334213433,89sleep310421-21wc00:33:213
28311213332,85sleep328333-21kernelversion20:18:423
13219213132,85sleep313294-21mailstats21:13:533
8780213050,20sleep30-21swapper/300:28:203
21732213016,97sleep00-21swapper/023:23:500
19775213022,84sleep00-21swapper/023:18:370
6739212938,70sleep30-21swapper/319:11:453
1574921289,106sleep315796-21ntp_offset19:38:463
20962212515,16sleep00-21swapper/021:38:410
4042212416,86sleep34187-21unixbench_multi20:43:533
4408212311,95sleep24409-21sshd20:45:382
21397212329,80sleep321457-21idleruntime23:23:413
5591212233,74sleep35719-21cat20:48:503
12890212124,76sleep012902-21modprobe22:58:190
31955212014,89sleep032085-21latency_hist23:58:220
5724211914,80sleep30-21swapper/300:18:233
5724211914,80sleep30-21swapper/300:18:233
2183211915,84sleep00-21swapper/020:38:460
3189421188,70sleep331953-21date23:58:213
25863211829,23sleep30-21swapper/321:53:503
117221179,16sleep00-21swapper/022:18:460
5364211515,21sleep00-21swapper/019:08:220
24430211535,39sleep30-21swapper/321:48:503
23154211414,81sleep122762-21/usr/sbin/munin23:28:441
23154211414,81sleep122762-21/usr/sbin/munin23:28:441
2156821140,6sleep20-21swapper/223:23:432
2269321139,91sleep321776-21/usr/sbin/munin19:59:253
31885211117,76sleep032028-21head20:28:530
19285211125,15sleep30-21swapper/321:33:393
1157021108,20sleep311585-21awk19:23:523
310721087,43sleep30-21swapper/320:43:213
26436210827,55sleep126455-21apt-get20:13:321
19778210811,79sleep219923-21latency_hist19:53:222
18380210717,40sleep018474-21memory23:13:450
19018210626,15sleep30-21swapper/319:48:513
15845210528,15sleep30-21swapper/321:23:223
10003210512,29sleep310045-21ntp_kernel_pll_21:03:483
501210336,24sleep30-21swapper/322:18:213
2889521039,67sleep30-21swapper/322:03:433
2889521039,67sleep30-21swapper/322:03:433
22495210223,62sleep1225-21systemd-journal23:28:191
13418210211,77sleep313436-21apt-get22:58:303
11809210116,22sleep30-21swapper/300:38:013
820821005,11sleep30-21swapper/322:42:473
820821005,11sleep30-21swapper/322:42:473
26971210019,64sleep30-21swapper/321:58:353
26971210019,64sleep30-21swapper/321:58:353
69329925,59sleep30-21swapper/320:33:483
3050429819,16sleep30-21swapper/322:08:443
238929820,60sleep22481-21tune2fs22:23:412
708529618,65sleep27203-21cat20:53:542
32182948,35sleep30-21swapper/300:08:463
276032945,36sleep30-21swapper/320:16:453
254729417,58sleep22614-21latency_hist00:08:212
1008629418,37sleep10-21swapper/122:48:351
560929323,40sleep00-21swapper/019:08:390
1850129316,63sleep218540-21sed23:13:472
16272927,65sleep11210-21/usr/sbin/munin00:03:441
186792918,65sleep30-21swapper/323:13:523
1664229110,62sleep30-21swapper/323:08:393
1641129013,59sleep116450-21memory21:23:441
12182908,26sleep31252-21ps22:18:493
456628916,54sleep04711-21/usr/sbin/munin22:28:520
2617228928,34sleep126201-21kworker/u8:020:13:191
2548328916,59sleep225561-21fschecks_time21:53:412
188872895,10sleep30-21swapper/321:33:193
969028820,41sleep19755-21fschecks_time21:03:421
2381928611,60sleep023825-21ntp_states20:03:460
935028511,58sleep09441-21users22:43:540
935028511,58sleep09441-21users22:43:540
1893928520,46sleep219067-21latency_hist21:33:222
949128411,57sleep29531-21ps00:28:482
33532847,32sleep30-21swapper/322:25:443
2861328424,35sleep20-21swapper/222:03:382
2861328424,35sleep20-21swapper/222:03:382
202612840,8sleep20-21swapper/219:53:412
959328116,22sleep00-21swapper/019:18:480
2051528124,27sleep20-21swapper/221:38:232
1820328129,31sleep20-21swapper/219:48:212
1820128123,30sleep10-21swapper/123:13:421
305752807,35sleep30-21swapper/320:24:483
2736728023,32sleep10-21swapper/121:58:451
2736728023,32sleep10-21swapper/121:58:451
880927916,18sleep00-21swapper/019:18:200
2302327930,24sleep10-21swapper/121:43:551
2302327930,24sleep10-21swapper/121:43:551
115482796,60sleep00-21swapper/000:34:430
98927811,36sleep31-21systemd00:03:213
3033327822,29sleep10-21swapper/123:53:221
2956327718,27sleep10-21swapper/120:23:291
2877027715,47sleep228912-21latency_hist23:48:232
2539727711,49sleep10-21swapper/121:53:361
2306627712,52sleep00-21swapper/020:03:130
1284427712,52sleep2225-21systemd-journal22:57:442
121472769,35sleep30-21swapper/321:09:493
121472769,35sleep30-21swapper/321:09:493
1094527620,38sleep111039-21/usr/sbin/munin19:23:421
589527511,53sleep00-21swapper/020:49:440
171022755,12sleep30-21swapper/319:43:423
2259427421,30sleep20-21swapper/219:59:242
2967227316,26sleep00-21swapper/020:23:360
1074427321,25sleep10-21swapper/100:33:391
3168927216,39sleep231756-21awk20:28:512
266052727,56sleep00-21swapper/021:57:430
2291527214,41sleep023021-21ls21:43:540
2291527214,41sleep023021-21ls21:43:540
201427217,38sleep12043-21awk22:23:201
1852727220,25sleep00-21swapper/019:48:400
450427127,25sleep10-21swapper/119:03:481
159282716,11sleep00-21swapper/023:04:000
2792227022,26sleep10-21swapper/123:43:501
2646327013,26sleep20-21swapper/220:13:372
1595427014,41sleep016041-21sort19:38:510
914326916,34sleep29217-21find22:43:512
914326916,34sleep29217-21find22:43:512
705926913,24sleep00-21swapper/022:38:370
705926913,24sleep00-21swapper/022:38:370
451726916,22sleep20-21swapper/200:13:422
451726916,22sleep20-21swapper/200:13:422
3006626912,40sleep230124-21/usr/sbin/munin20:23:462
2829026920,26sleep20-21swapper/222:03:222
2829026920,26sleep20-21swapper/222:03:222
189226912,39sleep21896-21diskstats20:38:402
1755026916,25sleep10-21swapper/119:43:521
1692826917,34sleep116973-21ntpq23:08:451
743426823,28sleep10-21swapper/122:38:461
743426823,28sleep10-21swapper/122:38:461
30326819,33sleep2313-21/usr/sbin/munin20:33:392
3026126817,34sleep230269-21modprobe23:53:192
2083726817,27sleep20-21swapper/221:38:392
1417226818,29sleep014376-21apt-config21:18:220
409426718,22sleep20-21swapper/200:13:222
3253426714,32sleep232582-21grep23:58:442
2653926717,34sleep026437-21/usr/sbin/munin20:13:390
249526713,36sleep02511-21ls00:08:190
2448026722,24sleep10-21swapper/123:33:401
1656026712,39sleep016609-21/usr/sbin/munin23:08:360
1568626716,25sleep20-21swapper/223:03:542
3121126613,37sleep131220-21df_inode20:28:371
2567026619,24sleep20-21swapper/223:38:222
201092667,47sleep00-21swapper/021:34:090
1655826614,21sleep20-21swapper/221:23:492
162226613,36sleep11762-21latency_hist20:38:221
1290526612,37sleep113014-21cat22:58:201
869026515,31sleep08817-21/usr/sbin/munin20:58:520
869026515,31sleep08817-21/usr/sbin/munin20:58:520
54212656,15sleep10-21swapper/120:48:481
2134826512,35sleep121514-21latency_hist19:58:221
1610626511,40sleep116108-21sshd23:06:191
1445326516,22sleep20-21swapper/219:33:522
1398026513,32sleep214067-21sh21:18:202
84126412,34sleep2936-21perl22:18:402
401926416,23sleep10-21swapper/122:28:411
2903826413,26sleep00-21swapper/022:03:480
2903826413,26sleep00-21swapper/022:03:480
2851526421,19sleep10-21swapper/120:18:491
579126316,31sleep25846-21/usr/sbin/munin19:08:452
382926319,17sleep00-21swapper/020:43:500
33622637,12sleep00-21swapper/022:25:490
3180726313,33sleep131850-21systemctl23:58:191
2324726316,28sleep00-21swapper/023:28:470
2324726316,28sleep00-21swapper/023:28:470
1222126320,26sleep112214-21apt-get19:28:341
805726218,21sleep10-21swapper/100:23:501
52026214,26sleep00-21swapper/022:18:220
2716826213,23sleep00-21swapper/023:43:200
2319026218,23sleep20-21swapper/223:28:452
2319026218,23sleep20-21swapper/223:28:452
1689426213,35sleep0363-21snmpd19:43:350
148342626,10sleep00-21swapper/019:35:480
1313626214,29sleep00-21swapper/019:28:540
1247626216,30sleep112620-21cat21:13:221
1247626216,30sleep112620-21cat21:13:221
1046426218,27sleep210618-21latency_hist19:23:222
936226118,23sleep10-21swapper/121:03:211
936226118,23sleep10-21swapper/121:03:211
741026119,19sleep10-21swapper/119:13:281
607226118,21sleep10-21swapper/119:08:521
562126111,32sleep15678-21irqstats22:33:411
1346426120,26sleep113532-21sshd22:58:441
1064826118,26sleep210667-21apt-get19:23:302
1377126018,25sleep113765-21apt-get19:33:351
46092597,37sleep04657-21sed00:13:440
46092597,37sleep04657-21sed00:13:440
42792596,11sleep10-21swapper/120:44:001
3122125916,17sleep20-21swapper/223:53:522
1122825919,18sleep00-21swapper/019:23:490
997325821,19sleep20-21swapper/221:03:472
825125820,21sleep20-21swapper/220:58:422
825125820,21sleep20-21swapper/220:58:422
81402586,12sleep20-21swapper/200:23:512
3075025815,28sleep030874-21sed22:08:490
2897425819,17sleep10-21swapper/122:03:451
2897425819,17sleep10-21swapper/122:03:451
2754425815,19sleep00-21swapper/023:43:400
227712586,12sleep10-21swapper/119:59:271
1167025810,36sleep011748-21mailstats21:08:490
1167025810,36sleep011748-21mailstats21:08:490
1092425814,26sleep00-21swapper/021:08:210
1043625817,21sleep10-21swapper/119:23:211
567925716,20sleep20-21swapper/222:33:422
300372575,40sleep00-21swapper/023:50:270
278825722,15sleep10-21swapper/100:08:361
2384725714,25sleep023841-21apt-get21:48:340
2333625715,25sleep123355-21apt-get20:03:311
1989825716,17sleep20-21swapper/223:18:402
1476925713,22sleep20-21swapper/223:03:212
1453225716,25sleep014526-21apt-get21:18:340
724025616,19sleep00-21swapper/000:23:210
724025616,19sleep00-21swapper/000:23:210
707425620,21sleep27123-21fschecks_count22:38:392
707425620,21sleep27123-21fschecks_count22:38:392
3213525613,28sleep132202-21df23:58:351
3046925612,20sleep20-21swapper/222:08:432
2620325617,23sleep226221-21munin-run20:13:202
1943225614,26sleep119445-21irqstats21:33:411
1668725612,27sleep016761-21find21:23:510
111582565,10sleep20-21swapper/222:50:582
91052559,31sleep00-21swapper/000:28:370
536325511,19sleep20-21swapper/220:48:452
2466225515,24sleep024701-21memory23:33:430
2462425517,23sleep224653-21munin-run20:08:202
2424525517,21sleep20-21swapper/221:48:442
231725514,22sleep10-21swapper/120:38:511
1844425511,26sleep118511-21/usr/sbin/munin19:48:371
1818225516,22sleep018258-21sed21:28:480
1655725514,25sleep216553-21apt-get23:08:342
747825413,23sleep07474-21apt-get00:23:330
311712545,11sleep10-21swapper/122:11:121
287325417,19sleep00-21swapper/000:08:390
2696525415,23sleep026966-21apt-get21:58:320
2696525415,23sleep026966-21apt-get21:58:320
2559025416,16sleep00-21swapper/020:08:500
1998725413,23sleep019983-21apt-get19:53:320
1808125411,28sleep118088-21sshd21:28:451
1346325414,24sleep013436-21apt-get22:58:430
1124825417,21sleep20-21swapper/221:08:392
1124825417,21sleep20-21swapper/221:08:392
71342538,28sleep10-21swapper/100:22:371
71342538,28sleep10-21swapper/100:22:371
54072535,10sleep10-21swapper/100:15:431
54072535,10sleep10-21swapper/100:15:431
3268425315,22sleep032680-21apt-get20:33:330
3230625316,16sleep20-21swapper/222:13:522
3189325317,16sleep10-21swapper/122:13:431
275825315,22sleep12764-21ntp_states22:23:481
2227325315,23sleep222290-21apt21:43:312
2227325315,23sleep222290-21apt21:43:312
1234225314,15sleep20-21swapper/219:28:392
86152529,18sleep10-21swapper/122:43:381
86152529,18sleep10-21swapper/122:43:381
593025213,23sleep25924-21apt-get00:18:342
593025213,23sleep25924-21apt-get00:18:342
577925215,19sleep00-21swapper/022:33:440
2802125212,26sleep028042-21apt-get20:18:310
2635325212,24sleep126364-21ntp_states23:38:451
2619425213,21sleep226276-21/usr/sbin/munin23:38:422
2384825214,21sleep123915-21df21:48:351
1437925110,13sleep10-21swapper/121:18:231
1035125114,18sleep00-21swapper/022:48:420
196912509,15sleep10-21swapper/123:18:341
1538725012,21sleep214508-21/usr/sbin/munin21:19:542
646724813,17sleep00-21swapper/020:53:350
49872486,30sleep20-21swapper/219:04:382
654624713,19sleep16552-21diskstats20:53:401
516524616,15sleep20-21swapper/222:33:222
1533924411,18sleep20-21swapper/219:38:322
280442439,19sleep228040-21apt-get20:18:332
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional