You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-17 - 08:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Wed Jun 17, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
311322755016,7521sleep118178-21kworker/1:019:48:581
15172269218,657sleep315247-21/usr/sbin/munin00:07:533
15172269218,657sleep315247-21/usr/sbin/munin00:07:533
17666268815,653sleep317789-21/usr/sbin/munin19:07:593
32402261919,579sleep332548-21/usr/sbin/munin19:53:003
170625697,550sleep21750-21ntpq21:42:542
27407255512,528sleep119773-21kworker/1:123:03:011
22594236015,241sleep222633-21wc19:22:562
1502723607,340sleep215016-21apt-get00:07:402
1502723607,340sleep215016-21apt-get00:07:402
230492347310,23sleep323051-21sshd22:52:013
4248231415,282sleep04254-21cat20:07:460
2221722598,238sleep1363-21snmpd19:22:471
245152243105,39sleep224565-21fschecks_count21:12:472
245152243105,39sleep224565-21fschecks_count21:12:472
13511223117,193sleep213533-21kernelversion00:02:522
9487222320,187sleep29523-21ntp_kernel_err22:07:482
30549220410,178sleep230604-21mailstats19:47:542
30549220410,178sleep230604-21mailstats19:47:542
26385220015,87sleep226433-21ntpq21:17:512
26385220015,87sleep226433-21ntpq21:17:512
13208219416,20sleep20-21swapper/220:37:272
6475219020,145sleep20-21swapper/221:57:552
7116218613,159sleep17197-21find23:42:561
17597218418,144sleep217664-21switchtime19:07:582
9574217513,146sleep19347-21latency_hist23:52:291
26568217534,26sleep20-21swapper/219:37:292
8880217410,147sleep28965-21latency_hist22:07:282
20761217212,147sleep220818-21find22:42:562
27661216919,137sleep227768-21tune2fs21:22:472
23286216815,139sleep223350-21ps00:32:552
13002216813,142sleep213015-21cat00:02:272
156802166118,24sleep20-21swapper/222:27:462
7955216112,134sleep28090-21cat20:17:572
7200215679,63sleep27338-21grep23:42:572
5064215311,19sleep20-21swapper/223:37:452
17317215315,108sleep20-21swapper/220:48:002
10371215030,27sleep20-21swapper/223:53:032
1600214915,122sleep21713-21latency_hist23:27:282
17162214812,123sleep20-21swapper/200:12:572
17162214812,123sleep20-21swapper/200:12:572
142921469,12sleep10-21swapper/121:42:491
21719214414,117sleep221766-21ntpq21:02:542
23891213521,98sleep123948-21interrupts19:27:471
16385213413,105sleep316353-21latency_hist20:47:293
23492133100,13sleep20-21swapper/220:02:392
2111213312,98sleep20-21swapper/223:27:482
2111213312,98sleep20-21swapper/223:27:482
10373213311,98sleep20-21swapper/220:27:382
21809213213,19sleep216816-21cyclictest00:27:582
11115212923,91sleep111175-40find23:57:251
28317212513,97sleep028336-21uname23:07:480
26891212518,94sleep226905-21grep19:37:452
15435212521,86sleep015469-21ntp_kernel_pll_00:07:580
15435212521,86sleep015469-21ntp_kernel_pll_00:07:580
18354212425,77sleep018494-21latency_hist19:12:290
6386212317,86sleep20-21swapper/220:12:572
19439212350,19sleep20-21swapper/222:37:592
27764212284,23sleep2307-21in:imuxsock23:07:252
32240212113,95sleep232289-21awk21:37:412
20224212111,17sleep20-21swapper/220:57:532
18409212026,70sleep118425-21modprobe22:37:261
8824211913,94sleep28840-21awk20:22:372
2959921198,17sleep20-21swapper/223:12:382
17606211979,25sleep20-21swapper/222:32:552
32492211813,18sleep20-21swapper/223:22:272
25387211810,92sleep025450-21sort19:32:470
22035211810,93sleep222087-21if_eth022:47:482
18753211879,24sleep218833-21switchtime20:52:572
23247211616,82sleep20-21swapper/221:07:502
3066021148,93sleep230656-21apt-get21:32:412
23949211311,16sleep20-21swapper/219:27:482
1899221128,91sleep219060-21df19:12:432
13612211210,84sleep20-21swapper/220:37:472
5474211112,84sleep20-21swapper/220:12:282
29201211113,85sleep029262-21fschecks_time21:27:460
27942211118,73sleep327984-21sed21:22:533
20981211110,16sleep20-21swapper/200:27:262
208421086,89sleep21-21systemd20:02:272
1876210812,81sleep01911-21df23:27:420
1876210812,81sleep01911-21df23:27:420
4432210610,80sleep20-21swapper/221:52:382
24239210517,70sleep1225-21systemd-journal21:12:291
24239210517,70sleep1225-21systemd-journal21:12:291
2087921056,14sleep20-21swapper/219:17:472
2087921056,14sleep20-21swapper/219:17:472
2362221047,12sleep20-21swapper/222:52:492
18258210410,78sleep20-21swapper/200:17:452
8828210322,67sleep08842-21idleruntime-cro22:07:260
1393521036,12sleep20-21swapper/222:22:292
11674210314,73sleep111675-21apt-get23:57:381
31352210244,17sleep20-21swapper/219:51:502
2921121028,10sleep20-21swapper/223:11:052
1182521026,13sleep20-21swapper/222:13:052
902121017,11sleep20-21swapper/223:48:312
464521006,11sleep20-21swapper/223:37:202
464521006,11sleep20-21swapper/223:37:202
404821000,11sleep10-21swapper/120:07:301
2646210016,69sleep02701-21latency_hist21:47:280
18965210016,66sleep118982-21/usr/sbin/munin19:12:381
1621721007,82sleep20-21swapper/220:45:582
4912997,77sleep20-21swapper/219:57:272
40582996,12sleep20-21swapper/221:51:212
413029820,58sleep33660-21apt-get20:07:373
2163929815,66sleep321643-21/usr/sbin/munin00:27:513
135272987,10sleep20-21swapper/222:19:312
2252929716,60sleep122530-21cron00:32:251
1870429713,64sleep118705-21munin-node22:37:381
261162967,10sleep20-21swapper/223:01:082
1986029617,56sleep119946-21fschecks_time20:57:441
2859329323,50sleep328564-21/usr/sbin/munin19:42:433
2859329323,50sleep328564-21/usr/sbin/munin19:42:433
1446629311,22sleep00-21swapper/000:07:010
303222916,10sleep20-21swapper/221:31:482
1981229021,45sleep119874-21grep00:22:491
1591429028,34sleep00-21swapper/019:02:520
419328913,20sleep00-21swapper/021:52:280
1599628724,31sleep20-21swapper/219:02:542
1516228720,51sleep015239-21tune2fs20:42:450
2312628617,54sleep123163-21latency_hist22:52:271
166642864,11sleep30-21swapper/322:30:013
2334628531,38sleep323348-21/usr/sbin/munin22:52:373
1380528519,35sleep10-21swapper/122:22:281
1042228515,50sleep110511-21apt-config22:12:271
1037828515,46sleep00-21swapper/022:12:260
2611028414,49sleep126206-21grep21:17:461
2611028414,49sleep126206-21grep21:17:461
1171328420,40sleep311714-21apt-get23:58:043
2350128317,43sleep123659-21sed00:32:581
308072825,11sleep00-21swapper/023:16:360
1999528117,27sleep00-21swapper/000:22:520
628128013,52sleep16354-21cat23:42:281
1572128029,33sleep315778-21latency19:02:473
3087227912,52sleep330929-21/usr/sbin/munin21:32:503
206492798,53sleep00-21swapper/019:17:410
206492798,53sleep00-21swapper/019:17:410
937127819,41sleep39462-21apt-config23:52:273
2540527816,30sleep30-21swapper/322:57:543
2415827822,27sleep30-21swapper/300:37:283
627527617,42sleep06287-21kernelversion21:57:490
569627618,40sleep15727-21modprobe21:57:261
2858227612,30sleep30-21swapper/323:07:553
2856127614,41sleep10-21swapper/119:42:371
2856127614,41sleep10-21swapper/119:42:371
1716827616,45sleep017239-21sensors_temp20:47:580
576327524,29sleep00-21swapper/020:12:420
376327521,29sleep30-21swapper/320:07:293
2765027522,34sleep127540-21/usr/sbin/munin21:22:461
2663427522,30sleep00-21swapper/021:17:570
2663427522,30sleep00-21swapper/021:17:570
1333427527,26sleep30-21swapper/300:02:493
1295027527,25sleep00-21swapper/000:02:250
982527424,29sleep00-21swapper/023:52:360
577527418,30sleep30-21swapper/321:57:283
237332736,20sleep00-21swapper/000:33:010
1582127320,30sleep10-21swapper/119:02:491
86227217,39sleep3940-21ls23:22:573
3252227217,32sleep30-21swapper/323:22:283
3128027216,36sleep131354-21fschecks_time23:17:461
288612728,14sleep00-21swapper/019:43:000
288612728,14sleep00-21swapper/019:43:000
260712725,57sleep00-21swapper/023:00:330
225672726,54sleep00-21swapper/021:07:200
90442716,13sleep00-21swapper/023:48:490
855827126,22sleep30-21swapper/320:22:263
2360927118,38sleep123767-21awk21:07:571
151322715,10sleep00-21swapper/022:25:140
110127117,26sleep10-21swapper/119:57:501
3145127018,28sleep00-21swapper/019:52:280
2252727019,22sleep30-21swapper/319:22:533
2173627019,27sleep00-21swapper/000:27:560
2012827020,29sleep30-21swapper/300:22:553
1123027011,34sleep311369-21sed23:57:283
375526912,41sleep33782-21netstat23:32:493
375526912,41sleep33782-21netstat23:32:493
3116626814,24sleep00-21swapper/023:17:370
2168626815,28sleep10-21swapper/121:02:531
1399226819,23sleep30-21swapper/320:37:563
702526712,20sleep00-21swapper/023:42:550
65226716,34sleep1681-21netstat23:22:501
2471226712,37sleep124813-21sed22:57:281
183826715,37sleep31872-21cpu23:27:413
183826715,37sleep31872-21cpu23:27:413
767026614,32sleep30-21swapper/322:02:493
384326615,26sleep04045-21cat20:07:300
306592669,44sleep10-21swapper/121:32:401
1951026617,22sleep10-21swapper/100:22:281
1908326615,36sleep119092-21vmstat20:53:001
326226513,36sleep33311-21ntp_kernel_pll_21:47:513
3095226524,24sleep30-21swapper/323:17:273
2926326514,37sleep129090-21/usr/sbin/munin21:27:471
2696926519,20sleep00-21swapper/023:02:560
2677726516,25sleep30-21swapper/323:02:503
2153526515,31sleep021559-21munin-run22:47:260
1561526523,27sleep10-21swapper/100:08:031
1561526523,27sleep10-21swapper/100:08:031
1312626513,24sleep00-21swapper/022:17:590
51262646,11sleep10-21swapper/120:08:361
3274126417,29sleep032735-21apt-get23:22:410
3206126415,21sleep00-21swapper/019:52:530
2680726414,25sleep30-21swapper/319:37:423
861226319,22sleep10-21swapper/120:22:281
2938326317,24sleep30-21swapper/321:27:493
2577726322,22sleep30-21swapper/321:17:283
2577726322,22sleep30-21swapper/321:17:283
191332637,11sleep00-21swapper/022:37:530
892126224,20sleep00-21swapper/020:22:430
2532226218,25sleep10-21swapper/122:57:521
2036826224,20sleep10-21swapper/122:42:461
2003826222,23sleep30-21swapper/322:42:283
3006826113,28sleep10-21swapper/123:12:531
2811826113,23sleep00-21swapper/021:22:580
257732617,11sleep10-21swapper/119:32:561
1885526114,29sleep318984-21/usr/sbin/munin00:17:583
1214126112,35sleep312198-21if_eth020:32:473
1130426119,20sleep30-21swapper/322:12:543
1086726119,28sleep010929-21processes20:27:560
473126021,20sleep10-21swapper/123:37:281
473126021,20sleep10-21swapper/123:37:281
470726014,30sleep04801-21/usr/sbin/munin21:52:490
2656126016,25sleep326629-21/usr/sbin/munin21:17:563
2656126016,25sleep326629-21/usr/sbin/munin21:17:563
2483126010,36sleep024878-21ntpq21:12:530
2483126010,36sleep024878-21ntpq21:12:530
99702595,11sleep10-21swapper/120:25:391
2851325910,26sleep10-21swapper/123:07:531
279112598,13sleep10-21swapper/119:40:271
2376725920,18sleep30-21swapper/319:27:453
1580525912,19sleep00-21swapper/022:27:480
3111225818,18sleep00-21swapper/021:32:550
2188925818,20sleep30-21swapper/319:22:283
2188925818,20sleep30-21swapper/319:22:283
162162588,10sleep10-21swapper/120:45:571
1410325812,19sleep10-21swapper/120:37:581
121625813,29sleep31258-21/usr/sbin/munin19:57:523
857725714,25sleep08611-21munin-run20:22:270
261592576,10sleep30-21swapper/319:33:143
2192125722,18sleep30-21swapper/321:02:593
1848925714,28sleep318564-21latency_hist22:37:283
1586325718,18sleep10-21swapper/122:27:491
708925612,27sleep37090-21sshd21:59:583
359625617,18sleep00-21swapper/023:32:460
359625617,18sleep00-21swapper/023:32:460
2417825616,18sleep30-21swapper/321:12:273
2349125611,24sleep00-21swapper/021:07:560
2084725615,22sleep10-21swapper/119:17:461
2084725615,22sleep10-21swapper/119:17:461
183625615,25sleep11831-21apt-get23:27:391
183625615,25sleep11831-21apt-get23:27:391
179492569,22sleep10-21swapper/100:17:281
179492569,22sleep10-21swapper/100:17:281
1175025616,17sleep10-21swapper/120:32:291
1148225622,19sleep00-21swapper/022:12:590
108625616,24sleep01095-21munin-run21:42:270
1815425514,21sleep00-21swapper/020:52:390
1805325517,19sleep00-21swapper/022:33:010
1695625513,24sleep00-21swapper/000:12:510
1695625513,24sleep00-21swapper/000:12:510
76322545,11sleep30-21swapper/323:45:333
496825413,24sleep04963-21apt-get23:37:390
2414925412,15sleep00-21swapper/019:27:550
2213125413,25sleep022126-21apt-get19:22:420
1959725414,17sleep00-21swapper/019:12:570
1855325416,22sleep00-21swapper/000:17:510
1364425414,24sleep113687-21/usr/sbin/munin00:02:551
1136325413,25sleep111369-21ntp_states22:12:561
99992536,36sleep30-21swapper/320:26:013
815525316,17sleep10-21swapper/122:02:591
2337525315,20sleep00-21swapper/022:52:420
2002925311,18sleep00-21swapper/020:57:470
1817825312,24sleep318173-21apt-get20:52:423
1741725312,25sleep117463-21ntp_kernel_pll_19:07:531
1670625314,22sleep116711-21diskstats20:47:471
1108025317,18sleep30-21swapper/320:27:593
297025214,23sleep12975-21/usr/sbin/munin21:47:451
2182825216,20sleep00-21swapper/022:47:420
2000925211,23sleep320027-21irqstats20:57:463
651825112,23sleep36610-21users20:12:583
609925112,21sleep10-21swapper/121:57:461
2687225112,24sleep026883-21df_inode19:37:430
1747625111,24sleep317537-21users20:48:013
1401825118,18sleep30-21swapper/322:22:383
2051025015,17sleep30-21swapper/322:42:483
1724325015,18sleep30-21swapper/300:12:583
1724325015,18sleep30-21swapper/300:12:583
140925014,19sleep00-21swapper/019:57:570
1328825012,17sleep00-21swapper/020:37:290
3223624913,19sleep30-21swapper/321:37:383
1360024913,21sleep013606-21df_inode20:37:450
571024810,23sleep15705-21apt-get20:12:391
41824813,15sleep10-21swapper/121:37:561
317652489,16sleep10-21swapper/119:52:461
919124710,22sleep39198-21df_inode22:07:413
1256724710,21sleep312576-21df_inode22:17:463
536724412,16sleep30-21swapper/323:37:503
186672439,15sleep30-21swapper/319:12:323
119462439,17sleep10-21swapper/120:32:411
144442419,15sleep10-21swapper/122:22:531
314402368,14sleep30-21swapper/319:52:273
17522366,14sleep30-21swapper/321:42:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional