You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 15:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu May 14, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
151732133216,1299sleep115182-21/usr/sbin/munin12:34:271
26421293921,901sleep324629-21kworker/u8:309:44:493
884327607,740sleep38879-21ntpq10:29:593
884327607,740sleep38879-21ntpq10:29:593
2520926358,613sleep025167-21apt-get08:00:030
18406257128,507sleep318476-21unin-run07:39:263
18406257128,507sleep318476-21unin-run07:39:263
7122251513,487sleep27130-21ntpq10:24:542
4366248621,449sleep24382-21/usr/sbin/munin08:34:502
4366248621,449sleep24382-21/usr/sbin/munin08:34:502
1011424670,7sleep20-21swapper/210:34:462
1660924518,429sleep316653-21ntpq10:54:513
26387244227,394sleep026403-21apt-get09:44:380
8051239924,357sleep28094-21/usr/sbin/munin12:10:092
1909623980,11sleep00-21swapper/007:39:590
3015223969,372sleep030190-21ntpq09:54:550
3015223969,372sleep030190-21ntpq09:54:550
71312364132,212sleep37171-21processes10:24:553
22442232713,249sleep322543-21timerandwakeup07:50:083
22442232713,249sleep322543-21timerandwakeup07:50:083
1124223169,292sleep311364-21cat08:54:583
1124223169,292sleep311364-21cat08:54:583
9004229016,175sleep39007-21apt-get08:49:373
20630229087,184sleep320785-21unixbench_multi09:24:583
25358226210,191sleep325429-21cat11:24:273
23070224219,208sleep123121-21unin-run07:54:261
23070224219,208sleep123121-21unin-run07:54:261
31753223622,192sleep231796-21fschecks_count08:19:482
28498223229,23sleep30-21swapper/309:49:533
28498223229,23sleep30-21swapper/309:49:533
463223012,170sleep3476-21idleruntime-cro08:24:253
20691222741,21sleep30-21swapper/311:09:293
16056222054,21sleep30-21swapper/307:30:093
13046221613,186sleep313179-21grep12:24:563
2017221312,180sleep32020-21cron08:29:263
16866220315,22sleep20-21swapper/209:14:442
16866220315,22sleep20-21swapper/209:14:442
16680219933,126sleep316720-21sendmail12:39:263
16680219933,126sleep316720-21sendmail12:39:263
3077221980,90sleep38485-21cyclictest08:14:553
3077221980,90sleep38485-21cyclictest08:14:553
185512198114,17sleep30-21swapper/309:19:453
10860219670,18sleep30-21swapper/307:14:473
12522219327,19sleep30-21swapper/307:19:473
7098218811,160sleep37147-21apt-config12:09:253
8807218035,18sleep30-21swapper/307:09:463
8807218035,18sleep30-21swapper/307:09:463
26923216627,101sleep326965-21cat08:04:463
25272216617,128sleep30-21swapper/308:00:143
7620215910,109sleep37617-21apt-get12:09:383
1739215843,25sleep30-21swapper/310:09:283
1739215843,25sleep30-21swapper/310:09:283
3523215611,106sleep30-21swapper/310:14:413
28566215611,68sleep30-21swapper/308:10:123
11698215425,17sleep30-21swapper/310:39:453
21486215110,102sleep321560-21cat11:09:533
21486215110,102sleep321560-21cat11:09:533
28684214622,99sleep128828-21grep09:49:551
28684214622,99sleep128828-21grep09:49:551
15044214658,74sleep315194-21latency_hist09:09:283
15044214658,74sleep315194-21latency_hist09:09:283
11145214418,102sleep20-21swapper/208:54:572
11145214418,102sleep20-21swapper/208:54:572
29873214035,90sleep329955-21fschecks_time09:54:503
29873214035,90sleep329955-21fschecks_time09:54:503
15173213616,93sleep215308-21latency_hist07:29:282
12417213620,93sleep20-21swapper/208:59:502
8886213518,102sleep08941-21idleruntime07:09:470
8886213518,102sleep08941-21idleruntime07:09:470
23932213531,37sleep323018-21/usr/sbin/munin09:35:003
2247213424,75sleep30-21swapper/310:09:483
2247213424,75sleep30-21swapper/310:09:483
2939213021,87sleep22437-21/usr/sbin/munin11:54:552
27479212916,98sleep227621-21unixbench_singl08:04:582
12084212759,55sleep312179-21latency_hist12:24:283
8618212614,98sleep28647-21fschecks_time10:29:542
8618212614,98sleep28647-21fschecks_time10:29:542
26060212631,26sleep326256-21apt-config09:44:283
22460212513,94sleep00-21swapper/011:14:390
26917212321,28sleep30-21swapper/311:29:283
3618211868,25sleep33705-21latency_hist08:34:273
571421168,67sleep30-21swapper/310:19:583
4557211655,47sleep34560-21ntp_states08:34:553
4557211655,47sleep34560-21ntp_states08:34:553
17393211613,59sleep30-21swapper/309:14:573
17393211613,59sleep30-21swapper/309:14:573
10218211316,85sleep310271-21cat10:34:483
31168211218,17sleep30-21swapper/311:39:583
20549211211,64sleep320632-21sed07:45:003
20549211211,64sleep320632-21sed07:45:003
1249021129,14sleep20-21swapper/207:19:462
32589211128,67sleep132602-21apt10:04:251
22552211112,79sleep00-21swapper/007:50:090
22552211112,79sleep00-21swapper/007:50:090
1092321119,88sleep010937-21irqstats07:14:480
7109210910,83sleep17121-21ntpq10:24:531
27724210918,63sleep20-21swapper/209:49:282
9028210827,55sleep09007-21apt-get08:49:460
15959210852,43sleep315992-21munin-run10:54:273
3943210610,79sleep34147-21apt-config11:59:283
6247210513,79sleep36339-21sed08:39:563
7899210448,42sleep38044-21sed08:44:573
1430321046,43sleep30-21swapper/310:48:173
18078210318,69sleep318135-21latency10:59:483
1611521038,82sleep216193-21mailstats07:30:122
14684210317,71sleep114731-21grep10:49:401
3182321029,52sleep30-21swapper/311:44:423
10241210218,66sleep210347-21munin-node07:14:302
4662210118,67sleep24707-21ntp_kernel_pll_12:00:002
4662210118,67sleep24707-21ntp_kernel_pll_12:00:002
4279210111,69sleep30-21swapper/311:59:443
4279210111,69sleep30-21swapper/311:59:443
27144210014,20sleep30-21swapper/311:29:403
27144210014,20sleep30-21swapper/311:29:403
376429916,68sleep13793-21irqstats10:14:471
2206729821,61sleep222101-21irqstats07:49:592
2206729821,61sleep222101-21irqstats07:49:592
118929813,68sleep01282-21/usr/sbin/munin11:49:480
1158429811,18sleep00-21swapper/010:39:380
197329722,37sleep10-21swapper/110:09:381
197329722,37sleep10-21swapper/110:09:381
1377229625,44sleep013791-21apt-get07:24:370
2413329517,64sleep024145-21fschecks_count11:19:430
1409929516,62sleep214133-21ntp_kernel_pll_09:04:502
1409929516,62sleep214133-21ntp_kernel_pll_09:04:502
1212129521,33sleep30-21swapper/308:59:393
2350629319,53sleep223018-21/usr/sbin/munin09:34:552
130629314,26sleep20-21swapper/211:49:502
1299229311,64sleep00-21swapper/007:19:570
715129119,57sleep07236-21wc12:09:260
2514729122,51sleep110219-21munin-node07:59:351
3003429018,56sleep130128-21latency_hist11:39:271
297662905,31sleep30-21swapper/311:37:033
198282906,32sleep30-21swapper/307:42:333
1359629016,52sleep213609-21/usr/sbin/munin10:44:532
1839628920,33sleep218402-21modprobe07:39:252
1839628920,33sleep218402-21modprobe07:39:252
1269928920,48sleep112768-21grep08:59:571
2217628820,46sleep121469-21/usr/sbin/munin09:29:571
3036128721,35sleep20-21swapper/211:39:442
2303728719,53sleep123106-21switchtime11:14:561
136712878,63sleep013739-21sensors_temp10:44:550
91842867,15sleep30-21swapper/312:14:383
1444728613,57sleep114588-21cut10:49:281
1190728523,28sleep10-21swapper/108:59:281
1190728523,28sleep10-21swapper/108:59:281
2735328418,50sleep127478-21timerandwakeup08:04:571
3242428317,44sleep132506-21grep11:44:561
1250628316,32sleep00-21swapper/012:24:450
865528224,37sleep28497-21latency_hist08:49:282
773428219,47sleep07826-21sed07:04:550
773428219,47sleep07826-21sed07:04:550
723728223,32sleep10-21swapper/112:09:271
2769828212,45sleep10-21swapper/111:29:541
2769828212,45sleep10-21swapper/111:29:541
206928224,32sleep10-21swapper/108:29:281
2031928215,51sleep020353-21/usr/sbin/munin09:24:510
1685128217,46sleep016861-21idleruntime-cro07:34:260
153382826,37sleep30-21swapper/309:09:443
153382826,37sleep30-21swapper/309:09:443
142522828,13sleep30-21swapper/307:24:523
809728016,40sleep00-21swapper/010:29:280
520128012,48sleep05226-21fschecks_count10:19:470
214612806,20sleep30-21swapper/309:29:303
2130928014,43sleep220889-21/usr/sbin/munin11:09:482
2130928014,43sleep220889-21/usr/sbin/munin11:09:482
2065028012,51sleep120695-21/usr/sbin/munin07:45:021
2065028012,51sleep120695-21/usr/sbin/munin07:45:021
1594328021,37sleep215956-21modprobe10:54:252
150192806,22sleep30-21swapper/312:32:593
1011228011,54sleep210113-21sshd08:52:192
965427914,50sleep19713-21processes12:14:531
900927914,50sleep19079-21sensors10:30:061
900927914,50sleep19079-21sensors10:30:061
2292427911,12sleep30-21swapper/311:14:543
2689827818,17sleep20-21swapper/211:29:272
234542786,12sleep30-21swapper/307:54:533
2803927716,25sleep00-21swapper/011:29:570
2803927716,25sleep00-21swapper/011:29:570
149027710,15sleep30-21swapper/311:49:553
1346427725,22sleep00-21swapper/009:04:280
3140327610,51sleep30-21swapper/309:59:433
1009627623,36sleep110111-21cut10:34:461
267427517,29sleep20-21swapper/208:29:492
1580527518,38sleep115875-21sensors_temp09:09:581
1580527518,38sleep115875-21sensors_temp09:09:581
2156727418,29sleep00-21swapper/011:09:540
2156727418,29sleep00-21swapper/011:09:540
2152327412,47sleep221566-21systemd-run07:49:272
2152327412,47sleep221566-21systemd-run07:49:272
2040027425,30sleep020439-21fschecks_time07:44:560
2040027425,30sleep020439-21fschecks_time07:44:560
1708727417,22sleep00-21swapper/009:14:500
1708727417,22sleep00-21swapper/009:14:500
1617527418,28sleep00-21swapper/012:34:580
1617527418,28sleep00-21swapper/012:34:580
771427310,42sleep00-21swapper/012:09:510
713027317,18sleep20-21swapper/207:04:412
713027317,18sleep20-21swapper/207:04:412
465227317,24sleep10-21swapper/111:59:591
465227317,24sleep10-21swapper/111:59:591
273427313,30sleep00-21swapper/011:54:480
2594527317,27sleep20-21swapper/209:44:272
2067927315,44sleep220690-21latency_hist11:09:282
1908727320,28sleep20-21swapper/209:19:572
1749127322,34sleep117495-21cron10:59:251
1509227317,34sleep115143-21unin-run07:29:271
2728527224,19sleep00-21swapper/008:04:560
246327220,23sleep10-21swapper/111:54:411
1549227216,34sleep115395-21/usr/sbin/munin12:34:431
1549227216,34sleep115395-21/usr/sbin/munin12:34:431
997427113,38sleep29991-21munin-run07:14:252
997427113,38sleep29991-21munin-run07:14:252
2832727118,31sleep228364-21ntpq09:49:482
2832727118,31sleep228364-21ntpq09:49:482
168727116,39sleep11720-21awk10:09:261
168727116,39sleep11720-21awk10:09:261
983127023,25sleep00-21swapper/010:34:290
1203427019,30sleep112058-21awk12:24:251
3227726814,21sleep10-21swapper/108:19:591
274626816,37sleep02875-21awk10:09:570
274626816,37sleep02875-21awk10:09:570
939126716,40sleep00-21swapper/010:30:130
939126716,40sleep00-21swapper/010:30:130
3171326713,36sleep231751-21/usr/sbin/munin09:59:502
317012677,12sleep20-21swapper/211:44:292
2890126723,22sleep00-21swapper/011:34:460
2722826719,22sleep20-21swapper/211:29:432
2722826719,22sleep20-21swapper/211:29:432
115712675,53sleep00-21swapper/008:56:080
115712675,53sleep00-21swapper/008:56:080
562126622,20sleep20-21swapper/210:19:572
3220526620,26sleep00-21swapper/008:19:580
312972666,10sleep00-21swapper/011:41:240
2255626613,35sleep222644-21/usr/sbin/munin11:14:452
1032926619,27sleep10-21swapper/108:54:281
67672655,47sleep00-21swapper/008:42:190
588126515,20sleep20-21swapper/208:39:472
2896526512,25sleep20-21swapper/211:34:482
2493526518,30sleep025059-21dpkg07:59:300
2100626513,26sleep10-21swapper/111:09:431
2100626513,26sleep10-21swapper/111:09:431
1977226519,23sleep10-21swapper/111:04:501
1349326523,22sleep20-21swapper/212:28:182
752526420,29sleep37569-21ntpq07:04:503
752526420,29sleep37569-21ntpq07:04:503
200726414,26sleep00-21swapper/008:29:190
1402326412,35sleep214080-21/usr/sbin/munin12:29:432
1202226421,25sleep20-21swapper/212:24:182
91726311,23sleep10-21swapper/111:49:411
894426315,32sleep18989-21irqstats07:09:481
894426315,32sleep18989-21irqstats07:09:481
67826317,30sleep00-21swapper/011:49:270
291726317,24sleep00-21swapper/008:29:560
2587926316,31sleep025886-21modprobe09:44:250
1915826316,21sleep00-21swapper/011:04:290
1788226310,31sleep217889-21df_inode10:59:432
1727626316,28sleep117308-21/usr/sbin/munin09:14:551
1727626316,28sleep117308-21/usr/sbin/munin09:14:551
1378926318,20sleep00-21swapper/009:04:440
1378926318,20sleep00-21swapper/009:04:440
855526220,18sleep10-21swapper/108:49:271
266162628,12sleep10-21swapper/109:45:031
2469926213,32sleep224788-21apt-config07:59:282
88526120,27sleep2900-21open_inodes10:04:502
88526120,27sleep2900-21open_inodes10:04:502
406226116,26sleep13576-21apt-get08:34:371
406226116,26sleep13576-21apt-get08:34:371
3158226110,28sleep10-21swapper/111:44:281
1218126119,22sleep20-21swapper/210:39:562
2983426017,21sleep20-21swapper/209:54:452
2983426017,21sleep20-21swapper/209:54:452
249672608,11sleep10-21swapper/111:20:051
446325910,34sleep04464-21sshd10:15:180
90592587,33sleep20-21swapper/212:14:292
583925822,18sleep20-21swapper/212:04:402
301825811,18sleep10-21swapper/108:29:571
2559625813,26sleep025664-21df11:24:400
2531825817,24sleep20-21swapper/208:00:172
1893325813,27sleep019003-21sensors_temp09:19:550
1771625814,30sleep017788-21cat07:35:030
1771625814,30sleep017788-21cat07:35:030
1678225818,19sleep00-21swapper/010:54:560
1367625815,25sleep113695-21apt-get09:04:371
1367625815,25sleep113695-21apt-get09:04:371
1127725818,15sleep10-21swapper/107:14:571
310142575,10sleep00-21swapper/008:15:180
310142575,10sleep00-21swapper/008:15:180
2851725715,25sleep228010-21apt-get08:09:382
2021925713,22sleep20-21swapper/209:24:492
186682576,12sleep10-21swapper/109:19:471
986725612,24sleep00-21swapper/012:14:560
768625617,20sleep00-21swapper/008:44:540
3264225614,18sleep00-21swapper/010:04:280
2539225613,25sleep125520-21/usr/sbin/munin09:39:581
2170125618,22sleep20-21swapper/209:29:462
1597125614,20sleep20-21swapper/212:34:562
1597125614,20sleep20-21swapper/212:34:562
1587625620,18sleep20-21swapper/209:09:592
1587625620,18sleep20-21swapper/209:09:592
1209225610,18sleep10-21swapper/110:39:551
776825512,30sleep17789-21grep12:10:001
712925512,25sleep17123-21apt-get07:04:401
712925512,25sleep17123-21apt-get07:04:401
352225512,24sleep23515-21apt-get10:14:402
2927125515,21sleep00-21swapper/009:54:270
2927125515,21sleep00-21swapper/009:54:270
1847925515,22sleep10-21swapper/107:39:281
1847925515,22sleep10-21swapper/107:39:281
1358425519,20sleep10-21swapper/112:29:261
567525417,19sleep10-21swapper/108:39:421
2896025414,16sleep10-21swapper/108:10:381
240552546,34sleep10-21swapper/109:36:151
1447525415,23sleep00-21swapper/012:29:540
1145525422,17sleep00-21swapper/012:19:560
100725416,23sleep21030-21irqstats08:24:492
642425316,20sleep16544-21date10:24:281
3033025311,17sleep10-21swapper/109:55:001
3033025311,17sleep10-21swapper/109:55:001
2580225317,16sleep10-21swapper/111:24:451
2462625313,23sleep124696-21sh07:59:271
20532537,12sleep20-21swapper/210:09:432
20532537,12sleep20-21swapper/210:09:432
1350625314,22sleep113507-21cron07:24:261
517125214,23sleep05218-21users12:00:220
517125214,23sleep05218-21users12:00:220
2411825211,22sleep224124-21df_inode11:19:412
1559125213,22sleep115576-21apt-get07:29:431
8402518,15sleep00-21swapper/010:04:480
8402518,15sleep00-21swapper/010:04:480
446925112,23sleep04500-21/usr/sbin/munin08:34:520
446925112,23sleep04500-21/usr/sbin/munin08:34:520
3007225111,23sleep130067-21apt-get08:14:381
3007225111,23sleep130067-21apt-get08:14:381
2886525113,22sleep028959-21memory08:10:370
2002425114,21sleep120030-21df_inode09:24:441
1962325115,22sleep119635-21vmstat07:40:091
1897325112,23sleep219002-21irqstats07:39:582
1781225116,17sleep10-21swapper/110:59:421
2157925015,16sleep00-21swapper/009:29:440
1623325016,19sleep20-21swapper/210:54:422
3052924915,17sleep20-21swapper/208:14:512
3052924915,17sleep20-21swapper/208:14:512
231272498,26sleep00-21swapper/009:34:440
1513724912,21sleep015143-21ntp_states10:49:510
1534624811,22sleep015352-21diskstats09:09:450
1534624811,22sleep015352-21diskstats09:09:450
1951224312,16sleep00-21swapper/011:04:450
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional