You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-25 - 20:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Mon May 25, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28626285969,8568sleep123347-21kworker/1:208:39:011
1808927657,746sleep318133-21ntpq08:04:113
501248051,415sleep3541-21uname10:34:143
25673246822,431sleep225735-21/usr/sbin/munin11:54:152
25673246822,431sleep225735-21/usr/sbin/munin11:54:152
250423179,295sleep32540-21ntpq12:24:123
19436225067,96sleep319583-21unixbench_singl11:34:183
1446122287,206sleep314502-21ntpq11:19:123
19863221788,109sleep319925-21fschecks_time09:54:063
19863221788,109sleep319925-21fschecks_time09:54:063
11034221735,157sleep211050-21fschecks_count11:09:052
3322221317,182sleep33380-21fschecks_count07:19:073
30588220699,22sleep30-21swapper/310:28:493
2360220319,61sleep30-21swapper/307:14:183
26813219910,115sleep326931-21latency_hist08:33:503
1599218514,153sleep11731-21/usr/sbin/munin08:54:191
1599218514,153sleep11731-21/usr/sbin/munin08:54:191
4261218117,67sleep34346-21threads12:29:163
31828218012,154sleep331939-21fschecks_time08:49:083
31828218012,154sleep331939-21fschecks_time08:49:083
20697217037,103sleep30-21swapper/311:39:313
24261216518,130sleep10-21swapper/110:08:491
31885216229,121sleep331999-21date07:08:493
15844216221,63sleep315903-21modprobe07:58:463
15844216221,63sleep315903-21modprobe07:58:463
30400215621,123sleep030418-21idleruntime-cro10:28:460
2925215617,123sleep02089-1kworker/0:1H10:43:480
2128421559,82sleep321330-21processes08:14:163
14082215559,26sleep30-21swapper/309:34:413
27037214921,27sleep30-21swapper/311:59:093
6264214713,108sleep30-21swapper/310:54:013
26462214710,17sleep00-21swapper/010:14:130
2016621478,43sleep320180-21awk08:09:203
15043214615,115sleep115078-21grep07:54:121
15043214615,115sleep115078-21grep07:54:121
16371214111,111sleep316137-21/usr/sbin/munin07:59:073
10008214013,25sleep30-21swapper/311:04:183
26362213815,100sleep00-21swapper/008:29:190
4231213611,17sleep30-21swapper/309:04:093
15344213659,19sleep30-21swapper/309:39:133
25267213324,18sleep30-21swapper/308:28:493
12136213232,84sleep312264-21timerwakeupswit07:44:173
1383521299,17sleep30-21swapper/311:18:473
1383521299,17sleep30-21swapper/311:18:473
25278212618,90sleep125365-21fschecks_time11:54:061
25278212618,90sleep125365-21fschecks_time11:54:061
8790212430,77sleep30-21swapper/310:59:463
13157212318,89sleep013043-21latency_hist09:33:490
18301212233,17sleep318307-21df_inode09:49:063
5505212121,72sleep00-21swapper/009:08:590
2964821207,99sleep329798-21cat12:08:493
30272211915,83sleep230369-21fschecks_time08:44:062
5345211810,96sleep35407-21latency07:24:483
5345211810,96sleep35407-21latency07:24:483
30571211831,20sleep330607-21ntpq08:44:113
24630211730,21sleep3363-21snmpd11:50:173
32138211625,17sleep30-21swapper/307:09:013
1583521166,12sleep10-21swapper/109:39:231
19349211311,17sleep00-21swapper/008:09:050
31503211113,77sleep031569-21/usr/sbin/munin10:29:310
31503211113,77sleep031569-21/usr/sbin/munin10:29:310
9116211012,22sleep30-21swapper/309:19:153
7204210811,20sleep30-21swapper/307:29:323
22991210810,27sleep30-21swapper/310:04:073
719210716,76sleep2870-21sed10:34:172
27274210625,60sleep127297-21cron10:18:461
14342210516,74sleep214376-21sed07:53:492
14342210516,74sleep214376-21sed07:53:492
9978210411,77sleep29944-21latency_hist09:23:502
5474210471,21sleep35190-21/usr/sbin/munin12:34:093
2917210415,73sleep23008-21cut08:59:162
2167210420,15sleep30-21swapper/308:58:493
2167210420,15sleep30-21swapper/308:58:493
10698210412,74sleep30-21swapper/309:24:153
8754210313,70sleep08814-21/usr/sbin/munin09:19:080
2882521039,79sleep328885-21cat08:39:073
16797210323,56sleep116923-21timerwakeupswit07:59:171
30549210140,34sleep20-21swapper/207:04:002
1779021017,58sleep30-21swapper/309:46:563
10833210119,58sleep110958-21timerwakeupswit09:24:171
31886210018,57sleep30-21swapper/312:14:143
23350210022,57sleep31-21systemd11:48:473
1762921006,11sleep20-21swapper/211:29:122
1762921006,11sleep20-21swapper/211:29:122
311272999,78sleep331185-21cat10:29:223
311272999,78sleep331185-21cat10:29:223
2735429913,70sleep30-21swapper/310:18:483
1953529916,67sleep219684-21latency_hist09:53:492
3112929855,26sleep331157-21modprobe12:13:463
1869729812,71sleep218716-21apt-get11:33:562
844829714,20sleep30-21swapper/309:18:503
343629712,12sleep00-21swapper/010:44:090
343629712,12sleep00-21swapper/010:44:090
2559129715,60sleep225480-21/usr/sbin/munin08:29:052
214722978,77sleep321558-21perl09:59:073
1904829614,66sleep119194-21latency_hist08:08:491
21922958,70sleep00-21swapper/012:24:050
3102929425,32sleep00-21swapper/007:04:140
265272947,76sleep326565-21sed11:58:483
265272947,76sleep326565-21sed11:58:483
2364129444,25sleep20-21swapper/211:49:022
1175929419,54sleep111783-21cpu09:29:021
2374429310,55sleep30-21swapper/308:23:493
2374429310,55sleep30-21swapper/308:23:493
776429218,56sleep07900-21unixbench_multi09:14:190
507929214,62sleep15146-21df07:24:421
507929214,62sleep15146-21df07:24:421
561529030,42sleep24693-21/usr/sbin/munin10:49:192
2705129011,58sleep20-21swapper/208:34:012
2259429010,65sleep222686-21mailstats11:44:142
109412899,37sleep30-21swapper/311:08:583
101492898,11sleep30-21swapper/307:39:093
285782886,11sleep30-21swapper/310:20:143
244232886,10sleep30-21swapper/310:08:513
2053628827,40sleep120531-21apt-get11:39:001
166922885,13sleep30-21swapper/311:26:143
166922885,13sleep30-21swapper/311:26:143
1271128810,66sleep0363-21snmpd07:48:220
34012878,13sleep30-21swapper/310:44:083
34012878,13sleep30-21swapper/310:44:083
270522878,63sleep30-21swapper/308:34:023
21012876,11sleep30-21swapper/310:39:153
36262866,12sleep30-21swapper/309:03:083
95972856,11sleep30-21swapper/307:38:323
119962857,62sleep30-21swapper/309:29:083
1011328517,29sleep00-21swapper/011:04:190
987928414,50sleep19944-21sendmail_mailqu11:04:161
67702847,11sleep30-21swapper/309:13:343
58682847,11sleep30-21swapper/310:52:233
456328417,52sleep14698-21latency_hist07:23:491
2391228422,35sleep00-21swapper/011:49:080
181902845,10sleep30-21swapper/311:30:343
181902845,10sleep30-21swapper/311:30:343
16812846,11sleep30-21swapper/312:21:583
1560528432,23sleep20-21swapper/209:39:202
140872835,30sleep30-21swapper/307:51:083
140872835,30sleep30-21swapper/307:51:083
710128210,56sleep17114-21irqstats07:29:301
2679028224,32sleep10-21swapper/108:33:481
1736228215,49sleep117512-21sed09:44:191
407228115,50sleep14175-21perl09:04:071
1156428116,16sleep00-21swapper/011:09:240
64628022,32sleep20-21swapper/212:19:062
633428016,40sleep06363-21munin-run07:28:490
633428016,40sleep06363-21munin-run07:28:490
2676728029,25sleep00-21swapper/011:59:020
1205828023,36sleep212133-21cat07:44:162
917227813,49sleep29326-21latency_hist11:03:492
3047827617,41sleep230581-21apt-config10:28:482
1731427610,46sleep017151-21/usr/sbin/munin11:29:060
1731427610,46sleep017151-21/usr/sbin/munin11:29:060
1259227622,34sleep212693-21fschecks_time11:14:062
1259227622,34sleep212693-21fschecks_time11:14:062
89627412,48sleep2908-21sshd08:54:032
89627412,48sleep2908-21sshd08:54:032
518127415,29sleep10-21swapper/110:49:141
3180727411,43sleep10-21swapper/108:49:061
3180727411,43sleep10-21swapper/108:49:061
1387127416,26sleep00-21swapper/011:18:480
1387127416,26sleep00-21swapper/011:18:480
496727322,29sleep20-21swapper/212:33:472
321932736,14sleep20-21swapper/208:49:152
321932736,14sleep20-21swapper/208:49:152
298302739,52sleep00-21swapper/008:43:030
2320527314,44sleep023279-21/usr/sbin/munin10:04:120
2051127320,36sleep220521-21find11:38:572
3257527212,13sleep20-21swapper/207:09:122
127682726,13sleep10-21swapper/111:14:091
127682726,13sleep10-21swapper/111:14:091
842527122,24sleep20-21swapper/209:18:492
678527115,29sleep26300-21apt-get07:28:592
997227023,26sleep10-21swapper/109:23:491
754827012,42sleep17589-21processes09:14:161
193072708,11sleep00-21swapper/009:51:290
170772707,44sleep00-21swapper/007:59:200
1077227021,33sleep210841-21/usr/sbin/munin07:39:202
3181426921,21sleep10-21swapper/107:08:471
3105226915,34sleep20-21swapper/210:29:212
3105226915,34sleep20-21swapper/210:29:212
251926918,35sleep12619-21perl08:59:071
14822699,48sleep00-21swapper/012:19:240
1071426919,31sleep210865-21latency_hist11:08:492
412426818,19sleep00-21swapper/012:29:130
211226819,30sleep12106-21apt-get12:24:011
2003726824,29sleep00-21swapper/011:38:470
1146826825,23sleep20-21swapper/209:28:462
447626711,24sleep20-21swapper/209:04:162
2838326718,28sleep10-21swapper/108:38:481
2705426726,19sleep00-21swapper/008:34:030
32726617,33sleep1477-21date12:18:491
2932626617,23sleep10-21swapper/110:24:071
240726618,25sleep00-21swapper/008:59:020
2404326613,38sleep124076-21grep08:24:071
2207926620,30sleep122137-21basename08:18:461
55842657,13sleep10-21swapper/112:34:111
2395526522,26sleep023956-21sshd08:24:020
2245726512,33sleep122488-21/usr/sbin/munin11:44:091
1902226516,32sleep019000-21latency_hist08:08:480
142512658,47sleep00-21swapper/007:53:140
142512658,47sleep00-21swapper/007:53:140
1076226519,25sleep00-21swapper/009:24:160
556526414,34sleep15605-21df09:09:041
3257126415,24sleep10-21swapper/110:34:041
308372648,11sleep00-21swapper/012:10:000
2445926418,29sleep224532-21/usr/sbin/munin10:09:032
2102826415,29sleep021086-21/usr/sbin/munin08:14:090
732926323,22sleep00-21swapper/007:29:370
470026323,18sleep20-21swapper/207:23:502
2862926314,36sleep228680-21cpuspeed08:39:022
2546126320,24sleep00-21swapper/011:54:090
2546126320,24sleep00-21swapper/011:54:090
1798826321,21sleep20-21swapper/209:48:482
50342626,16sleep00-21swapper/009:06:160
1899526220,18sleep10-21swapper/111:34:081
1395526216,18sleep20-21swapper/209:34:362
1351826216,27sleep113536-21apt-get09:33:581
975626111,33sleep19902-21modprobe07:38:511
801826111,38sleep18080-21idleruntime10:59:071
647526117,27sleep06536-21sendmail-msp12:38:460
3223926117,20sleep00-21swapper/007:09:060
3213626119,25sleep132133-21apt-get07:08:591
3157026117,18sleep10-21swapper/110:29:321
3157026117,18sleep10-21swapper/110:29:321
3085526117,20sleep10-21swapper/108:44:171
299726120,19sleep20-21swapper/207:18:492
234926120,17sleep20-21swapper/210:39:182
1850326119,20sleep10-21swapper/109:49:121
2899626015,32sleep229119-21cat12:04:212
2890226017,19sleep10-21swapper/112:04:191
2751026019,21sleep10-21swapper/111:59:191
2143026010,22sleep10-21swapper/109:59:051
209292605,10sleep00-21swapper/009:57:260
209292605,10sleep00-21swapper/009:57:260
1769326021,23sleep217712-21apt-get08:03:582
1349526019,21sleep10-21swapper/107:49:161
1349526019,21sleep10-21swapper/107:49:161
55625915,24sleep10-21swapper/112:19:011
3210525917,24sleep20-21swapper/212:14:172
316325917,18sleep10-21swapper/110:44:021
316325917,18sleep10-21swapper/110:44:021
3017125916,26sleep030166-21apt-get08:44:000
2830725912,26sleep10-21swapper/110:19:191
2808325913,30sleep228196-21latency_hist12:03:482
2286025915,28sleep222861-21ntp_states08:19:122
2286025915,28sleep222861-21ntp_states08:19:122
2254525918,20sleep00-21swapper/011:44:120
79925814,26sleep00-21swapper/008:53:570
79925814,26sleep00-21swapper/008:53:570
3213525816,18sleep00-21swapper/008:49:130
3213525816,18sleep00-21swapper/008:49:130
264932589,13sleep20-21swapper/210:14:152
2502925820,20sleep00-21swapper/010:09:170
2111225818,23sleep20-21swapper/209:58:492
2111225818,23sleep20-21swapper/209:58:492
1562625817,25sleep215620-21apt-get11:23:592
1562625817,25sleep215620-21apt-get11:23:592
574325716,23sleep25802-21irqstats09:09:092
442625716,18sleep20-21swapper/210:48:472
442625716,18sleep20-21swapper/210:48:472
366625725,15sleep10-21swapper/112:29:021
247162576,39sleep10-21swapper/111:51:241
2132225715,23sleep20-21swapper/209:58:582
211525715,26sleep22183-21/usr/sbin/munin12:24:022
2052225718,25sleep220545-21modprobe08:13:472
1935525717,21sleep10-21swapper/108:09:061
1584825714,24sleep115897-21irqstats11:24:061
1584825714,24sleep115897-21irqstats11:24:061
12932576,11sleep00-21swapper/010:38:440
1102725715,27sleep111033-21diskstats11:09:041
836025614,25sleep08354-21apt-get07:34:010
62102566,12sleep00-21swapper/010:53:490
179725615,26sleep01861-21idleruntime07:14:070
1197225616,22sleep10-21swapper/107:44:151
633825515,17sleep10-21swapper/110:54:031
223225519,16sleep10-21swapper/110:39:171
2174825513,26sleep021787-21ntpq09:59:120
1558925514,18sleep00-21swapper/009:39:180
1222625517,18sleep20-21swapper/209:29:152
2820325420,21sleep00-21swapper/010:19:180
251392545,11sleep20-21swapper/208:27:522
2053525412,25sleep020003-21apt-get11:38:590
2019225415,23sleep120229-21ntp_states09:54:121
2019225415,23sleep120229-21ntp_states09:54:121
1465025417,22sleep214555-21/usr/sbin/munin07:54:012
1465025417,22sleep214555-21/usr/sbin/munin07:54:012
96192536,37sleep10-21swapper/107:38:461
820625314,19sleep00-21swapper/010:59:100
671825316,21sleep20-21swapper/210:54:122
295832538,13sleep00-21swapper/012:08:460
2330225317,19sleep20-21swapper/210:04:142
2149725317,16sleep20-21swapper/208:14:192
502425213,24sleep25025-21sshd07:24:022
502425213,24sleep25025-21sshd07:24:022
1667625213,23sleep016670-21apt-get09:44:020
1489025211,24sleep014960-21sed09:38:490
561225114,19sleep20-21swapper/212:34:122
323725112,23sleep03309-21df07:19:030
3141825111,24sleep031414-21apt-get12:13:590
1533825112,17sleep10-21swapper/111:23:461
1417125111,18sleep20-21swapper/211:19:072
57042505,16sleep00-21swapper/010:50:140
2297525016,16sleep10-21swapper/108:19:151
2297525016,16sleep10-21swapper/108:19:151
1648325014,22sleep016530-21users11:24:180
1648325014,22sleep016530-21users11:24:180
1616325017,18sleep20-21swapper/207:59:022
1503625011,16sleep00-21swapper/007:54:110
1503625011,16sleep00-21swapper/007:54:110
2478824912,20sleep124827-21memory10:09:101
2236824917,19sleep00-21swapper/008:18:540
2236824917,19sleep00-21swapper/008:18:540
18652499,17sleep10-21swapper/107:14:081
1416124920,16sleep00-21swapper/011:19:050
2987024813,18sleep20-21swapper/212:09:012
2758824812,20sleep227586-21apt-get10:18:592
1717424711,18sleep10-21swapper/111:28:591
1717424711,18sleep10-21swapper/111:28:591
835724610,19sleep20-21swapper/207:33:592
3118224516,18sleep10-21swapper/112:13:481
1038724510,16sleep00-21swapper/007:39:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional