You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 20:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Thu Mar 12, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1686622242615,22394sleep014334-21kworker/u8:307:13:430
6258252058,5184sleep36186-21kworker/3:210:06:363
2553427857,765sleep325572-21ntpq09:23:413
2553427857,765sleep325572-21ntpq09:23:413
3164527617,699sleep331787-21cat11:28:183
8115275620,718sleep28148-21/usr/sbin/munin08:28:382
1486227458,725sleep314894-21ntpq10:33:403
1486227458,725sleep314894-21ntpq10:33:403
3849255816,526sleep03943-21/usr/sbin/munin09:58:380
1951325369,511sleep219558-21ntpq10:48:412
1940024766,460sleep313441-21snmpd07:19:313
7113244761,372sleep36176-21/usr/sbin/munin08:23:513
31441240098,209sleep331479-21fschecks_count09:43:353
12908234358,191sleep30-21swapper/310:28:323
12908234358,191sleep30-21swapper/310:28:323
17139231023,270sleep217170-21memory12:23:392
21846229921,255sleep121861-21cut09:13:341
21846229921,255sleep121861-21cut09:13:341
35022978,275sleep0390-21ntpq08:03:410
6201229316,259sleep16250-21/usr/sbin/munin08:23:321
31235229317,193sleep30-21swapper/307:58:393
3257226167,88sleep33286-21modprobe09:58:163
5260224616,210sleep35365-21timerandwakeup08:18:463
31965224666,99sleep331979-21fschecks_count11:28:343
18654222531,182sleep318655-21cpu09:03:303
17614219616,160sleep117706-21unixbench_multi07:14:121
18817219573,53sleep30-21swapper/312:28:453
21241219042,133sleep321328-21latency_hist12:38:173
27338218716,95sleep30-21swapper/311:13:413
21364218272,86sleep30-21swapper/310:53:473
974218178,72sleep30-21swapper/311:33:313
32397218029,58sleep30-21swapper/308:03:283
29002217122,126sleep129013-21sh07:53:161
9684216612,95sleep30-21swapper/311:58:463
9684216612,95sleep30-21swapper/311:58:463
17368216510,134sleep317432-21latency08:58:383
1082121649,143sleep213441-21snmpd10:20:032
14841215929,115sleep314891-21cat07:08:373
1889721518,131sleep318964-21latency_hist10:48:183
684421497,107sleep00-21swapper/008:23:480
10554214215,111sleep00-21swapper/008:38:170
25177214116,109sleep125141-21latency_hist11:08:181
2384021417,47sleep30-21swapper/311:03:273
23628214042,60sleep323764-21latency_hist11:03:193
20491214044,51sleep30-21swapper/309:08:393
20445214056,18sleep30-21swapper/310:53:183
15342213918,104sleep315431-21grep12:18:363
17167213743,74sleep30-21swapper/310:41:353
17167213743,74sleep30-21swapper/310:41:353
11553213410,111sleep311706-21awk08:38:483
1895213333,77sleep30-21swapper/308:08:413
13336213311,107sleep013354-21grep07:03:380
1048221338,72sleep310607-21cat10:18:473
29951213257,16sleep30-21swapper/307:53:473
23593212518,91sleep123577-21apt-get09:18:341
23593212518,91sleep123577-21apt-get09:18:341
3444212410,95sleep33485-21ntpq08:13:413
9574212236,67sleep30-21swapper/310:18:183
25009211811,89sleep125018-21/usr/sbin/munin07:38:391
29892211351,23sleep30-21swapper/309:38:343
11452211210,43sleep30-21swapper/310:23:353
18716211014,78sleep218725-21/usr/sbin/munin12:28:402
929621088,86sleep39342-21grep08:33:313
30095210735,16sleep30-21swapper/311:23:173
30095210735,16sleep30-21swapper/311:23:173
6948210649,22sleep36988-21latency10:08:383
31880210630,49sleep231876-21apt-get11:28:292
3153621069,82sleep231638-21timerwakeupswit07:58:472
5888210317,11sleep30-21swapper/311:48:323
5888210317,11sleep30-21swapper/311:48:323
29052210315,52sleep30-21swapper/307:53:183
13040210117,67sleep113045-21sort07:03:271
4887210015,69sleep24923-21/usr/sbin/munin08:18:372
23233210031,17sleep30-21swapper/307:33:363
23233210031,17sleep30-21swapper/307:33:363
386329913,64sleep20-21swapper/211:43:192
386329913,64sleep20-21swapper/211:43:192
236122998,61sleep30-21swapper/309:18:483
236122998,61sleep30-21swapper/309:18:483
1819529929,18sleep30-21swapper/307:18:183
312102989,76sleep231239-21users11:23:492
676229721,51sleep06827-21fschecks_time10:08:340
79552967,45sleep30-21swapper/308:28:353
1113929630,53sleep311223-21cat12:03:463
1113929630,53sleep311223-21cat12:03:463
108042967,80sleep00-21swapper/010:19:490
2152629527,19sleep321518-21apt-get07:28:323
2152629527,19sleep321518-21apt-get07:28:323
2895729417,58sleep013441-21snmpd09:33:470
268929421,47sleep22845-21grep09:53:472
743729312,38sleep30-21swapper/311:53:333
333929327,52sleep13428-21/usr/sbin/munin08:13:391
2688329316,58sleep027020-21unixbench_multi07:43:510
2210629314,65sleep022181-21grep07:28:490
2210629314,65sleep022181-21grep07:28:490
267612929,40sleep30-21swapper/309:28:333
835329118,54sleep08456-21timerandwakeup08:28:440
1707229120,54sleep117138-21wc12:23:381
1656229116,59sleep116602-21cut12:23:181
1261829124,45sleep012645-21modprobe10:28:160
85952906,65sleep00-21swapper/010:13:390
2828029018,51sleep227717-21/usr/sbin/munin07:48:452
46132897,67sleep34696-21mailstats11:43:443
2116228920,47sleep121201-21sendmail12:38:151
68428827,39sleep0695-21cron11:33:150
1987328811,61sleep019892-21apt-get12:33:260
1443028823,29sleep10-21swapper/110:33:261
1443028823,29sleep10-21swapper/110:33:261
1364028821,14sleep30-21swapper/312:13:273
357228718,15sleep30-21swapper/309:58:283
2115228719,35sleep30-21swapper/307:27:103
2522128523,32sleep10-21swapper/109:23:351
2522128523,32sleep10-21swapper/109:23:351
2076728510,58sleep120882-21users07:23:491
1996828524,39sleep219986-21grep09:08:182
1825728512,59sleep118374-21cat10:43:481
41928413,36sleep30-21swapper/309:48:273
283112848,35sleep30-21swapper/309:33:343
231428415,55sleep22451-21latency_hist11:38:182
2199428311,17sleep20-21swapper/207:28:472
2199428311,17sleep20-21swapper/207:28:472
2017928323,28sleep20-21swapper/207:23:382
155922836,42sleep30-21swapper/308:53:343
152732838,63sleep315282-21munin-run08:53:173
3261628125,29sleep00-21swapper/009:48:160
146562816,15sleep20-21swapper/207:08:322
2609427920,41sleep125388-21/usr/sbin/munin11:08:461
253427913,50sleep02604-21sensors09:53:450
3021027827,29sleep10-21swapper/109:38:401
2875627824,32sleep128247-21apt-get11:18:271
2875627824,32sleep128247-21apt-get11:18:271
2025627812,18sleep00-21swapper/007:23:400
1572127823,38sleep215723-21cron10:38:152
1572127823,38sleep215723-21cron10:38:152
152527826,24sleep10-21swapper/111:33:451
128752778,32sleep30-21swapper/308:43:453
291927648,17sleep32526-21/usr/sbin/munin11:38:393
291927648,17sleep32526-21/usr/sbin/munin11:38:393
2201627621,28sleep20-21swapper/210:58:182
197927613,44sleep22002-21/usr/sbin/munin08:08:442
1837527618,30sleep20-21swapper/210:43:492
289322759,54sleep00-21swapper/011:18:540
289322759,54sleep00-21swapper/011:18:540
266272757,12sleep00-21swapper/011:12:410
255722758,35sleep30-21swapper/307:38:573
1843327517,43sleep018408-21/usr/sbin/munin07:18:310
1432727526,29sleep314383-21systemd-run07:08:173
3185627413,30sleep10-21swapper/109:43:451
2975127422,28sleep00-21swapper/007:53:440
243352749,51sleep024399-21ps11:03:450
133327425,25sleep20-21swapper/208:08:192
2670727317,32sleep226801-21grep07:43:492
175027315,45sleep11763-21latency_hist09:53:181
1341227315,27sleep00-21swapper/010:28:450
1341227315,27sleep00-21swapper/010:28:450
1056627312,30sleep10-21swapper/112:03:321
1056627312,30sleep10-21swapper/112:03:321
97072720,7sleep10-21swapper/108:33:411
2782827225,23sleep00-21swapper/007:48:350
217592727,50sleep30-21swapper/309:13:273
217592727,50sleep30-21swapper/309:13:273
1621427218,21sleep20-21swapper/207:13:202
1472927222,19sleep00-21swapper/010:33:370
1472927222,19sleep00-21swapper/010:33:370
1267027213,13sleep20-21swapper/210:28:182
1251927217,38sleep012620-21fschecks_time08:43:380
489327121,26sleep10-21swapper/110:03:191
2147427117,32sleep021496-21awk09:13:160
1604027114,39sleep116107-21df10:38:311
1604027114,39sleep116107-21df10:38:311
940727018,37sleep19450-21ntpq11:58:401
940727018,37sleep19450-21ntpq11:58:401
51032706,53sleep00-21swapper/010:03:260
456627011,46sleep24612-21processes11:43:432
2202327013,42sleep022036-21kernelversion09:13:370
2202327013,42sleep022036-21kernelversion09:13:370
1063327021,23sleep20-21swapper/208:38:192
93726911,44sleep0999-21sendmail_mailst09:48:450
477926918,28sleep14910-21grep11:43:461
2831726921,22sleep00-21swapper/011:18:180
2521426914,39sleep025220-21diskstats09:23:340
2521426914,39sleep025220-21diskstats09:23:340
1602626923,25sleep10-21swapper/107:13:191
8842689,17sleep10-21swapper/108:03:501
880126824,23sleep20-21swapper/210:13:452
78912688,41sleep20-21swapper/211:53:442
2860126816,38sleep128644-21tr09:33:401
135972688,13sleep10-21swapper/108:47:141
119492686,11sleep30-21swapper/312:08:183
119492686,11sleep30-21swapper/312:08:183
2553426710,27sleep20-21swapper/211:08:352
254102677,44sleep30-21swapper/311:08:283
2115126714,32sleep020657-21/usr/sbin/munin10:53:440
209026717,25sleep00-21swapper/008:08:460
1660326715,33sleep216757-21latency_hist12:23:192
600226615,26sleep023822-1kworker/0:2H08:23:190
512926624,17sleep10-21swapper/108:18:441
3246526611,43sleep213441-21snmpd09:46:202
271242669,10sleep30-21swapper/307:44:023
260526622,26sleep10-21swapper/109:53:461
2213026614,36sleep222159-21nfs4_client09:13:392
2213026614,36sleep222159-21nfs4_client09:13:392
1374426611,37sleep213750-21/usr/sbin/munin12:13:342
42182656,12sleep00-21swapper/008:16:450
3027526511,38sleep030339-21ps09:38:430
179262656,11sleep30-21swapper/312:26:313
1784126515,26sleep00-21swapper/008:58:470
949526418,28sleep19537-21modprobe10:18:161
2877426416,29sleep228776-21apt-get11:18:282
2877426416,29sleep228776-21apt-get11:18:282
2315626421,21sleep20-21swapper/207:33:352
2315626421,21sleep20-21swapper/207:33:352
1599326414,35sleep216055-21ps08:53:452
1528726417,18sleep10-21swapper/108:53:191
927626322,21sleep20-21swapper/211:58:362
927626322,21sleep20-21swapper/211:58:362
3025926320,26sleep230261-21ntp_states09:38:412
1624526312,19sleep20-21swapper/210:38:362
1624526312,19sleep20-21swapper/210:38:362
1226026315,23sleep00-21swapper/012:08:360
1226026315,23sleep00-21swapper/012:08:360
633926220,27sleep00-21swapper/011:49:220
633926220,27sleep00-21swapper/011:49:220
294926217,22sleep00-21swapper/011:38:400
294926217,22sleep00-21swapper/011:38:400
2482926220,20sleep20-21swapper/207:38:362
2444526213,28sleep10-21swapper/107:38:191
2444526213,28sleep10-21swapper/107:38:191
2431426223,23sleep023559-21/usr/sbin/munin09:19:460
2431426223,23sleep023559-21/usr/sbin/munin09:19:460
2355826213,23sleep20-21swapper/209:18:272
2355826213,23sleep20-21swapper/209:18:272
1549726217,19sleep10-21swapper/108:53:271
1354526226,18sleep20-21swapper/207:03:442
323992618,36sleep132457-21sendmail_mailtr11:28:441
2152426119,19sleep20-21swapper/209:13:182
147132619,40sleep00-21swapper/012:15:220
107826115,34sleep21194-21timerwakeupswit09:48:472
78412609,33sleep17890-21ntp_states11:53:431
638426021,21sleep20-21swapper/208:23:372
3144326018,17sleep10-21swapper/107:58:461
1785826017,24sleep00-21swapper/010:43:380
541425912,22sleep00-21swapper/011:48:170
2968325917,22sleep10-21swapper/107:53:411
2695925912,30sleep126960-21cpu11:13:301
2357725920,25sleep023600-21munin-run11:03:160
1966325919,17sleep00-21swapper/012:33:180
1913525916,26sleep019129-21apt-get10:48:310
1735225913,26sleep00-21swapper/010:43:180
1735225913,26sleep00-21swapper/010:43:180
138232596,12sleep10-21swapper/110:28:501
138232596,12sleep10-21swapper/110:28:501
1218225917,24sleep112248-21latency_hist08:43:181
1064925920,23sleep00-21swapper/012:03:350
1064925920,23sleep00-21swapper/012:03:350
585625815,23sleep10-21swapper/111:48:261
585625815,23sleep10-21swapper/111:48:261
304025816,26sleep23059-21apt-get08:13:282
1913625818,19sleep10-21swapper/110:48:321
1905525816,21sleep10-21swapper/109:03:411
1612925819,20sleep00-21swapper/008:53:470
1095625813,21sleep20-21swapper/212:03:412
1095625813,21sleep20-21swapper/212:03:412
396725716,23sleep23993-21/usr/sbin/munin09:58:402
2616825716,21sleep10-21swapper/107:43:271
227272575,10sleep10-21swapper/107:32:331
227272575,10sleep10-21swapper/107:32:331
1456625713,27sleep014619-21switchtime08:48:460
829325614,24sleep08340-21users11:53:500
3086525614,24sleep030861-21apt-get07:58:280
299425615,25sleep13005-21ntp_states11:38:411
299425615,25sleep13005-21ntp_states11:38:411
2021625614,25sleep120213-21apt-get09:08:291
184292568,26sleep10-21swapper/107:18:281
1835425614,24sleep118348-21apt-get12:28:311
178672566,41sleep00-21swapper/012:25:430
1407125612,18sleep10-21swapper/112:13:411
587625513,25sleep25373-21apt-get11:48:272
587625513,25sleep25373-21apt-get11:48:272
3106825516,22sleep131209-21sed11:23:481
2986325520,22sleep20-21swapper/207:53:462
2314425515,24sleep023150-21df_inode07:33:330
2314425515,24sleep023150-21df_inode07:33:330
1596825517,20sleep00-21swapper/007:13:180
1443225516,19sleep20-21swapper/208:48:442
1248625511,32sleep112488-21sshd10:26:361
115625516,21sleep00-21swapper/011:33:360
1084425512,25sleep210863-21apt-get08:38:272
52242547,11sleep10-21swapper/110:03:351
2675425414,24sleep226760-21df_inode09:28:322
2526525419,19sleep00-21swapper/007:38:460
1799025417,19sleep10-21swapper/108:58:481
1764225417,18sleep20-21swapper/208:58:452
1256825412,24sleep212574-21ntp_states12:08:422
1256825412,24sleep212574-21ntp_states12:08:422
1134225414,14sleep00-21swapper/008:38:450
715025313,18sleep10-21swapper/110:08:441
2225725311,25sleep122251-21apt-get10:58:291
1107025313,23sleep111099-21munin-run10:23:171
772025214,19sleep10-21swapper/108:28:251
2753125213,21sleep127584-21users09:28:471
2695525213,23sleep226952-21apt-get11:13:272
1843025211,24sleep218426-21apt-get07:18:292
160125215,23sleep10-21swapper/108:08:351
2822825112,23sleep228223-21apt-get09:33:292
2194325116,22sleep10-21swapper/110:58:171
1290725111,22sleep212900-21apt-get10:28:312
1290725111,22sleep212900-21apt-get10:28:312
937825016,18sleep00-21swapper/011:58:390
937825016,18sleep00-21swapper/011:58:390
2799325010,17sleep00-21swapper/009:33:180
1525925013,15sleep00-21swapper/012:18:330
667424912,21sleep26671-21apt-get10:08:282
3033324914,19sleep00-21swapper/011:23:310
2524724911,20sleep225343-21perl09:23:362
2524724911,20sleep225343-21perl09:23:362
2235924913,19sleep222462-21perl10:58:342
2031524912,20sleep220352-21ntp_states12:33:402
1252524911,23sleep112567-21ntp_states12:08:411
1252524911,23sleep112567-21ntp_states12:08:411
2386524811,21sleep223859-21apt-get11:03:302
1285524711,22sleep212874-21ntp_states08:43:442
598524611,20sleep26032-21users10:03:482
1945824410,18sleep00-21swapper/009:03:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional