You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 16:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Mar 11, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
268552525612,5232sleep314326-21kworker/3:012:15:373
268552525612,5232sleep314326-21kworker/3:012:15:373
11743259823,558sleep311801-21/usr/sbin/munin11:28:383
31027256517,531sleep231090-21/usr/sbin/munin09:03:462
20011243818,404sleep220023-21/usr/sbin/munin08:28:382
1338240515,371sleep11414-21fschecks_time07:28:391
10356240529,319sleep110409-21sort07:58:211
9540233849,108sleep10-21swapper/109:38:361
1889230145,218sleep11933-21ntpq09:13:461
1889230145,218sleep11933-21ntpq09:13:461
155202272103,23sleep10-21swapper/109:58:221
155202272103,23sleep10-21swapper/109:58:221
23137223315,203sleep223197-21fschecks_time08:38:392
19233222694,25sleep10-21swapper/110:08:441
16513221611,183sleep116516-21cron08:18:191
30133221550,107sleep10-21swapper/107:18:211
25822221319,98sleep125966-21unixbench_singl10:28:501
6384221118,165sleep16402-21apt-get09:28:311
28129221014,77sleep128281-21unixbench_singl12:18:501
12222220222,164sleep112347-21timerwakeupswit11:28:481
2381322018,179sleep123871-21sed10:23:411
12419219912,162sleep10-21swapper/108:03:421
12419219912,162sleep10-21swapper/108:03:421
5938219313,166sleep36025-21unin-run09:28:213
5938219313,166sleep36025-21unin-run09:28:213
31743218715,115sleep131764-21modprobe12:33:191
1085721869,130sleep110871-21kernelversion07:58:421
17784218414,17sleep10-21swapper/111:48:221
13345217010,117sleep112832-21apt-get11:33:371
27156216971,83sleep127300-21latency_hist12:18:211
27156216971,83sleep127300-21latency_hist12:18:211
32516216810,141sleep132580-21mailstats12:33:471
23658216827,41sleep123743-21ls08:38:501
2830221666,16sleep30-21swapper/310:38:373
13783216526,20sleep10-21swapper/108:08:391
12902216511,111sleep112956-21latency09:48:411
17314216116,133sleep117315-21cpu10:03:351
1018216118,26sleep10-21swapper/107:28:221
32671216010,110sleep132739-21ls09:08:451
12772215814,125sleep312842-21fw_packets09:48:393
3652215412,27sleep10-21swapper/109:18:501
3652215412,27sleep10-21swapper/109:18:501
4846215318,21sleep10-21swapper/111:08:221
16486215340,92sleep10-21swapper/109:58:511
19777214733,19sleep10-21swapper/111:53:391
11571214581,45sleep10-21swapper/109:43:491
1984214228,85sleep126604-21cyclictest10:58:331
20176214183,43sleep10-21swapper/108:28:411
767721377,116sleep215336-1kworker/2:0H11:15:252
767721377,116sleep215336-1kworker/2:0H11:15:252
6156213428,23sleep16323-21store09:28:231
32006213212,105sleep032025-21apt-get12:33:300
1925121328,108sleep00-21swapper/008:23:520
2820021308,105sleep10-21swapper/110:38:311
30197212911,93sleep230235-21kworker/u8:012:28:192
16823212721,26sleep10-21swapper/108:18:321
31752212613,76sleep131888-21sed07:23:221
32163212333,62sleep032157-21apt-get07:23:370
24989212326,76sleep025135-21sed12:08:520
860421228,13sleep10-21swapper/111:19:201
1038621227,103sleep110432-21sed11:23:441
3443211621,78sleep33472-21nfs4_client09:18:443
3443211621,78sleep33472-21nfs4_client09:18:443
3012221155,13sleep10-21swapper/112:27:221
32473211412,20sleep00-21swapper/009:08:400
30845211311,80sleep10-21swapper/109:03:411
2396021138,93sleep124007-21systemctl12:08:201
1621921135,96sleep00-21swapper/008:14:390
13346211213,78sleep213488-21/usr/sbin/munin09:48:512
3030521115,11sleep10-21swapper/109:00:131
2714421115,12sleep10-21swapper/108:49:521
563621099,87sleep15639-21cron07:43:201
2787421095,17sleep10-21swapper/110:37:541
7142210810,73sleep10-21swapper/111:13:511
7142210810,73sleep10-21swapper/111:13:511
5554210712,79sleep25067-21/usr/sbin/munin11:08:472
838921058,15sleep10-21swapper/109:33:441
691621057,14sleep30-21swapper/311:13:453
691621057,14sleep30-21swapper/311:13:453
2085221048,83sleep13243-1kworker/1:1H10:13:441
2053210216,71sleep32063-21df_inode10:58:353
6644210114,73sleep06618-21/usr/sbin/munin11:13:380
6644210114,73sleep06618-21/usr/sbin/munin11:13:380
2977221008,80sleep129818-21grep10:43:351
2683421007,16sleep10-21swapper/107:08:361
112521007,19sleep1382-21/usr/sbin/munin10:53:531
222662996,14sleep10-21swapper/110:18:411
2156429819,63sleep221587-21fschecks_count08:33:422
63752976,15sleep10-21swapper/107:43:441
225922975,19sleep10-21swapper/108:36:321
149242967,50sleep10-21swapper/111:38:331
1037529618,62sleep010385-21uname11:23:430
285372957,14sleep10-21swapper/108:53:521
1939529520,59sleep218838-21/usr/sbin/munin10:08:492
1855929516,56sleep10-21swapper/111:48:511
900029415,56sleep38434-21/usr/sbin/munin11:19:303
2533529432,37sleep025399-21sort07:03:390
1502329421,50sleep315128-21fschecks_time11:38:393
56362936,11sleep10-21swapper/109:25:121
2746629317,55sleep027594-21latency_hist08:53:220
120142930,7sleep20-21swapper/208:03:222
3167229223,20sleep00-21swapper/007:23:210
312842925,12sleep10-21swapper/110:48:231
30742926,11sleep10-21swapper/107:33:431
243162926,48sleep10-21swapper/112:08:371
196332925,11sleep10-21swapper/108:28:161
1706229215,59sleep317080-21grep10:03:213
84002915,12sleep10-21swapper/107:49:051
2864529113,58sleep228670-21modprobe12:23:192
274512916,13sleep30-21swapper/310:33:523
220482915,12sleep10-21swapper/111:58:521
82529014,60sleep3382-21/usr/sbin/munin10:53:463
63432895,11sleep10-21swapper/111:13:141
3062328918,23sleep00-21swapper/009:03:230
2400428813,60sleep224017-21/usr/sbin/munin10:23:462
2639928718,53sleep026475-21sed10:33:210
2352628725,47sleep323585-21sendmail_mailqu08:38:483
1055928723,48sleep210633-21wc11:23:492
258102865,9sleep10-21swapper/108:47:421
564028412,58sleep25685-21unin-run07:43:212
2838628419,47sleep028532-21unixbench_multi08:53:510
1693628411,58sleep216447-21/usr/sbin/munin11:43:472
144242848,15sleep30-21swapper/309:53:403
144242848,15sleep30-21swapper/309:53:403
2141528318,39sleep221451-21latency11:58:412
2424728215,32sleep30-21swapper/312:08:343
1968828227,34sleep20-21swapper/208:28:222
2625328112,50sleep225767-21/usr/sbin/munin12:13:462
2625328112,50sleep225767-21/usr/sbin/munin12:13:462
149252816,15sleep20-21swapper/211:38:342
3163127921,43sleep331656-21munin-run07:23:203
99612788,15sleep00-21swapper/007:54:170
2592327723,27sleep30-21swapper/307:03:503
2407827726,20sleep00-21swapper/010:23:480
2330527714,45sleep223408-21/usr/sbin/munin12:03:492
1835027716,27sleep30-21swapper/308:23:303
125027713,39sleep21379-21latency_hist09:13:222
878027613,40sleep28909-21latency_hist07:53:222
2633627617,27sleep00-21swapper/012:13:480
2633627617,27sleep00-21swapper/012:13:480
1468827620,21sleep20-21swapper/211:38:222
138252767,13sleep30-21swapper/311:34:243
320302759,46sleep30-21swapper/312:33:333
157427430,24sleep30-21swapper/309:13:403
157427430,24sleep30-21swapper/309:13:403
1501727421,25sleep00-21swapper/008:13:220
78527317,41sleep2823-21ntp_kernel_err10:53:452
68812730,8sleep20-21swapper/209:29:082
852127216,33sleep28575-21cpuspeed11:19:172
1993727212,28sleep00-21swapper/011:53:420
75402716,53sleep00-21swapper/009:31:370
315912717,14sleep20-21swapper/210:48:412
2664327116,24sleep20-21swapper/208:48:482
1691827120,22sleep00-21swapper/008:18:390
966227017,25sleep30-21swapper/307:53:513
592527015,38sleep35994-21users11:08:523
2194027013,43sleep321974-21ps08:33:523
1673527013,45sleep016752-21kernelversion11:43:410
1513327020,29sleep00-21swapper/011:38:400
1483727017,24sleep00-21swapper/009:53:490
1483727017,24sleep00-21swapper/009:53:490
1250327019,25sleep20-21swapper/208:03:432
1250327019,25sleep20-21swapper/208:03:432
568626917,38sleep35755-21sed07:43:223
458726925,24sleep30-21swapper/311:05:373
3038426916,38sleep030486-21timerandwakeup10:43:500
2622826921,22sleep30-21swapper/308:48:373
2199426911,38sleep022047-21users11:58:510
1997826920,27sleep00-21swapper/008:28:360
193526912,43sleep21944-21ntp_states09:13:472
193526912,43sleep21944-21ntp_states09:13:472
832626824,24sleep38344-21netstat09:33:423
2556726822,24sleep20-21swapper/210:28:472
1346426815,37sleep013467-21entropy11:34:040
640026719,32sleep06490-21latency_hist11:13:210
3024926715,36sleep030256-21latency_hist12:28:210
2506026714,34sleep324572-21/usr/sbin/munin08:43:493
2143226712,42sleep321433-21cron10:15:203
2001326719,26sleep20-21swapper/211:53:442
1814026718,25sleep30-21swapper/308:23:223
1582226717,29sleep30-21swapper/308:13:493
1233226716,27sleep30-21swapper/308:03:403
1233226716,27sleep30-21swapper/308:03:403
3209426622,24sleep00-21swapper/009:08:200
290412669,41sleep328985-21latency_hist08:58:223
141726619,28sleep20-21swapper/207:28:402
1270226613,28sleep00-21swapper/009:48:360
3056826518,29sleep30-21swapper/312:28:373
2650726519,27sleep226572-21systemd-run07:08:212
2591026518,23sleep00-21swapper/008:48:220
1564126518,32sleep215676-21ntp_kernel_pll_08:13:442
3059826422,26sleep30-21swapper/309:03:223
2198126414,23sleep00-21swapper/008:33:530
1499726413,34sleep315011-21latency_hist08:13:213
1213326423,21sleep00-21swapper/011:28:470
115832646,46sleep00-21swapper/007:59:460
1015326419,23sleep00-21swapper/009:38:520
959126312,21sleep20-21swapper/207:53:502
3048226316,30sleep230476-21apt-get12:28:322
3247126218,25sleep00-21swapper/010:51:580
29252627,42sleep00-21swapper/010:59:120
2905626217,26sleep20-21swapper/207:13:422
2354426211,37sleep323614-21/usr/sbin/munin10:23:353
1028426214,22sleep30-21swapper/311:23:423
723426118,19sleep30-21swapper/307:48:223
391726116,28sleep23963-21/usr/sbin/munin11:03:472
282926116,24sleep00-21swapper/009:18:210
282926116,24sleep00-21swapper/009:18:210
2692026118,25sleep00-21swapper/007:08:390
2269726112,34sleep322727-21cpu12:03:343
179202610,7sleep00-21swapper/010:03:490
122512615,10sleep00-21swapper/009:47:070
1192526116,22sleep20-21swapper/211:28:412
1105726113,34sleep311097-21ps07:58:483
3177926013,31sleep331816-21ps10:48:463
2199426014,32sleep222017-21grep10:18:342
1946726012,19sleep30-21swapper/310:08:503
1704126011,27sleep20-21swapper/208:18:412
48862599,24sleep00-21swapper/007:38:510
462325917,22sleep00-21swapper/009:23:340
462325917,22sleep00-21swapper/009:23:340
417425919,21sleep00-21swapper/011:03:530
295052596,12sleep30-21swapper/312:23:483
289242599,34sleep028997-21fw_packets07:13:400
256425917,26sleep00-21swapper/007:33:220
1602625912,25sleep00-21swapper/009:58:420
7502588,38sleep30-21swapper/307:25:383
2924225811,26sleep30-21swapper/307:13:473
2867325818,24sleep328689-21systemd-run12:23:203
2843425824,20sleep20-21swapper/210:38:402
2822625814,27sleep028218-21apt-get10:38:340
1886325816,25sleep018856-21apt-get10:08:340
1222225813,18sleep00-21swapper/008:03:370
1222225813,18sleep00-21swapper/008:03:370
956425714,25sleep29613-21df09:38:372
747225714,25sleep07466-21apt-get07:48:360
37472577,38sleep00-21swapper/007:34:190
2978625717,17sleep20-21swapper/208:58:492
280225715,24sleep20-21swapper/207:33:362
2273225717,25sleep022769-21df12:03:350
1575225718,23sleep215749-21apt-get09:58:312
649925616,22sleep00-21swapper/007:43:470
592125617,18sleep20-21swapper/207:43:332
459925616,23sleep24618-21apt-get09:23:322
459925616,23sleep24618-21apt-get09:23:322
3127325615,24sleep331295-21ntp_states09:03:513
2869125613,18sleep00-21swapper/012:23:210
2123125615,25sleep321325-21fschecks_time11:58:373
1965325615,20sleep30-21swapper/311:53:373
977625517,23sleep39808-21irqstats09:38:423
929625519,19sleep20-21swapper/209:38:212
878025515,21sleep00-21swapper/011:19:230
795725514,25sleep07953-21apt-get09:33:310
705625521,19sleep30-21swapper/309:29:143
433425514,25sleep24364-21munin-run09:23:192
433425514,25sleep24364-21munin-run09:23:192
2704325513,22sleep20-21swapper/207:08:412
2040925513,25sleep220402-21apt-get10:13:332
43925418,18sleep20-21swapper/207:24:082
284025415,16sleep20-21swapper/210:58:502
2524025415,24sleep125242-21apt-get07:03:321
2091525416,19sleep00-21swapper/010:13:470
2066825415,16sleep30-21swapper/308:28:513
438625315,18sleep00-21swapper/009:23:210
438625315,18sleep00-21swapper/009:23:210
3226725312,17sleep20-21swapper/212:33:402
3031525316,16sleep30-21swapper/310:43:493
2814125321,17sleep30-21swapper/307:13:233
1753425315,22sleep317582-21irqstats10:03:413
40925216,17sleep00-21swapper/010:53:350
2710425214,22sleep227110-21ntp_states10:33:472
2487125213,22sleep224965-21/usr/sbin/munin08:43:432
132525214,23sleep31331-21df_inode07:28:373
298525113,24sleep33043-21/usr/sbin/munin07:33:413
2555425119,15sleep20-21swapper/212:13:222
2201825115,20sleep322065-21df10:18:353
1816825113,23sleep218215-21users10:03:522
1799725115,19sleep318015-21apt-get11:48:323
136982518,13sleep20-21swapper/208:08:362
2533225012,21sleep325361-21irqstats10:28:403
2358725015,19sleep00-21swapper/008:38:490
219072509,13sleep00-21swapper/010:18:230
313332497,33sleep20-21swapper/207:19:312
2896424914,21sleep229040-21systemd-run08:58:212
1933224815,17sleep00-21swapper/011:53:220
1697924815,16sleep30-21swapper/311:43:483
1151024812,20sleep211539-21ntp_states09:43:462
313272477,14sleep00-21swapper/012:28:500
755024613,19sleep37557-21diskstats07:48:393
2976824613,16sleep20-21swapper/210:43:322
42224511,16sleep30-21swapper/309:08:513
1829324513,17sleep20-21swapper/211:48:442
2671424411,15sleep00-21swapper/010:33:370
1437924216,13sleep20-21swapper/209:53:392
1437924216,13sleep20-21swapper/209:53:392
115402429,16sleep30-21swapper/309:43:473
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional