You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 22:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Fri May 15, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4950268306,6812sleep029506-21kworker/0:011:15:150
229072479316,4764sleep116745-21kworker/1:108:48:411
244892418318,4152sleep224411-21kworker/2:112:19:152
9419232629,3242sleep24824-21kworker/2:109:48:512
12476211818,1160sleep012522-21ntpq08:14:470
19859210767,1028sleep019038-1kworker/0:1H12:04:550
13258287828,815sleep013300-21ntpq09:59:480
1930828589,806sleep019330-21apt-get12:04:220
11859250021,463sleep111868-21/usr/sbin/munin08:14:241
16463245240,307sleep00-21swapper/011:54:370
16593239810,375sleep016679-21ls10:09:530
453236115,329sleep0491-21latency09:19:460
25218235216,323sleep325254-21ntpq12:19:483
30816233423,296sleep030931-21latency_hist09:14:240
26074233223,292sleep026092-21apt08:59:220
21228228018,244sleep121380-21gpgv12:09:241
29547227015,238sleep329583-21/usr/sbin/munin12:34:383
32078225873,167sleep00-21swapper/010:59:440
9545223165,87sleep09683-21latency_hist09:49:260
5312223126,119sleep00-21swapper/009:34:520
5312223126,119sleep00-21swapper/009:34:520
23018222513,193sleep023032-21latency_hist08:49:240
15494222181,56sleep015583-21switchtime11:49:520
1441022119,162sleep014414-21find10:04:320
17035218710,86sleep00-21swapper/008:29:380
6740218123,123sleep00-21swapper/011:22:540
12518217819,29sleep00-21swapper/009:58:510
25110217320,108sleep025217-21latency_hist10:39:250
2118421708,128sleep0363-21snmpd08:41:250
26079216838,115sleep026081-21cron12:24:210
649621609,136sleep06545-21fschecks_count09:39:410
12002216019,96sleep00-21swapper/011:39:420
785221578,133sleep27889-21ntpq07:59:472
785221578,133sleep27889-21ntpq07:59:472
31195215772,38sleep00-21swapper/007:30:090
31195215772,38sleep00-21swapper/007:30:090
8122215559,52sleep00-21swapper/007:59:530
8122215559,52sleep00-21swapper/007:59:530
17655215311,41sleep00-21swapper/010:14:420
999215110,87sleep01061-21fschecks_time11:04:410
999215110,87sleep01061-21fschecks_time11:04:410
24201215120,119sleep00-21swapper/008:50:130
22140215030,105sleep022186-21mailstats12:10:000
245721438,93sleep00-21swapper/011:09:330
245721438,93sleep00-21swapper/011:09:330
1163021425,29sleep00-21swapper/008:12:010
2791121418,93sleep028048-21grep07:19:550
2248021418,52sleep022522-21latency10:29:440
21381214017,99sleep221112-21apt-get12:09:252
1004021406,30sleep00-21swapper/008:06:280
5641213943,17sleep00-21swapper/007:54:240
31571213710,88sleep031672-21latency_hist10:59:240
29016213330,91sleep029107-21switchtime10:49:510
18764213122,46sleep00-21swapper/008:34:430
9942213041,66sleep00-21swapper/009:49:420
28153212912,100sleep028187-21apt10:49:220
28317212736,77sleep028342-21awk09:04:470
29186212617,95sleep00-21swapper/007:24:480
10723212636,76sleep010787-21sendmail_mailst11:34:500
9094212320,80sleep30-21swapper/308:04:413
27499212311,98sleep327571-21find10:44:533
23779212319,83sleep00-21swapper/012:14:520
19327212321,77sleep00-21swapper/010:19:420
15249212210,91sleep00-21swapper/008:24:250
15249212210,91sleep00-21swapper/008:24:250
15191212110,24sleep10-21swapper/111:49:451
27112211410,22sleep00-21swapper/008:59:550
26656211419,71sleep30-21swapper/310:44:243
28526211329,45sleep028637-21sed12:29:510
23854211318,77sleep123878-21fschecks_count10:34:411
5589211115,79sleep31-21systemd07:54:233
28220211112,80sleep328315-21memory09:04:463
27629211125,74sleep027670-21systemd-run12:29:220
18930211123,65sleep319073-21latency_hist10:19:253
1670921095,12sleep00-21swapper/008:28:570
1670921095,12sleep00-21swapper/008:28:570
12562210920,73sleep312566-21cron09:59:223
21263210824,45sleep021343-21awk10:24:510
28847210712,25sleep10-21swapper/109:04:561
24665210715,79sleep00-21swapper/007:10:460
3280210627,44sleep03275-21apt-get09:29:350
3156121069,62sleep031638-21mailstats09:14:530
30062210611,80sleep230137-21find12:34:502
8366210514,18sleep30-21swapper/307:59:563
8366210514,18sleep30-21swapper/307:59:563
604321050,9sleep30-21swapper/307:54:423
3044210519,70sleep23053-21latency_hist09:29:252
12852210212,74sleep212870-21/usr/sbin/munin09:59:332
8404210116,70sleep28454-21unin-run11:29:222
21923210022,61sleep321947-21munin-plugin-st10:29:223
19378210019,54sleep219495-21dpkg12:04:242
14628210014,62sleep2662-21cron11:49:222
187429926,56sleep22007-21wc07:39:522
1668129919,54sleep116832-21unixbench_multi10:09:541
1356229727,47sleep313705-21grep09:59:553
299229619,12sleep00-21swapper/009:29:230
1711829423,53sleep317135-21fschecks_count08:29:413
1227929311,20sleep30-21swapper/311:39:503
67229221,49sleep3722-21latency_hist11:04:233
922929110,51sleep20-21swapper/208:04:442
753829125,20sleep00-21swapper/011:24:480
92302908,18sleep00-21swapper/011:29:530
2209829030,37sleep222095-21apt-get07:04:352
492728920,50sleep23985-21/usr/sbin/munin11:14:572
38228913,54sleep10-21swapper/107:34:541
2699828916,58sleep327103-21timerandwakeup08:59:543
2343528913,34sleep00-21swapper/007:09:250
845528815,57sleep38515-21cat11:29:233
3053228818,49sleep230048-21apt-get07:29:342
3053228818,49sleep230048-21apt-get07:29:342
286628828,33sleep30-21swapper/307:44:423
1280528821,47sleep312955-21sed08:14:553
3151528713,61sleep231517-21ntp_states09:14:512
2564528621,16sleep00-21swapper/007:14:380
2534928614,55sleep225415-21cpuspeed10:39:392
1460228611,51sleep113904-21/usr/sbin/munin08:20:061
1460228611,51sleep113904-21/usr/sbin/munin08:20:061
99328510,48sleep30-21swapper/311:04:403
99328510,48sleep30-21swapper/311:04:403
1983828522,38sleep319862-21munin-run08:39:223
1887928552,17sleep20-21swapper/210:19:242
62242840,6sleep20-21swapper/207:54:452
325652838,34sleep00-21swapper/007:34:490
3232228321,40sleep132324-21cron09:19:221
2236128230,29sleep30-21swapper/307:04:443
2849928131,26sleep20-21swapper/209:04:532
27832819,34sleep00-21swapper/007:44:390
1991828129,27sleep10-21swapper/108:39:241
2985128013,51sleep230053-21cat10:54:252
2614428026,29sleep30-21swapper/307:14:533
2172328023,30sleep20-21swapper/212:09:492
2457527924,16sleep30-21swapper/308:54:243
2770827822,25sleep20-21swapper/209:04:252
24902789,26sleep00-21swapper/007:44:220
3022427713,44sleep329718-21apt-get10:54:343
1489827716,40sleep114936-21grep10:04:491
295042765,13sleep30-21swapper/309:09:353
2335327613,46sleep123453-21grep08:49:421
1557227627,22sleep10-21swapper/108:24:421
1557227627,22sleep10-21swapper/108:24:421
1343827622,13sleep00-21swapper/011:44:390
2542627521,13sleep00-21swapper/010:39:410
2109627524,30sleep321117-21munin-run12:09:223
1313027511,48sleep113236-21latency_hist11:44:241
298292745,30sleep00-21swapper/012:34:440
2167927415,44sleep221698-21apt-get08:44:342
1390327416,39sleep213407-21apt-get08:19:342
1390327416,39sleep213407-21apt-get08:19:342
1337827421,32sleep213401-21munin-run08:19:222
98927315,44sleep21021-21latency_hist07:39:242
721827310,43sleep00-21swapper/007:59:250
1949827325,22sleep30-21swapper/312:04:253
1683127317,36sleep116862-21/usr/sbin/munin11:54:461
1030927317,35sleep110323-21latency_hist08:09:231
192422725,12sleep10-21swapper/112:03:341
2950727119,31sleep10-21swapper/109:09:371
1360427119,20sleep10-21swapper/108:19:251
1341527120,29sleep10-21swapper/109:59:531
937727017,26sleep20-21swapper/211:29:552
725427014,40sleep17399-21latency_hist07:59:261
609227020,32sleep26094-21cron09:39:222
609227020,32sleep26094-21cron09:39:222
381227016,29sleep20-21swapper/211:14:242
3264327014,39sleep232711-21df09:19:392
3004027024,21sleep30-21swapper/307:29:233
3003427021,30sleep20-21swapper/207:29:222
2852527011,13sleep10-21swapper/110:49:391
2564627015,29sleep10-21swapper/107:14:391
3180926912,43sleep131857-21grep10:59:371
213232697,17sleep10-21swapper/108:43:161
3042726813,38sleep130490-21fw_packets10:55:171
1186926814,37sleep212010-21cut08:14:252
3240126721,21sleep30-21swapper/309:19:243
2061726722,28sleep30-21swapper/308:39:503
872326620,26sleep18735-21open_files09:45:351
559826624,21sleep20-21swapper/211:19:372
1159826613,26sleep30-21swapper/311:39:253
371526516,27sleep33738-21ntp_states09:29:503
338026521,20sleep10-21swapper/109:29:441
3272826515,33sleep1320-21tune2fs09:19:431
2523826517,22sleep20-21swapper/207:14:282
1679126514,35sleep216827-21cut08:29:232
1679126514,35sleep216827-21cut08:29:232
1469726517,33sleep314714-21kernelversion10:04:433
1138726516,18sleep20-21swapper/209:54:412
879526416,28sleep30-21swapper/309:45:373
2490726423,22sleep10-21swapper/112:19:421
2429226421,29sleep324379-21sort07:09:523
2360426420,27sleep30-21swapper/312:14:473
1682826416,28sleep316978-21latency_hist08:29:243
60642637,15sleep30-21swapper/311:19:503
2686826320,24sleep10-21swapper/108:59:511
2609926320,27sleep126151-21unin-run08:59:231
1938526315,33sleep119411-21kernelversion10:19:431
1569326314,30sleep315743-21/usr/sbin/munin08:24:443
1091926318,25sleep10-21swapper/109:53:151
102226312,26sleep30-21swapper/307:39:253
449826220,19sleep10-21swapper/111:14:521
408926217,20sleep10-21swapper/107:49:241
3112526211,31sleep331193-21/usr/sbin/munin07:30:083
3112526211,31sleep331193-21/usr/sbin/munin07:30:083
2863826227,21sleep10-21swapper/112:29:521
2839026217,27sleep20-21swapper/212:29:492
2347626225,21sleep30-21swapper/310:34:233
1866426218,23sleep20-21swapper/208:34:402
1834326214,23sleep20-21swapper/208:34:242
772026122,20sleep20-21swapper/209:44:232
586726115,24sleep10-21swapper/111:19:441
2765326122,22sleep20-21swapper/207:19:492
2229726123,16sleep10-21swapper/110:29:411
1002726118,19sleep30-21swapper/311:34:243
465526019,19sleep30-21swapper/311:14:543
298962609,12sleep20-21swapper/209:09:462
2219826018,16sleep10-21swapper/107:04:421
1493826013,34sleep215017-21systemctl10:04:502
1157126020,21sleep10-21swapper/109:54:441
984625916,19sleep30-21swapper/309:49:413
687625917,25sleep16940-21ps09:39:501
36812594,11sleep20-21swapper/209:29:492
2770425921,19sleep30-21swapper/312:29:253
2741225917,27sleep127465-21if_err_eth007:19:441
2364825918,24sleep223689-21ntpq08:49:472
460125824,19sleep20-21swapper/207:49:462
3018025813,18sleep10-21swapper/107:29:251
2526225812,28sleep225269-21ntp_states08:54:482
2074825816,19sleep10-21swapper/110:24:401
1909125815,24sleep20-21swapper/210:19:322
1570925819,21sleep20-21swapper/211:49:542
109572586,11sleep30-21swapper/308:09:463
39325712,21sleep30-21swapper/309:19:453
2636825716,24sleep226361-21apt-get12:24:362
2549025721,20sleep30-21swapper/308:54:533
2416525714,20sleep30-21swapper/310:34:483
2331025715,28sleep123317-21/usr/sbin/munin12:14:411
128125714,22sleep10-21swapper/111:04:461
128125714,22sleep10-21swapper/111:04:461
780925613,27sleep37893-21dpkg09:44:243
483625616,18sleep30-21swapper/309:34:393
483625616,18sleep30-21swapper/309:34:393
3180525617,23sleep231802-21apt-get10:59:342
3149325615,20sleep20-21swapper/210:59:222
303425619,18sleep20-21swapper/211:09:522
2388025616,23sleep223975-21perl10:34:422
2077425615,18sleep20-21swapper/210:24:412
1958425619,17sleep30-21swapper/310:19:503
180425615,20sleep10-21swapper/107:39:511
1742425615,22sleep10-21swapper/108:29:471
1694825614,27sleep217022-21mailstats11:54:512
15982567,24sleep30-21swapper/307:39:453
714625514,24sleep17140-21apt-get11:24:371
483325513,25sleep14827-21apt-get09:34:371
483325513,25sleep14827-21apt-get09:34:371
273425512,24sleep32827-21memory11:09:453
273425512,24sleep32827-21memory11:09:453
1336625514,26sleep113360-21apt-get11:44:361
2689025413,25sleep126887-21apt-get10:44:351
1753025415,23sleep117525-21apt-get10:14:341
1702325414,19sleep30-21swapper/311:54:523
3072725316,23sleep130745-21cut07:29:581
3072725316,23sleep130745-21cut07:29:581
2791125312,24sleep327907-21apt-get12:29:343
2383625319,20sleep10-21swapper/107:09:421
2302725315,23sleep223114-21users10:29:542
1023925312,24sleep210233-21apt-get11:34:352
98625212,16sleep20-21swapper/211:04:392
98625212,16sleep20-21swapper/211:04:392
878225214,24sleep18842-21/usr/sbin/munin11:29:421
486525218,17sleep10-21swapper/107:49:531
2850925213,22sleep328516-21df_inode10:49:373
2480325214,20sleep224815-21apt-get12:19:352
2078225215,16sleep20-21swapper/208:39:542
173425212,23sleep31802-21df09:24:373
1444525211,25sleep313904-21/usr/sbin/munin08:20:033
1444525211,25sleep313904-21/usr/sbin/munin08:20:033
782225113,20sleep10-21swapper/107:59:461
782225113,20sleep10-21swapper/107:59:461
2997025113,16sleep10-21swapper/112:34:491
1413725111,22sleep314267-21awk11:44:543
100232519,17sleep20-21swapper/209:49:442
2409525010,17sleep20-21swapper/207:09:472
2163725011,23sleep121615-21apt-get12:09:441
1605825013,21sleep316067-21df_inode10:09:403
1227225012,23sleep212278-21ntp_states11:39:492
225542496,14sleep30-21swapper/308:44:553
2164024912,22sleep321707-21df12:09:463
1078924913,23sleep110861-21switchtime11:34:511
30492489,17sleep20-21swapper/207:44:452
2848624810,24sleep328521-21awk07:24:223
25342488,16sleep10-21swapper/111:09:391
25342488,16sleep10-21swapper/111:09:391
2532224820,15sleep10-21swapper/110:39:341
181324713,17sleep10-21swapper/109:24:391
1613824718,16sleep20-21swapper/210:09:432
263672468,20sleep10-21swapper/112:24:351
1581924513,17sleep30-21swapper/311:49:553
2689124310,17sleep20-21swapper/210:44:362
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional