You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-29 - 15:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed Apr 29, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4822862124,8586sleep023320-21kworker/0:011:29:040
4822862124,8586sleep023320-21kworker/0:011:29:040
27247273418,697sleep227333-21/usr/sbin/munin11:10:582
1247525446,527sleep012486-21ntpq12:05:550
1247525446,527sleep012486-21ntpq12:05:550
4324249156,417sleep04445-21latency_hist08:15:300
6566235410,333sleep06575-21ntpq08:20:550
253802294152,37sleep025404-21latency_hist07:40:290
13753225070,44sleep013797-21ls12:10:470
25584224211,215sleep225596-21irqstats11:05:532
28730220454,81sleep028809-21date07:50:310
27425219510,169sleep127408-21apt-get07:45:471
21980218318,148sleep022023-21latency_hist10:55:290
14604217919,20sleep00-21swapper/007:05:490
11581217510,145sleep00-21swapper/010:20:490
12418217214,125sleep212513-21fschecks_time08:40:472
26056216717,70sleep026202-21unixbench_singl11:06:030
10739216414,125sleep00-21swapper/012:01:510
13640216121,121sleep013754-21latency_hist08:45:290
22144215940,108sleep013441-21snmpd07:29:310
94132157112,24sleep00-21swapper/008:30:480
31809215543,29sleep00-21swapper/008:00:280
4561215410,129sleep04609-21awk08:15:420
17743215410,129sleep017896-21unixbench_singl08:55:590
15174215419,19sleep00-21swapper/012:15:440
9878215212,126sleep05351-1kworker/dying11:59:330
1519821528,129sleep015341-21cat08:50:300
5723215112,103sleep05853-21wc10:00:590
5723215112,103sleep05853-21wc10:00:590
11021214315,113sleep311035-21awk12:01:583
2516212911,76sleep02619-21cmp08:10:290
8677212611,70sleep00-21swapper/010:10:560
8677212611,70sleep00-21swapper/010:10:560
19249212514,75sleep019293-21fschecks_time10:45:570
16891212514,67sleep00-21swapper/010:35:590
7092212410,29sleep00-21swapper/010:05:560
7092212410,29sleep00-21swapper/010:05:560
381321229,15sleep00-21swapper/011:40:310
2831221225,11sleep00-21swapper/009:30:470
32100212110,97sleep00-21swapper/008:00:440
1672321218,77sleep016715-21apt-get12:20:440
1672321218,77sleep016715-21apt-get12:20:440
5266212011,74sleep05311-21sh11:45:280
5266212011,74sleep05311-21sh11:45:280
1786321208,76sleep017856-21apt-get07:15:420
2435211812,87sleep32443-21/usr/sbin/munin11:35:373
16738211816,89sleep216750-21open_files07:10:562
21141211619,77sleep321212-21mailstats10:51:083
2691621155,45sleep00-21swapper/009:25:510
21763211513,19sleep20-21swapper/207:25:582
19578211212,25sleep00-21swapper/012:30:290
23960211166,24sleep024025-21if_eth011:00:500
19414211126,34sleep20-21swapper/207:20:452
8125210912,83sleep00-21swapper/008:25:550
27107210927,46sleep10-21swapper/111:10:401
8105210811,83sleep28116-21ntpq08:25:532
6213210812,42sleep10-21swapper/108:20:481
26144210859,40sleep015659-21munin-node09:22:480
3610210711,17sleep00-21swapper/009:55:460
3610210711,17sleep00-21swapper/009:55:460
18270210714,21sleep218337-21df12:25:432
11352210712,22sleep00-21swapper/008:35:570
31024210616,68sleep331056-21munin-plugin-st09:40:273
1585621066,58sleep00-21swapper/008:50:540
2253210519,66sleep32317-21latency_hist11:35:293
2198821057,57sleep00-21swapper/009:10:450
27201210375,15sleep00-21swapper/011:10:520
17439210213,73sleep117585-21awk12:21:011
17439210213,73sleep117585-21awk12:21:011
14408210212,16sleep30-21swapper/307:05:443
32620210112,23sleep00-21swapper/009:45:290
32620210112,23sleep00-21swapper/009:45:290
1409821015,12sleep30-21swapper/308:45:493
8164210013,68sleep17633-21/usr/sbin/munin08:25:561
7390210069,16sleep00-21swapper/011:50:530
7390210069,16sleep00-21swapper/011:50:530
23618210010,16sleep00-21swapper/011:00:300
20569210030,12sleep00-21swapper/007:24:100
2792029912,72sleep227935-21open_inodes07:46:252
2838029828,46sleep328399-21idleruntime-cro11:15:273
2407029827,12sleep00-21swapper/009:15:570
1762629811,72sleep317738-21cat08:55:583
1541029814,64sleep10-21swapper/108:50:391
484129512,66sleep24895-21latency_hist10:00:292
484129512,66sleep24895-21latency_hist10:00:292
2662029516,23sleep10-21swapper/111:10:281
1586329515,61sleep315950-21munin-node07:10:323
2906829412,66sleep229096-21/usr/sbin/munin11:15:542
2881029415,66sleep128932-21modprobe07:50:321
150262940,7sleep10-21swapper/107:05:571
1401329421,57sleep014081-21systemd07:05:290
1276029433,19sleep20-21swapper/207:00:512
227742938,13sleep00-21swapper/007:30:480
203592939,72sleep020360-21apt-get09:05:400
1635229345,32sleep00-21swapper/007:10:460
1294629317,56sleep113038-21fschecks_time10:25:471
932529212,64sleep39411-21fw_packets08:30:473
2976029259,20sleep029762-21apt-get09:35:400
2435029222,35sleep10-21swapper/111:00:591
198142928,16sleep00-21swapper/012:30:410
3000329110,64sleep230037-21irqstats09:35:502
2405829115,57sleep324077-21apt-get07:35:403
1856529143,32sleep00-21swapper/009:00:390
1836729115,58sleep318463-21mailstats07:15:583
1054029120,49sleep110591-21wc08:35:291
307129015,60sleep13074-21entropy08:10:541
185162905,73sleep00-21swapper/010:41:490
1799829016,54sleep117517-21/usr/sbin/munin10:40:571
37228944,12sleep00-21swapper/009:45:370
37228944,12sleep00-21swapper/009:45:370
289592898,15sleep00-21swapper/011:15:520
144142898,14sleep00-21swapper/010:30:380
2326228825,25sleep10-21swapper/109:15:301
677428743,11sleep00-21swapper/011:49:560
309722877,18sleep031086-21sed11:21:010
2667428742,12sleep00-21swapper/007:42:430
2667428742,12sleep00-21swapper/007:42:430
1286028533,36sleep00-21swapper/010:25:420
102062849,58sleep00-21swapper/010:15:530
315472830,11sleep331531-21/usr/sbin/munin11:25:273
2070628323,40sleep220726-21kernelversion09:06:432
1173928324,38sleep111766-21munin-run12:05:271
581728218,41sleep15819-21cron08:20:271
328628219,46sleep33399-21latency_hist09:55:293
2661928227,30sleep10-21swapper/109:25:391
145728220,30sleep10-21swapper/108:05:501
1393028212,49sleep30-21swapper/312:10:503
3140728117,40sleep331501-21fschecks_time09:40:473
3140728117,40sleep331501-21fschecks_time09:40:473
8912807,61sleep00-21swapper/011:30:460
610627917,36sleep20-21swapper/208:20:402
2679227923,27sleep30-21swapper/311:10:303
2296327919,42sleep123001-21grep07:30:531
320727817,43sleep10-21swapper/109:55:271
3041727815,21sleep10-21swapper/111:20:501
301402788,12sleep00-21swapper/007:54:220
29832786,12sleep00-21swapper/009:52:350
378027710,16sleep30-21swapper/309:55:493
378027710,16sleep30-21swapper/309:55:493
2845827716,46sleep228498-21latency09:30:492
1794827712,52sleep117969-21fschecks_count07:15:471
1086227719,36sleep110755-21/usr/sbin/munin08:35:451
1635927615,46sleep116363-21diskstats07:10:471
928827515,44sleep20-21swapper/208:30:462
2349627524,32sleep323493-21apt-get09:15:403
1544327516,37sleep315535-21/usr/sbin/munin12:15:513
1344027525,28sleep10-21swapper/107:01:061
2472827417,22sleep10-21swapper/109:20:061
2509527311,44sleep30-21swapper/311:05:303
2208427320,41sleep321890-21/usr/sbin/munin09:10:483
1997827320,28sleep20-21swapper/209:05:302
757027217,34sleep17607-21proc_pri11:50:581
757027217,34sleep17607-21proc_pri11:50:581
2995427217,37sleep329990-21awk11:20:283
833827123,19sleep30-21swapper/308:25:583
294327116,30sleep20-21swapper/211:36:012
2009027120,32sleep320304-21cat09:05:313
507927031,20sleep20-21swapper/210:00:412
507927031,20sleep20-21swapper/210:00:412
2688927024,27sleep226926-21modprobe07:45:282
2688927024,27sleep226926-21modprobe07:45:282
1715827020,31sleep10-21swapper/108:55:481
1552227021,29sleep215600-21fschecks_time08:50:482
1261227011,45sleep312627-21latency_hist10:25:293
1078027016,28sleep20-21swapper/208:35:422
449826915,39sleep14575-21mailstats11:41:011
449826915,39sleep14575-21mailstats11:41:011
2508226915,40sleep225093-21ls11:05:292
1418026916,33sleep30-21swapper/310:30:293
127426926,26sleep20-21swapper/208:05:472
120752690,7sleep10-21swapper/110:20:581
586826817,24sleep30-21swapper/308:20:293
2200026823,24sleep20-21swapper/209:10:472
1673626816,35sleep316888-21/usr/sbin/munin10:35:583
1082926816,37sleep210861-21latency12:01:532
2426926715,24sleep20-21swapper/209:15:592
1315326722,29sleep313192-21ntpq07:01:003
3218226616,20sleep30-21swapper/308:00:473
2353426620,27sleep30-21swapper/311:00:293
693826518,25sleep20-21swapper/211:50:302
318772658,37sleep10-21swapper/109:40:581
318772658,37sleep10-21swapper/109:40:581
3158526519,22sleep20-21swapper/209:40:502
3158526519,22sleep20-21swapper/209:40:502
1909426516,30sleep319089-21apt-get10:45:433
1592226517,25sleep30-21swapper/308:50:573
892326414,20sleep30-21swapper/310:10:593
892326414,20sleep30-21swapper/310:10:593
666526421,26sleep16704-21df10:05:441
666526421,26sleep16704-21df10:05:441
3014426422,25sleep130188-21ntpq09:35:531
2712726424,21sleep20-21swapper/209:25:572
1211426414,38sleep112150-21fschecks_count12:05:471
1211426414,38sleep112150-21fschecks_count12:05:471
429626316,26sleep10-21swapper/109:55:591
429626316,26sleep10-21swapper/109:55:591
293472636,11sleep10-21swapper/109:34:071
2506926312,23sleep10-21swapper/109:20:391
1987426312,24sleep30-21swapper/307:20:583
1847526312,24sleep30-21swapper/312:25:483
175432637,41sleep30-21swapper/310:40:433
1042726318,22sleep10-21swapper/110:15:581
914326220,27sleep19177-21ps11:55:541
596826211,34sleep36013-21ntp_states11:45:553
3107626215,33sleep131147-21/usr/sbin/munin07:55:571
3030426217,19sleep30-21swapper/307:55:303
2167826224,22sleep10-21swapper/109:10:291
2070726210,37sleep120719-21munin-run07:25:281
1864126216,21sleep10-21swapper/109:00:441
1285426215,28sleep212928-21/usr/sbin/munin12:06:012
1285426215,28sleep212928-21/usr/sbin/munin12:06:012
788926120,17sleep30-21swapper/310:10:273
788926120,17sleep30-21swapper/310:10:273
742026115,24sleep30-21swapper/308:25:303
719926110,25sleep20-21swapper/210:05:582
719926110,25sleep20-21swapper/210:05:582
2104326111,37sleep321143-21cat09:06:523
1576826117,22sleep30-21swapper/310:35:303
155582617,13sleep10-21swapper/112:15:531
1115626111,26sleep30-21swapper/308:35:513
882026022,21sleep20-21swapper/210:10:582
882026022,21sleep20-21swapper/210:10:582
244726018,21sleep30-21swapper/308:10:283
232302607,12sleep20-21swapper/210:57:132
1626426014,25sleep20-21swapper/210:35:492
141912607,12sleep20-21swapper/212:10:572
918025917,20sleep20-21swapper/211:55:552
3258125913,19sleep10-21swapper/108:00:571
296225916,24sleep20-21swapper/208:10:392
2798725917,26sleep328145-21latency_hist09:30:303
2549625911,28sleep325545-21/usr/sbin/munin09:20:533
2453625916,26sleep224542-21ntp_states07:35:552
199982594,13sleep20-21swapper/212:30:472
170225913,31sleep31720-21wc09:50:283
170225913,31sleep31720-21wc09:50:283
1468725914,25sleep214781-21/usr/sbin/munin10:30:492
1000825920,23sleep30-21swapper/312:00:283
99712587,13sleep30-21swapper/310:15:483
346825814,29sleep33531-21sendmail_mailst08:11:053
319225817,17sleep10-21swapper/111:36:041
3188325816,22sleep30-21swapper/311:25:483
3188325816,22sleep30-21swapper/311:25:483
3186025817,25sleep331954-21sed08:00:313
2556725812,21sleep20-21swapper/209:20:562
2376325811,23sleep323797-21munin-run07:35:283
2250025816,26sleep222497-21apt-get07:30:382
148425812,28sleep11567-21threads11:31:031
1387825811,31sleep213948-21df08:45:442
1304125818,23sleep20-21swapper/210:25:482
1241025814,28sleep112417-21/usr/sbin/munin08:40:461
2540625717,18sleep10-21swapper/107:40:301
222325719,21sleep20-21swapper/209:50:522
2065525715,26sleep120634-21apt-get10:50:541
193462576,17sleep10-21swapper/112:28:081
1333725717,22sleep30-21swapper/312:10:303
1333725717,22sleep30-21swapper/312:10:303
843925615,21sleep10-21swapper/110:10:491
843925615,21sleep10-21swapper/110:10:491
411025618,18sleep30-21swapper/311:40:503
411025618,18sleep30-21swapper/311:40:503
1251425616,18sleep30-21swapper/308:40:483
718425517,23sleep37190-21diskstats11:50:473
718425517,23sleep37190-21diskstats11:50:473
463525514,15sleep20-21swapper/208:15:442
400825514,24sleep24002-21apt-get11:40:412
400825514,24sleep24002-21apt-get11:40:412
37425513,17sleep20-21swapper/209:45:392
37425513,17sleep20-21swapper/209:45:392
3264425514,22sleep232714-21wc08:00:582
2226525518,22sleep122322-21sed07:30:291
1441525514,25sleep114410-21apt-get10:30:391
1149225515,18sleep20-21swapper/210:20:472
585425413,23sleep15980-21users10:01:001
585425413,23sleep15980-21users10:01:001
555825412,24sleep15553-21apt-get11:45:411
2934525414,23sleep229389-21ntpq07:50:532
1751625412,26sleep217534-21apt10:40:382
1722425412,14sleep20-21swapper/208:55:492
143472546,12sleep10-21swapper/108:45:551
508025314,23sleep35076-21apt-get10:00:423
508025314,23sleep35076-21apt-get10:00:423
208462537,35sleep10-21swapper/112:32:321
67325212,23sleep1727-21sed09:45:501
67325212,23sleep1727-21sed09:45:501
2938025218,16sleep10-21swapper/111:16:011
2228625213,22sleep122292-21df_inode10:55:431
221525221,17sleep20-21swapper/211:35:282
2111725214,19sleep20-21swapper/210:51:072
1641125212,15sleep10-21swapper/110:35:521
1498125217,15sleep20-21swapper/212:15:292
3114825113,16sleep20-21swapper/207:55:582
2743925111,23sleep327408-21apt-get07:45:573
2102625113,23sleep321032-21df_inode07:25:433
3179125012,22sleep131788-21apt-get11:25:381
3179125012,22sleep131788-21apt-get11:25:381
3078625017,15sleep20-21swapper/211:20:592
1909325015,19sleep20-21swapper/210:45:422
1695825012,23sleep316992-21irqstats12:20:513
1695825012,23sleep316992-21irqstats12:20:513
1379925011,22sleep113844-21irqstats12:10:481
97662498,15sleep20-21swapper/210:15:432
2595724911,16sleep10-21swapper/107:40:511
2595724911,16sleep10-21swapper/107:40:511
318672489,14sleep20-21swapper/211:25:432
318672489,14sleep20-21swapper/211:25:432
2850124813,20sleep328555-21memory09:30:503
167682488,24sleep216795-21df12:20:462
167682488,24sleep216795-21df12:20:462
1786524715,15sleep20-21swapper/207:15:442
2562624610,20sleep325622-21apt-get07:40:403
2562624610,20sleep325622-21apt-get07:40:403
2005624613,16sleep30-21swapper/312:30:483
1433124510,21sleep214333-21apt-get07:05:392
1909024410,14sleep10-21swapper/110:45:411
717724314,15sleep20-21swapper/211:50:462
717724314,15sleep20-21swapper/211:50:462
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional