You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-06 - 21:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Jul 06, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
79652132769,13251sleep24876-21kworker/u8:011:04:552
79652132769,13251sleep24876-21kworker/u8:011:04:552
3182921036621,10300sleep127179-21kworker/1:108:54:421
3182921036621,10300sleep127179-21kworker/1:108:54:421
261442173114,1703sleep326126-21apt-get10:19:333
261442173114,1703sleep326126-21apt-get10:19:333
189252122317,1189sleep018916-21apt-get08:14:300
26143298217,943sleep24876-21kworker/u8:010:19:322
26143298217,943sleep24876-21kworker/u8:010:19:322
1636925737,532sleep116410-21ntpq11:29:381
16272249527,431sleep116302-21unin-run09:49:141
13468249310,466sleep032679-1kworker/0:0H09:39:440
408423886,370sleep14095-21fschecks_count12:34:321
21899235544,207sleep122044-21unixbench_multi10:04:451
1944223317,286sleep119585-21latency_hist09:59:151
9718232937,174sleep10-21swapper/107:44:341
6640232316,291sleep36681-21ntpq07:34:393
26343231926,260sleep1364-21snmpd12:01:001
2958231710,293sleep12423-21/usr/sbin/munin12:29:481
3203023050,17sleep130275-21cyclictest07:14:161
15155227141,123sleep115302-21unixbench_singl11:24:461
412122649,240sleep14200-21systemctl09:09:441
2903722547,235sleep128642-21/usr/sbin/munin12:09:361
14649225442,197sleep114693-21ntp_kernel_pll_07:59:371
19747223310,204sleep119774-21fschecks_count09:59:321
16938223128,191sleep116975-21ntpq09:49:381
6597223078,138sleep16637-21processes10:59:411
1892622049,158sleep118997-21/usr/sbin/munin08:14:311
23307219517,162sleep123310-21cron08:29:131
31257218435,99sleep131319-21uname10:34:411
14530218111,12sleep00-21swapper/011:24:320
21826217912,149sleep021896-21find10:04:440
65021726,86sleep10-21swapper/112:24:141
21121216768,84sleep10-21swapper/111:44:401
18873216229,93sleep118915-21latency_hist11:39:151
24434216045,90sleep10-21swapper/111:54:461
2439621608,115sleep124421-21fschecks_count10:14:301
11417216026,95sleep111569-21latency_hist09:34:151
984321579,132sleep19909-21date09:29:151
30687214214,55sleep10-21swapper/112:14:411
3251214119,45sleep10-21swapper/109:08:271
26131213533,52sleep10-21swapper/110:19:271
26131213533,52sleep10-21swapper/110:19:271
27291213410,109sleep127317-21fschecks_time12:04:521
5371213133,79sleep15390-21apt-get09:14:241
1312213133,87sleep11360-21unin-run07:19:141
1312213133,87sleep11360-21unin-run07:19:141
423021306,112sleep00-21swapper/007:27:000
24285213015,96sleep024422-21grep08:29:450
2346921298,105sleep123481-21munin-run11:54:131
13607212829,83sleep113734-21grep11:19:451
14886212716,92sleep30-21swapper/307:59:433
295021267,96sleep00-21swapper/009:04:460
6963212522,16sleep10-21swapper/109:19:251
13170212412,74sleep113206-21grep07:54:441
30873212320,90sleep330880-21forks10:34:313
931421207,70sleep19360-21systemd07:44:141
6914212026,57sleep17035-21sed07:34:451
18215211616,26sleep00-21swapper/011:34:420
8723211521,53sleep18868-21unixbench_singl11:05:231
8723211521,53sleep18868-21unixbench_singl11:05:231
2384121158,72sleep1364-21snmpd10:11:511
32279211415,84sleep332291-21vmstat08:54:483
32279211415,84sleep332291-21vmstat08:54:483
18126211319,80sleep318213-21threads11:34:413
10112211311,83sleep10-21swapper/111:09:511
32711211215,74sleep2377-21grep08:59:152
32711211215,74sleep2377-21grep08:59:152
1362211120,73sleep31451-21latency_hist07:19:163
1362211120,73sleep31451-21latency_hist07:19:163
31802211019,77sleep00-21swapper/012:19:250
5220210927,18sleep15336-21ls10:54:451
15278210915,26sleep015362-21memory09:44:360
1411721086,92sleep00-21swapper/011:23:430
20880210710,72sleep10-21swapper/108:19:401
1249121078,27sleep10-21swapper/107:54:161
13535210532,33sleep10-21swapper/109:39:451
11717210426,45sleep111753-21fschecks_count09:34:321
97021036,23sleep10-21swapper/112:24:311
29978210216,73sleep029984-21latency_hist12:14:160
15500210216,66sleep214970-21/usr/sbin/munin09:44:422
13158210213,74sleep313174-21uname11:19:353
5162210127,44sleep25282-21latency_hist09:14:172
22176210114,25sleep222195-21apt-get11:49:222
1264121017,85sleep3308-21kworker/u8:209:35:593
399210014,65sleep0478-21/usr/sbin/munin10:39:410
31857210025,54sleep032029-21munin-node07:14:160
2268921007,78sleep00-21swapper/011:49:400
2935429916,69sleep229438-21perl10:29:332
973429814,17sleep20-21swapper/211:09:402
1582929816,39sleep10-21swapper/108:04:291
752629712,70sleep27689-21latency_hist11:04:152
309729614,65sleep23150-21fschecks_count10:49:372
1868029611,72sleep118735-21latency_hist08:14:151
41272947,75sleep10-21swapper/107:25:401
2744729436,39sleep027525-21latency_hist10:24:140
2744729436,39sleep027525-21latency_hist10:24:140
46932928,42sleep10-21swapper/107:29:281
154429116,58sleep21554-21/usr/sbin/munin10:44:322
1102429115,63sleep011120-21latency_hist11:14:140
1239629012,20sleep00-21swapper/011:17:130
30112897,46sleep13001-21apt-get10:49:321
127828921,49sleep01307-21munin-plugin-st07:19:130
127828921,49sleep01307-21munin-plugin-st07:19:130
970228829,39sleep39704-21apt-get11:09:253
824628813,60sleep08283-21idleruntime-cro09:24:130
2774528816,13sleep10-21swapper/110:24:321
2094528714,51sleep020633-21/usr/sbin/munin11:44:350
3189328517,47sleep331908-21cut12:19:323
3149128517,53sleep031544-21idleruntime-cro12:19:120
308582855,69sleep10-21swapper/108:51:181
308582855,69sleep10-21swapper/108:51:181
2842928512,59sleep30-21swapper/312:09:143
2649728527,32sleep30-21swapper/308:39:173
193852857,15sleep10-21swapper/111:39:361
1661728521,47sleep316734-21timerwakeupswit11:29:443
3114828417,47sleep331111-21latency_hist08:54:153
3114828417,47sleep331111-21latency_hist08:54:153
3103028417,49sleep231059-21/usr/sbin/munin10:34:342
2984628426,18sleep129958-21cat10:29:451
264712846,43sleep126480-21munin-run08:39:151
235952837,35sleep10-21swapper/108:29:251
229142830,8sleep30-21swapper/311:49:423
1000028313,55sleep210042-21ntp_kernel_pll_07:44:392
650328223,42sleep06521-21kernelversion07:34:360
2893328211,53sleep20-21swapper/207:04:292
2893328211,53sleep20-21swapper/207:04:292
199428224,17sleep10-21swapper/110:44:431
1642428221,43sleep316505-21awk08:04:443
1191628216,50sleep312034-21wc11:14:423
809528116,48sleep38181-21fw_packets11:05:053
809528116,48sleep38181-21fw_packets11:05:053
2110628110,58sleep30-21swapper/311:44:383
1298128112,52sleep013072-21apt-config09:39:150
1129628113,49sleep011362-21/usr/sbin/munin07:49:340
2762028027,24sleep20-21swapper/212:05:012
1932028013,30sleep30-21swapper/311:39:343
176427921,34sleep30-21swapper/309:04:133
1650727912,47sleep016636-21/usr/sbin/munin08:04:450
49952789,16sleep00-21swapper/010:54:400
142712780,6sleep30-21swapper/309:40:063
462727717,44sleep34636-21df_inode10:54:303
293792775,60sleep10-21swapper/108:47:181
146652779,48sleep30-21swapper/311:24:353
48827617,16sleep10-21swapper/108:59:261
3018527623,28sleep20-21swapper/208:49:382
3018527623,28sleep20-21swapper/208:49:382
220527612,43sleep22291-21ls07:19:432
2721227516,43sleep027298-21mailstats08:39:420
267982758,51sleep10-21swapper/108:39:311
3181927429,30sleep331835-21munin-node07:14:153
2331527410,47sleep223358-21unin-run08:29:142
2242527420,28sleep30-21swapper/308:24:393
2176227419,39sleep221798-21basename08:24:132
1973227427,27sleep30-21swapper/309:59:303
1654127411,16sleep30-21swapper/309:49:283
159827427,31sleep10-21swapper/107:19:261
936927311,28sleep30-21swapper/309:24:443
749527319,37sleep27575-21dpkg07:39:142
2927027311,50sleep0364-21snmpd08:45:520
3070227226,28sleep20-21swapper/207:09:462
302432725,10sleep30-21swapper/310:31:003
288792727,46sleep20-21swapper/208:44:452
2836627217,40sleep328459-21uniq08:44:343
2344527227,25sleep30-21swapper/310:09:433
2245927219,28sleep00-21swapper/008:24:400
2128927220,28sleep20-21swapper/210:04:292
2050827217,26sleep30-21swapper/308:19:303
1279027213,42sleep012878-21tune2fs07:54:350
9752715,53sleep20-21swapper/210:41:412
398527117,24sleep00-21swapper/009:09:390
331127119,31sleep31-21systemd09:09:133
2893527114,41sleep328950-21kernelversion12:09:343
2672227116,29sleep30-21swapper/308:39:293
182932707,11sleep00-21swapper/008:09:500
1575027019,37sleep215833-21wc11:29:152
1337927012,38sleep213440-21wc11:19:422
1153127016,22sleep20-21swapper/211:14:332
718426920,26sleep20-21swapper/209:19:342
285426922,20sleep00-21swapper/007:24:140
931026818,32sleep00-21swapper/007:44:130
2648126822,19sleep20-21swapper/208:39:162
1803926817,36sleep218125-21mailstats11:34:402
16042686,50sleep00-21swapper/009:02:080
8592676,28sleep10-21swapper/110:40:111
54332677,46sleep00-21swapper/009:14:480
338026713,35sleep33416-21ntpq10:49:423
264732677,50sleep30-21swapper/312:02:403
2002926714,25sleep20-21swapper/209:59:372
183126718,23sleep00-21swapper/010:44:380
613126620,33sleep06150-21grep10:59:280
3076326615,20sleep00-21swapper/010:34:240
2524526617,33sleep025348-21fschecks_time08:34:330
1911126620,20sleep00-21swapper/011:39:260
1469526615,35sleep214724-21ntp_states07:59:382
1240426616,23sleep30-21swapper/307:54:133
804626520,28sleep28042-21apt-get07:39:262
533826517,30sleep25457-21users10:54:462
3024126522,22sleep30-21swapper/307:09:143
3024126522,22sleep30-21swapper/307:09:143
2769126524,25sleep00-21swapper/010:24:310
3030126412,34sleep30-21swapper/312:14:313
299052647,46sleep20-21swapper/212:13:572
2603926418,24sleep20-21swapper/211:59:482
2426026415,34sleep324353-21grep11:54:443
238762647,24sleep20-21swapper/211:54:352
1630426418,20sleep30-21swapper/309:49:163
1058226419,20sleep00-21swapper/009:29:420
899426314,21sleep20-21swapper/209:24:392
79192636,42sleep00-21swapper/011:04:250
79192636,42sleep00-21swapper/011:04:250
540026312,26sleep30-21swapper/309:14:263
370626314,28sleep20-21swapper/212:34:132
360026319,27sleep23606-21ntp_states07:24:402
2019626317,31sleep00-21swapper/009:59:430
1360126320,28sleep213613-21vmstat07:54:492
121832635,28sleep10-21swapper/111:14:521
1015626319,20sleep30-21swapper/307:44:443
637026217,27sleep2225-21systemd-journal09:15:262
440026215,29sleep34420-21munin-run07:29:133
216026213,19sleep30-21swapper/312:29:133
1932626212,35sleep219363-21grep08:14:402
184592627,13sleep00-21swapper/009:54:380
147342628,13sleep00-21swapper/007:59:400
1326326214,19sleep20-21swapper/209:39:172
1114726212,25sleep20-21swapper/207:49:282
25382617,12sleep30-21swapper/312:29:333
181726111,20sleep20-21swapper/212:24:482
1781026116,26sleep217829-21ntp_states08:09:392
146126115,23sleep30-21swapper/310:44:293
489426018,25sleep30-21swapper/312:34:463
2809226017,25sleep228139-21ntp_offset10:24:412
2402326023,18sleep00-21swapper/010:14:130
1016826013,29sleep20-21swapper/209:29:322
69925916,27sleep2851-21sed07:14:472
69925916,27sleep2851-21sed07:14:472
3156925922,20sleep20-21swapper/212:19:152
2843125925,19sleep00-21swapper/012:09:150
491425814,20sleep30-21swapper/309:14:143
2573725819,18sleep00-21swapper/011:59:420
2516025816,26sleep325213-21users10:14:443
2493925818,18sleep30-21swapper/308:34:163
2420025820,22sleep024207-21ntp_states11:54:410
1855025810,31sleep318570-21/usr/sbin/munin09:54:413
594825714,28sleep25980-21basename07:34:132
3195325717,19sleep00-21swapper/008:54:450
3195325717,19sleep00-21swapper/008:54:450
1244025715,25sleep212532-21users09:34:452
474025615,18sleep20-21swapper/212:34:452
2442225615,19sleep20-21swapper/210:14:312
206025618,23sleep32061-21ntp_states07:19:403
1936425613,28sleep319380-21ntp_states08:14:413
1870325616,20sleep20-21swapper/209:54:442
1278325614,28sleep312788-21diskstats07:54:343
3055225517,22sleep030633-21latency_hist10:34:160
2981525515,21sleep00-21swapper/008:49:280
2981525515,21sleep00-21swapper/008:49:280
2768925515,24sleep327682-21apt-get10:24:303
244725514,25sleep02442-21apt-get12:29:260
101325512,24sleep01110-21perl12:24:340
936225417,22sleep30-21swapper/307:44:163
907725410,22sleep00-21swapper/009:24:410
724125414,24sleep37259-21irqstats09:19:353
50672545,38sleep00-21swapper/012:36:450
1923325315,23sleep219319-21perl11:39:332
1732825311,26sleep017443-21/usr/sbin/munin09:49:450
600025215,23sleep06078-21sed07:34:150
48925210,23sleep2483-21apt-get08:59:272
2916825215,22sleep029202-21irqstats07:04:360
2916825215,22sleep029202-21irqstats07:04:360
2083325213,23sleep020879-21memory08:19:390
1673725220,17sleep00-21swapper/011:29:450
1134025216,23sleep31-21systemd09:34:133
598125114,22sleep35998-21munin-run07:34:143
3180525111,23sleep231799-21apt-get12:19:272
2717725112,22sleep227185-21ntp_states08:39:402
2430925113,23sleep024304-21apt-get10:14:250
2073325113,21sleep220739-21df_inode11:44:292
2938625011,17sleep10-21swapper/107:04:411
2938625011,17sleep10-21swapper/107:04:411
2407525017,18sleep20-21swapper/210:14:152
2129925014,21sleep321302-21diskstats10:04:303
1711825014,16sleep20-21swapper/209:49:432
73152496,34sleep30-21swapper/311:03:113
431824912,17sleep00-21swapper/010:54:150
30302496,31sleep20-21swapper/209:05:332
1633224917,15sleep20-21swapper/208:04:432
2490324612,21sleep224881-21latency_hist08:34:152
968924512,15sleep00-21swapper/007:44:330
3273124411,18sleep30-21swapper/310:39:363
173742417,16sleep30-21swapper/308:09:283
1606924110,15sleep20-21swapper/211:29:312
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional