You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-18 - 04:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon May 18, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
273962521817,5189sleep225585-21kworker/2:020:31:002
66582363136,3546sleep06123-21kworker/u8:121:09:300
66582363136,3546sleep06123-21kworker/u8:121:09:300
230552275214,2721sleep34817-21kworker/u8:020:19:303
776828656,810sleep06242-21kworker/0:219:31:070
776828656,810sleep06242-21kworker/0:219:31:070
24496257516,542sleep224556-21/usr/sbin/munin22:04:432
915245825,417sleep21064-21/usr/sbin/munin20:49:462
915245825,417sleep21064-21/usr/sbin/munin20:49:462
1619226869,159sleep00-21swapper/000:19:330
89422670,6sleep30-21swapper/322:34:223
27173222915,191sleep027318-21grep20:30:260
2566422257,202sleep025700-21fschecks_count22:09:330
1472221714,190sleep31483-21sort00:19:243
4839220766,82sleep00-21swapper/019:24:160
18406220311,175sleep018441-21latency20:04:380
32381220010,175sleep032427-21latency_hist20:49:150
1451521917,173sleep0363-21snmpd19:51:420
15697218965,38sleep0662-21cron21:39:130
15697218965,38sleep0662-21cron21:39:130
3237021757,154sleep00-21swapper/000:14:320
29508216910,16sleep00-21swapper/020:39:260
15120216512,136sleep215273-21awk21:34:472
15120216512,136sleep215273-21awk21:34:472
13999216274,66sleep00-21swapper/023:14:530
10677215441,101sleep010724-21ntpq23:04:400
611621539,127sleep06173-21gpgv21:09:130
611621539,127sleep06173-21gpgv21:09:130
27354215245,93sleep027388-40mandb23:59:140
27354215245,93sleep027388-40mandb23:59:140
24872215287,16sleep024953-21latency20:24:360
21304214911,76sleep00-21swapper/021:54:480
561221477,12sleep30-21swapper/319:25:333
2286214638,90sleep00-21swapper/019:14:440
31896214520,105sleep231935-21cut22:29:342
12667214215,100sleep00-21swapper/021:29:160
6695213914,112sleep06771-21switchtime00:34:430
6695213914,112sleep06771-21switchtime00:34:430
24517213974,14sleep00-21swapper/023:49:290
11200213811,105sleep011289-21apt-config19:44:150
11200213811,105sleep011289-21apt-config19:44:150
8212213714,103sleep38263-21cpuspeed21:14:333
525921376,45sleep00-21swapper/022:48:410
2709921379,69sleep00-21swapper/022:14:260
1184213751,69sleep01298-21sed19:14:150
11935213323,33sleep012004-21sort19:44:390
12413213211,66sleep012503-21xargs23:09:450
32350213154,60sleep00-21swapper/022:29:440
1855221299,110sleep0363-21snmpd21:45:470
1855221299,110sleep0363-21snmpd21:45:470
7431212718,87sleep37572-21unixbench_multi19:29:453
7431212718,87sleep37572-21unixbench_multi19:29:453
4441212780,34sleep24484-21ntp_kernel_pll_22:44:402
16579212312,95sleep016631-21sort19:59:290
25259212121,80sleep125411-21sed23:49:471
20500212139,63sleep020648-21gpgv21:54:150
15273211945,16sleep00-21swapper/019:54:370
10519211831,69sleep010663-21unixbench_singl21:19:470
1481211748,57sleep01528-21cat20:54:160
1481211748,57sleep01528-21cat20:54:160
22668211619,79sleep122516-21latency_hist20:19:161
2660211527,76sleep00-21swapper/020:55:080
2660211527,76sleep00-21swapper/020:55:080
27939211414,81sleep127984-21/usr/sbin/munin20:34:301
13226211418,68sleep313347-21grep23:14:163
17537211115,81sleep317545-21acpi21:44:253
17537211115,81sleep317545-21acpi21:44:253
20227211016,22sleep00-21swapper/020:09:420
5858210919,76sleep35907-21unin-run00:34:133
5858210919,76sleep35907-21unin-run00:34:133
19603210945,17sleep019601-21munin-run23:34:150
13580210918,73sleep313096-21apt-get23:14:243
31898210840,15sleep00-21swapper/000:12:060
17272210825,67sleep317274-21cron21:44:133
17272210825,67sleep317274-21cron21:44:133
17269210710,86sleep0363-21snmpd21:44:100
17269210710,86sleep0363-21snmpd21:44:100
16103210722,70sleep216123-21fschecks_count21:39:362
16103210722,70sleep216123-21fschecks_count21:39:362
15777210713,79sleep315908-21cat21:39:163
15777210713,79sleep315908-21cat21:39:163
8610210636,50sleep00-21swapper/021:14:430
21506210642,47sleep00-21swapper/023:39:320
29818210527,37sleep00-21swapper/022:22:170
27055210514,79sleep327168-21cat20:30:253
30515210417,71sleep130533-21idleruntime-cro00:09:121
30515210417,71sleep130533-21idleruntime-cro00:09:121
10365210420,67sleep210445-21grep19:39:452
10365210420,67sleep210445-21grep19:39:452
236821036,84sleep00-21swapper/022:39:170
5802210215,74sleep25846-21if_eth022:49:362
2508121026,10sleep00-21swapper/022:06:340
21100210224,63sleep121105-21modprobe23:39:131
8989210118,63sleep28693-21/usr/sbin/munin22:59:372
22578210112,71sleep022595-21munin-run20:19:150
10741210115,71sleep310805-21ps23:04:433
16746210017,64sleep316800-21cpuspeed23:24:293
889129920,63sleep29044-21awk21:14:492
2436429913,71sleep024487-21latency_hist20:24:160
1864129915,66sleep318732-21cut20:04:453
2340429833,14sleep00-21swapper/023:44:410
1034529816,62sleep2363-21snmpd21:19:452
2029529730,15sleep00-21swapper/023:34:400
2002429722,61sleep220060-21idleruntime23:34:332
2919829611,68sleep025782-1kworker/0:2H20:39:140
1250529616,63sleep112621-21cat23:09:461
431029531,14sleep00-21swapper/000:29:140
375129520,60sleep13800-21unin-run22:44:141
2901529562,16sleep00-21swapper/000:04:150
2901529562,16sleep00-21swapper/000:04:150
981729212,23sleep00-21swapper/023:02:060
350429218,32sleep30-21swapper/320:59:363
350429218,32sleep30-21swapper/320:59:363
1725029210,34sleep017322-21mailstats23:24:420
92562916,39sleep00-21swapper/019:35:230
92562916,39sleep00-21swapper/019:35:230
536729122,18sleep30-21swapper/322:49:173
66442906,39sleep00-21swapper/022:51:320
163312906,41sleep00-21swapper/023:22:360
82112895,39sleep00-21swapper/022:56:490
380128911,55sleep23766-21latency_hist22:44:152
3200228936,31sleep332031-21/usr/sbin/munin19:09:143
3200228936,31sleep332031-21/usr/sbin/munin19:09:143
2951628915,38sleep30-21swapper/320:39:293
2732328918,45sleep327342-21apt23:59:133
2732328918,45sleep327342-21apt23:59:133
996628824,42sleep110054-21tune2fs19:39:361
996628824,42sleep110054-21tune2fs19:39:361
1973528820,53sleep319863-21ls21:49:463
19622885,37sleep00-21swapper/022:36:300
626528718,50sleep26387-21latency_hist21:09:152
626528718,50sleep26387-21latency_hist21:09:152
3060828710,63sleep230610-21sshd20:42:062
3232928515,56sleep232345-21grep19:09:292
2374128519,39sleep123766-21munin-run22:04:131
2247528520,45sleep222449-21/usr/sbin/munin21:59:302
349528419,49sleep23505-21/usr/sbin/munin19:19:342
2965128420,46sleep329227-21/usr/sbin/munin00:04:383
2965128420,46sleep329227-21/usr/sbin/munin00:04:383
2793328413,43sleep10-21swapper/122:14:471
3242328321,38sleep132506-21wc22:29:451
2765028313,54sleep127697-21systemd-run20:34:141
2741028322,44sleep127388-40mandb23:59:151
2741028322,44sleep127388-40mandb23:59:151
1214628316,50sleep212208-21/usr/sbin/munin19:44:452
55702827,25sleep0363-21snmpd00:30:310
55702827,25sleep0363-21snmpd00:30:310
1318128149,15sleep00-21swapper/021:29:360
1223328122,43sleep312242-21vmstat21:24:473
730028016,44sleep37353-21sort22:54:343
523328010,17sleep00-21swapper/019:24:530
390028010,56sleep00-21swapper/020:59:450
390028010,56sleep00-21swapper/020:59:450
174582790,5sleep10-21swapper/119:59:451
350627811,52sleep33609-21fschecks_time19:19:353
217427824,27sleep10-21swapper/122:39:131
1475827825,27sleep30-21swapper/319:54:153
614827719,36sleep20-21swapper/200:34:292
614827719,36sleep20-21swapper/200:34:292
2150027712,17sleep30-21swapper/323:39:313
2136927716,40sleep321404-21grep21:54:513
1260027732,25sleep10-21swapper/121:29:131
3138227618,23sleep10-21swapper/100:09:451
3101227627,28sleep10-21swapper/119:04:361
3101227627,28sleep10-21swapper/119:04:361
2399027616,46sleep2225-21systemd-journal20:20:062
1732427618,41sleep117400-21sed21:44:151
1732427618,41sleep117400-21sed21:44:151
436627516,27sleep20-21swapper/200:29:162
1208027524,28sleep212229-21grep21:24:462
1153327521,33sleep111552-21sh23:09:131
722427412,46sleep17225-21diskstats22:54:321
2839327424,32sleep328408-21modprobe22:19:133
2688927422,27sleep30-21swapper/322:14:163
2427727426,25sleep20-21swapper/223:49:152
1582827428,30sleep30-21swapper/323:19:453
747727313,45sleep37508-21latency_hist00:39:143
747727313,45sleep37508-21latency_hist00:39:143
2927827317,21sleep20-21swapper/220:39:162
2497227313,15sleep20-21swapper/220:24:382
1318227316,22sleep10-21swapper/119:49:171
315702727,19sleep031702-21cat22:29:150
791627125,28sleep37954-21kworker/u8:321:14:133
791627125,28sleep37954-21kworker/u8:321:14:133
590327119,37sleep26033-21sed19:25:402
3082827124,27sleep20-21swapper/219:04:332
3082827124,27sleep20-21swapper/219:04:332
1553527121,27sleep30-21swapper/319:54:443
1409927114,37sleep114159-21/usr/sbin/munin23:14:581
2528427023,20sleep10-21swapper/120:24:451
847226919,24sleep10-21swapper/121:14:391
3262526910,18sleep30-21swapper/320:49:303
3262526910,18sleep30-21swapper/320:49:303
2558026920,31sleep10-21swapper/122:09:301
159226920,30sleep21739-21grep22:34:422
52326823,32sleep3645-21sed00:14:483
3269726821,32sleep332741-21ntpq19:09:383
3126326811,41sleep231269-21ntp_states00:09:422
2892826810,45sleep128942-21apt-config00:04:081
2892826810,45sleep128942-21apt-config00:04:081
224732688,43sleep00-21swapper/021:59:280
1733226823,22sleep10-21swapper/123:24:431
1719026819,29sleep30-21swapper/319:59:443
963826719,24sleep10-21swapper/119:39:151
963826719,24sleep10-21swapper/119:39:151
280026718,32sleep12809-21unin-run00:24:141
2249726719,31sleep322576-21systemd-run20:19:143
1897326721,21sleep10-21swapper/123:29:441
1504826716,38sleep215074-21sh19:54:302
955826619,28sleep39601-21systemctl19:39:133
955826619,28sleep39601-21systemctl19:39:133
44122669,12sleep10-21swapper/122:44:391
3114926626,19sleep30-21swapper/320:44:343
2765126623,28sleep327692-21cpu23:59:293
2765126623,28sleep327692-21cpu23:59:293
2533926624,22sleep30-21swapper/322:09:163
2036826621,27sleep220393-21munin-run21:54:132
638826521,26sleep36135-21apt-get21:09:163
304242659,12sleep30-21swapper/322:24:343
296872657,44sleep00-21swapper/000:04:390
296872657,44sleep00-21swapper/000:04:390
181626520,24sleep20-21swapper/220:54:342
181626520,24sleep20-21swapper/220:54:342
69602648,39sleep10-21swapper/119:29:341
69602648,39sleep10-21swapper/119:29:341
617026413,14sleep30-21swapper/322:49:453
3219426410,19sleep20-21swapper/220:47:272
201882646,10sleep10-21swapper/121:51:541
721426318,31sleep17283-21expr21:09:471
721426318,31sleep17283-21expr21:09:471
70932637,14sleep10-21swapper/100:35:121
70932637,14sleep10-21swapper/100:35:121
683026318,23sleep20-21swapper/219:29:312
683026318,23sleep20-21swapper/219:29:312
646226315,21sleep10-21swapper/119:29:141
461626322,20sleep20-21swapper/221:04:152
461626322,20sleep20-21swapper/221:04:152
2879726315,19sleep30-21swapper/322:19:333
57852627,17sleep30-21swapper/321:05:013
35102629,39sleep33543-21proc_pri00:24:413
2119126211,21sleep20-21swapper/221:54:462
1890426218,20sleep10-21swapper/121:49:151
1890426218,20sleep10-21swapper/121:49:151
3095426117,26sleep031011-21irqstats19:04:350
3095426117,26sleep031011-21irqstats19:04:350
281026121,17sleep20-21swapper/200:24:152
2005226122,21sleep30-21swapper/320:09:373
1386126119,17sleep10-21swapper/119:50:091
114542617,10sleep20-21swapper/223:08:132
288832606,10sleep20-21swapper/200:03:482
288832606,10sleep20-21swapper/200:03:482
2615926018,25sleep20-21swapper/223:54:322
2615926018,25sleep20-21swapper/223:54:322
375625917,26sleep13760-21ntpq20:59:421
375625917,26sleep13760-21ntpq20:59:421
249025914,28sleep32485-21apt-get22:39:283
2038925919,17sleep20-21swapper/220:09:442
1828225919,24sleep218346-21/usr/sbin/munin20:04:362
1575125923,19sleep20-21swapper/223:19:442
1163625916,26sleep311653-21apt19:44:263
908325814,27sleep39095-21/usr/sbin/munin22:59:383
66025822,21sleep30-21swapper/322:34:153
3155025820,17sleep10-21swapper/120:44:441
3052825816,25sleep130606-21sed22:24:361
2500325820,17sleep20-21swapper/223:49:442
2086625819,19sleep10-21swapper/121:54:241
2028925817,17sleep10-21swapper/120:09:431
1866125814,20sleep20-21swapper/223:29:372
1345225815,19sleep20-21swapper/219:49:302
722625723,19sleep20-21swapper/222:54:332
617625716,21sleep10-21swapper/121:09:141
617625716,21sleep10-21swapper/121:09:141
48225715,29sleep1600-21cat19:09:441
231025711,25sleep10-21swapper/119:14:451
2296625715,25sleep322963-21apt-get23:44:283
2150025718,18sleep20-21swapper/220:14:372
143725722,18sleep20-21swapper/219:14:172
1313325719,21sleep20-21swapper/223:14:152
979325615,25sleep39788-21apt-get21:19:303
856025619,19sleep20-21swapper/219:34:362
856025619,19sleep20-21swapper/219:34:362
853125616,15sleep10-21swapper/122:59:161
3005825613,25sleep230156-21users00:04:452
3005825613,25sleep230156-21users00:04:452
254712568,15sleep30-21swapper/323:50:213
217425610,31sleep12184-21ntp_states20:54:411
217425610,31sleep12184-21ntp_states20:54:411
1813625616,25sleep118129-21apt-get20:04:311
1701825615,24sleep217111-21memory23:24:362
3093825512,24sleep331017-21fw_packets00:09:353
2327025516,22sleep323375-21users21:59:473
2007425516,17sleep30-21swapper/323:34:343
1869125516,23sleep30-21swapper/323:29:383
150825516,19sleep30-21swapper/319:14:183
32125412,24sleep2361-21ntp_states00:14:432
2932225411,27sleep229443-21timerandwakeup22:19:452
2765925416,23sleep20-21swapper/222:14:442
2271925422,17sleep20-21swapper/223:44:142
1287425414,24sleep112886-21apt-get21:29:251
79025321,19sleep10-21swapper/120:49:451
79025321,19sleep10-21swapper/120:49:451
690325311,22sleep10-21swapper/122:54:151
2595925318,20sleep30-21swapper/320:29:153
2305325313,24sleep123050-21apt-get20:19:281
178425317,19sleep30-21swapper/320:54:313
178425317,19sleep30-21swapper/320:54:313
1333725314,17sleep30-21swapper/321:29:393
304725211,23sleep13042-21apt-get00:24:261
2870925214,23sleep128704-21apt-get22:19:271
2190425216,16sleep10-21swapper/123:39:411
1443925212,23sleep10-21swapper/121:34:301
1443925212,23sleep10-21swapper/121:34:301
1657625111,24sleep216572-21apt-get19:59:272
1030325114,21sleep310332-21ntp_states19:39:423
1030325114,21sleep310332-21ntp_states19:39:423
66252509,22sleep20-21swapper/221:09:242
66252509,22sleep20-21swapper/221:09:242
476925011,22sleep10-21swapper/100:29:351
476925011,22sleep10-21swapper/100:29:351
315182509,15sleep20-21swapper/222:29:132
1066425011,23sleep110697-21users21:19:481
2578824911,22sleep225845-21irqstats22:09:352
2793624811,22sleep327932-21apt-get20:34:283
2342424813,20sleep123430-21ntp_states23:44:421
1524024816,16sleep10-21swapper/123:19:311
141122484,31sleep20-21swapper/221:33:462
2403124711,14sleep10-21swapper/122:04:291
1626124710,21sleep116316-21munin-run19:59:141
1361524710,21sleep213683-21df23:14:312
46762469,15sleep20-21swapper/200:29:322
46762469,15sleep20-21swapper/200:29:322
305292464,30sleep10-21swapper/120:41:041
384524411,14sleep10-21swapper/119:19:411
323402429,19sleep132359-21apt-get00:14:251
175622389,14sleep20-21swapper/221:44:282
175622389,14sleep20-21swapper/221:44:282
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional