You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-28 - 03:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sat Mar 28, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
103342101237,10084sleep38972-21kworker/3:120:43:093
303362456921,4536sleep222109-21kworker/2:020:06:092
665626766,657sleep36700-21ntpq20:32:473
164226317,590sleep31678-21ntpq23:42:463
26721262518,592sleep126792-21true21:37:231
20306241910,400sleep1191-21jbd2/mmcblk2p2-21:17:271
13193235112,321sleep313323-21unixbench_multi20:52:543
2344823350,6sleep00-21swapper/019:42:520
5624232123,273sleep15666-40ogrotate23:57:211
3563231914,207sleep33706-21head23:47:553
31167229227,215sleep331177-21ntp_states20:07:483
9121226653,155sleep30-21swapper/322:22:513
4814226611,239sleep24825-21ntp_states23:52:502
9169225477,68sleep39293-21latency_hist20:42:243
12716225221,212sleep012689-21latency_hist19:12:240
20800222842,130sleep30-21swapper/323:02:243
20800222842,130sleep30-21swapper/323:02:243
30974220811,180sleep331081-21sort23:32:523
30974220811,180sleep331081-21sort23:32:523
1302922058,181sleep312969-21latency_hist22:37:253
891220310,100sleep30-21swapper/323:41:083
566722010,5sleep30-21swapper/322:12:413
13240220015,169sleep313311-21cpuspeed19:12:313
11046220061,61sleep30-21swapper/320:47:413
11904219821,117sleep311982-21mailstats19:07:513
5239219410,124sleep35390-21latency_hist22:12:253
26004219410,47sleep30-21swapper/321:33:253
21597219136,79sleep321628-21munin-plugin-st21:22:223
25400218910,162sleep325478-21systemd23:17:223
2913321887,47sleep311199-21cyclictest21:43:063
15220218521,150sleep015284-21cat19:17:450
16937217728,136sleep317015-21fschecks_time00:32:413
24141217316,84sleep124161-21/usr/sbin/munin21:27:531
17173217071,56sleep30-21swapper/319:22:533
32242216924,59sleep30-21swapper/320:12:323
2472021686,145sleep00-21swapper/019:47:500
16562216112,23sleep016610-21/usr/sbin/munin00:32:220
16562216112,23sleep016610-21/usr/sbin/munin00:32:220
3625215547,60sleep30-21swapper/320:23:103
25967215212,124sleep226002-21grep21:33:242
19236214943,93sleep31-21systemd22:57:233
1013321467,122sleep30-21swapper/322:27:363
2580214520,109sleep12618-21date23:47:241
23186214335,92sleep323213-21munin-run21:27:233
14023214321,98sleep014087-21cat00:22:450
14023214321,98sleep014087-21cat00:22:450
17888214024,92sleep118019-21grep21:07:551
4129213660,26sleep00-21swapper/023:52:240
8761213512,62sleep38764-21cron00:07:213
350621347,11sleep30-21swapper/322:05:553
2843621328,73sleep313441-21snmpd23:26:143
6012212560,23sleep31-21systemd20:32:233
1561621258,101sleep315617-21/usr/sbin/munin21:02:313
3123521235,106sleep00-21swapper/021:50:220
26729212321,85sleep026859-21unixbench_multi19:52:590
17713212368,41sleep317804-21cut21:07:523
13348212323,81sleep113410-21fschecks_time22:37:391
2137221217,99sleep311199-21cyclictest19:37:433
15853211914,90sleep215927-21cat00:27:512
15853211914,90sleep215927-21cat00:27:512
6686211815,77sleep16829-21sed23:57:541
19948211714,85sleep019983-21proc_pri22:57:530
19948211714,85sleep019983-21proc_pri22:57:530
24873211320,40sleep30-21swapper/319:47:533
1498211319,71sleep11596-21fschecks_time20:17:431
29032211218,17sleep30-21swapper/323:27:443
28689211121,70sleep128743-21fschecks_count21:42:561
1082211120,73sleep01111-21sh20:17:220
16197211012,79sleep116210-21vmstat00:27:541
16197211012,79sleep116210-21vmstat00:27:541
1333521108,63sleep313441-21snmpd22:37:373
2499621099,49sleep325105-21apt-config21:32:243
28937210823,62sleep129065-21latency_hist20:02:251
137521079,86sleep31388-21acpi20:17:333
27704210615,70sleep227795-21fschecks_time19:57:412
31980210517,74sleep132014-21idleruntime-cro20:12:221
1344221045,51sleep30-21swapper/300:22:073
2633210313,74sleep02726-21memory22:02:450
1728721037,14sleep00-21swapper/021:07:410
10312210213,71sleep310331-21modprobe00:12:223
4925210110,76sleep14961-21idleruntime20:27:451
31678210119,65sleep131697-21grep21:52:371
212021019,31sleep30-21swapper/322:02:243
1530121018,38sleep30-21swapper/300:27:363
1530121018,38sleep30-21swapper/300:27:363
10287210117,63sleep09797-21/usr/sbin/munin19:02:520
2148521007,43sleep321496-21ntpq23:02:493
2148521007,43sleep321496-21ntpq23:02:493
19675210010,77sleep219696-21fschecks_count19:32:412
14642210018,67sleep014721-21switchtime20:57:520
317829920,58sleep03175-21apt-get20:22:350
436329743,18sleep34358-21apt-get23:52:353
427629712,70sleep14310-21ntp_kernel_err22:07:471
315429719,56sleep22671-21apt-get20:22:332
1446529636,16sleep30-21swapper/320:57:473
2631529512,61sleep30-21swapper/319:52:503
161802957,13sleep00-21swapper/021:02:530
1323629511,17sleep00-21swapper/022:37:300
88629420,58sleep1961-21fschecks_time21:57:421
3139129425,55sleep331438-21systemd-run21:52:233
1050229419,20sleep10-21swapper/122:27:461
3035029338,43sleep30-21swapper/320:06:183
2888429338,19sleep328905-21munin-run20:02:233
1345029326,51sleep113483-21grep19:12:421
1068629319,54sleep010699-21fschecks_count00:12:390
1384029214,57sleep113901-21fschecks_time00:22:411
1384029214,57sleep113901-21fschecks_time00:22:411
868329129,38sleep18765-21fschecks_time22:22:411
317502916,31sleep30-21swapper/321:52:393
118492916,39sleep30-21swapper/300:16:583
208492906,31sleep30-21swapper/319:37:233
760428917,57sleep17681-21cat22:17:571
290728915,59sleep22959-21fschecks_count23:47:412
2778628910,61sleep127862-21/usr/sbin/munin23:22:491
3236728822,51sleep132471-21fschecks_time20:12:421
2890328622,44sleep128991-21fschecks_time23:27:421
2010228618,46sleep220227-21grep21:17:242
1998928614,57sleep020031-21awk19:32:480
2724128432,39sleep327296-21interrupts21:37:413
3171928320,48sleep231823-21sed23:37:242
3171928320,48sleep231823-21sed23:37:242
3162428323,37sleep031650-21munin-run23:37:220
3162428323,37sleep031650-21munin-run23:37:220
309528311,55sleep13241-21awk22:02:541
2740228316,49sleep20-21swapper/221:37:442
239182835,11sleep30-21swapper/319:46:443
68732826,18sleep30-21swapper/323:58:043
599728222,46sleep26010-21idleruntime-cro20:32:222
192652828,23sleep30-21swapper/321:13:053
2258428119,29sleep10-21swapper/123:07:351
2258428119,29sleep10-21swapper/123:07:351
136028118,45sleep21417-21grep23:42:412
1291428114,51sleep112958-21ntp_offset20:52:481
3247228015,16sleep20-21swapper/220:12:432
2480428014,45sleep224872-21/usr/sbin/munin19:47:522
592727913,52sleep05928-21munin-node23:57:330
3228627915,41sleep132349-21cut23:37:551
2608227916,43sleep225692-21/usr/sbin/munin23:17:452
179827920,36sleep21838-21grep20:17:482
1703827912,47sleep016337-21/usr/sbin/munin22:47:560
2551827819,43sleep11-21systemd19:52:231
151692788,15sleep30-21swapper/319:17:443
2855527718,45sleep228560-21sort21:42:322
2712027723,23sleep10-21swapper/121:37:391
2116727718,41sleep221216-21cpuspeed19:37:382
1830427723,28sleep00-21swapper/022:52:480
1606227719,10sleep30-21swapper/322:47:143
1107927732,26sleep311142-21systemctl19:07:233
549927613,26sleep00-21swapper/020:27:560
436527620,27sleep00-21swapper/023:52:360
2232327611,49sleep022339-21ntp_states21:22:480
185782768,55sleep318622-21mailstats19:27:543
163622767,14sleep30-21swapper/322:47:383
3246927520,41sleep032511-21sort23:37:580
1625727518,35sleep20-21swapper/219:22:242
1465727520,43sleep30-21swapper/300:22:563
1465727520,43sleep30-21swapper/300:22:563
14212750,8sleep00-21swapper/021:57:520
952727413,41sleep29585-21/usr/sbin/munin00:07:482
3100727413,28sleep00-21swapper/020:07:450
249152748,50sleep024917-21sshd21:32:080
2255527420,11sleep30-21swapper/323:07:293
2255527420,11sleep30-21swapper/323:07:293
2188627416,35sleep221940-21sort21:22:372
1904427416,27sleep10-21swapper/121:12:591
1535527417,42sleep115363-21modprobe21:02:211
2571427310,48sleep30-21swapper/323:17:343
1507327323,32sleep10-21swapper/122:42:441
1195927314,44sleep212038-21cat22:32:432
1813227217,42sleep118209-21fschecks_time19:27:441
1279327216,28sleep10-21swapper/119:12:251
122072726,13sleep20-21swapper/219:07:542
1066827217,37sleep010682-21apt20:47:220
1022027225,28sleep10-21swapper/119:02:491
799327115,11sleep30-21swapper/322:18:273
582127116,25sleep20-21swapper/222:12:452
1767027114,13sleep20-21swapper/221:07:512
1745527118,37sleep217532-21/usr/sbin/munin00:32:522
2252827015,31sleep10-21swapper/121:22:531
1836227017,37sleep218368-21ntp_states22:52:502
1295127023,32sleep013010-21grep22:37:220
76822690,7sleep20-21swapper/222:17:572
2470626912,38sleep124801-21switchtime23:12:521
1693926925,26sleep20-21swapper/219:22:482
1377126920,32sleep213799-21munin-run20:57:212
2125026816,22sleep10-21swapper/119:37:411
1521226819,32sleep015254-21/usr/sbin/munin22:42:470
65892676,11sleep00-21swapper/022:15:370
2239026721,29sleep222543-21latency_hist23:07:242
2171926720,28sleep021862-21grep23:02:540
2171926720,28sleep021862-21grep23:02:540
96326612,40sleep21023-21basename21:57:432
906226621,27sleep20-21swapper/222:22:502
881626617,22sleep20-21swapper/200:07:242
3054626621,24sleep10-21swapper/123:32:421
3054626621,24sleep10-21swapper/123:32:421
325682658,11sleep20-21swapper/223:38:002
2651626515,27sleep10-21swapper/119:52:561
2150526517,36sleep121543-21processes23:02:511
2150526517,36sleep121543-21processes23:02:511
1988226512,37sleep119919-21sed22:57:501
1988226512,37sleep119919-21sed22:57:501
1200226520,20sleep10-21swapper/100:17:241
3079126414,28sleep20-21swapper/220:07:392
3037926417,32sleep130402-21irqstats21:47:441
212732647,45sleep00-21swapper/021:18:110
120152647,12sleep00-21swapper/020:49:400
2580226321,26sleep10-21swapper/123:17:401
1625126310,26sleep10-21swapper/121:02:541
1312826314,34sleep013129-21sshd00:18:080
1059926314,27sleep20-21swapper/200:12:332
2033826218,26sleep220405-21df21:17:372
1701726212,28sleep00-21swapper/000:32:420
611326114,31sleep26174-21sort23:57:432
2811626110,39sleep013441-21snmpd21:40:040
2628026118,18sleep00-21swapper/023:17:520
2402026122,19sleep20-21swapper/219:47:232
154112615,12sleep10-21swapper/119:17:501
1290826112,34sleep113026-21unixbench_multi00:17:531
366226018,20sleep20-21swapper/222:07:242
1779826016,30sleep117824-21cat19:27:231
655725920,19sleep10-21swapper/120:32:451
1934725916,19sleep00-21swapper/021:13:060
1288225920,18sleep00-21swapper/020:52:470
919625812,29sleep00-21swapper/022:22:520
811225813,29sleep08204-21cat00:02:540
748125811,35sleep0361-21lldpd20:36:320
309025821,23sleep10-21swapper/123:47:441
2752625811,20sleep20-21swapper/223:22:422
264592585,10sleep00-21swapper/021:34:020
2315925819,25sleep223246-21switchtime23:07:522
2315925819,25sleep223246-21switchtime23:07:522
1457525818,25sleep214639-21latency_hist22:42:242
1363125816,28sleep013648-21uname19:12:450
103402585,11sleep20-21swapper/220:43:122
981925715,25sleep29815-21apt-get19:02:382
55725718,18sleep00-21swapper/021:57:240
2440525716,24sleep224497-21memory23:12:452
2419725711,31sleep024208-21df_inode23:12:390
2256225715,24sleep022581-21apt-get23:07:340
2256225715,24sleep022581-21apt-get23:07:340
1648525714,24sleep116569-21perl22:47:441
101462575,10sleep10-21swapper/100:10:141
3058625614,25sleep230592-21ntp_states21:47:492
1256525619,22sleep20-21swapper/200:17:462
1227425612,28sleep212408-21cut20:52:252
846225516,18sleep10-21swapper/120:37:501
450225511,16sleep10-21swapper/120:27:241
3254625514,24sleep232579-21users21:52:542
2927025511,18sleep00-21swapper/023:27:490
2367625523,17sleep20-21swapper/221:27:382
2308425515,24sleep123125-21/usr/sbin/munin19:42:461
158162556,10sleep20-21swapper/222:44:062
783325416,20sleep20-21swapper/220:37:312
2925925416,23sleep10-21swapper/120:02:411
2918525414,20sleep20-21swapper/223:27:472
2110925415,24sleep221115-21df_inode23:02:392
185725415,22sleep01863-21ntp_states20:17:500
2848725312,23sleep028520-21users19:57:550
1956125313,24sleep219570-21diskstats22:57:432
1956125313,24sleep219570-21diskstats22:57:432
1676125312,21sleep20-21swapper/222:47:502
1506125313,21sleep20-21swapper/200:27:232
1506125313,21sleep20-21swapper/200:27:232
1013225316,23sleep210129-21apt-get22:27:352
545425212,23sleep15447-21apt-get22:12:351
483625215,19sleep20-21swapper/222:08:082
1885325215,20sleep218945-21users19:27:572
1685025211,24sleep116845-21apt-get00:32:351
1536625214,22sleep2281-21dbus-daemon21:02:222
134625213,24sleep01354-21df_inode23:42:390
1336625212,23sleep213429-21cpuspeed19:12:392
689325113,23sleep26980-21sort20:32:542
2887025112,17sleep00-21swapper/021:42:590
2772725114,18sleep00-21swapper/023:22:480
728725013,22sleep07316-21irqstats22:17:470
2093825015,18sleep10-21swapper/119:37:251
1804625015,18sleep00-21swapper/019:27:390
1686125015,21sleep016469-21/usr/sbin/munin19:22:460
1192125014,19sleep111958-21irqstats22:32:421
110332508,17sleep10-21swapper/120:47:391
932024915,16sleep00-21swapper/000:07:420
2656224912,16sleep20-21swapper/219:52:572
2174024910,23sleep021836-21sed19:37:520
1695724916,19sleep20-21swapper/221:07:232
1798724810,16sleep10-21swapper/122:52:411
986624614,17sleep10-21swapper/120:42:511
399424614,17sleep00-21swapper/022:07:420
2917324610,19sleep00-21swapper/020:02:360
792924410,20sleep07938-21df_inode20:37:370
1013524111,17sleep00-21swapper/022:27:370
78082406,17sleep10-21swapper/100:02:471
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional