You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-26 - 00:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Dec 25, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10047299627,935sleep210088-21ntpq09:54:412
8201268613,656sleep28343-21grep08:04:482
8201268613,656sleep28343-21grep08:04:482
2514726357,571sleep225175-21unin-run10:44:172
2514726357,571sleep225175-21unin-run10:44:172
3225325175,481sleep223215-1kworker/2:1H07:39:522
3225325175,481sleep223215-1kworker/2:1H07:39:522
2416824148,390sleep224217-21unin-run12:24:162
2416824148,390sleep224217-21unin-run12:24:162
2634823628,339sleep226396-21if_eth009:04:452
60572359134,200sleep26119-21fschecks_time07:59:392
2945233813,307sleep22964-21fschecks_count07:49:372
19870233114,296sleep219910-21fschecks_count12:09:342
19870233114,296sleep219910-21fschecks_count12:09:342
739522787,237sleep27399-21grep11:29:322
622224531,31sleep2763-21unixbench_singl09:24:482
31932224538,184sleep232116-21apt-key07:39:192
9489224471,58sleep20-21swapper/211:34:452
24460223729,22sleep20-21swapper/212:24:332
20843223453,152sleep20-21swapper/210:29:392
25603222868,55sleep20-21swapper/207:19:202
18743222870,25sleep20-21swapper/208:39:412
18748220746,110sleep218751-21munin-run10:24:172
14395220329,15sleep20-21swapper/208:24:472
388321998,176sleep23184-21/usr/sbin/munin09:35:272
6566218610,80sleep20-21swapper/209:44:312
43932184130,14sleep20-21swapper/207:54:302
43932184130,14sleep20-21swapper/207:54:302
11391218210,42sleep20-21swapper/211:39:462
27735218028,117sleep227779-21ntpq07:24:412
21414218071,33sleep221481-21fschecks_time12:14:352
23554217511,146sleep223592-21modprobe10:39:162
29961217216,135sleep330104-21grep09:14:483
16787217217,132sleep216871-21fschecks_time11:59:372
4651216516,18sleep30-21swapper/309:39:193
439521658,43sleep20-21swapper/211:19:352
1531721638,78sleep20-21swapper/211:54:362
23240215915,129sleep123274-21irqstats08:54:391
23240215915,129sleep123274-21irqstats08:54:391
16002215116,119sleep215823-21/usr/sbin/munin10:14:362
16002215116,119sleep215823-21/usr/sbin/munin10:14:362
26726215034,22sleep20-21swapper/212:29:502
25839215027,86sleep225889-21unin-run09:04:172
25839215027,86sleep225889-21unin-run09:04:172
1651321489,48sleep20-21swapper/208:34:162
501221477,116sleep00-21swapper/009:39:280
21530214546,58sleep20-21swapper/208:49:352
2890214412,115sleep03099-21modprobe09:34:190
11326214412,116sleep211470-21grep08:14:502
2470721426,12sleep10-21swapper/110:39:511
2470721426,12sleep10-21swapper/110:39:511
1254521429,44sleep20-21swapper/210:04:182
1254521429,44sleep20-21swapper/210:04:182
843321406,11sleep10-21swapper/108:04:511
843321406,11sleep10-21swapper/108:04:511
15558213611,110sleep20-21swapper/210:14:162
1386213615,106sleep01392-21df_inode07:44:340
1386213615,106sleep01392-21df_inode07:44:340
32215213320,92sleep331685-21apt-get07:39:293
32215213320,92sleep331685-21apt-get07:39:293
2833121339,108sleep20-21swapper/209:09:502
9063213211,104sleep29037-21/usr/sbin/munin08:09:322
2281321329,63sleep20-21swapper/207:09:472
13349213231,87sleep213513-21grep11:49:192
1830213020,90sleep31947-21timerwakeupswit11:09:463
14200212910,76sleep214329-21timerwakeupswit11:49:482
27167212816,43sleep20-21swapper/210:49:262
20451212710,47sleep20-21swapper/208:44:462
25193212512,93sleep124552-21/usr/sbin/munin08:59:471
25193212512,93sleep124552-21/usr/sbin/munin08:59:471
9639212424,82sleep09715-21switchtime08:09:470
24199212328,63sleep2369-21snmpd08:58:312
24199212328,63sleep2369-21snmpd08:58:312
6899212274,32sleep20-21swapper/211:26:332
20657212251,34sleep30-21swapper/307:04:273
20657212251,34sleep30-21swapper/307:04:273
18056212227,63sleep218082-21munin-run08:39:172
2686921210,8sleep30-21swapper/312:29:503
29221211828,34sleep20-21swapper/209:14:302
23743211827,71sleep323891-21unixbench_singl08:54:493
23743211827,71sleep323891-21unixbench_singl08:54:493
26003211713,22sleep10-21swapper/112:29:321
23514211612,86sleep123593-21find12:19:471
23514211612,86sleep123593-21find12:19:471
1245521159,39sleep20-21swapper/208:19:422
3576211410,36sleep20-21swapper/211:14:502
223221148,92sleep21290-21/usr/sbin/munin07:44:532
223221148,92sleep21290-21/usr/sbin/munin07:44:532
32336211311,16sleep10-21swapper/111:04:511
93921129,69sleep2951-21munin-run11:09:172
8917211234,13sleep20-21swapper/209:49:482
21183211225,26sleep00-21swapper/007:04:450
21183211225,26sleep00-21swapper/007:04:450
19245211215,79sleep319196-21apt-get10:24:593
19245211215,79sleep319196-21apt-get10:24:593
13560211031,34sleep30-21swapper/311:49:283
30862210914,79sleep330873-21ntp_states07:34:443
2941821099,36sleep20-21swapper/207:29:502
25793210910,39sleep20-21swapper/207:19:322
25793210910,39sleep20-21swapper/207:19:322
2874121078,37sleep20-21swapper/210:54:312
1925121078,38sleep20-21swapper/210:25:022
1925121078,38sleep20-21swapper/210:25:022
16538210715,77sleep31-21systemd08:34:173
3100210512,77sleep12706-21apt-get09:34:201
26667210513,72sleep026674-21modprobe10:49:150
20831210515,17sleep00-21swapper/010:29:370
17110210518,71sleep117151-21cat10:19:161
17110210518,71sleep117151-21cat10:19:161
27093210412,79sleep027169-21latency_hist07:24:190
27093210412,79sleep027169-21latency_hist07:24:190
20257210415,73sleep020258-21sshd10:26:340
20257210415,73sleep020258-21sshd10:26:340
26675210211,79sleep126757-21cmp10:49:161
22146210215,50sleep022296-21sed08:49:490
3368210112,75sleep03455-21grep07:49:470
12645210012,63sleep312157-21/usr/sbin/munin08:19:473
2720629920,64sleep327220-21cpu10:49:353
1867929813,69sleep018680-21cron10:24:150
2887529712,69sleep129003-21latency_hist12:39:181
2627229717,31sleep20-21swapper/210:44:492
1174929716,52sleep00-21swapper/009:59:480
1174929716,52sleep00-21swapper/009:59:480
3039729623,33sleep20-21swapper/210:59:352
1512829616,59sleep115103-21/usr/sbin/munin11:54:311
503329516,61sleep35010-21apt-get09:39:403
317872935,53sleep20-21swapper/209:20:442
2387629316,54sleep023847-21/usr/sbin/munin10:39:340
2387629316,54sleep023847-21/usr/sbin/munin10:39:340
1606029317,58sleep316117-21/usr/sbin/munin10:14:373
1606029317,58sleep316117-21/usr/sbin/munin10:14:373
1068829316,59sleep110748-21fschecks_time08:14:361
322929213,56sleep33258-21/usr/sbin/munin07:49:423
575229119,33sleep20-21swapper/209:40:232
49542910,13sleep124496-1kworker/1:1H07:54:441
49542910,13sleep124496-1kworker/1:1H07:54:441
3179229113,59sleep131586-21apt-get11:04:191
3031729113,62sleep030384-21df10:59:330
2606529114,63sleep326125-21latency07:19:413
2606529114,63sleep326125-21latency07:19:413
1561129124,45sleep115586-21latency_hist10:14:191
134112899,64sleep113533-21sed10:04:501
134112899,64sleep113533-21sed10:04:501
3223828613,54sleep332265-21/usr/sbin/munin11:04:493
205028428,39sleep20-21swapper/209:29:492
1090128421,40sleep110935-21munin-run09:59:161
399428321,41sleep33995-21cron11:19:153
342528219,41sleep13573-21grep11:14:491
2386428217,45sleep123882-21munin-node07:14:221
1916828221,29sleep20-21swapper/212:04:582
1916828221,29sleep20-21swapper/212:04:582
2865728019,30sleep20-21swapper/212:37:252
236928018,29sleep20-21swapper/211:13:172
2287428013,51sleep022950-21switchtime10:34:460
1225628017,47sleep312307-21cat11:44:403
1954727913,54sleep119600-21latency_hist12:09:171
1954727913,54sleep119600-21latency_hist12:09:171
1812527917,36sleep318276-21grep10:19:503
3010927814,46sleep030118-21/usr/sbin/munin09:14:490
2506927816,46sleep125171-21/usr/sbin/munin12:24:481
1489027817,39sleep015004-21latency_hist11:54:180
1375427819,28sleep20-21swapper/210:05:482
1375427819,28sleep20-21swapper/210:05:482
1031227821,36sleep110328-21ls08:14:161
942127722,41sleep19488-21systemctl11:34:441
150062770,6sleep10-21swapper/111:54:181
318927617,41sleep03196-21/usr/sbin/munin11:14:440
1999927627,29sleep00-21swapper/008:44:360
2763127519,23sleep00-21swapper/007:24:390
143327510,35sleep10-21swapper/109:29:321
717127415,46sleep17259-21timerandwakeup09:44:471
2754527416,28sleep30-21swapper/312:34:313
1630127425,31sleep00-21swapper/010:14:420
1630127425,31sleep00-21swapper/010:14:420
1322227424,24sleep00-21swapper/010:04:450
1322227424,24sleep00-21swapper/010:04:450
1182427421,24sleep10-21swapper/109:59:491
1182427421,24sleep10-21swapper/109:59:491
1030227412,15sleep00-21swapper/011:39:190
3227127317,32sleep032211-21apt-get07:40:060
3227127317,32sleep032211-21apt-get07:40:060
245342739,44sleep00-21swapper/012:24:350
410727214,41sleep14156-21unin-run07:54:171
2267527224,30sleep00-21swapper/012:19:180
1544827212,47sleep115498-21basename08:29:391
1058327219,23sleep00-21swapper/011:39:320
3250327117,29sleep30-21swapper/309:24:373
2756827115,40sleep127626-21latency_hist09:09:211
2342227118,31sleep00-21swapper/012:19:460
2342227118,31sleep00-21swapper/012:19:460
2110827126,19sleep10-21swapper/112:14:181
2110827126,19sleep10-21swapper/112:14:181
137627117,27sleep10-21swapper/111:09:361
1036227112,29sleep30-21swapper/308:14:183
556327019,30sleep30-21swapper/311:24:163
3135727022,19sleep10-21swapper/109:19:471
3099727017,27sleep30-21swapper/310:59:493
2901327022,25sleep30-21swapper/310:54:403
2018127017,26sleep30-21swapper/308:44:393
131727018,38sleep31384-21/usr/sbin/munin07:44:333
131727018,38sleep31384-21/usr/sbin/munin07:44:333
9162696,12sleep10-21swapper/107:42:541
9162696,12sleep10-21swapper/107:42:541
50026924,23sleep10-21swapper/109:24:471
43092696,49sleep10-21swapper/111:19:301
2735126921,20sleep10-21swapper/110:49:571
2145326919,31sleep321448-21apt-get08:49:323
192872698,48sleep00-21swapper/012:06:290
192872698,48sleep00-21swapper/012:06:290
192026918,31sleep00-21swapper/009:29:470
1497226926,20sleep30-21swapper/308:29:163
556026813,42sleep0369-21snmpd11:24:130
1846126817,31sleep118540-21fschecks_time08:39:361
172462689,14sleep00-21swapper/011:59:470
3170626712,40sleep031794-21cmp07:39:170
242182679,43sleep30-21swapper/312:24:173
242182679,43sleep30-21swapper/312:24:173
2295126722,22sleep10-21swapper/110:34:471
2143926721,31sleep321577-21grep10:29:523
1813626721,23sleep00-21swapper/008:39:190
1748526719,19sleep10-21swapper/108:34:481
1257726712,23sleep00-21swapper/008:19:440
2979626615,32sleep328856-21/usr/sbin/munin07:29:553
2092026624,18sleep10-21swapper/107:04:381
2092026624,18sleep10-21swapper/107:04:381
2076626625,21sleep20-21swapper/207:04:352
2076626625,21sleep20-21swapper/207:04:352
1868726613,37sleep118747-21systemd-run10:24:161
808226515,34sleep18090-21/usr/sbin/munin09:49:271
275326518,28sleep32885-21apt-config09:34:183
2595926516,30sleep326022-21/usr/sbin/munin10:44:463
2133226514,37sleep321398-21df12:14:323
798726417,28sleep30-21swapper/308:04:453
798726417,28sleep30-21swapper/308:04:453
2627326415,31sleep126335-21wc07:19:481
2627326415,31sleep126335-21wc07:19:481
2276026427,20sleep10-21swapper/108:54:181
1179726425,24sleep1369-21snmpd11:44:171
972426322,18sleep30-21swapper/309:54:343
759826319,19sleep00-21swapper/008:04:340
759826319,19sleep00-21swapper/008:04:340
3118226313,36sleep031226-21ntp_kernel_pll_09:19:410
3078826315,33sleep130390-21/usr/sbin/munin07:34:421
2883126323,22sleep00-21swapper/010:54:370
2589226319,21sleep00-21swapper/009:04:190
2589226319,21sleep00-21swapper/009:04:190
2193226320,27sleep122019-21timerandwakeup12:14:461
1999626315,25sleep00-21swapper/012:09:360
1999626315,25sleep00-21swapper/012:09:360
1572626313,34sleep015746-21/usr/sbin/munin08:29:460
1213526318,20sleep10-21swapper/111:44:381
956626217,19sleep30-21swapper/311:34:463
870726218,21sleep00-21swapper/009:49:460
715626219,21sleep00-21swapper/011:29:180
632726214,25sleep10-21swapper/109:44:181
37126222,18sleep00-21swapper/011:05:030
2846726217,26sleep328587-21/usr/sbin/munin09:09:513
2288826212,18sleep30-21swapper/307:09:483
1440726215,25sleep00-21swapper/010:09:370
1422026218,22sleep00-21swapper/008:24:450
2384526116,29sleep023863-21munin-node07:14:210
1645726118,21sleep30-21swapper/311:59:183
1498426117,20sleep30-21swapper/310:09:493
1087126114,28sleep00-21swapper/008:14:390
2886526013,31sleep028874-21munin-run12:39:170
2356826019,17sleep10-21swapper/107:14:181
227872607,11sleep30-21swapper/310:34:453
2075626015,27sleep120749-21apt-get10:29:341
139325917,25sleep11401-21diskstats07:44:351
139325917,25sleep11401-21diskstats07:44:351
2909025823,20sleep10-21swapper/107:29:411
1720625820,19sleep30-21swapper/308:34:413
1687325820,21sleep30-21swapper/311:59:383
1196225820,17sleep30-21swapper/308:19:193
592725716,26sleep35934-21diskstats11:24:323
2866325719,17sleep30-21swapper/307:29:193
2780125713,20sleep10-21swapper/107:24:441
2777225713,20sleep10-21swapper/109:09:371
2681625712,23sleep00-21swapper/009:04:550
2623125716,24sleep026274-21irqstats12:29:400
2245025718,25sleep022508-21if_err_eth007:09:370
1976725718,24sleep319779-21apt-get12:09:273
1976725718,24sleep319779-21apt-get12:09:273
1669925715,24sleep116693-21apt-get11:59:321
854825610,34sleep1369-21snmpd11:32:581
702025613,25sleep30-21swapper/308:01:343
477325614,27sleep04817-21ntpq07:54:400
477325614,27sleep04817-21ntpq07:54:400
2639725613,26sleep326411-21irqstats09:04:463
2318325615,20sleep00-21swapper/008:54:380
2318325615,20sleep00-21swapper/008:54:380
1317125619,20sleep30-21swapper/310:04:443
1317125619,20sleep30-21swapper/310:04:443
441125515,25sleep34464-21df07:54:313
441125515,25sleep34464-21df07:54:313
3256725516,23sleep032612-21irqstats09:24:380
3007525516,19sleep10-21swapper/110:59:181
2457425516,24sleep324570-21apt-get08:59:293
2457425516,24sleep324570-21apt-get08:59:293
2432625518,20sleep324332-21ntp_states10:39:453
2432625518,20sleep324332-21ntp_states10:39:453
2050425516,24sleep1287-21dbus-daemon10:29:171
2050425516,24sleep1287-21dbus-daemon10:29:171
2038825517,22sleep10-21swapper/108:44:451
132382555,14sleep10-21swapper/108:21:431
824425416,22sleep38297-21users11:29:493
1121525413,24sleep311211-21apt-get09:59:313
1121525413,24sleep311211-21apt-get09:59:313
869925318,19sleep00-21swapper/011:34:190
78112534,36sleep00-21swapper/009:49:100
727925315,21sleep10-21swapper/108:04:181
1371925313,23sleep113714-21apt-get08:24:301
1204725310,26sleep012041-21apt-get11:44:320
114832536,34sleep30-21swapper/311:40:403
2776525211,16sleep00-21swapper/009:09:360
272282528,17sleep00-21swapper/010:49:360
1860025212,16sleep30-21swapper/308:39:383
1837325211,24sleep018367-21apt-get08:39:310
741425113,22sleep37467-21users09:44:493
58262517,17sleep10-21swapper/111:24:261
1824425114,18sleep30-21swapper/312:04:313
1824425114,18sleep30-21swapper/312:04:313
119325110,28sleep01216-21grep11:09:310
3092425011,21sleep330986-21perl09:19:363
2929525016,15sleep10-21swapper/109:14:341
2380225010,19sleep323843-21perl07:14:203
1578725016,20sleep00-21swapper/011:54:460
817724915,16sleep30-21swapper/309:49:333
619024912,19sleep00-21swapper/007:59:410
2462724918,13sleep30-21swapper/312:24:383
973224715,17sleep00-21swapper/009:54:350
2895324613,15sleep00-21swapper/007:29:360
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional