You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-17 - 18:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun May 17, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25651247049,4679sleep314125-21kworker/u8:309:29:343
170162260012,2572sleep026633-1kworker/0:0H10:44:330
13315212189,1192sleep026633-1kworker/0:0H12:14:340
133212112612,1101sleep130628-1kworker/1:1H12:14:391
1188029997,958sleep311916-21ntpq12:09:403
1188029997,958sleep311916-21ntpq12:09:403
938827807,736sleep39425-21ntpq10:19:413
875276820,729sleep1969-21/usr/sbin/munin11:34:371
4249270910,685sleep026633-1kworker/0:0H10:04:270
1477425047,484sleep314810-21ntpq08:54:413
1035524865,468sleep310391-21ntpq12:04:393
10113246512,439sleep310153-21grep08:39:393
20734242622,336sleep331841-1kworker/3:2H09:14:353
844024098,281sleep38504-21wc08:34:383
28245232031,269sleep128264-21fschecks_count07:54:341
28245232031,269sleep128264-21fschecks_count07:54:341
7061231411,288sleep07090-21apt-config10:14:000
20870230019,228sleep30-21swapper/312:39:163
19220229744,214sleep319300-21fschecks_time09:09:353
20893229013,258sleep021037-21cat12:39:170
1235722577,239sleep312404-21cat10:29:383
1235722577,239sleep312404-21cat10:29:383
5039224245,152sleep35128-21latency_hist11:49:173
16736223953,166sleep316753-21modprobe07:19:153
22028222659,59sleep30-21swapper/307:34:373
19314221916,187sleep019321-21/usr/sbin/munin07:24:530
12932220927,168sleep313019-21fschecks_time08:49:363
30421220728,111sleep313298-21cyclictest09:44:363
13599220719,170sleep113616-21uniq07:09:361
13599220719,170sleep113616-21uniq07:09:361
2100122048,92sleep321123-21cat07:29:453
1138422019,174sleep311461-21fschecks_time08:44:353
24408218910,60sleep30-21swapper/309:24:453
232892188130,18sleep322982-21apt-get11:04:193
29947218419,120sleep329957-21modprobe09:44:153
14858217730,126sleep314795-21latency_hist07:14:183
14858217730,126sleep314795-21latency_hist07:14:183
7599217657,57sleep30-21swapper/310:14:363
3218221769,124sleep312492-21ntpd09:49:403
1266217233,14sleep30-21swapper/309:54:413
4988217115,85sleep15009-21awk11:49:151
2651221699,147sleep0363-21snmpd11:14:010
3236621656,32sleep30-21swapper/311:29:513
19497216215,127sleep019534-21ntp_kernel_pll_09:09:400
12391216226,117sleep212451-21processes07:04:452
2663216137,61sleep30-21swapper/311:39:443
5912215456,60sleep36039-21cat11:49:453
8173215064,19sleep30-21swapper/311:59:183
6181214614,51sleep26227-21/usr/sbin/munin10:09:422
13256214653,29sleep30-21swapper/307:09:183
908621449,115sleep39236-21unixbench_multi11:59:473
5516214128,71sleep35524-21modprobe10:09:153
21960214133,72sleep30-21swapper/310:59:383
21960214133,72sleep30-21swapper/310:59:383
1864921408,113sleep318675-21fschecks_count10:49:333
31778213717,19sleep30-21swapper/308:04:463
16529213711,73sleep316606-21fschecks_time12:24:343
16529213711,73sleep316606-21fschecks_time12:24:343
13592213710,111sleep013598-21diskstats07:09:350
13592213710,111sleep013598-21diskstats07:09:350
29308213433,63sleep329323-21vmstat11:19:493
18625213210,81sleep318753-21latency_hist07:24:203
13299212930,62sleep313289-21apt-get12:14:293
1214421270,6sleep10-21swapper/108:44:481
19123212527,59sleep30-21swapper/307:24:483
11595212513,89sleep111598-21/usr/sbin/munin12:09:341
11595212513,89sleep111598-21/usr/sbin/munin12:09:341
28508212014,84sleep228606-21fschecks_time11:19:362
15169212048,18sleep30-21swapper/312:19:383
15169212048,18sleep30-21swapper/312:19:383
21649211916,84sleep121668-21apt07:34:151
14972211719,73sleep01-21systemd10:39:160
354211517,78sleep1486-21latency_hist11:34:171
20484211413,86sleep220516-21awk10:54:412
16981211419,77sleep116491-21apt-get10:44:261
28781211221,70sleep128846-21fschecks_time09:39:371
868211014,80sleep3934-21sendmail_mailqu08:09:473
13184211041,44sleep113222-21modprobe07:09:161
6911210914,76sleep26959-21fschecks_count11:54:332
3562210911,83sleep03564-21sshd10:02:090
6991210814,81sleep27035-21ntp_kernel_err08:29:402
19073210826,55sleep219092-21apt-get09:09:262
2466321077,62sleep3363-21snmpd07:42:533
15848210731,64sleep315975-21expr10:39:473
15848210731,64sleep315975-21expr10:39:473
11024210714,75sleep211104-21grep10:24:442
11024210714,75sleep211104-21grep10:24:442
5582210648,44sleep30-21swapper/308:24:493
13354210612,80sleep013404-21unin-run10:34:150
13354210612,80sleep013404-21unin-run10:34:150
4123210519,70sleep24160-21sed11:44:402
2606021055,67sleep30-21swapper/307:45:563
15012210521,63sleep215101-21timerandwakeup08:54:482
22738210429,64sleep322824-21tr09:19:443
18810210416,69sleep118328-21apt-get07:24:261
1697021045,67sleep30-21swapper/309:00:423
3047621028,15sleep00-21swapper/009:44:370
15710210112,75sleep115772-21sendmail_mailst10:39:451
15710210112,75sleep115772-21sendmail_mailst10:39:451
9428210011,60sleep00-21swapper/010:19:420
2133421005,60sleep30-21swapper/310:58:493
14811210018,21sleep10-21swapper/110:38:081
312012995,60sleep30-21swapper/311:29:063
2711629835,47sleep30-21swapper/309:34:303
188112988,13sleep20-21swapper/207:24:272
995629712,64sleep00-21swapper/012:04:280
2709829717,24sleep30-21swapper/307:49:443
1615029720,51sleep016297-21grep07:14:480
899129616,65sleep29045-21ls10:19:312
246329643,39sleep32513-21awk08:14:433
246329643,39sleep32513-21awk08:14:433
100429616,59sleep1359-21/usr/sbin/munin08:09:491
2079629523,49sleep120937-21unixbench_multi10:54:481
3024029427,15sleep30-21swapper/307:59:493
3024029427,15sleep30-21swapper/307:59:493
1566529420,47sleep115689-21munin-run08:59:151
1298929317,53sleep013193-21latency_hist12:14:180
1298929317,53sleep013193-21latency_hist12:14:180
703729133,39sleep30-21swapper/308:29:413
3032629125,47sleep230419-21grep09:44:352
1004629120,54sleep110099-21fschecks_count12:04:331
2816528914,20sleep30-21swapper/311:19:173
239662898,66sleep123974-21ntp_states07:39:441
2295528819,50sleep023008-21systemd-run11:04:150
2295528819,50sleep023008-21systemd-run11:04:150
1739628817,56sleep30-21swapper/310:44:443
950028615,55sleep29529-21latency_hist08:39:172
47928619,47sleep2568-21fschecks_time08:09:372
423628621,27sleep00-21swapper/008:19:490
2732428619,43sleep10-21swapper/109:34:361
256932867,47sleep30-21swapper/311:09:523
2345628623,43sleep222982-21apt-get11:04:262
1490228629,43sleep314925-21munin-run10:39:153
1198528618,42sleep012140-21grep08:44:480
622828518,13sleep30-21swapper/310:09:433
3050228520,31sleep20-21swapper/211:24:452
1490028522,47sleep114968-21sh12:19:311
1490028522,47sleep114968-21sh12:19:311
891128420,40sleep09052-21grep08:34:470
51312846,12sleep00-21swapper/011:49:180
2658328415,55sleep126685-21sed11:14:181
1815928410,62sleep318224-21cat12:29:353
1815928410,62sleep318224-21cat12:29:353
156882849,61sleep015708-21open_inodes10:39:440
156882849,61sleep015708-21open_inodes10:39:440
281572838,16sleep30-21swapper/307:54:283
281572838,16sleep30-21swapper/307:54:283
180328338,11sleep30-21swapper/311:38:313
240228213,53sleep22462-21processes08:14:422
240228213,53sleep22462-21processes08:14:422
1826428224,39sleep118352-21latency12:29:371
1826428224,39sleep118352-21latency12:29:371
275982819,11sleep00-21swapper/007:50:460
2360528129,29sleep20-21swapper/207:39:372
972028013,52sleep19806-21latency_hist12:04:171
387128010,55sleep34084-21latency_hist10:04:183
1498528018,30sleep00-21swapper/012:19:350
1498528018,30sleep00-21swapper/012:19:350
2231827924,24sleep10-21swapper/109:19:341
2022427921,41sleep120374-21sed12:34:461
1867927916,47sleep018766-21grep10:49:340
1845627923,25sleep10-21swapper/107:24:181
709427838,21sleep10-21swapper/111:54:361
400027716,15sleep10-21swapper/108:19:461
1481827723,29sleep20-21swapper/207:14:172
1481827723,29sleep20-21swapper/207:14:172
96122769,10sleep00-21swapper/012:03:310
2288127610,54sleep00-21swapper/007:35:110
186127622,35sleep2225-21systemd-journal11:39:142
1753827621,14sleep30-21swapper/309:04:303
1042127617,42sleep210458-21ps12:04:432
325262756,19sleep00-21swapper/011:31:090
2845427519,28sleep20-21swapper/209:39:182
2535827515,41sleep125490-21latency_hist09:29:191
2309927520,28sleep20-21swapper/211:04:182
2309927520,28sleep20-21swapper/211:04:182
1174727530,11sleep30-21swapper/310:28:413
1174727530,11sleep30-21swapper/310:28:413
445827429,10sleep30-21swapper/308:20:043
3125927423,28sleep10-21swapper/111:29:161
268272749,50sleep026818-21apt-get11:14:330
788027321,32sleep17887-21modprobe08:34:151
2136927318,31sleep021398-21munin-run10:59:150
1775627316,33sleep10-21swapper/112:29:161
1775627316,33sleep10-21swapper/112:29:161
153092739,16sleep10-21swapper/107:14:241
2506627216,43sleep025120-21cpuspeed07:44:320
24532729,16sleep10-21swapper/111:39:381
1879727115,29sleep00-21swapper/009:09:150
96842707,17sleep00-21swapper/008:39:190
557027020,22sleep20-21swapper/210:09:182
2556827018,22sleep00-21swapper/009:29:300
1954127022,23sleep20-21swapper/212:34:302
116027012,12sleep10-21swapper/109:54:391
3159026924,20sleep20-21swapper/211:29:332
974426810,43sleep10-21swapper/108:39:281
2570026817,36sleep225757-21fschecks_time09:29:452
2552726822,25sleep10-21swapper/107:44:451
2286626713,37sleep122867-21sshd11:03:071
2286626713,37sleep122867-21sshd11:03:071
2112626716,33sleep021271-21grep07:29:460
797826618,25sleep00-21swapper/010:14:450
2576426621,19sleep20-21swapper/211:09:552
1156426618,27sleep20-21swapper/208:44:382
1024626617,33sleep110274-21cron10:24:151
646626517,21sleep10-21swapper/110:09:491
3039526512,29sleep10-21swapper/107:59:511
3039526512,29sleep10-21swapper/107:59:511
1855526524,20sleep20-21swapper/207:24:192
1281026528,21sleep20-21swapper/212:14:152
1281026528,21sleep20-21swapper/212:14:152
808426414,34sleep08148-21sendmail-msp11:59:150
617826415,33sleep16190-21vmstat11:49:471
567626417,22sleep00-21swapper/008:24:500
39292648,13sleep20-21swapper/208:19:432
3236526414,26sleep00-21swapper/009:49:450
2469026414,22sleep20-21swapper/209:24:482
544526315,31sleep25476-21/usr/sbin/munin08:24:432
348926313,15sleep00-21swapper/011:44:170
3176726316,28sleep231786-21apt-get09:49:272
2837626311,38sleep028379-21sshd11:19:220
2837526319,21sleep00-21swapper/009:39:160
2191426320,26sleep20-21swapper/209:19:152
211122638,12sleep00-21swapper/009:14:450
33792625,45sleep20-21swapper/209:59:532
2849826214,20sleep10-21swapper/111:19:351
2619226218,21sleep10-21swapper/109:29:561
2327126220,20sleep20-21swapper/207:39:172
2093326222,22sleep20-21swapper/207:29:442
1886326217,25sleep20-21swapper/209:09:172
3220726115,20sleep00-21swapper/008:04:510
3161326121,17sleep10-21swapper/109:49:181
3138026118,25sleep10-21swapper/108:04:361
2527226118,28sleep025054-21apt-get09:29:180
1876726114,33sleep118811-21cat10:49:351
62242605,11sleep20-21swapper/211:50:132
2971526014,20sleep00-21swapper/007:59:300
2971526014,20sleep00-21swapper/007:59:300
2249426014,22sleep00-21swapper/009:19:370
220426018,20sleep00-21swapper/008:14:360
220426018,20sleep00-21swapper/008:14:360
186392607,14sleep20-21swapper/210:49:322
1670926013,27sleep216903-21sed10:44:192
1670926013,27sleep216903-21sed10:44:192
1340526011,27sleep10-21swapper/110:34:161
1340526011,27sleep10-21swapper/110:34:161
3175525913,27sleep231777-21/usr/sbin/munin08:04:452
2862625918,25sleep20-21swapper/207:54:432
2862625918,25sleep20-21swapper/207:54:432
2524925918,19sleep20-21swapper/207:44:382
2369525920,20sleep10-21swapper/111:04:371
1618825920,18sleep00-21swapper/008:59:380
161825918,22sleep20-21swapper/209:54:482
1597525920,16sleep20-21swapper/207:14:462
867425813,30sleep28733-21latency11:59:382
458425815,27sleep14618-21irqstats10:05:451
1791625812,29sleep10-21swapper/109:04:401
1671525815,21sleep20-21swapper/212:24:372
1671525815,21sleep20-21swapper/212:24:372
2732925714,28sleep227405-21awk11:14:482
2661325712,18sleep20-21swapper/207:49:302
3042025616,20sleep00-21swapper/011:24:430
2974425614,23sleep229811-21df07:59:362
2974425614,23sleep229811-21df07:59:362
1579425615,24sleep015942-21date08:59:180
2392025516,20sleep00-21swapper/007:39:430
1808025516,18sleep20-21swapper/212:29:342
1808025516,18sleep20-21swapper/212:29:342
1625725517,22sleep216351-21memory08:59:402
516325416,25sleep15184-21fschecks_count08:24:381
328625415,18sleep10-21swapper/108:19:171
328625415,18sleep10-21swapper/108:19:171
2949825413,20sleep00-21swapper/007:59:180
2949825413,20sleep00-21swapper/007:59:180
2869425412,24sleep028687-21apt-get09:39:320
2359125412,23sleep023694-21perl11:04:360
2112725422,17sleep10-21swapper/109:14:461
1841425415,23sleep018449-21ntp_states12:29:400
1841425415,23sleep018449-21ntp_states12:29:400
983125318,16sleep20-21swapper/208:39:332
749925312,20sleep20-21swapper/210:14:352
1373125313,25sleep213767-21/usr/sbin/munin10:34:342
807125210,18sleep10-21swapper/110:14:461
2792325213,15sleep10-21swapper/107:54:181
2040425214,17sleep10-21swapper/109:14:171
31332516,36sleep20-21swapper/211:40:202
3000025116,17sleep20-21swapper/209:44:182
2304125112,23sleep223133-21users09:19:472
816625012,22sleep18161-21apt-get08:34:281
60725011,15sleep00-21swapper/008:09:390
2320025018,17sleep10-21swapper/107:39:161
1620625016,22sleep216231-21latency_hist12:24:172
1620625016,22sleep216231-21latency_hist12:24:172
1358925010,23sleep213625-21irqstats12:15:082
1302425015,18sleep00-21swapper/008:49:370
1051225011,21sleep110532-21apt-get10:24:271
1051225011,21sleep110532-21apt-get10:24:271
24382498,16sleep10-21swapper/109:59:271
1192424914,19sleep00-21swapper/007:04:310
1346724815,17sleep20-21swapper/208:49:472
3069424711,21sleep130700-21ntp_states09:44:421
2783624713,20sleep127837-21sshd11:16:081
1712324711,21sleep017129-21df_inode07:19:320
1710724713,19sleep20-21swapper/210:44:392
1606524711,21sleep116070-21diskstats08:59:351
131524710,15sleep00-21swapper/009:54:420
1099324711,21sleep010999-21ntp_states10:24:420
1099324711,21sleep010999-21ntp_states10:24:420
6782469,20sleep0686-21diskstats11:34:320
122992469,18sleep212356-21irqstats10:29:372
122992469,18sleep212356-21irqstats10:29:372
37282459,17sleep00-21swapper/011:44:290
164462439,19sleep016513-21df12:24:310
164462439,19sleep016513-21df12:24:310
130652418,16sleep10-21swapper/108:49:381
67082407,13sleep00-21swapper/008:29:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional