You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 01:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon May 18, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12161271249,7093sleep1463-21kworker/u8:110:14:381
12161271249,7093sleep1463-21kworker/u8:110:14:381
105542203027,1992sleep18454-21kworker/1:210:09:331
17648214857,1461sleep217620-21apt-get08:49:392
20762210069,982sleep1194-21jbd2/mmcblk2p2-08:59:271
15580244826,408sleep115652-21/usr/sbin/munin10:24:361
15580244826,408sleep115652-21/usr/sbin/munin10:24:361
20200242913,402sleep320210-21ntpq07:14:413
2738023346,316sleep127405-21awk09:19:341
30504226724,230sleep130540-21ntpq09:29:361
30504226724,230sleep130540-21ntpq09:29:361
22679225922,225sleep122718-21ntpq12:30:141
30984225140,103sleep130997-21grep07:49:311
31454224253,170sleep10-21swapper/111:14:431
29817221322,33sleep129820-21cron09:29:111
2578222120,11sleep117459-21cyclictest12:39:141
19116221210,62sleep119217-21timerwakeupswit07:14:151
20221221112,109sleep10-21swapper/107:14:431
505222018,181sleep3363-21snmpd09:49:573
505222018,181sleep3363-21snmpd09:49:573
1557122008,153sleep115720-21latency_hist12:09:141
32612219212,26sleep10-21swapper/111:19:311
32612219212,26sleep10-21swapper/111:19:311
21598218720,146sleep221617-21fschecks_count10:44:312
955021790,12sleep10-21swapper/110:04:441
955021790,12sleep10-21swapper/110:04:441
27085217513,47sleep10-21swapper/110:59:451
20935217290,24sleep120997-21uname12:24:411
758021687,110sleep17659-21date08:19:141
4625216810,57sleep14726-21latency_hist11:34:131
2142321677,122sleep121485-21grep07:19:321
2343021649,118sleep123472-21ntp_kernel_err10:49:351
2343021649,118sleep123472-21ntp_kernel_err10:49:351
11457216221,61sleep10-21swapper/111:54:361
11457216221,61sleep10-21swapper/111:54:361
32137216024,120sleep332138-21sshd07:54:013
28831215933,39sleep028899-21ls09:24:340
26310215817,124sleep326343-21fschecks_count07:34:323
17162214646,84sleep117285-21latency_hist08:49:141
23302214524,100sleep323325-21python3.9.prerm12:30:483
6931214010,88sleep17001-21systemd09:59:121
24428213630,59sleep10-21swapper/107:29:131
9366213541,53sleep10-21swapper/108:24:271
176621359,114sleep1363-21snmpd11:24:321
32440213436,85sleep132441-21apt-get07:54:241
2006821346,116sleep1363-21snmpd12:22:471
388221338,56sleep10-21swapper/109:49:141
21981213213,104sleep322002-21/usr/sbin/munin10:44:403
18055213227,93sleep118094-21ntpq08:50:151
20555212626,73sleep020595-21fschecks_count12:24:320
29967212330,53sleep10-21swapper/107:44:431
29967212330,53sleep10-21swapper/107:44:431
1205212311,18sleep10-21swapper/109:40:021
2032212226,52sleep10-21swapper/107:59:401
1331121205,106sleep00-21swapper/010:17:520
1331121205,106sleep00-21swapper/010:17:520
4634211828,51sleep10-21swapper/109:49:431
4634211828,51sleep10-21swapper/109:49:431
24199211817,48sleep2363-21snmpd07:26:522
1577421179,51sleep10-21swapper/112:09:261
25262211312,84sleep025328-21users12:34:450
20831211311,88sleep2363-21snmpd07:16:502
29107211241,15sleep10-21swapper/107:44:141
25929211214,47sleep10-21swapper/109:14:421
1545211112,84sleep01564-21apt-get07:59:230
27927211014,51sleep10-21swapper/111:04:331
21365211013,79sleep021442-21/usr/sbin/munin08:59:430
17289210949,28sleep00-21swapper/007:09:130
10642210817,13sleep10-21swapper/111:52:101
19966210713,78sleep320034-21/usr/sbin/munin10:39:283
19841210718,74sleep019860-21grep07:14:340
11812210716,76sleep011960-21sed11:54:430
11812210716,76sleep011960-21sed11:54:430
15771210631,20sleep30-21swapper/310:24:423
15771210631,20sleep30-21swapper/310:24:423
13786210613,71sleep313709-21/usr/sbin/munin10:19:293
13786210613,71sleep313709-21/usr/sbin/munin10:19:293
13727210610,61sleep113751-21apt-get08:39:121
5881210412,78sleep25882-21sshd11:36:042
1738821048,57sleep117599-21latency_hist07:09:141
13187210441,41sleep10-21swapper/111:59:411
13187210441,41sleep10-21swapper/111:59:411
2507521038,60sleep125120-21ntpq07:29:371
2507521038,60sleep125120-21ntpq07:29:371
17324210211,18sleep00-21swapper/012:14:260
3007821019,78sleep330081-21/usr/sbin/munin09:29:213
3007821019,78sleep330081-21/usr/sbin/munin09:29:213
2019210112,70sleep02059-21/usr/sbin/munin11:25:080
2875729910,74sleep228758-21sshd11:06:042
1535529918,12sleep10-21swapper/108:41:471
620329635,30sleep20-21swapper/208:14:232
2434429627,27sleep10-21swapper/109:09:411
1115429613,66sleep011188-21irqstats08:29:340
144629521,58sleep01594-21apt-get11:24:150
144629521,58sleep01594-21apt-get11:24:150
277522945,11sleep30-21swapper/307:39:163
2306029213,66sleep023063-21grep10:49:260
2306029213,66sleep023063-21grep10:49:260
211662926,74sleep00-21swapper/010:43:280
1430529216,59sleep214277-21apt-get08:39:422
349629110,62sleep13534-21ntp_kernel_err08:04:371
158252919,58sleep315749-21/usr/sbin/munin12:09:283
806729023,13sleep10-21swapper/108:19:341
3054929022,53sleep230572-21modprobe11:14:112
247902907,13sleep30-21swapper/312:34:363
1382229017,52sleep213915-21grep10:19:322
1382229017,52sleep213915-21grep10:19:322
1217328913,37sleep112176-21cron08:34:111
297452886,13sleep10-21swapper/111:09:381
2643228810,25sleep10-21swapper/107:34:341
2341028819,50sleep023422-21dpkg-deb12:30:580
2052628811,18sleep10-21swapper/110:39:421
1926528820,13sleep10-21swapper/108:54:311
1770028816,54sleep10-21swapper/112:14:351
1704228812,58sleep217100-21irqstats10:29:332
1244428818,44sleep212463-21apt-get08:34:252
723428719,13sleep10-21swapper/109:59:241
3226428725,35sleep00-21swapper/009:34:430
2271728714,55sleep322842-21find07:24:143
791028627,40sleep38005-21fschecks_time08:19:323
2446528621,39sleep323856-21/usr/sbin/munin09:09:433
733328516,55sleep07365-21fschecks_count09:59:310
529428516,22sleep00-21swapper/009:53:050
529428516,22sleep00-21swapper/009:53:050
2461528521,38sleep024609-21apt-get10:54:310
2461528521,38sleep024609-21apt-get10:54:310
2382528521,43sleep323956-21unixbench_multi07:24:593
749728424,40sleep37528-21modprobe08:19:123
2229228420,12sleep10-21swapper/109:04:241
220702849,21sleep122151-21cat10:44:431
3018328316,51sleep230189-21diskstats09:29:292
3018328316,51sleep230189-21diskstats09:29:292
1944528318,13sleep10-21swapper/110:36:351
1838628314,33sleep30-21swapper/310:34:273
1419028234,33sleep10-21swapper/110:19:391
1419028234,33sleep10-21swapper/110:19:391
133228115,27sleep00-21swapper/009:40:050
798428033,33sleep10-21swapper/111:44:231
798428033,33sleep10-21swapper/111:44:231
704828051,14sleep10-21swapper/111:39:421
704828051,14sleep10-21swapper/111:39:421
867027916,38sleep08816-21unixbench_multi11:44:420
867027916,38sleep08816-21unixbench_multi11:44:420
317512798,19sleep131841-21perl09:34:321
2515127918,39sleep324590-21/usr/sbin/munin10:54:463
2515127918,39sleep324590-21/usr/sbin/munin10:54:463
1222427915,16sleep30-21swapper/308:34:143
1063127915,28sleep30-21swapper/310:09:353
2669327820,41sleep326705-21modprobe09:19:123
904527717,29sleep30-21swapper/310:04:353
904527717,29sleep30-21swapper/310:04:353
620327723,28sleep00-21swapper/011:39:130
472727726,32sleep20-21swapper/209:49:442
472727726,32sleep20-21swapper/209:49:442
243642770,7sleep30-21swapper/310:54:123
243642770,7sleep30-21swapper/310:54:123
1745827721,22sleep00-21swapper/010:29:430
1373327712,52sleep00-21swapper/010:19:260
1373327712,52sleep00-21swapper/010:19:260
133127724,36sleep31379-21cat07:59:143
1085027713,49sleep310991-21cut11:54:143
973127611,53sleep20-21swapper/210:04:542
973127611,53sleep20-21swapper/210:04:542
312827626,27sleep00-21swapper/008:04:280
2278827612,30sleep20-21swapper/209:04:412
698627512,26sleep00-21swapper/011:39:410
698627512,26sleep00-21swapper/011:39:410
681127522,26sleep30-21swapper/311:39:363
681127522,26sleep30-21swapper/311:39:363
2943527520,36sleep229522-21fschecks_time07:44:322
2943527520,36sleep229522-21fschecks_time07:44:322
2817127517,39sleep028197-21netstat07:39:370
2440127514,46sleep224403-21/usr/sbin/munin12:34:232
1788027511,51sleep317889-21forks07:09:323
289427416,14sleep30-21swapper/308:04:143
2801627414,14sleep30-21swapper/311:04:353
2628427422,33sleep026391-21grep10:59:310
1779727411,48sleep017843-21grep07:09:290
1009127423,31sleep210158-21grep11:49:412
611827332,29sleep10-21swapper/111:39:061
51382737,13sleep10-21swapper/108:09:411
2758627312,49sleep327662-21switchtime09:19:403
1439927319,38sleep314455-21tune2fs08:39:513
2593427217,33sleep325960-21munin-run07:34:113
2593427217,33sleep325960-21munin-run07:34:113
1634527220,26sleep00-21swapper/008:44:440
250792719,15sleep117459-21cyclictest10:54:441
250792719,15sleep117459-21cyclictest10:54:441
1651927119,35sleep216654-21unixbench_multi08:44:462
1477527122,28sleep00-21swapper/008:40:000
957627013,27sleep30-21swapper/308:24:333
1975227011,43sleep219822-21sed08:54:442
1953427016,37sleep219649-21timerandwakeup12:19:452
3024326920,29sleep30-21swapper/307:44:453
3024326920,29sleep30-21swapper/307:44:453
2978426918,30sleep30-21swapper/311:09:403
2263526921,30sleep20-21swapper/207:24:132
1244926923,24sleep30-21swapper/310:15:013
1244926923,24sleep30-21swapper/310:15:013
714826813,18sleep20-21swapper/211:39:432
714826813,18sleep20-21swapper/211:39:432
3220926814,33sleep032288-21latency_hist07:54:140
2060426823,26sleep20-21swapper/210:39:432
1779726831,27sleep10-21swapper/110:30:171
1760026818,23sleep30-21swapper/312:14:333
1653826819,29sleep016559-21munin-run10:29:110
1653826819,29sleep016559-21munin-run10:29:110
98826722,25sleep30-21swapper/309:39:233
486726720,24sleep30-21swapper/308:09:353
2984226712,37sleep029917-21cat11:09:410
2622626719,20sleep00-21swapper/007:34:290
1910526715,28sleep00-21swapper/012:19:350
732526616,34sleep37331-21diskstats09:59:303
307626620,25sleep20-21swapper/211:29:132
2751126613,20sleep20-21swapper/209:19:392
2455926618,30sleep024676-21timerwakeupswit09:09:440
2428126613,39sleep224322-21ntpq09:09:382
17392668,36sleep20-21swapper/209:40:152
1586926616,32sleep215963-21fschecks_time12:09:312
1285226620,29sleep012913-21cat11:59:320
1285226620,29sleep012913-21cat11:59:320
601726519,25sleep20-21swapper/208:14:142
3170726525,22sleep20-21swapper/209:34:292
2207426522,23sleep30-21swapper/309:04:143
2051626520,21sleep30-21swapper/308:59:133
3265926418,19sleep00-21swapper/007:54:350
2630226420,22sleep20-21swapper/207:34:312
2284826415,16sleep00-21swapper/007:24:150
1216326412,35sleep212131-21apt-get10:14:392
1216326412,35sleep212131-21apt-get10:14:392
83012639,12sleep00-21swapper/008:19:410
594326321,19sleep30-21swapper/309:54:333
383126314,30sleep33873-21true09:49:123
1846426315,32sleep218361-21/usr/sbin/munin10:34:302
1722526319,19sleep30-21swapper/307:09:123
1686826320,24sleep00-21swapper/012:12:040
1127926314,32sleep20-21swapper/208:29:362
745326215,28sleep27506-21if_eth009:59:332
2514126222,21sleep00-21swapper/007:29:400
2514126222,21sleep00-21swapper/007:29:400
1478126222,21sleep20-21swapper/212:04:432
135126219,25sleep30-21swapper/311:24:143
135126219,25sleep30-21swapper/311:24:143
487126121,22sleep30-21swapper/311:34:313
25332617,11sleep20-21swapper/208:00:182
1750726114,31sleep317109-21apt-get08:49:163
1383726115,29sleep313999-21apt-config08:39:143
1264226118,27sleep212644-21/usr/sbin/munin11:59:222
1264226118,27sleep212644-21/usr/sbin/munin11:59:222
1110026120,19sleep00-21swapper/010:09:450
3136426021,22sleep20-21swapper/207:49:402
3110726021,16sleep00-21swapper/011:14:340
3010726015,26sleep030170-21cpuspeed09:29:270
3010726015,26sleep030170-21cpuspeed09:29:270
2369825915,30sleep223707-21systemd-gpt-aut12:31:162
2282425915,21sleep30-21swapper/309:04:423
2113025910,33sleep221164-21/usr/sbin/munin08:59:362
1939625913,29sleep019449-21/usr/sbin/munin08:54:340
1620725916,19sleep30-21swapper/308:44:393
2044125813,29sleep320459-21/usr/sbin/munin12:24:223
1822525816,24sleep218248-21ntp_states07:09:392
825625717,21sleep30-21swapper/311:44:333
825625717,21sleep30-21swapper/311:44:333
60325717,19sleep20-21swapper/207:54:452
567325718,18sleep00-21swapper/009:54:260
1885325714,30sleep018890-21ps10:34:400
1656425712,29sleep216642-21threads07:04:432
962625612,26sleep09721-21perl11:49:310
83912569,31sleep28440-21ntp_states11:44:362
83912569,31sleep28440-21ntp_states11:44:362
781725616,24sleep27811-21apt-get08:19:272
35752566,31sleep20-21swapper/211:29:342
2914625617,23sleep20-21swapper/207:44:152
285425615,20sleep20-21swapper/209:44:352
1449925616,17sleep30-21swapper/312:04:363
3120525517,18sleep20-21swapper/211:14:362
2989825517,19sleep00-21swapper/007:44:420
2989825517,19sleep00-21swapper/007:44:420
2878025516,24sleep328829-21irqstats09:24:333
2774525516,19sleep00-21swapper/011:04:280
2133625513,25sleep021330-21apt-get07:19:270
2086925517,24sleep220915-21ntp_offset12:24:382
1659025513,19sleep20-21swapper/210:29:132
1659025513,19sleep20-21swapper/210:29:132
398625415,24sleep34107-21timerwakeupswit11:29:433
367025414,16sleep20-21swapper/208:04:422
3173325416,22sleep331742-21diskstats09:34:303
274302548,13sleep20-21swapper/211:03:542
2475425413,25sleep224760-21diskstats07:29:302
2475425413,25sleep224760-21diskstats07:29:302
235402549,31sleep223598-21sendmail_mailqu10:49:402
235402549,31sleep223598-21sendmail_mailqu10:49:402
1135725414,17sleep30-21swapper/311:54:343
1135725414,17sleep30-21swapper/311:54:343
997225321,17sleep00-21swapper/008:24:420
574325314,24sleep25752-21df_inode09:54:282
53925315,24sleep2612-21wc11:19:402
53925315,24sleep2612-21wc11:19:402
2594925314,24sleep225888-21latency_hist10:59:132
2594925314,24sleep225888-21latency_hist10:59:132
2589725312,25sleep025926-21ps09:14:410
2587725313,17sleep20-21swapper/209:14:392
1890925314,23sleep318915-21apt-get12:19:263
1312925315,22sleep313130-21ntp_states11:59:393
1312925315,22sleep313130-21ntp_states11:59:393
1297825314,25sleep013072-21grep08:34:430
1150425312,23sleep211540-21ntp_states11:54:372
1150425312,23sleep211540-21ntp_states11:54:372
1035325313,24sleep210347-21apt-get10:09:262
753325216,17sleep00-21swapper/008:19:130
345025215,20sleep00-21swapper/011:29:320
2258825218,19sleep00-21swapper/009:04:350
1739725216,21sleep217403-21df_inode12:14:282
1420925212,23sleep014221-21apt-get12:04:250
358025114,21sleep33586-21ntp_states08:04:393
2627125116,17sleep30-21swapper/310:59:303
1710125115,21sleep317103-21irqstats10:29:343
468325013,19sleep00-21swapper/008:09:280
2701125012,23sleep027005-21apt-get09:19:250
256725013,21sleep02563-21apt-get09:44:250
175825013,21sleep31747-21apt-get11:24:263
936724913,21sleep29435-21df08:24:282
880524913,20sleep08798-21apt-get10:04:280
880524913,20sleep08798-21apt-get10:04:280
2200524919,16sleep00-21swapper/010:44:410
1894424713,21sleep00-21swapper/008:54:130
3091024512,18sleep330917-21df_inode11:14:283
285562438,18sleep20-21swapper/209:24:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional