You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 03:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue May 19, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
247112603813,6013sleep210951-21kworker/2:100:15:022
139592596413,5939sleep132173-21kworker/1:223:41:461
27622342114,3394sleep216621-21kworker/2:223:06:022
27622342114,3394sleep216621-21kworker/2:223:06:022
2138524847,409sleep321424-21ntpq20:39:353
31972244217,409sleep331987-21awk22:54:443
31972244217,409sleep331987-21awk22:54:443
307722770,63sleep324529-21cyclictest21:24:323
30102236138,26sleep31-21systemd23:09:103
30102236138,26sleep31-21systemd23:09:103
4568223068,81sleep34360-21/usr/sbin/munin21:29:293
28040221732,150sleep328054-21wc19:19:123
29767221418,142sleep329914-21unixbench_singl21:04:423
28191221295,21sleep30-21swapper/322:44:513
17716221095,19sleep30-21swapper/320:29:133
17895220313,177sleep317946-21cpuspeed20:29:253
12540220341,106sleep30-21swapper/323:38:453
18782218923,121sleep318848-21fschecks_time22:15:283
18782218923,121sleep318848-21fschecks_time22:15:283
12123218813,74sleep312133-21acpi21:54:203
12123218813,74sleep312133-21acpi21:54:203
2196218217,141sleep02286-21grep19:39:320
1315921829,157sleep013209-21sed23:39:310
178121788,99sleep31825-21awk19:39:103
6845217251,92sleep30-21swapper/323:19:323
19526217256,102sleep319540-21grep20:34:303
19526217256,102sleep319540-21grep20:34:303
15013217148,102sleep30-21swapper/323:44:433
24834216942,21sleep30-21swapper/322:34:223
5552216712,140sleep05561-21ntpq23:14:440
17199216452,92sleep30-21swapper/322:09:343
799216346,101sleep3882-21cat22:59:413
799216346,101sleep3882-21cat22:59:413
31234216213,103sleep331386-21latency_hist19:29:133
3396216119,34sleep30-21swapper/323:09:303
10697216019,27sleep310786-21fschecks_time21:49:283
12121215711,130sleep212129-21ntpq20:09:382
9271214317,26sleep30-21swapper/321:44:353
14994214050,78sleep315033-21modprobe22:04:093
27981213910,49sleep30-21swapper/321:00:303
17599213714,24sleep30-21swapper/323:54:243
30676213135,81sleep30-21swapper/321:09:273
6612213020,86sleep36756-21unixbench_multi21:34:423
22222213072,22sleep322268-21cpu00:09:263
22222213072,22sleep322268-21cpu00:09:263
6301212912,102sleep06332-21nfs_client21:34:350
402212917,39sleep30-21swapper/321:14:443
26026212820,93sleep026027-21sshd22:38:020
15743212862,25sleep315777-21ps22:04:383
20767212618,86sleep020858-21cut00:04:300
20767212618,86sleep020858-21cut00:04:300
14907212611,102sleep1363-21snmpd22:03:021
27823212515,94sleep227847-21cut22:44:182
3073721248,101sleep330882-21awk00:34:423
3073721248,101sleep330882-21awk00:34:423
22843212411,17sleep30-21swapper/322:27:033
22843212411,17sleep30-21swapper/322:27:033
14986212331,78sleep315037-21cat20:19:283
10603212261,44sleep30-21swapper/323:29:453
8767211930,61sleep20-21swapper/221:44:102
4469211915,81sleep14360-21/usr/sbin/munin21:29:271
32093211857,16sleep30-21swapper/319:29:423
10225211811,22sleep10-21swapper/123:29:391
28889211626,75sleep328966-21timerandwakeup19:19:403
1408521160,7sleep10-21swapper/120:14:411
26942211517,77sleep10-21swapper/120:59:111
23975211511,33sleep324101-21cat22:29:413
27103211415,81sleep327311-21latency_hist20:59:133
2884211313,84sleep22909-21/usr/sbin/munin21:24:282
13186211111,85sleep013205-21apt-get20:14:200
8303211013,82sleep38195-21/usr/sbin/munin23:24:283
14550211020,73sleep324529-21cyclictest20:19:123
11247211018,75sleep211373-21timerwakeupswit21:49:392
32403210916,79sleep132424-21sort22:59:111
32403210916,79sleep132424-21sort22:59:111
9832210755,33sleep29960-21/usr/sbin/munin21:44:472
29575210712,81sleep129603-21df22:49:261
8561210621,63sleep30-21swapper/319:59:253
26705210515,76sleep026799-21memory22:39:310
1256210411,78sleep31268-21cat21:19:203
20883210312,69sleep321025-21unixbench_multi22:19:403
20883210312,69sleep321025-21unixbench_multi22:19:403
10578210342,20sleep310612-21ps20:04:383
9331210216,65sleep00-21swapper/021:44:370
30100210218,65sleep230175-21switchtime22:49:392
29883210214,70sleep029948-21ls19:24:260
11399210114,70sleep111440-21uniq23:34:271
2606029922,51sleep026173-21latency_hist19:14:120
132962999,17sleep30-21swapper/323:39:343
65312987,17sleep30-21swapper/319:52:033
280012989,12sleep30-21swapper/300:27:023
219382988,78sleep321979-21systemd-run00:09:103
219382988,78sleep321979-21systemd-run00:09:103
37672978,77sleep33851-21perl19:44:313
1752629712,64sleep00-21swapper/023:54:210
1381729616,56sleep013904-21fschecks_time21:59:300
822029514,63sleep38366-21grep21:39:433
766129515,61sleep07787-21/usr/sbin/munin19:54:420
253602957,75sleep325357-21apt-get00:19:233
253602957,75sleep325357-21apt-get00:19:233
2324729519,51sleep023134-21/usr/sbin/munin19:04:280
1659429513,69sleep116708-21latency_hist22:09:121
2882229319,57sleep30-21swapper/300:29:333
2101829314,25sleep20-21swapper/220:39:262
2038229311,45sleep30-21swapper/300:04:093
68029216,55sleep1502-21/usr/sbin/munin19:34:401
96729142,26sleep10-21swapper/121:19:101
248242919,14sleep30-21swapper/319:09:273
2061129114,63sleep0225-21systemd-journal20:38:020
2061129114,63sleep0225-21systemd-journal20:38:020
266802909,42sleep30-21swapper/322:39:303
2521529017,53sleep225245-21proc_pri19:09:382
72432897,11sleep30-21swapper/319:54:333
70822899,66sleep27083-21192.168.115.60-21:37:482
435928920,46sleep24378-21apt-get21:29:202
2476628914,59sleep024888-21timerwakeupswit20:49:410
62652887,13sleep30-21swapper/323:18:023
241122887,43sleep30-21swapper/320:49:243
2230528816,59sleep222314-21forks00:09:292
2230528816,59sleep222314-21forks00:09:292
163882888,11sleep30-21swapper/323:49:353
623728718,46sleep16389-21head19:50:271
248212877,11sleep30-21swapper/300:16:203
2434028720,45sleep024421-21switchtime00:14:400
2069928719,50sleep220760-21modprobe20:39:092
2069928719,50sleep220760-21modprobe20:39:092
1992528713,60sleep119999-21mailstats20:34:401
1992528713,60sleep119999-21mailstats20:34:401
1596128717,46sleep115962-21/usr/sbin/munin23:49:211
121302876,11sleep30-21swapper/320:09:393
1060028725,39sleep010596-21apt-get21:49:210
30132855,11sleep30-21swapper/319:39:473
2146228528,33sleep00-21swapper/022:24:090
2146228528,33sleep00-21swapper/022:24:090
1683228525,27sleep20-21swapper/222:09:252
2460828415,53sleep224676-21mailstats20:49:392
2369928414,52sleep123740-21ntpq22:29:351
2043328415,21sleep10-21swapper/100:04:111
1210828414,54sleep112120-21ntp_states20:09:371
307782836,10sleep30-21swapper/322:54:043
235002835,12sleep30-21swapper/320:45:043
1443828320,31sleep20-21swapper/223:44:262
2693028223,36sleep026940-21modprobe20:59:100
3134028127,29sleep10-21swapper/121:09:411
3054828115,41sleep129977-21/usr/sbin/munin00:34:401
3054828115,41sleep129977-21/usr/sbin/munin00:34:401
2788728111,14sleep20-21swapper/219:18:062
2334328026,26sleep10-21swapper/119:04:291
220522790,7sleep10-21swapper/100:09:111
220522790,7sleep10-21swapper/100:09:111
169862794,9sleep00-21swapper/023:50:540
3028027823,39sleep129855-21/usr/sbin/munin19:24:351
2854127821,33sleep20-21swapper/200:29:282
28302788,15sleep10-21swapper/121:24:231
883027715,48sleep18882-21latency19:59:311
728227516,45sleep17356-21timerandwakeup23:19:421
613927527,27sleep06235-21sed19:50:260
392527512,46sleep24011-21memory19:44:352
2643527512,46sleep126408-21/usr/sbin/munin22:39:241
3030027433,23sleep230327-21modprobe21:09:092
2959327416,38sleep029609-21ls19:24:090
159882747,17sleep20-21swapper/223:49:262
1488927423,27sleep20-21swapper/220:19:272
146527421,40sleep00-21swapper/019:35:080
2816727312,18sleep00-21swapper/000:29:090
1920627314,43sleep119358-21sed20:34:121
1690527219,31sleep00-21swapper/022:09:270
947927114,41sleep0225-21systemd-journal23:29:100
503827114,26sleep00-21swapper/019:49:130
3153827114,26sleep20-21swapper/222:54:392
3153827114,26sleep20-21swapper/222:54:392
2195727115,16sleep10-21swapper/122:24:301
2195727115,16sleep10-21swapper/122:24:301
176022715,12sleep20-21swapper/220:29:082
312027016,39sleep13259-21latency_hist23:09:121
2892327022,20sleep00-21swapper/000:29:380
2667027022,26sleep20-21swapper/200:24:112
2667027022,26sleep20-21swapper/200:24:112
2523827021,25sleep20-21swapper/222:34:372
2354527016,23sleep00-21swapper/000:14:110
2354527016,23sleep00-21swapper/000:14:110
2279327020,28sleep10-21swapper/100:09:401
2279327020,28sleep10-21swapper/100:09:401
2264827020,20sleep10-21swapper/120:44:291
906326924,23sleep10-21swapper/121:44:291
3121626918,33sleep00-21swapper/021:09:400
727226815,36sleep17420-21latency_hist21:39:121
407526812,42sleep0363-21snmpd21:28:580
3182426812,31sleep10-21swapper/119:29:361
2951826819,29sleep029578-21find21:04:390
1886926818,28sleep20-21swapper/223:59:112
3140626720,27sleep031425-21apt-get19:29:220
3078626721,29sleep030795-21modprobe22:54:090
2880426720,21sleep10-21swapper/121:04:121
2812726714,26sleep20-21swapper/222:44:502
244622677,17sleep00-21swapper/022:32:530
2128626716,21sleep10-21swapper/120:39:331
3262626617,27sleep00-21swapper/022:59:220
3262626617,27sleep00-21swapper/022:59:220
236322666,12sleep00-21swapper/020:46:420
798526514,24sleep00-21swapper/021:39:400
238512657,12sleep10-21swapper/100:14:291
1522126516,20sleep10-21swapper/123:44:451
1427326514,23sleep20-21swapper/221:59:402
115792658,14sleep00-21swapper/023:34:310
2844726423,26sleep10-21swapper/119:19:301
2475226412,37sleep124783-21grep19:09:251
1928326419,28sleep219338-21/usr/sbin/munin23:59:322
1896526421,27sleep20-21swapper/222:15:312
1896526421,27sleep20-21swapper/222:15:312
2192926317,25sleep221936-21modprobe00:09:092
2192926317,25sleep221936-21modprobe00:09:092
91322629,17sleep20-21swapper/223:24:432
859726220,25sleep10-21swapper/123:24:341
306126211,35sleep03119-21latency_hist23:09:110
306126211,35sleep03119-21latency_hist23:09:110
3018526211,35sleep029977-21/usr/sbin/munin00:34:300
3018526211,35sleep029977-21/usr/sbin/munin00:34:300
2737126213,25sleep10-21swapper/100:24:381
2533726213,24sleep10-21swapper/100:19:211
2533726213,24sleep10-21swapper/100:19:211
2532226221,23sleep10-21swapper/122:34:401
229626224,20sleep10-21swapper/123:04:401
229626224,20sleep10-21swapper/123:04:401
2175226217,29sleep021815-21/usr/sbin/munin22:24:250
2175226217,29sleep021815-21/usr/sbin/munin22:24:250
1835826219,19sleep00-21swapper/020:29:390
1376726210,38sleep213791-21users23:39:412
1375526224,19sleep20-21swapper/220:14:392
2752126116,23sleep00-21swapper/020:59:510
248902619,33sleep125022-21/usr/sbin/munin20:49:421
832326015,24sleep20-21swapper/219:59:122
676526024,21sleep20-21swapper/223:19:312
3108526013,30sleep231147-21sendmail_mailst21:09:382
302732608,25sleep217420-1kworker/2:1H00:34:322
302732608,25sleep217420-1kworker/2:1H00:34:322
1906526016,28sleep019111-21/usr/sbin/munin22:15:330
1906526016,28sleep019111-21/usr/sbin/munin22:15:330
3047925910,32sleep230557-21find19:24:402
2881925915,17sleep20-21swapper/219:19:392
2361825916,18sleep30-21swapper/319:04:353
1261325914,28sleep012645-21/usr/sbin/munin21:54:380
1261325914,28sleep012645-21/usr/sbin/munin21:54:380
43712584,12sleep20-21swapper/223:11:432
2805625812,21sleep00-21swapper/019:19:130
2155225820,19sleep00-21swapper/020:39:400
193892589,23sleep00-21swapper/023:59:340
864525720,21sleep20-21swapper/219:59:282
74525717,21sleep2801-21irqstats19:34:412
3162625717,23sleep031691-21/usr/sbin/munin22:54:410
3162625717,23sleep031691-21/usr/sbin/munin22:54:410
258825711,33sleep22620-21date21:24:112
1144125716,23sleep211519-21perl23:34:282
598125618,18sleep20-21swapper/219:50:242
32082567,39sleep10-21swapper/119:42:151
2804225614,25sleep228100-21/usr/sbin/munin21:00:332
2315625614,25sleep223152-21apt-get19:04:222
2151225623,19sleep20-21swapper/222:24:112
2151225623,19sleep20-21swapper/222:24:112
1879025614,20sleep10-21swapper/123:59:101
1844925612,15sleep20-21swapper/220:29:412
162182565,38sleep00-21swapper/020:24:130
150952568,23sleep00-21swapper/023:44:440
128425614,17sleep00-21swapper/021:19:250
1018925613,25sleep010195-21df_inode20:04:270
90952557,35sleep00-21swapper/019:59:390
77425513,29sleep0363-21snmpd21:16:430
72725511,18sleep20-21swapper/222:59:402
72725511,18sleep20-21swapper/222:59:402
493725515,19sleep00-21swapper/021:29:390
379825516,20sleep00-21swapper/023:09:390
1528225512,25sleep215276-21apt-get22:04:252
1479325515,24sleep014788-21apt-get20:19:210
1235625515,19sleep10-21swapper/121:54:311
1235625515,19sleep10-21swapper/121:54:311
488525416,17sleep10-21swapper/123:14:131
2822025417,16sleep20-21swapper/200:29:112
2637525414,26sleep026377-21sshd00:21:020
2637525414,26sleep026377-21sshd00:21:020
253125417,23sleep12533-21sshd21:24:021
245402547,13sleep20-21swapper/222:33:522
1084425414,25sleep110872-21irqstats21:49:301
749925314,16sleep10-21swapper/121:39:211
721025323,16sleep00-21swapper/023:19:410
1710025315,16sleep10-21swapper/122:09:321
1106125314,24sleep011075-21munin-run23:34:100
1011525311,25sleep110109-21apt-get20:04:241
2896225216,17sleep10-21swapper/100:29:391
2806025212,24sleep027958-21/usr/sbin/munin22:44:280
2723925210,24sleep127310-21sensors_temp19:14:421
2041325211,27sleep120470-21awk22:19:301
2041325211,27sleep120470-21awk22:19:301
1924525217,17sleep10-21swapper/122:15:381
1924525217,17sleep10-21swapper/122:15:381
69612516,17sleep10-21swapper/119:54:231
3242225118,16sleep20-21swapper/221:14:352
3200525111,17sleep20-21swapper/219:29:412
1658725113,22sleep116612-21irqstats20:24:311
1578125113,25sleep015862-21run-parts22:04:390
104525112,22sleep11098-21users22:59:431
104525112,22sleep11098-21users22:59:431
2567025012,22sleep225666-21apt-get20:54:242
1332225012,19sleep20-21swapper/221:58:032
1332225012,19sleep20-21swapper/221:58:032
997724915,19sleep210030-21interrupts23:29:322
217724911,21sleep22184-21diskstats19:39:302
364424811,19sleep10-21swapper/119:44:251
560224711,15sleep20-21swapper/223:14:462
2182724712,19sleep221833-21diskstats22:24:272
2182724712,19sleep221833-21diskstats22:24:272
1524824711,20sleep115255-21ntp_states20:19:341
282792467,15sleep10-21swapper/122:44:531
1166424510,17sleep00-21swapper/020:09:240
1038824512,15sleep20-21swapper/220:04:332
2911623910,15sleep10-21swapper/121:04:281
225572389,14sleep20-21swapper/220:44:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional