You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 18:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Thu Sep 03, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13702265127,6493sleep37828-21kworker/3:109:28:093
32702218726,2141sleep33305-21install07:13:393
5162205211,2024sleep032684-1kworker/0:0H10:28:550
11430259310,534sleep311462-21latency_hist11:03:393
17716258716,556sleep117717-21sshd11:21:091
2446825787,561sleep3367-21snmpd10:01:163
9008249430,180sleep39160-21unixbench_singl12:34:123
17901247113,444sleep017948-21unin-run07:58:380
2071724126,395sleep3367-21snmpd08:04:563
4939238843,328sleep35017-21mailstats08:59:063
4939238843,328sleep35017-21mailstats08:59:063
1755023317,289sleep317586-21ntpq09:39:033
245202310230,19sleep30-21swapper/308:19:013
5920228395,58sleep30-21swapper/307:19:213
2816422517,206sleep328289-21grep11:54:063
1080224847,141sleep31147-21switchtime10:29:143
6309221318,139sleep3231-21systemd-journal12:27:093
883422067,162sleep38889-21cat10:54:183
883422067,162sleep38889-21cat10:54:183
17870219654,127sleep317889-21modprobe07:58:373
15136219210,167sleep215144-21df07:48:552
6959219111,164sleep07016-21awk10:48:580
708721899,167sleep27117-21awk12:29:062
27710218972,57sleep30-21swapper/308:28:583
21692218910,163sleep121735-21ntpq08:09:011
168352179141,19sleep30-21swapper/311:19:463
3832217015,141sleep13840-21ntpq12:19:031
2095421697,150sleep2367-21snmpd11:32:572
24565216610,23sleep30-21swapper/311:43:563
1689216551,101sleep31743-21ntpq08:49:053
16920215818,25sleep00-21swapper/009:38:400
25142215623,110sleep225284-21unixbench_singl11:44:072
15450215326,105sleep215582-21unixbench_multi11:14:072
2958215212,23sleep33052-21fschecks_time08:53:573
30198215016,23sleep20-21swapper/210:19:082
25333214810,12sleep10-21swapper/111:44:111
21899214847,52sleep30-21swapper/311:34:073
2284721467,57sleep30-21swapper/308:13:563
21513214411,76sleep321529-21modprobe09:53:373
21513214411,76sleep321529-21modprobe09:53:373
3920214230,56sleep33994-21cat10:38:593
641021418,95sleep30-21swapper/309:04:073
641021418,95sleep30-21swapper/309:04:073
18792214118,81sleep30-21swapper/307:59:103
1482821409,92sleep314961-21latency_hist07:48:403
109221399,115sleep31240-21date08:48:403
1039121397,38sleep30-21swapper/307:33:543
7795213721,76sleep30-21swapper/309:08:593
16111213230,45sleep316184-21sensors09:34:073
1599921307,44sleep3367-21snmpd07:49:293
11940213013,27sleep10-21swapper/107:38:491
4634212752,30sleep34667-21awk12:23:383
1772212612,52sleep31775-21apt-get12:13:483
26063212510,54sleep326157-21cut08:23:573
21056212522,43sleep30-21swapper/311:33:393
14498212416,94sleep11-21systemd11:13:381
26962212015,91sleep227007-21processes10:09:112
9235211711,89sleep09271-21/usr/sbin/munin07:29:030
30650211616,84sleep130643-21apt-get08:38:531
2250421158,48sleep322618-21unixbench_multi09:54:103
2250421158,48sleep322618-21unixbench_multi09:54:103
5283211350,46sleep30-21swapper/307:18:403
3117021136,14sleep20-21swapper/210:23:522
3117021136,14sleep20-21swapper/210:23:522
29159211220,26sleep10-21swapper/108:33:571
14484211220,75sleep014494-21modprobe11:13:370
28856211117,33sleep10-21swapper/111:58:401
1867921107,46sleep318698-21apt-get09:43:483
20426210918,71sleep120569-21unixbench_multi08:04:091
1403121088,39sleep30-21swapper/309:28:543
22603210718,72sleep022692-21latency_hist08:13:410
5282210619,68sleep05374-21tune2fs12:24:190
15776210518,13sleep30-21swapper/311:15:183
711821049,79sleep37155-21grep12:29:073
737221036,37sleep30-21swapper/307:23:573
287921029,12sleep20-21swapper/207:09:142
26891210248,18sleep326936-21ntpq10:09:083
7965210147,18sleep3367-21snmpd10:53:053
4608210116,67sleep04736-21unixbench_multi07:14:130
14148210149,14sleep30-21swapper/311:09:203
5154210012,73sleep24648-21apt-get12:23:492
3164421006,40sleep3367-21snmpd08:40:133
2703210016,68sleep12826-21cat10:34:091
2946429945,16sleep329509-21awk11:59:023
570129812,69sleep35177-21/usr/sbin/munin10:44:083
2824629816,18sleep30-21swapper/310:14:003
3124729713,70sleep031254-21/usr/sbin/munin10:23:540
3124729713,70sleep031254-21/usr/sbin/munin10:23:540
302892975,38sleep30-21swapper/308:38:073
249342977,31sleep30-21swapper/310:03:513
1887429726,45sleep00-21swapper/007:59:110
2361429618,59sleep023757-21grep11:39:090
234862969,72sleep323611-21cat11:39:083
1951229621,59sleep119521-21wc11:28:401
1951229621,59sleep119521-21wc11:28:401
1919529615,66sleep219279-21grep09:44:072
815629518,59sleep28352-21latency_hist10:53:402
321742957,16sleep00-21swapper/008:43:490
1277629518,63sleep112817-21ls09:24:161
1137829521,52sleep011527-21grep09:19:100
614429410,66sleep06168-21/usr/sbin/munin07:19:260
259782937,13sleep30-21swapper/311:48:573
1230429236,42sleep312435-21timerwakeupswit11:04:073
647929112,62sleep26539-21sendmail_mailqu09:04:102
647929112,62sleep26539-21sendmail_mailqu09:04:102
2938629110,62sleep229479-21latency_hist10:18:402
1164629114,51sleep00-21swapper/011:03:480
227529016,53sleep02309-21grep12:14:220
99822888,13sleep10-21swapper/107:32:111
1688028816,59sleep116948-21latency07:53:591
1449928824,45sleep2374-21sshd09:29:092
2021328710,63sleep220251-21/usr/sbin/munin08:04:062
881628526,16sleep30-21swapper/307:28:493
324162854,52sleep318867-1kworker/3:1H12:08:523
3064828521,49sleep30-21swapper/312:03:513
3011828517,41sleep129599-21/usr/sbin/munin10:19:071
1069228522,17sleep30-21swapper/309:18:523
105728419,51sleep21091-21latency_hist08:48:392
2246928318,10sleep30-21swapper/308:12:353
20682839,13sleep30-21swapper/310:33:513
1913928318,10sleep30-21swapper/311:24:303
1913928318,10sleep30-21swapper/311:24:303
1046828316,50sleep110474-21diskstats07:33:561
83628220,43sleep2859-21/usr/sbin/munin07:04:062
2538428213,54sleep025395-21ntpq10:04:040
1725428216,50sleep217357-21cut09:38:582
1172828122,34sleep00-21swapper/007:38:400
94752800,5sleep20-21swapper/207:29:082
420028016,50sleep14256-21latency_hist08:58:401
3156728018,29sleep30-21swapper/310:24:013
3156728018,29sleep30-21swapper/310:24:013
2053028021,20sleep20-21swapper/209:49:042
2053028021,20sleep20-21swapper/209:49:042
1036228018,10sleep30-21swapper/309:18:283
298962796,14sleep30-21swapper/310:19:013
129632785,23sleep30-21swapper/307:40:163
921927721,34sleep29235-21sshd12:34:392
736427610,52sleep27371-21sh07:23:562
651227614,47sleep26588-21sed10:48:392
1065127614,46sleep110722-21sensors10:59:081
290522757,55sleep00-21swapper/010:15:260
1574527521,32sleep015873-21/usr/sbin/munin07:49:100
1684027414,40sleep116874-21modprobe09:38:371
1360027416,45sleep013613-21fschecks_count07:43:570
977027330,34sleep032684-1kworker/0:0H10:58:150
977027330,34sleep032684-1kworker/0:0H10:58:150
797227319,35sleep17992-21modprobe12:33:371
3043127316,29sleep00-21swapper/008:38:400
2306327324,33sleep223065-21cron09:58:372
2306327324,33sleep223065-21cron09:58:372
1780427321,38sleep117917-21timerwakeupswit09:39:101
128727322,34sleep11319-21munin-run12:13:381
775527218,35sleep17793-21grep07:24:071
2877427218,31sleep228803-21munin-run11:58:372
2729727225,17sleep20-21swapper/211:53:402
2470027216,33sleep024823-21latency_hist10:03:390
2048427216,40sleep120617-21awk11:29:131
1874727213,43sleep018825-21cat11:24:070
1874727213,43sleep018825-21cat11:24:070
947827119,20sleep00-21swapper/012:34:570
661727124,25sleep00-21swapper/009:04:120
661727124,25sleep00-21swapper/009:04:120
2540327114,43sleep225457-21processes10:04:062
2493727123,27sleep10-21swapper/110:03:531
243912717,52sleep00-21swapper/011:43:450
2295027116,29sleep20-21swapper/211:38:552
1453527125,27sleep20-21swapper/211:13:392
425727011,43sleep24400-21modprobe08:58:412
425727011,43sleep24400-21modprobe08:58:412
1037527012,34sleep110408-21munin-run09:18:381
698126916,39sleep06998-21cut12:29:040
2936526920,29sleep10-21swapper/111:59:001
2883426919,22sleep10-21swapper/108:33:401
861426819,25sleep10-21swapper/110:54:121
861426819,25sleep10-21swapper/110:54:121
3162026810,44sleep131621-21sendmail-mta12:04:541
2637726820,22sleep10-21swapper/108:24:031
2306626813,37sleep123126-21if_eth011:38:581
3191126716,28sleep20-21swapper/212:08:382
2909626723,25sleep20-21swapper/211:58:532
277712670,7sleep00-21swapper/008:28:580
2624526721,28sleep10-21swapper/111:49:031
1522326713,39sleep115230-21ntp_states11:14:021
1204826715,35sleep112098-21awk11:04:011
519026621,29sleep15203-21modprobe07:18:381
454726616,26sleep10-21swapper/110:39:101
331826616,30sleep03358-21munin-node07:13:400
326882669,12sleep10-21swapper/108:44:061
490826524,22sleep20-21swapper/212:23:412
2723326511,36sleep027280-21systemd-run08:28:390
2111526516,28sleep00-21swapper/011:33:400
127126513,22sleep10-21swapper/110:29:161
111722659,47sleep00-21swapper/011:00:530
804626418,21sleep20-21swapper/209:09:062
231426415,32sleep02343-21/usr/sbin/munin10:34:000
193326421,27sleep10-21swapper/108:49:111
178126416,32sleep21783-21cron10:33:372
1183726418,25sleep10-21swapper/109:22:091
1164826416,28sleep211704-21munin-run07:38:382
316672635,11sleep00-21swapper/012:05:290
855326216,21sleep00-21swapper/007:28:380
467126216,24sleep00-21swapper/012:23:390
274132625,13sleep00-21swapper/010:09:170
262026219,22sleep20-21swapper/208:53:402
1952226218,29sleep219661-21latency_hist11:28:412
1952226218,29sleep219661-21latency_hist11:28:412
1391226212,32sleep213954-21ntpq07:44:032
1222326215,18sleep20-21swapper/211:04:062
64426121,27sleep1724-21cat07:04:011
281526116,31sleep2231-21systemd-journal12:15:372
2652426110,37sleep226592-21switchtime08:24:092
1859226122,24sleep20-21swapper/207:59:052
887126012,26sleep00-21swapper/009:13:410
3162326016,26sleep10-21swapper/110:24:031
3162326016,26sleep10-21swapper/110:24:031
2999626017,27sleep030008-21vmstat11:59:110
2228126018,24sleep00-21swapper/009:54:070
2228126018,24sleep00-21swapper/009:54:070
1693326019,27sleep016983-21if_eth011:19:550
1388226014,19sleep10-21swapper/107:44:021
1259326017,20sleep20-21swapper/209:24:122
927825914,26sleep19270-21apt-get12:34:501
439325910,28sleep04542-21unixbench_singl10:39:090
3267325910,31sleep232714-21uname12:09:032
2718025916,19sleep10-21swapper/110:09:141
1974625914,26sleep019740-21apt-get08:03:510
1208225915,20sleep20-21swapper/209:23:402
103062597,39sleep00-21swapper/009:17:440
2311425814,17sleep10-21swapper/108:14:021
2176925812,18sleep10-21swapper/111:34:051
705425719,16sleep20-21swapper/207:23:402
646125718,21sleep00-21swapper/010:48:370
296162577,13sleep00-21swapper/010:18:500
2789125714,24sleep20-21swapper/208:29:012
245052577,11sleep10-21swapper/108:18:591
2129425716,24sleep221288-21apt-get08:08:502
1907625716,21sleep10-21swapper/109:44:021
1430825716,25sleep114400-21load09:29:041
1204525711,30sleep012051-21diskstats07:38:560
41342566,12sleep20-21swapper/207:14:032
2338025613,17sleep10-21swapper/109:58:521
2036725611,28sleep020483-21sed11:29:120
177752567,15sleep20-21swapper/211:21:552
1715725614,25sleep017152-21apt-get09:38:520
1594925612,18sleep20-21swapper/209:34:022
1038625615,17sleep20-21swapper/210:59:012
443725512,25sleep14431-21apt-get08:58:521
443725512,25sleep14431-21apt-get08:58:521
3123525512,17sleep00-21swapper/008:39:090
2568425512,20sleep00-21swapper/008:23:380
1206425515,19sleep20-21swapper/207:38:582
231125414,20sleep10-21swapper/112:14:231
1950825415,17sleep10-21swapper/108:03:401
672325313,24sleep16859-21unixbench_singl09:04:131
672325313,24sleep16859-21unixbench_singl09:04:131
36025312,23sleep3358-21apt-get07:03:513
246482538,17sleep20-21swapper/210:03:372
2281525314,21sleep00-21swapper/008:13:490
2111625313,22sleep121168-21users09:49:151
2111625313,22sleep121168-21users09:49:151
979825211,16sleep20-21swapper/210:58:372
979825211,16sleep20-21swapper/210:58:372
674825212,23sleep16742-21apt-get10:48:511
487125213,24sleep04894-21ntp_states08:59:030
487125213,24sleep04894-21ntp_states08:59:030
44425216,19sleep00-21swapper/007:03:560
2180325213,23sleep121798-21apt-get09:53:511
2180325213,23sleep121798-21apt-get09:53:511
1898125213,15sleep00-21swapper/009:44:010
696425113,18sleep10-21swapper/107:23:101
586825117,17sleep10-21swapper/110:44:101
46942515,35sleep20-21swapper/210:40:482
3145625111,23sleep231554-21users08:39:112
285825111,23sleep22852-21apt-get08:53:522
2025825113,22sleep020254-21apt-get09:48:540
2025825113,22sleep020254-21apt-get09:48:540
1248825112,23sleep012470-21apt-get09:23:580
516225013,22sleep15151-21apt-get12:23:521
3253125014,15sleep00-21swapper/012:09:010
291912506,30sleep10-21swapper/110:17:091
176482506,33sleep20-21swapper/207:55:462
1446725011,15sleep00-21swapper/009:29:070
934824910,15sleep20-21swapper/209:13:592
764624913,16sleep10-21swapper/109:08:571
2906524912,21sleep229063-21apt-get08:33:502
2293124911,18sleep20-21swapper/208:13:592
161012495,35sleep10-21swapper/107:50:481
130424912,24sleep01306-21cpu08:48:540
400724815,18sleep00-21swapper/008:57:090
2829524812,18sleep00-21swapper/011:54:070
2605024810,23sleep026056-21df_inode11:48:580
20732475,16sleep20-21swapper/210:33:542
238824611,19sleep10-21swapper/107:09:051
2443324511,17sleep20-21swapper/208:18:562
2129124411,18sleep221287-21apt-get11:33:502
153042429,13sleep00-21swapper/011:14:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional