You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-19 - 14:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed Aug 19, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13851268221,644sleep113882-21/usr/sbin/munin20:54:591
15283259647,534sleep315289-21diskstats00:24:523
23227259211,569sleep223263-21ntpq21:25:002
423523857,346sleep24243-21df20:24:522
423523857,346sleep24243-21df20:24:522
9343234412,318sleep39382-21ntpq00:05:023
311962295132,30sleep20-21swapper/221:50:032
259302290107,53sleep225971-21proc_pri19:50:032
2478222868,267sleep0367-21snmpd23:10:560
2478222868,267sleep0367-21snmpd23:10:560
15565228017,244sleep115659-21grep21:00:031
260312277119,29sleep20-21swapper/221:34:532
260312277119,29sleep20-21swapper/221:34:532
2117222747,217sleep221237-21awk23:00:042
16514225811,102sleep216550-21latency_hist00:29:352
4688225044,191sleep24735-21ntpq23:49:592
5183224111,204sleep25246-21latency22:10:182
2638224024,193sleep12760-21latency_hist22:04:351
2638224024,193sleep12760-21latency_hist22:04:351
26624221721,176sleep126694-21true23:19:331
13250220515,73sleep213404-21latency_hist20:54:352
18675220010,176sleep318719-21ntpq00:34:573
22346219613,165sleep222413-21fschecks_time23:04:522
9339218612,158sleep29448-21latency_hist22:24:352
9339218612,158sleep29448-21latency_hist22:24:352
15720218214,19sleep20-21swapper/222:44:172
4355217812,152sleep24389-21idleruntime-cro22:09:332
16701217818,140sleep20-21swapper/221:04:532
18297217531,126sleep218349-21cpuspeed00:34:482
1431721668,24sleep20-21swapper/220:55:082
9773216510,131sleep02412-1kworker/0:1H22:24:450
2743216211,135sleep22744-21cpu23:44:502
27025216119,18sleep20-21swapper/223:19:502
155072156116,23sleep20-21swapper/221:00:002
23401215314,124sleep323464-21swap21:25:053
20242214721,11sleep20-21swapper/222:57:422
1853214725,21sleep20-21swapper/223:40:462
22620214612,29sleep222709-21latency_hist21:24:352
9061214515,18sleep20-21swapper/200:04:562
3011214416,108sleep10-21swapper/123:44:571
2903221447,15sleep20-21swapper/223:25:202
5397214313,116sleep05459-21sendmail_mailst22:10:240
27575214221,81sleep20-21swapper/221:39:492
3124221415,11sleep20-21swapper/223:30:492
1817721388,95sleep218152-21/usr/sbin/munin21:09:492
11113213810,89sleep211253-21latency_hist22:29:362
3179921379,107sleep20-21swapper/223:34:442
2513721368,117sleep2367-21snmpd19:49:022
6518213314,99sleep16549-21tune2fs22:14:531
31643213116,19sleep20-21swapper/220:09:332
2751921310,10sleep30-21swapper/319:55:043
11473213036,72sleep311629-21grep19:05:033
818521297,15sleep20-21swapper/200:00:082
4294212915,88sleep14141-21/usr/sbin/munin20:24:551
4294212915,88sleep14141-21/usr/sbin/munin20:24:551
2713921299,104sleep227223-21perl19:54:562
7232212511,99sleep37246-21acpi20:34:463
643421247,73sleep20-21swapper/222:14:502
2422221248,59sleep224135-21sh21:29:362
2012021248,19sleep220164-21awk21:14:582
1174221245,11sleep20-21swapper/200:14:092
2572921237,103sleep225772-21ntpq23:14:582
1697921235,46sleep20-21swapper/222:45:352
1262621237,103sleep212615-21apt-get19:09:502
9751212225,61sleep20-21swapper/220:40:142
9751212225,61sleep20-21swapper/220:40:142
14044212220,73sleep20-21swapper/222:37:592
571121218,17sleep20-21swapper/220:29:482
3193321217,16sleep20-21swapper/220:09:502
1045921217,14sleep20-21swapper/220:44:542
1045921217,14sleep20-21swapper/220:44:542
2407721209,96sleep20-21swapper/223:09:572
2407721209,96sleep20-21swapper/223:09:572
18290212010,95sleep018373-21find19:25:020
97621197,81sleep2367-21snmpd21:59:282
1336721188,99sleep213417-21munin-run00:19:342
733521176,13sleep20-21swapper/220:34:532
28699211713,88sleep328213-21apt-get23:24:443
2186921179,69sleep20-21swapper/219:37:112
2186921179,69sleep20-21swapper/219:37:112
2859021167,52sleep20-21swapper/219:59:482
25178211521,79sleep025200-21awk19:49:330
246421156,97sleep20-21swapper/222:03:152
246421156,97sleep20-21swapper/222:03:152
17906211516,84sleep317962-21interrupts19:24:533
367821145,74sleep20-21swapper/220:22:012
367821145,74sleep20-21swapper/220:22:012
29160211425,72sleep20-21swapper/221:44:542
1470521136,96sleep00-21swapper/020:59:050
1633621127,45sleep20-21swapper/219:19:552
7178211149,47sleep27254-21latency_hist23:59:352
1455621118,45sleep20-21swapper/219:14:462
12543211120,39sleep20-21swapper/222:33:352
20798211013,78sleep320865-21/usr/sbin/munin19:34:503
20798211013,78sleep320865-21/usr/sbin/munin19:34:503
1947021106,46sleep20-21swapper/219:29:572
1947021106,46sleep20-21swapper/219:29:572
13989211022,67sleep30-21swapper/320:55:053
13936211021,68sleep013983-21latency_hist19:14:360
1815821087,63sleep20-21swapper/219:25:002
20994210722,63sleep121027-21idleruntime-cro21:19:331
1386621076,15sleep10-21swapper/100:19:531
30627210614,76sleep030690-21/usr/sbin/munin21:49:490
198821055,42sleep20-21swapper/220:15:432
198821055,42sleep20-21swapper/220:15:432
17579210518,44sleep00-21swapper/022:49:480
23975210416,69sleep10-21swapper/119:44:541
18652210416,69sleep318714-21wc21:10:033
1425821046,44sleep20-21swapper/200:20:032
9199210312,15sleep30-21swapper/320:39:593
9199210312,15sleep30-21swapper/320:39:593
757221035,88sleep00-21swapper/022:17:550
1794421025,42sleep20-21swapper/200:33:432
32604210013,72sleep032726-21users23:35:070
27693210012,67sleep127819-21/usr/sbin/munin19:55:071
2861029911,73sleep328611-21sshd21:42:173
175452996,40sleep20-21swapper/222:49:382
3067029713,64sleep00-21swapper/020:05:240
123812977,15sleep30-21swapper/320:50:003
3027629629,42sleep130273-21apt-get23:29:471
1566229614,67sleep015670-21ntp_states00:25:010
1232429620,59sleep112502-21apt-config19:09:371
68192957,75sleep00-21swapper/020:32:370
502629521,49sleep35030-21fschecks_count22:10:113
1981029416,26sleep00-21swapper/021:14:520
416629325,33sleep00-21swapper/020:24:500
416629325,33sleep00-21swapper/020:24:500
2672629217,57sleep026753-21/usr/sbin/munin19:54:330
1670429116,55sleep30-21swapper/319:20:043
1674928918,33sleep30-21swapper/321:04:543
620328814,57sleep16249-21/usr/sbin/munin23:54:581
2196728828,36sleep30-21swapper/323:04:333
189928715,58sleep31977-21find22:00:053
189928715,58sleep31977-21find22:00:053
2546928541,29sleep125462-21apt-get19:49:501
2459228518,25sleep30-21swapper/319:45:083
2131628515,57sleep021320-21grep21:19:530
579728420,49sleep35865-21fschecks_time20:29:533
1972728415,50sleep119791-21find19:30:041
1972728415,50sleep119791-21find19:30:041
2672828219,42sleep326642-21latency_hist23:19:353
1230328214,32sleep10-21swapper/120:49:581
1100628232,28sleep10-21swapper/100:10:041
629428013,49sleep00-21swapper/023:55:010
3070928015,48sleep330739-21fschecks_count21:49:523
1483728014,44sleep114459-21/usr/sbin/munin22:39:561
223382795,42sleep20-21swapper/219:39:382
1182627819,30sleep00-21swapper/000:14:350
2728427722,40sleep027302-21latency_hist21:39:350
2728427722,40sleep027302-21latency_hist21:39:350
2054727716,46sleep0367-21snmpd19:34:350
2054727716,46sleep0367-21snmpd19:34:350
193182776,61sleep00-21swapper/021:13:020
1885227716,48sleep018928-21switchtime00:35:030
1163627717,18sleep10-21swapper/120:49:341
1163627717,18sleep10-21swapper/120:49:341
195627621,34sleep32105-21sed23:40:473
138727617,19sleep00-21swapper/021:59:540
138727617,19sleep00-21swapper/021:59:540
1027727621,29sleep00-21swapper/000:09:350
3240127515,16sleep00-21swapper/020:10:040
1900627522,23sleep10-21swapper/119:29:351
2202227418,31sleep10-21swapper/123:04:351
219832740,7sleep00-21swapper/023:04:330
1981027417,24sleep30-21swapper/322:55:123
161527415,36sleep01650-21ntp_kernel_err23:40:400
1583427426,31sleep115886-21munin-run19:19:341
335727320,34sleep03511-21sed20:20:050
335727320,34sleep03511-21sed20:20:050
2447727313,45sleep124483-21df_inode21:29:511
1793827320,21sleep00-21swapper/021:09:350
1031827317,41sleep110470-21cat00:09:361
2715027216,41sleep027206-21/usr/sbin/munin23:19:520
1334127218,35sleep313490-21sed19:10:113
2706227119,30sleep326916-21/usr/sbin/munin23:19:513
1831827122,33sleep118465-21unixbench_singl22:50:061
1640027125,23sleep30-21swapper/322:44:593
1135127119,23sleep30-21swapper/322:29:493
542127024,26sleep35448-21munin-run20:29:333
542127024,26sleep35448-21munin-run20:29:333
2046427015,40sleep120504-21cut22:59:341
944926916,22sleep30-21swapper/322:24:363
944926916,22sleep30-21swapper/322:24:363
3188526920,24sleep10-21swapper/121:54:341
2956526916,35sleep329627-21/usr/sbin/munin21:45:073
261926915,37sleep12689-21df20:19:501
261926915,37sleep12689-21df20:19:501
1217826917,32sleep012322-21latency_hist19:09:360
1205626920,29sleep212090-21munin-run19:09:332
996226813,25sleep30-21swapper/322:24:563
2923726815,35sleep029296-21/usr/sbin/munin23:25:260
1949426821,24sleep10-21swapper/121:14:371
1477926814,23sleep30-21swapper/319:14:563
2562626714,28sleep00-21swapper/023:14:560
1603726722,23sleep10-21swapper/100:25:061
314372664,12sleep10-21swapper/120:06:541
3033626618,21sleep10-21swapper/121:49:331
2884426620,27sleep00-21swapper/021:44:360
2730426618,25sleep10-21swapper/121:39:361
2730426618,25sleep10-21swapper/121:39:361
2294526611,25sleep10-21swapper/121:24:541
1624426621,23sleep10-21swapper/119:19:541
1333926621,28sleep113366-21munin-run00:19:331
750326521,20sleep00-21swapper/023:59:540
378626521,21sleep30-21swapper/322:05:143
2876026522,22sleep10-21swapper/123:25:101
2426826515,24sleep30-21swapper/323:10:023
2426826515,24sleep30-21swapper/323:10:023
1573926522,20sleep30-21swapper/321:00:053
1433626524,17sleep30-21swapper/300:20:043
1190126518,31sleep00-21swapper/020:49:450
8032646,17sleep00-21swapper/021:57:160
3226026412,23sleep10-21swapper/121:54:541
2868326418,19sleep00-21swapper/019:59:540
1491626419,29sleep014955-21modprobe00:24:330
1398526422,25sleep114149-21fw_packets19:14:371
611026314,32sleep16147-21ntp_kernel_pll_20:29:591
2998626320,18sleep10-21swapper/120:04:361
2663426314,30sleep326768-21unixbench_multi21:35:063
2663426314,30sleep326768-21unixbench_multi21:35:063
171326315,33sleep31865-21unixbench_multi20:15:083
171326315,33sleep31865-21unixbench_multi20:15:083
1600626317,17sleep10-21swapper/122:44:451
156052635,16sleep00-21swapper/022:42:460
1172026317,22sleep00-21swapper/022:29:580
324726218,26sleep00-21swapper/023:45:040
1740926114,26sleep017437-21munin-run19:24:340
69526016,19sleep30-21swapper/323:39:343
496726015,17sleep00-21swapper/023:50:050
2283326013,18sleep00-21swapper/021:24:450
211526025,21sleep10-21swapper/120:17:121
211526025,21sleep10-21swapper/120:17:121
693625910,24sleep30-21swapper/322:15:033
3271925917,24sleep3323-21/usr/sbin/munin21:55:053
309125911,20sleep00-21swapper/022:04:470
2709525915,27sleep027102-21diskstats19:54:540
2449125919,27sleep324504-21fschecks_count21:29:533
2357025915,18sleep10-21swapper/123:09:361
233542595,11sleep00-21swapper/023:07:250
1695325913,33sleep016954-21sshd22:45:160
1428025912,22sleep00-21swapper/019:14:400
1084325922,21sleep00-21swapper/020:45:040
1084325922,21sleep00-21swapper/020:45:040
94125812,24sleep10-21swapper/123:39:361
859625811,24sleep10-21swapper/120:39:361
805925812,26sleep18183-21sed00:00:071
78625821,20sleep30-21swapper/320:14:353
644725811,19sleep30-21swapper/323:55:043
59482584,10sleep10-21swapper/122:12:021
1755825812,29sleep317650-21users00:30:053
1244225813,21sleep00-21swapper/000:15:000
771925716,26sleep00-21swapper/020:35:030
455525720,20sleep30-21swapper/323:49:563
2908225716,17sleep00-21swapper/021:44:520
2878725714,28sleep128816-21munin-run21:44:331
1489825715,25sleep314935-21ntp_states22:39:583
883125615,24sleep08827-21apt-get20:39:480
883125615,24sleep08827-21apt-get20:39:480
544425611,30sleep15445-21sshd23:53:171
309872566,12sleep00-21swapper/023:30:060
2586725616,19sleep30-21swapper/319:50:003
229522569,32sleep323040-21cat19:40:073
1424725613,28sleep014338-21latency_hist22:39:350
1307225613,25sleep30-21swapper/300:16:163
1079225614,24sleep310885-21memory00:09:583
3193225512,26sleep131927-21apt-get20:09:491
303552559,22sleep30-21swapper/320:04:463
281672554,11sleep10-21swapper/123:24:091
2216825512,19sleep00-21swapper/019:39:360
2216825512,19sleep00-21swapper/019:39:360
2079625514,25sleep120791-21apt-get19:34:481
2079625514,25sleep120791-21apt-get19:34:481
1892925516,22sleep119051-21cat00:35:041
1661525513,25sleep116610-21apt-get21:04:481
115425512,28sleep01240-21fw_packets20:14:560
115425512,28sleep01240-21fw_packets20:14:560
749725418,21sleep37500-21diskstats23:59:533
3027825412,25sleep330273-21apt-get23:29:493
298412547,35sleep30-21swapper/320:04:223
2422325413,17sleep10-21swapper/123:10:001
2422325413,17sleep10-21swapper/123:10:001
1923425415,23sleep00-21swapper/022:55:010
1299325417,25sleep112882-21/usr/sbin/munin22:34:561
1174925420,22sleep111784-21sed22:29:591
3221025322,17sleep10-21swapper/123:35:001
2451025211,16sleep00-21swapper/021:29:540
2148825217,18sleep10-21swapper/121:19:581
1587725212,24sleep015960-21users21:00:060
1972225112,23sleep319718-21apt-get21:14:473
273925013,22sleep32736-21apt-get23:44:473
2139525014,20sleep321399-21diskstats21:19:563
1674825013,22sleep016744-21apt-get00:29:460
1041525021,16sleep10-21swapper/122:25:071
801724911,22sleep08012-21apt-get22:19:490
801724911,22sleep08012-21apt-get22:19:490
322924911,22sleep33352-21timerandwakeup20:20:043
322924911,22sleep33352-21timerandwakeup20:20:043
2235124913,22sleep022370-21apt-get19:39:480
89102487,15sleep10-21swapper/120:39:531
89102487,15sleep10-21swapper/120:39:531
2274024814,15sleep30-21swapper/323:05:023
1682724812,21sleep116833-21diskstats00:29:511
927524713,19sleep10-21swapper/122:24:341
927524713,19sleep10-21swapper/122:24:341
231682479,20sleep123278-21users19:40:091
309524610,19sleep13086-21apt-get22:04:481
1339324614,18sleep30-21swapper/322:35:063
197042458,15sleep30-21swapper/319:30:023
197042458,15sleep30-21swapper/319:30:023
1779624415,15sleep30-21swapper/322:49:563
1038224212,16sleep30-21swapper/320:44:513
1038224212,16sleep30-21swapper/320:44:513
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional