You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-15 - 15:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Sep 15, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26115282517,790sleep226150-21/usr/sbin/munin07:18:212
4572244213,385sleep14646-21grep09:33:261
3154323940,114sleep131517-21cron09:17:541
26089232732,283sleep126114-21awk07:18:201
915723239,298sleep09177-21/usr/sbin/munin09:48:240
1416423047,262sleep1367-21snmpd11:46:381
5751229328,205sleep15894-21date07:57:581
18461229191,150sleep118572-21apt-config08:37:571
9078228150,185sleep19218-21unixbench_singl11:28:291
9263227174,42sleep10-21swapper/108:08:161
19588227116,241sleep019610-21latency_hist10:22:570
30767225454,180sleep130809-21ntpq07:33:231
10000224453,152sleep110107-21latency_hist09:52:571
15524224217,210sleep315553-21nfs4_client10:08:183
29525223436,101sleep129671-21unixbench_singl12:33:271
3180223381,18sleep10-21swapper/111:12:581
18527222144,45sleep10-21swapper/111:58:251
16039221716,186sleep016113-21tr08:28:270
5996220724,143sleep16124-21unixbench_singl11:18:391
16417220710,179sleep016462-21systemd-run10:12:550
2142722008,155sleep121561-21unixbench_singl08:43:511
1955921989,103sleep119579-21fschecks_count12:03:141
32536218653,92sleep10-21swapper/111:02:561
19919217473,43sleep121978-21cyclictest10:23:141
14893216612,142sleep0367-21snmpd08:23:380
1253221659,14sleep10-21swapper/109:58:471
1253221659,14sleep10-21swapper/109:58:471
27632215821,114sleep227771-21grep09:03:262
11139215510,129sleep111142-21cron11:37:541
11139215510,129sleep111142-21cron11:37:541
30121215213,124sleep330151-21latency_hist07:32:563
2473221529,101sleep124135-21/usr/sbin/munin12:18:241
15732215117,117sleep10-21swapper/110:08:241
809221497,47sleep18099-21munin-run11:27:561
163521469,16sleep10-21swapper/111:07:571
1551921449,40sleep10-21swapper/108:28:111
6856214311,117sleep16900-21true09:42:551
23407214159,16sleep10-21swapper/110:33:241
12142214145,34sleep30-21swapper/309:58:163
12142214145,34sleep30-21swapper/309:58:163
2867221408,52sleep10-21swapper/109:08:061
2867221408,52sleep10-21swapper/109:08:061
1371121395,11sleep00-21swapper/010:03:140
22803213831,66sleep122925-21cat07:08:261
15013213816,81sleep10-21swapper/111:48:241
6606213615,59sleep10-21swapper/107:58:271
6606213615,59sleep10-21swapper/107:58:271
2293221367,40sleep10-21swapper/108:48:271
22164213528,76sleep222183-21apt-get07:08:072
12817213512,64sleep212400-21/usr/sbin/munin08:18:232
31828213419,100sleep231847-21cpu09:18:092
2823021345,11sleep10-21swapper/112:29:111
24643213412,17sleep30-21swapper/307:13:253
7827213325,90sleep37977-21unixbench_multi09:43:263
2308321328,55sleep10-21swapper/112:13:241
13944213210,85sleep10-21swapper/108:23:061
3084821295,12sleep10-21swapper/110:57:151
2985212910,105sleep13031-21grep07:48:211
2985212910,105sleep13031-21grep07:48:211
29111212910,79sleep129162-21apt-config10:52:541
27352212919,87sleep227392-21fschecks_count07:23:142
27352212919,87sleep227392-21fschecks_count07:23:142
27621212710,55sleep127767-21latency_hist10:47:581
23024212717,89sleep023069-21grep10:33:150
6932212315,92sleep37065-21cat09:42:573
1504211913,79sleep10-21swapper/109:23:251
1504211913,79sleep10-21swapper/109:23:251
18742211811,86sleep10-21swapper/110:18:211
25403211717,81sleep025407-21cron07:17:550
2344211751,19sleep10-21swapper/107:47:561
22652211630,51sleep11-21systemd10:32:551
2513121125,73sleep10-21swapper/107:14:321
23871211218,77sleep20-21swapper/208:53:172
21243211214,80sleep320633-21/usr/sbin/munin08:43:493
3693211136,58sleep10-21swapper/111:13:181
2826221110,6sleep00-21swapper/009:07:560
20265211129,43sleep120362-21date08:42:581
20265211129,43sleep120362-21date08:42:581
10908211135,50sleep112-21rcu_sched08:13:131
26450211018,67sleep226585-21grep12:23:262
797210916,77sleep3906-21sed07:42:583
9514210813,74sleep09026-21/usr/sbin/munin08:08:230
4388210818,42sleep30-21swapper/309:33:203
29453210715,78sleep029588-21cmp10:52:580
7680210621,16sleep10-21swapper/109:43:241
27768210518,29sleep20-21swapper/212:28:272
9307210419,60sleep39412-21timerandwakeup09:48:273
2428721030,8sleep20-21swapper/210:37:562
23176210218,68sleep223254-21find12:13:252
21782210220,69sleep221823-21ntpq10:28:202
21365210231,30sleep10-21swapper/107:03:261
17898210224,67sleep1367-21snmpd10:17:001
17381210213,28sleep20-21swapper/210:13:262
6135210113,68sleep25575-21/usr/sbin/munin09:38:242
27019210117,71sleep027052-21latency_hist12:27:560
26325210142,41sleep10-21swapper/112:23:251
17321210116,67sleep00-21swapper/008:33:190
30364210029,46sleep330357-21apt-get07:33:123
2822621005,29sleep10-21swapper/107:24:061
2822621005,29sleep10-21swapper/107:24:061
23208210012,73sleep323240-21latency10:33:183
1477721006,12sleep10-21swapper/110:06:461
125232988,75sleep112622-21grep08:18:181
75902979,45sleep10-21swapper/108:03:151
464829720,53sleep24773-21timerwakeupswit09:33:272
324422977,52sleep132564-21timerandwakeup09:18:241
3007029723,53sleep030109-21idleruntime-cro12:37:540
2555529723,61sleep125607-21cat08:58:161
275429618,35sleep10-21swapper/109:28:211
112372965,12sleep10-21swapper/109:54:351
2110629522,34sleep10-21swapper/112:08:191
3098029413,67sleep231049-21latency_hist10:57:562
2879029412,38sleep10-21swapper/107:28:051
55429323,42sleep10-21swapper/107:40:291
40529315,54sleep30-21swapper/311:03:153
1691529312,64sleep016967-21/usr/sbin/munin10:13:150
1426629220,57sleep314384-21grep10:03:253
101328920,49sleep21045-21tune2fs09:23:132
101328920,49sleep21045-21tune2fs09:23:132
815628827,30sleep30-21swapper/311:27:583
311932887,12sleep00-21swapper/010:58:070
1119728813,58sleep011210-21/usr/sbin/munin08:13:200
3159228714,58sleep231663-21latency_hist09:17:562
301492878,58sleep3364-21ntpd12:37:573
3015628624,33sleep00-21swapper/007:32:570
1682528622,50sleep216855-21sed08:32:572
1623028612,58sleep315822-21apt-get11:52:583
1623028612,58sleep315822-21apt-get11:52:583
268232850,6sleep30-21swapper/309:02:583
2451428425,46sleep124508-21apt-get10:38:081
242282848,16sleep10-21swapper/108:53:241
127142842,68sleep1289-21dbus-daemon11:42:541
127142842,68sleep1289-21dbus-daemon11:42:541
2003628311,57sleep0367-21snmpd08:41:390
2003628311,57sleep0367-21snmpd08:41:390
1964128329,30sleep30-21swapper/308:38:483
1964128329,30sleep30-21swapper/308:38:483
1731428333,36sleep10-21swapper/111:54:131
1731428333,36sleep10-21swapper/111:54:131
2186128227,34sleep221890-21munin-run07:07:552
570828116,32sleep00-21swapper/007:57:570
2732928112,56sleep21-21systemd10:47:552
3151428028,33sleep031540-21munin-run09:17:540
153828014,51sleep31634-21mailstats07:43:223
407027820,33sleep23855-21apt-get07:52:582
407027820,33sleep23855-21apt-get07:52:582
2233727818,42sleep222314-21latency_hist12:12:562
1958127812,50sleep319586-21systemd-run10:22:563
1424227818,42sleep214304-21latency08:23:202
288162779,16sleep20-21swapper/212:33:112
255072777,12sleep00-21swapper/010:39:290
468127617,38sleep04682-21cron11:17:540
277022760,7sleep30-21swapper/312:28:233
2087227624,24sleep00-21swapper/007:03:170
1151327615,46sleep31-21systemd09:57:563
797427510,41sleep20-21swapper/208:03:242
2918527515,39sleep228791-21/usr/sbin/munin07:28:192
2267227520,32sleep30-21swapper/307:08:243
1309727419,33sleep20-21swapper/211:43:162
652427314,43sleep06605-21switchtime07:58:260
652427314,43sleep06605-21switchtime07:58:260
2242727331,29sleep022459-21irqstats07:08:170
2513027219,30sleep30-21swapper/308:57:583
2416427224,31sleep30-21swapper/312:18:103
1452327211,47sleep014525-21sshd11:48:020
1192427213,16sleep20-21swapper/211:38:232
1192427213,16sleep20-21swapper/211:38:232
1021627117,34sleep010217-21/usr/sbin/munin09:53:080
3025027020,30sleep30-21swapper/309:13:133
248127019,28sleep22549-21sshd09:28:142
2821126919,29sleep328239-21munin-run09:07:563
272652698,14sleep00-21swapper/012:28:130
521326820,28sleep05232-21apt-get11:18:060
2766526815,25sleep00-21swapper/007:23:200
2766526815,25sleep00-21swapper/007:23:200
2191326821,29sleep321930-21/usr/sbin/munin07:07:563
2065126819,31sleep220652-21apt-get08:43:072
197326823,25sleep00-21swapper/011:08:160
1635926822,23sleep00-21swapper/011:53:200
1635926822,23sleep00-21swapper/011:53:200
1580826821,31sleep315834-21munin-run11:52:543
1149626819,31sleep211509-21modprobe09:57:552
93026712,38sleep21007-21/usr/sbin/munin11:03:262
388226715,35sleep23904-21/usr/sbin/munin11:13:232
2800826710,14sleep00-21swapper/010:48:210
3206326613,22sleep00-21swapper/009:18:150
2922126622,20sleep30-21swapper/309:08:363
2922126622,20sleep30-21swapper/309:08:363
2635226616,26sleep00-21swapper/007:18:270
2489026617,30sleep20-21swapper/210:38:222
1567226622,28sleep30-21swapper/308:28:173
130112668,11sleep00-21swapper/009:58:580
130112668,11sleep00-21swapper/009:58:580
1030726619,28sleep310323-21apt-get08:12:543
381126511,24sleep30-21swapper/311:13:203
378526521,26sleep00-21swapper/009:32:570
3198826511,31sleep231914-21/usr/sbin/munin07:38:122
268752655,27sleep10-21swapper/109:03:051
2654726521,24sleep30-21swapper/310:43:233
151432655,10sleep30-21swapper/308:26:523
2424726416,30sleep224259-21/usr/sbin/munin12:18:132
2111926414,23sleep30-21swapper/312:08:213
1797126421,26sleep317999-21munin-run10:17:553
2460426316,30sleep224642-21/usr/sbin/munin07:13:242
1239926315,21sleep00-21swapper/008:18:090
1153026316,25sleep30-21swapper/311:38:123
1153026316,25sleep30-21swapper/311:38:123
50426214,32sleep0537-21/usr/sbin/munin11:03:160
384226215,27sleep33870-21munin-run07:52:553
384226215,27sleep33870-21munin-run07:52:553
207272625,46sleep30-21swapper/312:07:523
2040926223,18sleep00-21swapper/010:23:250
1782826212,33sleep217961-21uptime08:33:292
719526119,22sleep00-21swapper/009:43:100
2747826117,26sleep012-21rcu_sched10:47:570
2592226116,30sleep225995-21grep08:58:262
2437126114,22sleep00-21swapper/012:18:150
2263326120,22sleep30-21swapper/308:48:213
1888026117,28sleep218871-21apt-get08:38:102
1888026117,28sleep218871-21apt-get08:38:102
1605726113,22sleep20-21swapper/211:52:572
1605726113,22sleep20-21swapper/211:52:572
1110926118,22sleep30-21swapper/308:13:193
3256426015,19sleep20-21swapper/211:02:572
2898226016,28sleep029000-21uname09:08:290
2898226016,28sleep029000-21uname09:08:290
1954426018,27sleep219569-21munin-run10:22:552
1051226017,21sleep20-21swapper/208:12:582
104826016,25sleep31134-21perl09:23:143
104826016,25sleep31134-21perl09:23:143
85652595,17sleep00-21swapper/008:05:240
615625912,23sleep30-21swapper/307:58:163
615625912,23sleep30-21swapper/307:58:163
484825910,36sleep34861-21/usr/sbin/munin07:53:363
312625910,30sleep23159-21munin-run11:12:552
2275825910,24sleep30-21swapper/310:32:573
2259225915,24sleep222632-21grep08:48:202
200512599,23sleep00-21swapper/012:03:250
1727325918,19sleep30-21swapper/308:33:183
876625817,25sleep38795-21ntp_states11:28:233
465625815,23sleep20-21swapper/207:53:312
304825811,31sleep03088-21ps07:48:240
304825811,31sleep03088-21ps07:48:240
2918125823,20sleep20-21swapper/209:08:352
2918125823,20sleep20-21swapper/209:08:352
2206825814,22sleep00-21swapper/010:28:260
1122225815,18sleep00-21swapper/011:37:570
1122225815,18sleep00-21swapper/011:37:570
248012576,14sleep00-21swapper/008:55:000
988525615,25sleep29879-21apt-get11:33:102
988525615,25sleep29879-21apt-get11:33:102
869325614,26sleep28689-21apt-get09:48:092
840525614,27sleep28416-21df_inode11:28:132
3239325613,19sleep00-21swapper/007:38:220
2827925613,26sleep228316-21ntp_states10:48:272
2609525618,21sleep326137-21ntpq12:23:193
2228725614,27sleep022306-21awk12:12:540
268825513,28sleep30-21swapper/309:28:193
25502559,32sleep32676-21cat11:08:283
253482556,14sleep00-21swapper/008:58:110
1789425515,17sleep20-21swapper/211:58:062
132825513,23sleep00-21swapper/009:23:190
132825513,23sleep00-21swapper/009:23:190
1324725514,27sleep013281-21/usr/sbin/munin11:43:180
103225515,24sleep01026-21apt-get07:43:070
2198125417,24sleep022063-21cut08:47:570
190342547,34sleep00-21swapper/012:00:590
1524125411,24sleep20-21swapper/210:08:132
135502544,11sleep20-21swapper/210:03:002
1202525412,26sleep211517-21apt-get09:58:072
1202525412,26sleep211517-21apt-get09:58:072
1990425312,26sleep319913-21df_inode10:23:123
1433725313,18sleep30-21swapper/311:47:573
1373025315,23sleep313781-21ls08:22:563
897825216,18sleep00-21swapper/011:28:280
678525212,23sleep06782-21apt-get11:23:070
550525214,22sleep35531-21irqstats11:18:293
536325216,17sleep00-21swapper/009:37:570
3147025213,23sleep331496-21irqstats10:58:223
1453225210,18sleep20-21swapper/211:48:092
1107125214,22sleep211108-21irqstats08:13:182
2384525115,20sleep01-21systemd12:17:550
1923125116,16sleep30-21swapper/312:02:563
125572515,37sleep00-21swapper/011:40:520
125572515,37sleep00-21swapper/011:40:520
594925012,22sleep26016-21df07:58:112
594925012,22sleep26016-21df07:58:112
2784825011,23sleep327845-21apt-get10:48:083
2726825010,22sleep327335-21df07:23:113
2726825010,22sleep327335-21df07:23:113
2615125013,22sleep326174-21ntp_states07:18:223
236132506,31sleep30-21swapper/312:14:553
2360025015,15sleep00-21swapper/008:52:580
1947625012,22sleep219543-21df12:03:112
750124910,24sleep37497-21apt-get08:03:093
557424911,23sleep05593-21apt-get09:38:060
524524914,18sleep20-21swapper/211:18:122
254372496,30sleep30-21swapper/310:38:353
2029724913,19sleep20-21swapper/210:23:232
3033724811,19sleep230338-21/usr/sbin/munin07:33:072
845624717,17sleep20-21swapper/209:47:572
74424717,15sleep30-21swapper/309:22:583
3037824713,14sleep30-21swapper/310:53:333
988624513,15sleep30-21swapper/311:33:113
988624513,15sleep30-21swapper/311:33:113
739624511,21sleep20-21swapper/211:23:252
3048924511,12sleep00-21swapper/007:33:170
11432448,18sleep20-21swapper/207:43:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional