You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-19 - 10:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Apr 19, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1705287036,818sleep31770-21awk22:56:453
14580286911,841sleep214604-21df20:16:202
25503277612,603sleep325591-21fschecks_time20:51:253
1930727698,724sleep319353-21ntpq22:11:313
19159268615,654sleep219161-21/usr/sbin/munin20:31:182
3212126569,603sleep332133-21apt-get21:11:163
1743924119,388sleep317453-21grep22:06:223
13963240312,378sleep11234-21kworker/1:120:13:001
19569238412,353sleep319635-40find23:56:043
19443230116,271sleep119456-21irqstats20:31:291
2329723008,213sleep323325-21awk22:26:043
5027228826,246sleep35121-21grep19:46:253
5027228826,246sleep35121-21grep19:46:253
243282280165,18sleep30-21swapper/300:11:063
7810227694,33sleep37850-21latency_hist19:56:063
7810227694,33sleep37850-21latency_hist19:56:063
2394822418,188sleep324033-21grep20:46:253
266172240129,61sleep30-21swapper/300:16:313
1733722248,198sleep317408-21switchtime23:46:353
2104221312,181sleep32168-21latency19:36:273
28813220611,169sleep328945-21unixbench_singl19:16:373
4708219527,142sleep34770-21fschecks_time21:26:303
31830218742,94sleep30-21swapper/321:11:083
16557218545,48sleep316619-21sendmail_mailqu20:21:343
6018218310,157sleep06008-21apt-key21:31:070
11243218214,19sleep30-21swapper/320:06:233
11243218214,19sleep30-21swapper/320:06:233
2580321798,39sleep30-21swapper/322:31:333
7779217616,105sleep30-21swapper/321:36:073
3251421769,145sleep332643-21timerwakeupswit22:51:363
3251421769,145sleep332643-21timerwakeupswit22:51:363
31315217315,95sleep30-21swapper/319:26:263
19994216313,64sleep320032-21grep23:56:243
22761216114,23sleep30-21swapper/300:06:053
21479216012,129sleep321632-21unixbench_singl20:36:363
30196215912,126sleep30-21swapper/319:21:383
307821577,100sleep313441-21snmpd21:21:253
3566215515,123sleep13713-21awk21:21:361
26993215312,128sleep327045-21cpuspeed20:56:183
335521439,96sleep33375-21apt-get19:41:163
16737214313,22sleep00-21swapper/023:46:160
23966214111,85sleep30-21swapper/322:26:283
23966214111,85sleep30-21swapper/322:26:283
2949421398,118sleep329541-21grep00:26:213
10133213814,76sleep324305-21cyclictest21:41:363
1227121376,47sleep30-21swapper/323:31:253
25918213515,95sleep30-21swapper/300:16:073
17111213515,26sleep30-21swapper/322:06:063
30401213230,58sleep330485-21fschecks_time22:46:233
1478821325,29sleep30-21swapper/323:39:203
19175212913,93sleep00-21swapper/022:11:280
6937212815,18sleep10-21swapper/119:51:311
6937212815,18sleep10-21swapper/119:51:311
31436212783,26sleep30-21swapper/300:31:303
12985212623,18sleep30-21swapper/321:51:293
12985212623,18sleep30-21swapper/321:51:293
30751212517,48sleep230774-21awk00:31:042
12914212510,99sleep312933-21apt-get20:11:163
28758212074,26sleep30-21swapper/321:01:153
19597211917,77sleep119748-21grep22:11:371
561921177,60sleep35684-21latency_hist23:11:063
18028211614,83sleep118072-21awk20:26:281
18028211614,83sleep118072-21awk20:26:281
28963211311,55sleep30-21swapper/300:26:043
1092121139,54sleep311051-21cat20:06:073
915821125,12sleep30-21swapper/321:40:023
3133211211,56sleep30-21swapper/323:01:293
342211117,73sleep1438-21cut19:31:251
2182121119,24sleep321965-21cat22:21:073
1545421115,11sleep30-21swapper/320:16:453
9430210913,22sleep29449-21ntp_states23:21:312
2473210817,75sleep225884-21systemd-journal23:01:042
1019421086,45sleep30-21swapper/323:25:193
5570210717,71sleep25618-21unin-run23:11:052
1551021078,83sleep315519-21munin-run22:01:053
29689210610,17sleep00-21swapper/021:01:400
12333210626,48sleep30-21swapper/321:51:043
12333210626,48sleep30-21swapper/321:51:043
2816921055,11sleep30-21swapper/322:40:253
2816921055,11sleep30-21swapper/322:40:253
14208210525,54sleep114198-21apt-get21:56:231
30383210418,63sleep230398-21cut22:46:222
1135221038,39sleep30-21swapper/323:26:373
31162210217,70sleep231245-21fschecks_time00:31:252
2875821028,37sleep30-21swapper/322:41:263
2875821028,37sleep30-21swapper/322:41:263
26645210223,56sleep226676-21modprobe22:36:042
24347210256,23sleep30-21swapper/319:06:073
2039121025,11sleep30-21swapper/320:35:263
1894521018,81sleep318944-21/usr/sbin/munin20:31:063
1894521018,81sleep318944-21/usr/sbin/munin20:31:063
117121016,36sleep30-21swapper/319:31:393
59422997,57sleep35834-21/usr/sbin/munin23:11:223
20632997,15sleep30-21swapper/321:16:373
1028429921,59sleep19585-21/usr/sbin/munin20:01:371
1820629715,68sleep218302-21switchtime20:26:332
1820629715,68sleep218302-21switchtime20:26:332
389529611,72sleep00-21swapper/021:23:400
344529516,63sleep23604-21unixbench_multi23:01:362
637229419,56sleep06373-21/usr/sbin/munin21:31:150
2872629415,61sleep128235-21apt-get22:41:161
2872629415,61sleep128235-21apt-get22:41:161
2037029416,56sleep019882-21/usr/sbin/munin23:56:330
129802948,13sleep00-21swapper/020:11:450
104029415,63sleep11117-21latency_hist21:16:061
322102937,20sleep20-21swapper/221:11:392
264852937,33sleep30-21swapper/319:11:203
2512529116,58sleep225142-21awk20:51:042
2365229129,37sleep023827-21head19:01:370
1894729014,59sleep019090-21date20:31:070
1894729014,59sleep019090-21date20:31:070
440428819,50sleep14437-21grep23:06:281
276428813,55sleep10-21swapper/123:01:191
1388528820,47sleep213909-21munin-run21:56:042
1388528820,47sleep213909-21munin-run21:56:042
806428713,59sleep00-21swapper/023:16:360
931128511,59sleep09313-21sshd20:00:590
2946228522,43sleep128978-21apt-get00:26:151
2754428519,47sleep227618-21ls20:56:332
408428422,43sleep24108-21awk21:26:042
1240228319,42sleep212351-21latency_hist21:51:072
1240228319,42sleep212351-21latency_hist21:51:072
851028231,35sleep30-21swapper/319:56:343
773928218,40sleep27778-21sed21:36:062
637528219,37sleep15886-21apt-get21:31:161
2347328212,56sleep123502-21proc_pri00:06:321
1604128210,38sleep20-21swapper/222:01:292
65328114,44sleep01-21systemd22:56:050
65328114,44sleep01-21systemd22:56:050
2268428110,55sleep222816-21ls22:21:352
162732805,29sleep30-21swapper/323:43:323
279422799,16sleep00-21swapper/020:57:040
1521327921,25sleep10-21swapper/120:16:351
3161427813,44sleep00-21swapper/000:31:350
2334127828,26sleep20-21swapper/219:01:302
2112827813,45sleep021167-21ntpq20:36:280
181492780,7sleep00-21swapper/022:06:340
1396427816,49sleep014052-21wc21:56:060
1396427816,49sleep014052-21wc21:56:060
86182775,27sleep30-21swapper/323:19:583
713027719,40sleep17193-21sendmail-msp23:16:051
587227722,36sleep21-21systemd21:31:052
2624427715,42sleep126315-21munin-node19:11:101
1366827722,30sleep10-21swapper/123:36:191
1133127710,51sleep011388-21cat20:06:240
1133127710,51sleep011388-21cat20:06:240
3079427610,52sleep030808-21latency_hist00:31:060
2669227622,37sleep126763-21/usr/sbin/munin00:16:331
1831127618,45sleep118339-21grep23:51:221
81927522,32sleep21038-21apt-key22:56:072
81927522,32sleep21038-21apt-key22:56:072
2584427521,35sleep225945-21munin-node19:11:072
1792827514,45sleep218022-21cat22:06:332
1669227517,43sleep113441-21snmpd20:21:361
2630427417,40sleep026132-21/usr/sbin/munin00:16:240
84862736,44sleep20-21swapper/221:36:342
812827311,13sleep10-21swapper/121:36:251
2874227322,32sleep228738-21apt-get22:41:172
2874227322,32sleep228738-21apt-get22:41:172
2517727319,26sleep00-21swapper/020:51:060
2278927318,23sleep00-21swapper/020:41:330
2210227312,45sleep222255-21latency_hist20:41:072
218132739,16sleep10-21swapper/120:38:271
3053927213,17sleep10-21swapper/121:06:261
2353827228,10sleep30-21swapper/320:45:443
170642727,10sleep00-21swapper/020:22:310
2924427142,16sleep20-21swapper/219:19:592
2679627121,27sleep00-21swapper/020:56:070
254142719,43sleep10-21swapper/120:51:191
2160627117,36sleep221670-21idleruntime00:01:242
1839527121,23sleep00-21swapper/023:51:250
993827012,13sleep00-21swapper/023:22:010
407527015,39sleep04180-21cat23:06:060
214282706,11sleep00-21swapper/022:16:570
1784827014,24sleep10-21swapper/122:06:321
100472706,13sleep00-21swapper/020:01:320
3071826913,41sleep030726-21needreboot21:06:290
2751426920,19sleep20-21swapper/200:21:072
2499526915,36sleep025019-21/usr/sbin/munin19:06:330
2395226917,24sleep20-21swapper/222:26:272
2395226917,24sleep20-21swapper/222:26:272
641926823,20sleep20-21swapper/223:11:342
3229826818,35sleep132343-21basename00:36:041
279512687,37sleep00-21swapper/000:21:290
2685226816,22sleep00-21swapper/019:11:290
193126813,39sleep22046-21timerwakeupswit21:16:362
322282678,48sleep00-21swapper/019:28:330
3028126715,35sleep030396-21timerwakeupswit19:21:390
21752670,8sleep00-21swapper/019:36:270
177772678,48sleep10-21swapper/123:47:581
167492677,17sleep00-21swapper/022:02:400
1350126717,26sleep20-21swapper/220:12:162
1083726717,29sleep20-21swapper/221:46:072
458326611,37sleep14602-21apt-get21:26:161
293412668,12sleep10-21swapper/121:01:371
2871826613,39sleep028235-21apt-get22:41:150
2871826613,39sleep028235-21apt-get22:41:150
1588726620,22sleep20-21swapper/223:41:332
1393726615,23sleep00-21swapper/023:36:270
1218126622,24sleep20-21swapper/223:31:242
1158226616,20sleep00-21swapper/021:46:330
1158226616,20sleep00-21swapper/021:46:330
785326523,20sleep00-21swapper/019:56:070
50152657,12sleep20-21swapper/219:46:242
50152657,12sleep20-21swapper/219:46:242
2566826519,25sleep10-21swapper/100:13:591
2500026516,26sleep10-21swapper/122:31:061
2500026516,26sleep10-21swapper/122:31:061
2335026517,32sleep123442-21sort22:26:061
2044126519,31sleep020473-21munin-run20:36:040
866126425,19sleep00-21swapper/021:36:360
2910426414,29sleep129247-21sed00:26:061
2672626429,18sleep00-21swapper/022:36:060
253726417,24sleep10-21swapper/123:01:071
3204226318,25sleep10-21swapper/122:51:261
3204226318,25sleep10-21swapper/122:51:261
2906426315,28sleep20-21swapper/221:01:302
2602526320,22sleep10-21swapper/122:31:391
2490826319,26sleep024997-21apt-config22:31:050
2490826319,26sleep024997-21apt-config22:31:050
2338826316,29sleep30-21swapper/319:01:313
2116326313,36sleep121164-21sshd00:00:591
1807026312,37sleep218109-21cat23:51:062
1552126319,17sleep00-21swapper/022:01:060
1480126318,18sleep00-21swapper/020:16:260
1238526320,22sleep10-21swapper/121:51:061
1238526320,22sleep10-21swapper/121:51:061
3174626218,26sleep231828-21apt-config21:11:072
2938126218,28sleep129508-21latency_hist19:21:071
2340926221,20sleep10-21swapper/119:01:341
100032627,11sleep10-21swapper/121:41:341
782626121,25sleep20-21swapper/223:16:302
755226112,28sleep10-21swapper/123:16:251
2407726113,32sleep124127-21irqstats20:46:271
832426017,31sleep28370-21sed19:56:292
74426014,17sleep20-21swapper/219:31:342
33026014,29sleep0341-21/usr/sbin/munin19:31:240
1617926018,25sleep216279-21fschecks_time20:21:242
1244826017,19sleep10-21swapper/120:11:061
1244826017,19sleep10-21swapper/120:11:061
99322598,38sleep10-21swapper/123:21:581
721225913,26sleep00-21swapper/023:16:080
38432596,12sleep00-21swapper/019:41:330
2705425918,24sleep027141-21tune2fs22:36:220
2705425918,24sleep027141-21tune2fs22:36:220
260325915,29sleep12751-21wc21:21:071
233792596,11sleep10-21swapper/120:43:441
1937125914,29sleep219436-21awk22:11:342
1430325917,17sleep20-21swapper/221:56:282
1367125916,19sleep20-21swapper/223:36:202
1248425917,18sleep20-21swapper/220:11:072
1248425917,18sleep20-21swapper/220:11:072
961625822,19sleep00-21swapper/021:41:250
812625815,26sleep18135-21df_inode19:56:241
2656125815,22sleep20-21swapper/219:11:232
2309525815,20sleep00-21swapper/000:06:230
1647425817,25sleep11-21systemd23:46:051
52792579,36sleep00-21swapper/023:07:200
469425717,22sleep20-21swapper/219:46:072
314325717,17sleep20-21swapper/219:41:072
257925718,20sleep00-21swapper/021:21:060
2468725714,18sleep00-21swapper/000:11:230
970925617,24sleep29816-21perl20:01:262
2841925611,27sleep228564-21/usr/sbin/munin00:21:392
2278025616,18sleep00-21swapper/000:06:060
2203525614,26sleep022031-21apt-get22:21:170
492925516,25sleep14923-21apt-get19:46:191
492925516,25sleep14923-21apt-get19:46:191
1208825514,24sleep012082-21apt-get23:31:180
1188725516,23sleep212039-21sed23:31:082
1062125516,22sleep110628-21diskstats23:26:221
647525412,25sleep06469-21apt-get19:51:180
388625416,22sleep10-21swapper/119:41:341
3039725413,23sleep10-21swapper/119:21:401
134025415,18sleep00-21swapper/022:56:340
62825317,16sleep00-21swapper/021:11:580
1075125318,16sleep00-21swapper/023:26:250
76722526,16sleep20-21swapper/219:55:162
76722526,16sleep20-21swapper/219:55:162
2453025211,24sleep124527-21apt-get19:06:171
2049425216,21sleep120486-21apt-get22:16:211
1575825213,23sleep115753-21apt-get22:01:201
1208925212,24sleep112090-21http23:31:191
2747825113,23sleep127543-21awk20:56:321
270925110,15sleep10-21swapper/119:36:371
3029625011,22sleep030292-21apt-get22:46:160
279112506,32sleep20-21swapper/222:37:052
279112506,32sleep20-21swapper/222:37:052
2744325012,19sleep10-21swapper/122:36:321
2744325012,19sleep10-21swapper/122:36:321
2483325014,15sleep20-21swapper/200:11:252
2145925010,15sleep10-21swapper/100:01:191
2086225010,17sleep20-21swapper/222:16:302
3162424912,22sleep231630-21ntp_states19:26:332
2568624914,17sleep20-21swapper/220:51:282
2300624911,22sleep223003-21apt-get00:06:172
1998824912,21sleep219993-21diskstats23:56:232
1286524915,17sleep10-21swapper/121:51:271
1286524915,17sleep10-21swapper/121:51:271
1528024811,21sleep015290-21df_inode23:41:190
2895124710,23sleep028975-21users19:16:380
234622466,30sleep20-21swapper/220:44:462
542524412,18sleep00-21swapper/019:46:340
542524412,18sleep00-21swapper/019:46:340
3072824311,17sleep10-21swapper/122:46:291
1834024310,14sleep20-21swapper/223:51:232
312502429,17sleep10-21swapper/119:26:241
295922416,16sleep20-21swapper/219:21:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional