You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 23:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Apr 03, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
181321785925,17815sleep212406-21kworker/u8:207:12:432
18469215149,1493sleep317375-21kworker/3:209:44:413
2839828475,831sleep028051-1kworker/0:2H08:37:180
18431254210,518sleep118467-21ntpq11:27:231
19266249022,450sleep219272-21/usr/sbin/munin08:07:312
631524017,383sleep013441-21snmpd10:48:220
631524017,383sleep013441-21snmpd10:48:220
12329234813,222sleep012458-21latency_hist07:47:020
28764227216,241sleep228771-21ntp_states10:17:272
28764227216,241sleep228771-21ntp_states10:17:272
265022271102,55sleep00-21swapper/011:52:260
265022271102,55sleep00-21swapper/011:52:260
20058224918,217sleep320106-21unin-run08:12:003
1516122466,12sleep00-21swapper/007:54:170
22026223924,155sleep022116-21fschecks_time08:17:190
22026223924,155sleep022116-21fschecks_time08:17:190
2485223630,22sleep031713-21cyclictest08:57:170
203192234108,71sleep00-21swapper/008:12:100
1388022320,19sleep031713-21cyclictest07:52:010
15288223115,193sleep015313-21munin-plugin-st09:36:580
16247222115,150sleep016217-21latency_hist11:22:010
684722149,19sleep00-21swapper/007:27:270
109921960,110sleep31103-21kworker/u8:112:16:593
2792121857,79sleep00-21swapper/010:16:120
2792121857,79sleep00-21swapper/010:16:120
13998218418,142sleep00-21swapper/011:12:280
13248218220,146sleep113376-21unixbench_multi07:47:311
19080218115,60sleep00-21swapper/009:47:200
21028217922,139sleep121183-21grep08:12:311
22803217722,137sleep122909-21tune2fs11:42:171
2348221737,152sleep023534-21sort08:22:150
2348221737,152sleep023534-21sort08:22:150
301412171108,44sleep00-21swapper/008:42:180
688621709,143sleep00-21swapper/010:52:130
688621709,143sleep00-21swapper/010:52:130
3156421647,37sleep00-21swapper/010:27:100
1317121615,12sleep00-21swapper/009:27:460
1317121615,12sleep00-21swapper/009:27:460
1150721615,11sleep00-21swapper/011:06:510
1929215827,78sleep01932-21cron10:36:590
8466215194,35sleep18471-21cpu10:57:151
1426214911,123sleep01438-21munin-node07:12:080
1323214827,99sleep00-21swapper/008:52:490
8385214610,119sleep01-21systemd09:17:000
3750214618,104sleep23829-21mailstats07:17:282
3325214113,27sleep20-21swapper/212:22:242
1161213932,55sleep01117-21latency_hist12:17:010
32028213752,73sleep01-21systemd12:12:000
807521368,111sleep00-21swapper/007:32:200
30292213612,102sleep00-21swapper/010:22:210
30292213612,102sleep00-21swapper/010:22:210
387621345,10sleep00-21swapper/012:22:380
5010213215,91sleep00-21swapper/012:27:270
5010213215,91sleep00-21swapper/012:27:270
316121326,11sleep00-21swapper/010:37:360
29778213121,89sleep20-21swapper/212:02:292
26844213117,14sleep00-21swapper/008:32:210
985521295,12sleep00-21swapper/009:18:010
2173221275,12sleep00-21swapper/009:56:490
2565621265,13sleep00-21swapper/011:50:440
2565621265,13sleep00-21swapper/011:50:440
3249521256,85sleep00-21swapper/008:48:370
1732421258,62sleep00-21swapper/008:02:160
3331212414,94sleep03365-21cut07:17:170
5507212228,51sleep00-21swapper/009:07:100
525521227,103sleep05289-21ps07:22:260
29510212227,61sleep029408-21apt-get10:19:310
29510212227,61sleep029408-21apt-get10:19:310
5211212117,69sleep00-21swapper/009:06:460
3049211915,88sleep12948-21latency_hist07:17:021
2270621178,49sleep00-21swapper/011:42:120
30541211614,82sleep030599-21/usr/sbin/munin07:02:200
859321150,9sleep28589-21latency_hist09:17:012
485521145,57sleep00-21swapper/010:44:300
485521145,57sleep00-21swapper/010:44:300
27620211422,14sleep00-21swapper/011:57:140
1587521149,68sleep015937-21cat07:57:200
1587521149,68sleep015937-21cat07:57:200
247121136,56sleep00-21swapper/012:19:270
16290211322,50sleep00-21swapper/009:37:510
501211227,60sleep2669-21sed12:12:312
426211227,58sleep30-21swapper/310:32:013
15315211216,78sleep115380-21apt-config09:37:001
25171211118,52sleep00-21swapper/010:07:150
25171211118,52sleep00-21swapper/010:07:150
1076921117,53sleep010779-21munin-run07:42:010
3029921105,70sleep00-21swapper/012:05:180
18038210734,50sleep00-21swapper/011:27:140
25724210516,73sleep325804-21sed10:07:303
25724210516,73sleep325804-21sed10:07:303
14795210416,71sleep314956-21awk07:52:303
11977210418,28sleep00-21swapper/011:07:200
17337210323,57sleep217400-21fschecks_time08:02:182
1266210311,74sleep21309-21/usr/sbin/munin08:52:472
341210214,71sleep1437-21dpkg08:52:011
27402210215,69sleep227564-21sed10:12:302
27402210215,69sleep227564-21sed10:12:302
23242210220,66sleep223368-21latency_hist08:22:012
23242210220,66sleep223368-21latency_hist08:22:012
626221018,81sleep00-21swapper/012:32:190
626221018,81sleep00-21swapper/012:32:190
32005210117,65sleep30-21swapper/307:07:193
16120210012,63sleep10-21swapper/107:57:251
16120210012,63sleep10-21swapper/107:57:251
223852999,15sleep30-21swapper/308:17:263
223852999,15sleep30-21swapper/308:17:263
216029920,60sleep32198-21grep07:12:563
197312997,56sleep019791-21perl11:32:190
2806629719,61sleep128131-21cat10:17:011
2806629719,61sleep128131-21cat10:17:011
2893029614,66sleep228962-21latency_hist12:02:012
87802950,13sleep30-21swapper/309:17:023
2510829516,62sleep125113-21diskstats08:27:171
2510829516,62sleep125113-21diskstats08:27:171
3165829227,43sleep231715-21cut07:07:012
1176329228,21sleep011790-21munin-run09:27:010
846429128,37sleep18590-21latency_hist09:17:011
2424029121,54sleep124363-21/usr/sbin/munin10:02:321
2424029121,54sleep124363-21/usr/sbin/munin10:02:321
72429058,18sleep0765-21http07:12:000
1901729040,33sleep00-21swapper/008:07:250
1533529017,54sleep315374-21grep11:17:243
2739728915,59sleep327548-21latency_hist11:57:013
2739728915,59sleep327548-21latency_hist11:57:013
157382896,13sleep20-21swapper/211:17:312
465028821,40sleep14796-21sed09:02:311
2458228814,57sleep10-21swapper/111:47:221
2458228814,57sleep10-21swapper/111:47:221
183732885,40sleep00-21swapper/009:43:250
2214828717,31sleep30-21swapper/309:57:173
3076328624,38sleep130913-21grep10:22:311
3076328624,38sleep130913-21grep10:22:311
241682868,17sleep00-21swapper/010:02:310
241682868,17sleep00-21swapper/010:02:310
1779928510,21sleep017923-21cat11:27:020
232232846,43sleep00-21swapper/010:00:370
217728324,35sleep10-21swapper/108:57:021
955628219,42sleep39648-21tune2fs07:37:193
2666228220,26sleep10-21swapper/108:32:181
254262829,21sleep00-21swapper/008:27:240
254262829,21sleep00-21swapper/008:27:240
2502828121,36sleep225022-21apt-get08:27:142
2502828121,36sleep225022-21apt-get08:27:142
1293628112,52sleep212967-21nfs_client07:47:242
987228011,49sleep19444-21/usr/sbin/munin07:37:261
581328029,33sleep15873-21latency_hist12:32:001
581328029,33sleep15873-21latency_hist12:32:001
534128018,30sleep10-21swapper/110:47:141
534128018,30sleep10-21swapper/110:47:141
1372128022,32sleep10-21swapper/111:12:211
1233428019,45sleep212345-21/usr/sbin/munin09:27:232
1233428019,45sleep212345-21/usr/sbin/munin09:27:232
90702798,15sleep00-21swapper/010:57:300
3060027926,30sleep130637-21latency07:02:211
3050227924,24sleep20-21swapper/210:22:282
3050227924,24sleep20-21swapper/210:22:282
439927817,40sleep23768-21/usr/sbin/munin10:42:302
439927817,40sleep23768-21/usr/sbin/munin10:42:302
1016427828,33sleep30-21swapper/309:21:593
63827711,14sleep20-21swapper/207:10:572
447627722,38sleep34570-21cut09:02:293
2575227713,49sleep225777-21munin-run11:51:592
2575227713,49sleep225777-21munin-run11:51:592
1658027715,45sleep116673-21uniq11:22:181
1667427615,38sleep216461-21/usr/sbin/munin11:22:202
288027519,22sleep30-21swapper/308:57:283
33232749,14sleep30-21swapper/307:17:163
2168627421,34sleep321705-21sort08:17:013
133522749,22sleep20-21swapper/211:12:102
1226927418,37sleep311782-21/usr/sbin/munin11:07:273
2783527318,40sleep227905-21gpgv08:36:592
219072739,46sleep222054-21awk11:37:342
219072739,46sleep222054-21awk11:37:342
1925827314,43sleep119265-21/usr/sbin/munin08:07:301
1748027322,22sleep20-21swapper/209:42:182
1443527312,29sleep20-21swapper/209:32:322
387827231,25sleep30-21swapper/310:42:193
387827231,25sleep30-21swapper/310:42:193
261227219,22sleep30-21swapper/310:37:233
2054027215,30sleep30-21swapper/309:52:163
605427111,39sleep16083-21idleruntime-cro07:27:001
3044527128,22sleep30-21swapper/307:02:193
2977727128,21sleep30-21swapper/310:22:003
2977727128,21sleep30-21swapper/310:22:003
292027119,30sleep22219-21/usr/sbin/munin10:37:302
2914927118,29sleep30-21swapper/312:02:103
1967027123,27sleep30-21swapper/311:32:183
1715627117,39sleep217242-21grep09:42:012
1646027021,24sleep30-21swapper/307:57:313
1646027021,24sleep30-21swapper/307:57:313
3257926918,35sleep2367-21sshd10:28:412
307412698,15sleep30-21swapper/308:42:303
2411126920,31sleep30-21swapper/308:22:313
2411126920,31sleep30-21swapper/308:22:313
211626915,34sleep21385-21/usr/sbin/munin12:17:312
1701126912,44sleep217113-21sed08:02:022
1701126912,44sleep217113-21sed08:02:022
1308726919,30sleep313146-21awk07:47:293
1057126918,25sleep20-21swapper/211:02:532
928126812,14sleep20-21swapper/207:37:022
2254626810,12sleep10-21swapper/109:57:271
57026727,19sleep30-21swapper/308:52:033
2909026713,37sleep329216-21unixbench_multi08:37:453
2706926710,41sleep327150-21/usr/sbin/munin08:32:283
749426617,32sleep17571-21/usr/sbin/munin10:52:311
749426617,32sleep17571-21/usr/sbin/munin10:52:311
1856126616,19sleep10-21swapper/108:07:021
1766326620,20sleep10-21swapper/109:42:211
937726518,29sleep29401-21/usr/sbin/munin09:17:342
2965226516,28sleep1222-21systemd-journal08:41:591
2161526521,28sleep221685-21systemd-run08:17:002
1083026519,30sleep110904-21mailstats11:03:001
1060126522,20sleep30-21swapper/311:02:543
64626415,20sleep10-21swapper/110:32:121
613826422,21sleep20-21swapper/212:32:172
613826422,21sleep20-21swapper/212:32:172
286026421,19sleep10-21swapper/108:57:261
2807226410,40sleep328078-21ntp_states11:57:253
991226317,20sleep30-21swapper/311:02:013
659726316,32sleep20-21swapper/210:51:592
659726316,32sleep20-21swapper/210:51:592
560326313,37sleep25613-21df_inode09:07:162
1960626318,24sleep30-21swapper/309:47:323
1045326314,21sleep10-21swapper/109:22:141
973926216,28sleep29770-21/usr/sbin/munin07:37:232
3232726217,30sleep132372-21ntp_states07:07:251
299622620,8sleep10-21swapper/110:22:011
299622620,8sleep10-21swapper/110:22:011
2553426216,25sleep10-21swapper/110:07:241
2553426216,25sleep10-21swapper/110:07:241
2370926213,23sleep20-21swapper/210:02:212
2370926213,23sleep20-21swapper/210:02:212
1740326215,29sleep317460-21/usr/sbin/munin08:02:193
271626118,21sleep30-21swapper/312:22:013
2058426113,30sleep10-21swapper/111:33:411
1951726113,35sleep219605-21sed09:47:312
1587926121,22sleep10-21swapper/109:37:401
117326126,17sleep10-21swapper/112:17:021
1049926121,22sleep20-21swapper/209:22:152
922726012,33sleep19280-21latency_hist07:37:011
221452605,11sleep00-21swapper/011:38:290
221452605,11sleep00-21swapper/011:38:290
127962606,12sleep30-21swapper/309:27:373
127962606,12sleep30-21swapper/309:27:373
85125911,22sleep30-21swapper/308:52:273
43825915,25sleep2566-21apt-config08:52:022
361425914,22sleep10-21swapper/107:17:221
3072325911,27sleep10-21swapper/112:07:131
304325927,15sleep10-21swapper/112:22:191
282725915,22sleep10-21swapper/110:37:291
2278225916,26sleep322789-21diskstats11:42:153
1491425923,21sleep10-21swapper/111:17:101
1242025922,19sleep20-21swapper/211:07:302
822625811,25sleep30-21swapper/310:57:013
822625811,25sleep30-21swapper/310:57:013
52472589,34sleep35253-21ntp_states07:22:253
200825821,17sleep20-21swapper/210:37:012
158425810,19sleep30-21swapper/310:32:343
1461325817,27sleep314700-21sed09:32:373
709625716,25sleep27153-21irqstats10:52:192
709625716,25sleep27153-21irqstats10:52:192
684225713,29sleep36984-21latency_hist09:12:023
631625715,26sleep26334-21apt07:27:092
3086925719,16sleep30-21swapper/312:07:193
3019825710,32sleep130212-21uname08:42:191
3002525720,19sleep20-21swapper/208:42:162
110882576,25sleep30-21swapper/307:42:163
677625617,15sleep30-21swapper/312:32:303
677625617,15sleep30-21swapper/312:32:303
670925614,19sleep30-21swapper/307:27:223
2851625612,24sleep10-21swapper/111:57:311
2230925611,18sleep10-21swapper/108:17:241
2230925611,18sleep10-21swapper/108:17:241
2058625617,16sleep20-21swapper/209:52:192
1655925613,23sleep316568-21diskstats11:22:163
1419625613,25sleep214200-21diskstats07:52:172
1110425621,18sleep10-21swapper/107:42:181
1098725619,21sleep30-21swapper/309:22:283
86412555,13sleep30-21swapper/307:32:313
663925513,27sleep36664-21cat10:52:003
663925513,27sleep36664-21cat10:52:003
510325518,15sleep10-21swapper/110:47:011
510325518,15sleep10-21swapper/110:47:011
327242557,14sleep10-21swapper/110:30:341
316632556,13sleep20-21swapper/212:07:502
3133125515,24sleep331465-21latency_hist10:27:013
3133125515,24sleep331465-21latency_hist10:27:013
2690025514,18sleep30-21swapper/310:12:193
2690025514,18sleep30-21swapper/310:12:193
2554725520,18sleep20-21swapper/210:07:252
2554725520,18sleep20-21swapper/210:07:252
244972555,39sleep10-21swapper/108:23:541
244972555,39sleep10-21swapper/108:23:541
2396425514,26sleep223993-21munin-run11:46:592
1958425514,25sleep219580-21apt-get11:32:132
168082555,10sleep30-21swapper/309:38:113
1117025514,24sleep211227-21fw_packets07:42:192
763825414,24sleep17652-21munin-run07:32:001
646825413,24sleep16505-21ntp_states12:32:241
646825413,24sleep16505-21ntp_states12:32:241
487125417,18sleep20-21swapper/207:22:162
2450325416,24sleep324570-21df11:47:203
2450325416,24sleep324570-21df11:47:203
216412547,34sleep10-21swapper/109:55:411
168125416,14sleep10-21swapper/112:17:221
861125320,17sleep20-21swapper/210:57:202
2493325319,17sleep30-21swapper/310:07:023
2493325319,17sleep30-21swapper/310:07:023
2124525314,24sleep121249-21diskstats11:37:211
2124525314,24sleep121249-21diskstats11:37:211
1222025313,24sleep112261-21ntp_states11:07:251
2957825213,22sleep129626-21ntp_states12:02:241
238825216,18sleep20-21swapper/208:57:112
1021625210,25sleep10-21swapper/109:22:011
459125113,21sleep20-21swapper/212:27:162
459125113,21sleep20-21swapper/212:27:162
2133425118,15sleep30-21swapper/311:37:233
2133425118,15sleep30-21swapper/311:37:233
1830725112,21sleep318401-21memory11:27:213
1453025114,22sleep114567-21ntp_states07:52:241
1237125114,18sleep10-21swapper/109:27:261
1237125114,18sleep10-21swapper/109:27:261
630325015,15sleep10-21swapper/109:07:321
450725014,21sleep34574-21df12:27:133
450725014,21sleep34574-21df12:27:133
426524916,16sleep30-21swapper/312:27:013
2885824912,16sleep20-21swapper/208:37:402
2347924910,23sleep223475-21apt-get08:22:132
2347924910,23sleep223475-21apt-get08:22:132
2029524814,19sleep120439-21cat09:52:021
1343024812,20sleep313437-21diskstats11:12:153
409824718,15sleep20-21swapper/209:02:202
2604124711,21sleep226038-21apt-get11:52:122
2604124711,21sleep226038-21apt-get11:52:122
251502479,14sleep30-21swapper/308:27:193
251502479,14sleep30-21swapper/308:27:193
1803524610,16sleep20-21swapper/211:27:122
2228524510,19sleep222333-21irqstats09:57:202
260452418,16sleep10-21swapper/111:52:151
260452418,16sleep10-21swapper/111:52:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional