You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-06 - 09:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Sep 06, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17738280922,770sleep217755-21/usr/sbin/munin23:43:382
7175270819,672sleep37233-21/usr/sbin/munin21:28:543
7175270819,672sleep37233-21/usr/sbin/munin21:28:543
5970262016,585sleep0231-21systemd-journal19:40:320
28463250714,478sleep328555-21install19:13:313
2066250115,471sleep12058-21apt-get21:13:461
5839248348,421sleep05888-21ntpq21:24:100
14911239623,354sleep314942-21/usr/sbin/munin23:33:503
8608234610,313sleep08685-21switchtime19:49:000
29676229118,260sleep129714-21ntpq22:38:541
15324225032,200sleep015462-21unixbench_multi23:33:590
23071223768,75sleep00-21swapper/023:59:010
20583223054,163sleep020602-21fschecks_count20:28:480
1771223011,117sleep01780-21modprobe21:13:290
105602230132,25sleep00-21swapper/021:38:510
13533222315,192sleep213537-21diskstats21:48:502
20056221217,182sleep218487-21kworker/2:120:26:322
2254922095,11sleep00-21swapper/022:16:090
2254922095,11sleep00-21swapper/022:16:090
17485220715,176sleep117490-21entropy20:18:481
267422019,136sleep02778-21wc21:14:010
428822007,183sleep0367-21snmpd23:00:040
28561219116,151sleep028741-21cut19:13:320
1089021887,170sleep0367-21snmpd23:21:480
19672218716,94sleep00-21swapper/022:08:300
1782921767,14sleep00-21swapper/020:18:550
10922217616,145sleep211025-21cat19:58:312
17407217517,140sleep00-21swapper/023:42:350
32284217348,108sleep032348-21mailstats00:29:010
32284217348,108sleep032348-21mailstats00:29:010
14422217272,82sleep00-21swapper/023:33:300
16010216444,108sleep016061-21cat20:13:510
8984215911,70sleep00-21swapper/021:33:510
8984215911,70sleep00-21swapper/021:33:510
26892215559,39sleep027024-21latency_hist19:08:310
2504921555,42sleep00-21swapper/000:05:400
26065215244,96sleep026149-21mailstats00:08:590
14222215038,93sleep00-21swapper/021:49:050
31027214947,52sleep0367-21snmpd21:02:230
27728214314,69sleep00-21swapper/000:13:590
27728214314,69sleep00-21swapper/000:13:590
1743421439,73sleep017556-21cat21:59:100
1743421439,73sleep017556-21cat21:59:100
29002214250,79sleep029013-21ls19:13:360
108621405,10sleep00-21swapper/022:49:110
2051721385,45sleep00-21swapper/023:52:440
6260213715,106sleep16410-21latency_hist19:43:321
912321366,42sleep00-21swapper/023:14:290
29279213633,73sleep329274-21apt-get22:38:443
12383213417,103sleep112465-21mailstats21:43:571
295721335,41sleep00-21swapper/022:58:190
2006621335,42sleep00-21swapper/020:26:380
26517213244,74sleep026553-21ntpq22:28:560
31031213119,90sleep130545-21apt-get22:43:391
23697213116,91sleep00-21swapper/022:19:040
1908213124,58sleep30-21swapper/319:28:493
356212839,66sleep00-21swapper/019:23:490
356212839,66sleep00-21swapper/019:23:490
28703212725,71sleep128697-21apt-get00:18:451
28703212725,71sleep128697-21apt-get00:18:451
3553212518,19sleep00-21swapper/019:33:500
31346212353,16sleep00-21swapper/022:43:580
19918212242,67sleep019911-21apt-get22:08:460
2913921208,95sleep00-21swapper/000:18:560
2913921208,95sleep00-21swapper/000:18:560
2306212014,51sleep02370-21awk19:28:580
28781211819,70sleep20-21swapper/220:53:572
12858211713,88sleep012829-21latency_hist23:28:300
6201211623,76sleep26228-21modprobe19:43:292
480921166,99sleep0367-21snmpd23:03:350
15929211642,64sleep0367-21snmpd21:54:210
27850211518,77sleep127999-21unixbench_singl00:14:001
27850211518,77sleep127999-21unixbench_singl00:14:001
23223211531,72sleep00-21swapper/020:37:280
21045211510,25sleep00-21swapper/023:53:490
25598211436,17sleep00-21swapper/020:43:540
7534211328,74sleep00-21swapper/023:11:590
5457211320,46sleep20-21swapper/221:23:562
24683211335,15sleep00-21swapper/022:23:480
1382211314,82sleep21463-21grep00:34:002
1382211314,82sleep21463-21grep00:34:002
12123211144,13sleep00-21swapper/019:59:340
25270211017,75sleep225357-21fschecks_time20:43:482
4099210768,20sleep00-21swapper/021:18:580
4099210768,20sleep00-21swapper/021:18:580
13327210711,23sleep00-21swapper/020:04:060
28952210615,67sleep00-21swapper/020:53:590
6620210513,74sleep00-21swapper/019:43:510
11134210315,74sleep211150-21awk19:58:392
2242210212,25sleep00-21swapper/000:38:300
2242210212,25sleep00-21swapper/000:38:300
6946210117,63sleep26953-21/usr/sbin/munin19:43:572
31906210032,14sleep00-21swapper/021:04:000
30929210045,37sleep231058-21unixbench_multi00:24:032
30929210045,37sleep231058-21unixbench_multi00:24:032
313029815,67sleep13105-21latency_hist19:33:311
51829714,70sleep2582-21cpuspeed21:08:462
3150229624,51sleep331530-21idleruntime-cro00:28:283
3150229624,51sleep331530-21idleruntime-cro00:28:283
2370129621,58sleep223705-21diskstats20:38:472
2362329561,16sleep00-21swapper/020:38:450
2089729517,56sleep220940-21ntp_offset20:28:542
36582945,12sleep30-21swapper/322:58:543
1719129415,25sleep20-21swapper/221:59:042
1719129415,25sleep20-21swapper/221:59:042
3248629320,49sleep232630-21latency_hist19:23:312
3248629320,49sleep232630-21latency_hist19:23:312
2671929221,53sleep00-21swapper/020:48:460
2593029239,32sleep20-21swapper/219:03:542
1590129227,42sleep115927-21munin-run23:38:281
2212029110,63sleep322121-21cron23:58:283
1023529110,66sleep010360-21cat19:54:010
874429024,47sleep18256-21apt-get21:33:401
874429024,47sleep18256-21apt-get21:33:401
3061829026,47sleep130678-21sendmail-msp19:18:291
2986129021,42sleep229254-21/usr/sbin/munin22:39:002
2723529024,53sleep127255-21fschecks_count19:08:481
2402229011,20sleep10-21swapper/120:38:541
144312908,28sleep014492-21basename20:08:500
144312908,28sleep014492-21basename20:08:500
87328920,45sleep1892-21apt-get00:33:431
87328920,45sleep1892-21apt-get00:33:431
344428920,54sleep13346-21/usr/sbin/munin19:33:471
2827428913,58sleep00-21swapper/022:34:010
2827428913,58sleep00-21swapper/022:34:010
3243328856,15sleep00-21swapper/019:23:290
3243328856,15sleep00-21swapper/019:23:290
3042428810,65sleep330516-21switchtime20:58:593
294072889,56sleep00-21swapper/022:38:490
2698828817,54sleep327004-21uname20:48:543
2371928811,60sleep223739-21modprobe00:03:282
127162888,62sleep20-21swapper/220:03:432
3032728758,14sleep00-21swapper/000:23:480
3032728758,14sleep00-21swapper/000:23:480
617128640,22sleep30-21swapper/321:24:173
1870728527,26sleep30-21swapper/320:23:303
1545928527,33sleep30-21swapper/321:54:003
7362840,7sleep30-21swapper/322:49:053
2370628416,48sleep323742-21/usr/sbin/munin20:38:483
649528315,52sleep36548-21ls19:43:463
3155528317,50sleep131526-21latency_hist00:28:301
3155528317,50sleep131526-21latency_hist00:28:301
2590228320,14sleep20-21swapper/222:28:302
222982830,12sleep222299-21unixbench_multi22:14:002
222982830,12sleep222299-21unixbench_multi22:14:002
3021628213,54sleep130242-21netstat20:58:531
178982826,17sleep30-21swapper/322:01:153
178982826,17sleep30-21swapper/322:01:153
177622828,16sleep00-21swapper/023:43:400
892728123,38sleep29077-21grep23:14:072
1968628118,47sleep119836-21latency_hist22:08:311
1184128127,28sleep20-21swapper/223:24:082
2602028028,31sleep10-21swapper/119:03:571
34427921,27sleep30-21swapper/319:23:483
34427921,27sleep30-21swapper/319:23:483
2845827920,27sleep20-21swapper/200:18:292
2845827920,27sleep20-21swapper/200:18:292
1644127920,46sleep216571-21timerwakeupswit20:14:012
1103227922,33sleep211163-21unixbench_multi21:39:022
881227819,26sleep20-21swapper/219:49:022
846027823,25sleep10-21swapper/121:33:321
846027823,25sleep10-21swapper/121:33:321
67602789,16sleep10-21swapper/123:08:531
2751727814,41sleep227126-21/usr/sbin/munin19:08:542
1875927816,48sleep318796-21ntp_kernel_pll_22:03:533
1755727821,33sleep10-21swapper/121:59:111
1755727821,33sleep10-21swapper/121:59:111
74192776,15sleep00-21swapper/021:28:590
74192776,15sleep00-21swapper/021:28:590
2687027722,39sleep327028-21sed22:29:043
119852778,49sleep00-21swapper/021:43:480
572227616,35sleep10-21swapper/121:24:071
190327613,35sleep21907-21diskstats19:28:482
809027518,41sleep38115-21/usr/sbin/munin19:48:473
774727524,32sleep17772-21munin-run19:48:281
2835027512,48sleep328355-21diskstats20:53:463
824227418,37sleep28259-21sh21:33:292
824227418,37sleep28259-21sh21:33:292
2765927416,34sleep327090-21/usr/sbin/munin00:13:583
2765927416,34sleep327090-21/usr/sbin/munin00:13:583
504927316,24sleep30-21swapper/323:03:513
1836627321,32sleep218338-21/usr/sbin/munin22:03:442
1786127320,23sleep20-21swapper/220:18:572
521727219,30sleep34934-21apt-get21:23:333
521727219,30sleep34934-21apt-get21:23:333
3118927222,23sleep20-21swapper/222:43:522
2895727220,33sleep329012-21tr22:38:283
2895727220,33sleep329012-21tr22:38:283
2122327220,36sleep121287-21sed22:13:321
2086527218,28sleep10-21swapper/120:28:531
1943927210,44sleep219525-21/usr/sbin/munin23:48:502
143412728,14sleep10-21swapper/120:08:471
143412728,14sleep10-21swapper/120:08:471
1246827219,34sleep112484-21cat20:03:301
446027016,32sleep24552-21users21:19:042
446027016,32sleep24552-21users21:19:042
355827020,24sleep10-21swapper/122:58:521
3272127020,29sleep132718-21apt-get19:23:421
3272127020,29sleep132718-21apt-get19:23:421
3173027023,21sleep30-21swapper/319:19:083
3173027023,21sleep30-21swapper/319:19:083
316542708,13sleep30-21swapper/322:44:053
3137927021,28sleep20-21swapper/221:03:422
191827011,14sleep30-21swapper/322:53:503
1508327020,21sleep10-21swapper/121:53:501
984626911,26sleep30-21swapper/323:18:483
968126913,30sleep10-21swapper/119:53:491
2686926921,28sleep327008-21latency_hist00:13:303
2668326921,31sleep126754-21/usr/sbin/munin22:29:021
2613326924,21sleep30-21swapper/319:03:593
1485326920,30sleep214910-21/usr/sbin/munin23:33:492
1319526923,25sleep10-21swapper/123:28:471
3150126823,30sleep331610-21perl21:03:513
2491026816,34sleep124921-21vmstat00:04:031
929526722,31sleep11-21systemd19:53:291
814626722,24sleep327-21ksoftirqd/323:13:403
329026725,22sleep20-21swapper/222:58:452
2116926713,39sleep221192-21modprobe22:13:282
1689126720,29sleep30-21swapper/323:38:583
1435426717,27sleep30-21swapper/320:08:493
1435426717,27sleep30-21swapper/320:08:493
1171526722,21sleep30-21swapper/323:24:053
2486526617,25sleep30-21swapper/322:23:513
1636526614,29sleep116439-21switchtime20:14:001
1384426620,25sleep10-21swapper/121:48:581
1323326621,21sleep30-21swapper/321:48:313
1079826620,23sleep10-21swapper/121:38:571
3130626514,34sleep131367-21fschecks_time19:18:581
3130626514,34sleep131367-21fschecks_time19:18:581
55026418,25sleep20-21swapper/222:49:002
3180026416,28sleep30-21swapper/300:28:483
3180026416,28sleep30-21swapper/300:28:483
2642326417,32sleep326465-21systemd-run20:48:293
134162647,16sleep10-21swapper/120:04:071
1162126412,11sleep30-21swapper/319:58:563
1015826419,28sleep310227-21/usr/sbin/munin19:53:593
2709526313,30sleep126694-21/usr/sbin/munin20:48:561
225932637,14sleep30-21swapper/322:16:403
225932637,14sleep30-21swapper/322:16:403
1641826316,24sleep216616-21cat21:58:312
97832627,43sleep20-21swapper/221:35:182
97832627,43sleep20-21swapper/221:35:182
2911126217,26sleep228420-21apt-get19:13:382
2909726220,28sleep329131-21ntp_kernel_pll_00:18:553
2909726220,28sleep329131-21ntp_kernel_pll_00:18:553
2679126214,30sleep226797-21/usr/sbin/munin20:48:482
2649926212,37sleep20-21swapper/200:09:252
2583826215,29sleep125963-21cat20:43:591
337126121,20sleep30-21swapper/319:33:453
2836526119,20sleep10-21swapper/122:34:021
2836526119,20sleep10-21swapper/122:34:021
2317726120,20sleep10-21swapper/123:59:021
484326013,27sleep14837-21apt-get23:03:451
3001526013,19sleep20-21swapper/200:23:302
3001526013,19sleep20-21swapper/200:23:302
2873826025,21sleep10-21swapper/120:53:561
2338026016,25sleep20-21swapper/220:38:312
1114926017,24sleep111316-21sed23:23:301
68025915,20sleep20-21swapper/219:23:552
68025915,20sleep20-21swapper/219:23:552
2990025911,27sleep20-21swapper/220:58:452
218432597,11sleep10-21swapper/123:54:551
1837725916,27sleep118472-21timerandwakeup23:43:571
119225912,23sleep10-21swapper/121:09:021
1002225918,22sleep20-21swapper/223:18:512
70325815,26sleep1850-21cut00:33:301
497125813,19sleep10-21swapper/121:23:311
497125813,19sleep10-21swapper/121:23:311
1971925819,25sleep30-21swapper/323:48:553
400725717,24sleep24133-21sed19:34:002
2893925712,26sleep30-21swapper/319:13:353
2401225715,25sleep224006-21apt-get00:03:442
2231125715,22sleep30-21swapper/320:33:523
1978125711,24sleep30-21swapper/320:24:033
1435425716,25sleep214382-21/usr/sbin/munin23:33:282
266562565,39sleep10-21swapper/100:11:261
171325616,25sleep11707-21apt-get22:53:441
1591025624,17sleep30-21swapper/320:13:503
1453725613,18sleep20-21swapper/220:08:522
1453725613,18sleep20-21swapper/220:08:522
1071925612,26sleep310750-21/usr/sbin/munin21:38:553
2330025515,24sleep323323-21irqstats22:18:543
2272925514,25sleep122782-21sendmail22:18:281
2272925514,25sleep122782-21sendmail22:18:281
1439425516,19sleep30-21swapper/323:33:293
888125412,24sleep38982-21perl21:33:503
888125412,24sleep38982-21perl21:33:503
494125413,24sleep34938-21apt-get19:38:423
340925420,17sleep20-21swapper/221:18:312
2434825417,21sleep20-21swapper/222:23:292
1961725416,22sleep119685-21memory23:48:531
124662548,12sleep20-21swapper/221:43:582
785525310,18sleep30-21swapper/323:13:313
3187325311,22sleep131882-21/usr/sbin/munin00:28:501
27802536,17sleep20-21swapper/222:56:032
2675525313,23sleep226868-21ls22:29:032
2401325216,18sleep30-21swapper/300:03:453
2086225213,22sleep220856-21apt-get23:53:432
1992125216,16sleep30-21swapper/322:08:493
1353825217,17sleep30-21swapper/321:48:513
3184425113,21sleep131857-21ntp_states21:03:571
200972517,32sleep10-21swapper/120:27:041
1648325111,21sleep20-21swapper/223:38:492
136882516,11sleep30-21swapper/320:04:493
3207025017,16sleep20-21swapper/200:28:552
3207025017,16sleep20-21swapper/200:28:552
2302325018,17sleep10-21swapper/122:18:401
2222525014,15sleep10-21swapper/120:33:501
1775625011,15sleep30-21swapper/323:43:393
97124913,21sleep3980-21diskstats00:33:493
97124913,21sleep3980-21diskstats00:33:493
813724911,22sleep17664-21apt-get23:13:331
493924910,17sleep20-21swapper/223:03:502
2287524913,21sleep322911-21ntp_states23:58:563
2779124811,21sleep227883-21perl22:33:512
2779124811,21sleep227883-21perl22:33:512
171352487,30sleep10-21swapper/123:39:051
1154824810,20sleep10-21swapper/123:23:431
2477924710,16sleep10-21swapper/122:23:491
689124610,14sleep20-21swapper/223:08:582
2094324610,20sleep320950-21ntp_states20:28:553
53624510,16sleep10-21swapper/122:48:591
5842448,15sleep30-21swapper/321:08:473
26002418,17sleep32038-21/usr/sbin/munin21:14:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional