You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 00:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Nov 24, 2025 12:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
244412463011,4607sleep25654-21kworker/2:007:26:522
2945421632114,1501sleep329453-21ntp_states09:23:563
165632100312,973sleep324545-21kworker/u8:212:09:483
27714280423,766sleep227727-21irqstats09:18:552
18729276916,735sleep018742-21/usr/sbin/munin08:49:040
18729276916,735sleep018742-21/usr/sbin/munin08:49:040
32240272141,662sleep132314-21fschecks_time09:33:521
10459263312,606sleep210465-21ntpq11:48:592
1638124770,64sleep116454-21cyclictest10:24:031
1638124770,64sleep116454-21cyclictest10:24:031
17270231112,248sleep117423-21unixbench_multi12:10:191
20682230310,213sleep120745-21processes07:13:581
14308228193,95sleep10-21swapper/110:18:551
16076227023,202sleep116105-21apt-key12:08:331
31785226510,242sleep231796-21ntpq07:49:012
31785226510,242sleep231796-21ntpq07:49:012
5033225715,226sleep25068-21ps09:49:002
5033225715,226sleep25068-21ps09:49:002
27666224414,163sleep127669-21cron11:03:301
17592222645,87sleep10-21swapper/108:48:321
17592222645,87sleep10-21swapper/108:48:321
19726222014,152sleep119820-21cut12:19:071
19726222014,152sleep119820-21cut12:19:071
20261218850,76sleep10-21swapper/110:38:511
976218120,142sleep01001-21dpkg09:38:310
5244217921,98sleep10-21swapper/108:08:551
28541217841,54sleep10-21swapper/107:39:021
14827217016,66sleep114840-21fschecks_count08:39:031
5144216939,86sleep10-21swapper/111:33:561
31803216619,58sleep10-21swapper/107:49:041
31803216619,58sleep10-21swapper/107:49:041
27216216534,74sleep127299-21latency_hist09:18:331
17883216337,109sleep118029-21latency_hist12:13:341
31497216111,113sleep131543-21ntpq11:13:561
31497216111,113sleep131543-21ntpq11:13:561
21170216046,27sleep10-21swapper/112:23:481
21170216046,27sleep10-21swapper/112:23:481
844215936,81sleep1883-21ntp_kernel_err07:54:051
17977215818,56sleep10-21swapper/107:08:331
24123215629,101sleep00-21swapper/010:49:030
23028215637,47sleep10-21swapper/110:48:321
3319215411,129sleep33307-21latency_hist08:03:323
2698621548,133sleep127051-21swap07:34:021
4258215313,63sleep10-21swapper/108:04:231
24860215323,16sleep10-21swapper/107:28:441
4651215146,92sleep14541-21/usr/sbin/munin09:48:501
4651215146,92sleep14541-21/usr/sbin/munin09:48:501
29263215016,110sleep229302-21latency_hist11:08:322
2565421507,131sleep125674-21latency_hist09:13:321
11118214610,121sleep111261-21latency_hist08:28:331
309521418,74sleep10-21swapper/109:43:501
309521418,74sleep10-21swapper/109:43:501
17500214013,109sleep016446-21/usr/sbin/munin12:10:220
8607213910,94sleep18621-21vmstat09:59:081
22426213828,91sleep322436-21modprobe12:28:303
22426213828,91sleep322436-21modprobe12:28:303
15231213818,59sleep30-21swapper/308:39:173
30529213120,12sleep10-21swapper/109:28:431
30529213120,12sleep10-21swapper/109:28:431
29417213120,95sleep029397-21apt-key07:43:340
2779213137,78sleep10-21swapper/109:43:331
2779213137,78sleep10-21swapper/109:43:331
29305212920,82sleep30-21swapper/311:08:333
23134212823,80sleep30-21swapper/312:29:113
23134212823,80sleep30-21swapper/312:29:113
13613212411,72sleep113673-21grep11:59:001
6419212221,58sleep10-21swapper/109:53:551
725921219,15sleep10-21swapper/111:39:211
3464212039,64sleep10-21swapper/111:28:511
8491211623,45sleep10-21swapper/111:44:181
24611211519,54sleep124744-21latency_hist10:53:331
23579211316,11sleep10-21swapper/109:05:491
22375211210,53sleep10-21swapper/107:19:031
28035211037,14sleep10-21swapper/111:03:511
18353211015,78sleep218370-21cpu07:08:382
7225210923,68sleep10-21swapper/108:14:081
26709210913,79sleep226775-21wc07:33:542
2785221089,83sleep127889-21sed09:18:581
579210614,77sleep0582-21sshd07:53:520
8449210411,78sleep28439-21apt-get11:43:462
23705210441,39sleep10-21swapper/107:23:561
11629210420,61sleep211330-21/usr/sbin/munin08:28:552
32590210213,26sleep00-21swapper/007:53:310
32590210213,26sleep00-21swapper/007:53:310
29468210223,64sleep10-21swapper/109:23:581
9256210121,42sleep10-21swapper/110:03:491
19482210121,15sleep10-21swapper/108:53:491
1963521009,17sleep00-21swapper/012:18:580
1963521009,17sleep00-21swapper/012:18:580
11967210014,72sleep012004-21ntp_kernel_pll_11:53:540
658429913,73sleep26629-21latency_hist08:13:362
2976729920,51sleep20-21swapper/207:43:442
2976729920,51sleep20-21swapper/207:43:442
2467329916,66sleep324817-21date07:28:343
1183129921,16sleep10-21swapper/111:53:511
3106629812,69sleep031159-21mailstats09:29:020
3106629812,69sleep031159-21mailstats09:29:020
3127529616,65sleep031068-21/usr/sbin/munin11:13:510
3127529616,65sleep031068-21/usr/sbin/munin11:13:510
2979829641,14sleep10-21swapper/107:43:591
2979829641,14sleep10-21swapper/107:43:591
1487129616,57sleep314662-21/usr/sbin/munin12:03:543
1742029417,15sleep10-21swapper/110:28:571
3103829317,59sleep031050-21sh07:48:310
3103829317,59sleep031050-21sh07:48:310
3092529310,65sleep330530-21/usr/sbin/munin09:28:573
3092529310,65sleep330530-21/usr/sbin/munin09:28:573
1871029223,48sleep218742-21fschecks_count10:33:512
1675729213,60sleep016851-21cut07:03:510
2691929115,59sleep00-21swapper/007:34:010
1908929118,53sleep219099-21modprobe08:53:302
1908929118,53sleep219099-21modprobe08:53:302
1051229127,41sleep210542-21munin-plugin-st10:08:302
33462907,18sleep00-21swapper/008:03:330
1549029017,51sleep015465-21/usr/sbin/munin10:23:470
1549029017,51sleep015465-21/usr/sbin/munin10:23:470
2392028912,56sleep224082-21cat09:08:332
202922890,9sleep20-21swapper/210:38:512
1711728910,62sleep30-21swapper/307:03:593
117028913,19sleep20-21swapper/209:38:332
117028913,19sleep20-21swapper/209:38:332
24532887,65sleep12600-21grep11:24:021
27942879,35sleep10-21swapper/107:59:181
260852875,54sleep10-21swapper/110:58:051
2144328717,52sleep220966-21/usr/sbin/munin12:23:572
2144328717,52sleep220966-21/usr/sbin/munin12:23:572
1435328718,47sleep014554-21gpgv08:38:340
1435328718,47sleep014554-21gpgv08:38:340
1088328712,56sleep310949-21fschecks_time10:08:483
595828627,27sleep30-21swapper/309:53:333
2553328524,25sleep30-21swapper/310:54:033
674728325,39sleep06765-21fschecks_count11:38:510
2753928319,47sleep327551-21/usr/sbin/munin09:18:513
182728314,48sleep00-21swapper/011:23:490
1602428319,42sleep216041-21grep08:43:322
1565628330,24sleep10-21swapper/106:59:001
1312028329,35sleep10-21swapper/108:33:541
1312028329,35sleep10-21swapper/108:33:541
1402328223,41sleep014091-21fschecks_time10:18:500
1342528211,58sleep213508-21mailstats08:34:042
1342528211,58sleep213508-21mailstats08:34:042
844228123,39sleep08439-21apt-get11:43:440
123142815,65sleep10-21swapper/108:29:381
2461628018,31sleep20-21swapper/209:09:132
2331228012,52sleep023356-21cpu07:23:470
2140128022,17sleep10-21swapper/108:58:581
2140128022,17sleep10-21swapper/108:58:581
805227921,29sleep20-21swapper/208:18:332
1783127922,34sleep30-21swapper/308:48:463
1783127922,34sleep30-21swapper/308:48:463
1508127915,17sleep00-21swapper/006:58:470
1311327911,53sleep113260-21head10:14:031
115427930,33sleep10-21swapper/111:19:421
115427930,33sleep10-21swapper/111:19:421
1044827929,13sleep10-21swapper/111:48:581
83332780,6sleep30-21swapper/309:59:053
190692787,32sleep10-21swapper/110:33:581
1846527815,24sleep00-21swapper/012:13:570
1750727814,47sleep217514-21ntp_states10:29:022
154027819,38sleep01477-21latency_hist11:23:320
154027819,38sleep01477-21latency_hist11:23:320
686327711,31sleep00-21swapper/008:13:590
3264527721,29sleep30-21swapper/311:18:473
3264527721,29sleep30-21swapper/311:18:473
1818527723,36sleep318257-21fschecks_time12:13:523
463227629,25sleep30-21swapper/311:33:333
2950727614,15sleep00-21swapper/011:08:460
393027524,23sleep00-21swapper/011:29:020
338627517,29sleep00-21swapper/009:43:570
338627517,29sleep00-21swapper/009:43:570
2668027518,25sleep20-21swapper/210:58:552
300012747,33sleep10-21swapper/111:09:001
201227415,45sleep02086-21mailstats09:39:120
201227415,45sleep02086-21mailstats09:39:120
1842627416,19sleep10-21swapper/107:08:411
305427323,29sleep30-21swapper/311:28:323
2495427322,27sleep30-21swapper/309:09:223
2213527319,33sleep321740-21/usr/sbin/munin07:18:573
206752739,44sleep00-21swapper/007:13:570
80127221,31sleep2903-21/usr/sbin/munin11:19:022
80127221,31sleep2903-21/usr/sbin/munin11:19:022
3123727227,25sleep231367-21grep09:29:042
3123727227,25sleep231367-21grep09:29:042
2198227220,32sleep20-21swapper/210:43:522
209927211,42sleep20-21swapper/207:58:512
2066427217,35sleep320182-21/usr/sbin/munin07:13:563
147832727,14sleep20-21swapper/210:19:052
1300127216,29sleep00-21swapper/010:14:020
87272718,46sleep10-21swapper/108:19:051
81127122,24sleep30-21swapper/309:36:533
2711727115,38sleep327244-21grep10:59:043
1384927116,36sleep013990-21/usr/sbin/munin11:59:030
106482715,29sleep10-21swapper/110:08:331
3235527019,33sleep332364-21modprobe11:18:313
3235527019,33sleep332364-21modprobe11:18:313
2104527020,23sleep20-21swapper/212:23:452
2104527020,23sleep20-21swapper/212:23:452
1646927016,38sleep215960-21apt-get12:08:422
1374327018,36sleep313847-21timerandwakeup11:59:023
3016426916,41sleep330286-21sort11:09:023
227162696,28sleep10-21swapper/112:28:441
227162696,28sleep10-21swapper/112:28:441
2222926910,43sleep222242-21/usr/sbin/munin07:19:002
2218326912,41sleep00-21swapper/009:02:570
2218326912,41sleep00-21swapper/009:02:570
2024526914,41sleep320252-21df_inode10:38:483
189322695,52sleep10-21swapper/108:51:271
189322695,52sleep10-21swapper/108:51:271
1704926913,27sleep30-21swapper/310:28:473
1610326922,28sleep30-21swapper/310:24:023
1610326922,28sleep30-21swapper/310:24:023
933126823,29sleep39338-21/usr/sbin/munin10:03:513
2486426812,22sleep00-21swapper/007:28:480
2416926819,21sleep20-21swapper/212:33:342
2416926819,21sleep20-21swapper/212:33:342
228272687,48sleep00-21swapper/007:21:000
1984626812,43sleep319851-21/usr/sbin/munin12:19:093
1984626812,43sleep319851-21/usr/sbin/munin12:19:093
1935026829,23sleep30-21swapper/307:09:043
922926724,20sleep00-21swapper/010:03:440
2807226714,21sleep20-21swapper/207:38:512
2503226715,22sleep00-21swapper/010:53:520
1902526712,41sleep019068-21/usr/sbin/munin10:33:570
1367526720,27sleep20-21swapper/211:59:012
316292666,12sleep20-21swapper/211:14:012
316292666,12sleep20-21swapper/211:14:012
2545526616,27sleep20-21swapper/210:54:022
2354426613,37sleep223628-21cat10:48:532
1523526614,37sleep015301-21/usr/sbin/munin12:04:040
672026517,19sleep00-21swapper/009:54:020
396526516,32sleep03540-21/usr/sbin/munin08:04:140
3162726513,34sleep331705-21/usr/sbin/munin07:48:583
3162726513,34sleep331705-21/usr/sbin/munin07:48:583
3085626515,34sleep230966-21latency_hist11:13:332
2940526514,33sleep229453-21/usr/sbin/munin09:23:552
2015426512,35sleep020301-21awk08:54:050
1968126512,23sleep20-21swapper/208:53:542
1911226515,32sleep019169-21wc07:09:000
2483326424,26sleep30-21swapper/307:28:423
1935626421,23sleep30-21swapper/310:34:043
1838626414,32sleep00-21swapper/010:33:320
1540126416,25sleep20-21swapper/212:04:062
1278826425,25sleep212833-21ntp_offset10:13:562
1167026421,21sleep30-21swapper/311:53:493
2643826315,31sleep326517-21systemctl09:14:233
2640326312,37sleep226435-21ps09:14:222
2180626315,22sleep30-21swapper/310:43:493
118072638,18sleep20-21swapper/210:10:122
108962636,11sleep00-21swapper/008:26:180
54862627,45sleep00-21swapper/009:49:100
54862627,45sleep00-21swapper/009:49:100
537626216,19sleep20-21swapper/211:34:092
511726210,38sleep00-21swapper/008:08:460
2715926216,27sleep327296-21/usr/sbin/munin07:34:043
174742627,13sleep00-21swapper/008:47:380
174742627,13sleep00-21swapper/008:47:380
1707426223,20sleep20-21swapper/207:03:582
1660826216,28sleep216649-21sshd08:43:532
1660826216,28sleep216649-21sshd08:43:532
1184126213,26sleep00-21swapper/008:29:010
977926120,20sleep20-21swapper/208:23:422
825526118,27sleep28251-21apt-get08:18:452
373826119,25sleep33776-21fschecks_time08:04:083
1754126116,19sleep30-21swapper/308:48:303
1754126116,19sleep30-21swapper/308:48:303
1640626116,26sleep20-21swapper/207:03:332
1050726117,26sleep010597-21mailstats11:49:010
747026014,23sleep20-21swapper/209:58:332
328426017,24sleep20-21swapper/209:43:552
328426017,24sleep20-21swapper/209:43:552
2908626019,18sleep00-21swapper/009:23:490
2677826011,25sleep00-21swapper/010:58:570
242322604,11sleep30-21swapper/310:49:393
1852526010,33sleep218531-21ntp_states12:13:592
477425913,28sleep30-21swapper/309:48:523
477425913,28sleep30-21swapper/309:48:523
686825815,18sleep30-21swapper/311:39:113
432625816,22sleep30-21swapper/309:48:323
432625816,22sleep30-21swapper/309:48:323
2835025820,18sleep30-21swapper/307:38:563
2209225815,27sleep022121-21/usr/sbin/munin10:43:540
2106225819,24sleep30-21swapper/308:58:523
2106225819,24sleep30-21swapper/308:58:523
1977525815,20sleep30-21swapper/308:53:563
137425816,24sleep20-21swapper/207:54:142
854725719,19sleep30-21swapper/311:44:243
529025714,23sleep05327-21ntp_states11:33:590
212042576,11sleep20-21swapper/207:15:042
1684925717,16sleep20-21swapper/210:28:342
826225616,23sleep20-21swapper/209:59:052
816025615,24sleep08166-21ntp_states09:58:590
545625621,21sleep30-21swapper/311:34:103
291632567,36sleep00-21swapper/011:07:490
192525618,18sleep20-21swapper/211:23:512
1393625615,25sleep313931-21apt-get10:18:453
1183225620,20sleep00-21swapper/010:10:300
519125513,27sleep35197-21df_inode08:08:493
2979525516,23sleep329765-21apt-get07:43:573
2979525516,23sleep329765-21apt-get07:43:573
2261125514,24sleep022615-21diskstats09:03:500
1351025513,23sleep313581-21/usr/sbin/munin08:34:053
1351025513,23sleep313581-21/usr/sbin/munin08:34:053
2880125412,27sleep028947-21cat09:23:330
2076025416,15sleep00-21swapper/010:39:020
1572625414,24sleep215760-21/usr/sbin/munin10:23:532
1157425414,24sleep211568-21apt-get11:53:442
102725418,18sleep30-21swapper/307:54:113
955725314,22sleep09566-21munin-run08:23:320
215432539,31sleep0362-21snmpd12:23:590
215432539,31sleep0362-21snmpd12:23:590
1469125313,22sleep00-21swapper/008:38:420
143825313,24sleep3951-21apt-get09:38:433
143825313,24sleep3951-21apt-get09:38:433
1146425313,20sleep30-21swapper/308:28:523
100022539,17sleep30-21swapper/311:48:483
978525215,22sleep39799-21apt-get08:23:433
900125218,15sleep30-21swapper/308:19:153
717725213,23sleep37178-21ntp_states08:14:063
3231825111,17sleep20-21swapper/209:33:532
2329525118,17sleep00-21swapper/012:29:160
2329525118,17sleep00-21swapper/012:29:160
2142525112,23sleep221431-21ntp_states08:58:592
2142525112,23sleep221431-21ntp_states08:58:592
336524910,23sleep23371-21df_inode11:28:482
68052489,15sleep20-21swapper/211:39:102
386524811,17sleep20-21swapper/208:04:112
2590024812,20sleep025896-21apt-get09:13:440
2512124817,16sleep20-21swapper/207:28:542
242252486,15sleep00-21swapper/009:08:350
1648624815,14sleep30-21swapper/307:03:343
1246424610,20sleep312470-21diskstats10:13:483
2260224211,14sleep30-21swapper/309:03:493
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional