You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-31 - 02:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sat Aug 30, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
197721840217,18364sleep31396-21kworker/u8:209:39:093
123912865224,8609sleep310392-21kworker/3:108:28:533
319612701517,6987sleep327552-21kworker/3:009:29:203
180022426234,4212sleep17865-21kworker/u8:010:28:491
1624329447,924sleep019202-1kworker/0:0H10:24:110
23199263616,604sleep223208-21/usr/sbin/munin09:03:442
23199263616,604sleep223208-21/usr/sbin/munin09:03:442
1323825727,410sleep113284-21ntpq10:13:591
6577256514,534sleep06595-21/usr/sbin/munin09:53:470
1794238814,312sleep11921-21timerwakeupswit11:19:071
17895231116,282sleep317904-21ntpq12:09:023
2549122917,29sleep10-21swapper/110:53:231
2514226713,184sleep10-21swapper/107:58:351
2514226713,184sleep10-21swapper/107:58:351
6434225072,83sleep10-21swapper/108:08:571
11425223715,207sleep311536-21ls08:24:043
3975223264,93sleep13978-21cron11:28:311
20978223217,190sleep021009-21fschecks_count07:13:540
9299222954,114sleep19384-21fschecks_time08:18:541
2005922295,11sleep00-21swapper/008:53:400
15681222662,98sleep115706-21munin-run10:23:321
16180221536,161sleep115695-21apt-get10:23:441
399322109,13sleep00-21swapper/009:44:030
12277220512,179sleep012367-21/usr/sbin/munin11:53:330
29325220482,25sleep10-21swapper/107:38:571
24732219910,174sleep224733-21sh07:24:012
27647219193,21sleep10-21swapper/110:59:001
23061218214,72sleep10-21swapper/110:43:591
30756217511,113sleep130829-21sed11:09:011
25487217567,85sleep10-21swapper/112:33:511
21422217210,126sleep121458-21sed08:58:341
31211216613,86sleep10-21swapper/109:28:541
1073521637,106sleep110706-21latency_hist11:48:331
19002216149,78sleep10-21swapper/112:13:481
22133214810,125sleep122141-21awk08:59:021
30497214726,18sleep10-21swapper/107:43:501
32629214511,119sleep332707-21find11:14:043
32426214422,106sleep132464-21ntp_kernel_pll_11:13:581
11278214431,100sleep111280-21apt-get10:08:461
12368214130,39sleep112378-21munin-run11:53:341
4350214034,20sleep10-21swapper/111:28:511
4350214034,20sleep10-21swapper/111:28:511
13409213613,107sleep313478-21switchtime10:14:053
14131213422,87sleep00-21swapper/011:58:340
4072213250,16sleep10-21swapper/109:44:041
20559213153,22sleep120701-21munin-node07:13:431
32144212925,23sleep10-21swapper/107:48:541
8332212837,21sleep10-21swapper/108:14:051
2842921280,9sleep20-21swapper/209:19:012
18941212814,98sleep119007-21cpuspeed07:08:501
5826212719,79sleep30-21swapper/308:08:343
26495212710,102sleep126623-21sort10:54:401
26338212442,17sleep10-21swapper/109:13:481
24271212413,31sleep10-21swapper/107:23:481
25292212315,34sleep125212-21latency_hist12:33:351
12728212311,95sleep312248-21apt-get11:53:403
11046212312,23sleep111164-21date10:08:341
6138211936,62sleep10-21swapper/111:33:521
14961211917,85sleep215085-21cat11:59:042
12886211938,15sleep10-21swapper/108:29:071
21535211817,28sleep10-21swapper/110:38:581
23639211516,85sleep31-21systemd12:28:323
7888211412,88sleep07956-21ls08:13:560
15606211419,29sleep115633-21apt12:03:311
21335211324,27sleep30-21swapper/307:14:013
602221129,20sleep00-21swapper/011:33:430
24484211011,32sleep124487-21cron09:08:311
24484211011,32sleep124487-21cron09:08:311
21324211034,23sleep121456-21awk12:19:091
21324211034,23sleep121456-21awk12:19:091
2761021088,21sleep20-21swapper/207:33:542
13939210817,65sleep213957-21fschecks_count08:33:522
1669221077,18sleep116717-21munin-run08:43:321
1669221077,18sleep116717-21munin-run08:43:321
28509210618,40sleep10-21swapper/107:37:041
25268210615,76sleep00-21swapper/012:33:340
2052121066,12sleep10-21swapper/110:34:381
19152210219,69sleep219255-21timerandwakeup08:49:062
30215210016,19sleep10-21swapper/109:24:051
1974421006,11sleep10-21swapper/108:52:491
2682029914,70sleep326822-21sshd07:30:163
1900629912,75sleep319014-21open_files08:49:033
1743929937,33sleep00-21swapper/007:03:530
1743929937,33sleep00-21swapper/007:03:530
85942986,28sleep10-21swapper/111:39:411
3253629815,21sleep20-21swapper/211:14:032
2862529815,22sleep00-21swapper/007:38:320
2368929819,55sleep323201-21/usr/sbin/munin09:04:013
2368929819,55sleep323201-21/usr/sbin/munin09:04:013
198529810,73sleep02053-21sh09:39:140
17112988,72sleep21409-21apt-get09:38:352
3212629715,65sleep032225-21fschecks_time11:13:530
3176229722,58sleep031901-21grep09:29:050
2941629620,58sleep029479-21sendmail_mailqu11:04:080
191462956,11sleep20-21swapper/207:08:552
2760729322,23sleep00-21swapper/009:18:310
1137229311,65sleep211415-21ntp_offset11:48:592
1465929212,22sleep314757-21users08:34:063
398429111,67sleep14023-21http08:03:321
1350229111,60sleep10-21swapper/111:54:311
190928917,59sleep11409-21apt-get09:38:381
1572128917,50sleep215726-21apt-key10:23:332
43222887,48sleep10-21swapper/108:03:361
2628228714,37sleep10-21swapper/107:29:001
167912870,7sleep00-21swapper/012:04:060
2002228616,54sleep020027-21ntpq10:34:000
1087928611,60sleep010731-21/usr/sbin/munin08:23:530
2966328524,19sleep30-21swapper/311:04:113
2735128510,60sleep027365-21acpi10:58:470
2643428520,19sleep20-21swapper/209:13:532
1626328516,38sleep30-21swapper/312:03:573
1616228511,63sleep116180-21grep12:03:551
635928424,31sleep20-21swapper/209:53:332
635928424,31sleep20-21swapper/209:53:332
2801428418,49sleep028082-21fschecks_time09:18:510
1509128420,54sleep10-21swapper/108:38:151
1295828413,52sleep013044-21/usr/sbin/munin10:13:540
479928317,43sleep04242-21/usr/sbin/munin11:29:030
479928317,43sleep04242-21/usr/sbin/munin11:29:030
337728316,50sleep33492-21timerwakeupswit07:59:063
3099928312,56sleep231017-21apt09:28:422
183972830,6sleep30-21swapper/308:48:343
2184928220,40sleep321864-21apt-get12:23:313
2184928220,40sleep321864-21apt-get12:23:313
155528214,50sleep11710-21latency_hist09:38:341
634528111,14sleep10-21swapper/109:53:321
634528111,14sleep10-21swapper/109:53:321
2873028112,49sleep10-21swapper/109:19:051
2359528113,48sleep023201-21/usr/sbin/munin09:03:580
2359528113,48sleep023201-21/usr/sbin/munin09:03:580
1868128111,13sleep10-21swapper/107:08:341
1868128111,13sleep10-21swapper/107:08:341
108262819,12sleep10-21swapper/108:23:501
817228010,37sleep10-21swapper/109:58:451
812728014,46sleep28205-21/usr/sbin/munin11:39:022
2555128030,23sleep30-21swapper/310:53:333
2139628012,51sleep221410-21irqstats10:38:552
174392807,53sleep10-21swapper/108:44:011
1605828014,50sleep016135-21threads08:39:190
1605828014,50sleep016135-21threads08:39:190
370427917,48sleep33745-21if_eth009:43:533
182502796,14sleep10-21swapper/112:09:081
2918927812,48sleep229218-21irqstats07:38:542
2282527811,48sleep10-21swapper/112:24:221
1902727814,41sleep20-21swapper/212:13:492
744527712,46sleep10-21swapper/109:54:091
289062778,33sleep10-21swapper/111:03:451
1932427615,42sleep219450-21wc10:33:342
1552327617,44sleep215601-21grep08:38:532
1552327617,44sleep215601-21grep08:38:532
148722768,10sleep00-21swapper/008:35:240
1201127615,43sleep212022-21modprobe08:28:312
2335427510,10sleep10-21swapper/107:19:281
2929827418,28sleep20-21swapper/211:03:582
1921627410,49sleep019306-21memory07:08:570
308627312,46sleep03129-21ntp_kernel_err11:23:580
2615827311,48sleep326160-21/usr/sbin/munin10:53:583
2609227315,40sleep026194-21/usr/sbin/munin12:34:060
1707727320,27sleep20-21swapper/208:43:542
1068327319,34sleep310712-21awk11:48:313
516927211,45sleep35240-21grep09:48:543
516927211,45sleep35240-21grep09:48:543
3180527218,36sleep231930-21grep07:48:352
2681827216,37sleep326758-21latency_hist12:38:333
2535727218,27sleep20-21swapper/212:33:362
1762127226,23sleep30-21swapper/307:03:563
1762127226,23sleep30-21swapper/307:03:563
252432717,31sleep10-21swapper/109:08:591
252432717,31sleep10-21swapper/109:08:591
1006927117,22sleep20-21swapper/211:44:512
944527021,24sleep30-21swapper/308:18:563
2746427014,24sleep20-21swapper/210:58:572
2562027023,24sleep30-21swapper/307:28:343
2293627023,21sleep20-21swapper/209:03:312
1168527012,45sleep011728-21ntp_kernel_pll_10:09:010
1808526922,19sleep00-21swapper/010:28:520
1410326911,43sleep214174-21systemctl10:18:322
1123826918,28sleep30-21swapper/311:48:563
671026820,24sleep30-21swapper/311:34:233
617626810,44sleep0362-21snmpd09:51:500
617626810,44sleep0362-21snmpd09:51:500
2267926810,37sleep222749-21cat12:24:192
1980026812,42sleep019844-21systemd-run08:53:320
146326817,18sleep00-21swapper/009:38:330
2470126712,39sleep0362-21snmpd07:23:590
2134826713,22sleep30-21swapper/308:58:323
2129126717,26sleep20-21swapper/207:14:002
2067926718,30sleep020712-21fschecks_count12:18:560
2067926718,30sleep020712-21fschecks_count12:18:560
1579226722,24sleep30-21swapper/310:23:343
123026713,35sleep31290-21/usr/sbin/munin07:53:533
123026713,35sleep31290-21/usr/sbin/munin07:53:533
1165426716,23sleep30-21swapper/310:09:003
820626615,35sleep38274-21switchtime11:39:033
565426616,28sleep30-21swapper/311:33:343
565426616,28sleep30-21swapper/311:33:343
240412669,16sleep00-21swapper/007:23:340
2295926612,37sleep322975-21/usr/sbin/munin10:43:563
872626515,35sleep38789-21/usr/sbin/munin09:59:033
2819626519,26sleep30-21swapper/309:18:553
2621126516,30sleep225796-21/usr/sbin/munin07:28:572
2567426510,39sleep225732-21if_eth012:33:562
2379526511,43sleep323796-21sshd07:21:153
1635226515,37sleep216356-21ntp_states12:04:012
796926415,33sleep28120-21latency_hist09:58:342
1691226416,24sleep30-21swapper/310:25:073
1560326424,22sleep30-21swapper/308:38:543
1560326424,22sleep30-21swapper/308:38:543
415326314,15sleep00-21swapper/008:03:350
3094726315,26sleep20-21swapper/211:09:062
2458526314,19sleep30-21swapper/312:29:093
239826219,18sleep20-21swapper/207:58:322
239826219,18sleep20-21swapper/207:58:322
1257426222,21sleep20-21swapper/208:28:562
2683626112,32sleep326914-21/usr/sbin/munin09:14:023
2084226114,26sleep220990-21/usr/sbin/munin08:54:072
1944926117,21sleep30-21swapper/307:09:043
138826118,27sleep31418-21munin-run09:38:323
479826020,18sleep20-21swapper/209:48:332
402626021,18sleep30-21swapper/311:28:333
209712607,13sleep20-21swapper/212:19:022
209712607,13sleep20-21swapper/212:19:022
826225922,19sleep00-21swapper/009:58:520
515025915,19sleep20-21swapper/208:04:172
261622596,12sleep00-21swapper/010:53:590
2478925917,26sleep024783-21apt-get09:08:460
2478925917,26sleep024783-21apt-get09:08:460
2350425912,36sleep019202-1kworker/0:0H12:26:470
3103125816,21sleep00-21swapper/011:09:080
2268625817,27sleep222181-21apt-get07:18:442
221252588,22sleep00-21swapper/008:59:010
2122325819,21sleep30-21swapper/310:38:523
1490625816,17sleep00-21swapper/010:19:030
1280625815,17sleep00-21swapper/008:29:060
940925715,22sleep00-21swapper/011:43:440
705525716,24sleep37061-21ntp_states09:54:033
3244525713,26sleep232486-21ntpq07:48:592
3090425713,27sleep330949-21ntp_states07:44:003
2707625714,29sleep027095-21munin-run07:33:310
2295625714,25sleep30-21swapper/309:03:323
69852565,39sleep30-21swapper/308:09:113
3082025615,17sleep20-21swapper/209:28:342
2487925615,22sleep224971-21perl09:08:522
2487925615,22sleep224971-21perl09:08:522
247925612,29sleep02616-21latency_hist11:23:340
2470725612,19sleep20-21swapper/210:49:042
2111425614,24sleep021133-21apt-get10:38:450
1799925612,28sleep317995-21apt-get10:28:463
136525613,29sleep21366-21sshd09:38:152
958225518,21sleep29183-21/usr/sbin/munin08:18:592
929325514,18sleep00-21swapper/008:18:530
672025515,19sleep00-21swapper/008:09:040
3110225513,26sleep031197-21sed07:44:050
2903225514,20sleep30-21swapper/307:38:513
2333525511,21sleep00-21swapper/010:44:060
1650525513,24sleep216561-21irqstats10:24:542
1156025512,24sleep211653-21memory10:08:592
1016925517,22sleep310188-21ntp_states10:04:003
83052549,15sleep00-21swapper/011:39:040
3048925415,19sleep30-21swapper/311:08:553
2611625420,19sleep30-21swapper/309:13:343
2611625420,19sleep30-21swapper/309:13:343
206825414,24sleep32079-21vmstat11:19:093
2006825415,20sleep20-21swapper/210:34:022
185152546,10sleep20-21swapper/207:07:222
185152546,10sleep20-21swapper/207:07:222
865925317,20sleep20-21swapper/209:59:022
424125313,24sleep24260-21apt-get11:28:442
424125313,24sleep24260-21apt-get11:28:442
2479925316,23sleep024877-21cat10:49:060
2317825315,16sleep20-21swapper/210:44:042
663325214,21sleep26709-21sed11:34:222
332825217,16sleep20-21swapper/211:24:042
2947725211,25sleep329471-21apt-get09:23:473
2736725211,23sleep327362-21apt-get07:33:473
184682526,35sleep00-21swapper/012:10:280
1698225213,24sleep016976-21apt-get08:43:480
761525112,23sleep37611-21apt-get08:13:473
1276725111,23sleep212753-21apt-get11:53:512
809525014,20sleep20-21swapper/208:14:022
1795325012,18sleep20-21swapper/212:09:052
1431425010,15sleep30-21swapper/311:58:453
3262224911,17sleep30-21swapper/307:49:053
3250124912,23sleep032507-21ntp_states07:49:010
200524911,20sleep02047-21users07:54:060
200524911,20sleep02047-21users07:54:060
188672494,10sleep30-21swapper/312:13:343
994124816,16sleep20-21swapper/210:03:552
973524812,18sleep00-21swapper/010:03:490
3009324812,15sleep00-21swapper/009:24:040
1433824811,22sleep014334-21apt-get11:58:460
65522478,14sleep20-21swapper/208:09:022
2738024712,20sleep327447-21df10:58:543
1956224716,19sleep30-21swapper/310:33:473
1900124712,18sleep00-21swapper/012:13:470
116724711,20sleep21163-21apt-get11:18:472
271824611,20sleep02712-21apt-get07:58:460
2630524514,13sleep20-21swapper/210:54:372
329624212,13sleep20-21swapper/207:59:052
289212429,17sleep00-21swapper/007:38:440
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional