You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 08:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Jul 01, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
109942117711,1155sleep211029-1kworker/2:2H23:29:572
109942117711,1155sleep211029-1kworker/2:2H23:29:572
23744292611,871sleep323788-21unin-run00:14:383
11940264616,616sleep111960-21users20:05:161
23201262117,587sleep123268-21/usr/sbin/munin18:59:521
23201262117,587sleep123268-21/usr/sbin/munin18:59:521
29095242771,330sleep329255-21unixbench_multi21:00:123
24067242715,396sleep324159-21fschecks_time22:30:003
24067242715,396sleep324159-21fschecks_time22:30:003
31624233061,139sleep30-21swapper/321:09:583
2661322969,272sleep326639-21latency_hist20:54:393
6158227615,199sleep36198-21ntp_kernel_pll_23:15:043
20123225512,229sleep020131-21ntp_states18:50:100
97282225113,53sleep30-21swapper/321:44:403
14561222011,197sleep314638-21switchtime23:40:083
2248622179,192sleep322547-21latency20:40:373
23822221650,55sleep30-21swapper/320:44:573
31247221443,151sleep331250-21cron21:09:373
29600221022,167sleep129671-21sort19:20:041
21673220613,128sleep321809-21unixbench_singl00:05:133
27946219011,168sleep1370-21snmpd20:57:271
281652184113,55sleep328143-21latency_hist20:59:393
24759217815,107sleep324897-21sed00:15:103
1405217520,140sleep01424-21idleruntime-cro19:34:370
14702217336,119sleep30-21swapper/320:15:123
14702217336,119sleep30-21swapper/320:15:123
22517217115,136sleep322598-21fschecks_time22:24:593
4889216734,111sleep30-21swapper/323:10:113
11059216530,115sleep30-21swapper/320:04:583
1758215914,130sleep31886-21unixbench_multi23:00:093
23054215616,20sleep30-21swapper/318:59:463
23054215616,20sleep30-21swapper/318:59:463
22304215520,113sleep022367-21cut20:40:340
1510521548,132sleep315164-21find22:00:113
28078215258,59sleep30-21swapper/319:15:013
3619214614,115sleep13658-21ntp_kernel_pll_19:40:041
28290214614,117sleep00-21swapper/022:44:070
2927821418,121sleep329322-21ntpq22:45:193
20688214163,57sleep321030-21cyclictest00:04:403
12923214115,108sleep30-21swapper/323:35:073
21036214019,101sleep30-21swapper/322:20:023
276721395,12sleep00-21swapper/023:05:030
19101213916,16sleep30-21swapper/323:59:393
11648213929,89sleep30-21swapper/321:50:013
19281213869,20sleep30-21swapper/320:30:073
15596213833,93sleep315644-21awk23:44:533
15596213833,93sleep315644-21awk23:44:533
26098213412,63sleep326224-21latency_hist19:09:403
19672213412,20sleep30-21swapper/322:15:053
267121308,58sleep32747-21mailstats21:20:123
2332421297,105sleep00-21swapper/022:25:120
30740212813,99sleep130853-21unixbench_singl21:05:151
30027212814,50sleep30-21swapper/319:20:143
29673212815,94sleep20-21swapper/219:20:052
16743212615,51sleep30-21swapper/322:05:123
16743212615,51sleep30-21swapper/322:05:123
24163212514,97sleep024201-21idleruntime22:30:010
24163212514,97sleep024201-21idleruntime22:30:010
26781212410,75sleep31-21systemd22:39:373
19483212411,96sleep219595-21timerwakeupswit20:30:102
22556212311,93sleep022583-21sort18:59:390
10731212311,61sleep30-21swapper/320:04:383
13823212222,74sleep213974-21unixbench_singl21:55:172
3105121219,72sleep331131-21timerandwakeup22:50:123
3087821210,13sleep20-21swapper/221:05:152
2377821207,85sleep20-21swapper/220:44:562
931421188,75sleep39352-21ls23:25:013
931421188,75sleep39352-21ls23:25:013
20378211823,71sleep120513-21grep18:50:151
7517211717,80sleep07032-21apt-get23:19:490
7466211722,50sleep30-21swapper/321:35:103
389211710,73sleep3410-21irqstats19:30:003
1147211714,68sleep31235-21mailstats21:15:113
26734211523,50sleep30-21swapper/319:10:043
13052211554,40sleep30-21swapper/321:54:543
18788211420,59sleep30-21swapper/322:11:503
22161211322,69sleep121650-21apt-get20:39:501
25577211220,65sleep10-21swapper/100:19:531
12220211249,16sleep30-21swapper/323:34:393
12220211249,16sleep30-21swapper/323:34:393
694321117,90sleep3370-21snmpd19:50:073
18468211119,75sleep218625-21tail23:55:062
30085210720,70sleep030136-21fschecks_count21:05:020
25500210525,48sleep125521-21apt-get22:34:511
29752210224,58sleep229838-21grep21:04:392
26009210213,74sleep226074-21mailstats22:35:082
21473210214,74sleep121589-21cat22:20:121
20991210212,75sleep020350-21/usr/sbin/munin20:35:080
14279210214,71sleep00-21swapper/023:40:010
11702210218,61sleep00-21swapper/023:30:240
11702210218,61sleep00-21swapper/023:30:240
1607621016,79sleep00-21swapper/023:45:060
1607621016,79sleep00-21swapper/023:45:060
2959229928,48sleep229756-21unixbench_multi22:45:262
2845929917,64sleep028528-21/usr/sbin/munin19:15:100
2034929933,32sleep10-21swapper/120:34:491
13152990,11sleep10-21swapper/121:15:121
1044929916,63sleep210508-21/usr/sbin/munin21:45:112
973829817,65sleep29762-21kernelversion20:00:042
1943529813,67sleep119533-21fschecks_time23:59:571
168029710,71sleep21757-21/usr/sbin/munin23:00:082
3236129614,42sleep332472-21date19:29:393
2129929629,50sleep220906-21/usr/sbin/munin00:05:032
188929620,51sleep21938-21unin-run21:19:382
803929533,13sleep30-21swapper/323:20:373
49729514,67sleep0499-21sshd22:55:500
1727329524,15sleep30-21swapper/320:24:533
126462957,17sleep00-21swapper/023:35:000
324229410,35sleep30-21swapper/319:39:533
1192629415,63sleep011970-21ntpq21:50:060
33029315,57sleep2350-21awk21:14:372
322002929,67sleep30-21swapper/322:55:003
1752229218,22sleep20-21swapper/222:09:392
1752229218,22sleep20-21swapper/222:09:392
2379129116,24sleep10-21swapper/100:14:401
495329020,13sleep30-21swapper/319:44:583
495329020,13sleep30-21swapper/319:44:583
869928912,16sleep30-21swapper/319:55:163
996828829,21sleep39964-21apt-get21:44:523
2707428820,14sleep30-21swapper/322:39:533
2679328813,56sleep126797-21ntp_states19:10:061
2022128710,63sleep220244-21users00:00:112
879628611,55sleep20-21swapper/221:40:062
312828617,15sleep30-21swapper/323:05:103
3015228622,44sleep130180-21grep22:49:371
1257528638,32sleep30-21swapper/320:09:513
1257528638,32sleep30-21swapper/320:09:513
907328514,50sleep09199-21/usr/sbin/munin21:40:120
637128526,33sleep00-21swapper/019:49:520
622528517,44sleep20-21swapper/223:15:072
1930328514,53sleep20-21swapper/222:14:562
119812850,6sleep20-21swapper/221:50:072
1042928515,53sleep010476-21apt-config23:29:390
1042928515,53sleep010476-21apt-config23:29:390
2764628417,32sleep00-21swapper/020:55:120
1773728421,47sleep117774-21ps20:25:081
383328310,54sleep30-21swapper/321:24:563
3174828321,29sleep20-21swapper/222:54:392
607228218,46sleep16208-21/usr/sbin/munin21:30:201
236542827,55sleep00-21swapper/019:00:040
236542827,55sleep00-21swapper/019:00:040
491228114,44sleep24951-21grep19:44:572
491228114,44sleep24951-21grep19:44:572
2326928112,50sleep20-21swapper/218:59:532
2326928112,50sleep20-21swapper/218:59:532
494228014,52sleep0227-21systemd-journal21:28:450
2014128038,26sleep20-21swapper/218:50:122
1024528033,23sleep00-21swapper/021:45:050
666627912,51sleep16808-21modprobe21:34:401
644027914,45sleep20-21swapper/219:49:542
51727912,50sleep2563-21/usr/sbin/munin19:30:032
1761527914,14sleep00-21swapper/023:54:390
449827719,30sleep10-21swapper/121:25:101
2528527721,37sleep125311-21munin-run00:19:371
2925027618,21sleep20-21swapper/219:19:412
2258427618,32sleep122624-21munin-node18:59:401
2183027613,41sleep30-21swapper/318:55:103
801627522,22sleep20-21swapper/219:55:032
3009627512,42sleep030193-21timerandwakeup19:20:150
27312758,56sleep00-21swapper/019:36:470
174722757,57sleep00-21swapper/023:54:150
2492327420,35sleep024983-21grep19:05:080
554427338,19sleep30-21swapper/321:30:053
175512738,53sleep317613-21sed23:54:383
1629427317,22sleep00-21swapper/022:05:000
1629427317,22sleep00-21swapper/022:05:000
3114627222,26sleep00-21swapper/019:25:040
2168127227,25sleep20-21swapper/220:39:392
2142127217,35sleep020906-21/usr/sbin/munin00:05:100
1985327230,23sleep30-21swapper/318:50:013
1439127224,23sleep20-21swapper/223:40:032
1317527224,23sleep10-21swapper/121:55:041
1083527222,32sleep110975-21latency_hist20:04:401
2967227118,24sleep00-21swapper/021:04:370
295042718,11sleep10-21swapper/121:02:281
237727122,24sleep10-21swapper/119:35:111
215842716,45sleep321030-21cyclictest20:38:553
2712527018,32sleep227140-21kernelversion20:55:022
182242708,15sleep20-21swapper/222:10:122
265132685,11sleep30-21swapper/300:20:323
2350726819,32sleep20-21swapper/220:44:402
594826716,26sleep00-21swapper/021:30:180
2683126717,33sleep026845-21modprobe00:24:360
1534226719,25sleep215482-21grep22:00:142
673826613,34sleep16764-21netstat19:50:011
637826620,31sleep16494-21timerandwakeup23:15:101
1695126615,15sleep30-21swapper/320:24:373
3165226516,26sleep00-21swapper/021:09:590
272832656,13sleep00-21swapper/019:10:380
216426515,21sleep20-21swapper/219:35:082
19072656,37sleep30-21swapper/319:35:013
1750026516,24sleep10-21swapper/122:09:381
1750026516,24sleep10-21swapper/122:09:381
1566926519,19sleep00-21swapper/020:19:490
1477926522,22sleep00-21swapper/020:15:130
1477926522,22sleep00-21swapper/020:15:130
146142658,11sleep10-21swapper/120:15:101
146142658,11sleep10-21swapper/120:15:101
1350126519,23sleep20-21swapper/220:10:142
1350126519,23sleep20-21swapper/220:10:142
120412657,45sleep00-21swapper/020:06:120
113202658,12sleep10-21swapper/123:30:161
113202658,12sleep10-21swapper/123:30:161
3245026413,34sleep132456-21/usr/sbin/munin22:55:061
3193526421,25sleep10-21swapper/121:10:041
255326416,32sleep22576-21/usr/sbin/munin21:20:072
253372646,37sleep20-21swapper/219:05:192
780326314,24sleep20-21swapper/223:20:322
739926312,25sleep20-21swapper/221:35:092
73132636,13sleep00-21swapper/021:35:070
1927626316,29sleep119287-21acpi22:14:511
1778626316,27sleep117780-21apt-get23:54:481
2744326214,33sleep227473-21awk22:40:052
2530726217,27sleep125300-21apt-get20:49:561
2885526115,27sleep00-21swapper/022:44:470
2435426119,20sleep10-21swapper/120:45:111
218626114,29sleep02180-21apt-get21:19:560
1686026117,23sleep216886-21munin-run23:49:362
1686026117,23sleep216886-21munin-run23:49:362
85926015,18sleep20-21swapper/222:59:402
3113626016,24sleep031283-21grep22:50:130
3082326020,23sleep10-21swapper/122:50:061
1615626024,19sleep10-21swapper/120:20:071
142726018,27sleep11456-21munin-run19:34:381
1254726021,24sleep212559-21fschecks_count23:34:582
2060825918,26sleep020619-21modprobe00:04:370
2060625917,21sleep00-21swapper/022:19:390
108025916,26sleep11073-21apt-get22:59:511
940125818,19sleep10-21swapper/123:25:031
940125818,19sleep10-21swapper/123:25:031
613325815,30sleep06233-21latency_hist19:49:400
613325815,30sleep06233-21latency_hist19:49:400
2613925814,30sleep026232-21sed22:35:100
2419125817,22sleep00-21swapper/020:45:060
130712585,14sleep00-21swapper/021:54:550
1269725819,21sleep10-21swapper/120:10:011
1269725819,21sleep10-21swapper/120:10:011
584125712,18sleep00-21swapper/023:14:570
2373825720,23sleep223743-21modprobe00:14:372
1735925710,33sleep017388-21fschecks_count20:24:590
1489225718,19sleep00-21swapper/022:00:040
1463625716,17sleep20-21swapper/220:15:112
1463625716,17sleep20-21swapper/220:15:112
1068125718,23sleep10-21swapper/121:45:141
446325615,25sleep14486-21irqstats23:10:011
3155425615,19sleep20-21swapper/219:25:142
3106325616,24sleep131130-21df19:25:011
2528625619,20sleep20-21swapper/222:34:392
2528625619,20sleep20-21swapper/222:34:392
2489925616,24sleep024932-21users00:15:110
2266025615,24sleep122700-21irqstats22:25:011
2084925614,25sleep020844-21apt-get22:19:550
1392325613,20sleep00-21swapper/020:14:400
1392325613,20sleep00-21swapper/020:14:400
819925513,19sleep10-21swapper/119:55:061
282025514,25sleep12942-21timerwakeupswit21:20:141
251582554,11sleep20-21swapper/200:18:002
2243125516,23sleep222425-21apt-get22:24:542
2013425512,29sleep020211-21sed20:34:400
1750025514,25sleep117505-21modprobe23:54:361
90732546,39sleep00-21swapper/019:58:160
838025414,24sleep18392-21apt-get21:39:531
42162545,18sleep00-21swapper/019:41:000
41852545,37sleep00-21swapper/023:09:480
348425413,23sleep13576-21users23:05:161
2716125411,27sleep10-21swapper/122:40:001
2224125418,15sleep10-21swapper/100:09:391
174812546,11sleep20-21swapper/220:25:012
2538725315,21sleep025391-21diskstats20:49:590
2248325311,26sleep00-21swapper/000:09:540
220752536,10sleep20-21swapper/222:24:092
166892535,17sleep10-21swapper/123:47:251
166892535,17sleep10-21swapper/123:47:251
1507825313,24sleep115084-21ntp_states22:00:091
1437325316,24sleep114384-21netstat23:40:021
123725314,17sleep00-21swapper/021:15:120
759125214,22sleep17536-21apt-get23:20:231
2797125212,27sleep127982-21awk19:14:591
2420525212,21sleep124249-21irqstats22:30:021
2420525212,21sleep124249-21irqstats22:30:021
2175025210,21sleep20-21swapper/218:55:092
1833125212,16sleep00-21swapper/022:10:140
1780425212,22sleep117828-21df22:09:581
169382525,35sleep20-21swapper/220:24:282
1113925213,24sleep211142-21diskstats20:05:012
424025110,24sleep24302-21/usr/sbin/munin21:25:072
1939725116,18sleep10-21swapper/120:30:091
1246225111,24sleep112457-21apt-get23:34:531
421625014,17sleep20-21swapper/223:09:542
2604725011,22sleep126095-21systemd-run19:09:381
2567425013,23sleep225708-21/usr/sbin/munin20:50:052
2564825013,22sleep225651-21df_inode00:19:542
62822495,33sleep20-21swapper/221:31:052
324124912,21sleep23237-21apt-get19:39:522
189024911,22sleep01982-21users23:00:100
508624815,18sleep10-21swapper/119:45:001
508624815,18sleep10-21swapper/119:45:001
2713124810,16sleep20-21swapper/219:10:112
984024712,21sleep09447-21/usr/sbin/munin20:00:060
1197224711,22sleep111979-21ntp_states21:50:071
292224612,19sleep20-21swapper/223:05:052
959624510,20sleep29597-21ntp_states23:25:132
959624510,20sleep29597-21ntp_states23:25:132
175062439,19sleep20-21swapper/223:54:372
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional