You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 14:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Nov 05, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26827228797,2859sleep125168-21kworker/1:100:09:341
22140230223,262sleep322174-40ogrotate23:54:383
22140230223,262sleep322174-40ogrotate23:54:383
16762229665,211sleep116929-21latency_hist21:54:411
7879228896,173sleep17724-21/usr/sbin/munin19:40:001
17219223265,99sleep117199-21apt-get21:55:011
1803222690,64sleep10-21swapper/121:05:121
1655422177,197sleep116593-21ntpq23:35:031
22549221622,91sleep122694-21unixbench_multi22:10:101
29456221029,55sleep10-21swapper/122:34:411
19372220988,100sleep119434-21fschecks_time23:45:001
17179220438,120sleep117222-21fschecks_count20:10:031
25385220211,142sleep125421-21ntp_kernel_err22:20:071
2528322019,171sleep225304-21munin-plugin-st00:04:382
22265219017,156sleep222383-21date23:54:412
22265219017,156sleep222383-21date23:54:412
20028219026,143sleep2225-21systemd-journal22:04:382
20028219026,143sleep2225-21systemd-journal22:04:382
1005421888,159sleep19935-21/usr/sbin/munin23:15:011
19042218117,62sleep10-21swapper/123:44:391
28610216732,118sleep128742-21latency_hist19:04:401
28610216732,118sleep128742-21latency_hist19:04:401
13603215823,117sleep213648-21latency_hist21:44:412
4329215369,70sleep14371-21true19:29:391
2429321449,61sleep10-21swapper/100:00:011
6590214214,106sleep10-21swapper/119:35:041
1614214263,20sleep10-21swapper/119:19:581
31863214113,111sleep10-21swapper/100:24:551
31863214113,111sleep10-21swapper/100:24:551
1218721419,119sleep112326-21latency_hist19:54:421
31793213643,47sleep00-21swapper/000:24:500
31793213643,47sleep00-21swapper/000:24:500
8634213457,61sleep18637-21cron21:29:381
4670213366,20sleep10-21swapper/121:15:051
20198213215,23sleep10-21swapper/120:19:521
20198213215,23sleep10-21swapper/120:19:521
12694213224,93sleep10-21swapper/119:55:021
30882213029,86sleep131010-21cat00:20:111
28333212332,76sleep128439-21tune2fs20:44:581
5174212231,79sleep15244-21switchtime19:30:111
13693212212,23sleep213741-21systemd-run19:59:392
19206212036,16sleep10-21swapper/120:15:121
19206212036,16sleep10-21swapper/120:15:121
9572211834,20sleep19666-21memory19:45:051
30478211717,79sleep130623-21unixbench_multi20:50:111
1403621179,14sleep10-21swapper/119:59:581
13819211713,88sleep013838-21apt-get21:44:510
30827211613,23sleep030828-21apt-get19:09:520
25576211618,22sleep325643-21df00:04:553
28637211514,53sleep228684-21ntpq20:45:042
25116211423,15sleep10-21swapper/120:34:531
13890211426,65sleep10-21swapper/123:25:141
13890211426,65sleep10-21swapper/123:25:141
2421021137,89sleep10-21swapper/122:15:111
2176211324,64sleep32237-21idleruntime-cro00:34:383
210021135,13sleep00-21swapper/000:33:380
888211211,89sleep1927-21latency_hist21:04:401
22400211211,88sleep122532-21timerwakeupswit20:25:131
22400211211,88sleep122532-21timerwakeupswit20:25:131
13017211222,69sleep313165-21grep21:40:143
4478211115,19sleep00-21swapper/022:55:100
421211119,17sleep10-21swapper/122:44:581
16084211123,13sleep10-21swapper/120:05:231
746211011,79sleep0795-21/usr/sbin/munin22:45:050
27255211018,72sleep023611-21munin-node18:59:520
27255211018,72sleep023611-21munin-node18:59:520
14877210910,80sleep114913-21cat23:30:021
3020621079,18sleep10-21swapper/122:35:131
30382210617,72sleep330534-21unixbench_multi22:35:153
13742210518,67sleep313798-21latency_hist19:59:403
785321045,60sleep10-21swapper/123:06:461
21279210424,15sleep10-21swapper/123:50:041
21279210424,15sleep10-21swapper/123:50:041
10716210420,64sleep310201-21apt-get21:34:533
1018221039,20sleep10-21swapper/121:34:391
1018221039,20sleep10-21swapper/121:34:391
28083210132,37sleep00-21swapper/022:29:510
2683121017,21sleep10-21swapper/122:25:041
10341210131,39sleep30-21swapper/321:34:413
10341210131,39sleep30-21swapper/321:34:413
2436210015,64sleep22528-21memory22:50:242
2436210015,64sleep22528-21memory22:50:242
2032221007,26sleep119767-1kworker/1:1H22:04:541
2032221007,26sleep119767-1kworker/1:1H22:04:541
3059029920,15sleep10-21swapper/119:09:451
690729844,29sleep00-21swapper/023:04:590
56692989,76sleep1360-21ntpd23:00:041
2677229811,20sleep10-21swapper/120:39:581
2368629711,71sleep323698-21netstat20:30:033
1885029711,72sleep318855-21diskstats21:59:593
1885029711,72sleep318855-21diskstats21:59:593
2372729612,68sleep10-21swapper/123:59:331
1879529614,51sleep00-21swapper/020:15:030
1879529614,51sleep00-21swapper/020:15:030
1555029624,48sleep315545-21apt-get20:05:093
147632969,70sleep10-21swapper/121:45:201
321622955,58sleep10-21swapper/122:40:181
3014729513,22sleep230154-21ntp_states22:35:102
70702945,60sleep10-21swapper/121:24:301
1885729416,62sleep018879-21fschecks_count22:00:000
1885729416,62sleep018879-21fschecks_count22:00:000
9912925,74sleep10-21swapper/119:16:351
436729214,63sleep04458-21fschecks_time21:15:000
2791029214,62sleep228063-21latency_hist22:29:412
122962928,66sleep10-21swapper/121:39:551
118692925,74sleep10-21swapper/121:37:541
81629117,58sleep3300-21/usr/sbin/munin22:45:083
194302919,69sleep119561-21cat22:00:141
194302919,69sleep119561-21cat22:00:141
26852908,66sleep10-21swapper/121:09:531
120572908,12sleep10-21swapper/123:20:101
2471928915,56sleep024744-21cut22:19:380
2214128914,57sleep022230-21/usr/sbin/munin22:10:010
317192865,11sleep10-21swapper/120:59:271
88782858,60sleep10-21swapper/123:10:091
271262857,14sleep10-21swapper/100:09:541
1249628538,32sleep012502-21df_inode19:54:570
46772845,11sleep10-21swapper/122:55:481
297692847,67sleep1362-21snmpd00:17:001
2435528415,47sleep223435-21/usr/sbin/munin22:15:122
462528214,47sleep04695-21/usr/sbin/munin19:29:580
298612825,11sleep10-21swapper/119:06:181
298612825,11sleep10-21swapper/119:06:181
2633328215,50sleep026482-21unixbench_singl00:05:250
19402825,11sleep10-21swapper/100:31:371
2510228122,27sleep00-21swapper/022:20:020
440628020,42sleep34544-21cat19:29:413
3230028010,49sleep20-21swapper/200:25:082
3230028010,49sleep20-21swapper/200:25:082
263232807,64sleep00-21swapper/020:38:400
2032028026,26sleep00-21swapper/022:04:530
2032028026,26sleep00-21swapper/022:04:530
1198028014,53sleep3362-21snmpd19:53:023
172052799,11sleep00-21swapper/023:36:350
106072797,50sleep00-21swapper/023:15:120
32842785,11sleep10-21swapper/122:53:141
32842785,11sleep10-21swapper/122:53:141
3251327820,27sleep30-21swapper/319:15:003
245642785,11sleep10-21swapper/120:34:251
26872778,14sleep30-21swapper/321:09:553
2552427711,52sleep225548-21netstat20:35:192
224722777,47sleep00-21swapper/023:54:560
1781827723,38sleep317880-21fschecks_time23:39:573
913627614,31sleep00-21swapper/021:30:020
913627614,31sleep00-21swapper/021:30:020
144032769,53sleep014528-21date23:29:410
144032769,53sleep014528-21date23:29:410
1124827616,24sleep20-21swapper/221:35:372
2839327514,47sleep228469-21systemd-run00:14:392
1720227514,34sleep37460-1kworker/3:2H21:54:513
1594827521,34sleep316069-21latency_hist23:34:403
905827423,25sleep20-21swapper/219:44:422
3035627420,25sleep20-21swapper/200:20:002
2400927419,24sleep30-21swapper/322:15:093
2359327419,30sleep20-21swapper/220:30:022
99927313,45sleep21003-21diskstats00:29:572
3000427317,24sleep30-21swapper/320:50:013
3043527223,28sleep30-21swapper/300:20:013
1506827219,31sleep215076-21open_files23:30:072
1186627218,38sleep311880-21/usr/sbin/munin23:20:033
1094527215,37sleep211037-21fschecks_time19:50:022
3006927114,37sleep030100-21/usr/sbin/munin22:35:080
1586627124,31sleep015877-21modprobe23:34:370
1346327120,27sleep20-21swapper/223:25:072
1346327120,27sleep20-21swapper/223:25:072
1332827120,33sleep313038-21/usr/sbin/munin23:25:043
1332827120,33sleep313038-21/usr/sbin/munin23:25:043
2932927015,41sleep029352-21/usr/sbin/munin19:05:080
2932927015,41sleep029352-21/usr/sbin/munin19:05:080
1396227020,30sleep013981-21apt-get19:59:530
708926916,31sleep30-21swapper/323:05:023
1755526918,28sleep20-21swapper/220:10:132
589626820,22sleep00-21swapper/021:19:570
3210226816,32sleep332192-21fschecks_time20:59:543
284626812,39sleep32953-21sed22:50:333
284626812,39sleep32953-21sed22:50:333
868426718,35sleep38750-21latency_hist21:29:403
845626721,26sleep30-21swapper/323:09:583
665526716,33sleep06693-21proc_pri19:35:070
502926712,15sleep20-21swapper/222:59:402
3210126717,33sleep230298-21munin-node19:14:382
2874326723,22sleep30-21swapper/322:30:113
2677626717,26sleep20-21swapper/220:39:592
2471826718,31sleep00-21swapper/020:34:400
1724326711,41sleep017244-21dpkg21:55:200
1510126724,20sleep30-21swapper/321:49:383
1151426715,25sleep00-21swapper/023:19:560
922726616,32sleep29293-21/usr/sbin/munin21:30:042
922726616,32sleep29293-21/usr/sbin/munin21:30:042
665526625,23sleep30-21swapper/323:04:423
2898826623,22sleep00-21swapper/000:15:010
2585226616,31sleep225918-21/usr/sbin/munin22:20:172
239626617,22sleep30-21swapper/321:09:383
1585426621,24sleep00-21swapper/021:50:080
181262659,45sleep00-21swapper/020:11:440
783426410,32sleep07878-21fschecks_count19:39:590
750626415,22sleep20-21swapper/221:25:032
598426418,23sleep30-21swapper/319:34:423
524526414,20sleep20-21swapper/219:30:122
3091826411,37sleep330808-21/usr/sbin/munin19:09:593
172426420,23sleep20-21swapper/219:19:592
2955826320,21sleep30-21swapper/320:49:413
214402639,41sleep20-21swapper/220:24:092
214402639,41sleep20-21swapper/220:24:092
1716726319,27sleep317171-21/usr/sbin/munin20:10:013
150626327,19sleep20-21swapper/221:05:052
2324426214,17sleep30-21swapper/322:14:403
2050726220,24sleep320591-21memory20:20:063
2050726220,24sleep320591-21memory20:20:063
1955526216,31sleep019619-21latency23:45:030
1749626221,21sleep30-21swapper/323:39:413
96202617,45sleep20-21swapper/223:14:212
3129426113,27sleep00-21swapper/022:39:560
2723426112,27sleep00-21swapper/020:40:090
1998226115,27sleep20-21swapper/220:19:412
1998226115,27sleep20-21swapper/220:19:412
1717426113,33sleep216673-21apt-get21:54:462
2908726018,28sleep229123-21ntpq00:15:042
1970026013,18sleep30-21swapper/323:45:063
619325915,25sleep26189-21apt-get19:34:532
323522599,12sleep30-21swapper/300:25:093
323522599,12sleep30-21swapper/300:25:093
2792125922,21sleep30-21swapper/319:00:113
2792125922,21sleep30-21swapper/319:00:113
251322597,16sleep30-21swapper/300:02:413
2041425924,20sleep30-21swapper/322:05:003
2041425924,20sleep30-21swapper/322:05:003
1896625914,25sleep319039-21memory20:15:063
1896625914,25sleep319039-21memory20:15:063
1666325917,21sleep20-21swapper/223:35:082
152382596,43sleep30-21swapper/320:04:363
1142325919,18sleep00-21swapper/019:50:130
3122025815,23sleep30-21swapper/322:39:513
2979725816,25sleep029792-21apt-get20:49:540
210325821,23sleep00-21swapper/022:49:490
210325821,23sleep00-21swapper/022:49:490
1250625814,18sleep00-21swapper/021:40:030
735625714,20sleep20-21swapper/223:05:092
2903125711,20sleep20-21swapper/222:30:142
2892425718,16sleep00-21swapper/020:45:100
2534025710,13sleep30-21swapper/320:35:163
1908725717,24sleep219241-21cut23:44:402
1569625713,22sleep00-21swapper/020:05:140
102672577,11sleep20-21swapper/219:45:592
937325616,24sleep39379-21df_inode19:44:593
837425612,26sleep08441-21df23:09:550
582325618,21sleep20-21swapper/221:19:552
33025614,26sleep2322-21apt-get22:44:552
3241925615,23sleep232413-21apt-get19:14:552
317132565,38sleep00-21swapper/020:59:220
2533525611,23sleep00-21swapper/000:04:400
2308225615,23sleep00-21swapper/020:29:400
2308225615,23sleep00-21swapper/020:29:400
208532566,10sleep00-21swapper/023:49:470
208532566,10sleep00-21swapper/023:49:470
2069225618,23sleep020730-21processes20:20:110
2069225618,23sleep020730-21processes20:20:110
1976625613,27sleep219830-21/usr/sbin/munin23:45:092
764725517,21sleep00-21swapper/021:25:050
437225513,17sleep20-21swapper/219:29:402
3228525512,21sleep20-21swapper/220:59:572
3049325513,24sleep030538-21irqstats00:20:020
300225516,23sleep23052-21memory21:10:022
2758725512,29sleep327594-21open_files00:10:083
2705925517,24sleep327104-21ntp_kernel_pll_20:40:043
1665225517,22sleep30-21swapper/321:54:383
159425515,25sleep31600-21diskstats19:19:563
128225512,23sleep00-21swapper/019:19:410
113925511,26sleep01134-21apt-get21:04:550
667525413,25sleep36722-21users21:20:123
306252546,19sleep20-21swapper/220:50:122
2703625419,19sleep30-21swapper/322:25:103
2175025413,24sleep221769-21apt-get20:24:542
2175025413,24sleep221769-21apt-get20:24:542
2077625410,29sleep220782-21ntp_states22:05:072
2077625410,29sleep220782-21ntp_states22:05:072
582325313,20sleep20-21swapper/223:00:092
2557525312,24sleep225569-21apt-get00:04:542
1539325313,24sleep215389-21apt-get21:49:532
1391625317,17sleep20-21swapper/221:44:572
104672535,37sleep00-21swapper/019:48:270
1000425313,24sleep310031-21df23:14:553
283225213,23sleep02926-21sed19:24:410
1773025214,22sleep017726-21apt-get23:39:500
149022525,35sleep20-21swapper/220:00:202
526525113,21sleep05261-21apt-get22:59:500
363325114,18sleep00-21swapper/019:25:100
3124525112,23sleep231239-21apt-get22:39:542
2749025113,22sleep227547-21irqstats19:00:022
2749025113,22sleep227547-21irqstats19:00:022
227422519,15sleep30-21swapper/323:55:033
1276925112,23sleep212814-21systemd-run23:24:382
834925014,17sleep30-21swapper/319:40:103
1615625013,17sleep20-21swapper/220:05:242
427524912,21sleep34270-21apt-get21:14:553
2797524910,22sleep028028-21users00:10:130
2656024911,23sleep026555-21apt-get22:24:550
2528524917,17sleep30-21swapper/322:20:053
368324813,20sleep33702-21apt-get22:54:493
222572488,17sleep30-21swapper/320:25:113
222572488,17sleep30-21swapper/320:25:113
1049424816,16sleep00-21swapper/021:34:420
64652476,16sleep30-21swapper/319:35:023
2932124713,17sleep30-21swapper/319:05:073
2932124713,17sleep30-21swapper/319:05:073
471124614,16sleep20-21swapper/221:15:062
402424617,16sleep00-21swapper/021:14:400
2886524610,20sleep228861-21apt-get19:04:542
2886524610,20sleep228861-21apt-get19:04:542
99342459,21sleep29953-21apt-get23:14:502
320182459,19sleep032088-21df20:59:510
222462459,18sleep20-21swapper/222:10:032
3178424210,17sleep20-21swapper/220:59:402
2879124010,14sleep30-21swapper/300:14:563
224692398,16sleep20-21swapper/223:54:542
313462288,12sleep30-21swapper/320:54:493
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional