You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-11 - 13:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Jan 11, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1569210177,995sleep21639-21/usr/sbin/munin19:09:592
14625290711,880sleep1656-1kworker/1:1H21:31:231
14625290711,880sleep1656-1kworker/1:1H21:31:231
648827106,693sleep028871-1kworker/0:3H00:31:180
27881265713,626sleep027957-21/usr/sbin/munin23:55:120
27881265713,626sleep027957-21/usr/sbin/munin23:55:120
26760264217,610sleep326777-21/usr/sbin/munin22:09:533
2884426338,616sleep37694-1kworker/3:1H22:16:193
23389261212,591sleep1656-1kworker/1:1H20:15:241
23389261212,591sleep1656-1kworker/1:1H20:15:241
3300260319,568sleep33333-21/usr/sbin/munin20:55:073
1625325806,563sleep216289-21ntpq21:35:082
3236924709,446sleep032405-21ntpq00:10:060
24277240322,363sleep024368-21fschecks_time23:45:030
1719523856,369sleep230095-21snmpd19:56:142
2540123678,349sleep230095-21snmpd23:49:132
409623506,336sleep37694-1kworker/3:1H00:23:423
22637232220,278sleep022843-21latency_hist20:14:460
527422886,276sleep04744-1kworker/0:3H21:01:100
570722658,219sleep25807-21timerandwakeup19:20:122
13158224537,160sleep213262-21timerandwakeup21:25:142
23212224337,187sleep223285-21switchtime23:40:162
24233223215,202sleep124255-21df23:45:001
29348222717,191sleep129469-21wc22:17:021
6997222034,137sleep27150-21unixbench_multi21:05:162
20253221446,106sleep220393-21grep23:30:152
7699221224,56sleep20-21swapper/222:50:142
24900220717,133sleep224936-21ntpq20:20:092
24900220717,133sleep224936-21ntpq20:20:092
3258218164,53sleep20-21swapper/200:21:022
30723217947,59sleep20-21swapper/220:39:562
28140216918,138sleep228256-21cat22:14:452
886221686,154sleep028871-1kworker/0:3H00:36:310
14949216533,118sleep215025-21systemctl21:31:322
14949216533,118sleep215025-21systemctl21:31:322
8220216344,79sleep28358-21date19:29:452
18426216335,57sleep20-21swapper/223:25:102
18426216335,57sleep20-21swapper/223:25:102
1038621629,134sleep210427-21ntp_kernel_pll_19:35:112
9852216143,53sleep29875-21munin-run22:59:432
20039215810,136sleep220056-21open_inodes20:05:112
28692215729,84sleep228603-40mandb23:59:472
2696215728,111sleep22782-21latency_hist20:54:462
5037215640,62sleep20-21swapper/221:00:192
11269215618,123sleep211307-21latency21:20:052
11269215618,123sleep211307-21latency21:20:052
16387215024,51sleep20-21swapper/223:19:532
16387215024,51sleep20-21swapper/223:19:532
21815214869,64sleep221818-21cron21:54:422
27684214571,17sleep20-21swapper/220:30:022
28048214221,101sleep028077-21idleruntime-cro22:14:430
13310214045,38sleep20-21swapper/219:45:042
13201213924,94sleep113308-21grep19:45:031
2694421385,11sleep10-21swapper/120:25:171
12459213712,57sleep20-21swapper/223:05:162
11297213711,73sleep20-21swapper/223:02:582
22366213216,57sleep20-21swapper/221:55:062
9694212712,99sleep314386-21systemd-journal19:34:433
5912212728,73sleep00-21swapper/000:29:450
24051212710,50sleep20-21swapper/222:00:072
31951212644,59sleep20-21swapper/200:09:522
15325212513,91sleep115399-21/usr/sbin/munin23:15:121
107921215,33sleep20-21swapper/200:14:572
5519212030,19sleep20-21swapper/200:27:082
32188212023,78sleep332248-21grep22:25:123
7054211836,67sleep27133-21xargs00:32:132
3454211818,81sleep13478-21proc_pri00:21:101
1752211869,31sleep31775-21/usr/sbin/munin20:50:063
3883211511,85sleep00-21swapper/019:15:060
3883211511,85sleep00-21swapper/019:15:060
2139921159,48sleep20-21swapper/220:10:082
938621129,91sleep29406-21dpkg21:14:532
9173211229,70sleep29216-21date21:14:442
29268211233,61sleep20-21swapper/220:35:032
19519211143,44sleep20-21swapper/221:45:132
5708210912,84sleep35709-21date00:29:153
17460210751,39sleep20-21swapper/221:39:562
17460210751,39sleep20-21swapper/221:39:562
14378210631,17sleep20-21swapper/223:12:432
26986210516,71sleep127033-21if_eth022:10:031
6290210414,73sleep15787-21apt-get00:29:541
19231210460,11sleep20-21swapper/220:03:582
16886210412,71sleep00-21swapper/019:55:150
8440210312,76sleep08475-21cpu19:30:000
13036210316,65sleep313166-21latency_hist23:09:453
27307210240,47sleep20-21swapper/220:29:422
5763210115,28sleep30-21swapper/322:45:063
1910210127,18sleep20-21swapper/222:32:342
8038210023,60sleep18130-21fschecks_time00:36:121
1509429915,60sleep20-21swapper/223:15:052
680529717,56sleep16866-21latency_hist22:49:441
918929427,52sleep20-21swapper/200:39:442
324872936,13sleep10-21swapper/120:45:081
190112939,71sleep019022-21cpu21:44:590
2210329213,17sleep00-21swapper/021:54:560
2927928917,60sleep029347-21expr22:17:010
2854428934,15sleep20-21swapper/222:14:532
2570528924,50sleep20-21swapper/222:05:122
230052895,71sleep20-21swapper/220:15:042
230052895,71sleep20-21swapper/220:15:042
2121628920,50sleep321281-21uniq20:10:043
1473828928,46sleep20-21swapper/219:50:012
313728813,60sleep03200-21/usr/sbin/munin00:21:000
3069128831,43sleep230132-21/usr/sbin/munin22:20:102
2105328819,53sleep20-21swapper/223:34:552
938228716,17sleep20-21swapper/219:30:422
445628710,18sleep20-21swapper/220:59:462
3076228734,25sleep30-21swapper/322:20:113
130528732,26sleep00-21swapper/019:09:440
855628613,59sleep08675-21unixbench_multi21:10:140
285362859,33sleep228586-40ogrotate23:59:422
285362859,33sleep228586-40ogrotate23:59:422
1384928529,33sleep30-21swapper/321:29:433
497928313,56sleep15036-21/usr/sbin/munin21:00:181
314422835,13sleep20-21swapper/200:06:332
314422835,13sleep20-21swapper/200:06:332
245328310,56sleep20-21swapper/222:35:082
13282838,16sleep10-21swapper/100:15:041
651228216,18sleep30-21swapper/321:05:043
33692829,61sleep23440-21expr19:14:512
33692829,61sleep23440-21expr19:14:512
750428115,48sleep37536-21/usr/sbin/munin22:50:113
228832819,16sleep00-21swapper/020:14:500
228832819,16sleep00-21swapper/020:14:500
135228116,50sleep11506-21latency_hist19:09:451
119282819,15sleep20-21swapper/219:40:072
2694828020,12sleep20-21swapper/223:54:132
2694828020,12sleep20-21swapper/223:54:132
2590528010,46sleep125214-21/usr/sbin/munin22:05:151
1170328016,31sleep10-21swapper/123:04:541
1124828021,43sleep31-21systemd19:39:443
606527917,33sleep30-21swapper/321:04:443
3254327919,16sleep20-21swapper/220:45:092
3180527916,13sleep20-21swapper/222:25:032
1823927914,21sleep00-21swapper/020:00:280
1806727916,31sleep30-21swapper/321:40:143
1806727916,31sleep30-21swapper/321:40:143
1043627913,50sleep310474-21wc23:00:043
73927842,19sleep20-21swapper/219:05:152
391927817,12sleep20-21swapper/222:39:522
306127820,40sleep03197-21unixbench_multi22:35:220
194762789,57sleep119518-21processes21:45:121
1491227811,51sleep314949-21fschecks_count23:15:023
948427712,42sleep00-21swapper/021:14:590
890827718,29sleep10-21swapper/119:30:131
403227722,27sleep00-21swapper/022:40:030
2231427616,44sleep022333-21apt23:39:420
213612760,11sleep10-21swapper/121:50:161
54932757,14sleep20-21swapper/222:44:572
2424027522,26sleep00-21swapper/022:00:130
2912827421,26sleep10-21swapper/120:34:541
213027419,29sleep30-21swapper/322:34:453
1782927424,27sleep00-21swapper/021:40:060
1782927424,27sleep00-21swapper/021:40:060
2641727315,42sleep026025-21/usr/sbin/munin20:25:080
1511427321,36sleep015218-21timerwakeupswit21:31:340
1511427321,36sleep015218-21timerwakeupswit21:31:340
1012927318,31sleep10-21swapper/119:35:051
2850027219,28sleep00-21swapper/020:30:170
2727627218,41sleep127313-21grep23:54:571
2727627218,41sleep127313-21grep23:54:571
1977027211,30sleep00-21swapper/023:30:050
859027118,34sleep18643-21cpuspeed22:54:571
776027112,42sleep37899-21/usr/sbin/munin19:27:203
1657727114,39sleep116614-21ntp_kernel_err19:55:081
3134027018,34sleep131443-21grep20:40:151
3099727022,21sleep00-21swapper/020:40:060
2890127015,35sleep328933-21latency_hist20:34:443
1940727011,46sleep119540-21modprobe20:04:451
1391927013,41sleep014062-21find21:29:440
1388927026,28sleep313995-21sshd23:10:133
1796026913,39sleep00-21swapper/023:24:560
1796026913,39sleep00-21swapper/023:24:560
835426816,30sleep18494-21latency_hist22:54:451
406426812,41sleep14154-21fschecks_time22:40:041
2655026822,22sleep30-21swapper/322:09:453
2602426810,12sleep30-21swapper/320:24:553
2219126824,25sleep30-21swapper/321:55:033
1999426819,27sleep30-21swapper/323:30:123
1143626823,31sleep111475-21modprobe23:04:421
713526725,23sleep30-21swapper/300:32:143
684926719,22sleep00-21swapper/021:05:140
309132679,17sleep30-21swapper/300:05:133
309132679,17sleep30-21swapper/300:05:133
1453726722,24sleep00-21swapper/023:14:420
371026622,20sleep30-21swapper/322:39:453
3048826622,21sleep30-21swapper/320:39:453
301582667,49sleep00-21swapper/020:36:310
1855126615,33sleep118625-21mailstats20:00:381
125632666,12sleep00-21swapper/019:41:260
124126624,22sleep30-21swapper/319:09:433
919626517,31sleep09297-21sed22:55:130
912326510,40sleep39194-21sensors_temp22:55:123
759426519,28sleep17657-21/usr/sbin/munin19:27:181
3019226516,26sleep30-21swapper/300:04:443
2619626516,27sleep00-21swapper/023:50:120
2619626516,27sleep00-21swapper/023:50:120
2064326516,20sleep30-21swapper/321:50:033
147082658,16sleep00-21swapper/019:49:580
630626419,23sleep10-21swapper/121:04:581
56726420,23sleep00-21swapper/022:29:450
3104126412,36sleep131130-21timerandwakeup00:05:151
3104126412,36sleep131130-21timerandwakeup00:05:151
2581526417,24sleep30-21swapper/323:50:023
2581526417,24sleep30-21swapper/323:50:023
2575826419,22sleep30-21swapper/320:24:423
2575826419,22sleep30-21swapper/320:24:423
2463526412,30sleep30-21swapper/323:45:103
1794226416,22sleep10-21swapper/121:40:111
1794226416,22sleep10-21swapper/121:40:111
1502626411,27sleep30-21swapper/321:31:333
1502626411,27sleep30-21swapper/321:31:333
129126418,22sleep00-21swapper/000:15:030
12542640,9sleep30-21swapper/320:49:453
1245626419,23sleep30-21swapper/319:40:163
274726317,22sleep10-21swapper/122:35:151
530726215,22sleep00-21swapper/019:20:020
371026215,22sleep10-21swapper/119:15:021
371026215,22sleep10-21swapper/119:15:021
25832626,11sleep00-21swapper/020:53:550
2186626223,16sleep00-21swapper/021:54:440
215322625,38sleep10-21swapper/120:10:111
1656726217,28sleep116694-21/usr/sbin/munin21:35:151
977926116,24sleep30-21swapper/321:15:063
600626120,19sleep10-21swapper/100:29:461
3179326110,23sleep10-21swapper/122:25:021
2703926119,18sleep10-21swapper/123:54:451
2703926119,18sleep10-21swapper/123:54:451
2361426116,16sleep30-21swapper/320:15:303
2361426116,16sleep30-21swapper/320:15:303
2014526115,31sleep10-21swapper/123:30:141
1799126111,26sleep317485-21apt-get19:59:553
1526626117,28sleep30-21swapper/319:50:143
145326112,27sleep30-21swapper/300:15:063
61332607,39sleep10-21swapper/119:21:491
290626015,30sleep12923-21/usr/sbin/munin20:54:531
2896226012,32sleep129022-21fschecks_time00:00:021
2861626019,25sleep328603-40mandb23:59:433
2861626019,25sleep328603-40mandb23:59:433
2511126022,18sleep00-21swapper/020:20:150
2511126022,18sleep00-21swapper/020:20:150
1585926014,31sleep015861-21/usr/sbin/munin21:34:530
3269225921,17sleep30-21swapper/300:10:133
232282597,40sleep10-21swapper/121:57:561
1803625916,26sleep318042-21/usr/sbin/munin23:24:593
1803625916,26sleep318042-21/usr/sbin/munin23:24:593
1688025929,15sleep00-21swapper/023:20:120
1688025929,15sleep00-21swapper/023:20:120
2416925816,18sleep30-21swapper/322:00:123
165225814,22sleep10-21swapper/119:10:021
1132225822,17sleep00-21swapper/019:39:450
941125711,22sleep10-21swapper/121:14:561
725425712,30sleep07309-21interrupts22:50:040
2520225719,20sleep10-21swapper/120:20:161
2520225719,20sleep10-21swapper/120:20:161
2239525715,19sleep30-21swapper/323:39:453
1900925715,26sleep319002-21apt-get21:44:583
1355725717,16sleep00-21swapper/019:45:100
1014325711,19sleep00-21swapper/022:59:530
3168825617,25sleep031693-21modprobe00:09:420
3168825617,25sleep031693-21modprobe00:09:420
2365925616,19sleep10-21swapper/121:59:541
2263825617,18sleep30-21swapper/323:40:013
1680325616,22sleep30-21swapper/319:55:143
1482925613,29sleep014896-21df23:14:590
1220425617,20sleep10-21swapper/119:40:141
2582525514,29sleep00-21swapper/022:05:140
847125415,24sleep38550-21wc21:10:133
78322546,38sleep10-21swapper/121:09:471
60512545,10sleep30-21swapper/319:20:463
394625413,25sleep33942-21apt-get22:39:573
3080225416,22sleep330808-21diskstats20:40:013
2914525416,17sleep00-21swapper/000:00:050
1099825415,24sleep010994-21apt-get21:19:550
1099825415,24sleep010994-21apt-get21:19:550
3240725314,23sleep132431-21ntp_states00:10:071
318225314,22sleep33261-21ntp_states19:14:483
318225314,22sleep33261-21ntp_states19:14:483
3100625314,22sleep131059-21users22:20:131
2803825312,24sleep128051-21ntp_states20:30:101
2146625315,22sleep121505-21ntp_states23:35:071
2042425311,22sleep30-21swapper/320:05:163
18382536,36sleep00-21swapper/022:31:390
164325313,24sleep11656-21irqstats20:50:041
1376025314,18sleep10-21swapper/123:10:111
2340925213,19sleep00-21swapper/023:40:180
62325112,22sleep1545-21latency_hist22:29:461
2271425113,23sleep122717-21df_inode23:40:031
1693825115,19sleep10-21swapper/123:20:131
1693825115,19sleep10-21swapper/123:20:131
1550925112,20sleep115562-21users19:50:161
1361325113,23sleep313699-21mailstats19:45:133
90672507,32sleep30-21swapper/300:39:093
706425013,22sleep00-21swapper/019:26:120
3026925011,17sleep00-21swapper/022:20:000
204825012,19sleep32101-21/usr/sbin/munin19:10:123
2011825010,22sleep020186-21sensors_temp20:05:130
1804925012,22sleep118141-21perl23:25:011
1804925012,22sleep118141-21perl23:25:011
156525012,22sleep01559-21apt-get19:09:570
1137025013,20sleep10-21swapper/121:20:081
1137025013,20sleep10-21swapper/121:20:081
1097525012,19sleep30-21swapper/321:19:543
1097525012,19sleep30-21swapper/321:19:543
427824815,18sleep00-21swapper/020:59:440
1677324812,22sleep316780-21memory23:20:073
1677324812,22sleep316780-21memory23:20:073
1254524813,18sleep10-21swapper/121:24:571
1227724814,14sleep00-21swapper/023:05:140
159582459,17sleep30-21swapper/321:35:013
80724413,18sleep00-21swapper/020:46:120
55842448,14sleep00-21swapper/022:45:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional