You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-29 - 04:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Jun 29, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18667229867,2966sleep28866-21kworker/2:119:22:202
123872113121,1089sleep08673-1kworker/0:0H22:29:580
28414268614,656sleep327864-21kworker/u8:423:20:003
30495248813,460sleep230503-21ntpq20:00:112
1856524378,408sleep118621-21latency_hist22:49:421
2732239923,360sleep32749-21/usr/sbin/munin21:59:533
69823847,329sleep1364-21snmpd21:52:161
2738123309,305sleep127393-21find21:34:511
29381231113,284sleep329419-21ntpq21:40:063
13333229721,261sleep313425-21/usr/sbin/munin00:15:043
872227996,154sleep10-21swapper/123:35:041
30623225111,224sleep130702-21fschecks_time21:45:011
10284224941,24sleep10-21swapper/120:39:561
4869224616,201sleep14935-21switchtime22:05:121
21999224432,20sleep10-21swapper/119:34:421
19510222243,163sleep119554-21ntpq19:25:051
20954222114,191sleep321101-21latency_hist21:14:423
20954222114,191sleep321101-21latency_hist21:14:423
7427221860,25sleep10-21swapper/120:30:031
7427221860,25sleep10-21swapper/120:30:031
13879221841,100sleep114024-21sed19:09:431
13741221730,165sleep313770-21/usr/sbin/munin19:09:413
13741221730,165sleep313770-21/usr/sbin/munin19:09:413
1946221027,26sleep11969-21modprobe20:14:401
1946221027,26sleep11969-21modprobe20:14:401
7123220832,139sleep17165-21true22:14:411
32169220416,174sleep232182-21fschecks_count21:49:592
10364220417,97sleep10-21swapper/100:05:051
10364220417,97sleep10-21swapper/100:05:051
1257122000,25sleep113855-21cyclictest22:30:141
6586219930,95sleep16597-21latency_hist23:54:431
1960521937,174sleep119625-21apt-get21:09:521
1960521937,174sleep119625-21apt-get21:09:521
2163821918,170sleep121154-21/usr/sbin/munin21:15:091
12329218518,143sleep10-21swapper/120:45:091
19325218312,113sleep119329-21cron21:09:401
19325218312,113sleep119329-21cron21:09:401
29059218110,24sleep129166-21sort23:20:391
2383421818,15sleep10-21swapper/121:22:201
30306218010,153sleep130324-21kernelversion20:00:061
16767217712,148sleep116782-21munin-run22:44:411
31854217514,139sleep10-21swapper/120:05:031
31854217514,139sleep10-21swapper/120:05:031
935621749,110sleep19484-21timerwakeupswit20:35:111
935621749,110sleep19484-21timerwakeupswit20:35:111
1847521739,151sleep118629-21grep00:30:121
14091216710,119sleep114093-21modprobe00:16:391
22615216638,19sleep10-21swapper/119:35:071
13158216619,126sleep013251-21fschecks_time00:15:010
2012221638,140sleep120276-21date22:54:421
27592216119,117sleep20-21swapper/219:50:142
2997721579,136sleep0364-21snmpd21:41:160
10050215520,109sleep310088-21grep00:04:583
10050215520,109sleep310088-21grep00:04:583
27945215110,87sleep127868-21latency_hist23:19:421
1941215112,29sleep30-21swapper/323:39:433
17680215097,16sleep10-21swapper/119:20:011
16197215015,17sleep00-21swapper/020:59:410
8956214928,19sleep10-21swapper/100:00:121
8956214928,19sleep10-21swapper/100:00:121
2290721487,104sleep1364-21snmpd23:00:561
4376214517,103sleep24501-21cat23:45:112
15333214066,55sleep114914-21/usr/sbin/munin20:55:061
30421213817,105sleep230443-21pluginstate23:25:082
20375213115,21sleep10-21swapper/100:37:191
29641212736,76sleep129765-21timerwakeupswit21:40:121
11981212714,17sleep10-21swapper/100:10:091
4117212514,95sleep00-21swapper/023:45:050
16671212511,97sleep116800-21timerwakeupswit19:15:121
160921197,71sleep10-21swapper/121:55:061
11935211915,21sleep112012-21gpgv22:29:431
26360211714,84sleep326510-21latency_hist23:14:433
26360211714,84sleep326510-21latency_hist23:14:433
8679211612,88sleep00-21swapper/022:19:400
25362211617,70sleep113855-21cyclictest19:44:561
8083211519,77sleep28133-21readlink23:59:402
8083211519,77sleep28133-21readlink23:59:402
31595211510,91sleep10-21swapper/123:30:001
24082211518,79sleep224148-21wc19:40:052
2050321150,12sleep20-21swapper/221:10:122
2050321150,12sleep20-21swapper/221:10:122
1651321157,18sleep10-21swapper/120:59:551
1645721150,6sleep30-21swapper/320:59:433
806021125,30sleep10-21swapper/123:59:231
806021125,30sleep10-21swapper/123:59:231
27181211210,50sleep127332-21latency_hist21:34:431
265621128,32sleep12715-21find23:40:091
443221118,80sleep10-21swapper/120:20:131
28429211114,16sleep20-21swapper/223:20:112
223621118,18sleep10-21swapper/120:14:561
2688521108,17sleep10-21swapper/119:49:531
824521096,61sleep10-21swapper/122:15:131
6921210910,83sleep27027-21cat20:29:432
6921210910,83sleep27027-21cat20:29:432
19766210824,66sleep219280-21/usr/sbin/munin00:35:102
10257210723,69sleep310263-21modprobe22:24:403
1020021075,46sleep10-21swapper/122:23:571
29960210610,15sleep10-21swapper/123:24:551
2596321069,83sleep026038-21find19:45:110
15267210510,73sleep10-21swapper/100:20:111
30342210311,79sleep3360-21ntpd23:25:053
15554210323,58sleep015609-21fschecks_count22:40:000
275621017,43sleep12753-21apt-get21:59:551
23785210115,62sleep10-21swapper/119:39:511
11781210019,67sleep311809-21kernelversion00:10:033
2394829837,44sleep113855-21cyclictest23:05:081
2158029814,66sleep321630-21/usr/sbin/munin21:15:073
67392976,27sleep10-21swapper/120:28:011
67392976,27sleep10-21swapper/120:28:011
1627729711,11sleep00-21swapper/019:15:030
47252958,14sleep10-21swapper/123:46:191
1698229514,68sleep017028-21processes21:00:100
146572958,14sleep10-21swapper/120:54:421
141412956,14sleep10-21swapper/122:35:041
141412956,14sleep10-21swapper/122:35:041
907929451,29sleep29081-21diskstats22:19:592
419329417,61sleep34227-21ntp_kernel_err20:20:073
1406529418,36sleep30-21swapper/319:09:533
643929217,60sleep26480-21switchtime22:10:142
2842629219,55sleep028397-21apt-get23:20:090
133552926,71sleep00-21swapper/020:49:520
527629121,57sleep35305-21cpu23:49:563
1256929131,42sleep212610-21uniq19:05:002
1256929131,42sleep212610-21uniq19:05:002
284552907,32sleep10-21swapper/119:54:531
659928914,59sleep26742-21latency_hist23:54:442
2081828912,60sleep220855-21ps22:55:082
927928814,59sleep09354-21switchtime20:35:100
927928814,59sleep09354-21switchtime20:35:100
633728820,51sleep06490-21unixbench_multi20:25:370
633728820,51sleep06490-21unixbench_multi20:25:370
840428726,38sleep28401-21apt-get23:59:532
840428726,38sleep28401-21apt-get23:59:532
243812878,63sleep10-21swapper/121:24:581
357228612,58sleep23597-21sed20:19:422
2759428610,59sleep027652-21/usr/sbin/munin21:35:030
216082869,14sleep00-21swapper/022:59:340
3051328516,53sleep030583-21switchtime23:25:100
2388228510,57sleep223916-21ntp_kernel_pll_23:05:052
203302856,30sleep10-21swapper/122:54:511
1710428516,53sleep317112-21vmstat00:25:143
3187728415,32sleep20-21swapper/223:30:052
1501628415,55sleep215025-21needreboot00:20:042
262762835,29sleep10-21swapper/123:14:251
262762835,29sleep10-21swapper/123:14:251
2512328214,51sleep025205-21fschecks_time23:10:000
2512328214,51sleep025205-21fschecks_time23:10:000
174332825,29sleep10-21swapper/100:29:191
146822826,28sleep10-21swapper/119:10:121
948628115,46sleep29605-21users20:35:122
948628115,46sleep29605-21users20:35:122
88382819,17sleep00-21swapper/000:00:070
88382819,17sleep00-21swapper/000:00:070
713228113,54sleep27151-21apt-get20:29:512
713228113,54sleep27151-21apt-get20:29:512
40128119,48sleep0556-21latency_hist23:34:430
1746428115,25sleep30-21swapper/322:45:113
2389928015,48sleep323992-21grep19:40:013
2822127921,28sleep30-21swapper/319:54:433
1948827918,43sleep30-21swapper/300:35:033
121027914,52sleep21300-21timerandwakeup23:35:142
918127611,49sleep09224-21basename22:20:010
73527611,51sleep0763-21df20:09:590
73527611,51sleep0763-21df20:09:590
592527618,40sleep36009-21memory20:25:283
592527618,40sleep36009-21memory20:25:283
1815627623,24sleep20-21swapper/221:05:002
1815627623,24sleep20-21swapper/221:05:002
1389327615,46sleep313863-21/usr/sbin/munin22:34:583
1389327615,46sleep313863-21/usr/sbin/munin22:34:583
562427522,25sleep30-21swapper/322:09:423
284727527,27sleep00-21swapper/022:00:020
1819727516,38sleep018280-21grep19:20:120
1812127518,35sleep318195-21/usr/sbin/munin19:20:113
1537927416,26sleep00-21swapper/019:14:430
2872927314,44sleep328434-21/usr/sbin/munin19:55:043
2855027328,29sleep028645-21sort19:55:010
1975227323,23sleep30-21swapper/319:25:123
402427210,16sleep20-21swapper/222:04:402
2081227219,30sleep00-21swapper/019:30:010
3121227126,21sleep20-21swapper/221:45:142
2131027116,25sleep20-21swapper/221:15:022
1003327123,20sleep30-21swapper/320:39:423
1003327123,20sleep30-21swapper/320:39:423
846727020,25sleep00-21swapper/020:34:420
846727020,25sleep00-21swapper/020:34:420
2381027017,28sleep00-21swapper/019:39:540
1659927014,39sleep216601-21/usr/sbin/munin21:00:002
1303527021,33sleep013119-21sed19:05:100
1303527021,33sleep013119-21sed19:05:100
1076727019,28sleep30-21swapper/320:40:103
2622926919,21sleep00-21swapper/021:30:060
81526821,23sleep00-21swapper/023:35:030
1843826812,39sleep318461-21/usr/sbin/munin21:05:053
1843826812,39sleep318461-21/usr/sbin/munin21:05:053
1827926823,20sleep00-21swapper/021:05:020
1827926823,20sleep00-21swapper/021:05:020
813426713,41sleep38169-21munin-run23:59:423
813426713,41sleep38169-21munin-run23:59:423
632826710,13sleep00-21swapper/022:10:120
521126714,24sleep20-21swapper/220:24:422
35126716,28sleep30-21swapper/323:34:423
2944626719,24sleep20-21swapper/221:40:092
148162675,53sleep00-21swapper/022:35:230
148162675,53sleep00-21swapper/022:35:230
124352675,17sleep00-21swapper/000:10:210
2045126620,16sleep20-21swapper/219:29:432
1773226619,26sleep217767-21ps22:45:192
1502626622,29sleep315071-21/usr/sbin/munin00:20:053
752626513,22sleep30-21swapper/320:30:053
752626513,22sleep30-21swapper/320:30:053
1908326515,35sleep319234-21latency_hist00:34:423
1184626517,30sleep01-21systemd22:29:420
1161426511,36sleep00-21swapper/020:44:430
1033226516,22sleep00-21swapper/000:05:040
1033226516,22sleep00-21swapper/000:05:040
3106626410,15sleep00-21swapper/021:45:120
255712647,12sleep30-21swapper/319:45:023
2784926320,28sleep3278-21dbus-daemon23:19:403
2698626316,21sleep20-21swapper/223:15:072
2329026321,19sleep00-21swapper/023:04:430
2326226314,31sleep223331-21mailstats21:20:112
976426221,21sleep30-21swapper/300:04:423
2479826215,31sleep324940-21cat23:09:423
2320026220,23sleep20-21swapper/223:04:412
137862625,12sleep20-21swapper/220:50:062
127452625,47sleep00-21swapper/020:45:160
68626115,33sleep3733-21/usr/sbin/munin20:09:583
68626115,33sleep3733-21/usr/sbin/munin20:09:583
348926121,18sleep20-21swapper/223:44:422
215426116,27sleep02173-21apt-get23:39:520
1719526119,20sleep30-21swapper/321:00:133
1081826112,20sleep20-21swapper/222:25:042
104922617,12sleep20-21swapper/200:05:102
104922617,12sleep20-21swapper/200:05:102
34592607,11sleep00-21swapper/020:19:070
3255026016,19sleep30-21swapper/321:50:083
2874526013,33sleep30-21swapper/321:39:423
2477326011,36sleep024810-21ps21:25:090
2244426023,20sleep00-21swapper/021:19:400
1862326020,18sleep20-21swapper/222:49:432
1158426011,35sleep211591-21df_inode00:09:582
781325923,19sleep20-21swapper/222:15:062
3053425916,26sleep330530-21apt-get21:44:553
2654125917,23sleep30-21swapper/321:30:133
2428525920,19sleep30-21swapper/321:24:523
2230025918,21sleep30-21swapper/323:00:053
1814525916,19sleep20-21swapper/200:30:052
1693625923,19sleep30-21swapper/322:44:433
1602425915,21sleep20-21swapper/220:57:292
135392595,12sleep20-21swapper/222:34:002
128972595,11sleep20-21swapper/200:14:432
498525827,16sleep30-21swapper/323:49:403
435925816,23sleep00-21swapper/020:20:120
2056525813,27sleep320623-21/usr/sbin/munin22:55:013
200072586,10sleep30-21swapper/322:53:563
1784125818,27sleep017911-21switchtime22:45:210
1677925815,20sleep00-21swapper/000:25:100
118125819,25sleep31226-21/usr/sbin/munin21:54:563
95825717,22sleep20-21swapper/220:10:052
95825717,22sleep20-21swapper/220:10:052
322925712,27sleep33230-21sshd20:16:203
3167825719,24sleep231710-21tune2fs20:05:002
3167825719,24sleep231710-21tune2fs20:05:002
2630725721,17sleep20-21swapper/221:30:082
1985825713,19sleep00-21swapper/000:35:120
89025610,32sleep0932-21systemd-run21:54:410
59512569,12sleep30-21swapper/322:10:023
3199225610,27sleep00-21swapper/020:05:060
3199225610,27sleep00-21swapper/020:05:060
3134525615,23sleep01-21systemd20:04:420
2233325615,25sleep022420-21perl19:35:020
219172569,33sleep221919-21sshd19:34:192
1647225617,22sleep216494-21proc_pri19:15:092
1233525615,25sleep312385-21/usr/sbin/munin22:29:573
767925519,21sleep30-21swapper/322:15:033
431725513,25sleep24309-21apt-get22:04:572
2726725514,18sleep00-21swapper/023:15:130
1887625513,17sleep00-21swapper/022:49:570
1567725517,17sleep20-21swapper/222:40:022
952225413,21sleep30-21swapper/300:02:443
817125412,21sleep00-21swapper/023:59:430
817125412,21sleep00-21swapper/023:59:430
2415725416,18sleep00-21swapper/023:05:110
2269125414,23sleep322697-21ntp_states19:35:093
2066325414,21sleep30-21swapper/319:29:523
2039925411,28sleep020428-21munin-run19:29:410
2024225418,21sleep00-21swapper/021:10:110
2024225418,21sleep00-21swapper/021:10:110
1314125420,15sleep30-21swapper/320:49:433
1256125415,22sleep10-21swapper/119:04:591
2741025311,25sleep227403-21apt-get21:34:562
2276425313,24sleep022758-21apt-get21:19:560
178725315,16sleep20-21swapper/221:55:122
1065325318,19sleep20-21swapper/220:40:052
2845625211,25sleep228452-21apt-get19:54:542
2526625210,28sleep225305-21irqstats23:10:022
2526625210,28sleep225305-21irqstats23:10:022
161922529,16sleep20-21swapper/200:24:522
1452425214,18sleep20-21swapper/222:35:132
1452425214,18sleep20-21swapper/222:35:132
117725212,23sleep01173-21apt-get21:54:530
2438925115,17sleep20-21swapper/221:24:592
217825112,23sleep32173-21apt-get23:39:553
1480925119,17sleep20-21swapper/219:10:132
1346125113,23sleep313465-21diskstats20:49:593
1465325012,22sleep014648-21apt-get00:19:530
531424912,21sleep05349-21df23:49:570
3015124915,18sleep330011-21/usr/sbin/munin20:00:043
1775324911,22sleep017747-21apt-get00:29:550
3171324811,15sleep30-21swapper/320:05:013
3171324811,15sleep30-21swapper/320:05:013
171782484,31sleep30-21swapper/319:18:253
1063524811,22sleep310727-21perl22:25:003
269782479,16sleep00-21swapper/019:50:000
2582824712,16sleep20-21swapper/219:45:092
2301524611,19sleep323034-21irqstats21:20:043
1313124610,23sleep213140-21munin-run20:49:422
554524210,18sleep05546-21sshd22:09:190
2279024210,20sleep222868-21switchtime19:35:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional