You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-25 - 13:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sat Apr 25, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
140522700219,6970sleep111153-21kworker/1:022:33:031
16433215725,1556sleep212966-21kworker/2:100:24:062
12172252014,487sleep112174-21sshd20:45:081
5625241228,367sleep15675-21/usr/sbin/munin22:06:101
5625241228,367sleep15675-21/usr/sbin/munin22:06:101
29444228516,210sleep229591-21latency_hist21:40:462
32497227290,138sleep21-21systemd21:50:442
25356226444,197sleep224965-21/usr/sbin/munin21:26:082
10735225918,224sleep010740-21/usr/sbin/munin00:06:070
2163322530,6sleep20-21swapper/219:31:122
14278225228,185sleep214339-21find00:16:512
14278225228,185sleep214339-21find00:16:512
14582224531,198sleep214610-21cpu22:35:592
14582224531,198sleep214610-21cpu22:35:592
2438223418,192sleep12518-21fschecks_time23:41:041
16400222831,73sleep216432-21grep22:41:082
16400222831,73sleep216432-21grep22:41:082
25823222332,170sleep20-21swapper/223:11:082
11210221449,104sleep20-21swapper/222:25:452
18102221243,151sleep218192-21fschecks_time19:21:032
3942221171,82sleep20-21swapper/222:01:062
3942221171,82sleep20-21swapper/222:01:062
19310221110,183sleep219367-21fschecks_count22:51:012
18062221181,77sleep20-21swapper/222:46:092
7418220987,54sleep27488-21fschecks_time23:56:272
2520220632,134sleep22567-21idleruntime23:41:052
1158220674,117sleep20-21swapper/221:51:212
676922007,149sleep26770-21cpu22:10:592
7004219713,135sleep20-21swapper/220:26:192
2239219227,149sleep22346-21sort20:11:202
4501218951,59sleep2281-21dbus-daemon20:20:452
4501218951,59sleep2281-21dbus-daemon20:20:452
29925218957,114sleep230075-21latency_hist23:25:462
19235218936,137sleep219317-21find00:31:132
30810218638,67sleep230931-21timerwakeupswit23:26:152
24170218434,131sleep224234-21latency23:06:032
9203218318,149sleep09353-21latency_hist20:35:460
15272216715,138sleep115311-21ntpq19:11:091
2106215624,100sleep22165-21latency_hist23:40:442
2106215624,100sleep22165-21latency_hist23:40:442
6698215411,126sleep21-21systemd23:55:432
27055215414,87sleep20-21swapper/221:31:132
17751215210,92sleep20-21swapper/221:01:122
14683215210,130sleep214787-21wc20:51:152
2006921469,52sleep220156-21latency_hist21:10:462
2460121455,129sleep20-21swapper/221:24:272
8584214431,99sleep28731-21awk20:31:152
16142214388,38sleep20-21swapper/220:56:122
13086214214,56sleep20-21swapper/222:31:022
5902214116,60sleep36033-21grep22:06:163
5902214116,60sleep36033-21grep22:06:163
25564214112,38sleep00-21swapper/021:26:150
28033213927,62sleep20-21swapper/219:51:052
10726213920,104sleep310864-21sed20:40:463
20913213010,60sleep20-21swapper/221:11:142
30028212915,48sleep230104-21xargs19:56:142
1422421279,77sleep214348-21latency_hist19:10:452
1646021268,64sleep20-21swapper/219:15:582
31680212353,48sleep20-21swapper/223:30:592
5031212224,74sleep10-21swapper/122:05:451
12508212138,45sleep10-21swapper/120:45:571
1478821190,11sleep30-21swapper/320:51:153
60621179,95sleep0675-21sensors20:06:130
15662211713,89sleep015778-21users19:11:160
24550211620,78sleep324582-21grep19:41:043
1920211620,77sleep02071-21latency_hist21:55:470
1193421168,43sleep212000-21cat22:26:142
8114211511,90sleep38185-21cat22:15:453
18297211210,86sleep018414-21cat22:46:150
15539211214,82sleep315655-21ls19:11:153
12507211215,17sleep00-21swapper/020:45:560
4349211027,40sleep20-21swapper/223:46:092
26311211016,27sleep30-21swapper/321:30:463
9426210921,66sleep39513-21sort00:01:133
6150210830,13sleep20-21swapper/222:06:312
6150210830,13sleep20-21swapper/222:06:312
27248210833,24sleep20-21swapper/223:16:052
17439210820,70sleep317467-21latency_hist22:45:453
17439210820,70sleep317467-21latency_hist22:45:453
2361321077,36sleep20-21swapper/223:05:432
1238121078,90sleep00-21swapper/019:03:040
1238121078,90sleep00-21swapper/019:03:040
21654210513,67sleep121734-21latency_hist21:15:451
13351210426,53sleep113502-21apt-key00:15:451
13351210426,53sleep113502-21apt-key00:15:451
11330210413,75sleep111374-21cpu19:01:001
11330210413,75sleep111374-21cpu19:01:001
30954210326,49sleep330973-21apt-get20:00:553
4981210129,41sleep00-21swapper/022:05:440
4981210129,41sleep00-21swapper/022:05:440
20451210110,79sleep220464-21vmstat19:26:172
20451210110,79sleep220464-21vmstat19:26:172
3262229822,55sleep20-21swapper/220:06:032
2974829824,53sleep329764-21sort21:41:043
2461229820,24sleep30-21swapper/323:06:123
3097929720,17sleep20-21swapper/220:00:582
2662529615,66sleep026672-21fschecks_count21:31:030
2209729624,51sleep20-21swapper/221:16:032
2178829621,52sleep021942-21grep19:31:150
131992967,74sleep213341-21sed20:46:142
2856029511,20sleep30-21swapper/323:20:543
265402959,19sleep20-21swapper/219:46:282
1169829511,67sleep20-21swapper/200:10:452
67629415,63sleep1753-21timerandwakeup20:06:141
577429416,12sleep20-21swapper/220:22:432
577429416,12sleep20-21swapper/220:22:432
1540629410,70sleep015484-21cat20:55:450
119902949,17sleep20-21swapper/200:11:012
119902949,17sleep20-21swapper/200:11:012
803529327,41sleep18057-21ls22:15:431
475129314,63sleep14821-21df20:21:011
475129314,63sleep14821-21df20:21:011
34362937,72sleep23515-21latency_hist22:00:462
34362937,72sleep23515-21latency_hist22:00:462
2698029314,60sleep027026-21timerandwakeup19:46:380
2328329329,51sleep223379-21sed19:36:142
2211329316,61sleep122142-21latency_hist23:00:451
680029217,61sleep06767-40mandb23:55:450
2992429228,28sleep20-21swapper/221:41:072
2549629231,29sleep20-21swapper/219:42:122
965229026,15sleep20-21swapper/222:20:442
2295929016,59sleep023047-21timerandwakeup23:01:130
2121729023,51sleep20-21swapper/222:56:102
2088328917,55sleep320774-21/usr/sbin/munin22:56:033
181322896,50sleep20-21swapper/200:27:582
1545128929,21sleep215538-21sed19:11:142
1227328929,16sleep20-21swapper/220:45:452
1166928913,55sleep111650-21latency_hist00:10:441
2918828813,58sleep329276-21latency_hist19:55:463
1012928844,27sleep210184-21latency_hist00:05:442
725228619,49sleep16716-21apt-get23:55:541
426328512,53sleep03870-21/usr/sbin/munin23:46:070
99012847,41sleep29964-21mailstats20:36:122
2942528420,38sleep329420-21apt-get19:55:583
1929528420,31sleep00-21swapper/022:50:590
1747128414,19sleep00-21swapper/022:45:460
1747128414,19sleep00-21swapper/022:45:460
882628316,46sleep18327-21/usr/sbin/munin22:16:111
768828315,52sleep37722-21ntp_kernel_pll_23:56:333
3246828320,42sleep132494-21munin-plugin-st21:50:431
1573828314,55sleep115825-21fschecks_time20:56:021
88328214,33sleep10-21swapper/123:36:031
88328214,33sleep10-21swapper/123:36:031
2030628218,15sleep30-21swapper/321:10:593
1387328212,56sleep113890-21fschecks_count00:16:301
1387328212,56sleep113890-21fschecks_count00:16:301
1187828214,19sleep00-21swapper/020:41:190
1159328230,33sleep30-21swapper/319:01:063
1159328230,33sleep30-21swapper/319:01:063
2250428116,44sleep022626-21timerandwakeup21:16:130
1842428115,43sleep018564-21latency_hist00:30:470
1838428123,36sleep318423-21sshd19:21:083
1299928124,27sleep30-21swapper/319:06:023
1264328132,26sleep20-21swapper/219:05:452
1264328132,26sleep20-21swapper/219:05:452
1512828018,29sleep00-21swapper/022:36:130
1512828018,29sleep00-21swapper/022:36:130
1252527911,51sleep312625-21sed00:11:153
1252527911,51sleep312625-21sed00:11:153
1045227915,45sleep310470-21fschecks_count00:06:023
2048127822,37sleep320498-21cron22:55:433
987827713,15sleep10-21swapper/120:36:111
674627718,38sleep36767-40mandb23:55:443
3097427723,30sleep00-21swapper/021:45:450
1207727714,28sleep00-21swapper/022:26:160
2320327621,26sleep10-21swapper/121:20:461
1754727614,47sleep117553-21ntp_states00:26:111
1557527614,24sleep10-21swapper/100:21:061
133832769,48sleep013441-21snmpd19:06:110
3221427512,48sleep032285-21sensors_temp23:31:130
3147727517,25sleep00-21swapper/021:46:060
1548627517,43sleep315544-21if_err_eth000:21:043
1200127528,24sleep30-21swapper/322:26:153
651127417,37sleep06568-21irqstats20:26:090
182552738,54sleep029062-21kworker/0:121:03:080
3204727214,29sleep30-21swapper/323:31:083
2680727218,38sleep326838-21/usr/sbin/munin21:31:063
2545327213,45sleep125520-21/usr/sbin/munin23:10:591
248662724,15sleep10-21swapper/123:06:551
2475127223,21sleep30-21swapper/321:25:453
2304827219,30sleep123199-21grep23:01:141
2989227121,25sleep10-21swapper/123:25:451
2352627118,25sleep10-21swapper/121:21:021
1746827011,29sleep30-21swapper/321:01:053
1505427020,35sleep315083-21sed00:20:443
1505427020,35sleep315083-21sed00:20:443
63826918,25sleep30-21swapper/321:51:103
3228826919,29sleep10-21swapper/120:05:461
2157226920,19sleep10-21swapper/119:31:121
2010926917,22sleep30-21swapper/319:26:133
2010926917,22sleep30-21swapper/319:26:133
1686026914,24sleep00-21swapper/019:16:090
1312426914,22sleep10-21swapper/119:06:041
895726816,27sleep30-21swapper/322:16:133
1937326816,34sleep319480-21ls21:06:143
162626824,21sleep30-21swapper/320:11:053
205526716,37sleep02062-21modprobe23:40:420
205526716,37sleep02062-21modprobe23:40:420
690726617,35sleep17003-21timerandwakeup20:26:181
2590226617,30sleep126099-21latency_hist19:45:461
248182660,7sleep00-21swapper/021:25:450
237992669,12sleep00-21swapper/019:38:590
1371726611,39sleep013213-21apt-get00:15:520
1371726611,39sleep013213-21apt-get00:15:520
438326510,38sleep34528-21unixbench_singl22:01:153
438326510,38sleep34528-21unixbench_singl22:01:153
436526516,35sleep14423-21processes23:46:131
34262658,41sleep10-21swapper/122:00:451
34262658,41sleep10-21swapper/122:00:451
3176326521,25sleep10-21swapper/123:31:021
2607326514,21sleep00-21swapper/023:11:140
1949326518,21sleep10-21swapper/122:51:041
600626412,36sleep36035-21munin-run20:25:433
600626412,36sleep36035-21munin-run20:25:433
366426418,26sleep30-21swapper/320:16:103
366426418,26sleep30-21swapper/320:16:103
2993226411,34sleep130026-21mailstats19:56:131
2762526416,25sleep30-21swapper/323:16:143
1872426417,29sleep318782-21cut00:31:023
1841626417,21sleep10-21swapper/122:46:161
1663326414,28sleep10-21swapper/122:41:151
1663326414,28sleep10-21swapper/122:41:151
149702646,12sleep30-21swapper/322:36:083
149702646,12sleep30-21swapper/322:36:083
133422647,11sleep30-21swapper/320:46:153
1275426411,39sleep10-21swapper/122:30:451
103662647,46sleep00-21swapper/020:36:470
832626315,19sleep00-21swapper/022:15:540
3255226320,21sleep30-21swapper/321:50:453
2755926318,29sleep127575-21apt19:50:431
134672636,11sleep30-21swapper/322:31:113
483526211,16sleep00-21swapper/020:21:040
483526211,16sleep00-21swapper/020:21:040
377526212,35sleep13841-21switchtime20:16:121
377526212,35sleep13841-21switchtime20:16:121
2937326217,26sleep329402-21munin-run21:40:433
245852625,16sleep00-21swapper/019:41:050
210426214,28sleep02168-21find20:11:180
1810526223,23sleep318131-21ntp_states22:46:103
1026226217,20sleep10-21swapper/122:21:081
653026120,22sleep00-21swapper/022:10:450
653026120,22sleep00-21swapper/022:10:450
1982626114,19sleep00-21swapper/019:26:060
1982626114,19sleep00-21swapper/019:26:060
170226116,29sleep11708-21diskstats20:11:071
917126012,31sleep39118-21latency_hist20:35:453
450026013,31sleep30-21swapper/323:46:153
211926027,16sleep00-21swapper/021:55:590
211926027,16sleep00-21swapper/021:55:590
2107626012,33sleep321095-21apt-get19:30:543
1880826014,18sleep00-21swapper/021:05:590
928225914,28sleep09289-21/usr/sbin/munin00:01:100
2989425911,12sleep00-21swapper/019:56:120
2749825915,19sleep00-21swapper/023:16:110
1690625913,30sleep116916-21ntp_states19:16:101
1145025920,20sleep30-21swapper/320:41:143
3040025819,23sleep030414-21kernelversion23:26:040
2812125820,18sleep00-21swapper/019:51:070
2630225816,25sleep326288-21apt-get19:46:013
1572725812,23sleep00-21swapper/020:56:010
320462577,39sleep00-21swapper/020:03:140
2857725715,26sleep10-21swapper/121:36:131
1809325715,26sleep10-21swapper/121:01:151
1406725714,27sleep114062-21apt-get20:50:571
128622575,43sleep00-21swapper/000:11:360
128622575,43sleep00-21swapper/000:11:360
1007925713,30sleep110128-21systemd-run00:05:431
2704225618,23sleep127109-21df23:16:001
2438025617,24sleep00-21swapper/023:06:090
1924325621,18sleep10-21swapper/121:06:121
1771825622,19sleep30-21swapper/319:20:443
784125513,24sleep17835-21apt-get20:30:581
72692555,37sleep00-21swapper/023:56:010
688425514,26sleep16994-21cmp23:55:461
676725515,24sleep06762-21apt-get22:10:570
368925516,22sleep00-21swapper/020:16:110
368925516,22sleep00-21swapper/020:16:110
292652557,36sleep00-21swapper/021:39:190
2499025514,25sleep124984-21apt-get21:25:591
1944325512,17sleep00-21swapper/000:31:150
1855625511,30sleep018632-21switchtime19:21:140
1620025517,18sleep30-21swapper/320:56:133
62125412,17sleep10-21swapper/121:51:081
2552825416,22sleep30-21swapper/323:11:013
2308725412,24sleep123131-21ntp_states19:36:091
2276225416,17sleep30-21swapper/319:36:033
2230425414,21sleep322340-21ntp_states21:16:083
1708925413,23sleep317081-21apt-get00:25:573
1530325412,26sleep015296-21apt-get00:20:570
98125322,14sleep30-21swapper/323:36:053
98125322,14sleep30-21swapper/323:36:053
35462535,36sleep00-21swapper/023:44:580
2667325318,18sleep10-21swapper/121:31:041
1870925315,22sleep118718-21df_inode00:31:001
711925215,16sleep30-21swapper/320:26:203
3013325218,16sleep10-21swapper/123:25:571
2905025212,21sleep00-21swapper/023:21:120
256925211,23sleep32614-21irqstats23:41:063
1955525213,21sleep119622-21df19:25:591
1955525213,21sleep119622-21df19:25:591
1051225216,15sleep00-21swapper/022:21:150
2821125117,18sleep10-21swapper/119:51:081
1305425112,22sleep013078-21df_inode22:31:000
408825015,21sleep14127-21ntp_states22:01:091
1863325013,23sleep118759-21expr19:21:151
156652495,32sleep10-21swapper/122:38:281
156652495,32sleep10-21swapper/122:38:281
2987124816,17sleep10-21swapper/121:41:061
2210524815,18sleep00-21swapper/023:00:440
2030324811,21sleep020299-21apt-get21:10:560
997524712,21sleep39981-21diskstats22:21:023
2234924717,15sleep30-21swapper/323:00:553
3053124511,21sleep330575-21ntpq23:26:073
268424510,17sleep30-21swapper/321:56:143
268424510,17sleep30-21swapper/321:56:143
4332408,12sleep30-21swapper/320:06:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional