You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-14 - 06:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Jul 14, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11419225915,2574sleep38182-21kworker/3:122:36:583
6606282215,790sleep26607-21ntp_states00:04:152
9104269923,658sleep19113-21/usr/sbin/munin22:29:091
38125278,507sleep3463-21mailstats23:44:143
2576222629,241sleep325798-21ntpq21:39:113
32606225311,224sleep232739-21awk21:59:322
17783225047,184sleep317831-21readlink22:58:463
21273224032,188sleep321315-21latency_hist19:43:483
18579223222,192sleep318639-21sendmail_mailqu22:59:143
13193221111,176sleep313353-21wc22:43:483
22546221015,180sleep322708-21cmp23:13:473
2373022010,7sleep10-21swapper/119:49:181
5731219921,163sleep25786-21fschecks_count22:19:052
28946219413,168sleep128953-21sh21:49:141
1351821768,155sleep313399-21/usr/sbin/munin22:44:043
27434217213,90sleep11-21systemd23:28:471
88172170125,23sleep30-21swapper/322:29:033
11475216222,31sleep311516-21ntp_kernel_pll_20:54:113
19497215930,97sleep3364-21snmpd19:36:453
7687215712,33sleep10-21swapper/100:08:591
4992215514,120sleep30-21swapper/323:59:123
721215315,116sleep30-21swapper/322:03:493
16936215310,131sleep317019-21cut19:29:063
3118215113,122sleep00-21swapper/020:28:490
1212721519,122sleep312141-21munin-run00:23:463
26459215014,114sleep30-21swapper/319:59:093
1401821498,106sleep314085-21latency19:19:093
13988214717,27sleep30-21swapper/321:03:473
13988214717,27sleep30-21swapper/321:03:473
21600214610,21sleep00-21swapper/019:44:050
24343214412,118sleep30-21swapper/323:18:483
454214015,61sleep30-21swapper/320:19:193
454214015,61sleep30-21swapper/320:19:193
3207421385,11sleep30-21swapper/320:18:143
141721327,15sleep30-21swapper/322:04:133
141721327,15sleep30-21swapper/322:04:133
1002221328,113sleep310085-21ps20:49:183
1002221328,113sleep310085-21ps20:49:183
1932221299,108sleep319358-21ntpq21:19:133
795921285,20sleep30-21swapper/320:43:533
1529021285,13sleep30-21swapper/321:06:093
1529021285,13sleep30-21swapper/321:06:093
4338212610,80sleep34356-21ls23:58:483
23785212626,63sleep30-21swapper/321:33:583
2812321250,11sleep10-21swapper/123:29:101
29521212430,56sleep30-21swapper/323:34:093
665021235,11sleep30-21swapper/322:20:313
3021121238,94sleep330273-21basename21:54:083
1004921229,73sleep30-21swapper/300:14:173
1004921229,73sleep30-21swapper/300:14:173
29798212110,67sleep30-21swapper/321:53:483
1330821215,17sleep30-21swapper/319:17:113
11226212014,93sleep211266-21ntp_kernel_pll_00:19:092
3023821195,11sleep30-21swapper/320:09:433
28472211912,18sleep30-21swapper/320:04:223
32194211719,82sleep032308-21latency_hist20:18:480
258521175,11sleep30-21swapper/323:52:103
311821137,61sleep30-21swapper/322:09:173
23443211323,66sleep223304-21latency_hist21:33:492
1678421136,78sleep30-21swapper/300:38:433
1677721126,77sleep30-21swapper/321:10:133
13688211253,47sleep313735-21systemctl00:28:463
1574621118,61sleep30-21swapper/319:24:143
609321105,74sleep30-21swapper/320:38:203
3080421098,59sleep30-21swapper/323:38:583
18515210968,30sleep30-21swapper/321:17:313
2070321089,59sleep30-21swapper/321:24:093
1500210713,80sleep11518-21/usr/sbin/munin20:23:581
30116210612,79sleep030125-21/usr/sbin/munin21:54:050
20237210615,74sleep220295-21cut21:23:482
743121055,87sleep30-21swapper/320:40:183
9965210415,18sleep00-21swapper/019:08:490
464921048,79sleep30-21swapper/320:33:483
3237210421,62sleep03385-21grep22:09:180
22713210420,24sleep00-21swapper/023:13:480
21191210418,58sleep120453-21/usr/sbin/munin21:24:191
2202221038,56sleep30-21swapper/321:28:573
814521026,56sleep30-21swapper/300:09:133
24582210269,17sleep30-21swapper/323:19:003
159121027,55sleep30-21swapper/320:24:043
1513621025,86sleep30-21swapper/300:32:263
7463210016,72sleep17541-21sed22:24:091
2743721006,62sleep30-21swapper/321:44:153
911429816,60sleep28717-21/usr/sbin/munin22:29:102
200782985,14sleep00-21swapper/023:04:120
200782985,14sleep00-21swapper/023:04:120
1570029818,62sleep215745-21/usr/sbin/munin19:24:132
1472629822,57sleep114735-21munin-run22:48:471
3196129718,65sleep131982-21fschecks_count21:59:181
2674229719,26sleep10-21swapper/121:43:491
207272965,53sleep30-21swapper/323:06:163
207272965,53sleep30-21swapper/323:06:163
1442629510,72sleep214437-21ntp_states00:29:142
50322947,53sleep30-21swapper/322:15:073
3211829417,55sleep132266-21latency_hist23:43:481
313022949,17sleep20-21swapper/220:14:142
1001429416,62sleep210021-21/usr/sbin/munin20:49:172
1001429416,62sleep210021-21/usr/sbin/munin20:49:172
930929319,53sleep18601-21/usr/sbin/munin19:04:181
300429312,64sleep13083-21sendmail_mailtr22:09:151
268052938,65sleep00-21swapper/019:59:230
828929240,41sleep30-21swapper/322:26:413
743529110,63sleep00-21swapper/022:24:080
739929116,45sleep07423-21awk00:08:450
2246629123,40sleep122484-21munin-run23:13:451
115929130,37sleep20-21swapper/223:48:452
435929011,62sleep04501-21latency_hist23:58:490
159272905,50sleep30-21swapper/322:49:503
156229010,65sleep21636-21find22:04:162
156229010,65sleep21636-21find22:04:162
111712905,52sleep30-21swapper/319:09:573
65982898,22sleep10-21swapper/100:04:141
1085728923,44sleep010943-21latency_hist20:53:470
1085728923,44sleep010943-21latency_hist20:53:470
2744428814,60sleep227504-21cat23:28:482
241612875,50sleep30-21swapper/319:51:393
1868728629,40sleep218731-21latency_hist21:18:472
2587228519,48sleep025891-21modprobe19:58:460
277802847,12sleep30-21swapper/323:29:053
2147828413,55sleep021492-21kernelversion23:09:080
1185228410,24sleep011871-21apt-get22:38:570
3065628321,33sleep10-21swapper/121:54:181
415828216,51sleep04267-21perl22:14:070
2577728240,31sleep30-21swapper/319:57:323
1785528211,54sleep017951-21mailstats21:14:180
994128028,32sleep39915-21/usr/sbin/munin19:08:483
2296528014,29sleep10-21swapper/123:13:571
2178928017,15sleep20-21swapper/221:28:482
796727915,43sleep07989-21sshd20:43:580
708727940,28sleep30-21swapper/300:04:453
55427923,31sleep10-21swapper/120:19:221
55427923,31sleep10-21swapper/120:19:221
2718527941,27sleep30-21swapper/323:26:063
2694327917,29sleep10-21swapper/121:44:011
2535027919,24sleep10-21swapper/123:19:181
184752797,43sleep10-21swapper/119:34:051
643927823,30sleep16436-21apt-get20:38:591
283027825,35sleep22856-21munin-run20:28:472
2463027819,38sleep024782-21unixbench_singl21:35:120
2251227840,29sleep30-21swapper/319:45:213
2099927812,16sleep10-21swapper/123:08:491
132792780,6sleep10-21swapper/120:59:201
132792780,6sleep10-21swapper/120:59:201
100682780,7sleep00-21swapper/022:33:470
49142779,14sleep10-21swapper/123:59:101
254802779,12sleep20-21swapper/221:39:062
2270627715,45sleep122835-21unixbench_multi21:29:161
2262027721,38sleep022703-21threads21:29:150
1625727717,41sleep016295-21grep21:09:130
1535227715,47sleep015353-21sshd21:06:590
1535227715,47sleep015353-21sshd21:06:590
1423327717,45sleep214266-21cpu21:04:022
1423327717,45sleep214266-21cpu21:04:022
116927718,43sleep11221-21interrupts22:04:071
116927718,43sleep11221-21interrupts22:04:071
2161827624,26sleep20-21swapper/219:44:072
2131827619,26sleep00-21swapper/019:43:490
1743827621,16sleep20-21swapper/219:29:382
165302769,51sleep00-21swapper/022:54:020
1289227622,26sleep00-21swapper/019:14:180
572127520,43sleep00-21swapper/000:02:030
346127515,22sleep20-21swapper/223:54:142
3161127519,23sleep00-21swapper/021:58:490
2944227420,31sleep10-21swapper/123:34:071
2950027318,37sleep229518-21kernelversion23:34:082
2527627318,27sleep10-21swapper/119:54:191
208327314,28sleep10-21swapper/123:49:171
140932736,46sleep00-21swapper/019:19:100
1185627313,44sleep20-21swapper/220:54:182
728527217,27sleep20-21swapper/222:24:062
695327216,37sleep00-21swapper/022:23:470
2455627214,30sleep00-21swapper/023:18:570
2057927219,29sleep10-21swapper/119:39:181
2133227116,35sleep221418-21/usr/sbin/munin23:09:062
644627021,27sleep00-21swapper/020:39:020
1038827019,21sleep00-21swapper/022:34:040
2839326924,25sleep20-21swapper/220:04:212
988526827,23sleep20-21swapper/219:08:472
25702689,44sleep00-21swapper/023:51:580
142626818,27sleep00-21swapper/022:04:140
142626818,27sleep00-21swapper/022:04:140
1377826819,28sleep10-21swapper/100:28:481
929926715,37sleep19441-21cat20:48:481
2515426718,25sleep10-21swapper/121:38:481
2179426715,28sleep10-21swapper/119:44:101
1907726717,33sleep219198-21/usr/sbin/munin19:34:182
1629726725,23sleep116374-21cat21:09:141
1505626718,24sleep20-21swapper/219:23:482
519526619,24sleep20-21swapper/223:59:192
2353226614,36sleep223556-21/usr/sbin/munin19:49:162
164662666,47sleep00-21swapper/000:34:370
1328126613,36sleep013334-21users00:24:190
1303726619,28sleep20-21swapper/200:24:172
540626519,20sleep10-21swapper/122:18:481
425526519,29sleep14316-21sendmail-msp23:58:451
3262926512,36sleep132722-21memory23:44:081
2875326512,36sleep028765-21/usr/sbin/munin21:49:090
2818326521,26sleep00-21swapper/023:29:140
1974826517,31sleep219836-21tune2fs23:04:052
1974826517,31sleep219836-21tune2fs23:04:052
1692726522,21sleep20-21swapper/222:54:132
34726428,20sleep00-21swapper/020:19:120
34726428,20sleep00-21swapper/020:19:120
3136226419,28sleep231448-21cat21:58:472
2825226422,18sleep00-21swapper/021:48:480
2740426414,34sleep227436-21/usr/sbin/munin21:44:142
263672646,11sleep10-21swapper/119:59:071
1431426418,21sleep10-21swapper/121:04:051
1431426418,21sleep10-21swapper/121:04:051
121126418,25sleep00-21swapper/023:48:480
1155826416,27sleep111602-21awk22:38:461
76562635,10sleep20-21swapper/200:08:522
699126321,25sleep20-21swapper/220:39:162
3006526319,31sleep030134-21/usr/sbin/munin23:34:190
2704926319,27sleep227102-21/usr/sbin/munin19:59:252
1818426313,33sleep118188-21diskstats22:59:041
1534626318,27sleep00-21swapper/022:49:100
1532626327,19sleep10-21swapper/100:33:491
856726212,36sleep28636-21switchtime20:44:172
3138226216,25sleep00-21swapper/023:39:150
3092526211,35sleep130932-21diskstats20:14:051
2899326220,18sleep20-21swapper/223:33:482
181926216,31sleep21847-21/usr/sbin/munin23:49:102
1080226216,21sleep00-21swapper/000:18:540
106742628,12sleep10-21swapper/119:09:141
612526118,26sleep06150-21munin-run20:38:460
2144126111,35sleep021442-21sshd21:24:590
1689726113,31sleep116926-21/usr/sbin/munin22:54:111
1197826120,21sleep20-21swapper/219:13:562
3081026014,19sleep20-21swapper/223:39:012
2459226017,26sleep224613-21apt-get19:54:002
1535026016,24sleep10-21swapper/119:24:041
1529226014,23sleep00-21swapper/000:33:480
1219026017,20sleep10-21swapper/100:23:481
120112607,11sleep00-21swapper/020:54:200
1176726014,24sleep10-21swapper/119:13:501
76932597,12sleep00-21swapper/000:09:020
2364525919,21sleep00-21swapper/019:49:180
138802597,13sleep20-21swapper/222:44:112
302412586,10sleep10-21swapper/120:09:451
2669725814,26sleep00-21swapper/021:43:480
2390125817,26sleep223903-21diskstats21:34:052
1973225820,24sleep019813-21latency_hist19:38:480
308425714,21sleep20-21swapper/222:09:162
2989825714,31sleep229942-21switchtime20:09:182
1807525718,22sleep218094-21apt-get22:58:572
1553225711,20sleep10-21swapper/100:34:011
137822578,10sleep00-21swapper/021:01:420
137822578,10sleep00-21swapper/021:01:420
1334125716,27sleep213455-21timerwakeupswit20:59:222
1334125716,27sleep213455-21timerwakeupswit20:59:222
1238125715,26sleep212460-21systemctl22:39:152
1029725714,27sleep210293-21apt-get22:33:582
1029525712,31sleep210296-21sshd00:15:582
1029525712,31sleep210296-21sshd00:15:582
62425614,23sleep20-21swapper/220:19:232
62425614,23sleep20-21swapper/220:19:232
529925616,24sleep15318-21ntp_states20:34:111
45402564,18sleep20-21swapper/220:33:192
2800325616,25sleep128026-21irqstats20:04:121
1529925612,27sleep015349-21/usr/sbin/munin19:24:030
1374425615,24sleep113741-21apt-get19:18:591
1039525615,18sleep20-21swapper/219:09:072
572625514,26sleep15730-21diskstats22:19:041
331925513,25sleep13306-21apt-get20:29:021
2006025523,18sleep20-21swapper/219:39:062
1684525514,25sleep116838-21apt-get19:29:001
964025417,17sleep00-21swapper/000:14:100
964025417,17sleep00-21swapper/000:14:100
405925413,25sleep24053-21apt-get22:14:012
1734725415,23sleep117342-21apt-get21:14:021
1890325313,26sleep118906-21munin-node21:18:591
1497625313,23sleep214968-21apt-get22:49:002
1015925313,20sleep10-21swapper/120:49:201
1015925313,20sleep10-21swapper/120:49:201
296242525,16sleep20-21swapper/221:52:092
1900925212,23sleep019015-21diskstats21:19:060
1623725214,22sleep216248-21ntp_states21:09:112
1188225211,17sleep10-21swapper/122:39:021
923725111,23sleep19229-21apt-get00:13:591
923725111,23sleep19229-21apt-get00:13:591
1963325119,18sleep10-21swapper/123:03:561
1684225113,15sleep00-21swapper/019:28:590
160725114,21sleep01613-21diskstats20:24:050
1172925115,18sleep10-21swapper/120:54:171
620425016,17sleep20-21swapper/220:38:482
1468425012,17sleep00-21swapper/022:48:460
1832224911,22sleep018371-21irqstats22:59:070
2588324813,15sleep20-21swapper/223:23:472
1549224812,16sleep10-21swapper/122:49:151
2640424714,18sleep00-21swapper/023:24:090
2299124712,16sleep20-21swapper/223:13:582
1066724612,20sleep010673-21ntp_states19:09:130
2793924516,15sleep00-21swapper/020:04:110
293032449,18sleep029370-21df20:09:030
1678824315,15sleep20-21swapper/200:38:462
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional