You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-18 - 21:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Mar 18, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
194172221012,2182sleep319387-21apt-get12:18:243
5885280312,775sleep122524-1kworker/1:1H08:08:071
1566327186,699sleep215706-21ntpq08:38:202
6518245912,431sleep06552-21/usr/sbin/munin09:53:210
8732237717,336sleep08752-21apt08:17:550
13207233113,156sleep213301-21threads10:13:342
23922232017,285sleep123998-21/usr/sbin/munin07:23:241
22258225694,45sleep20-21swapper/212:27:562
7406225040,194sleep27442-21awk09:57:552
7376224317,203sleep27542-21unixbench_multi11:38:262
14568222231,175sleep214676-21fschecks_time12:03:172
32691221013,180sleep132732-21latency_hist11:17:571
11935220432,125sleep212020-21awk11:53:282
2008422020,7sleep00-21swapper/008:53:150
2008422020,7sleep00-21swapper/008:53:150
264192196110,63sleep20-21swapper/209:13:172
19389218811,123sleep20-21swapper/212:18:072
18841218512,99sleep218872-21grep12:17:552
1190621809,98sleep212015-21cut08:27:582
22057217534,115sleep20-21swapper/210:43:142
22896217412,147sleep222940-21ntpq12:28:212
23734217135,120sleep223865-21grep09:03:282
23734217135,120sleep223865-21grep09:03:282
635216953,73sleep20-21swapper/211:18:152
19402216918,29sleep00-21swapper/012:18:130
2581921677,126sleep213441-21snmpd09:12:162
2581921677,126sleep213441-21snmpd09:12:162
31459216667,81sleep231593-21timerwakeupswit09:28:272
1784216510,143sleep01823-21expr09:38:230
1775216117,126sleep31777-21cron07:57:553
31147216035,109sleep231298-21latency_hist11:12:582
784215994,40sleep20-21swapper/207:53:172
3178821596,141sleep00-21swapper/009:28:580
20015215635,21sleep20-21swapper/207:13:032
31427215432,101sleep20-21swapper/211:13:142
6293215318,21sleep00-21swapper/008:08:260
18419215380,55sleep218476-21fschecks_count07:08:152
18419215380,55sleep218476-21fschecks_count07:08:152
11047215378,57sleep20-21swapper/211:52:552
10246215320,115sleep010272-21/usr/sbin/munin11:48:210
29063214855,32sleep229217-21latency_hist09:22:582
26876214848,29sleep226941-21grep07:33:142
16246214512,117sleep316247-21sshd08:39:223
2476214264,66sleep22481-21fschecks_count07:58:342
5385214119,88sleep20-21swapper/208:07:552
22814214122,95sleep00-21swapper/009:02:570
18071214017,100sleep118118-21latency_hist07:07:581
4113213970,16sleep20-21swapper/211:28:242
1891621376,10sleep10-21swapper/110:33:111
17328213641,73sleep20-21swapper/212:12:582
24042213517,26sleep20-21swapper/212:33:092
22140213420,96sleep20-21swapper/207:18:202
18840213315,21sleep00-21swapper/008:48:240
18840213315,21sleep00-21swapper/008:48:240
5655212812,66sleep20-21swapper/211:33:252
17274212821,91sleep317306-21basename12:12:553
12181212612,98sleep212336-21latency_hist10:12:572
23521212534,45sleep20-21swapper/210:48:102
9741212434,19sleep29894-21sed08:18:252
7904212335,18sleep27946-21cat09:58:152
29926212232,13sleep20-21swapper/207:43:122
29926212232,13sleep20-21swapper/207:43:122
27192212112,18sleep20-21swapper/210:58:242
15020212116,15sleep30-21swapper/308:37:573
5840212013,80sleep05976-21grep11:33:270
15268212018,26sleep00-21swapper/008:38:100
690721195,64sleep20-21swapper/208:09:292
1844821177,45sleep218345-21/usr/sbin/munin08:48:132
1844821177,45sleep218345-21/usr/sbin/munin08:48:132
19113211612,89sleep219133-21kernelversion10:33:162
23705211510,92sleep023757-21cat10:48:160
23145211516,75sleep023278-21ls12:28:270
17564211520,72sleep317616-21cpuspeed12:13:113
15043211513,87sleep015188-21latency_hist08:37:580
8878211426,12sleep20-21swapper/211:43:262
6470211419,74sleep36622-21latency_hist11:37:583
23516211416,83sleep323557-21grep09:03:253
23516211416,83sleep323557-21grep09:03:253
21497211417,84sleep221490-21apt-get08:58:112
1513121136,14sleep00-21swapper/010:18:450
1513121136,14sleep00-21swapper/010:18:450
14171211322,18sleep20-21swapper/210:18:062
14171211322,18sleep20-21swapper/210:18:062
414921126,10sleep20-21swapper/209:45:522
4130211116,75sleep04114-21apt-get08:03:130
9371211014,78sleep09459-21/usr/sbin/munin10:03:140
19368210914,78sleep118860-21apt-get12:18:061
1322921095,10sleep20-21swapper/208:30:452
29288210812,16sleep00-21swapper/009:23:080
2945121075,66sleep20-21swapper/211:07:122
2061621060,6sleep00-21swapper/007:13:210
114221065,10sleep20-21swapper/209:37:202
568321046,64sleep20-21swapper/209:50:402
26553210414,65sleep20-21swapper/207:32:582
10578210420,63sleep210590-21latency_hist10:07:562
8464210219,60sleep18616-21sed09:58:261
26326210210,22sleep226359-21sendmail10:57:552
13397210212,78sleep213478-21mailstats11:58:242
1501210169,20sleep21503-21apt-get09:38:082
3026121007,74sleep00-21swapper/011:08:230
3218029815,64sleep132331-21latency_hist09:32:581
2531829813,25sleep00-21swapper/009:08:260
2531829813,25sleep00-21swapper/009:08:260
2318429712,69sleep323304-21latency_hist07:22:583
1611929613,70sleep20-21swapper/210:23:192
1611929613,70sleep20-21swapper/210:23:192
2538429511,66sleep10-21swapper/112:37:571
247642940,6sleep10-21swapper/109:08:141
247642940,6sleep10-21swapper/109:08:141
236702947,13sleep20-21swapper/207:23:172
1724529431,34sleep30-21swapper/307:03:243
2874629318,20sleep00-21swapper/011:03:230
2636529312,68sleep126418-21/usr/sbin/munin09:13:161
235982939,55sleep10-21swapper/110:48:131
1786329314,64sleep217943-21cat10:28:242
177922936,15sleep20-21swapper/208:44:222
206412927,14sleep20-21swapper/210:38:172
206412927,14sleep20-21swapper/210:38:172
3171129115,59sleep331317-21/usr/sbin/munin11:13:193
3113629121,51sleep131152-21apt07:47:551
3113629121,51sleep131152-21apt07:47:551
2955829122,51sleep129606-21latency09:23:171
256429110,61sleep20-21swapper/211:23:232
1948629127,19sleep30-21swapper/310:33:253
1137529115,58sleep010802-21/usr/sbin/munin10:08:270
3273429014,17sleep20-21swapper/207:52:572
292902908,67sleep20-21swapper/209:23:102
2879129014,59sleep128815-21/usr/sbin/munin07:38:221
2640729023,50sleep00-21swapper/010:57:570
1030729024,46sleep010331-21awk08:22:560
1641028812,60sleep016449-21ntpq12:08:200
379828722,46sleep13948-21grep09:43:281
2288628718,49sleep021945-21/usr/sbin/munin10:43:280
2103028715,19sleep30-21swapper/312:23:133
1893928724,33sleep00-21swapper/012:17:570
1495028711,23sleep20-21swapper/208:37:562
85412867,10sleep20-21swapper/208:15:312
2584428617,46sleep125947-21grep07:28:411
1267428626,28sleep30-21swapper/311:57:573
294552855,12sleep20-21swapper/207:41:142
210932857,12sleep20-21swapper/208:56:492
210932857,12sleep20-21swapper/208:56:492
1665528520,45sleep315718-21/usr/sbin/munin10:23:283
1665528520,45sleep315718-21/usr/sbin/munin10:23:283
976328413,53sleep19846-21mailstats10:03:231
3204028413,52sleep332167-21/usr/sbin/munin07:48:263
2398428416,50sleep323999-21/usr/sbin/munin10:48:233
3216928317,46sleep032291-21grep07:48:270
126728315,45sleep31394-21latency_hist09:37:573
359828212,54sleep33636-21http08:02:563
1683228224,35sleep216919-21fschecks_time07:03:152
2257728113,49sleep322580-21diskstats12:28:143
418128019,41sleep34288-21timerandwakeup11:28:253
1101028017,50sleep311046-21idleruntime10:08:183
288542797,60sleep00-21swapper/009:21:020
73927811,51sleep0828-21sed11:18:170
232802780,7sleep10-21swapper/112:28:271
997327717,31sleep10-21swapper/111:48:141
2712727716,17sleep10-21swapper/110:58:231
780627512,47sleep07811-21/usr/sbin/munin09:58:130
347527522,27sleep30-21swapper/309:43:223
2517627412,48sleep00-21swapper/007:28:040
2178627420,25sleep30-21swapper/310:42:583
187227414,41sleep11987-21latency_hist07:57:571
1749527419,34sleep017645-21tail08:43:270
1153527413,15sleep10-21swapper/111:53:191
219322730,11sleep10-21swapper/107:18:141
2042827315,43sleep00-21swapper/010:38:110
2042827315,43sleep00-21swapper/010:38:110
1886927310,16sleep00-21swapper/007:08:250
1886927310,16sleep00-21swapper/007:08:250
2661727213,44sleep026636-21apt-get10:58:060
2488027122,27sleep30-21swapper/309:08:173
2488027122,27sleep30-21swapper/309:08:173
2350627115,36sleep323399-21/usr/sbin/munin07:23:143
333327017,27sleep30-21swapper/311:27:573
248027020,29sleep00-21swapper/011:23:210
1278327015,25sleep00-21swapper/010:13:240
97302699,41sleep30-21swapper/310:03:213
637826916,21sleep10-21swapper/111:37:561
432926917,38sleep14387-21if_err_eth008:03:301
2841526915,29sleep00-21swapper/007:38:130
1775026916,28sleep00-21swapper/010:28:220
255282688,12sleep10-21swapper/110:53:221
2188226814,22sleep00-21swapper/007:18:140
2159426824,22sleep10-21swapper/108:58:181
1212026824,29sleep112150-21munin-run10:12:561
783026722,25sleep10-21swapper/108:13:201
429126714,24sleep00-21swapper/011:28:260
2356026714,26sleep00-21swapper/009:03:260
2356026714,26sleep00-21swapper/009:03:260
1283426715,32sleep10-21swapper/110:13:251
1143326714,37sleep011534-21fschecks_time11:53:180
955226618,29sleep10-21swapper/111:47:581
277842667,17sleep00-21swapper/007:34:580
271122669,11sleep00-21swapper/009:14:000
2089826620,23sleep30-21swapper/307:13:293
1807726616,31sleep10-21swapper/108:47:561
1453926616,36sleep314546-21df_inode12:03:143
118826626,22sleep10-21swapper/109:37:551
921526513,36sleep39272-21basename08:18:143
824126524,19sleep30-21swapper/311:43:113
656726516,22sleep30-21swapper/308:08:333
32892656,11sleep10-21swapper/107:58:581
177226525,19sleep30-21swapper/311:22:563
121326526,20sleep10-21swapper/111:18:261
881226416,16sleep30-21swapper/308:17:583
799226416,20sleep00-21swapper/011:42:560
47052647,43sleep14707-21sshd11:31:221
2079526417,28sleep120839-21ntpq10:38:201
2079526417,28sleep120839-21ntpq10:38:201
1964426419,23sleep119708-21munin-node07:12:581
1964426419,23sleep119708-21munin-node07:12:581
1330326418,29sleep313431-21/usr/sbin/munin10:13:353
623526320,20sleep30-21swapper/309:53:163
2342426315,22sleep00-21swapper/007:23:110
206632636,13sleep30-21swapper/308:53:273
206632636,13sleep30-21swapper/308:53:273
199326319,22sleep30-21swapper/309:38:303
189062639,41sleep118976-21sensors08:48:251
189062639,41sleep118976-21sensors08:48:251
3007026218,21sleep30-21swapper/311:08:183
1660026218,27sleep10-21swapper/112:08:251
1454826212,17sleep00-21swapper/012:03:150
93892617,18sleep00-21swapper/011:46:310
1385926114,28sleep00-21swapper/011:58:550
2965726016,29sleep329691-21/usr/sbin/munin09:23:193
2624226020,20sleep30-21swapper/309:13:143
1116826018,19sleep10-21swapper/108:23:251
2824725912,26sleep30-21swapper/311:03:093
2131325911,29sleep00-21swapper/012:23:180
2066025912,26sleep10-21swapper/107:13:231
1883025914,29sleep318831-21ntp_states08:48:233
1883025914,29sleep318831-21ntp_states08:48:233
2441025815,27sleep324444-21/usr/sbin/munin12:33:183
1800925822,18sleep00-21swapper/007:07:560
178125815,26sleep01871-21apt-config07:57:560
130612577,39sleep10-21swapper/108:28:341
558225610,31sleep15584-21sshd09:49:221
501125613,17sleep00-21swapper/009:48:200
2835025615,26sleep328424-21find09:18:273
238925620,20sleep10-21swapper/111:23:181
2115325614,21sleep10-21swapper/112:23:151
1327825619,23sleep113305-21awk11:58:191
931525514,21sleep10-21swapper/108:18:161
669425515,22sleep30-21swapper/311:38:103
585825511,29sleep35859-21cron09:52:553
279125513,23sleep12812-21munin-run09:42:561
278102557,40sleep10-21swapper/107:35:191
2444625515,22sleep00-21swapper/012:33:190
2128625515,17sleep10-21swapper/108:57:571
2128625515,17sleep10-21swapper/108:57:571
2039925515,24sleep30-21swapper/310:38:063
2039925515,24sleep30-21swapper/310:38:063
1779925516,20sleep10-21swapper/112:13:171
1425025515,24sleep314401-21latency_hist12:02:583
986925411,18sleep30-21swapper/311:48:123
800025420,15sleep10-21swapper/111:42:571
636825412,25sleep16417-21irqstats09:53:181
2987525412,25sleep329871-21apt-get07:43:093
2987525412,25sleep329871-21apt-get07:43:093
2506425414,23sleep325059-21apt-get10:53:083
2207525417,19sleep30-21swapper/308:58:283
1259825415,23sleep312604-21ntp_states08:28:243
3054125312,24sleep030671-21sed07:43:260
3054125312,24sleep030671-21sed07:43:260
2224125314,17sleep10-21swapper/110:43:171
1708025320,20sleep30-21swapper/308:43:183
363825218,17sleep00-21swapper/008:02:570
3159425219,18sleep30-21swapper/309:28:283
2954325214,22sleep129557-21munin-run11:07:561
2727225211,25sleep327370-21find10:58:253
1728825212,23sleep317283-21apt-get10:28:083
1573025214,23sleep10-21swapper/108:38:231
1449725212,21sleep114589-21users08:33:261
1059225213,24sleep010588-21apt-get08:23:080
305625112,23sleep03050-21apt-get09:43:110
2517425112,25sleep324698-21apt-get07:28:033
163792519,26sleep016508-21sed10:23:250
163792519,26sleep016508-21sed10:23:250
1574425113,23sleep115771-21grep10:23:101
1574425113,23sleep115771-21grep10:23:101
1514825111,16sleep10-21swapper/112:03:281
3142225012,21sleep131420-21apt-get07:48:071
3017925016,18sleep10-21swapper/111:08:201
2675425010,24sleep326750-21apt-get07:33:073
1832524910,22sleep318328-21apt-get07:08:083
1832524910,22sleep318328-21apt-get07:08:083
1550624913,15sleep10-21swapper/110:22:581
1550624913,15sleep10-21swapper/110:22:581
3238624813,20sleep032381-21apt-get09:33:100
3187624812,19sleep00-21swapper/011:13:240
2905324815,19sleep10-21swapper/109:22:571
2832924812,20sleep328325-21apt-get07:38:083
1223524813,15sleep00-21swapper/008:28:170
2331124710,15sleep10-21swapper/109:03:191
2331124710,15sleep10-21swapper/109:03:191
1139224712,18sleep30-21swapper/308:23:273
441724614,16sleep30-21swapper/308:03:323
1735824510,20sleep117365-21df_inode10:28:111
813724411,18sleep38173-21ntp_states09:58:203
200252439,20sleep020027-21sshd10:36:540
141902439,19sleep314191-21apt-get10:18:073
141902439,19sleep314191-21apt-get10:18:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional