You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-05 - 22:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Jun 05, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25263286616,832sleep025289-21/usr/sbin/munin09:13:370
793527399,667sleep17907-21/usr/sbin/munin10:03:091
20889267813,650sleep020890-21sshd10:43:360
14461248511,459sleep014505-21ntpq12:03:310
1142624097,390sleep111462-21ntpq11:54:281
1124224028,382sleep111276-21ps08:28:341
272342315134,164sleep127328-21fschecks_time11:03:271
245622300140,136sleep10-21swapper/109:13:091
28781229223,253sleep328803-21fschecks_count11:08:253
3249223713,208sleep33319-21sensors11:28:393
30478222642,163sleep130603-21latency_hist07:48:101
25328219517,41sleep125480-21latency_hist10:58:101
3605219011,164sleep33679-21find08:03:383
21967218912,154sleep122057-21fschecks_time12:28:261
21949218813,159sleep121664-21/usr/sbin/munin09:03:291
6102218718,157sleep36168-21sort08:13:223
2180218127,134sleep32197-21apt-get11:28:063
15540217913,154sleep115606-21fschecks_time08:43:271
26006217619,52sleep10-21swapper/110:58:341
32409216612,138sleep032441-21tune2fs07:53:260
11420216415,44sleep10-21swapper/110:13:261
23523216214,87sleep123622-21cut12:33:261
6586215819,77sleep16652-21awk08:13:371
6563215815,128sleep06585-21awk08:13:360
21439215711,12sleep10-21swapper/110:44:141
16144215521,120sleep117481-21cyclictest12:08:361
16144215521,120sleep117481-21cyclictest12:08:361
13231215217,122sleep113274-21ntp_kernel_pll_10:18:311
20902215113,117sleep120961-21basename07:18:081
20097214617,15sleep10-21swapper/112:23:091
1054521438,37sleep10-21swapper/108:28:101
1655821405,10sleep20-21swapper/212:08:412
1655821405,10sleep20-21swapper/212:08:412
30751213421,90sleep230755-21find07:48:192
19159213010,21sleep10-21swapper/108:53:361
10301212918,97sleep110393-21awk10:08:341
5603212612,39sleep10-21swapper/111:38:091
507212619,59sleep10-21swapper/111:21:351
1384521246,14sleep10-21swapper/112:03:071
21105212318,38sleep10-21swapper/112:23:381
8441212288,16sleep10-21swapper/110:03:301
792321209,16sleep10-21swapper/108:18:291
31739211913,89sleep10-21swapper/109:33:391
18530211913,31sleep218532-21/usr/sbin/munin08:53:172
16519211916,44sleep116586-21cat10:28:361
7777211815,87sleep27812-21ntp_kernel_pll_11:43:312
5333211811,41sleep10-21swapper/109:53:451
27697211861,42sleep127837-21latency_hist09:23:101
11535211814,91sleep311594-21latency10:13:293
3176321178,11sleep10-21swapper/107:49:401
1785211711,87sleep31845-21/usr/sbin/munin09:43:263
23228211414,37sleep10-21swapper/107:23:361
25946211313,86sleep025953-21forks07:33:280
25946211313,86sleep025953-21forks07:33:280
1546021130,9sleep20-21swapper/208:43:232
14307211313,31sleep10-21swapper/112:03:281
917921128,91sleep19161-21apt-get11:48:261
65021129,38sleep10-21swapper/109:38:361
297221127,16sleep00-21swapper/008:03:140
22935211213,77sleep022996-21sendmail_mailqu10:49:050
333221119,36sleep10-21swapper/109:48:241
30678211118,81sleep10-21swapper/109:32:221
1566221118,43sleep115688-21munin-run10:28:071
9320211016,67sleep20-21swapper/208:23:262
2834211017,79sleep12863-21grep11:28:291
18333211010,35sleep10-21swapper/110:33:401
888921098,83sleep11222-1kworker/1:0H11:48:101
11279210912,82sleep211359-21cat08:28:352
26316210811,32sleep10-21swapper/107:33:371
26316210811,32sleep10-21swapper/107:33:371
16587210816,75sleep216720-21timerwakeupswit10:28:372
24889210739,53sleep10-21swapper/107:28:371
15077210715,58sleep10-21swapper/108:42:021
310521066,32sleep10-21swapper/108:03:281
29170210613,79sleep029206-21idleruntime07:43:290
19819210314,75sleep019885-21modprobe08:58:070
2227521026,10sleep10-21swapper/107:20:341
31592210117,66sleep231639-21grep11:18:092
229992997,30sleep10-21swapper/110:49:061
3000329716,68sleep230076-21latency_hist11:13:082
1855729714,69sleep318624-21df08:53:223
25492966,29sleep10-21swapper/108:01:021
2040129613,70sleep020486-21cat08:58:270
200332966,30sleep10-21swapper/110:39:091
2908829512,61sleep10-21swapper/111:08:311
1477029519,57sleep00-21swapper/010:23:300
1039429515,55sleep210525-21unixbench_multi10:08:352
2378129415,43sleep20-21swapper/209:08:342
2322329416,65sleep023229-21cat09:08:160
213112949,68sleep021402-21fschecks_time07:18:270
70942935,30sleep10-21swapper/111:43:061
286152936,29sleep10-21swapper/107:42:051
286152936,29sleep10-21swapper/107:42:051
2059229324,55sleep10-21swapper/108:58:321
2818529216,59sleep328334-21uptime07:38:403
2818529216,59sleep328334-21uptime07:38:403
515129153,18sleep10-21swapper/108:08:361
2480029021,55sleep10-21swapper/109:13:211
2889328921,54sleep10-21swapper/109:24:191
2319228912,62sleep023245-21latency_hist12:33:090
226128815,57sleep12403-21apt-config11:28:081
1409128828,39sleep214085-21apt-get12:03:212
3236828713,59sleep332471-21latency_hist09:38:093
2971528717,49sleep329860-21sed07:43:403
1736728731,27sleep20-21swapper/207:08:082
1716428712,58sleep10-21swapper/112:13:161
44152869,64sleep0363-21snmpd09:51:540
3269128618,54sleep10-21swapper/107:53:311
2428628610,17sleep20-21swapper/207:28:182
1326228623,51sleep10-21swapper/108:33:551
714728511,57sleep07216-21latency_hist11:43:090
263328520,54sleep10-21swapper/109:43:561
1882728522,52sleep10-21swapper/107:11:341
1905928415,52sleep219076-21apt-get10:38:172
2222828310,59sleep26311-21kworker/u8:410:48:182
1744028314,55sleep017558-21sed07:08:100
1737628327,34sleep20-21swapper/212:13:262
833228215,51sleep38148-21/usr/sbin/munin10:03:283
3122128221,51sleep10-21swapper/111:14:331
2665128220,38sleep026671-21uname09:18:290
2103028213,51sleep221043-21awk08:58:372
1769128224,39sleep317856-21grep08:48:383
1720828251,14sleep10-21swapper/108:48:271
2351328128,33sleep30-21swapper/309:08:273
2228728121,28sleep30-21swapper/310:48:483
3111228014,48sleep031119-21/usr/sbin/munin09:33:260
295928019,32sleep00-21swapper/009:48:070
2874728013,52sleep028821-21latency_hist07:43:090
2874728013,52sleep028821-21latency_hist07:43:090
1385528017,39sleep213951-21latency_hist12:03:082
1102827920,34sleep00-21swapper/010:13:090
580327817,41sleep25919-21grep09:53:542
472327718,30sleep30-21swapper/311:33:353
3253127711,52sleep232582-21if_eth007:53:282
2880627721,29sleep00-21swapper/011:08:260
154127722,22sleep00-21swapper/007:58:260
2433527614,18sleep20-21swapper/210:53:312
2196227611,15sleep20-21swapper/209:03:302
1544527621,26sleep30-21swapper/312:08:103
1170627616,37sleep311811-21sort11:54:343
2719327515,43sleep20-21swapper/211:03:242
2477327516,43sleep324895-21latency_hist12:38:083
2248927513,43sleep322587-21/usr/sbin/munin09:03:393
629927423,31sleep36325-21munin-run09:58:073
339227416,40sleep23222-21/usr/sbin/munin09:48:252
3209927415,43sleep232175-21latency11:18:292
2956327424,30sleep20-21swapper/209:28:242
2616227424,25sleep00-21swapper/010:58:370
2307127414,43sleep223187-21timerandwakeup10:49:072
1924627411,31sleep20-21swapper/212:18:342
43927312,44sleep3363-21snmpd07:53:373
2480727316,33sleep024887-21sshd07:28:360
148627314,37sleep31641-21latency_hist09:43:103
72227212,42sleep3879-21sed11:23:103
3208727214,36sleep032057-21latency_hist07:53:100
2805427225,28sleep20-21swapper/207:38:392
2805427225,28sleep20-21swapper/207:38:392
1670727223,26sleep30-21swapper/307:03:373
1294627213,40sleep013037-21fschecks_time10:18:260
1817427129,24sleep30-21swapper/307:08:373
3151427020,25sleep20-21swapper/209:33:362
3021527021,21sleep30-21swapper/311:13:173
1950427020,26sleep20-21swapper/207:13:162
1938927017,29sleep30-21swapper/307:13:133
1439127010,17sleep20-21swapper/208:38:362
1285327014,37sleep212849-21apt-get10:18:202
1057827018,25sleep00-21swapper/011:53:100
196872695,13sleep30-21swapper/312:18:393
1448526917,32sleep314519-21fschecks_count10:23:253
2382426818,26sleep20-21swapper/212:33:312
2243426815,31sleep30-21swapper/312:28:363
92252675,51sleep00-21swapper/010:05:300
227626719,27sleep20-21swapper/209:43:372
2081926711,14sleep20-21swapper/212:23:352
942926618,31sleep20-21swapper/210:08:072
2987426614,35sleep029919-21awk09:28:300
2906826616,34sleep229072-21diskstats07:43:272
187302667,44sleep20-21swapper/207:10:182
1747026618,27sleep017522-21processes08:48:350
1572326615,35sleep015781-21sed08:43:300
1367226620,27sleep00-21swapper/008:38:090
877926414,25sleep00-21swapper/011:48:090
81472647,12sleep30-21swapper/308:18:353
2606826416,27sleep326092-21munin-run09:18:083
238212646,12sleep00-21swapper/007:25:330
2183426415,32sleep321195-21/usr/sbin/munin07:18:383
2176526413,22sleep20-21swapper/207:18:372
2104426411,11sleep30-21swapper/308:58:383
2001926418,24sleep00-21swapper/012:22:480
1145526416,32sleep010757-21/usr/sbin/munin08:28:370
970026320,22sleep30-21swapper/308:23:363
682726319,24sleep26593-21/usr/sbin/munin09:58:302
1900526318,26sleep00-21swapper/007:13:100
1103126322,22sleep00-21swapper/011:54:190
102662639,41sleep0363-21snmpd08:25:140
2993926216,24sleep30-21swapper/309:28:333
2686126213,35sleep226907-21munin-run11:03:072
2278726211,37sleep322826-21memory10:49:003
805626122,22sleep00-21swapper/008:18:320
2477326119,20sleep30-21swapper/307:28:353
206052617,36sleep00-21swapper/012:23:280
1928926115,30sleep30-21swapper/310:38:263
1744526113,27sleep217457-21sh08:48:322
1228126121,25sleep21-21systemd11:58:082
670526021,19sleep00-21swapper/009:58:280
379026021,21sleep30-21swapper/309:48:343
350926021,18sleep20-21swapper/208:03:372
2537626016,20sleep20-21swapper/209:13:392
2436726013,20sleep30-21swapper/310:53:323
2236426012,33sleep222433-21systemctl12:28:352
1776926020,22sleep30-21swapper/312:13:353
1564326015,21sleep30-21swapper/312:08:223
1564326015,21sleep30-21swapper/312:08:223
632325916,24sleep36406-21sshd11:38:363
559125913,33sleep05599-21systemd-run11:38:080
272262590,6sleep20-21swapper/209:18:382
1592525913,26sleep315944-21apt-get10:28:173
2919225814,25sleep20-21swapper/209:28:072
1634025813,23sleep00-21swapper/010:28:310
976925718,20sleep30-21swapper/311:48:533
6062578,37sleep0647-21ps09:38:350
505425715,25sleep05029-21apt-get09:53:320
455325715,25sleep24547-21apt-get08:08:212
2714725718,25sleep027141-21apt-get11:03:220
3260325614,26sleep232597-21apt-get09:38:202
1796925610,26sleep00-21swapper/010:33:350
665425518,17sleep20-21swapper/208:13:382
323102559,33sleep032374-21sendmail_mailst11:18:350
2678625516,19sleep30-21swapper/309:18:323
2585925515,24sleep225856-21apt-get07:33:222
2585925515,24sleep225856-21apt-get07:33:222
241825515,23sleep22598-21apt-config11:28:102
1126525513,26sleep211260-21apt-get10:13:192
722525416,24sleep30-21swapper/309:58:393
3074225413,26sleep330260-21apt-get07:48:163
2530025421,19sleep00-21swapper/010:58:090
156725413,21sleep20-21swapper/211:23:382
1390825414,25sleep013902-21apt-get08:38:210
1043225420,16sleep20-21swapper/211:53:082
594325314,24sleep26043-21perl11:38:272
434625315,17sleep20-21swapper/211:33:262
144625312,25sleep31441-21apt-get07:58:213
1279425311,25sleep30-21swapper/308:33:363
1233125311,25sleep212325-21apt-get08:33:232
2298225213,23sleep023013-21munin-run09:08:080
164542529,17sleep00-21swapper/012:08:390
164542529,17sleep00-21swapper/012:08:390
139492527,34sleep30-21swapper/310:21:033
1077925212,23sleep310775-21apt-get08:28:203
868525115,22sleep38774-21systemd-run11:48:083
2845825114,16sleep30-21swapper/311:08:093
2666925112,22sleep326767-21users07:33:433
2666925112,22sleep326767-21users07:33:433
2327125111,25sleep223351-21tr07:23:372
221725113,22sleep00-21swapper/011:28:070
1951925110,17sleep00-21swapper/010:38:320
1784725116,18sleep00-21swapper/012:13:360
494925011,17sleep00-21swapper/008:08:310
3151225013,21sleep031537-21munin-run11:18:070
2813925015,17sleep30-21swapper/309:23:283
206582505,12sleep00-21swapper/007:15:080
202025017,15sleep20-21swapper/207:58:362
1750825011,22sleep317503-21apt-get10:33:223
1545225012,21sleep315448-21apt-get08:43:213
1398125014,20sleep30-21swapper/308:38:253
2554224912,21sleep325538-21apt-get10:58:203
1758224914,21sleep217588-21df_inode10:33:252
1135324914,17sleep00-21swapper/010:13:250
38892485,30sleep00-21swapper/011:32:070
2370124811,14sleep00-21swapper/012:33:290
1466724813,15sleep20-21swapper/210:23:282
2284524710,22sleep322741-21/usr/sbin/munin07:23:263
2793224615,15sleep00-21swapper/009:23:210
2090524613,17sleep30-21swapper/312:23:363
169632467,16sleep10-21swapper/107:03:391
829124511,14sleep20-21swapper/208:18:382
49192459,15sleep30-21swapper/308:08:303
1017624212,15sleep30-21swapper/310:08:323
128122418,19sleep012906-21memory11:58:300
1253424112,15sleep30-21swapper/311:58:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional