You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 21:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Mar 31, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6546280915,777sleep16549-21diskstats10:27:281
2875827529,731sleep24452-1kworker/2:1H11:37:292
24215268821,652sleep124248-21/usr/sbin/munin11:22:341
78402582541,25sleep07843-21sshd07:06:270
13988244513,273sleep314152-21grep07:22:413
31727243613,411sleep126377-21kworker/1:211:46:171
24395235015,189sleep324465-21switchtime11:22:413
27058231075,23sleep30-21swapper/308:07:113
27058231075,23sleep30-21swapper/308:07:113
14346228948,228sleep314416-21switchtime09:07:403
115262288116,21sleep30-21swapper/307:17:123
306832269109,136sleep30-21swapper/311:42:483
10874226838,216sleep310905-21latency12:22:313
12452224877,25sleep30-21swapper/310:47:123
7412223364,98sleep37404-21apt-get12:12:263
162802233133,26sleep30-21swapper/310:57:373
15409223244,41sleep315428-21fschecks_count09:12:293
20245223047,140sleep30-21swapper/309:27:233
6152222265,20sleep30-21swapper/312:07:343
6152222265,20sleep30-21swapper/312:07:343
10742221829,158sleep30-21swapper/308:57:273
12221221312,188sleep012229-21df12:27:250
12221221312,188sleep012229-21df12:27:250
27543221129,168sleep327644-21timerandwakeup11:32:423
1269922077,188sleep312191-21/usr/sbin/munin09:02:393
3162622060,12sleep00-21swapper/010:02:390
3162622060,12sleep00-21swapper/010:02:390
31092220653,103sleep331137-21ntpq08:17:533
23877220210,176sleep323934-21grep07:57:103
30215220130,116sleep330390-21cat08:17:123
30215220130,116sleep330390-21cat08:17:123
14001219722,151sleep314147-21latency_hist10:52:123
14001219722,151sleep314147-21latency_hist10:52:123
2682121930,14sleep38018-21cyclictest09:47:323
2682121930,14sleep38018-21cyclictest09:47:323
12834218838,21sleep312983-21tail12:27:403
12834218838,21sleep312983-21tail12:27:403
13113218267,47sleep313261-21latency_hist07:22:133
10342218213,144sleep30-21swapper/308:57:103
10342218213,144sleep30-21swapper/308:57:103
1959921798,22sleep30-21swapper/307:42:283
23603217612,142sleep323574-21latency_hist11:22:123
15497217469,25sleep315501-21cron10:57:093
2039421730,28sleep120398-21fschecks_time09:27:281
8289216310,69sleep30-21swapper/310:32:313
7289216213,76sleep30-21swapper/308:47:133
31497215619,67sleep30-21swapper/310:02:353
31497215619,67sleep30-21swapper/310:02:353
2343921558,93sleep30-21swapper/309:37:283
2343921558,93sleep30-21swapper/309:37:283
138421526,128sleep30-21swapper/308:27:303
25851215012,124sleep325882-21proc_pri11:27:373
23124214946,36sleep323142-21latency_hist09:37:123
23124214946,36sleep323142-21latency_hist09:37:123
1846421448,68sleep30-21swapper/307:37:393
2844213910,116sleep32886-21fschecks_count11:57:293
4994213721,90sleep25126-21tail12:02:412
1883121369,68sleep30-21swapper/309:22:303
30530213510,103sleep130534-21kworker/u8:309:58:091
1465621356,11sleep30-21swapper/312:33:173
695213322,24sleep10-21swapper/110:07:381
9141213114,102sleep09151-21forks12:17:280
20837213025,83sleep220874-21fschecks_count11:12:582
2879321269,61sleep30-21swapper/311:37:553
8276212519,80sleep18425-21grep08:47:391
3255212510,99sleep33299-21ntpq08:32:353
7198212311,97sleep07201-21sshd08:46:410
18684211920,81sleep118810-21latency_hist11:07:121
889621179,87sleep30-21swapper/307:07:433
17680211716,81sleep117764-21grep09:18:121
8383211614,88sleep08399-21netstat10:32:320
4521211625,71sleep24560-21cut08:37:312
31764211615,84sleep231883-21timerwakeupswit10:02:422
31764211615,84sleep231883-21timerwakeupswit10:02:422
28605211311,84sleep328652-21latency_hist08:12:123
13066211213,83sleep213112-21latency_hist07:22:122
24686211134,13sleep30-21swapper/307:57:383
24686211134,13sleep30-21swapper/307:57:383
17661210921,71sleep217664-21cron07:37:102
3049210821,72sleep03073-21sendmail10:17:100
17120210820,68sleep216612-21apt-get09:17:202
15327210814,75sleep015394-21df09:12:260
753021065,11sleep30-21swapper/310:29:443
3210021066,13sleep30-21swapper/308:22:133
8324210524,53sleep08414-21fschecks_time07:07:320
6613210530,36sleep20-21swapper/207:02:222
3027210416,71sleep23066-21latency11:57:322
1489210416,74sleep01518-21awk10:12:090
12121210411,75sleep212131-21find12:27:192
12121210411,75sleep212131-21find12:27:192
16803210321,67sleep116844-21ntpq07:32:371
29522210217,68sleep129648-21unixbench_multi08:12:421
29522210217,68sleep129648-21unixbench_multi08:12:421
18072210225,54sleep218164-21grep07:37:302
1758021025,18sleep30-21swapper/307:36:103
9115210013,65sleep29131-21sshd12:17:262
94852997,74sleep00-21swapper/008:52:350
94852997,74sleep00-21swapper/008:52:350
29872996,64sleep30-21swapper/310:16:243
1806429912,65sleep117956-21/usr/sbin/munin07:37:291
622629816,62sleep16355-21latency_hist10:27:121
210742978,34sleep30-21swapper/307:47:253
1036229633,47sleep310459-21latency_hist12:22:113
2930829551,22sleep20-21swapper/208:12:392
2930829551,22sleep20-21swapper/208:12:392
1531529530,53sleep315316-21ntp_states07:27:383
141562959,56sleep00-21swapper/007:22:420
2205429433,45sleep30-21swapper/311:17:113
1006429417,52sleep30-21swapper/310:37:393
1006429417,52sleep30-21swapper/310:37:393
326882937,31sleep30-21swapper/310:07:243
2197829320,52sleep122015-21munin-run11:17:091
191312937,30sleep30-21swapper/311:07:303
171132937,30sleep30-21swapper/309:17:173
113542935,11sleep20-21swapper/210:42:312
71929216,60sleep2804-21grep10:07:392
23842928,74sleep313441-21snmpd11:56:283
321722918,20sleep30-21swapper/311:47:283
1800229156,19sleep30-21swapper/311:02:423
1060029120,48sleep210596-21apt-get12:22:222
45252908,63sleep00-21swapper/012:02:300
3064129015,58sleep030136-21apt-get08:17:220
2299729016,58sleep023038-21/usr/sbin/munin07:52:340
94932880,6sleep10-21swapper/110:37:121
251828715,55sleep22637-21latency_hist11:57:122
879328628,47sleep38832-21modprobe08:52:093
831928513,56sleep18332-21/usr/sbin/munin12:12:451
789828531,35sleep37951-21systemd-run07:07:113
2958928532,39sleep30-21swapper/309:57:263
1693528515,23sleep00-21swapper/011:00:170
260862845,68sleep30-21swapper/309:46:043
260862845,68sleep30-21swapper/309:46:043
491528316,39sleep24968-21cpuspeed10:22:242
44312837,18sleep34427-21apt-get08:37:243
2872128311,60sleep128742-21apt-get11:37:201
2861728220,27sleep10-21swapper/109:52:441
15732820,7sleep00-21swapper/008:27:340
430528123,44sleep30-21swapper/312:02:213
1581328117,48sleep115896-21mailstats09:12:391
1503528021,42sleep115084-21unin-run09:12:101
548827920,19sleep00-21swapper/010:22:390
329227913,44sleep12734-21/usr/sbin/munin11:57:391
2911527920,32sleep00-21swapper/008:12:330
2911527920,32sleep00-21swapper/008:12:330
1283927912,53sleep112964-21timerwakeupswit09:02:411
750127818,29sleep00-21swapper/012:12:310
2622427822,27sleep10-21swapper/109:47:111
2622427822,27sleep10-21swapper/109:47:111
72602770,14sleep17262-21basename08:47:101
2303927725,34sleep123074-21ntp_states07:52:361
1017727719,33sleep20-21swapper/207:12:272
3169427617,35sleep131761-21switchtime10:02:411
3169427617,35sleep131761-21switchtime10:02:411
2967327519,31sleep20-21swapper/209:57:302
2848827517,42sleep30-21swapper/309:52:423
225627512,46sleep22315-21/usr/sbin/munin10:12:402
3063727417,40sleep030646-21diskstats11:42:450
2287527419,33sleep00-21swapper/011:17:410
1031127423,32sleep110338-21munin-run12:22:091
458327325,22sleep10-21swapper/112:02:311
33612736,14sleep30-21swapper/310:17:213
965527211,13sleep10-21swapper/107:12:131
616827220,41sleep30-21swapper/310:27:063
377027227,26sleep00-21swapper/010:17:340
131372727,18sleep00-21swapper/009:02:530
1223027219,34sleep112235-21sshd12:27:261
1223027219,34sleep112235-21sshd12:27:261
1105727213,41sleep10-21swapper/108:57:331
2926527121,32sleep20-21swapper/209:57:092
2364627115,25sleep10-21swapper/109:37:341
2364627115,25sleep10-21swapper/109:37:341
1616427116,33sleep116173-21wc07:32:121
121262716,52sleep00-21swapper/010:43:380
93827015,38sleep2995-21sed11:52:112
3017627014,33sleep030380-21readlink11:42:120
1433127018,34sleep00-21swapper/010:52:300
1378227017,34sleep113806-21sshd12:32:261
1333127017,34sleep20-21swapper/210:47:432
1333127017,34sleep20-21swapper/210:47:432
1077227013,40sleep010841-21fschecks_time08:57:280
191032696,12sleep00-21swapper/009:22:350
1589726926,27sleep20-21swapper/209:12:402
1357626920,29sleep20-21swapper/207:22:322
321842686,16sleep00-21swapper/008:22:140
2777126810,43sleep127906-21latency_hist09:52:121
2777126810,43sleep127906-21latency_hist09:52:121
2305226820,31sleep123093-21munin-run09:37:101
2305226820,31sleep123093-21munin-run09:37:101
186826824,23sleep10-21swapper/110:12:301
2024526718,25sleep00-21swapper/011:12:120
110742678,15sleep10-21swapper/112:22:381
1025326714,37sleep010259-21diskstats07:12:290
672226626,23sleep10-21swapper/107:02:291
5862666,17sleep00-21swapper/011:48:180
1797226612,36sleep018094-21users09:18:150
1493626615,34sleep214940-21diskstats07:27:292
2787526425,17sleep00-21swapper/008:07:410
2775426413,35sleep127760-21ntp_states08:07:381
2588426413,29sleep00-21swapper/011:27:380
880426317,22sleep20-21swapper/207:07:422
667526318,20sleep00-21swapper/010:27:310
202212639,40sleep00-21swapper/009:27:200
1878126319,27sleep118785-21diskstats09:22:281
884826215,26sleep20-21swapper/208:52:132
397226216,25sleep10-21swapper/110:17:391
3066226213,20sleep10-21swapper/108:17:231
2551326214,25sleep10-21swapper/108:02:131
2551326214,25sleep10-21swapper/108:02:131
2346926222,17sleep20-21swapper/209:37:312
2346926222,17sleep20-21swapper/209:37:312
1969226213,28sleep219737-21/usr/sbin/munin07:42:312
1878626213,30sleep218830-21grep09:22:292
1517326215,29sleep215322-21apt-key12:37:122
1441126221,19sleep10-21swapper/110:52:311
1405026217,19sleep00-21swapper/009:07:330
968126113,21sleep10-21swapper/110:37:291
968126113,21sleep10-21swapper/110:37:291
797026119,19sleep20-21swapper/212:12:422
615026111,35sleep26210-21idleruntime08:42:312
613326123,18sleep20-21swapper/212:07:332
613326123,18sleep20-21swapper/212:07:332
2509826115,28sleep025124-21modprobe11:27:100
2462626117,22sleep00-21swapper/009:42:090
2462626117,22sleep00-21swapper/009:42:090
2289326112,32sleep222982-21/usr/sbin/munin07:52:322
1961326113,35sleep119691-21fschecks_time07:42:301
183052617,41sleep00-21swapper/011:02:590
1301826118,19sleep00-21swapper/007:22:100
1106626121,21sleep00-21swapper/012:22:370
2700826018,25sleep227017-21df_inode11:32:292
2528126013,26sleep00-21swapper/009:42:340
2528126013,26sleep00-21swapper/009:42:340
1841026010,15sleep00-21swapper/007:37:360
1579226022,19sleep20-21swapper/210:57:202
136932609,39sleep00-21swapper/010:48:500
136932609,39sleep00-21swapper/010:48:500
3115525914,27sleep231187-21/usr/sbin/munin08:17:562
2225725911,33sleep222278-21awk09:32:382
2225725911,33sleep222278-21awk09:32:382
2155925911,34sleep121627-21systemctl07:47:391
2107625921,16sleep00-21swapper/007:47:260
1327725919,19sleep10-21swapper/107:22:191
2627725818,23sleep226425-21modprobe09:47:122
2627725818,23sleep226425-21modprobe09:47:122
24282586,10sleep10-21swapper/111:57:021
193812585,13sleep00-21swapper/007:42:130
107325816,22sleep10-21swapper/108:27:121
957025718,21sleep20-21swapper/210:37:222
957025718,21sleep20-21swapper/210:37:222
712425714,21sleep20-21swapper/210:27:412
421525718,20sleep00-21swapper/008:37:120
3122125722,19sleep20-21swapper/211:42:592
285325717,24sleep22850-21apt-get08:32:222
149322576,16sleep00-21swapper/012:36:490
1478725710,16sleep20-21swapper/210:52:402
1230725713,26sleep212397-21fschecks_time09:02:302
1110325715,15sleep20-21swapper/208:57:342
1042225720,17sleep10-21swapper/108:57:121
1042225720,17sleep10-21swapper/108:57:121
717025614,19sleep00-21swapper/012:12:110
717025614,19sleep00-21swapper/012:12:110
672325615,23sleep16770-21users12:07:441
672325615,23sleep16770-21users12:07:441
3208625614,25sleep132083-21apt-get11:47:221
313925616,24sleep13223-21/usr/sbin/munin08:32:331
2663525619,16sleep00-21swapper/009:47:300
2663525619,16sleep00-21swapper/009:47:300
2419525615,25sleep024190-21apt-get07:57:230
2419525615,25sleep024190-21apt-get07:57:230
1961025613,20sleep10-21swapper/111:07:411
1324925612,24sleep10-21swapper/110:47:421
1324925612,24sleep10-21swapper/110:47:421
1232125615,18sleep10-21swapper/107:17:391
828025513,25sleep10-21swapper/107:07:291
451525513,25sleep14520-21diskstats08:37:301
2739825518,24sleep027404-21open_files11:32:390
2575225518,17sleep20-21swapper/208:02:262
2575225518,17sleep20-21swapper/208:02:262
2384425512,26sleep223840-21apt-get11:22:232
2209525523,15sleep00-21swapper/011:17:120
1638825514,24sleep016400-21apt-get07:32:230
1426225514,18sleep20-21swapper/209:07:392
336725412,25sleep23360-21apt-get10:17:242
322625422,18sleep00-21swapper/011:57:380
2739525410,28sleep227491-21grep08:07:302
2583525415,26sleep125863-21cut08:02:291
2583525415,26sleep125863-21cut08:02:291
1976425413,25sleep219822-21users11:07:422
117525414,23sleep01176-21apt-get11:52:210
2596625311,18sleep10-21swapper/111:27:391
2238225316,18sleep20-21swapper/207:52:122
1792825315,17sleep20-21swapper/211:02:412
1532325318,18sleep00-21swapper/007:27:390
917025218,16sleep20-21swapper/208:52:292
917025218,16sleep20-21swapper/208:52:292
241712528,16sleep20-21swapper/207:57:212
241712528,16sleep20-21swapper/207:57:212
2359325212,24sleep223602-21munin-run11:22:112
2584425110,16sleep20-21swapper/211:27:362
2221325110,15sleep00-21swapper/009:32:360
2221325110,15sleep00-21swapper/009:32:360
2031725115,21sleep220326-21df_inode09:27:262
1194625114,19sleep112044-21users10:42:421
1002425113,22sleep010050-21ntp_states10:37:360
1002425113,22sleep010050-21ntp_states10:37:360
927625013,21sleep29328-21munin-run10:37:102
761025016,18sleep00-21swapper/008:47:260
2958625012,22sleep029582-21apt-get09:57:230
2808925013,21sleep028095-21df_inode09:52:310
253452509,17sleep20-21swapper/209:42:362
253452509,17sleep20-21swapper/209:42:362
1624025014,20sleep116272-21ntp_states10:57:351
168224914,17sleep20-21swapper/208:27:372
269312489,16sleep10-21swapper/111:32:241
261824814,14sleep10-21swapper/108:32:131
232452485,33sleep10-21swapper/111:18:101
60524711,19sleep2666-21users08:22:452
2582824311,16sleep00-21swapper/008:02:280
2582824311,16sleep00-21swapper/008:02:280
450623910,12sleep00-21swapper/008:37:290
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional