You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-21 - 20:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Apr 21, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
68442115611,1129sleep06846-21diskstats07:06:140
4907280510,717sleep25008-21fschecks_time08:41:152
3711252812,500sleep33719-21ntp_states08:36:233
354824867,442sleep23586-21ntpq12:01:212
1534238538,331sleep01537-21/usr/sbin/munin08:31:150
22630237714,349sleep022679-21unin-run11:20:560
17203231616,82sleep217357-21unixbench_multi07:36:292
20032229420,256sleep020066-21/usr/sbin/munin07:46:200
2134227527,44sleep12153-21apt10:15:561
32517226876,104sleep232658-21grep10:06:282
31484226377,104sleep231514-21grep10:05:552
20476226027,215sleep320541-21uniq09:31:153
11936226057,189sleep211968-21fschecks_count07:21:152
5311225410,187sleep25398-21latency_hist10:25:582
14245225149,27sleep20-21swapper/209:11:152
22467224367,86sleep222497-21latency_hist07:55:552
9281223411,212sleep215670-21sshd12:20:102
13153223114,199sleep213308-21date10:50:582
12708222818,190sleep012805-21fschecks_time09:06:160
11531222522,181sleep20-21swapper/210:45:572
22922222420,190sleep322915-21apt-get11:21:113
2589122238,201sleep225927-21latency_hist08:05:572
2836221120,103sleep22723-21latency_hist08:35:582
680822067,22sleep00-21swapper/008:46:220
17418220584,23sleep20-21swapper/211:01:282
26112220212,106sleep226198-21fschecks_time11:31:162
2322021989,104sleep223223-21cron09:40:552
2322021989,104sleep223223-21cron09:40:552
1859221919,129sleep218609-21grep09:25:572
3154121909,67sleep031614-21cat11:47:090
53121838,164sleep013441-21snmpd10:10:250
16438218313,153sleep316449-21unin-run11:00:563
9938217946,120sleep21-21systemd10:40:562
24932217775,21sleep20-21swapper/211:26:232
542621729,150sleep25478-21processes12:06:382
9393217016,139sleep19507-21latency_hist12:20:581
8901215247,16sleep20-21swapper/207:11:172
31084215241,26sleep20-21swapper/208:21:202
31084215241,26sleep20-21swapper/208:21:202
12376214351,19sleep20-21swapper/209:05:582
30840214210,117sleep130954-21sed10:01:261
20324213936,17sleep20-21swapper/211:11:252
119921398,118sleep21211-21latency_hist08:30:572
13396213713,104sleep213450-21sort07:26:112
10808213737,85sleep20-21swapper/209:00:572
3761213620,90sleep20-21swapper/208:36:262
18552213656,64sleep20-21swapper/207:41:262
19549213322,93sleep319650-21latency_hist11:10:583
29698213229,91sleep229778-21switchtime11:41:262
20388213056,16sleep20-21swapper/209:31:102
31773212828,77sleep131767-21apt-get10:06:101
28213212813,25sleep20-21swapper/209:55:582
24770212318,88sleep124801-21munin-run09:45:551
27601212016,89sleep027668-21df11:36:110
16937211912,17sleep00-21swapper/007:36:230
1264211836,70sleep21304-21ntp_kernel_pll_10:11:202
16890211712,19sleep30-21swapper/307:36:223
1460521176,31sleep20-21swapper/210:53:122
13386211719,79sleep113508-21timerwakeupswit12:31:281
13386211719,79sleep113508-21timerwakeupswit12:31:281
8323211612,87sleep38372-21/usr/sbin/munin08:51:213
2482121167,96sleep00-21swapper/008:02:280
23192211210,22sleep20-21swapper/211:21:182
9607211126,34sleep30-21swapper/312:21:083
2723211118,13sleep20-21swapper/211:59:112
10573210919,69sleep20-21swapper/207:16:182
495321089,82sleep14925-21apt-get12:06:201
4372210811,84sleep14404-21awk10:21:211
2026721089,22sleep20-21swapper/207:46:272
615421078,81sleep20-21swapper/210:26:272
28900210715,77sleep328938-21latency_hist11:40:583
1572121068,22sleep20-21swapper/210:56:272
5260210536,48sleep15318-21cut07:01:151
1861621057,15sleep20-21swapper/211:06:202
2592821049,72sleep37018-1kworker/3:0H08:05:583
7045210321,60sleep20-21swapper/212:11:262
7920210215,16sleep20-21swapper/207:09:442
6893210218,69sleep17029-21latency_hist10:30:581
28657210220,64sleep028129-21apt-get09:56:080
10303210214,73sleep310305-21/usr/sbin/munin07:16:113
3754210010,24sleep20-21swapper/210:20:572
25776210024,57sleep025755-21latency_hist11:30:580
2337210018,60sleep02486-21grep11:56:260
2397429925,16sleep20-21swapper/209:41:242
2742329812,70sleep327467-21awk09:51:263
2786529725,15sleep20-21swapper/211:36:172
998829610,67sleep30-21swapper/310:40:573
704629613,67sleep17167-21timerwakeupswit08:46:271
1945229516,60sleep20-21swapper/209:26:262
658429437,29sleep36595-21awk12:11:153
2134729410,52sleep30-21swapper/311:16:073
2796929311,67sleep027970-21sshd09:54:100
1240829311,68sleep112486-21switchtime07:21:261
1352529210,63sleep20-21swapper/209:06:312
1142029120,58sleep011460-21grep09:01:200
473828921,43sleep04421-21apt-get12:05:590
3119028912,62sleep331231-21if_eth011:47:003
1777228915,26sleep00-21swapper/009:21:230
1601428914,59sleep016108-21memory09:16:180
1601428914,59sleep016108-21memory09:16:180
643228824,43sleep26440-21modprobe07:05:562
162272887,43sleep216304-21basename09:16:242
162272887,43sleep216304-21basename09:16:242
8302878,13sleep20-21swapper/211:51:352
80732878,66sleep28137-21basename08:51:162
1649528522,29sleep10-21swapper/109:16:271
1649528522,29sleep10-21swapper/109:16:271
440728418,37sleep14440-21munin-run12:05:561
2263028418,49sleep122717-21ls09:36:261
2263028418,49sleep122717-21ls09:36:261
1966128412,54sleep319669-21grep07:46:113
379228319,29sleep30-21swapper/310:20:583
1398328327,37sleep313994-21modprobe12:35:543
1398328327,37sleep313994-21modprobe12:35:543
1161128318,50sleep111669-21latency_hist07:20:571
262852815,65sleep20-21swapper/209:47:222
1303028112,41sleep00-21swapper/010:50:560
61222808,58sleep06185-21latency_hist08:45:570
234622808,33sleep20-21swapper/207:56:282
218282807,14sleep20-21swapper/211:16:222
1377528023,42sleep313816-21ntpq07:26:203
277622797,53sleep20-21swapper/208:11:142
119912796,15sleep20-21swapper/212:26:292
2894627821,35sleep328964-21apt08:15:553
1550527816,27sleep00-21swapper/010:56:210
36227710,53sleep3401-21ps08:26:263
308902777,13sleep20-21swapper/211:46:022
108942777,14sleep20-21swapper/212:25:562
3062527614,46sleep113441-21snmpd08:20:581
3062527614,46sleep113441-21snmpd08:20:581
286642767,57sleep228653-21apt-get09:56:102
2703027612,47sleep127039-21/usr/sbin/munin08:06:291
908627522,27sleep10-21swapper/110:36:231
2735527513,47sleep327465-21latency_hist11:35:573
1885727523,26sleep00-21swapper/011:06:260
70792747,11sleep20-21swapper/210:31:072
658027416,31sleep30-21swapper/308:46:163
3276227412,45sleep0336-21/usr/sbin/munin08:26:220
1489827412,17sleep10-21swapper/110:55:571
272722735,11sleep20-21swapper/208:09:242
2320227316,26sleep00-21swapper/007:56:220
2026427314,43sleep120323-21/usr/sbin/munin11:11:241
158652736,11sleep20-21swapper/207:31:362
644327229,30sleep36501-21systemd-run07:05:573
371127220,35sleep11-21systemd10:20:561
207927223,24sleep10-21swapper/111:56:231
1861127212,44sleep318750-21latency_hist09:25:583
917627122,35sleep19201-21munin-run08:55:561
719527111,42sleep07224-21fschecks_count10:31:170
67382716,29sleep20-21swapper/208:46:192
302482715,12sleep20-21swapper/208:17:242
2569027118,30sleep00-21swapper/009:46:390
1555327119,25sleep10-21swapper/109:15:591
1470527114,36sleep014814-21latency_hist07:30:580
852427018,33sleep00-21swapper/008:51:260
607027028,26sleep30-21swapper/308:45:563
490427020,31sleep34923-21apt12:06:063
372427025,23sleep30-21swapper/312:01:263
188362708,48sleep10-21swapper/109:26:081
139712705,11sleep20-21swapper/212:35:452
139712705,11sleep20-21swapper/212:35:452
86526916,24sleep00-21swapper/010:11:100
2486826921,28sleep30-21swapper/309:45:573
1261426912,39sleep112680-21/usr/sbin/munin10:46:281
115826913,41sleep11198-21sh08:30:561
76902685,11sleep00-21swapper/012:15:000
2507626815,37sleep125166-21timerandwakeup11:26:261
958126711,42sleep09471-21/usr/sbin/munin08:56:150
458926715,35sleep34659-21sed10:21:273
3092626714,17sleep10-21swapper/111:46:091
2286126716,29sleep30-21swapper/309:36:283
2286126716,29sleep30-21swapper/309:36:283
1921326718,33sleep019259-21ntpq09:26:190
1510826718,26sleep10-21swapper/110:56:061
335226620,30sleep03408-21tune2fs08:36:160
2190426620,21sleep30-21swapper/307:51:283
2092326619,24sleep10-21swapper/109:31:251
2049226627,20sleep00-21swapper/011:11:270
2005926616,24sleep30-21swapper/311:11:183
1920126622,22sleep30-21swapper/309:26:183
1455526613,22sleep00-21swapper/009:11:210
3226026527,21sleep30-21swapper/310:06:253
3061226520,27sleep330816-21gpgv11:45:593
2983926515,24sleep00-21swapper/008:16:250
1815226518,17sleep00-21swapper/007:41:160
1679126518,26sleep10-21swapper/107:36:201
1288426511,35sleep312978-21memory09:06:203
1233626519,30sleep012407-21tr07:21:250
998626418,30sleep010021-21munin-run07:15:560
113526417,30sleep31166-21/usr/sbin/munin10:11:173
923926316,28sleep39317-21mailstats07:11:263
397526320,26sleep03994-21apt-get10:21:070
3156626314,26sleep30-21swapper/308:21:293
3156626314,26sleep30-21swapper/308:21:293
278162636,13sleep10-21swapper/111:36:161
275826316,28sleep12801-21munin-run08:35:571
1400426315,33sleep014062-21munin-run12:35:550
1400426315,33sleep014062-21munin-run12:35:550
3171026214,32sleep031721-21/usr/sbin/munin08:21:300
3171026214,32sleep031721-21/usr/sbin/munin08:21:300
2992426219,22sleep10-21swapper/108:16:261
2498726215,20sleep00-21swapper/009:45:580
2268026215,24sleep10-21swapper/111:20:571
2254826214,24sleep00-21swapper/007:55:570
1482026225,21sleep00-21swapper/010:55:560
1462026222,21sleep30-21swapper/309:11:243
1011526219,28sleep10-21swapper/112:21:271
61802615,45sleep00-21swapper/007:02:430
311782616,12sleep00-21swapper/010:02:010
298052617,11sleep10-21swapper/109:59:161
270752616,15sleep00-21swapper/011:33:200
2204426117,20sleep00-21swapper/009:36:130
2204426117,20sleep00-21swapper/009:36:130
1630526115,27sleep30-21swapper/309:16:253
1630526115,27sleep30-21swapper/309:16:253
1562526112,31sleep115761-21awk07:31:271
1155126125,19sleep30-21swapper/310:45:583
113902616,44sleep00-21swapper/007:18:460
768526018,28sleep37780-21switchtime10:31:283
3052226021,21sleep30-21swapper/308:20:563
2807326012,27sleep128111-21/usr/sbin/munin08:11:211
2409526016,26sleep324180-21gpgv08:00:573
2373126020,19sleep10-21swapper/107:56:551
1898426019,23sleep10-21swapper/111:06:271
1502626014,20sleep00-21swapper/007:31:140
1117926014,28sleep111180-21sshd12:26:111
111772609,34sleep00-21swapper/012:26:100
975325912,34sleep39766-21irqstats08:56:183
480825910,25sleep10-21swapper/108:41:101
3043225917,18sleep30-21swapper/310:01:163
2813225918,26sleep128212-21apt-config09:55:571
1627925915,28sleep30-21swapper/307:35:583
1248825916,20sleep30-21swapper/312:30:583
2727525814,25sleep127339-21sendmail-msp11:35:551
2594325818,22sleep30-21swapper/309:46:463
1806325818,23sleep318064-21sshd07:41:103
1802625818,18sleep00-21swapper/011:05:580
1756625820,18sleep30-21swapper/311:01:293
86922579,18sleep10-21swapper/107:11:081
676025714,22sleep10-21swapper/107:06:111
2962625712,26sleep129696-21mailstats11:41:251
2864125716,25sleep328653-21apt-get09:56:073
2687725716,17sleep00-21swapper/008:06:280
2436325719,22sleep124426-21apt-key08:00:591
2042025716,18sleep00-21swapper/009:31:120
135725715,25sleep01496-21sed11:55:570
1356225713,26sleep313558-21apt-get10:51:113
1177525714,26sleep311770-21apt-get10:46:103
1148725710,26sleep10-21swapper/110:45:561
1075925716,25sleep110806-21systemd-run09:00:561
844725615,27sleep38456-21needreboot12:16:203
67452566,38sleep00-21swapper/010:30:040
2640825615,24sleep326500-21memory08:06:183
2422225615,23sleep00-21swapper/011:25:570
1669125616,23sleep116685-21apt-get11:01:101
1197125621,16sleep30-21swapper/307:21:163
686925513,26sleep16913-21ntpq12:11:211
1104825514,25sleep111043-21apt-get09:01:091
91122545,37sleep00-21swapper/012:17:580
2424925416,15sleep10-21swapper/111:25:581
1852825413,18sleep10-21swapper/107:41:251
1280725411,19sleep10-21swapper/109:06:171
322925213,22sleep12725-21apt-get08:36:051
2527125214,24sleep125290-21apt-get09:46:071
1054925211,25sleep010595-21memory10:41:190
962925112,22sleep09625-21apt-get12:21:090
360625112,16sleep00-21swapper/012:01:230
2528525112,22sleep325371-21users11:26:283
1965925115,21sleep119654-21apt-get07:46:091
70425012,16sleep10-21swapper/111:51:341
675725012,21sleep36753-21apt-get07:06:093
144725011,21sleep31443-21apt-get08:31:103
1113725013,18sleep30-21swapper/309:01:153
915224911,19sleep00-21swapper/010:36:260
134424913,23sleep3281-21dbus-daemon11:55:563
290924813,16sleep30-21swapper/312:00:573
276732489,14sleep00-21swapper/008:11:080
2950824714,15sleep00-21swapper/011:41:200
1339524711,20sleep113389-21apt-get07:26:101
554924511,19sleep35542-21apt-get10:26:113
24262457,13sleep00-21swapper/010:16:070
175272459,16sleep10-21swapper/109:21:161
711224411,19sleep17179-21df10:31:141
216272427,15sleep10-21swapper/107:51:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional