You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-18 - 10:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sat Apr 18, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
278472546915,5441sleep216217-21kworker/2:120:44:292
274662130321,1259sleep01770-21kworker/u8:222:26:260
274662130321,1259sleep01770-21kworker/u8:222:26:260
319932121718,1184sleep031438-21kworker/u8:219:16:250
10490271021,670sleep310532-21/usr/sbin/munin21:31:383
25174268617,653sleep225175-21sshd22:18:282
27071259748,526sleep127123-21apt-key22:26:101
931924009,377sleep29323-21ntpq19:46:352
23119226061,172sleep123082-40mandb23:56:101
23119226061,172sleep123082-40mandb23:56:101
28030224866,113sleep10-21swapper/120:46:101
8051224320,207sleep38190-21awk19:41:403
31531223418,192sleep131654-21cat19:16:101
16393223215,193sleep116408-21fschecks_count21:51:261
16393223215,193sleep116408-21fschecks_count21:51:261
31807222814,198sleep331855-21ntp_states20:56:333
2486922200,6sleep10-21swapper/100:01:101
21564221513,182sleep021632-21latency_hist20:26:100
25431220912,171sleep125524-21latency_hist22:21:101
821422060,15sleep127971-21cyclictest19:41:411
1357220276,89sleep11437-21cat19:21:311
21801220128,100sleep10-21swapper/120:26:231
17230219919,116sleep117266-21fschecks_count23:36:361
17230219919,116sleep117266-21fschecks_count23:36:361
7019219717,85sleep17055-21latency21:21:311
7019219717,85sleep17055-21latency21:21:311
19392218813,160sleep019482-21sed20:17:100
9661218717,24sleep10-21swapper/123:11:531
9661218717,24sleep10-21swapper/123:11:531
28731218049,33sleep128741-21modprobe22:31:071
28731218049,33sleep128741-21modprobe22:31:071
7669217927,28sleep10-21swapper/123:06:281
7669217927,28sleep10-21swapper/123:06:281
3057217797,53sleep10-21swapper/100:36:071
21487217412,121sleep121565-21sendmail_mailtr22:06:371
1371421746,131sleep113759-21latency_hist23:26:101
29507217097,21sleep10-21swapper/119:11:091
29507217097,21sleep10-21swapper/119:11:091
19151216922,106sleep10-21swapper/122:01:071
2358521689,122sleep123077-21apt-get20:31:241
15557216848,47sleep10-21swapper/120:06:371
25535216621,56sleep10-21swapper/120:36:331
3984216514,29sleep20-21swapper/221:11:342
546321638,132sleep10-21swapper/121:16:301
546321638,132sleep10-21swapper/121:16:301
19483215229,75sleep119620-21head20:17:111
999214927,71sleep11041-21date22:46:091
999214927,71sleep11041-21date22:46:091
20253214333,51sleep10-21swapper/120:21:231
9248213328,87sleep38747-21apt-get23:11:183
9248213328,87sleep38747-21apt-get23:11:183
3003213015,62sleep10-21swapper/119:26:321
14144212916,93sleep314227-21grep20:01:383
3208521287,63sleep10-21swapper/120:56:391
23040212817,64sleep10-21swapper/122:11:371
23040212817,64sleep10-21swapper/122:11:371
1496921289,106sleep115021-21if_eth021:46:301
1496921289,106sleep115021-21if_eth021:46:301
24492212718,93sleep324537-21awk22:16:323
11907212441,71sleep111930-21sshd21:36:291
31758212316,39sleep10-21swapper/122:39:511
4185212116,85sleep10-21swapper/119:31:241
31431212022,74sleep331455-21munin-run19:16:083
29586211912,93sleep129710-21latency_hist20:51:101
2640321185,102sleep00-21swapper/020:40:480
7393211715,81sleep27440-21fschecks_count19:41:262
30999211615,41sleep10-21swapper/120:55:021
18007211513,83sleep318088-21fw_packets21:56:253
9554211410,66sleep19587-21awk21:31:081
27328211432,52sleep227503-21grep19:01:382
9826211316,78sleep29897-21/usr/sbin/munin23:11:582
9826211316,78sleep29897-21/usr/sbin/munin23:11:582
32007211311,84sleep231979-21apt-get19:16:352
28805211216,78sleep228927-21/usr/sbin/munin00:11:422
1695821129,58sleep10-21swapper/123:36:121
1695821129,58sleep10-21swapper/123:36:121
26591211032,47sleep326610-21apt-get19:01:193
24109211021,25sleep20-21swapper/220:32:182
16919211031,14sleep10-21swapper/120:11:541
16919211031,14sleep10-21swapper/120:11:541
2689721097,90sleep113441-21snmpd00:06:321
2689721097,90sleep113441-21snmpd00:06:321
2034821088,80sleep10-21swapper/123:46:291
2034821088,80sleep10-21swapper/123:46:291
6010210629,23sleep10-21swapper/123:01:271
6010210629,23sleep10-21swapper/123:01:271
17679210512,71sleep00-21swapper/020:12:170
17679210512,71sleep00-21swapper/020:12:170
1423210511,42sleep10-21swapper/122:46:291
8185210322,58sleep00-21swapper/023:06:390
8185210322,58sleep00-21swapper/023:06:390
1407921039,35sleep10-21swapper/123:26:311
26460210212,71sleep326510-21/usr/sbin/munin00:06:223
26460210212,71sleep326510-21/usr/sbin/munin00:06:223
5311210140,49sleep15314-21modprobe22:57:071
30776210042,48sleep113441-21snmpd00:19:561
42872988,39sleep10-21swapper/121:11:411
121602987,36sleep10-21swapper/123:21:101
2062729613,68sleep220669-21ntp_offset23:46:342
2062729613,68sleep220669-21ntp_offset23:46:342
1452129618,19sleep20-21swapper/221:46:112
1452129618,19sleep20-21swapper/221:46:112
57929510,20sleep30-21swapper/321:01:263
283002957,35sleep10-21swapper/100:11:321
150929514,68sleep21550-21systemd-run00:31:082
320132947,34sleep10-21swapper/119:16:381
229812949,71sleep222991-21ntpq22:11:342
229812949,71sleep222991-21ntpq22:11:342
1856329423,52sleep218725-21grep21:56:362
1748229413,66sleep217555-21switchtime20:12:152
1748229413,66sleep217555-21switchtime20:12:152
1176129419,55sleep311846-21fschecks_time21:36:273
6702939,32sleep10-21swapper/100:26:341
2234929315,64sleep322435-21switchtime23:51:403
2234929315,64sleep322435-21switchtime23:51:403
2039429313,61sleep220226-21/usr/sbin/munin20:21:282
3009429210,66sleep030189-21cut20:51:300
1320829211,17sleep30-21swapper/321:41:233
1320829211,17sleep30-21swapper/321:41:233
1048229212,33sleep10-21swapper/119:51:271
651029118,57sleep06586-21sed21:21:090
651029118,57sleep06586-21sed21:21:090
1867529115,18sleep30-21swapper/323:41:233
1867529115,18sleep30-21swapper/323:41:233
1324829126,19sleep10-21swapper/120:01:081
1324829126,19sleep10-21swapper/120:01:081
3124229016,61sleep331326-21fschecks_time00:21:283
674828918,28sleep20-21swapper/221:21:242
674828918,28sleep20-21swapper/221:21:242
61342899,67sleep30-21swapper/323:01:293
61342899,67sleep30-21swapper/323:01:293
279228912,56sleep02764-21/usr/sbin/munin22:51:250
2518028913,20sleep30-21swapper/300:01:293
1667728914,60sleep216700-21netstat21:51:322
1667728914,60sleep216700-21netstat21:51:322
132062899,32sleep10-21swapper/121:41:211
132062899,32sleep10-21swapper/121:41:211
1130828818,55sleep011318-21ntpq23:16:410
445428725,41sleep30-21swapper/319:31:303
3108628724,31sleep30-21swapper/320:56:093
2645728728,33sleep00-21swapper/000:06:190
2645728728,33sleep00-21swapper/000:06:190
2466628711,60sleep324705-21cat00:01:093
1994628413,51sleep220027-21mailstats22:01:372
1297128415,43sleep213102-21latency_hist21:41:102
2422328315,52sleep024314-21sed20:32:200
1908428310,61sleep019132-21cat23:41:360
1908428310,61sleep019132-21cat23:41:360
16682837,62sleep10-21swapper/121:04:551
894128213,54sleep39130-21apt-key23:11:103
894128213,54sleep39130-21apt-key23:11:103
3020928224,26sleep20-21swapper/220:51:322
2928728223,43sleep329326-21grep22:31:303
2701928126,31sleep00-21swapper/019:01:320
1767028124,24sleep00-21swapper/021:56:100
1767028124,24sleep00-21swapper/021:56:100
100532818,58sleep110074-21apt-get21:31:201
919328015,46sleep28272-21/usr/sbin/munin21:26:402
613228027,17sleep16168-21ntpq19:36:331
460828016,47sleep00-21swapper/022:56:300
284628011,52sleep02847-21sshd00:33:280
239728011,56sleep02515-21date19:26:100
2029728011,51sleep00-21swapper/023:46:240
2029728011,51sleep00-21swapper/023:46:240
3138727910,48sleep10-21swapper/100:21:301
2791627921,41sleep327965-21ntpq22:27:263
2791627921,41sleep327965-21ntpq22:27:263
2788027927,33sleep127927-21awk19:06:091
2697927917,40sleep027069-21latency_hist22:26:090
1760827913,50sleep317646-21processes23:36:463
1760827913,50sleep317646-21processes23:36:463
1488427920,29sleep00-21swapper/020:06:110
1092627921,41sleep210957-21grep21:31:452
287527816,48sleep32940-21sort22:51:283
2777527826,29sleep20-21swapper/200:11:092
2777527826,29sleep20-21swapper/200:11:092
259927819,43sleep10-21swapper/121:06:411
153342785,16sleep20-21swapper/221:46:392
153342785,16sleep20-21swapper/221:46:392
976527711,50sleep09920-21latency_hist21:31:110
2916027729,11sleep10-21swapper/122:31:271
1656527716,42sleep316361-21apt-get20:11:113
1261027719,32sleep212372-21/usr/sbin/munin23:21:312
62227620,31sleep00-21swapper/000:26:330
588227622,30sleep20-21swapper/221:16:392
588227622,30sleep20-21swapper/221:16:392
2970327627,25sleep30-21swapper/300:16:273
2206527624,28sleep00-21swapper/020:26:300
2067027616,46sleep320681-21sh23:46:353
2067027616,46sleep320681-21sh23:46:353
1841227615,25sleep00-21swapper/021:56:340
912627510,47sleep10-21swapper/119:46:301
2692327512,28sleep00-21swapper/020:41:280
1251827515,47sleep312519-21sshd23:21:283
3086927415,44sleep330919-21sed22:36:303
2829227425,30sleep328306-21grep19:06:273
2829227425,30sleep328306-21grep19:06:273
2227127416,41sleep222312-21processes20:26:372
1701427422,31sleep30-21swapper/320:12:043
1701427422,31sleep30-21swapper/320:12:043
1587727412,44sleep215911-21/usr/sbin/munin23:31:372
1587727412,44sleep215911-21/usr/sbin/munin23:31:372
135092746,57sleep00-21swapper/023:24:120
369327329,11sleep10-21swapper/122:51:421
289942737,15sleep00-21swapper/019:06:400
289942737,15sleep00-21swapper/019:06:400
2613227316,40sleep226152-21/usr/sbin/munin22:21:352
2041527328,11sleep10-21swapper/122:02:291
196952736,56sleep10-21swapper/123:43:021
196952736,56sleep10-21swapper/123:43:021
1484427315,38sleep314862-21fschecks_count21:46:283
1484427315,38sleep314862-21fschecks_count21:46:283
58527222,24sleep00-21swapper/021:01:270
404627217,36sleep24066-21apt22:56:072
2601127230,10sleep10-21swapper/100:04:061
2229727222,29sleep30-21swapper/322:11:113
1914027226,25sleep20-21swapper/223:41:382
1914027226,25sleep20-21swapper/223:41:382
143892729,17sleep10-21swapper/120:01:401
143827218,34sleep21061-21/usr/sbin/munin19:21:322
1049027215,15sleep20-21swapper/219:51:282
644627114,31sleep06610-21unixbench_multi19:36:400
2002927115,36sleep319415-21/usr/sbin/munin22:01:383
2537427025,27sleep20-21swapper/220:36:302
262502699,38sleep00-21swapper/000:06:100
2284126910,49sleep313441-21snmpd20:28:193
51652686,15sleep00-21swapper/019:32:320
3198226717,25sleep30-21swapper/319:16:203
2910526713,16sleep00-21swapper/022:31:260
2135526725,21sleep00-21swapper/022:06:320
1450326613,37sleep014512-21systemd-run21:46:090
1450326613,37sleep014512-21systemd-run21:46:090
124126618,28sleep00-21swapper/022:46:240
882326522,27sleep28810-21/usr/sbin/munin23:11:092
882326522,27sleep28810-21/usr/sbin/munin23:11:092
2397926517,25sleep20-21swapper/223:56:402
1952526511,30sleep00-21swapper/022:01:270
185472655,13sleep30-21swapper/320:16:123
780826420,23sleep00-21swapper/019:41:370
728326415,31sleep07355-21/usr/sbin/munin21:21:380
728326415,31sleep07355-21/usr/sbin/munin21:21:380
3153026417,20sleep00-21swapper/000:21:330
2980526415,24sleep00-21swapper/000:16:280
277292646,11sleep30-21swapper/320:42:573
2581226410,21sleep20-21swapper/200:01:402
3095026317,24sleep20-21swapper/200:21:102
299962635,13sleep20-21swapper/222:32:292
2471226319,23sleep00-21swapper/000:01:100
2358926310,27sleep30-21swapper/320:31:263
120552638,14sleep00-21swapper/021:36:320
61702626,12sleep20-21swapper/219:36:342
473326213,27sleep20-21swapper/222:56:322
3057326219,26sleep230621-21ntp_states19:11:352
2320426219,16sleep30-21swapper/322:11:393
2320426219,16sleep30-21swapper/322:11:393
2204926215,28sleep022143-21memory23:51:330
2204926215,28sleep022143-21memory23:51:330
1855126215,22sleep00-21swapper/023:41:110
1855126215,22sleep00-21swapper/023:41:110
95126115,24sleep00-21swapper/022:46:080
95126115,24sleep00-21swapper/022:46:080
936526118,27sleep00-21swapper/019:46:370
321026118,22sleep30-21swapper/319:26:383
3176626114,27sleep20-21swapper/200:21:392
1760126118,21sleep20-21swapper/223:36:452
1760126118,21sleep20-21swapper/223:36:452
1591226120,21sleep30-21swapper/323:31:383
1591226120,21sleep30-21swapper/323:31:383
101052615,20sleep30-21swapper/319:51:003
2834826022,20sleep20-21swapper/220:46:232
279022605,12sleep20-21swapper/222:27:252
279022605,12sleep20-21swapper/222:27:252
2124226018,26sleep221258-21irqstats22:06:302
2076126015,22sleep30-21swapper/320:21:373
156826013,27sleep31630-21/usr/sbin/munin19:21:373
1416626010,19sleep20-21swapper/223:26:322
419325912,31sleep04286-21cat21:11:400
3177225913,28sleep231806-21/usr/sbin/munin20:56:322
290825915,17sleep00-21swapper/019:26:310
234025930,15sleep20-21swapper/221:06:342
1562625913,20sleep00-21swapper/023:31:300
1562625913,20sleep00-21swapper/023:31:300
1329925914,22sleep30-21swapper/320:01:103
1329925914,22sleep30-21swapper/320:01:103
1225425911,31sleep012337-21cat19:56:300
1225425911,31sleep012337-21cat19:56:300
3146025819,20sleep00-21swapper/019:16:090
300525811,35sleep33007-21sshd21:07:033
174252585,10sleep30-21swapper/321:54:193
174252585,10sleep30-21swapper/321:54:193
1085625817,25sleep210923-21df23:16:262
1053225816,21sleep00-21swapper/023:16:080
1053225816,21sleep00-21swapper/023:16:080
160302576,10sleep20-21swapper/220:07:022
159722576,10sleep00-21swapper/021:50:330
159722576,10sleep00-21swapper/021:50:330
14172576,11sleep30-21swapper/300:29:563
436125615,25sleep34358-21apt-get22:56:203
3222925615,18sleep20-21swapper/222:41:262
3222925615,18sleep20-21swapper/222:41:262
3067225610,11sleep20-21swapper/222:36:252
1867225612,27sleep218164-21apt-get20:16:192
1204525612,20sleep20-21swapper/219:56:242
1204525612,20sleep20-21swapper/219:56:242
1005225616,26sleep00-21swapper/021:31:190
325902555,37sleep00-21swapper/021:00:060
313725513,28sleep33168-21latency_hist00:36:093
261532556,14sleep30-21swapper/322:21:363
19472556,11sleep00-21swapper/021:06:110
1422325521,21sleep30-21swapper/323:26:333
1121025515,17sleep30-21swapper/319:51:413
723825410,29sleep07381-21cat23:06:110
723825410,29sleep07381-21cat23:06:110
282625418,21sleep22883-21if_eth019:26:282
226225418,20sleep22302-21ps00:31:362
1060425421,19sleep00-21swapper/019:51:300
601825314,18sleep30-21swapper/321:16:403
601825314,18sleep30-21swapper/321:16:403
2953425316,19sleep20-21swapper/219:11:102
2953425316,19sleep20-21swapper/219:11:102
217712539,19sleep20-21swapper/223:51:232
217712539,19sleep20-21swapper/223:51:232
1517425316,16sleep00-21swapper/020:06:280
51622526,34sleep20-21swapper/219:32:302
3214425214,22sleep332212-21df22:41:233
3214425214,22sleep332212-21df22:41:233
2253025212,23sleep022526-21apt-get22:11:200
2253025212,23sleep022526-21apt-get22:11:200
1441625213,17sleep00-21swapper/023:26:380
855225113,22sleep38568-21irqstats21:26:293
638925115,19sleep20-21swapper/223:01:362
638925115,19sleep20-21swapper/223:01:362
2957625113,21sleep029585-21munin-run20:51:090
2851325016,21sleep328599-21mailstats00:11:383
1092425014,20sleep310933-21df_inode23:16:273
804924914,16sleep30-21swapper/321:26:093
804924914,16sleep30-21swapper/321:26:093
49424912,21sleep2490-21apt-get21:01:212
3092224912,21sleep030968-21memory22:36:310
2559924912,22sleep325623-21ntp_states20:36:353
2810324811,19sleep00-21swapper/000:11:270
636724710,21sleep30-21swapper/319:36:393
342724711,18sleep30-21swapper/321:11:113
131424711,21sleep21323-21diskstats22:46:262
2542024611,18sleep00-21swapper/022:21:090
2872924510,20sleep328735-21ntp_states20:46:323
185724515,15sleep30-21swapper/322:46:393
1395624311,16sleep20-21swapper/220:01:332
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional