You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 08:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed May 06, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22892401816,3989sleep223968-21kworker/2:022:11:322
1594329248,900sleep015987-21ntpq22:55:200
1392129096,891sleep013961-21ntpq21:05:230
1013285119,817sleep11071-21/usr/sbin/munin20:25:251
938028347,791sleep09413-21ntpq19:10:220
9304256826,506sleep09310-21ntpq20:50:240
9304256826,506sleep09310-21ntpq20:50:240
10391252413,495sleep210384-21apt-get20:55:112
23076247926,418sleep023113-21unin-run23:19:560
613724478,400sleep06177-21ntpq20:40:230
13797243826,397sleep013944-21cat22:49:590
3240423377,296sleep0363-21snmpd20:20:360
28394231617,284sleep228438-21ntpq23:35:212
335230211,275sleep1417-21find22:05:251
20307229349,195sleep020345-21fschecks_count23:10:150
1605722776,259sleep0363-21snmpd21:11:200
24515226646,161sleep024530-21fschecks_count21:40:160
24515226646,161sleep024530-21fschecks_count21:40:160
6788226518,228sleep26873-21grep00:10:152
2867522655,33sleep00-21swapper/023:35:270
2686224854,151sleep02824-21unixbench_multi23:55:270
12652222941,66sleep012680-21munin-run00:29:560
12652222941,66sleep012680-21munin-run00:29:560
5146222323,114sleep05297-21unixbench_singl22:20:270
22029221612,115sleep022161-21grep21:30:270
22029221612,115sleep022161-21grep21:30:270
12928220518,166sleep313045-21cat00:29:593
12928220518,166sleep313045-21cat00:29:593
1260522059,150sleep012661-21fschecks_count19:20:150
29684219754,124sleep029801-21timerandwakeup21:55:270
3224219144,92sleep03265-21ntpq22:15:220
296622189130,42sleep00-21swapper/020:14:580
5045218519,150sleep35142-21timerandwakeup22:20:263
23949218216,125sleep024029-21xargs23:20:250
4668217720,146sleep00-21swapper/000:02:250
22125217346,113sleep0929-1kworker/0:2H19:50:110
28113217232,126sleep028250-21date20:09:580
28067217019,58sleep00-21swapper/021:50:260
26564216917,64sleep00-21swapper/020:04:570
26564216917,64sleep00-21swapper/020:04:570
25199216524,122sleep125302-21unixbench_singl21:40:291
25199216524,122sleep125302-21unixbench_singl21:40:291
1587521618,17sleep00-21swapper/019:30:250
1571216113,126sleep01703-21head20:25:360
12254216019,84sleep00-21swapper/022:44:580
1234221548,19sleep00-21swapper/019:19:590
3144921537,134sleep031472-21cut23:45:170
21210215156,55sleep021336-21timerwakeupswit19:45:270
30732214940,87sleep00-21swapper/022:00:150
103521498,102sleep01118-21cat23:50:260
6962214713,44sleep26963-21sshd20:44:332
14162214531,100sleep014271-21fschecks_time19:25:160
17501214328,71sleep017587-21fschecks_time19:35:180
1934521399,94sleep019390-21awk19:40:180
24132213616,79sleep00-21swapper/019:55:240
16285213616,104sleep316429-21unixbench_singl22:55:283
8341212821,78sleep28430-21fschecks_time00:15:162
8341212821,78sleep28430-21fschecks_time00:15:162
1847221285,11sleep10-21swapper/121:20:181
15216212828,61sleep015294-21mailstats00:35:250
15216212828,61sleep015294-21mailstats00:35:250
3062921270,6sleep10-21swapper/122:00:071
888421235,86sleep00-21swapper/022:32:590
1520921205,11sleep00-21swapper/022:54:190
7388211823,66sleep27450-21apt-key22:29:582
1761421185,101sleep00-21swapper/021:16:290
11727211847,54sleep00-21swapper/020:59:570
32107211713,87sleep132245-21awk23:45:301
9127211615,83sleep19266-21latency_hist22:34:581
32656211414,86sleep232664-21open_files22:05:222
2583521145,74sleep00-21swapper/023:25:460
15078211317,84sleep315105-21grep21:10:103
29777211224,72sleep129779-21munin-node23:40:051
16711211221,70sleep316745-21fschecks_count21:15:153
5237211118,48sleep05320-21fschecks_time00:05:160
29804211116,79sleep129937-21sed21:55:281
26757211015,18sleep30-21swapper/321:45:283
19044211020,46sleep00-21swapper/023:05:220
19044211020,46sleep00-21swapper/023:05:220
9326210921,46sleep30-21swapper/322:35:123
23035210716,75sleep323037-21cron23:19:553
2647121069,57sleep026546-21sendmail_mailtr21:45:250
5612210516,65sleep15644-21proc_pri00:05:251
4367210533,30sleep04402-21idleruntime20:35:380
4367210533,30sleep04402-21idleruntime20:35:380
11397210421,65sleep211404-21/usr/sbin/munin19:15:252
7651210212,76sleep1363-21snmpd20:45:201
7651210212,76sleep1363-21snmpd20:45:201
14147210212,74sleep214265-21timerwakeupswit21:05:292
13144210121,64sleep212660-21apt-get00:30:062
12888210116,64sleep212495-21/usr/sbin/munin19:20:212
4313210019,62sleep34365-21cut20:35:373
4313210019,62sleep34365-21cut20:35:373
31605210011,20sleep131691-21fw_packets20:20:161
3026721007,56sleep030308-21ntpq23:41:210
1448221008,39sleep00-21swapper/000:34:570
10309210022,38sleep00-21swapper/000:20:270
80072989,53sleep08081-21latency_hist00:14:570
1843729823,61sleep118408-21latency_hist23:04:591
1737729816,68sleep217378-21/usr/sbin/munin19:35:072
2651229716,62sleep226514-21cron20:04:552
2651229716,62sleep226514-21cron20:04:552
2644729729,43sleep226443-21apt-get23:30:082
2390629713,64sleep223940-21/usr/sbin/munin19:55:182
2387129719,44sleep00-21swapper/021:36:490
1984729618,58sleep119862-21cut21:25:191
1065829610,18sleep00-21swapper/022:39:580
789829539,41sleep00-21swapper/020:45:270
789829539,41sleep00-21swapper/020:45:270
2024729518,62sleep320309-21mailstats21:25:293
1772729417,55sleep317828-21/usr/sbin/munin23:00:283
1711529428,41sleep30-21swapper/319:34:563
886529320,55sleep18969-21wc00:15:271
886529320,55sleep18969-21wc00:15:271
859029311,62sleep28617-21latency_hist20:49:582
859029311,62sleep28617-21latency_hist20:49:582
25752938,74sleep02627-21cat20:30:170
2306129322,54sleep323111-21if_eth021:35:183
200102938,50sleep00-21swapper/021:25:220
3113229210,48sleep00-21swapper/020:19:310
3113229210,48sleep00-21swapper/020:19:310
3091329214,65sleep330926-21cut22:00:183
2712129115,57sleep30-21swapper/323:30:253
2268629020,20sleep30-21swapper/319:50:273
52242899,66sleep25231-21df_inode00:05:142
1766328937,33sleep20-21swapper/223:00:272
911328822,34sleep30-21swapper/320:50:193
911328822,34sleep30-21swapper/320:50:193
2707328824,49sleep227104-21latency20:05:202
2498528829,39sleep10-21swapper/123:25:141
1958028720,51sleep219666-21timerandwakeup19:40:242
2002828627,26sleep30-21swapper/323:09:583
1477928621,44sleep114806-21munin-run21:09:561
1332028624,40sleep00-21swapper/021:04:580
65928526,25sleep0363-21snmpd22:05:330
184262856,42sleep018436-21munin-run23:04:580
273852845,39sleep00-21swapper/023:30:470
1895728415,51sleep319042-21/usr/sbin/munin23:05:213
184192847,40sleep018471-21cat21:20:170
353128220,40sleep33693-21unixbench_multi22:15:293
1716628217,51sleep117140-21latency_hist19:34:581
861928121,36sleep38768-21ls20:49:593
861928121,36sleep38768-21ls20:49:593
699428113,50sleep17041-21systemctl20:44:561
638128114,53sleep36382-21sshd00:09:353
345128122,36sleep20-21swapper/222:15:282
2183528119,43sleep21-21systemd19:49:572
1529628118,45sleep11-21systemd22:54:571
2526728014,47sleep225295-21/usr/sbin/munin23:25:192
2694927916,24sleep10-21swapper/123:30:231
14932798,58sleep01549-21if_eth022:10:170
1034227813,46sleep210520-21latency_hist19:15:002
427027612,50sleep24309-21grep20:35:362
427027612,50sleep24309-21grep20:35:362
2605827627,23sleep20-21swapper/221:45:152
2012927612,30sleep20-21swapper/221:25:242
1452327620,23sleep10-21swapper/100:34:581
2249127522,27sleep10-21swapper/119:50:201
1531327522,37sleep115347-21irqstats21:10:161
1127027516,24sleep20-21swapper/222:40:202
873027425,36sleep00-21swapper/019:09:570
3262027424,23sleep20-21swapper/223:49:552
3010627421,30sleep10-21swapper/120:15:161
3010627421,30sleep10-21swapper/120:15:161
2987127419,34sleep30-21swapper/320:15:053
2987127419,34sleep30-21swapper/320:15:053
1215927417,41sleep312212-21if_eth021:00:173
849227320,14sleep00-21swapper/000:15:180
849227320,14sleep00-21swapper/000:15:180
285582738,52sleep30-21swapper/320:10:173
207927321,32sleep32087-21modprobe20:29:553
1545627321,33sleep215607-21gpgv19:29:582
935327217,25sleep20-21swapper/220:50:262
935327217,25sleep20-21swapper/220:50:262
768027225,30sleep37739-21/usr/sbin/munin19:05:243
751127025,25sleep37255-21apt-get22:29:593
111927019,30sleep10-21swapper/123:50:271
3169226911,31sleep20-21swapper/220:20:172
3044426915,25sleep30-21swapper/321:59:583
2883126919,24sleep30-21swapper/321:54:573
2343226921,30sleep20-21swapper/219:54:582
1489126915,28sleep30-21swapper/300:35:163
1489126915,28sleep30-21swapper/300:35:163
141126917,26sleep30-21swapper/322:10:163
1270526914,26sleep20-21swapper/221:00:282
339126819,15sleep30-21swapper/323:59:583
2688926822,22sleep30-21swapper/320:05:163
256132689,16sleep30-21swapper/320:00:203
256132689,16sleep30-21swapper/320:00:203
245226817,25sleep20-21swapper/220:30:152
2057626821,29sleep120620-21/usr/sbin/munin19:45:121
972326716,37sleep19751-21ntp_kernel_pll_22:35:221
2945926724,18sleep20-21swapper/221:55:212
1906826721,29sleep119093-21/usr/sbin/munin23:05:231
1906826721,29sleep119093-21/usr/sbin/munin23:05:231
1797126723,20sleep10-21swapper/121:19:581
935726616,37sleep39376-21netstat19:10:213
773726620,26sleep30-21swapper/322:30:073
362626615,28sleep30-21swapper/320:34:563
1801126620,22sleep20-21swapper/221:19:592
1242426610,36sleep112443-21apt-get22:45:081
15502657,13sleep10-21swapper/122:10:181
82726419,23sleep20-21swapper/223:50:202
776626415,29sleep27760-21apt-get22:30:102
741426420,25sleep17484-21df19:05:181
2878026419,20sleep10-21swapper/121:54:551
76752637,46sleep30-21swapper/300:11:213
490626317,33sleep24927-21latency_hist00:04:582
1966826325,19sleep30-21swapper/319:40:253
1263226319,24sleep20-21swapper/222:45:172
72392628,16sleep07293-21apt-config22:29:560
2925426215,31sleep129284-21/usr/sbin/munin23:39:561
2826326213,34sleep328304-21ls23:35:183
2386026216,22sleep30-21swapper/323:20:243
2349626219,23sleep10-21swapper/121:35:281
1882526217,28sleep218939-21sed21:20:272
1536426223,22sleep20-21swapper/200:35:272
1536426223,22sleep20-21swapper/200:35:272
1406726220,21sleep20-21swapper/222:50:132
1158326217,28sleep10-21swapper/122:40:271
49772618,13sleep20-21swapper/222:20:252
359326122,24sleep10-21swapper/100:00:121
3192326117,22sleep30-21swapper/323:45:283
2788726117,24sleep20-21swapper/221:50:202
2492326116,17sleep20-21swapper/221:40:262
2492326116,17sleep20-21swapper/221:40:262
242126111,32sleep22457-21ntp_states23:55:202
2231226112,17sleep20-21swapper/223:15:262
1639026118,21sleep20-21swapper/221:14:582
1562826124,20sleep20-21swapper/222:55:142
1443426114,20sleep30-21swapper/300:34:563
1020926119,28sleep310243-21ntp_kernel_err00:20:223
839926012,35sleep2369-21sshd19:06:322
327426011,36sleep13319-40mandb23:59:561
3236726019,26sleep30-21swapper/322:05:153
2499926015,29sleep125010-21systemd19:59:571
1806326017,26sleep318178-21grep19:35:293
1611626021,23sleep10-21swapper/122:55:261
1403926021,22sleep30-21swapper/319:25:073
1220926021,17sleep30-21swapper/322:44:573
598425917,20sleep10-21swapper/122:25:121
493225913,22sleep10-21swapper/120:35:511
493225913,22sleep10-21swapper/120:35:511
2989625916,27sleep229892-21apt-get20:15:092
2989625916,27sleep229892-21apt-get20:15:092
2421025922,18sleep20-21swapper/221:39:582
2261325917,25sleep222684-21cut21:34:572
2261325917,25sleep222684-21cut21:34:572
212332595,11sleep20-21swapper/223:11:462
1894025916,20sleep30-21swapper/321:20:283
1446225921,21sleep30-21swapper/322:50:273
135442595,11sleep10-21swapper/119:21:541
1499125814,30sleep115057-21wc00:35:181
1499125814,30sleep115057-21wc00:35:181
1285025818,22sleep20-21swapper/200:29:582
1285025818,22sleep20-21swapper/200:29:582
1015125820,19sleep10-21swapper/120:54:581
1015125820,19sleep10-21swapper/120:54:581
730025716,17sleep10-21swapper/100:10:261
248842577,17sleep30-21swapper/319:59:233
2471625721,19sleep10-21swapper/123:24:581
2442925715,26sleep324424-21apt-get21:40:113
2442925715,26sleep324424-21apt-get21:40:113
236725715,26sleep12362-21apt-get20:30:101
1047125714,26sleep110475-21diskstats20:55:141
1016925714,18sleep20-21swapper/200:20:212
3113025617,19sleep20-21swapper/222:00:262
2720625618,23sleep10-21swapper/120:05:231
2581925612,21sleep10-21swapper/120:00:261
2581925612,21sleep10-21swapper/120:00:261
2510725620,18sleep30-21swapper/323:25:163
1877625619,16sleep10-21swapper/119:39:591
130462566,36sleep10-21swapper/121:02:331
1085625621,17sleep30-21swapper/320:55:243
442925516,23sleep14427-21apt-get22:20:071
2898525516,19sleep10-21swapper/120:10:271
2153025519,21sleep321502-21latency_hist23:14:583
1072525515,16sleep10-21swapper/119:15:031
862225416,23sleep30-21swapper/300:15:213
862225416,23sleep30-21swapper/300:15:213
440125413,23sleep34497-21users00:00:303
2987925414,23sleep329798-21apt-get23:41:043
2257025416,21sleep122611-21systemd-run21:34:561
2257025416,21sleep122611-21systemd-run21:34:561
219425415,23sleep22328-21date20:29:582
2021825414,23sleep120215-21apt-get23:10:081
1596125417,20sleep20-21swapper/219:30:312
1491925414,23sleep114966-21users19:25:291
1031425415,25sleep110341-21latency_hist19:14:591
50025315,18sleep30-21swapper/320:24:593
2082025314,22sleep320844-21irqstats19:45:183
573425212,23sleep35731-21apt-get20:40:093
2191625214,16sleep30-21swapper/321:30:263
2191625214,16sleep30-21swapper/321:30:263
190942529,18sleep20-21swapper/223:05:242
190942529,18sleep20-21swapper/223:05:242
1897425214,22sleep118969-21apt-get19:40:071
1273725216,21sleep10-21swapper/100:29:571
1273725216,21sleep10-21swapper/100:29:571
986225114,17sleep20-21swapper/222:35:272
2335225115,20sleep223419-21df23:20:102
2978025014,23sleep229299-21apt-get23:40:062
1383925014,15sleep30-21swapper/319:24:593
1093125012,21sleep310932-21apt-get19:15:093
1011025018,15sleep10-21swapper/100:20:201
326724913,21sleep13299-21ntp_states22:15:231
2183124913,20sleep121842-21df_inode23:15:131
1484224916,16sleep30-21swapper/321:09:583
66624710,16sleep30-21swapper/323:50:173
2654724615,17sleep10-21swapper/121:45:261
1375924614,18sleep30-21swapper/322:49:583
283532397,12sleep20-21swapper/220:10:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional