You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-22 - 04:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Apr 22, 2026 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
273622167522,1633sleep323741-21kworker/u8:220:46:053
273622167522,1633sleep323741-21kworker/u8:220:46:053
312562125011,1223sleep25353-1kworker/2:2H22:41:172
7377277822,737sleep37460-21users23:06:263
849126564,641sleep031233-1kworker/0:1H23:11:320
2046926528,627sleep120512-21/usr/sbin/munin22:06:191
15315241368,244sleep015451-21unixbench_singl23:31:270
9146236516,330sleep39202-21/usr/sbin/munin21:31:163
9146236516,330sleep39202-21/usr/sbin/munin21:31:163
25360233522,290sleep225446-21latency_hist20:40:552
25360233522,290sleep225446-21latency_hist20:40:552
11191228313,256sleep211236-21unin-run23:20:552
50542273125,90sleep05195-21unixbench_singl21:16:270
30494226615,235sleep330545-21fschecks_count00:21:133
1534224723,210sleep11598-21awk00:31:241
3272224020,195sleep327278-21kworker/u8:019:30:543
12188222614,197sleep312257-21switchtime19:56:353
12188222614,197sleep312257-21switchtime19:56:353
23815220811,175sleep023940-21latency_hist20:35:570
23815220811,175sleep023940-21latency_hist20:35:570
1820122088,85sleep00-21swapper/020:16:180
1820122088,85sleep00-21swapper/020:16:180
5383220514,173sleep15414-21/usr/sbin/munin23:01:191
16315220311,178sleep016362-21idleruntime23:36:150
28949220016,162sleep028963-21fschecks_count00:16:130
18062218516,154sleep318064-21unin-run22:00:563
18062218516,154sleep318064-21unin-run22:00:563
1348821838,159sleep013514-21proc_pri23:26:220
12224218310,157sleep012291-21sort21:41:140
12224218310,157sleep012291-21sort21:41:140
11476218221,140sleep00-21swapper/023:21:090
767721809,159sleep07757-21latency21:26:160
767721809,159sleep07757-21latency21:26:160
5689217910,154sleep05809-21sed23:01:260
16340217927,84sleep00-21swapper/020:11:130
26135217818,141sleep00-21swapper/022:25:570
3051821758,149sleep00-21swapper/020:56:080
9962217342,72sleep09965-21sshd21:32:420
9962217342,72sleep09965-21sshd21:32:420
28906217311,147sleep028987-21sed19:10:560
2329821718,17sleep00-21swapper/022:16:110
483221688,145sleep04972-21date23:00:580
24139216711,136sleep00-21swapper/020:36:130
24139216711,136sleep00-21swapper/020:36:130
19065216415,45sleep00-21swapper/022:02:060
19005216350,93sleep00-21swapper/023:45:560
27240216163,27sleep00-21swapper/000:11:040
2991921598,140sleep029953-21ps22:36:230
3772215812,124sleep00-21swapper/019:31:160
241621486,10sleep00-21swapper/021:10:370
668821458,125sleep06726-21cpu19:41:090
668821458,125sleep06726-21cpu19:41:090
2268721435,11sleep00-21swapper/022:13:000
12022214120,106sleep212094-21cat23:21:232
1108321405,11sleep00-21swapper/023:19:310
17404213919,100sleep217417-21modprobe23:40:532
1454213810,17sleep00-21swapper/019:22:270
31998213517,102sleep00-21swapper/022:41:360
14941213411,42sleep00-21swapper/020:06:170
24262213310,69sleep00-21swapper/000:01:150
1643213311,71sleep01680-21sed22:50:560
17410213174,32sleep00-21swapper/021:56:290
17410213174,32sleep00-21swapper/021:56:290
12757213120,84sleep312895-21grep21:41:263
12757213120,84sleep312895-21grep21:41:263
31370212819,91sleep330842-21apt-get19:16:083
15909212721,86sleep116029-21latency_hist23:35:561
7815212518,89sleep27816-21cron23:10:532
878212423,75sleep1914-21kworker/u8:021:05:541
878212423,75sleep1914-21kworker/u8:021:05:541
30792212312,96sleep130872-21latency20:56:171
8347212216,45sleep20-21swapper/223:11:102
781721205,103sleep322168-1kworker/3:2H19:44:153
781721205,103sleep322168-1kworker/3:2H19:44:153
28090212015,86sleep228138-21/usr/sbin/munin22:31:162
28090212015,86sleep228138-21/usr/sbin/munin22:31:162
2668321205,74sleep00-21swapper/020:43:360
2617121170,14sleep226151-21awk20:41:232
2138021178,54sleep00-21swapper/022:10:540
2040721175,101sleep00-21swapper/020:22:520
2174721165,68sleep00-21swapper/023:52:020
2916321157,31sleep00-21swapper/022:35:540
2916321157,31sleep00-21swapper/022:35:540
13833211518,24sleep00-21swapper/021:46:150
3215021148,31sleep00-21swapper/021:01:130
3215021148,31sleep00-21swapper/021:01:130
30280211419,79sleep230416-21awk19:11:272
26692211418,76sleep126725-21/usr/sbin/munin22:26:181
26692211418,76sleep126725-21/usr/sbin/munin22:26:181
9227211311,84sleep19321-21memory21:31:181
9227211311,84sleep19321-21memory21:31:181
2922921098,84sleep00-21swapper/019:11:010
1914210917,66sleep32056-21sed21:06:293
27030210820,19sleep00-21swapper/000:10:560
25546210816,72sleep325675-21/usr/sbin/munin22:21:273
18159210818,74sleep118215-21latency_hist22:00:581
18159210818,74sleep118215-21latency_hist22:00:581
5113210618,74sleep15130-21awk19:36:051
5113210618,74sleep15130-21awk19:36:051
15600210516,60sleep314678-21/usr/sbin/munin20:06:283
27644210422,69sleep127671-21awk00:11:171
750221025,85sleep00-21swapper/023:06:510
22995210111,55sleep023120-21cat23:56:260
22995210111,55sleep023120-21cat23:56:260
13405210111,74sleep213543-21latency_hist21:45:572
13405210111,74sleep213543-21latency_hist21:45:572
6250210015,74sleep00-21swapper/019:38:590
6250210015,74sleep00-21swapper/019:38:590
454021005,83sleep00-21swapper/022:58:320
29135210021,57sleep329156-21latency_hist22:35:533
29135210021,57sleep329156-21latency_hist22:35:533
2120021006,12sleep10-21swapper/120:26:171
13625210013,73sleep313689-21cpuspeed21:46:103
1053121006,41sleep00-21swapper/021:36:100
82042995,45sleep00-21swapper/019:46:020
3028229811,33sleep00-21swapper/020:55:560
3028229811,33sleep00-21swapper/020:55:560
3020229818,60sleep230264-21grep20:55:532
3020229818,60sleep230264-21grep20:55:532
3087129730,56sleep030382-21/usr/sbin/munin00:21:220
3211329634,50sleep032256-21sed19:17:100
267992967,78sleep313441-21snmpd20:45:083
262922965,81sleep00-21swapper/022:26:010
262922965,81sleep00-21swapper/022:26:010
2193929619,28sleep00-21swapper/020:27:410
713429513,65sleep30-21swapper/321:25:553
711729530,44sleep27130-21modprobe21:25:542
414029527,51sleep14215-21wc19:31:251
364929515,56sleep20-21swapper/219:31:142
3207929510,70sleep332147-21df21:01:123
3207929510,70sleep332147-21df21:01:123
2732829522,60sleep027361-21latency_hist19:05:560
151672959,33sleep00-21swapper/021:51:080
1371029517,54sleep213723-21fschecks_count21:46:132
1274429518,29sleep00-21swapper/019:59:180
1274429518,29sleep00-21swapper/019:59:180
2486229434,36sleep224923-21fschecks_time22:21:142
137482946,66sleep20-21swapper/220:01:262
137482946,66sleep20-21swapper/220:01:262
201629315,27sleep00-21swapper/000:32:120
73829218,27sleep00-21swapper/000:30:380
73829218,27sleep00-21swapper/000:30:380
287929218,27sleep00-21swapper/022:52:340
2856429218,28sleep00-21swapper/020:49:420
2856429218,28sleep00-21swapper/020:49:420
2019729218,29sleep00-21swapper/023:47:010
2019729218,29sleep00-21swapper/023:47:010
109842925,75sleep00-21swapper/019:55:100
2609729122,34sleep10-21swapper/119:01:131
2504529123,50sleep125081-21cat22:21:171
2772029015,55sleep227806-21/usr/sbin/munin19:06:142
905728913,62sleep19181-21timerwakeupswit23:11:531
800128922,40sleep18197-21wc23:10:561
303028815,28sleep00-21swapper/019:27:440
448528712,62sleep24486-21sshd22:57:532
448528712,62sleep24486-21sshd22:57:532
2954628716,48sleep129704-21unixbench_multi00:16:261
2647528729,24sleep20-21swapper/219:01:222
2542728618,47sleep213441-21snmpd00:05:532
223762866,13sleep20-21swapper/223:56:082
223762866,13sleep20-21swapper/223:56:082
2985428512,59sleep129892-21ntpq22:36:201
27262849,41sleep00-21swapper/021:11:100
2049928413,50sleep120525-21/usr/sbin/munin23:50:541
2049928413,50sleep120525-21/usr/sbin/munin23:50:541
2571828325,32sleep20-21swapper/200:06:092
794828115,53sleep17995-21unin-run19:45:541
794828115,53sleep17995-21unin-run19:45:541
7812810,7sleep20-21swapper/222:46:172
185728116,40sleep21876-21apt-get22:51:062
1481028117,46sleep214844-21/usr/sbin/munin23:31:162
1156828117,45sleep111062-21apt-get19:56:041
1156828117,45sleep111062-21apt-get19:56:041
2921728026,27sleep10-21swapper/122:35:551
2921728026,27sleep10-21swapper/122:35:551
150328011,50sleep31509-21ntp_states00:31:223
131282806,28sleep00-21swapper/020:01:040
131282806,28sleep00-21swapper/020:01:040
636427943,21sleep06444-21tr21:21:240
95227815,20sleep10-21swapper/119:21:231
2369827817,43sleep123784-21apt-config00:00:551
2369827817,43sleep123784-21apt-config00:00:551
2257427817,36sleep322674-21fschecks_time20:31:153
2257427817,36sleep322674-21fschecks_time20:31:153
159127820,42sleep11592-21cron22:50:531
2870727725,37sleep128713-21latency_hist20:50:561
2870727725,37sleep128713-21latency_hist20:50:561
1441327720,38sleep114426-21sh20:05:531
1441327720,38sleep114426-21sh20:05:531
1104127719,34sleep311062-21apt-get19:55:543
1417827615,43sleep114249-21ls21:46:241
869827512,48sleep18721-21/usr/sbin/munin19:46:231
2616227526,27sleep30-21swapper/319:01:153
1029227516,46sleep210376-21latency_hist21:35:552
1029227516,46sleep210376-21latency_hist21:35:552
2862327422,28sleep30-21swapper/300:15:553
2599127414,43sleep126045-21latency00:06:161
238027420,38sleep125884-21systemd-journal21:10:101
2246427416,45sleep322499-21fschecks_count23:56:133
2246427416,45sleep322499-21fschecks_count23:56:133
1806327418,30sleep217668-21/usr/sbin/munin23:41:182
1479127420,38sleep214883-21tune2fs20:06:152
2104627312,41sleep10-21swapper/123:51:141
2799827216,36sleep328043-21grep19:06:193
558927114,38sleep30-21swapper/323:01:253
414627114,39sleep34296-21unixbench_multi22:56:253
414627114,39sleep34296-21unixbench_multi22:56:253
2898827121,32sleep129062-21ntp_states19:10:571
2566827116,35sleep10-21swapper/120:41:111
1625527115,18sleep10-21swapper/120:11:101
1033027124,25sleep20-21swapper/219:51:252
3268227023,23sleep20-21swapper/219:20:562
1985627012,42sleep219993-21latency_hist22:05:562
1809327012,35sleep30-21swapper/323:41:193
1623527026,19sleep30-21swapper/323:36:143
1073127023,20sleep10-21swapper/121:36:151
2006826929,22sleep30-21swapper/322:06:053
1959126916,30sleep10-21swapper/120:21:161
795126812,41sleep38018-21swap21:26:233
795126812,41sleep38018-21swap21:26:233
3136826819,30sleep130842-21apt-get19:16:061
2749426822,25sleep20-21swapper/220:46:212
2749426822,25sleep20-21swapper/220:46:212
2332626822,17sleep30-21swapper/322:16:143
326922677,39sleep20-21swapper/221:01:272
326922677,39sleep20-21swapper/221:01:272
3019026720,25sleep30-21swapper/300:20:563
2927826717,25sleep10-21swapper/119:11:021
248726718,28sleep20-21swapper/221:10:562
3098126618,25sleep20-21swapper/220:56:222
2857026620,32sleep21-21systemd00:15:542
2174326618,27sleep10-21swapper/122:11:161
1315926611,40sleep313178-21grep20:01:113
1315926611,40sleep313178-21grep20:01:113
407126514,38sleep14016-21latency_hist21:15:551
2933226513,27sleep20-21swapper/200:16:232
275826512,38sleep22800-21df21:11:122
2632026518,29sleep226316-21apt-get22:26:072
2632026518,29sleep226316-21apt-get22:26:072
262326515,26sleep20-21swapper/219:26:252
2432326519,28sleep124385-21/usr/sbin/munin00:01:161
1995326522,23sleep20-21swapper/220:21:252
3272726412,22sleep30-21swapper/300:26:253
3272726412,22sleep30-21swapper/300:26:253
321726420,19sleep30-21swapper/322:55:563
1878226421,25sleep10-21swapper/122:01:551
147862640,7sleep10-21swapper/120:06:131
104526415,30sleep21041-21apt-get00:31:092
103526415,20sleep20-21swapper/219:21:242
3153126318,29sleep131532-21sshd00:24:091
217552635,10sleep30-21swapper/323:52:093
2533426215,31sleep125359-21munin-run20:40:541
2533426215,31sleep125359-21munin-run20:40:541
1886826212,26sleep30-21swapper/322:01:573
1302426220,25sleep313019-21apt-get23:26:093
101326217,29sleep31172-21unixbench_multi22:46:233
431026116,29sleep34311-21sshd21:16:103
332626113,26sleep10-21swapper/119:30:561
3255426122,20sleep10-21swapper/122:45:571
3071126112,21sleep30-21swapper/322:40:543
3070326121,16sleep30-21swapper/320:56:153
3003226117,19sleep20-21swapper/222:36:252
1359926115,28sleep213668-21/usr/sbin/munin23:26:242
2213726014,25sleep20-21swapper/223:55:562
1909426014,19sleep30-21swapper/320:20:533
1909426014,19sleep30-21swapper/320:20:533
1597426014,24sleep30-21swapper/321:51:273
1338526016,25sleep10-21swapper/121:45:561
1338526016,25sleep10-21swapper/121:45:561
1000426014,28sleep30-21swapper/323:16:113
993525916,29sleep110001-21basename19:51:151
356625919,21sleep30-21swapper/319:31:113
3089925915,28sleep331042-21readlink19:15:563
2331325913,21sleep20-21swapper/222:16:132
2245025911,23sleep20-21swapper/220:31:062
2245025911,23sleep20-21swapper/220:31:062
2002025910,18sleep30-21swapper/320:21:263
1454525910,19sleep30-21swapper/323:31:053
740625825,20sleep10-21swapper/121:26:091
740625825,20sleep10-21swapper/121:26:091
2891925817,24sleep328938-21apt-get20:51:063
2891925817,24sleep328938-21apt-get20:51:063
2659225815,24sleep30-21swapper/322:26:163
2659225815,24sleep30-21swapper/322:26:163
1614525816,26sleep116140-21apt-get23:36:081
852425713,29sleep38562-21fschecks_time23:11:393
841925718,21sleep20-21swapper/219:46:162
2432125716,22sleep30-21swapper/320:36:163
2432125716,22sleep30-21swapper/320:36:163
2129925720,23sleep321339-21grep20:26:193
206392577,11sleep20-21swapper/222:06:242
1118725719,23sleep111189-21cron23:20:541
758025622,16sleep20-21swapper/221:26:142
758025622,16sleep20-21swapper/221:26:142
2160525620,18sleep20-21swapper/220:26:262
1759125613,22sleep217608-21latency_hist20:15:562
1201525611,28sleep20-21swapper/219:56:302
1201525611,28sleep20-21swapper/219:56:302
3040625514,24sleep230402-21apt-get00:21:072
2164325517,22sleep30-21swapper/322:11:103
114422556,37sleep30-21swapper/323:21:003
2255525412,26sleep122562-21diskstats20:31:131
2255525412,26sleep122562-21diskstats20:31:131
2055025417,16sleep30-21swapper/323:50:563
2055025417,16sleep30-21swapper/323:50:563
1584825418,17sleep20-21swapper/221:51:262
1236625413,27sleep112377-21users23:21:261
412925313,23sleep20-21swapper/221:15:562
3203625315,22sleep132041-21diskstats00:26:111
3203625315,22sleep132041-21diskstats00:26:111
2161625317,18sleep20-21swapper/222:11:052
1981525318,20sleep20-21swapper/223:46:232
1981525318,20sleep20-21swapper/223:46:232
1821625310,27sleep118255-21/usr/sbin/munin20:16:191
1821625310,27sleep118255-21/usr/sbin/munin20:16:191
1727925319,18sleep30-21swapper/320:12:543
1109325311,13sleep20-21swapper/221:36:242
70212524,38sleep10-21swapper/121:24:401
430825213,24sleep14305-21apt-get21:16:081
296825214,22sleep32997-21irqstats21:11:173
2948125213,18sleep10-21swapper/120:51:241
2948125213,18sleep10-21swapper/120:51:241
266622525,17sleep30-21swapper/300:06:513
1561725223,16sleep10-21swapper/121:51:211
880325116,17sleep30-21swapper/319:46:253
645025116,18sleep30-21swapper/319:40:563
645025116,18sleep30-21swapper/319:40:563
3122725114,20sleep30-21swapper/322:41:063
1839125012,22sleep218029-21apt-get22:01:072
1839125012,22sleep218029-21apt-get22:01:072
178052505,33sleep20-21swapper/220:16:002
178052505,33sleep20-21swapper/220:16:002
1341725012,23sleep113455-21/usr/sbin/munin23:26:191
1098425015,21sleep310990-21ntp_states21:36:213
73824910,20sleep3832-21memory19:21:173
710324812,20sleep27141-21ntp_states19:41:192
710324812,20sleep27141-21ntp_states19:41:192
62622485,31sleep20-21swapper/219:39:092
62622485,31sleep20-21swapper/219:39:092
2365824811,19sleep10-21swapper/122:16:211
398924716,16sleep10-21swapper/122:56:231
398924716,16sleep10-21swapper/122:56:231
1778124715,15sleep10-21swapper/123:41:131
188324610,18sleep10-21swapper/122:51:101
2414124513,16sleep30-21swapper/300:01:053
322382408,13sleep20-21swapper/200:26:162
322382408,13sleep20-21swapper/200:26:162
541523910,16sleep20-21swapper/223:01:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional