You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-26 - 09:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed Aug 26, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3293241611,388sleep03265-21latency_hist20:09:110
30566239220,352sleep230629-21fschecks_time21:34:282
30566239220,352sleep230629-21fschecks_time21:34:282
1635231817,281sleep31696-21latency_hist23:29:103
25237229111,266sleep325294-21fw_packets19:34:303
28734228195,73sleep128867-21timerwakeupswit23:09:381
18050226613,236sleep118120-21latency_hist22:39:101
19590225214,224sleep219644-21sort20:59:242
30848223913,213sleep330892-21ntpq21:34:333
30848223913,213sleep330892-21ntpq21:34:333
20436223853,163sleep120592-21latency_hist00:29:111
3682222117,185sleep23697-21kernelversion23:34:302
2933822177,197sleep129367-21ntpq21:29:341
17470220547,109sleep10-21swapper/100:19:221
21299220114,169sleep121510-21sed22:49:111
1310218916,159sleep21373-21idleruntime21:44:292
25927218514,62sleep10-21swapper/121:19:291
2226021827,165sleep1367-21snmpd21:07:181
27844217610,93sleep127860-21latency_hist23:09:101
19860217521,50sleep10-21swapper/120:59:311
5688217411,145sleep05763-21grep20:14:400
19923216812,61sleep120013-21fschecks_time22:44:271
7252216618,132sleep27335-21/usr/sbin/munin20:19:392
18704216585,54sleep10-21swapper/119:14:171
28640216449,67sleep10-21swapper/121:29:061
270421648,145sleep12770-21sort21:49:241
19353216010,66sleep119303-21latency_hist20:59:101
10092216020,116sleep310183-21sort20:29:293
28210215510,58sleep10-21swapper/119:44:251
31822215212,59sleep10-21swapper/121:39:111
31822215212,59sleep10-21swapper/121:39:111
27324214510,120sleep127438-21cat19:39:411
23854214373,28sleep124003-21tail22:54:401
10570214368,27sleep110633-21switchtime20:29:391
3266721418,118sleep1316-21cat21:39:391
3157321380,8sleep20-21swapper/219:54:302
14223213711,113sleep114336-21timerandwakeup22:24:411
18835213511,111sleep018837-21acpi19:14:200
6288213327,94sleep16322-21ps21:59:381
25171213222,89sleep225234-21cut19:34:292
17103213032,50sleep10-21swapper/120:49:421
978721298,106sleep19942-21date20:29:111
23624212914,98sleep123683-21sendmail-msp00:39:081
2564621269,51sleep125659-21awk22:59:411
2465212511,100sleep32434-21latency_hist21:49:103
13209212556,56sleep113255-21fschecks_count20:39:281
18899212429,81sleep119006-21wc22:39:381
15971212412,93sleep115854-21/usr/sbin/munin00:14:291
3222521235,44sleep10-21swapper/123:20:481
3222521235,44sleep10-21swapper/123:20:481
12255212319,15sleep10-21swapper/100:01:051
12255212319,15sleep10-21swapper/100:01:051
12542212213,92sleep312516-21latency_hist00:04:103
12542212213,92sleep312516-21latency_hist00:04:103
32648212010,67sleep1339-21latency_hist19:59:111
29825212011,90sleep10-21swapper/119:49:261
25354212010,98sleep125406-21latency19:34:321
7165211920,13sleep10-21swapper/120:19:381
22882211911,95sleep1312-21in:imuxsock22:54:081
3616211817,23sleep10-21swapper/120:09:281
3507211817,83sleep33520-21grep23:34:273
26439211817,85sleep026517-21find21:19:400
23374211811,92sleep123495-21cat21:09:401
13054211710,19sleep10-21swapper/100:04:321
14729211610,92sleep114772-21awk00:09:331
10744211626,57sleep110814-21cat22:14:311
23697211412,85sleep10-21swapper/119:29:311
20840211315,76sleep020135-21kworker/u8:219:19:330
20772211325,52sleep10-21swapper/100:29:151
15134211325,67sleep10-21swapper/122:29:211
9640211221,16sleep10-21swapper/123:54:231
7565211030,64sleep10-21swapper/122:04:301
755621095,12sleep10-21swapper/123:45:551
622721095,11sleep10-21swapper/123:43:451
14041210918,74sleep014063-21awk00:09:070
4835210813,79sleep24895-21cat20:14:102
27485210819,69sleep027592-21grep21:24:280
535721075,11sleep10-21swapper/121:57:111
8315210624,50sleep10-21swapper/123:49:301
506210620,13sleep10-21swapper/119:59:271
3025321069,59sleep130342-21cat23:14:391
13318210615,51sleep11-21systemd22:24:081
32135210514,74sleep332201-21users19:54:413
26738210510,17sleep30-21swapper/319:39:263
21095210551,33sleep20-21swapper/222:49:082
21095210551,33sleep20-21swapper/222:49:082
1874210518,14sleep10-21swapper/123:29:241
14560210512,73sleep014771-21latency_hist20:44:120
641221048,83sleep16421-21munin-run20:19:091
21289210414,76sleep321351-21sendmail_mailqu00:29:413
1678321049,21sleep00-21swapper/022:34:250
11969210410,74sleep212121-21sed23:59:412
11969210410,74sleep212121-21sed23:59:412
16208210273,14sleep10-21swapper/120:49:091
13700210220,52sleep213795-21sort22:24:302
1816221019,14sleep10-21swapper/119:14:091
1498629915,68sleep014993-21diskstats20:44:280
401029822,26sleep30-21swapper/321:54:103
124129866,16sleep10-21swapper/121:44:281
194792977,12sleep30-21swapper/300:24:373
40529610,69sleep1471-21fschecks_time23:24:271
1496029524,45sleep314982-21df20:44:273
647229412,66sleep36618-21latency_hist20:19:113
204432946,12sleep10-21swapper/119:19:221
2247329218,51sleep022441-21latency_hist21:09:110
2559229112,62sleep025683-21mailstats19:34:390
1408329120,39sleep20-21swapper/200:09:092
780628925,42sleep37812-21modprobe23:49:083
379928915,56sleep03830-21/usr/sbin/munin20:09:320
1280228911,16sleep30-21swapper/300:04:263
496928716,49sleep05119-21grep21:54:400
2111828718,47sleep321164-21unin-run22:49:093
2111828718,47sleep321164-21unin-run22:49:093
3262928616,50sleep032646-21grep19:59:100
2935128620,46sleep329377-21idleruntime-cro23:14:083
1666228618,45sleep016832-21grep19:09:110
322952850,6sleep00-21swapper/021:39:290
2492628511,56sleep10-21swapper/121:14:411
1149728510,61sleep211498-21sshd22:16:062
2238928417,54sleep122432-21ntp_kernel_pll_19:24:341
1996528418,48sleep320011-21ntp_kernel_pll_20:59:333
1821228414,23sleep30-21swapper/319:14:113
31728315,43sleep2474-21unixbench_singl21:39:402
2464828326,38sleep324687-21grep21:14:353
1563928319,28sleep20-21swapper/200:14:102
45452825,10sleep10-21swapper/123:37:231
2085728210,58sleep020883-21sh00:29:260
126412825,11sleep10-21swapper/120:37:021
960928022,38sleep39747-21grep22:09:413
31032805,10sleep10-21swapper/120:07:231
278728018,41sleep02819-21fschecks_count21:49:270
2170228014,51sleep021708-21df_inode22:49:260
1950428015,20sleep00-21swapper/000:24:380
2325027919,40sleep323266-21idleruntime-cro19:29:093
99952785,41sleep10-21swapper/122:11:411
1558827815,42sleep00-21swapper/000:14:080
865627712,18sleep20-21swapper/220:25:082
738327714,20sleep00-21swapper/022:04:250
2902427710,50sleep30-21swapper/321:29:283
2285027711,52sleep322920-21grep00:34:383
768227615,47sleep07683-21sshd20:21:060
2235927616,29sleep00-21swapper/019:24:330
1152227616,22sleep20-21swapper/220:34:212
1568627521,35sleep215725-21ntpq19:04:332
1755627427,24sleep30-21swapper/300:19:283
2422427318,26sleep00-21swapper/019:29:420
2400527316,26sleep20-21swapper/222:54:412
1686027312,25sleep30-21swapper/320:49:363
818227213,41sleep38242-21fschecks_time23:49:283
3196027214,43sleep232106-21awk23:19:402
3196027214,43sleep232106-21awk23:19:402
2651227223,31sleep226531-21apt-get23:04:182
2628627220,28sleep30-21swapper/323:04:113
1541627218,32sleep30-21swapper/322:29:313
1214927216,39sleep312240-21/usr/sbin/munin22:19:263
312927112,43sleep2374-21sshd23:34:052
2307227115,42sleep223099-21netstat21:09:332
68382707,11sleep20-21swapper/222:00:522
1577827019,26sleep10-21swapper/119:04:361
126672705,54sleep00-21swapper/020:37:210
971426910,16sleep00-21swapper/023:54:260
3268026912,12sleep20-21swapper/223:24:122
234826913,37sleep32389-21grep20:04:343
2160426916,34sleep221610-21ntp_states21:04:412
93926819,23sleep00-21swapper/023:24:390
89312688,14sleep00-21swapper/022:09:260
1585326813,40sleep015855-21/usr/sbin/munin00:14:200
1164926812,16sleep00-21swapper/023:59:350
1164926812,16sleep00-21swapper/023:59:350
935126716,33sleep29352-21cron23:54:072
897126727,18sleep30-21swapper/320:25:173
2313626713,38sleep023158-21ntp_states21:09:350
161312676,12sleep20-21swapper/220:48:422
158692667,36sleep30-21swapper/319:04:383
478326511,33sleep20-21swapper/221:54:382
3140826521,24sleep30-21swapper/323:19:293
3140826521,24sleep30-21swapper/323:19:293
2176726517,23sleep20-21swapper/222:49:452
2054626514,31sleep320631-21fschecks_time19:19:283
146862655,11sleep30-21swapper/322:26:493
2878226414,23sleep30-21swapper/319:44:393
142426414,35sleep01501-21latency21:44:310
1362126411,32sleep30-21swapper/320:39:383
1130826413,35sleep011360-21latency_hist20:34:100
104412647,11sleep30-21swapper/322:14:133
73362639,40sleep37448-21sed20:19:403
400026313,35sleep24009-21/usr/sbin/munin21:54:092
2845626313,24sleep20-21swapper/223:09:312
2717526312,34sleep027246-21tr23:06:260
1852126318,21sleep20-21swapper/220:54:392
1796926314,32sleep317976-21modprobe22:39:083
130322634,13sleep00-21swapper/022:20:280
964126216,24sleep20-21swapper/223:54:242
89122625,12sleep00-21swapper/020:25:140
2866426215,19sleep00-21swapper/023:09:370
2714726219,19sleep20-21swapper/221:24:102
2479826216,28sleep30-21swapper/319:34:093
1899326212,33sleep218998-21sort00:24:192
1877226218,25sleep30-21swapper/322:39:363
174326215,30sleep01833-21sort20:04:100
1725026217,20sleep20-21swapper/200:19:112
7862617,11sleep20-21swapper/219:59:322
373926111,27sleep30-21swapper/320:09:313
252232618,42sleep00-21swapper/021:15:240
940326017,23sleep00-21swapper/023:54:100
582726013,21sleep00-21swapper/021:59:240
307912605,43sleep20-21swapper/219:51:162
3034326013,21sleep20-21swapper/223:14:402
2901826019,24sleep20-21swapper/221:29:272
2153726021,23sleep20-21swapper/200:29:442
110092607,12sleep00-21swapper/022:14:390
467925915,23sleep04692-21modprobe23:39:080
3168925910,36sleep0367-21snmpd21:38:290
3168925910,36sleep0367-21snmpd21:38:290
2939825912,31sleep029432-21ps21:29:370
2869825917,19sleep20-21swapper/221:29:112
2115225919,18sleep30-21swapper/321:04:293
1173525915,27sleep311787-21/usr/sbin/munin20:34:303
265702587,37sleep30-21swapper/323:04:433
2634525818,25sleep326435-21grep21:19:393
236062588,39sleep00-21swapper/000:38:550
2052825817,17sleep20-21swapper/219:19:272
195032586,12sleep00-21swapper/022:43:330
1824225819,18sleep00-21swapper/019:14:120
318525716,24sleep33313-21ls23:34:103
234422579,12sleep00-21swapper/022:54:310
1439125713,25sleep214454-21apt-config20:44:092
3178025611,19sleep00-21swapper/023:19:380
3178025611,19sleep00-21swapper/023:19:380
3100725610,37sleep00-21swapper/019:54:050
2797025615,17sleep20-21swapper/219:44:102
2792025615,26sleep027936-21apt19:44:080
2020725614,23sleep20-21swapper/219:19:102
1826525616,24sleep218284-21apt-get22:39:192
1645825615,25sleep016451-21apt-get20:49:240
1524425614,22sleep00-21swapper/022:29:280
1008125614,19sleep20-21swapper/220:29:282
485525516,19sleep30-21swapper/321:54:393
3207825514,24sleep332131-21df21:39:253
307642555,38sleep00-21swapper/023:17:160
2870625515,19sleep20-21swapper/219:44:382
1360825514,25sleep013603-21apt-get22:24:240
654325413,25sleep06539-21apt-get23:44:210
2964625412,23sleep329713-21df23:14:243
291232544,42sleep00-21swapper/019:44:440
2368625415,23sleep223703-21munin-run00:39:092
2243325414,22sleep222446-21ntp_states19:24:352
2029025412,25sleep020296-21ntp_states22:44:340
2029025412,25sleep020296-21ntp_states22:44:340
195525414,19sleep20-21swapper/220:04:212
1690625413,24sleep216899-21apt-get19:09:252
1646525415,15sleep20-21swapper/222:34:102
871825315,18sleep20-21swapper/222:09:112
3002625312,24sleep330120-21memory19:49:323
2339125311,15sleep30-21swapper/322:54:303
2310025315,19sleep30-21swapper/321:09:343
1645125313,23sleep316534-21sed00:14:393
966925212,24sleep39712-21df23:54:253
242282528,16sleep20-21swapper/221:14:212
2253625215,20sleep30-21swapper/322:49:583
1813725213,23sleep018221-21perl20:54:290
1667925214,18sleep20-21swapper/222:34:182
59242517,16sleep20-21swapper/220:14:422
1754925113,23sleep217555-21diskstats00:19:272
718425010,23sleep26521-21/usr/sbin/munin23:44:392
56862504,11sleep30-21swapper/321:59:113
500125011,21sleep05068-21df23:39:240
2747225014,18sleep30-21swapper/321:24:273
1893125012,23sleep318940-21df_inode19:14:273
1692925014,23sleep216432-21/usr/sbin/munin20:49:392
1441325014,17sleep30-21swapper/300:09:273
137125017,18sleep30-21swapper/320:00:063
1018525010,16sleep20-21swapper/222:14:082
2090724913,23sleep020976-21sed21:04:100
545124810,23sleep25457-21ntp_states23:39:342
2482524814,20sleep224831-21diskstats22:59:262
226624812,16sleep00-21swapper/023:29:350
2473124611,20sleep024743-21apt-get22:59:190
1801524611,18sleep30-21swapper/320:54:243
1336824613,20sleep213444-21sed22:24:092
2395624512,16sleep20-21swapper/219:29:362
74652439,15sleep30-21swapper/322:04:283
499824312,18sleep30-21swapper/323:39:233
1758324210,16sleep00-21swapper/000:19:290
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional