You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-25 - 20:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Tue Aug 25, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26858257357,5690sleep324312-21kworker/3:108:40:093
26858257357,5690sleep324312-21kworker/3:108:40:093
3754233467,3326sleep026086-21kworker/0:207:28:090
28638274925,705sleep128730-21/usr/sbin/munin07:04:321
6615261712,586sleep16617-21sshd07:35:091
10257258721,453sleep310397-21grep09:29:443
27882256123,497sleep328026-21cat12:09:123
227512507106,383sleep322899-21head10:09:423
2898823859,337sleep329009-21dpkg08:49:213
30353231614,258sleep330484-21grep12:14:413
58662275110,147sleep35902-21run-parts09:19:103
281222261114,129sleep328318-21apt-config10:29:123
8742223742,62sleep30-21swapper/309:24:433
15757223431,189sleep315813-21sendmail_mailqu11:30:433
13902222938,168sleep313923-21modprobe09:44:093
16060222317,192sleep216152-21load09:49:342
12408221935,164sleep312534-21latency_hist09:39:133
12408221935,164sleep312534-21latency_hist09:39:133
920221013,181sleep2977-21awk10:44:302
15118220421,56sleep315224-21fschecks_time08:04:313
14663220169,53sleep30-21swapper/309:44:383
11944219910,174sleep011955-21awk11:19:300
423121815,13sleep00-21swapper/009:13:090
423121815,13sleep00-21swapper/009:13:090
25142217411,123sleep30-21swapper/310:19:233
7100216934,122sleep37216-21timerwakeupswit09:20:113
23618216413,136sleep323673-21if_err_eth011:54:303
6271215816,127sleep26349-21find07:34:412
8914215612,130sleep08915-21sshd09:25:080
860215210,127sleep30-21swapper/312:24:393
2022215029,98sleep30-21swapper/310:49:123
32169214311,119sleep332219-21latency07:14:353
394721417,120sleep34087-21date07:29:133
22159213936,64sleep30-21swapper/311:49:333
198221369,18sleep30-21swapper/309:05:043
12433213627,97sleep312488-21switchtime11:19:413
23730213421,21sleep30-21swapper/310:14:303
24846213244,71sleep324318-21/usr/sbin/munin08:34:393
30968212917,98sleep031005-21sed08:54:350
20265212922,39sleep30-21swapper/310:04:133
12157212926,14sleep30-21swapper/307:54:333
2632921289,107sleep326393-21latency_hist12:04:113
31902212513,73sleep332010-21timerwakeupswit12:19:423
23709212422,41sleep30-21swapper/308:29:473
14069212414,94sleep014171-21/usr/sbin/munin07:59:390
2342212226,35sleep30-21swapper/307:24:123
8722212121,37sleep30-21swapper/311:09:233
8220211757,22sleep31-21systemd11:09:113
15884211518,38sleep30-21swapper/309:49:303
28796211213,18sleep00-21swapper/007:04:350
1706211114,88sleep00-21swapper/010:45:450
28468211025,63sleep30-21swapper/310:29:213
27309211021,60sleep227458-21grep12:04:432
18484210812,74sleep118515-21/usr/sbin/munin08:14:341
1088210718,69sleep01208-21sed09:04:130
1088210718,69sleep01208-21sed09:04:130
3190221066,89sleep0364-21ntpd07:14:280
2773321067,63sleep327816-21sed08:44:333
13496210620,65sleep213560-21sort11:24:322
20465210522,45sleep30-21swapper/310:04:253
17580210536,22sleep30-21swapper/311:35:003
13406210527,15sleep30-21swapper/311:24:253
13285210534,14sleep30-21swapper/307:59:143
32249210416,72sleep232366-21ls10:39:422
2062921045,36sleep30-21swapper/308:19:573
4585210333,33sleep30-21swapper/307:29:373
16948210315,69sleep116464-21apt-get11:34:211
11474210318,50sleep30-21swapper/307:52:353
9628210116,18sleep30-21swapper/307:44:443
20057210133,27sleep320087-21modprobe11:44:123
7930210027,33sleep30-21swapper/307:39:433
575921008,33sleep35790-21tune2fs07:34:303
4133210020,24sleep10-21swapper/112:34:421
24916210010,77sleep025011-21switchtime08:34:400
1859029921,53sleep118636-21modprobe09:59:101
1354929913,66sleep113620-21fschecks_time07:59:281
2541029819,63sleep225478-21latency10:19:302
1089929813,67sleep111023-21latency_hist09:34:141
2996129711,17sleep30-21swapper/307:09:303
1324229729,46sleep013324-21find09:39:420
215682965,57sleep30-21swapper/311:48:493
3215229542,34sleep30-21swapper/308:59:273
3215229542,34sleep30-21swapper/308:59:273
2873129525,48sleep20-21swapper/207:04:332
2477429513,65sleep024892-21date11:59:120
206529512,50sleep32127-21ls12:29:313
1155329410,69sleep111629-21tr09:34:381
1155329410,69sleep111629-21tr09:34:381
2794629321,58sleep027971-21latency_hist10:29:090
785329211,69sleep27927-21find07:39:422
1779729216,35sleep30-21swapper/309:54:393
219222918,14sleep30-21swapper/308:24:423
1662029160,15sleep30-21swapper/308:09:273
316162909,67sleep331643-21grep10:39:263
1724429018,54sleep117339-21timerwakeupswit08:09:421
1045729011,63sleep210521-21if_eth011:14:312
1876528911,63sleep118768-21/usr/sbin/munin11:39:221
2228328811,65sleep222343-21latency10:09:332
3003828715,34sleep30-21swapper/310:34:263
212328710,66sleep1367-21snmpd07:22:041
2343728611,25sleep20-21swapper/211:54:252
58362858,41sleep35837-21ntp_states10:59:383
3162859,63sleep1323-21acpi12:24:201
389028252,12sleep30-21swapper/310:54:293
2677928238,27sleep226840-21fschecks_time10:24:312
538628116,32sleep00-21swapper/010:59:260
35702819,50sleep30-21swapper/310:54:123
302728119,17sleep30-21swapper/309:09:203
302728119,17sleep30-21swapper/309:09:203
1811528119,16sleep30-21swapper/308:14:243
173528117,44sleep11792-21/usr/sbin/munin09:04:581
1518328115,49sleep214696-21apt-get11:29:212
1152428111,55sleep111526-21sshd11:18:581
2501228010,56sleep225065-21ls11:59:262
3168727918,27sleep30-21swapper/308:57:213
2886727927,22sleep30-21swapper/307:04:383
2364627925,28sleep20-21swapper/210:14:292
1926527914,48sleep219272-21/usr/sbin/munin11:39:472
1830827916,47sleep218396-21gpgv11:39:102
2255527828,32sleep10-21swapper/108:29:121
1787927823,24sleep00-21swapper/008:14:130
155722780,6sleep10-21swapper/108:04:401
3194527713,43sleep132098-21latency_hist08:59:131
2907127718,28sleep30-21swapper/312:10:173
207227610,51sleep20-21swapper/209:05:082
2022427630,28sleep20-21swapper/210:04:122
62202757,12sleep20-21swapper/210:59:452
2320027525,25sleep10-21swapper/111:54:121
2102927511,49sleep221178-21latency_hist08:24:122
1663727520,24sleep10-21swapper/111:34:131
1132227518,27sleep30-21swapper/311:16:223
2176627415,44sleep121776-21/usr/sbin/munin10:09:111
3184427317,24sleep10-21swapper/107:14:211
2848127315,41sleep10-21swapper/110:29:271
2746627314,27sleep00-21swapper/008:44:260
1091727325,23sleep00-21swapper/007:49:410
981827213,39sleep29498-21/usr/sbin/munin09:29:352
947727218,40sleep29627-21sed07:44:432
470827219,36sleep04729-21modprobe12:39:100
2717227216,40sleep027205-21awk08:44:090
2717227216,40sleep027205-21awk08:44:090
143262726,56sleep00-21swapper/011:24:510
3201827114,39sleep032155-21uptime12:19:430
2554727115,39sleep025617-21sensors_temp11:59:400
1427727115,41sleep214329-21cut09:44:292
2627627013,40sleep126280-21cron12:04:091
2356727017,27sleep10-21swapper/108:29:451
113792706,52sleep10-21swapper/107:51:201
3054726910,44sleep230555-21acpi08:54:212
149326913,39sleep21517-21/usr/sbin/munin07:19:392
819626815,32sleep28219-21basename11:09:102
38542687,46sleep33901-21ntpq12:34:363
1783526710,14sleep00-21swapper/009:54:400
1395426712,42sleep114035-21latency_hist09:44:111
96426613,38sleep21004-21http09:04:102
96426613,38sleep21004-21http09:04:102
449526612,24sleep00-21swapper/007:29:340
3272326623,26sleep10-21swapper/108:59:411
3272326623,26sleep10-21swapper/108:59:411
271152667,12sleep00-21swapper/012:04:410
122322668,44sleep00-21swapper/007:54:340
736026518,25sleep10-21swapper/109:20:131
3256926521,25sleep10-21swapper/112:24:121
306992657,17sleep20-21swapper/212:16:102
2325126514,35sleep023258-21ntp_states08:29:400
2094526527,21sleep20-21swapper/211:44:402
1938226518,30sleep019404-21idleruntime-cro08:19:100
797226422,25sleep27966-21apt-get09:24:262
434126419,22sleep20-21swapper/210:54:402
354926417,27sleep20-21swapper/209:09:402
354926417,27sleep20-21swapper/209:09:402
3089926413,36sleep130900-21sshd07:10:091
227026414,33sleep02394-21unixbench_singl09:05:100
2138026414,31sleep021458-21/usr/sbin/munin08:24:310
1322826421,29sleep213284-21latency_hist07:59:132
610526318,17sleep20-21swapper/209:19:132
542726322,18sleep10-21swapper/107:34:121
478926311,20sleep20-21swapper/212:39:122
473026313,34sleep24826-21cat07:29:402
3185426314,31sleep231874-21acpi07:14:222
1472826315,24sleep10-21swapper/111:29:121
2980026219,20sleep20-21swapper/210:34:122
2255926211,34sleep121990-21/usr/sbin/munin10:09:401
2186526219,25sleep021884-21apt-get11:49:220
2066326213,36sleep220718-21if_eth010:04:322
1446126213,26sleep10-21swapper/109:44:321
1035826221,21sleep00-21swapper/011:14:290
567826112,33sleep05746-21df07:34:270
2948126117,18sleep00-21swapper/008:49:390
263362618,11sleep10-21swapper/110:23:211
256232615,12sleep00-21swapper/010:19:360
2151726112,37sleep221527-21kernelversion08:24:332
326126017,22sleep10-21swapper/109:09:311
326126017,22sleep10-21swapper/109:09:311
2696226016,27sleep126993-21/usr/sbin/munin10:24:341
2074326014,27sleep10-21swapper/111:44:351
1859026022,19sleep00-21swapper/011:39:130
1652026018,19sleep00-21swapper/011:34:120
1222626021,21sleep10-21swapper/111:19:351
893525920,25sleep18968-21/usr/sbin/munin11:09:531
856025915,28sleep08671-21date07:44:120
2751625912,29sleep10-21swapper/108:44:271
244025911,19sleep00-21swapper/012:29:410
243692595,11sleep10-21swapper/111:55:011
2364125912,19sleep10-21swapper/110:14:281
886825811,22sleep00-21swapper/007:44:280
305102589,25sleep20-21swapper/207:09:462
234825816,26sleep20-21swapper/210:49:312
99282575,40sleep00-21swapper/011:13:280
693825713,26sleep16932-21apt-get11:04:241
3186325715,25sleep131886-21irqstats10:39:321
2882325717,25sleep028962-21/usr/sbin/munin12:09:420
277452575,43sleep10-21swapper/112:08:091
2409525715,27sleep024166-21/usr/sbin/munin10:14:400
1890925718,23sleep10-21swapper/109:59:261
1693725712,20sleep20-21swapper/208:09:342
1183725717,21sleep00-21swapper/009:34:410
1183725717,21sleep00-21swapper/009:34:410
71225617,22sleep1779-21df10:44:251
3125825613,25sleep231254-21apt-get12:19:252
2609825615,19sleep10-21swapper/108:39:301
2609825615,19sleep10-21swapper/108:39:301
1709625615,19sleep00-21swapper/009:54:120
1554425616,21sleep00-21swapper/009:49:110
1005125613,22sleep10-21swapper/107:49:101
668425514,23sleep26727-21irqstats09:19:572
2992725510,32sleep0367-21snmpd07:09:190
2217825518,20sleep00-21swapper/010:09:310
2092125514,25sleep020927-21ntp_states10:04:380
1946125516,17sleep20-21swapper/208:19:122
888525418,17sleep10-21swapper/109:24:451
61625414,26sleep0628-21/usr/sbin/munin12:24:320
46925415,23sleep0577-21latency_hist10:44:120
448825413,20sleep00-21swapper/010:54:420
3003725413,25sleep230030-21apt-get10:34:252
2479625415,16sleep10-21swapper/108:34:371
2298925416,19sleep20-21swapper/208:29:332
2033825415,23sleep020334-21apt-get11:44:210
199325415,19sleep20-21swapper/212:29:302
1316825417,19sleep20-21swapper/211:24:122
311925313,23sleep03166-21users10:49:450
3048525311,28sleep130491-21ntp_states10:34:361
2188825313,23sleep221884-21apt-get11:49:242
2095925312,21sleep00-21swapper/008:24:100
161825315,24sleep20-21swapper/212:29:102
146772537,10sleep20-21swapper/208:03:292
1277625311,18sleep20-21swapper/209:39:322
1080525316,18sleep20-21swapper/207:49:392
869925210,16sleep20-21swapper/211:09:222
701225213,25sleep07018-21df_inode11:04:270
584525212,25sleep15846-21ntp_states10:59:401
35952526,36sleep20-21swapper/207:26:052
306002527,10sleep10-21swapper/112:14:561
253125214,24sleep12589-21sed10:49:341
462525112,21sleep14708-21users10:54:431
2966825112,22sleep029736-21df12:14:260
252932515,34sleep20-21swapper/208:35:152
217625116,19sleep10-21swapper/112:29:331
2100125113,26sleep121028-21latency_hist08:24:111
1801025117,19sleep20-21swapper/209:54:422
1633025116,22sleep016370-21sed08:09:120
130125114,17sleep00-21swapper/007:19:330
1131425114,21sleep211337-21irqstats09:34:312
1131425114,21sleep211337-21irqstats09:34:312
107762515,34sleep00-21swapper/009:34:090
503025010,24sleep15054-21ntp_states09:14:351
2923425012,22sleep129282-21irqstats08:49:321
2901425011,23sleep229008-21apt-get08:49:242
2859125013,23sleep10-21swapper/112:09:381
2811225011,23sleep228108-21apt-get12:09:232
2722525011,22sleep20-21swapper/208:44:112
2722525011,22sleep20-21swapper/208:44:112
257325010,16sleep10-21swapper/107:24:221
1766025012,16sleep00-21swapper/011:35:010
1502225012,22sleep215016-21apt-get08:04:262
732124914,21sleep17327-21df_inode07:39:291
465924912,22sleep04675-21df09:14:260
283272497,15sleep00-21swapper/010:29:140
2877424815,19sleep20-21swapper/208:49:122
1204724812,16sleep10-21swapper/107:54:311
636924711,20sleep05888-21apt-get09:19:150
302462469,17sleep00-21swapper/010:34:310
1985424514,17sleep00-21swapper/010:00:100
1578723910,16sleep10-21swapper/109:49:241
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional