You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-18 - 16:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sat Apr 18, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
175112617614,6149sleep15183-21kworker/1:212:27:011
1437229028,881sleep314408-21ntpq08:51:323
18094276316,729sleep018107-21/usr/sbin/munin10:46:330
18094276316,729sleep018107-21/usr/sbin/munin10:46:330
20965271211,684sleep021018-21/usr/sbin/munin10:56:270
1743252517,490sleep11814-21true09:56:061
1421223650,6sleep00-21swapper/012:16:370
1421223650,6sleep00-21swapper/012:16:370
19754226252,167sleep319833-21sort10:51:373
15703226067,59sleep30-21swapper/312:21:383
14503225775,125sleep314656-21find07:11:093
260242249101,43sleep30-21swapper/311:11:363
5744222768,134sleep35850-21fschecks_time11:51:253
5744222768,134sleep35850-21fschecks_time11:51:253
27230220376,67sleep327247-21apt07:51:073
22986219773,50sleep30-21swapper/311:01:393
5409219411,167sleep25469-21latency_hist11:51:082
32698219112,117sleep30-21swapper/309:51:083
30416218947,23sleep30-21swapper/308:01:083
30416218947,23sleep30-21swapper/308:01:083
22653218449,16sleep30-21swapper/307:36:093
3064217345,72sleep32853-21/usr/sbin/munin08:16:263
21055217247,113sleep321121-21date07:31:083
21055217247,113sleep321121-21date07:31:083
12870217041,52sleep30-21swapper/307:06:083
30123216715,93sleep330156-21latency09:41:393
808216510,139sleep1954-21latency_hist11:36:091
808216510,139sleep1954-21latency_hist11:36:091
28545216014,133sleep228520-21latency_hist11:21:082
28545216014,133sleep228520-21latency_hist11:21:082
470521529,127sleep34816-21timerandwakeup11:46:373
1262321519,21sleep30-21swapper/308:46:273
18733214911,124sleep118778-21awk12:31:331
2368821467,12sleep10-21swapper/109:21:221
844721455,104sleep30-21swapper/312:00:343
2045921459,52sleep320469-21acpi09:11:183
18689214315,72sleep318717-21apt-key09:06:093
3396214116,105sleep30-21swapper/310:01:093
18905213812,111sleep118414-21apt-get09:06:161
25251213710,115sleep325284-21grep09:26:253
2562213211,76sleep32564-21cron08:16:063
907621307,14sleep30-21swapper/312:01:313
27915212817,45sleep30-21swapper/311:16:373
27915212817,45sleep30-21swapper/311:16:373
23702212610,58sleep30-21swapper/309:21:243
14055212612,75sleep30-21swapper/310:34:013
16915212549,25sleep30-21swapper/309:01:093
12004212424,80sleep112041-21ps07:01:351
443021217,59sleep34500-21df08:21:233
10121211836,25sleep30-21swapper/310:21:293
8867211732,29sleep39001-21cut08:36:093
8867211732,29sleep39001-21cut08:36:093
31481211610,78sleep013441-21snmpd11:29:420
31481211610,78sleep013441-21snmpd11:29:420
28014211644,23sleep30-21swapper/307:51:353
23959211619,31sleep20-21swapper/211:06:262
23959211619,31sleep20-21swapper/211:06:262
11669211510,49sleep30-21swapper/310:26:293
10360211530,23sleep30-21swapper/308:41:063
1863121149,63sleep30-21swapper/307:21:353
1863121149,63sleep30-21swapper/307:21:353
4819211311,79sleep04961-21unixbench_singl11:46:380
24182211246,22sleep30-21swapper/307:41:093
11652211248,17sleep30-21swapper/308:42:473
11652211248,17sleep30-21swapper/308:42:473
478921117,70sleep313441-21snmpd10:05:013
1887221110,6sleep00-21swapper/009:06:090
32468211046,49sleep332488-21sed08:06:293
5556210910,80sleep35592-21ntpq10:06:313
5556210910,80sleep35592-21ntpq10:06:313
1062210936,51sleep30-21swapper/309:51:363
68521070,7sleep20-21swapper/209:51:262
5919210713,18sleep20-21swapper/211:51:282
5919210713,18sleep20-21swapper/211:51:282
25852210714,78sleep025905-21if_eth011:11:330
3007721060,6sleep00-21swapper/011:26:050
3007721060,6sleep00-21swapper/011:26:050
22062210629,64sleep322065-21apt-get09:16:193
6996210514,19sleep30-21swapper/308:28:013
18633210527,56sleep30-21swapper/312:31:313
16668210422,67sleep316758-21sort12:26:263
26716210350,42sleep326712-21apt-get09:31:213
12151210326,52sleep212170-21grep08:46:062
12151210326,52sleep212170-21grep08:46:062
3860210215,30sleep33984-21latency_hist11:46:083
32411210213,71sleep232467-21interrupts08:06:282
21902210211,78sleep021993-21timerandwakeup07:31:370
7796210118,59sleep07815-21sed08:31:300
7796210118,59sleep07815-21sed08:31:300
5645210126,35sleep00-21swapper/011:51:180
5645210126,35sleep00-21swapper/011:51:180
512210120,56sleep132177-21/usr/sbin/munin08:06:391
2826521009,12sleep20-21swapper/209:36:172
24755210039,44sleep30-21swapper/311:06:393
24755210039,44sleep30-21swapper/311:06:393
23878210010,22sleep20-21swapper/209:21:272
17901210023,62sleep30-21swapper/310:46:283
17901210023,62sleep30-21swapper/310:46:283
1848429814,68sleep018537-21cut12:31:280
1740829813,68sleep017500-21/usr/sbin/munin09:01:300
2138429622,49sleep121471-21fschecks_time07:31:261
857129520,13sleep30-21swapper/310:16:283
2106129516,27sleep20-21swapper/209:11:382
1086429521,12sleep30-21swapper/312:06:363
1086229511,23sleep00-21swapper/008:41:190
1086229511,23sleep00-21swapper/008:41:190
798529424,47sleep18030-21grep10:16:061
719129426,50sleep30-21swapper/310:11:353
3093429428,48sleep30-21swapper/311:26:343
3093429428,48sleep30-21swapper/311:26:343
2717329418,56sleep127180-21ntp_states09:31:351
166252949,73sleep316635-21find07:16:183
29752939,68sleep23063-21fschecks_time08:16:252
2728329219,59sleep227392-21sed07:51:102
2254129214,62sleep122582-21cut09:16:371
1557329211,15sleep20-21swapper/207:11:372
3236329112,18sleep30-21swapper/311:31:353
3236329112,18sleep30-21swapper/311:31:353
3077629113,22sleep330852-21fschecks_time08:01:283
3077629113,22sleep330852-21fschecks_time08:01:283
2931929117,22sleep30-21swapper/311:21:373
2931929117,22sleep30-21swapper/311:21:373
2087729122,13sleep30-21swapper/310:56:263
1453529117,12sleep30-21swapper/310:36:193
1928529013,52sleep119368-21fschecks_time10:51:271
1408829028,47sleep30-21swapper/312:16:373
1408829028,47sleep30-21swapper/312:16:373
1224829018,52sleep112292-21ntpq12:11:301
1224829018,52sleep112292-21ntpq12:11:301
2507928812,60sleep025197-21wc07:41:390
2207528718,44sleep022065-21apt-get09:16:240
1662428715,13sleep30-21swapper/310:41:363
956028619,52sleep19577-21idleruntime-cro10:21:061
373828614,13sleep00-21swapper/011:45:120
2525228423,32sleep00-21swapper/011:11:080
2525228423,32sleep00-21swapper/011:11:080
171128418,48sleep11856-21unixbench_singl11:36:371
171128418,48sleep11856-21unixbench_singl11:36:371
1656228439,20sleep20-21swapper/210:41:352
1623828414,55sleep116360-21cat08:56:391
440228312,57sleep24423-21apt-get08:21:182
2540228313,17sleep10-21swapper/111:11:091
171342838,16sleep30-21swapper/309:01:213
253272825,65sleep30-21swapper/307:41:423
1584728219,27sleep00-21swapper/010:41:080
2336128027,27sleep20-21swapper/207:36:362
646927920,31sleep00-21swapper/010:11:080
646927920,31sleep00-21swapper/010:11:080
421427917,44sleep04294-21awk10:01:350
3264727916,42sleep132649-21cron09:51:061
1654627915,33sleep10-21swapper/110:41:341
1478327912,53sleep114838-21latency_hist12:21:071
1478327912,53sleep114838-21latency_hist12:21:071
1204427926,33sleep20-21swapper/207:01:362
1514827815,15sleep30-21swapper/307:11:253
1469327821,37sleep214849-21grep08:51:392
2214027714,40sleep221270-21/usr/sbin/munin07:31:392
671727612,47sleep26724-21grep10:11:222
3249427618,34sleep00-21swapper/008:06:300
1820127620,31sleep00-21swapper/007:21:240
1820127620,31sleep00-21swapper/007:21:240
75282748,53sleep10-21swapper/108:31:221
75282748,53sleep10-21swapper/108:31:221
637727420,35sleep06414-21ntp_kernel_pll_08:26:330
2763827413,46sleep127683-21ntp_offset11:16:301
2763827413,46sleep127683-21ntp_offset11:16:301
1038627421,30sleep010435-21rm08:41:070
251982738,16sleep10-21swapper/107:41:401
2100527324,30sleep10-21swapper/107:31:061
2100527324,30sleep10-21swapper/107:31:061
296527222,28sleep12853-21/usr/sbin/munin08:16:241
1983527221,25sleep00-21swapper/010:51:380
195352726,11sleep20-21swapper/212:35:012
1379527214,39sleep013911-21ls07:06:380
1196627217,41sleep012037-21fschecks_time12:11:250
1196627217,41sleep012037-21fschecks_time12:11:250
2828327119,12sleep30-21swapper/309:36:183
2737227121,31sleep00-21swapper/011:16:250
2737227121,31sleep00-21swapper/011:16:250
23102719,19sleep30-21swapper/311:41:083
23102719,19sleep30-21swapper/311:41:083
1889227125,27sleep10-21swapper/110:51:071
1889227125,27sleep10-21swapper/110:51:071
1656927113,37sleep00-21swapper/012:26:190
1443927120,29sleep20-21swapper/207:11:082
1316827115,26sleep10-21swapper/110:31:281
1195227116,40sleep211959-21df_inode12:11:232
1195227116,40sleep211959-21df_inode12:11:232
2810327023,23sleep20-21swapper/209:36:092
1526127015,33sleep115296-21munin-run08:56:061
1187827010,17sleep30-21swapper/312:11:203
1187827010,17sleep30-21swapper/312:11:203
3105126911,14sleep00-21swapper/008:01:330
3105126911,14sleep00-21swapper/008:01:330
166526920,21sleep10-21swapper/108:11:321
1021526920,26sleep00-21swapper/010:21:300
849026818,34sleep28503-21sh12:01:052
648726818,22sleep10-21swapper/110:11:091
648726818,22sleep10-21swapper/110:11:091
3129226818,23sleep20-21swapper/208:01:392
3129226818,23sleep20-21swapper/208:01:392
1633326816,36sleep016394-21sendmail-msp07:16:070
1463526820,23sleep20-21swapper/210:36:262
975226711,42sleep19754-21sshd12:02:261
1600626711,44sleep20-21swapper/212:22:332
135602675,11sleep10-21swapper/108:49:111
68426610,40sleep2754-21systemd-run11:36:062
68426610,40sleep2754-21systemd-run11:36:062
2837226612,38sleep128377-21diskstats09:36:241
1782926622,26sleep217869-21http12:31:052
1074726618,31sleep210760-21/usr/sbin/munin10:21:402
498226523,23sleep10-21swapper/108:21:371
45026518,27sleep00-21swapper/009:51:170
3121026518,32sleep131289-21switchtime08:01:381
3121026518,32sleep131289-21switchtime08:01:381
264326515,22sleep20-21swapper/208:16:092
1542326519,25sleep10-21swapper/112:21:311
3069826425,22sleep10-21swapper/111:26:281
3069826425,22sleep10-21swapper/111:26:281
2024826418,21sleep20-21swapper/209:11:092
1548926418,26sleep00-21swapper/012:21:340
1055226418,25sleep210750-21sed08:41:092
904326322,19sleep00-21swapper/010:16:370
3016326310,36sleep00-21swapper/009:41:400
2909726316,25sleep20-21swapper/207:56:232
2909726316,25sleep20-21swapper/207:56:232
2797726318,26sleep00-21swapper/009:36:070
2562826312,26sleep00-21swapper/009:26:340
239842636,11sleep00-21swapper/007:39:120
1795326317,24sleep10-21swapper/107:21:081
1088326321,24sleep110880-21apt-get08:41:201
1088326321,24sleep110880-21apt-get08:41:201
698726217,20sleep10-21swapper/111:56:081
698726217,20sleep10-21swapper/111:56:081
418626216,29sleep04235-21latency_hist08:21:080
1322526219,22sleep20-21swapper/210:31:292
1003326213,32sleep110091-21/usr/sbin/munin10:21:271
78362615,11sleep00-21swapper/010:14:110
358826117,20sleep10-21swapper/110:01:151
314722615,14sleep20-21swapper/209:46:262
29962617,10sleep10-21swapper/109:56:591
2021926113,27sleep00-21swapper/007:26:370
2021926113,27sleep00-21swapper/007:26:370
1615426116,24sleep00-21swapper/008:56:380
1041726122,20sleep10-21swapper/112:06:251
93742607,15sleep10-21swapper/108:36:271
915026014,24sleep00-21swapper/012:01:320
723626017,28sleep17262-21awk08:31:061
332726014,31sleep03366-21modprobe10:01:060
2462026012,15sleep20-21swapper/207:41:292
1665526019,19sleep00-21swapper/007:16:230
1308726017,26sleep013167-21/usr/sbin/munin10:31:270
749325919,18sleep10-21swapper/111:56:281
1798425919,18sleep20-21swapper/207:21:092
134832597,11sleep20-21swapper/208:48:122
2846125811,25sleep10-21swapper/107:52:011
160825812,30sleep01710-21timerandwakeup11:36:360
160825812,30sleep01710-21timerandwakeup11:36:360
922825717,26sleep29238-21ntp_states12:01:342
789525719,21sleep20-21swapper/208:31:322
789525719,21sleep20-21swapper/208:31:322
776825715,18sleep00-21swapper/011:56:350
573825716,24sleep00-21swapper/008:26:080
2606225712,31sleep026145-21fschecks_time07:46:270
2386125714,27sleep123855-21apt-get11:06:211
2386125714,27sleep123855-21apt-get11:06:211
2362325713,29sleep023681-21latency_hist11:06:080
1391325713,21sleep10-21swapper/107:06:391
1162525714,19sleep00-21swapper/007:01:260
2231225615,23sleep122307-21apt-get11:01:201
2056125614,19sleep20-21swapper/210:56:082
337125513,22sleep20-21swapper/210:01:082
2985725513,26sleep129924-21df09:41:331
2565125513,30sleep225719-21/usr/sbin/munin11:11:232
248452555,10sleep00-21swapper/009:25:410
1731925518,19sleep20-21swapper/209:01:282
1608425515,24sleep016078-21apt-get10:41:200
1140125514,25sleep011392-21apt-get10:26:220
663725415,19sleep20-21swapper/208:26:392
2523625415,23sleep125243-21df_inode09:26:231
2073125414,24sleep220733-21/usr/sbin/munin10:56:162
1568425413,23sleep215786-21perl08:56:272
1033025412,24sleep010324-21apt-get12:06:200
76525317,18sleep00-21swapper/011:36:080
76525317,18sleep00-21swapper/011:36:080
2911325316,20sleep129153-21memory11:21:311
2911325316,20sleep129153-21memory11:21:311
2275625314,22sleep222767-21ntp_states11:01:332
2228525314,20sleep20-21swapper/209:16:302
2128825312,16sleep00-21swapper/009:11:400
1827125310,27sleep20-21swapper/207:21:262
1827125310,27sleep20-21swapper/207:21:262
1587625312,17sleep10-21swapper/107:11:401
50142526,34sleep10-21swapper/111:46:431
361525211,17sleep20-21swapper/210:01:212
2992525211,22sleep229934-21df_inode09:41:342
2020025213,24sleep220205-21ntp_states07:26:352
2020025213,24sleep220205-21ntp_states07:26:352
1723225213,17sleep10-21swapper/109:01:271
2847725116,21sleep028539-21cat09:36:270
181642518,32sleep10-21swapper/107:21:121
181642518,32sleep10-21swapper/107:21:121
162525110,15sleep00-21swapper/008:11:310
1481125113,22sleep114904-21memory10:36:291
1407525113,23sleep014084-21df_inode08:51:240
1355225113,22sleep213558-21diskstats12:16:242
1355225113,22sleep213558-21diskstats12:16:242
837725012,23sleep28383-21diskstats10:16:232
300625011,23sleep23012-21ntp_states11:41:352
2809725011,23sleep028189-21sed07:51:360
2119925014,22sleep121210-21ntp_states10:56:321
1300325015,17sleep00-21swapper/008:46:360
2907024912,23sleep10-21swapper/107:56:171
2907024912,23sleep10-21swapper/107:56:171
25262499,16sleep10-21swapper/111:41:181
185924911,21sleep21902-21users11:36:392
185924911,21sleep21902-21users11:36:392
1770624913,22sleep217712-21df_inode10:46:232
1770624913,22sleep217712-21df_inode10:46:232
1147324911,22sleep211480-21df_inode10:26:242
303272488,26sleep20-21swapper/211:26:172
303272488,26sleep20-21swapper/211:26:172
2933824811,21sleep029367-21irqstats07:56:290
2933824811,21sleep029367-21irqstats07:56:290
2646924815,19sleep226479-21unin-run09:31:082
2614724815,14sleep10-21swapper/107:46:281
1187024813,17sleep10-21swapper/110:26:361
315172479,15sleep00-21swapper/009:46:290
2963324610,20sleep229686-21users11:21:402
2963324610,20sleep229686-21users11:21:402
228472466,16sleep10-21swapper/107:36:191
2188224611,17sleep10-21swapper/109:16:091
2751024410,19sleep227513-21apt-get07:51:182
590824310,14sleep20-21swapper/210:06:382
590824310,14sleep20-21swapper/210:06:382
105082419,18sleep20-21swapper/212:06:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional