You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-28 - 22:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Mar 28, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
302012376814,3741sleep121502-21kworker/1:007:41:361
29080282413,795sleep229086-21diskstats07:37:382
24455252124,477sleep324600-21grep09:02:553
12727242224,377sleep312878-21head11:52:523
19285239517,315sleep319439-21grep08:47:533
22531230172,208sleep322622-21switchtime08:58:113
27480226916,190sleep327604-21cat09:12:533
11119224711,217sleep311195-21wc08:22:493
19093224175,113sleep319189-21timerandwakeup12:12:583
31052223510,210sleep331139-21mailstats07:42:483
30545222776,84sleep30-21swapper/311:07:453
2444222527,180sleep22464-21latency_hist07:57:202
8153221516,182sleep38293-21grep08:12:523
374221410,189sleep3522-21ls09:32:233
28621220110,172sleep30-21swapper/309:17:413
27360219315,76sleep30-21swapper/310:57:433
29683218211,156sleep329844-21unixbench_multi07:37:513
5878217920,68sleep30-21swapper/309:47:503
2527021749,138sleep022474-21kworker/u8:107:23:140
22522217013,141sleep222599-21fschecks_time10:42:402
21171217051,106sleep321185-21cat07:12:313
21171217051,106sleep321185-21cat07:12:313
6221216623,64sleep36283-21mailstats11:32:493
14717216511,72sleep30-21swapper/310:17:423
24523216283,60sleep30-21swapper/310:47:503
11197216014,90sleep011324-21timerwakeupswit08:22:500
19278215854,44sleep319348-21sshd10:32:363
22867215732,39sleep322883-21modprobe12:27:203
22757215615,76sleep30-21swapper/307:17:323
23049215419,120sleep223069-21cut07:17:432
2263215071,61sleep30-21swapper/309:37:403
20944214925,43sleep321001-21grep10:37:413
1364621468,123sleep313761-21sed08:32:223
17500214348,80sleep317654-21latency_hist10:27:233
94321425,12sleep20-21swapper/209:32:422
536821406,78sleep30-21swapper/311:31:193
1438621369,68sleep314436-21grep10:17:213
8194213313,103sleep38346-21latency_hist09:57:223
294721339,81sleep33002-21if_err_eth007:57:423
6992213121,91sleep27032-21awk11:37:202
25473212833,50sleep00-21swapper/010:52:310
303212748,64sleep30-21swapper/307:47:523
26124212615,89sleep326248-21timerwakeupswit10:52:513
17685212517,88sleep217818-21unixbench_multi08:42:512
16442212551,62sleep316411-21latency_hist12:07:213
16442212551,62sleep316411-21latency_hist12:07:213
621821224,106sleep00-21swapper/009:48:190
18206212120,80sleep118271-21mailstats10:27:491
26560211910,95sleep326688-21unixbench_multi07:27:543
29001211717,82sleep029070-21/usr/sbin/munin07:37:360
29888211510,28sleep329913-21munin-run11:07:213
25310211517,77sleep125400-21timerandwakeup12:32:511
2057821116,18sleep30-21swapper/308:52:493
2913211037,53sleep30-21swapper/311:22:433
25552210936,15sleep30-21swapper/312:32:533
23242210911,81sleep223257-21fschecks_count12:27:392
20568210912,82sleep320694-21grep07:12:233
19987210917,72sleep220137-21head10:32:512
7768210833,13sleep30-21swapper/311:37:493
3726210837,53sleep30-21swapper/309:42:373
19036210816,26sleep30-21swapper/310:32:233
750210736,55sleep30-21swapper/309:32:393
439321076,86sleep30-21swapper/308:02:383
9075210622,27sleep00-21swapper/009:57:520
32238210634,13sleep30-21swapper/311:12:463
25209210612,81sleep125256-21systemd-run10:52:201
1120521069,81sleep311228-21munin-run10:07:203
1120521069,81sleep311228-21munin-run10:07:203
9486210514,70sleep30-21swapper/311:42:543
29485210511,78sleep029533-21users11:02:530
22096210411,77sleep022166-21tr12:22:490
31692210311,75sleep131750-21/usr/sbin/munin09:27:411
31040210319,73sleep30-21swapper/309:25:043
16261210316,65sleep016349-21tune2fs10:22:390
6008210213,65sleep15711-21/usr/sbin/munin11:32:431
32193210219,63sleep032288-21fschecks_time07:47:410
13968210010,25sleep30-21swapper/308:32:383
3038629917,62sleep129902-21apt-get11:07:311
2789429918,60sleep10-21swapper/107:32:481
2787429928,14sleep30-21swapper/307:32:463
2441929811,71sleep30-21swapper/307:22:413
1714329717,61sleep016656-21/usr/sbin/munin12:07:460
16722977,73sleep00-21swapper/007:52:500
21632965,34sleep30-21swapper/307:53:423
210082967,34sleep30-21swapper/312:18:193
503329513,54sleep00-21swapper/009:47:220
1826329510,72sleep318292-21awk08:47:203
2201429414,65sleep122021-21/usr/sbin/munin08:57:401
498129316,56sleep25009-21idleruntime-cro09:47:202
1808329310,70sleep318121-21munin-run12:12:223
1553429313,65sleep215543-21/usr/sbin/munin08:37:382
2785229113,64sleep127859-21kworker/u8:309:14:201
2479129121,49sleep224830-21cut12:32:402
2019129119,59sleep120198-21/usr/sbin/munin08:52:391
106382918,67sleep110700-21if_err_eth011:47:411
1366229010,67sleep013748-21cut11:57:370
1366229010,67sleep013748-21cut11:57:370
897528913,16sleep30-21swapper/308:17:323
897528913,16sleep30-21swapper/308:17:323
1676928950,25sleep316912-21date08:42:243
2200628811,61sleep022013-21diskstats08:57:390
227982868,20sleep322833-21ntpq10:42:453
148302865,46sleep30-21swapper/312:02:083
148302865,46sleep30-21swapper/312:02:083
596028519,46sleep25999-21cut08:07:422
265372859,62sleep326565-21latency_hist09:12:203
1829628515,55sleep018314-21munin-run08:47:210
1195028511,15sleep10-21swapper/110:07:461
1195028511,15sleep10-21swapper/110:07:461
1061528516,53sleep210623-21df_inode08:22:362
516028414,51sleep25227-21users08:02:542
1343128312,58sleep20-21swapper/208:30:152
1066928314,51sleep210810-21awk10:02:512
1066928314,51sleep210810-21awk10:02:512
3054428214,34sleep10-21swapper/107:42:331
236192817,62sleep323653-21ps12:27:483
815128012,17sleep20-21swapper/209:57:212
803128013,49sleep28152-21timerandwakeup08:12:512
2972028023,34sleep00-21swapper/009:22:220
2591028014,51sleep226032-21expr09:07:512
1691728019,44sleep216951-21/usr/sbin/munin12:07:402
16482809,15sleep30-21swapper/311:17:513
1546028017,27sleep00-21swapper/008:37:360
1515028015,48sleep115219-21unin-run08:37:211
58762798,15sleep30-21swapper/308:07:393
2618127920,41sleep00-21swapper/009:07:530
118127912,41sleep00-21swapper/011:17:390
876327818,24sleep10-21swapper/109:57:451
2488127810,58sleep10-21swapper/109:06:041
185727819,38sleep11071-21/usr/sbin/munin11:17:531
104262788,58sleep310445-21apt-get11:47:303
2154027720,27sleep00-21swapper/008:57:230
853627614,17sleep20-21swapper/211:41:402
84742768,13sleep30-21swapper/309:57:393
1898927636,17sleep30-21swapper/307:07:223
1898927636,17sleep30-21swapper/307:07:223
1573627612,48sleep015810-21find12:02:500
1573627612,48sleep015810-21find12:02:500
129232767,20sleep00-21swapper/011:53:010
1289427612,25sleep00-21swapper/008:27:590
3122127523,35sleep231268-21unin-run09:27:212
780027419,25sleep10-21swapper/111:37:511
272827417,41sleep12798-21expr09:37:501
2107927419,23sleep00-21swapper/007:12:280
2107927419,23sleep00-21swapper/007:12:280
892127311,42sleep28367-21/usr/sbin/munin09:57:502
2810827316,38sleep028266-21unixbench_multi07:32:510
1903627220,30sleep00-21swapper/008:47:500
125542726,10sleep30-21swapper/310:09:463
125542726,10sleep30-21swapper/310:09:463
876227120,27sleep10-21swapper/108:17:221
768327111,15sleep00-21swapper/011:37:460
549927114,37sleep15634-21latency_hist11:32:221
3002827115,24sleep00-21swapper/009:22:350
2740327111,45sleep227479-21wc09:12:522
2482127113,36sleep124906-21cut07:22:511
2057027112,14sleep20-21swapper/208:52:482
1917327119,29sleep20-21swapper/208:47:522
394727022,25sleep10-21swapper/111:27:211
1769727013,42sleep017716-21apt-get10:27:320
338126926,22sleep00-21swapper/011:22:520
2839226917,39sleep128489-21date11:02:221
1280226921,23sleep00-21swapper/010:12:220
1280226921,23sleep00-21swapper/010:12:220
1246326914,40sleep212464-21sshd10:08:362
1246326914,40sleep212464-21sshd10:08:362
1021526916,29sleep20-21swapper/211:47:212
630926726,22sleep00-21swapper/011:32:500
1489726714,21sleep20-21swapper/210:17:452
5832665,12sleep20-21swapper/207:48:202
550726613,29sleep20-21swapper/209:47:402
398826620,17sleep20-21swapper/211:27:222
3017726630,14sleep20-21swapper/211:07:242
2355526621,24sleep00-21swapper/012:27:450
231872669,16sleep10-21swapper/107:17:461
138842657,13sleep00-21swapper/008:32:350
899926419,23sleep10-21swapper/111:42:401
2598226424,18sleep10-21swapper/110:52:491
2050526421,28sleep020528-21munin-plugin-st07:12:200
2041826420,26sleep120432-21/usr/sbin/munin12:17:451
1261326412,36sleep212726-21cat11:52:502
3079526310,37sleep030823-21/usr/sbin/munin11:07:500
2257926321,23sleep20-21swapper/207:17:232
2216726313,37sleep122251-21sed12:22:501
2169426324,25sleep221583-21/usr/sbin/munin12:22:382
964526215,24sleep20-21swapper/208:17:522
964526215,24sleep20-21swapper/208:17:522
942526217,25sleep19454-21ntp_states08:17:461
942526217,25sleep19454-21ntp_states08:17:461
77726217,19sleep10-21swapper/111:17:201
2278926218,23sleep00-21swapper/007:17:370
2061926223,19sleep10-21swapper/110:37:221
1974426219,21sleep00-21swapper/012:17:190
899126115,22sleep00-21swapper/011:42:390
428226120,22sleep14349-21switchtime09:42:511
371326111,37sleep113441-21snmpd07:58:201
34026118,18sleep20-21swapper/209:32:222
3114126115,28sleep00-21swapper/007:42:490
2619526110,13sleep10-21swapper/109:07:541
2136426119,18sleep10-21swapper/110:37:511
2088626116,27sleep00-21swapper/010:37:370
205452619,12sleep20-21swapper/212:17:502
1826526115,20sleep10-21swapper/112:12:241
1091326121,20sleep10-21swapper/108:22:431
53926016,29sleep1548-21vmstat07:47:541
2525726020,23sleep225314-21sed10:52:212
244826016,18sleep20-21swapper/209:37:432
1935126016,27sleep019358-21/usr/sbin/munin10:32:370
1826126026,17sleep10-21swapper/107:02:481
1826126026,17sleep10-21swapper/107:02:481
1744926017,27sleep117459-21ntp_states08:42:461
1581326017,20sleep10-21swapper/112:02:511
1581326017,20sleep10-21swapper/112:02:511
3121725918,25sleep10-21swapper/109:27:201
2628025916,25sleep126320-21grep07:27:471
2461625916,20sleep00-21swapper/010:47:510
2434025920,18sleep20-21swapper/209:02:542
240125920,15sleep20-21swapper/211:22:222
1522025912,33sleep215280-21latency_hist08:37:222
1296325920,20sleep10-21swapper/108:28:001
937925813,25sleep00-21swapper/008:17:450
937925813,25sleep00-21swapper/008:17:450
870225819,21sleep00-21swapper/008:17:210
564425812,19sleep10-21swapper/109:47:431
2033525814,30sleep00-21swapper/012:17:440
1910525816,23sleep10-21swapper/108:47:511
1374925817,24sleep10-21swapper/111:57:381
1374925817,24sleep10-21swapper/111:57:381
2820725718,21sleep20-21swapper/211:00:372
268682575,11sleep00-21swapper/007:28:510
2680425715,25sleep026824-21apt-get09:12:330
266025714,25sleep00-21swapper/009:37:490
2558725711,30sleep225604-21ls07:27:202
2449725719,16sleep10-21swapper/112:32:231
1897025719,23sleep218988-21munin-run07:07:212
1897025719,23sleep218988-21munin-run07:07:212
180542579,30sleep018098-21ntpq07:02:430
180542579,30sleep018098-21ntpq07:02:430
1364925716,20sleep00-21swapper/010:12:500
101425715,25sleep01050-21ntp_states09:32:440
500725614,29sleep15158-21unixbench_multi08:02:521
3150925618,19sleep00-21swapper/009:27:350
2754225620,16sleep00-21swapper/010:57:490
2372325614,26sleep023716-21apt-get09:02:360
2326225615,21sleep20-21swapper/209:02:222
204662564,10sleep20-21swapper/207:11:492
2005325614,22sleep120150-21users07:07:531
1657525615,27sleep216621-21ntp_offset10:22:452
1582925616,20sleep10-21swapper/108:37:451
1035525614,27sleep010400-21ntp_states10:02:450
1035525614,27sleep010400-21ntp_states10:02:450
594525510,25sleep00-21swapper/008:07:400
3188625519,19sleep20-21swapper/207:47:232
283925513,18sleep00-21swapper/007:57:390
2742125515,23sleep227418-21apt-get07:32:332
2237525513,24sleep122394-21apt-get10:42:311
2127225515,23sleep121278-21df_inode07:12:371
2127225515,23sleep121278-21df_inode07:12:371
1855625516,19sleep10-21swapper/112:12:401
1701625513,23sleep117077-21users10:22:531
1444125513,25sleep014588-21date10:17:220
1351125519,20sleep20-21swapper/210:12:482
1312725512,27sleep113187-21/usr/sbin/munin10:12:391
2922125416,21sleep229313-21users09:17:522
2868725415,23sleep128780-21memory09:17:431
2683725413,25sleep226985-21latency_hist10:57:222
1839225414,24sleep217953-21apt-get12:12:252
1795925414,23sleep20-21swapper/210:27:422
151342549,26sleep20-21swapper/212:02:362
151342549,26sleep20-21swapper/212:02:362
70672539,17sleep00-21swapper/009:52:430
58722539,18sleep10-21swapper/108:07:371
464325312,18sleep20-21swapper/211:27:472
3204525312,18sleep00-21swapper/011:12:390
2085625311,17sleep20-21swapper/210:37:352
1357425313,23sleep213570-21apt-get11:57:312
1357425313,23sleep213570-21apt-get11:57:312
1224325316,19sleep10-21swapper/111:52:411
1002725314,22sleep110032-21diskstats10:02:371
1002725314,22sleep110032-21diskstats10:02:371
67825212,22sleep1682-21diskstats09:32:371
1513725212,18sleep10-21swapper/110:17:521
418525113,21sleep04181-21apt-get11:27:330
2885025113,22sleep128899-21irqstats11:02:421
319652498,15sleep20-21swapper/211:12:372
1176424913,15sleep20-21swapper/211:52:222
74932489,16sleep10-21swapper/109:52:521
312824816,16sleep20-21swapper/207:57:452
2614224814,15sleep20-21swapper/207:27:442
2491624610,20sleep024973-21irqstats12:32:420
2248324613,17sleep00-21swapper/010:42:380
1122524511,18sleep00-21swapper/011:47:520
3041824416,15sleep20-21swapper/211:07:362
2091624416,15sleep20-21swapper/207:12:262
477424311,17sleep00-21swapper/008:02:470
11852408,17sleep10-21swapper/107:52:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional