You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-16 - 06:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sun Aug 16, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
283532297113,2945sleep328250-21kworker/3:223:07:513
797222998,2274sleep119224-1kworker/1:2H21:40:101
22426279627,736sleep122462-21ntpq00:30:111
25551246715,437sleep025618-21/usr/sbin/munin19:35:020
176112451144,221sleep117676-21awk19:10:151
433623578,338sleep14344-21ntpq21:50:111
433623578,338sleep14344-21ntpq21:50:111
326252351144,192sleep10-21swapper/123:20:131
238352320146,16sleep10-21swapper/122:54:451
119222301134,151sleep111985-21latency22:15:071
18714229863,185sleep118766-21cpuspeed20:54:591
23710228912,261sleep023759-21unin-run19:29:450
15539228826,243sleep215606-21df19:05:002
11605227720,231sleep111747-21unixbench_singl23:55:171
207682276104,25sleep10-21swapper/100:25:061
19738225679,56sleep10-21swapper/122:40:071
20763225264,161sleep10-21swapper/119:20:041
5341224611,218sleep25342-21sendmail-mta23:35:402
12462224516,211sleep112513-21cpuspeed00:00:021
23553223143,146sleep10-21swapper/100:34:551
23553223143,146sleep10-21swapper/100:34:551
3644222931,48sleep13783-21latency_hist21:49:471
3644222931,48sleep13783-21latency_hist21:49:471
14496220819,162sleep1231-21systemd-journal22:24:441
8214220534,128sleep18315-21timerandwakeup20:20:141
25026220225,70sleep10-21swapper/121:15:021
25026220225,70sleep10-21swapper/121:15:021
28651220112,171sleep128855-21latency_hist23:09:471
26148219171,27sleep10-21swapper/123:00:101
6459219018,156sleep15959-21/usr/sbin/munin23:40:151
30545218717,39sleep10-21swapper/119:50:051
20348218613,156sleep120395-21sendmail19:19:441
30242218427,99sleep130278-21ntpq21:30:121
28698218411,29sleep128292-21/usr/sbin/munin21:25:161
14573218317,139sleep10-21swapper/120:40:331
15985218219,102sleep10-21swapper/120:45:141
16261217722,128sleep00-21swapper/020:45:210
31068217619,18sleep10-21swapper/123:15:161
17115217411,145sleep017230-21latency_hist00:14:470
24945217016,139sleep325083-21sed00:39:463
24945217016,139sleep325083-21sed00:39:463
1722021686,139sleep00-21swapper/019:10:060
24856216532,72sleep124997-21grep22:55:211
3204215914,124sleep10-21swapper/123:30:101
13317215910,17sleep00-21swapper/022:20:030
2825821589,138sleep1367-21snmpd19:43:101
8294215713,107sleep18371-21mailstats23:45:321
8294215713,107sleep18371-21mailstats23:45:321
21981215613,125sleep221944-21latency_hist19:24:462
31688215513,38sleep10-21swapper/121:35:091
2055121548,112sleep120565-21kernelversion21:00:081
12013215419,28sleep00-21swapper/020:34:460
31418215017,23sleep20-21swapper/221:35:022
9763214610,97sleep19132-21/usr/sbin/munin20:25:191
1011121398,95sleep110101-21apt-get22:10:021
9798213421,95sleep39855-21find23:50:143
16861212946,59sleep116916-21cyclictest19:09:461
1743821279,82sleep117458-21grep00:15:041
19761212618,86sleep319892-21grep19:15:183
23864212421,44sleep10-21swapper/121:10:131
1660212410,77sleep10-21swapper/123:25:091
2928521238,106sleep3201-21jbd2/mmcblk2p2-21:25:583
30152212119,28sleep10-21swapper/123:12:581
986721207,54sleep19953-21cat22:09:461
1316211913,67sleep11354-21ls20:00:061
31716211724,73sleep031751-21modprobe19:54:440
18477211714,67sleep118515-21sed20:54:471
170321140,10sleep20-21swapper/223:25:092
32675211320,71sleep032738-21sendmail-msp21:39:440
5984211011,81sleep05985-21/usr/sbin/munin23:40:020
23760210913,85sleep123825-21date19:29:461
1716721090,12sleep30-21swapper/322:30:193
3239321088,15sleep30-21swapper/323:20:063
3198321078,38sleep10-21swapper/119:54:581
31263210720,68sleep031396-21sed23:15:190
444821069,39sleep14467-21apt-get20:09:561
13764210611,77sleep013851-21gpgv00:04:450
7236210524,57sleep27268-21cut23:44:442
16092210412,78sleep216126-21latency_hist22:29:462
12471210422,17sleep10-21swapper/120:35:081
3465210310,59sleep13541-21systemctl20:05:401
30341210314,73sleep00-21swapper/023:14:470
19973210220,63sleep2231-21systemd-journal20:59:442
5995210115,34sleep10-21swapper/120:14:571
27388210111,34sleep10-21swapper/121:20:191
15316210110,52sleep10-21swapper/122:25:131
2947210010,43sleep20-21swapper/220:05:162
28528210014,71sleep017687-1kworker/0:2H19:44:470
609329921,52sleep10-21swapper/121:55:161
609329921,52sleep10-21swapper/121:55:161
147072996,34sleep10-21swapper/100:05:151
167882989,73sleep10-21swapper/122:30:131
239762978,19sleep10-21swapper/119:29:581
239762978,19sleep10-21swapper/119:29:581
1550529723,43sleep215522-21apt00:09:442
2227129511,68sleep122310-21modprobe22:49:441
1300029515,64sleep013118-21cut22:19:460
94042947,29sleep10-21swapper/123:50:041
2183829427,30sleep10-21swapper/121:04:561
1879729418,54sleep018811-21grep20:55:020
698129211,22sleep30-21swapper/321:59:593
2871429222,30sleep10-21swapper/119:44:481
2404929215,57sleep224090-21processes00:35:122
2404929215,57sleep224090-21processes00:35:122
2122329212,67sleep021257-21if_err_eth022:45:070
1851629222,29sleep20-21swapper/220:54:482
1058029218,58sleep010640-21processes22:10:170
77429130,41sleep3771-21apt-get21:39:553
3216729112,58sleep20-21swapper/223:20:012
2965029114,58sleep229000-21/usr/sbin/munin23:10:252
221182918,14sleep00-21swapper/021:05:070
2554628910,15sleep10-21swapper/119:34:591
2720928822,50sleep327301-21tune2fs19:40:053
1891728811,42sleep118927-21find00:19:571
1406628819,50sleep314051-21apt-get20:40:033
924428718,52sleep29306-21fschecks_time20:25:072
1943028713,59sleep319504-21mailstats00:20:153
1834228618,47sleep317867-21/usr/sbin/munin22:35:123
1057228621,48sleep010581-21modprobe23:54:440
1019828512,60sleep310257-21fschecks_count22:10:083
226382849,13sleep30-21swapper/322:50:023
2108828416,46sleep021170-21cat00:25:130
1931428410,59sleep319389-21find20:55:133
1095028420,47sleep011011-21fschecks_time23:55:040
1555828317,46sleep015527-21latency_hist00:09:460
2792428220,39sleep028070-21latency_hist21:24:460
2448528229,31sleep20-21swapper/219:30:152
2448528229,31sleep20-21swapper/219:30:152
3124428119,44sleep231359-21/usr/sbin/munin19:50:182
3085428113,15sleep30-21swapper/319:50:113
1914128119,29sleep10-21swapper/122:39:381
1683628122,26sleep10-21swapper/100:11:531
794428019,28sleep10-21swapper/122:00:411
680328023,38sleep36714-21latency_hist21:59:463
44032807,32sleep10-21swapper/123:34:581
1484028023,40sleep014949-21sed20:40:360
1093728010,30sleep10-21swapper/120:30:071
289862796,55sleep00-21swapper/019:45:030
2451027915,30sleep20-21swapper/222:55:142
432327812,50sleep04335-21ntpq21:50:100
432327812,50sleep04335-21ntpq21:50:100
1490627816,29sleep00-21swapper/022:25:030
81427621,36sleep3847-21munin-run19:59:443
440527615,17sleep30-21swapper/323:35:003
2888427622,29sleep20-21swapper/219:44:572
262152768,10sleep20-21swapper/223:00:112
2340427616,15sleep00-21swapper/000:34:460
2340427616,15sleep00-21swapper/000:34:460
728727411,48sleep06982-21/usr/sbin/munin22:00:120
56392748,16sleep30-21swapper/321:55:063
424627412,48sleep24277-21nfs_client21:50:082
424627412,48sleep24277-21nfs_client21:50:082
88627328,25sleep00-21swapper/021:40:170
2576827316,36sleep025810-21fschecks_count23:00:010
2147627323,25sleep30-21swapper/322:45:143
2076627317,39sleep020797-21sed22:44:460
1333527316,36sleep20-21swapper/222:20:052
2318227222,20sleep30-21swapper/321:09:473
1767927212,19sleep20-21swapper/219:10:162
1079527211,28sleep30-21swapper/320:30:053
2883227111,27sleep20-21swapper/221:25:212
2381827120,30sleep023831-21apt22:54:440
2299627118,22sleep20-21swapper/219:25:432
2211827113,44sleep322011-21/usr/sbin/munin00:30:043
676027021,26sleep20-21swapper/221:59:452
676027021,26sleep20-21swapper/221:59:452
317627015,40sleep03203-21awk23:30:090
2855727019,22sleep00-21swapper/023:09:460
261252705,55sleep00-21swapper/021:18:320
261252705,55sleep00-21swapper/021:18:320
1402927014,41sleep213525-21apt-get20:39:532
943826923,26sleep20-21swapper/223:50:052
749226913,29sleep20-21swapper/222:00:182
2594726923,29sleep30-21swapper/323:00:043
256262697,44sleep20-21swapper/219:35:042
22442696,11sleep20-21swapper/220:02:462
97026817,34sleep0990-21modprobe23:24:440
2764426814,15sleep20-21swapper/223:05:122
2121926824,27sleep30-21swapper/319:20:153
185762689,16sleep00-21swapper/022:35:170
128526813,39sleep21352-21run-parts21:40:272
874026711,42sleep08791-21cat22:05:040
2904026719,28sleep329019-21apt-get23:10:053
2256026718,31sleep222557-21apt-get22:49:572
2241926726,23sleep30-21swapper/319:24:553
2096626711,40sleep221090-21sed21:00:172
1687326721,23sleep00-21swapper/022:30:160
1582526722,22sleep20-21swapper/220:45:102
1097326713,23sleep20-21swapper/220:30:082
652026614,22sleep30-21swapper/320:15:153
457126615,29sleep30-21swapper/320:10:063
293126614,34sleep02915-21apt-get20:05:060
2402326612,27sleep00-21swapper/021:10:160
194062669,12sleep20-21swapper/200:20:142
1774126619,25sleep00-21swapper/020:50:130
1373326619,23sleep30-21swapper/322:20:143
43792659,13sleep20-21swapper/220:09:492
1221126519,21sleep20-21swapper/223:59:462
625326413,32sleep36342-21/usr/sbin/munin23:40:093
2503826415,25sleep30-21swapper/321:15:043
2503826415,25sleep30-21swapper/321:15:043
247726413,33sleep32541-21/usr/sbin/munin21:45:043
247726413,33sleep32541-21/usr/sbin/munin21:45:043
1994926415,27sleep30-21swapper/322:40:133
1914726420,28sleep219193-21systemctl22:39:432
1615026423,19sleep00-21swapper/019:05:140
126326416,30sleep01330-21/usr/sbin/munin23:25:000
1038026414,34sleep210405-21kernelversion22:10:112
2712326319,27sleep227117-21apt-get19:40:002
2072226315,32sleep220755-21awk22:44:442
193526323,20sleep30-21swapper/323:25:153
1784126311,33sleep317923-21mailstats00:15:143
1367526318,27sleep00-21swapper/020:39:470
1286326314,33sleep212932-21timerandwakeup20:35:172
924126211,24sleep00-21swapper/023:49:550
924126211,24sleep00-21swapper/023:49:550
831926216,27sleep20-21swapper/220:20:152
2395726217,27sleep324018-21/usr/sbin/munin21:10:153
2379826220,22sleep20-21swapper/221:10:102
2224026222,24sleep20-21swapper/221:05:092
1266726214,21sleep20-21swapper/200:00:072
411426122,25sleep30-21swapper/323:34:443
3265426114,28sleep0311-21/usr/sbin/munin19:55:170
3188126116,25sleep30-21swapper/321:35:153
969326013,18sleep00-21swapper/020:25:170
736626015,22sleep20-21swapper/220:19:462
427826020,20sleep30-21swapper/321:50:093
427826020,20sleep30-21swapper/321:50:093
2653926015,27sleep326532-21apt-get21:20:013
2174326015,19sleep30-21swapper/300:29:443
1945126014,29sleep019484-21/usr/sbin/munin19:15:110
1772226018,29sleep217329-21/usr/sbin/munin00:15:092
174926011,25sleep00-21swapper/020:00:180
88322596,40sleep00-21swapper/020:24:210
341425917,28sleep33420-21ntp_states20:05:383
239725917,24sleep32435-21http20:04:453
2085325915,21sleep20-21swapper/200:25:072
181592597,11sleep30-21swapper/320:51:213
1233825915,25sleep312398-21fschecks_time20:35:063
578325813,21sleep00-21swapper/020:14:470
3118425817,27sleep331259-21find23:15:183
3054825818,18sleep00-21swapper/021:30:200
279025818,18sleep20-21swapper/221:45:122
279025818,18sleep20-21swapper/221:45:122
2600225811,32sleep326008-21ntp_states19:35:133
1765025815,19sleep20-21swapper/222:34:472
1371925813,28sleep313747-21munin-run00:04:443
1198925815,25sleep212020-21memory22:15:082
338225715,20sleep20-21swapper/223:30:152
2201025715,25sleep022029-21apt-get00:29:570
152832575,43sleep30-21swapper/300:06:543
927825612,27sleep39412-21/usr/sbin/munin22:05:163
864125617,24sleep28646-21/usr/sbin/munin22:05:022
2521825616,26sleep225265-21sed21:15:072
2521825616,26sleep225265-21sed21:15:072
1570025614,26sleep315701-21diskstats20:45:073
569825513,17sleep00-21swapper/021:55:070
569825513,17sleep00-21swapper/021:55:070
3120125520,18sleep30-21swapper/321:34:473
2258925514,17sleep20-21swapper/200:30:162
1167725513,17sleep30-21swapper/322:15:013
808925412,20sleep30-21swapper/323:45:263
808925412,20sleep30-21swapper/323:45:263
3085125415,23sleep20-21swapper/223:15:092
1612825413,24sleep316273-21latency_hist22:29:473
1086325413,23sleep310858-21apt-get23:54:593
759925314,23sleep37596-21apt-get20:19:553
2887525315,22sleep328394-21apt-get19:44:543
219382535,11sleep20-21swapper/222:45:292
2112025310,26sleep021126-21ntp_states19:20:120
659725214,19sleep00-21swapper/020:15:160
2454525213,24sleep324567-21ntp_states22:55:153
2028325213,23sleep320285-21apt-get20:59:583
2456925111,16sleep00-21swapper/022:55:160
1758125113,22sleep317587-21ntp_states19:10:133
1743125111,15sleep00-21swapper/000:15:030
3207325012,23sleep232084-21/usr/sbin/munin19:55:032
3028025018,17sleep20-21swapper/221:30:132
1625625012,21sleep316262-21ntp_states00:10:143
915924917,18sleep30-21swapper/320:25:043
53192495,34sleep00-21swapper/023:35:220
251882495,33sleep00-21swapper/019:33:530
251882495,33sleep00-21swapper/019:33:530
2442924913,20sleep324452-21ntp_states19:30:123
2442924913,20sleep324452-21ntp_states19:30:123
1575224816,15sleep30-21swapper/319:05:053
1891924715,16sleep20-21swapper/219:14:522
787024612,20sleep27874-21diskstats23:45:212
787024612,20sleep27874-21diskstats23:45:212
2828524610,21sleep027806-21apt-get21:24:500
2680724615,17sleep20-21swapper/221:20:082
25432469,14sleep00-21swapper/021:45:050
25432469,14sleep00-21swapper/021:45:050
206902459,18sleep00-21swapper/021:00:110
190202448,14sleep00-21swapper/000:20:040
1580024410,16sleep20-21swapper/200:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional