You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 03:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Thu May 28, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11682363511,3611sleep06727-21kworker/0:021:59:590
20437248422,444sleep120486-21/usr/sbin/munin23:03:521
7015242659,201sleep37028-21sshd00:03:593
20834241813,387sleep220872-21ntpq23:04:012
426223986,380sleep3194-21jbd2/mmcblk2p2-20:29:223
23101227590,166sleep323122-21awk19:48:383
138322259101,33sleep313857-21fschecks_count20:59:003
19886224913,211sleep3225-21systemd-journal21:18:373
1208622370,10sleep30-21swapper/319:13:423
1208622370,10sleep30-21swapper/319:13:423
28895223413,23sleep30-21swapper/320:04:563
28895223413,23sleep30-21swapper/320:04:563
25039222134,22sleep30-21swapper/321:33:593
2456322219,200sleep324608-21unin-run21:33:393
16024222014,190sleep116046-21ntpq19:24:021
16024222014,190sleep116046-21ntpq19:24:021
7036220613,177sleep17196-21latency_hist20:38:401
12086220224,24sleep30-21swapper/319:13:433
4838219252,34sleep30-21swapper/322:13:503
9388218812,78sleep30-21swapper/320:44:073
9388218812,78sleep30-21swapper/320:44:073
4877217615,145sleep25002-21awk20:29:452
10299217210,20sleep30-21swapper/300:14:003
30307216817,107sleep30-21swapper/320:09:003
1395216819,130sleep01543-21grep23:44:110
1927321677,77sleep319309-21ntpq22:59:023
11672216617,112sleep311794-21latency_hist20:53:393
2146216430,46sleep31716-21/usr/sbin/munin22:04:053
3001021638,16sleep30-21swapper/323:33:533
11828216216,121sleep211324-21apt-get00:18:472
12709215914,128sleep112734-21df_inode22:38:541
12309215923,99sleep312383-21fschecks_time20:54:483
25787215766,25sleep30-21swapper/323:19:403
25787215766,25sleep30-21swapper/323:19:403
1993215014,114sleep12127-21latency_hist23:48:401
28221214915,43sleep20-21swapper/223:28:392
28221214915,43sleep20-21swapper/223:28:392
5394214816,117sleep05473-21switchtime22:14:070
27594214824,105sleep027745-21sed23:24:090
27594214824,105sleep027745-21sed23:24:090
11382214825,101sleep111476-21latency_hist00:18:381
3194821458,71sleep331959-21uname23:39:013
2682021450,7sleep10-21swapper/121:39:001
16267214391,27sleep30-21swapper/319:24:083
16267214391,27sleep30-21swapper/319:24:083
13759213922,91sleep20-21swapper/219:18:382
2347213611,112sleep02422-21find22:04:110
23750213516,23sleep323799-21irqstats23:13:593
14571213354,58sleep30-21swapper/322:44:013
29380213117,94sleep029500-21head20:05:050
29380213117,94sleep029500-21head20:05:050
20689213115,33sleep20-21swapper/221:19:052
1750921318,91sleep00-21swapper/022:53:580
17012213015,102sleep317033-21modprobe22:53:373
16525212518,85sleep016568-21munin-plugin-st21:08:370
956212376,27sleep30-21swapper/323:44:023
2718021218,100sleep327222-21ls23:24:003
2718021218,100sleep327222-21ls23:24:003
23597212011,94sleep023116-21apt-get19:48:470
14726211920,79sleep114852-21latency_hist00:28:391
30837211714,84sleep330995-21latency_hist21:53:403
11856211715,86sleep111902-21latency_hist19:13:411
11856211715,86sleep111902-21latency_hist19:13:411
1415921169,15sleep30-21swapper/319:18:563
599521158,92sleep26115-21timerwakeupswit23:59:102
3735211523,75sleep23937-21latency_hist20:28:402
30776211330,44sleep030800-21awk21:53:370
21313211241,13sleep30-21swapper/319:40:313
17410211180,17sleep30-21swapper/300:34:103
13394211113,81sleep113450-21basename20:58:381
6899211013,80sleep06904-21sort00:03:480
30097211022,42sleep20-21swapper/220:08:512
1084121109,86sleep310956-21latency_hist22:33:393
6191210920,75sleep16202-21latency_hist22:18:401
32621210918,75sleep232623-21sshd23:40:462
17536210810,82sleep317688-21unixbench_singl21:09:093
6134210719,65sleep30-21swapper/323:59:113
31915210717,68sleep30-21swapper/320:14:003
13412210712,18sleep20-21swapper/200:23:522
7137210614,72sleep17293-21unixbench_singl22:19:091
10429210619,67sleep110490-21fschecks_time20:48:571
18718210542,18sleep30-21swapper/319:33:533
770321047,87sleep37743-21ps20:39:063
770321047,87sleep37743-21ps20:39:063
17385210417,69sleep217418-21fschecks_count22:53:562
23670210311,70sleep30-21swapper/319:49:223
8435210217,62sleep17948-21/usr/sbin/munin22:24:051
29230210245,14sleep30-21swapper/323:29:093
23293210210,74sleep30-21swapper/323:13:393
1924621026,21sleep20-21swapper/222:59:012
1046210244,13sleep30-21swapper/320:19:013
16271210115,71sleep016313-21cut00:33:390
15768210146,14sleep30-21swapper/300:29:103
12880210117,64sleep213036-21grep20:54:592
2773221008,13sleep30-21swapper/321:43:393
2425210018,58sleep12571-21sed22:04:121
1161729912,67sleep211670-21apt-config20:53:382
385029845,13sleep30-21swapper/323:53:543
2688529825,51sleep026920-21grep21:39:040
73172978,14sleep00-21swapper/020:38:550
73172978,14sleep00-21swapper/020:38:550
280532978,76sleep328099-21apt-config20:03:393
450429613,71sleep14506-21sshd22:12:591
3261529526,36sleep20-21swapper/221:58:482
185602959,20sleep318594-21munin-run22:58:373
123232958,16sleep30-21swapper/300:19:133
2704229412,57sleep10-21swapper/123:23:581
2704229412,57sleep10-21swapper/123:23:581
260542947,64sleep10-21swapper/123:19:461
260542947,64sleep10-21swapper/123:19:461
166682949,11sleep00-21swapper/022:49:100
90922938,19sleep30-21swapper/300:09:083
2587329313,60sleep225896-21/usr/sbin/munin23:19:432
2587329313,60sleep225896-21/usr/sbin/munin23:19:432
1244829210,17sleep30-21swapper/322:38:403
1190429226,37sleep20-21swapper/219:13:422
1190429226,37sleep20-21swapper/219:13:422
76972917,19sleep37735-21munin-run22:23:393
293622917,12sleep00-21swapper/023:29:100
211329110,64sleep22145-21awk22:04:042
42702908,71sleep3363-21snmpd22:10:033
3207129012,62sleep032139-21mailstats23:39:060
3019428913,59sleep230325-21unixbench_multi21:49:072
292972898,16sleep30-21swapper/321:48:403
231728911,52sleep30-21swapper/320:23:493
1659928913,54sleep216689-21fschecks_time00:33:572
1371128922,32sleep00-21swapper/020:58:530
261612885,12sleep20-21swapper/219:55:022
251772887,14sleep30-21swapper/319:53:503
217032888,69sleep321746-21unin-run23:08:383
178512889,12sleep00-21swapper/019:29:100
2810128717,50sleep028237-21latency_hist20:03:400
64892867,59sleep30-21swapper/320:34:103
504328613,42sleep25092-40ogrotate23:58:382
3232428617,46sleep032345-21modprobe21:58:380
136762866,14sleep30-21swapper/322:40:343
1256028522,17sleep00-21swapper/000:19:180
1698628411,57sleep30-21swapper/319:28:403
145402846,13sleep30-21swapper/300:26:563
311522839,57sleep30-21swapper/321:53:573
2802328324,37sleep228038-21apt-get20:03:382
2313228324,37sleep00-21swapper/019:48:390
172802839,12sleep20-21swapper/221:09:042
1407628324,31sleep00-21swapper/019:18:530
214622827,13sleep30-21swapper/321:23:383
1754328219,47sleep117566-21/usr/sbin/munin19:29:031
2480228123,35sleep124828-21munin-run23:18:371
2868328026,27sleep10-21swapper/121:44:101
2484828014,30sleep20-21swapper/221:33:542
2151328025,31sleep10-21swapper/121:23:401
155922807,57sleep30-21swapper/321:04:003
542327918,24sleep10-21swapper/120:33:401
2266527919,36sleep022829-21unixbench_multi23:09:080
220022796,55sleep30-21swapper/321:23:583
2019027913,17sleep00-21swapper/023:03:390
1205227912,48sleep10-21swapper/122:34:591
92842787,35sleep39344-21latency_hist22:28:393
672727823,25sleep00-21swapper/000:03:400
47392787,12sleep00-21swapper/023:54:440
3134827822,26sleep20-21swapper/223:38:402
1432227828,27sleep20-21swapper/220:59:112
575027715,48sleep25789-21fschecks_count20:33:572
32482776,11sleep30-21swapper/323:50:313
3182727714,45sleep131858-21/usr/sbin/munin23:38:591
257882775,60sleep00-21swapper/021:34:320
2683927619,27sleep10-21swapper/119:58:571
1165327611,36sleep20-21swapper/222:34:072
11232766,10sleep30-21swapper/321:59:293
337327511,47sleep03380-21/usr/sbin/munin22:08:550
287022750,8sleep10-21swapper/123:28:571
2651427513,47sleep026459-21latency_hist19:58:400
146292758,17sleep10-21swapper/122:44:031
114682757,11sleep00-21swapper/019:09:240
114682757,11sleep00-21swapper/019:09:240
862427433,26sleep18688-21idleruntime00:08:581
2368327418,32sleep223506-21/usr/sbin/munin23:13:582
2330027425,26sleep10-21swapper/121:28:521
286827312,43sleep02936-21/usr/sbin/munin20:24:060
208427320,24sleep20-21swapper/220:23:402
143522737,54sleep00-21swapper/000:24:310
818727214,43sleep18223-21awk00:08:371
620427220,36sleep26351-21latency_hist22:18:412
259827219,30sleep10-21swapper/123:49:001
1021327225,29sleep310251-21munin-run19:08:383
669627125,26sleep00-21swapper/022:19:000
922227023,32sleep09293-21sort19:04:000
2482127014,33sleep10-21swapper/121:33:491
2162427015,39sleep221782-21cat19:43:402
1918227014,38sleep019214-21/usr/sbin/munin19:34:060
364826920,29sleep13654-21munin-run20:28:391
1885126920,33sleep118867-21awk22:58:481
1637026918,35sleep116497-21timerwakeupswit22:49:071
1608626814,37sleep016109-21/usr/sbin/munin19:24:050
1608626814,37sleep016109-21/usr/sbin/munin19:24:050
2655826715,29sleep10-21swapper/119:58:411
2192826714,41sleep221964-21fschecks_count19:43:572
145132679,15sleep20-21swapper/219:19:032
1396826719,24sleep20-21swapper/222:43:392
892626624,29sleep18932-21cat19:03:491
287312666,39sleep20-21swapper/223:28:592
2380326615,29sleep00-21swapper/023:14:000
1552626614,26sleep20-21swapper/222:48:402
1302826612,39sleep013056-21nfs4_client22:39:010
95362659,16sleep00-21swapper/000:11:030
3172426521,25sleep20-21swapper/220:13:552
2375126519,25sleep20-21swapper/219:49:302
1553326515,33sleep215591-21latency21:03:592
1210126515,31sleep011614-21apt-get20:53:480
768526422,27sleep27695-21modprobe22:23:382
289122645,11sleep00-21swapper/021:44:330
1713726412,24sleep10-21swapper/100:34:081
1457626415,24sleep10-21swapper/119:19:061
97922637,12sleep20-21swapper/222:28:582
824426310,28sleep00-21swapper/022:24:000
2367526319,27sleep123617-21apt-get19:49:241
2235326313,27sleep10-21swapper/123:09:011
20132635,11sleep00-21swapper/020:23:210
1996626316,25sleep10-21swapper/121:18:391
1554926314,23sleep10-21swapper/100:29:081
1693126215,28sleep216984-21latency_hist19:28:392
1693126215,28sleep216984-21latency_hist19:28:392
1411826219,27sleep114167-21/usr/sbin/munin20:59:061
902326113,33sleep29091-21/usr/sbin/munin00:09:072
353926112,34sleep03585-21latency_hist23:53:390
1147926122,19sleep20-21swapper/200:18:392
1075926123,18sleep00-21swapper/000:14:090
3201326015,27sleep032037-21netstat20:14:010
3010026016,26sleep030094-21apt-get20:08:530
2318026017,23sleep10-21swapper/119:48:401
228026013,21sleep20-21swapper/223:48:532
1914026016,24sleep20-21swapper/221:14:532
1011426014,23sleep010268-21ls20:48:400
1011426014,23sleep010268-21ls20:48:400
98125911,23sleep10-21swapper/120:18:591
892825918,23sleep10-21swapper/120:43:571
892825918,23sleep10-21swapper/120:43:571
699625917,24sleep00-21swapper/020:38:390
3203925915,20sleep10-21swapper/120:14:021
2184325917,26sleep121941-21fschecks_time21:23:561
1775125916,17sleep10-21swapper/122:54:031
53202585,12sleep00-21swapper/020:32:580
3047425815,22sleep00-21swapper/023:34:060
2917125813,32sleep229247-21switchtime20:05:032
2917125813,32sleep229247-21switchtime20:05:032
289902586,38sleep20-21swapper/221:45:322
849625710,20sleep20-21swapper/220:43:372
849625710,20sleep20-21swapper/220:43:372
2852025714,27sleep128038-21apt-get20:03:491
2852025714,27sleep128038-21apt-get20:03:491
2005825715,18sleep20-21swapper/219:38:402
1735725717,23sleep217421-21sshd19:28:592
1693125715,17sleep00-21swapper/000:34:030
3000825615,24sleep130003-21apt-get23:33:511
1892725611,16sleep00-21swapper/021:14:470
1690025616,20sleep10-21swapper/121:08:551
1317325614,20sleep10-21swapper/100:23:391
1188225613,15sleep00-21swapper/022:34:090
98932556,12sleep00-21swapper/022:29:000
804225513,26sleep28048-21df_inode22:23:542
536825515,24sleep15361-21apt-get23:58:531
377925515,23sleep13773-21apt-get23:53:521
2329925514,24sleep023293-21apt-get21:28:510
2172325512,28sleep021740-21apt21:23:470
1649825519,17sleep20-21swapper/222:49:082
1420225514,24sleep014198-21apt-get22:43:490
975725412,22sleep10-21swapper/122:28:571
63625414,25sleep1642-21ntp_states21:59:031
3037425412,24sleep130468-21memory20:09:021
2205325416,23sleep222059-21df_inode23:08:542
2174625413,25sleep221743-21apt-get21:23:492
328625313,24sleep23283-21apt-get22:08:492
2959625314,23sleep129601-21diskstats21:48:541
2409625314,21sleep224201-21users21:29:102
213162537,14sleep10-21swapper/119:40:331
112902535,36sleep00-21swapper/020:49:370
734825211,18sleep20-21swapper/200:04:062
3234825215,23sleep132403-21munin-run21:58:391
885225111,22sleep28861-21df_inode20:43:542
885225111,22sleep28861-21df_inode20:43:542
2535125111,23sleep00-21swapper/023:18:480
2535125111,23sleep00-21swapper/023:18:480
2150325112,23sleep021512-21munin-run21:23:390
2111325112,17sleep00-21swapper/023:04:070
1911125111,23sleep119149-21/usr/sbin/munin19:34:031
1290325114,22sleep012948-21ntp_states19:14:040
1682425014,20sleep016820-21apt-get21:08:500
1533525012,21sleep10-21swapper/121:03:541
213102496,34sleep00-21swapper/019:40:280
1525224912,18sleep20-21swapper/200:29:002
9262487,15sleep20-21swapper/223:44:012
768424810,23sleep17695-21ntp_states20:39:041
768424810,23sleep17695-21ntp_states20:39:041
89332477,15sleep20-21swapper/219:03:502
315182467,15sleep20-21swapper/221:54:042
1249424614,16sleep10-21swapper/119:13:531
264992407,14sleep20-21swapper/221:38:542
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional