You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 23:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Mar 22, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1340529796,961sleep032089-1kworker/0:1H09:28:460
18015259014,560sleep118077-21fschecks_time09:43:001
8609255218,516sleep08701-21/usr/sbin/munin09:13:040
627251512,488sleep1663-21/usr/sbin/munin08:48:021
627251512,488sleep1663-21/usr/sbin/munin08:48:021
29581240839,218sleep1588-21cron08:37:411
32724235231,272sleep1318-21grep12:13:011
201612331130,135sleep120327-21unixbench_multi09:48:131
201612331130,135sleep120327-21unixbench_multi09:48:131
29624233013,200sleep129662-21fschecks_count12:02:591
1553230513,275sleep01637-21memory10:33:050
1553230513,275sleep01637-21memory10:33:050
1208228520,236sleep01227-21apt-get07:07:540
28589228110,259sleep128681-21cat11:58:111
3251422596,243sleep113441-21snmpd08:45:161
1812122450,101sleep11010-21cyclictest07:58:121
1812122450,101sleep11010-21cyclictest07:58:121
9402224417,201sleep39451-21latency_hist07:32:433
26861223749,173sleep126918-21fschecks_count08:28:001
26861223749,173sleep126918-21fschecks_count08:28:001
8464223441,34sleep10-21swapper/107:28:051
25226222219,47sleep10-21swapper/108:22:551
20448222251,18sleep10-21swapper/108:07:451
20448222251,18sleep10-21swapper/108:07:451
14588220411,177sleep314732-21latency_hist09:32:453
25494220147,54sleep125430-21latency_hist10:07:431
972220012,81sleep11043-21sort07:07:431
5220220025,26sleep10-21swapper/107:18:081
5220220025,26sleep10-21swapper/107:18:081
3436219923,162sleep13527-21latency_hist08:57:431
13496218911,66sleep10-21swapper/107:43:151
11442218937,103sleep10-21swapper/109:22:501
6839218555,23sleep10-21swapper/107:23:211
7251218413,156sleep17395-21latency_hist10:52:431
1540721647,117sleep115481-21find09:33:111
16130215249,46sleep10-21swapper/107:53:041
2172215162,24sleep10-21swapper/112:18:051
14073214719,67sleep114112-21fschecks_count11:13:221
16856214622,102sleep10-21swapper/111:22:431
2394121439,122sleep123955-21grep08:18:031
30381213816,109sleep130458-21tr08:38:091
7998213132,85sleep18086-21latency_hist12:37:431
11879213040,76sleep111882-21cron11:07:411
1407212815,64sleep10-21swapper/107:08:031
4523212742,22sleep10-21swapper/107:17:441
21853212721,91sleep021909-21fschecks_count11:38:000
27012212614,98sleep10-21swapper/111:53:111
27012212614,98sleep10-21swapper/111:53:111
22464212633,44sleep122501-21ntpq08:13:061
1044212614,96sleep21193-21modprobe07:07:442
2094721248,103sleep120991-21ntpq08:08:061
7917212220,83sleep31-21systemd12:37:413
3414212217,85sleep10-21swapper/112:23:021
11588211913,51sleep10-21swapper/107:38:061
3214721188,95sleep032089-1kworker/0:1H10:27:520
2934211817,84sleep32939-21/usr/sbin/munin10:38:003
4993211710,53sleep15131-21latency_hist09:02:431
13560211619,78sleep313729-21latency_hist11:12:443
13560211619,78sleep313729-21latency_hist11:12:443
15627211550,15sleep10-21swapper/107:52:441
22036211312,89sleep00-21swapper/009:54:300
22036211312,89sleep00-21swapper/009:54:300
1892421139,36sleep10-21swapper/111:28:041
29450211221,74sleep00-21swapper/010:18:070
2820211249,47sleep32917-21munin-node07:12:453
31445211110,53sleep10-21swapper/110:23:131
21486211118,74sleep321521-21grep09:53:073
21486211118,74sleep321521-21grep09:53:073
509921109,62sleep113441-21snmpd12:28:021
509921109,62sleep113441-21snmpd12:28:021
6793210812,54sleep10-21swapper/109:07:561
6627210819,67sleep26144-21apt-get12:32:522
3356210827,59sleep33403-21munin-run08:57:413
10193210815,72sleep210285-21/usr/sbin/munin09:18:022
10193210815,72sleep210285-21/usr/sbin/munin09:18:022
23681210717,76sleep323732-21cpuspeed08:17:573
2710921059,84sleep00-21swapper/010:11:330
28041210419,66sleep028106-21switchtime10:13:110
2176221039,52sleep10-21swapper/111:37:531
2390921018,57sleep123918-21munin-run10:02:421
2390921018,57sleep123918-21munin-run10:02:421
1340621019,50sleep10-21swapper/109:28:471
971721009,49sleep10-21swapper/107:32:581
971721009,49sleep10-21swapper/107:32:581
3399210016,67sleep23463-21switchtime10:38:112
20791210019,58sleep020918-21latency_hist09:52:440
20791210019,58sleep020918-21latency_hist09:52:440
1034210013,71sleep31110-21latency_hist10:32:433
3263629828,33sleep132680-21processes10:28:091
2531429815,69sleep225377-21fschecks_time08:23:002
12662989,34sleep10-21swapper/110:32:531
12662989,34sleep10-21swapper/110:32:531
466929736,43sleep10-21swapper/110:43:071
1904129718,61sleep319086-21ntp_offset11:28:063
1330429714,63sleep213247-21apt-get09:28:322
749029618,58sleep10-21swapper/110:52:561
648829620,60sleep26492-21cron09:07:412
648829620,60sleep26492-21cron09:07:412
316922968,36sleep10-21swapper/112:08:121
3000529612,62sleep230069-21ps12:03:092
2494629613,66sleep324948-21/usr/sbin/munin11:48:003
2494629613,66sleep324948-21/usr/sbin/munin11:48:003
2216629616,17sleep20-21swapper/211:38:062
1919129620,58sleep2222-21systemd-journal09:47:412
785729518,61sleep37955-21latency_hist07:27:433
602929517,62sleep25918-21/usr/sbin/munin10:48:002
3253929510,56sleep00-21swapper/007:03:060
257772957,40sleep113441-21snmpd11:48:181
257772957,40sleep113441-21snmpd11:48:181
2594229439,43sleep125933-21apt-get10:07:551
2399329418,45sleep324140-21grep11:43:123
333329344,27sleep13397-21sendmail_mailst10:38:101
3145729316,57sleep331488-21/usr/sbin/munin12:08:063
1606529130,42sleep316084-21apt09:37:413
1220828925,39sleep312352-21grep09:23:123
914328811,58sleep09041-21/usr/sbin/munin10:57:590
3141028817,50sleep231062-21/usr/sbin/munin12:08:052
521028727,45sleep10-21swapper/109:02:561
521028727,45sleep10-21swapper/109:02:561
260928624,27sleep20-21swapper/210:37:432
260928624,27sleep20-21swapper/210:37:432
1286128616,55sleep212758-21latency_hist09:27:432
3120728519,19sleep30-21swapper/308:42:433
2677528516,50sleep326808-21ntp_kernel_err11:53:053
2677528516,50sleep326808-21ntp_kernel_err11:53:053
2250328512,52sleep222071-21/usr/sbin/munin08:13:072
100928427,35sleep21050-21proc_pri08:48:112
100928427,35sleep21050-21proc_pri08:48:112
2541528320,40sleep325443-21idleruntime-cro10:07:413
1512128314,17sleep00-21swapper/007:48:150
2803328221,39sleep228065-21modprobe08:32:412
2803328221,39sleep228065-21modprobe08:32:412
187728218,36sleep32025-21grep08:52:443
187728218,36sleep32025-21grep08:52:443
862928116,50sleep28700-21sensors_temp07:28:102
227928111,56sleep22357-21mailstats12:18:102
2029728119,41sleep220320-21latency_hist08:07:422
2029728119,41sleep220320-21latency_hist08:07:422
879928020,41sleep10-21swapper/109:13:091
818228024,41sleep08247-21uniq07:28:000
2867728016,31sleep30-21swapper/308:33:073
274882809,15sleep10-21swapper/110:12:551
2206028032,12sleep10-21swapper/109:54:471
2206028032,12sleep10-21swapper/109:54:471
2044228033,30sleep10-21swapper/111:33:041
2449227919,11sleep10-21swapper/111:46:411
1769327922,15sleep10-21swapper/109:42:441
718927812,48sleep27230-21ntp_kernel_pll_09:08:052
463327827,28sleep20-21swapper/212:27:432
2915127832,31sleep10-21swapper/110:18:001
235827815,42sleep32451-21/usr/sbin/munin12:18:113
2180527813,49sleep021815-21modprobe08:12:410
316792770,6sleep00-21swapper/008:43:040
23162776,18sleep20-21swapper/208:53:042
2028727717,36sleep320381-21fschecks_time11:33:023
888027610,13sleep10-21swapper/110:57:441
2237127625,31sleep30-21swapper/311:38:113
170942767,14sleep10-21swapper/111:22:551
1330127631,12sleep10-21swapper/111:11:041
1330127631,12sleep10-21swapper/111:11:041
1092827618,11sleep10-21swapper/109:19:271
1092827618,11sleep10-21swapper/109:19:271
311427511,49sleep03262-21date12:22:440
2970527518,25sleep00-21swapper/008:37:440
1179327514,46sleep311875-21switchtime07:38:133
1130127514,24sleep00-21swapper/011:03:120
703027415,42sleep27050-21/usr/sbin/munin07:23:262
3219027420,27sleep20-21swapper/208:43:152
2969127419,31sleep20-21swapper/210:18:132
261492746,16sleep00-21swapper/008:23:230
1557427411,15sleep30-21swapper/307:52:423
1098427419,25sleep00-21swapper/007:37:450
1098427419,25sleep00-21swapper/007:37:450
3163027320,26sleep30-21swapper/308:43:043
1254927313,46sleep212593-21ntpq11:08:072
1254927313,46sleep212593-21ntpq11:08:072
964427222,34sleep30-21swapper/307:32:563
964427222,34sleep30-21swapper/307:32:563
627227218,29sleep20-21swapper/212:32:442
627227218,29sleep20-21swapper/212:32:442
2859527213,43sleep228654-21latency08:33:062
2414627213,44sleep024214-21users11:43:130
874627122,26sleep30-21swapper/310:57:423
354527115,30sleep00-21swapper/007:13:070
308852718,14sleep30-21swapper/310:23:033
2611727121,23sleep00-21swapper/011:52:410
2611727121,23sleep00-21swapper/011:52:410
1680227110,44sleep216804-21sshd07:53:382
1411527116,37sleep214196-21/usr/sbin/munin11:13:232
126062715,10sleep00-21swapper/009:26:080
3246426922,29sleep232492-21nfs4_client07:03:042
1068626914,40sleep210697-21df_inode11:02:582
266126813,38sleep22819-21munin-node07:12:442
2621726823,24sleep20-21swapper/210:08:082
2268526822,25sleep20-21swapper/209:57:592
2268526822,25sleep20-21swapper/209:57:592
1324626820,27sleep213307-21grep07:43:122
98942676,49sleep00-21swapper/009:17:480
98942676,49sleep00-21swapper/009:17:480
613126727,23sleep00-21swapper/012:32:410
613126727,23sleep00-21swapper/012:32:410
2236526717,21sleep20-21swapper/209:57:422
2236526717,21sleep20-21swapper/209:57:422
1533826717,25sleep30-21swapper/311:17:443
1501326716,24sleep00-21swapper/009:33:020
888426616,33sleep38952-21/usr/sbin/munin09:13:113
540926620,26sleep00-21swapper/007:18:110
540926620,26sleep00-21swapper/007:18:110
2737926616,25sleep00-21swapper/008:28:110
2737926616,25sleep00-21swapper/008:28:110
1338326618,18sleep00-21swapper/007:43:140
853726512,37sleep28579-21cat09:13:022
2720026522,28sleep327218-21munin-run10:12:413
1719626512,25sleep00-21swapper/007:57:430
925126418,23sleep30-21swapper/310:58:023
565026417,28sleep35668-21ls10:47:413
289282646,45sleep00-21swapper/011:58:390
2542926420,22sleep20-21swapper/211:48:112
2542926420,22sleep20-21swapper/211:48:112
1875626423,22sleep30-21swapper/308:02:433
1875626423,22sleep30-21swapper/308:02:433
3069026319,21sleep20-21swapper/210:22:572
3014326315,25sleep00-21swapper/012:03:110
2807126315,25sleep20-21swapper/211:58:002
2694026311,33sleep027011-21sensors_temp11:53:100
2694026311,33sleep027011-21sensors_temp11:53:100
2684526314,30sleep326854-21df_inode08:27:583
2684526314,30sleep326854-21df_inode08:27:583
1665626313,24sleep20-21swapper/209:38:052
1373626313,32sleep00-21swapper/011:12:450
1142826315,28sleep20-21swapper/207:38:032
45592628,17sleep00-21swapper/012:27:250
2537926215,27sleep325437-21/usr/sbin/munin08:23:013
2435926222,22sleep20-21swapper/208:18:122
189612626,12sleep00-21swapper/009:44:450
549326118,25sleep35507-21uname09:03:063
549326118,25sleep35507-21uname09:03:063
3083626115,23sleep30-21swapper/312:07:423
2996926114,31sleep229978-21diskstats08:37:582
191052615,12sleep20-21swapper/211:28:092
153626120,18sleep30-21swapper/312:17:433
3239626014,24sleep00-21swapper/012:12:430
6982597,11sleep30-21swapper/307:05:073
65582596,39sleep00-21swapper/007:23:040
40482595,10sleep00-21swapper/010:41:570
171625924,20sleep30-21swapper/310:33:083
171625924,20sleep30-21swapper/310:33:083
73742586,11sleep30-21swapper/309:08:103
4892585,11sleep00-21swapper/008:47:450
4892585,11sleep00-21swapper/008:47:450
3264025818,23sleep20-21swapper/212:12:582
631025714,26sleep36341-21/usr/sbin/munin10:48:053
590225716,25sleep06027-21sed09:03:150
590225716,25sleep06027-21sed09:03:150
2415825711,18sleep00-21swapper/010:02:570
1751825710,32sleep017524-21diskstats07:58:000
1751825710,32sleep017524-21diskstats07:58:000
1736725716,20sleep30-21swapper/311:23:053
533125615,20sleep30-21swapper/307:18:103
533125615,20sleep30-21swapper/307:18:103
2494525616,26sleep024987-21systemd-run08:22:420
2456025614,26sleep224609-21/usr/sbin/munin10:03:072
2099325619,17sleep20-21swapper/208:08:072
2038725611,24sleep00-21swapper/008:07:440
2038725611,24sleep00-21swapper/008:07:440
1759825612,30sleep21-21systemd11:23:122
1489125617,21sleep20-21swapper/209:33:002
882625516,17sleep00-21swapper/010:57:430
810225518,18sleep00-21swapper/009:12:440
3268125514,23sleep2304-21mailstats10:28:102
3062125513,25sleep030615-21apt-get10:22:550
2331225513,26sleep223306-21apt-get11:42:542
2207025512,27sleep022088-21apt08:12:530
2019325513,24sleep220185-21apt-get11:32:572
1706525514,25sleep00-21swapper/011:22:500
1638525518,17sleep30-21swapper/309:37:583
367225413,24sleep03667-21apt-get08:57:540
2260225414,22sleep322669-21df09:57:563
2260225414,22sleep322669-21df09:57:563
2217625413,24sleep322180-21diskstats08:13:003
2122625411,19sleep30-21swapper/308:08:123
1994725414,27sleep220011-21ps09:48:102
1994725414,27sleep220011-21ps09:48:102
1121225414,20sleep30-21swapper/311:03:113
2943725315,23sleep329447-21needreboot10:18:063
2753325311,26sleep327560-21df10:12:573
2001325312,22sleep320082-21sensors_temp09:48:113
2001325312,22sleep320082-21sensors_temp09:48:113
1864225313,24sleep018637-21apt-get11:27:550
1833325317,19sleep30-21swapper/309:43:063
1585925314,21sleep315858-21apt-get07:52:543
1431825311,24sleep214312-21apt-get07:47:562
1122225313,23sleep011215-21apt-get07:37:570
594125214,23sleep05937-21apt-get10:47:540
487225214,21sleep30-21swapper/312:27:563
487225214,21sleep30-21swapper/312:27:563
461125214,19sleep00-21swapper/010:43:060
2058925217,18sleep00-21swapper/011:33:070
1683925212,23sleep016853-21munin-run11:22:420
943425113,24sleep29451-21netstat10:58:052
690025112,21sleep06949-21irqstats12:33:180
415725114,23sleep34219-21sendmail_mailst08:58:093
2637925110,17sleep20-21swapper/211:52:512
2637925110,17sleep20-21swapper/211:52:512
1611825115,21sleep216197-21sort11:18:122
1280825116,18sleep00-21swapper/011:08:130
1280825116,18sleep00-21swapper/011:08:130
372925013,18sleep30-21swapper/312:23:083
248825012,13sleep30-21swapper/307:12:413
218425011,25sleep02255-21fschecks_time08:53:020
751424912,22sleep27561-21df10:52:572
52442499,19sleep20-21swapper/207:18:092
52442499,19sleep20-21swapper/207:18:092
2299924912,24sleep013441-21snmpd09:58:050
2299924912,24sleep013441-21snmpd09:58:050
1755124910,25sleep217415-21/usr/sbin/munin07:58:022
1755124910,25sleep217415-21/usr/sbin/munin07:58:022
1028024917,15sleep20-21swapper/207:33:112
1028024917,15sleep20-21swapper/207:33:112
448624814,17sleep20-21swapper/210:43:042
2845224811,21sleep30-21swapper/311:58:093
1680124811,18sleep30-21swapper/311:22:413
63624711,21sleep0642-21ntp_states12:13:080
298712478,17sleep00-21swapper/008:37:520
1643124711,21sleep016458-21df09:37:590
1123324712,15sleep30-21swapper/309:22:443
1123324712,15sleep30-21swapper/309:22:443
2748924615,18sleep20-21swapper/210:12:562
1317724611,19sleep313222-21ntp_states07:43:093
364824513,17sleep20-21swapper/208:57:522
194942429,17sleep319571-21/usr/sbin/munin08:03:113
194942429,17sleep319571-21/usr/sbin/munin08:03:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional