You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 15:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Mon Mar 23, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15692436124,4321sleep214817-21kworker/u8:107:58:102
19560286512,834sleep119594-21/usr/sbin/munin10:38:021
19560286512,834sleep119594-21/usr/sbin/munin10:38:021
9448245197,273sleep39590-21grep11:48:113
31117234413,315sleep031175-21/usr/sbin/munin07:48:110
170192320107,198sleep317053-21proc_pri12:13:063
1745231622,195sleep30-21swapper/309:42:383
1745231622,195sleep30-21swapper/309:42:383
29812227579,25sleep30-21swapper/309:27:533
2859226813,197sleep32922-21latency_hist08:02:393
3286223322,63sleep30-21swapper/308:03:003
28927223212,198sleep328988-21latency_hist07:42:413
28927223212,198sleep328988-21latency_hist07:42:413
801922168,193sleep38157-21latency_hist10:02:403
15232221019,64sleep30-21swapper/310:23:113
20111220937,154sleep3222-21systemd-journal10:38:373
20111220937,154sleep3222-21systemd-journal10:38:373
3255222076,190sleep00-21swapper/011:18:210
3255222076,190sleep00-21swapper/011:18:210
10273220719,82sleep30-21swapper/310:08:093
10273220719,82sleep30-21swapper/310:08:093
19106220431,27sleep20-21swapper/207:12:412
2802721998,31sleep328164-21latency_hist09:22:403
2379121999,175sleep023803-21ntpq09:08:040
31008218610,65sleep30-21swapper/307:48:063
24980217818,75sleep325022-21fschecks_count09:12:593
30340217323,69sleep3222-21systemd-journal07:47:383
2893721738,16sleep30-21swapper/311:08:033
687621729,147sleep37034-21sed08:13:093
2687217217,143sleep117378-21kworker/1:008:01:041
28635216814,140sleep128675-21ntp_kernel_err09:23:021
8385216510,140sleep28459-21find08:18:092
3208121619,40sleep30-21swapper/311:18:043
3208121619,40sleep30-21swapper/311:18:043
8462216015,128sleep38602-21tail08:18:103
1145021595,10sleep30-21swapper/311:56:313
25837214910,15sleep00-21swapper/010:58:030
23005214913,48sleep30-21swapper/310:48:093
19039214916,117sleep319043-21cron07:12:383
15317214920,48sleep30-21swapper/308:42:403
15317214920,48sleep30-21swapper/308:42:403
2729821459,21sleep30-21swapper/307:37:403
9508214469,24sleep30-21swapper/308:22:583
19913214438,86sleep019960-21awk08:57:380
12508214213,114sleep212562-21fschecks_count08:32:582
16464214155,68sleep30-21swapper/308:43:123
10631214013,112sleep010605-21latency_hist08:27:390
3577213923,100sleep30-21swapper/309:47:513
2008213815,49sleep20-21swapper/209:42:492
2008213815,49sleep20-21swapper/209:42:492
2918521376,11sleep30-21swapper/309:23:163
2442821377,12sleep30-21swapper/309:10:213
31390213615,96sleep30-21swapper/311:17:403
1945121369,112sleep319513-21latency_hist12:22:383
1945121369,112sleep319513-21latency_hist12:22:383
12938213510,106sleep10-21swapper/110:17:521
1294021348,114sleep312944-21grep10:17:543
5124213312,39sleep30-21swapper/309:52:513
23335213167,53sleep323372-21ps07:23:073
574521305,10sleep30-21swapper/308:10:353
8357212811,100sleep38462-21fschecks_time10:03:003
8357212811,100sleep38462-21fschecks_time10:03:003
22981212814,90sleep30-21swapper/312:33:003
2067821286,55sleep30-21swapper/312:24:083
2067821286,55sleep30-21swapper/312:24:083
376921258,44sleep33793-21munin-run11:32:373
11380212586,25sleep311452-21mailstats08:28:073
69021245,52sleep30-21swapper/307:53:513
2228521235,72sleep30-21swapper/307:18:513
2226021237,51sleep30-21swapper/312:29:333
2226021237,51sleep30-21swapper/312:29:333
20317212330,56sleep320386-21fschecks_time08:57:583
1391321228,42sleep313985-21mailstats12:03:053
16805212124,77sleep016830-21munin-run08:47:370
3098621195,102sleep00-21swapper/009:31:290
19737211815,77sleep30-21swapper/307:12:543
1658211631,26sleep31788-21grep11:23:213
1658211631,26sleep31788-21grep11:23:213
787821150,7sleep00-21swapper/008:17:570
19167211513,18sleep00-21swapper/010:37:530
19167211513,18sleep00-21swapper/010:37:530
18916211419,78sleep20-21swapper/210:37:392
24868211212,22sleep30-21swapper/310:55:033
18610211212,85sleep118119-21/usr/sbin/munin12:18:041
18610211212,85sleep118119-21/usr/sbin/munin12:18:041
1909821108,19sleep30-21swapper/308:53:053
16007211028,40sleep00-21swapper/012:09:560
16007211028,40sleep00-21swapper/012:09:560
5317210824,32sleep30-21swapper/311:37:383
29339210814,77sleep029373-21irqstats07:43:110
12036210823,66sleep30-21swapper/310:13:103
26730210713,80sleep026797-21df09:18:010
1699121077,54sleep30-21swapper/310:28:443
1699121077,54sleep30-21swapper/310:28:443
12563210724,13sleep30-21swapper/308:32:593
5721210621,61sleep25787-21fschecks_time11:37:572
22497210622,62sleep022505-21modprobe12:32:370
22497210622,62sleep022505-21modprobe12:32:370
21193210421,11sleep30-21swapper/310:43:053
13762210414,73sleep013852-21sort08:37:400
8264210324,11sleep30-21swapper/311:45:213
24045210310,79sleep124079-21/usr/sbin/munin10:52:591
17893210324,63sleep30-21swapper/310:33:033
10081210318,62sleep210145-21cat10:08:042
10081210318,62sleep210145-21cat10:08:042
6003210260,27sleep36049-21/usr/sbin/munin11:38:023
3129210226,11sleep30-21swapper/309:45:333
3129210226,11sleep30-21swapper/309:45:333
1287821015,84sleep30-21swapper/311:59:303
983210027,49sleep21008-21munin-run07:57:382
3786210023,53sleep23924-21grep08:03:112
28041210018,11sleep30-21swapper/311:05:333
23137210015,72sleep123161-21kernelversion07:23:011
317772997,15sleep10-21swapper/109:33:031
317772997,15sleep10-21swapper/109:33:031
313622998,16sleep30-21swapper/309:32:533
313622998,16sleep30-21swapper/309:32:533
2284329918,41sleep30-21swapper/309:05:043
426329824,54sleep14401-21grep09:48:091
246232989,17sleep317541-21cyclictest07:28:003
2450229814,70sleep124525-21fschecks_count07:27:581
2304129823,50sleep223055-21modprobe09:07:372
2039229511,65sleep020450-21/usr/sbin/munin08:57:590
1687429519,58sleep317006-21unixbench_multi07:03:093
2899929315,59sleep229092-21mailstats11:08:062
266452935,77sleep30-21swapper/311:02:303
2553629216,60sleep225597-21sort10:57:572
180532926,76sleep313441-21snmpd08:48:373
150062927,12sleep30-21swapper/312:07:533
150062927,12sleep30-21swapper/312:07:533
1134229223,56sleep211377-21ps08:28:062
1808129113,64sleep118143-21awk10:33:101
1601129114,61sleep116046-21proc_pri08:43:061
2182029019,55sleep221901-21run-parts07:18:062
718428920,54sleep27185-21cpu11:42:542
2402828917,44sleep124192-21grep09:08:091
46542880,7sleep10-21swapper/111:33:071
2526528812,58sleep025299-21/usr/sbin/munin09:13:040
2288428815,47sleep222237-21/usr/sbin/munin10:48:082
2281028813,61sleep122882-21switchtime10:48:071
1456128710,65sleep114568-21df_inode10:22:561
358128616,56sleep23634-21ls09:47:542
1095928611,59sleep111009-21fschecks_count08:27:561
225082850,9sleep10-21swapper/112:32:371
225082850,9sleep10-21swapper/112:32:371
2835328325,28sleep00-21swapper/009:22:570
998928220,41sleep21-21systemd11:52:382
160082828,60sleep016027-21apt-get10:27:480
160082828,60sleep016027-21apt-get10:27:480
1805628114,49sleep018080-21/usr/sbin/munin10:33:090
1778128126,34sleep017777-21apt-get07:07:500
1147128112,52sleep011553-21/usr/sbin/munin08:28:080
1617328026,21sleep00-21swapper/007:02:500
1339828014,49sleep213400-21apt-get12:02:482
849427923,26sleep00-21swapper/011:47:400
253002790,8sleep10-21swapper/109:13:041
2132127710,49sleep00-21swapper/007:17:510
1882627725,23sleep00-21swapper/012:18:070
1882627725,23sleep00-21swapper/012:18:070
93727615,30sleep00-21swapper/011:22:490
93727615,30sleep00-21swapper/011:22:490
1782627623,32sleep217853-21munin-run12:17:372
1530027614,47sleep215315-21grep08:42:392
1530027614,47sleep215315-21grep08:42:392
3234527513,47sleep032346-21sshd09:34:030
3234527513,47sleep032346-21sshd09:34:030
3036727513,45sleep030405-21memory11:13:010
2266127514,47sleep022691-21ntp_kernel_pll_10:48:020
1424827518,15sleep00-21swapper/010:22:390
941627418,36sleep29507-21/usr/sbin/munin08:22:572
2288527413,48sleep222979-21fschecks_time12:32:592
2131427417,28sleep20-21swapper/212:27:542
2131427417,28sleep20-21swapper/212:27:542
590427226,23sleep10-21swapper/108:12:391
195952727,13sleep20-21swapper/210:38:032
195952727,13sleep20-21swapper/210:38:032
1752827222,27sleep20-21swapper/207:07:402
1421327210,13sleep10-21swapper/108:38:021
1421327210,13sleep10-21swapper/108:38:021
1669327128,23sleep10-21swapper/107:03:071
349427014,40sleep03495-21sshd11:29:030
250732700,8sleep20-21swapper/207:28:102
1785027017,35sleep118003-21sed08:48:111
1729227016,34sleep117309-21modprobe10:32:371
1729227016,34sleep117309-21modprobe10:32:371
1275027010,41sleep20-21swapper/211:58:112
1042727018,28sleep00-21swapper/011:52:570
746026920,31sleep07527-21modprobe08:17:380
230202695,20sleep10-21swapper/109:07:201
1349226921,20sleep10-21swapper/112:02:551
307626816,34sleep13077-21/usr/sbin/munin08:02:501
295826823,24sleep20-21swapper/211:28:042
2339726823,22sleep10-21swapper/112:33:101
213626821,25sleep00-21swapper/007:59:050
1836226814,30sleep00-21swapper/008:52:380
1634826821,26sleep10-21swapper/112:12:391
107526817,32sleep01215-21dpkg07:57:400
779126717,31sleep17866-21timerandwakeup11:43:091
754026717,28sleep20-21swapper/208:17:402
646726716,34sleep00-21swapper/009:57:400
3220126728,18sleep20-21swapper/211:18:072
3220126728,18sleep20-21swapper/211:18:072
3039226724,23sleep10-21swapper/107:47:401
2825626721,23sleep00-21swapper/011:07:400
229862676,19sleep00-21swapper/009:06:550
179526719,26sleep00-21swapper/009:42:390
179526719,26sleep00-21swapper/009:42:390
1685726721,20sleep20-21swapper/208:47:392
3040626615,37sleep130445-21ntpq11:13:021
2799526612,36sleep028001-21ntp_states07:38:060
2799526612,36sleep028001-21ntp_states07:38:060
249142666,20sleep00-21swapper/007:28:090
1580026614,38sleep015947-21cat10:27:400
130322666,37sleep10-21swapper/108:33:091
621026523,22sleep10-21swapper/111:38:081
334226520,24sleep00-21swapper/009:47:390
334226520,24sleep00-21swapper/009:47:390
277572655,45sleep10-21swapper/109:19:471
267726517,20sleep10-21swapper/109:43:081
267726517,20sleep10-21swapper/109:43:081
2202626518,28sleep222116-21sort10:47:402
996726425,22sleep10-21swapper/108:23:081
473126411,22sleep20-21swapper/208:07:552
382126417,33sleep13908-21grep11:32:391
1916326414,33sleep219257-21mailstats08:53:082
1447826416,23sleep00-21swapper/008:38:090
1447826416,23sleep00-21swapper/008:38:090
116202645,11sleep20-21swapper/210:13:002
3072226315,33sleep130723-21diskstats07:48:001
2896026324,21sleep10-21swapper/111:08:051
2550026322,21sleep20-21swapper/209:13:102
2417926323,22sleep00-21swapper/010:53:030
1982126314,21sleep20-21swapper/207:12:572
1969326315,16sleep10-21swapper/112:22:531
1969326315,16sleep10-21swapper/112:22:531
1351626315,35sleep213592-21switchtime10:18:092
1068226316,20sleep10-21swapper/111:53:021
3270326218,23sleep20-21swapper/209:37:402
3270326218,23sleep20-21swapper/209:37:402
122142627,17sleep10-21swapper/111:58:021
1186526223,19sleep10-21swapper/110:13:071
959626111,33sleep09636-21users11:48:120
94242618,43sleep00-21swapper/010:06:240
94242618,43sleep00-21swapper/010:06:240
908526118,25sleep19194-21sed08:22:401
520126113,22sleep20-21swapper/209:52:542
2020126117,28sleep020214-21/usr/sbin/munin07:13:070
1987426117,27sleep219927-21/usr/sbin/munin12:22:582
1987426117,27sleep219927-21/usr/sbin/munin12:22:582
17232615,11sleep10-21swapper/109:42:201
17232615,11sleep10-21swapper/109:42:201
853326021,22sleep28626-21memory10:03:032
853326021,22sleep28626-21memory10:03:032
42326019,24sleep1484-21systemd-run11:22:381
42326019,24sleep1484-21systemd-run11:22:381
1977626016,27sleep00-21swapper/012:22:560
1977626016,27sleep00-21swapper/012:22:560
1510326015,24sleep20-21swapper/210:23:092
47925914,23sleep20-21swapper/209:37:532
47925914,23sleep20-21swapper/209:37:532
2917325920,24sleep029314-21sed11:08:080
2620025911,19sleep00-21swapper/007:33:010
217792596,10sleep00-21swapper/010:45:070
2129625915,25sleep121315-21apt-get07:17:481
873225815,26sleep18727-21apt-get11:47:531
513725811,25sleep05199-21awk08:08:050
3112525813,26sleep231258-21sed09:32:402
2691825816,26sleep226937-21apt-get11:02:492
1946325813,30sleep119522-21irqstats07:12:441
1054425817,18sleep20-21swapper/211:52:592
2946325718,20sleep20-21swapper/207:43:132
2657225717,17sleep10-21swapper/107:33:101
150525717,18sleep10-21swapper/111:23:191
150525717,18sleep10-21swapper/111:23:191
121252577,12sleep00-21swapper/011:58:010
65372569,33sleep06539-21users11:38:110
476325613,30sleep013441-21snmpd09:51:390
2734225623,19sleep10-21swapper/111:03:051
2154925615,25sleep121696-21latency_hist09:02:401
2116225614,26sleep121168-21ntp_states10:43:021
1655025613,27sleep016546-21apt-get12:12:510
124325615,20sleep20-21swapper/211:23:122
124325615,20sleep20-21swapper/211:23:122
1036125613,26sleep110426-21switchtime10:08:111
1036125613,26sleep110426-21switchtime10:08:111
833825511,28sleep18344-21diskstats10:02:581
833825511,28sleep18344-21diskstats10:02:581
78472557,36sleep00-21swapper/010:01:040
533325516,17sleep10-21swapper/109:52:571
473925520,18sleep20-21swapper/211:33:092
2957525521,17sleep20-21swapper/209:27:392
2680525515,21sleep20-21swapper/209:18:032
2286925514,24sleep222863-21apt-get07:22:542
1844025515,17sleep20-21swapper/208:52:402
149312559,18sleep20-21swapper/212:07:482
149312559,18sleep20-21swapper/212:07:482
2800225419,21sleep10-21swapper/107:38:071
2800225419,21sleep10-21swapper/107:38:071
2404625412,16sleep00-21swapper/012:37:370
2381325411,31sleep223852-21processes09:08:062
1657725416,23sleep216645-21/usr/sbin/munin10:28:062
1657725416,23sleep216645-21/usr/sbin/munin10:28:062
405725316,16sleep00-21swapper/011:32:500
3072425313,18sleep20-21swapper/207:48:012
3005525315,20sleep130084-21irqstats09:27:591
177822538,20sleep10-21swapper/107:07:511
1107825315,23sleep011115-21munin-run10:12:380
1107825315,23sleep011115-21munin-run10:12:380
352125215,21sleep03558-21ntp_states08:03:050
3010825216,20sleep230117-21df_inode11:12:542
2441025213,21sleep224490-21awk10:53:092
1662925217,21sleep116527-21/usr/sbin/munin12:12:561
670125112,23sleep26698-21apt-get09:57:512
2547025117,16sleep10-21swapper/110:57:561
1912025113,22sleep118629-21/usr/sbin/munin08:53:071
1182125114,20sleep00-21swapper/010:13:060
2802925012,17sleep20-21swapper/207:38:082
2802925012,17sleep20-21swapper/207:38:082
2031225013,22sleep220316-21diskstats08:57:572
1495625010,24sleep114950-21apt-get12:07:511
679024916,17sleep10-21swapper/109:57:581
3263624911,21sleep132644-21ntp_states07:53:051
1400324920,16sleep00-21swapper/012:03:060
1940524813,22sleep2281-21dbus-daemon12:22:372
1940524813,22sleep2281-21dbus-daemon12:22:372
2417724714,16sleep00-21swapper/007:27:400
61912459,17sleep10-21swapper/108:12:511
259402456,16sleep20-21swapper/207:32:512
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional