You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-10 - 17:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed Jun 10, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
284962766313,7636sleep122815-21kworker/1:009:19:231
7225251412,489sleep27233-21ntpq08:13:292
9958247922,438sleep39967-21/usr/sbin/munin08:23:073
27146242812,310sleep127287-21head10:58:201
3266233011,268sleep13269-21acpi11:27:591
14066231111,283sleep114172-21latency_hist12:02:521
3055230810,256sleep13195-21latency_hist11:27:511
4937225483,98sleep10-21swapper/109:48:191
2903422510,6sleep20-21swapper/209:22:592
4142223427,186sleep24286-21cat09:47:512
26322223267,91sleep10-21swapper/107:32:521
15159222417,128sleep10-21swapper/112:03:321
27294222348,112sleep127417-21sed07:33:211
27294222348,112sleep127417-21sed07:33:211
15300221821,177sleep115441-21latency_hist10:22:511
32728221637,107sleep10-21swapper/111:18:061
9833221526,103sleep10-21swapper/110:03:211
9833221526,103sleep10-21swapper/110:03:211
17064221578,89sleep10-21swapper/110:28:041
29025221415,178sleep129026-21/usr/sbin/munin09:22:591
24604219266,108sleep124678-21latency_hist10:52:511
24141219212,110sleep124209-21latency_hist09:07:521
32713218769,26sleep10-21swapper/107:53:001
24742218012,155sleep124718-21latency_hist07:27:521
19721218046,34sleep10-21swapper/108:53:301
11905217319,127sleep30-21swapper/308:28:163
1595021689,147sleep115970-21munin-run08:42:521
8608216654,88sleep10-21swapper/108:18:121
12042215845,18sleep10-21swapper/108:28:181
28475215528,88sleep128520-21grep07:38:141
24132215511,126sleep10-21swapper/112:33:131
23894215517,123sleep323932-21processes07:23:173
18717215314,119sleep10-21swapper/110:33:091
4557215217,121sleep2225-21systemd-journal11:32:492
3029021487,127sleep130419-21awk11:08:211
3029021487,127sleep130419-21awk11:08:211
23475214537,33sleep123491-21modprobe12:32:491
32407214014,28sleep132422-21latency_hist11:17:501
11430214013,29sleep011424-21apt-get08:28:010
672121387,68sleep10-21swapper/108:12:591
5473213657,19sleep15566-21mailstats08:08:191
864721359,109sleep18796-21tail11:43:221
22259213413,97sleep122302-21ntp_kernel_pll_07:18:141
2178621345,33sleep10-21swapper/112:26:021
11282213457,19sleep111281-21ntp_states10:08:141
18814213314,94sleep218858-21/usr/sbin/munin10:33:102
16741213048,21sleep116777-21ntpq08:43:161
19157212920,35sleep119186-21idleruntime-cro08:52:491
26180212722,89sleep126307-21latency_hist10:57:521
26180212722,89sleep126307-21latency_hist10:57:521
2321212755,14sleep10-21swapper/107:58:171
13962212719,90sleep014062-21latency_hist12:02:510
6399212631,73sleep10-21swapper/109:53:201
6399212631,73sleep10-21swapper/109:53:201
14316212630,26sleep10-21swapper/108:37:511
7698212527,83sleep17717-21apt-get09:58:001
7698212527,83sleep17717-21apt-get09:58:001
6422212574,15sleep10-21swapper/111:38:051
20354212316,89sleep10-21swapper/107:13:051
8005212224,73sleep38022-21cut09:58:233
8005212224,73sleep38022-21cut09:58:233
17242212233,77sleep11-21systemd12:12:491
1152321226,105sleep111557-21ps11:53:191
11143212112,89sleep111164-21modprobe08:27:491
30841211923,29sleep10-21swapper/107:47:021
4820211728,18sleep10-21swapper/111:33:001
4820211728,18sleep10-21swapper/111:33:001
2968921167,12sleep00-21swapper/011:08:080
2968921167,12sleep00-21swapper/011:08:080
27591211421,70sleep327607-21fschecks_count09:18:083
20430211431,26sleep120524-21latency_hist12:22:511
1471211425,24sleep11242-21/usr/sbin/munin09:38:091
7694211310,88sleep27191-21apt-get09:57:572
7694211310,88sleep27191-21apt-get09:57:572
1741521125,95sleep00-21swapper/008:45:250
7729211112,84sleep37845-21cut11:42:513
30828210810,79sleep230929-21latency_hist11:12:502
30828210810,79sleep230929-21latency_hist11:12:502
2825321088,14sleep00-21swapper/011:03:120
9343210611,13sleep30-21swapper/311:47:523
4842210632,49sleep24839-21apt-get11:33:012
4842210632,49sleep24839-21apt-get11:33:012
14926210614,72sleep214984-21/usr/sbin/munin12:03:292
1882321056,85sleep10-21swapper/107:08:171
1368821040,11sleep013683-21modprobe10:17:490
17217210318,18sleep10-21swapper/107:03:161
4678210216,68sleep04709-21munin-run08:07:490
27096210218,65sleep327165-21switchtime07:33:193
27096210218,65sleep327165-21switchtime07:33:193
321221017,82sleep13256-21ntpq09:43:141
31157210120,57sleep331199-21fschecks_count11:13:123
3270210013,70sleep33278-21ntpq09:43:163
26525210028,45sleep026519-21apt-get07:33:040
26525210028,45sleep026519-21apt-get07:33:040
21645210013,72sleep221715-21latency_hist07:17:512
38782997,37sleep13912-21ps08:03:181
2659829911,74sleep226647-21idleruntime10:58:092
379529714,61sleep33878-21mailstats11:28:183
1837429720,58sleep318389-21latency_hist10:32:513
163772967,25sleep10-21swapper/110:23:201
717229529,44sleep27177-21modprobe09:57:492
717229529,44sleep27177-21modprobe09:57:492
3191329559,22sleep132054-21date09:32:521
3191329559,22sleep132054-21date09:32:521
2021629514,62sleep320289-21apt07:12:593
2510129421,52sleep00-21swapper/010:53:100
2510129421,52sleep00-21swapper/010:53:100
2881429316,32sleep20-21swapper/209:22:512
1312629310,62sleep10-21swapper/110:13:181
9492929,71sleep1993-21basename09:37:491
1996329116,54sleep020089-21latency_hist10:37:520
187222915,34sleep10-21swapper/112:16:491
1239929016,55sleep212525-21latency_hist11:57:522
2455128913,18sleep30-21swapper/310:52:493
1733228914,59sleep017402-21cat10:28:110
2156728815,18sleep30-21swapper/310:42:523
2134728819,44sleep021497-21unixbench_multi12:23:190
1527028616,55sleep015299-21latency_hist10:22:500
1417328633,26sleep214372-21sed12:02:532
108142867,67sleep110823-21munin-run11:52:501
622328517,53sleep36240-21awk08:12:493
213228519,34sleep00-21swapper/007:58:120
121592850,6sleep20-21swapper/208:28:182
904728424,40sleep09024-21latency_hist10:02:520
904728424,40sleep09024-21latency_hist10:02:520
2125928419,18sleep10-21swapper/108:58:031
1231728423,32sleep30-21swapper/311:57:493
713628322,42sleep37167-21/usr/sbin/munin08:13:263
284322828,63sleep128469-21ps11:03:171
259202827,13sleep10-21swapper/109:13:011
2252528224,39sleep322539-21apt09:02:493
41528015,44sleep0549-21unixbench_singl09:33:200
243702806,12sleep10-21swapper/109:08:001
1923328019,46sleep319356-21latency_hist08:52:523
120328013,53sleep0363-21snmpd11:19:350
311912797,31sleep10-21swapper/107:48:031
311912797,31sleep10-21swapper/107:48:031
1827527920,36sleep318404-21grep12:13:203
1093927923,25sleep20-21swapper/210:08:072
310592788,44sleep00-21swapper/011:13:010
55682778,55sleep25644-21find08:08:202
2156427725,33sleep021598-21modprobe07:17:490
155752776,10sleep00-21swapper/012:06:220
609627612,48sleep06149-21if_err_eth009:53:100
609627612,48sleep06149-21if_err_eth009:53:100
2305627620,27sleep20-21swapper/210:47:512
2229327613,42sleep322192-21/usr/sbin/munin12:28:063
207312766,11sleep10-21swapper/110:38:181
1574327613,47sleep315841-21date12:07:503
34942758,19sleep30-21swapper/308:03:083
2873927516,17sleep30-21swapper/307:38:203
2573527513,25sleep30-21swapper/310:53:223
2573527513,25sleep30-21swapper/310:53:223
1713227532,23sleep30-21swapper/307:03:143
225422747,31sleep10-21swapper/110:43:221
1968027422,32sleep019811-21latency_hist07:12:520
1959927414,37sleep219093-21/usr/sbin/munin12:18:162
511627312,26sleep00-21swapper/011:33:110
511627312,26sleep00-21swapper/011:33:110
2425827322,28sleep20-21swapper/212:33:182
2203227318,29sleep00-21swapper/010:43:130
1612927317,35sleep316224-21timerandwakeup10:23:183
540327210,49sleep0363-21snmpd09:50:000
2952027212,28sleep329515-21apt-get11:08:033
2952027212,28sleep329515-21apt-get11:08:033
2923327222,34sleep21-21systemd11:07:502
2483527212,47sleep024841-21ntpq09:08:150
2399727219,31sleep20-21swapper/210:48:202
2372727211,43sleep223795-21memory07:23:122
1643127213,37sleep316474-21fschecks_count08:43:103
284952715,15sleep00-21swapper/009:19:220
27692716,12sleep30-21swapper/307:58:233
2433827123,20sleep30-21swapper/312:33:193
234327116,37sleep22405-21/usr/sbin/munin07:58:182
1604127119,33sleep216111-21latency_hist08:42:532
907026914,40sleep29211-21unixbench_singl08:18:212
562426915,35sleep35659-21awk09:52:493
2819326922,26sleep00-21swapper/007:38:090
1622826927,22sleep00-21swapper/010:23:190
154226933,18sleep20-21swapper/207:57:492
1495326917,32sleep314994-21ntp_kernel_pll_08:38:133
1364826925,30sleep313673-21munin-run10:17:493
54826815,38sleep0590-21latency07:53:110
2464426822,25sleep30-21swapper/309:08:103
1417326714,40sleep314211-21if_err_eth010:18:093
539326616,25sleep30-21swapper/311:33:183
539326616,25sleep30-21swapper/311:33:183
3124126627,22sleep30-21swapper/307:48:053
3124126627,22sleep30-21swapper/307:48:053
302562666,19sleep30-21swapper/309:27:093
1869826612,38sleep318705-21diskstats10:33:073
182826616,33sleep21838-21/usr/sbin/munin11:23:082
1685026620,22sleep20-21swapper/207:03:092
163226614,32sleep01661-21/usr/sbin/munin09:38:120
1424026618,22sleep00-21swapper/008:37:500
1106326618,25sleep00-21swapper/011:53:060
3209126518,28sleep20-21swapper/211:14:482
3191226517,29sleep232055-21grep07:48:202
3191226517,29sleep232055-21grep07:48:202
293152655,10sleep20-21swapper/207:42:192
2385026523,27sleep00-21swapper/012:33:080
1338526515,21sleep30-21swapper/308:33:173
1119626514,23sleep00-21swapper/008:27:520
1059326522,23sleep20-21swapper/210:07:502
1059326522,23sleep20-21swapper/210:07:502
974826416,32sleep39785-21ps10:03:193
974826416,32sleep39785-21ps10:03:193
1861626414,31sleep318675-21/usr/sbin/munin07:08:113
1760926421,26sleep20-21swapper/212:13:072
113242647,12sleep00-21swapper/010:08:170
988426317,28sleep00-21swapper/011:48:130
94182635,11sleep00-21swapper/008:20:330
879826315,22sleep20-21swapper/211:43:232
44392635,47sleep00-21swapper/011:31:220
3214326317,28sleep232139-21apt-get09:33:012
2326826310,38sleep023275-21ntp_states09:03:150
2150726324,18sleep20-21swapper/210:42:512
2123426319,24sleep20-21swapper/208:58:002
17202635,15sleep30-21swapper/309:38:153
170826313,34sleep21719-21ntp_states09:38:142
1677426319,20sleep20-21swapper/210:27:492
1327226318,28sleep213285-21vmstat10:13:202
3275826221,26sleep3194-21jbd2/mmcblk2p2-09:33:193
3031026221,20sleep00-21swapper/007:43:220
2369026215,25sleep20-21swapper/209:03:212
2275126216,19sleep00-21swapper/012:28:190
1653526212,25sleep20-21swapper/212:08:222
2773526112,20sleep00-21swapper/011:02:510
2559726114,27sleep225734-21/usr/sbin/munin10:53:212
2559726114,27sleep225734-21/usr/sbin/munin10:53:212
1816926120,26sleep218258-21load08:48:102
1793726113,21sleep00-21swapper/008:48:040
1408726115,19sleep20-21swapper/210:18:082
12522615,12sleep30-21swapper/311:20:143
2320626017,19sleep20-21swapper/212:29:222
1990026018,22sleep00-21swapper/008:54:220
1791626011,34sleep013899-21kworker/0:112:13:130
968225910,34sleep09720-21ntpq10:03:160
968225910,34sleep09720-21ntpq10:03:160
78725913,33sleep3881-21mailstats07:53:183
3120925916,29sleep231298-21sed09:28:212
3120925916,29sleep231298-21sed09:28:212
2854525921,21sleep30-21swapper/311:03:193
2518825915,29sleep325245-21irqstats07:28:103
2215325913,28sleep322242-21/usr/sbin/munin07:18:123
76062585,13sleep00-21swapper/011:41:550
702825817,24sleep27102-21awk11:38:222
244292588,37sleep00-21swapper/010:51:170
2018425815,24sleep20-21swapper/208:54:282
1920225811,35sleep019209-21/usr/sbin/munin12:18:050
164462589,13sleep00-21swapper/012:08:200
1635025812,30sleep316380-21awk12:08:153
727325715,26sleep00-21swapper/008:13:310
533725718,18sleep00-21swapper/008:08:130
1374925715,27sleep213866-21awk08:33:242
391525615,21sleep20-21swapper/208:03:192
2659425623,17sleep20-21swapper/207:33:062
2659425623,17sleep20-21swapper/207:33:062
193272565,11sleep20-21swapper/207:09:152
1121825613,22sleep30-21swapper/310:08:123
436425513,26sleep24357-21apt-get09:48:042
3250225520,20sleep00-21swapper/007:52:510
3250225520,20sleep00-21swapper/007:52:510
305442556,14sleep20-21swapper/207:43:242
30002555,39sleep00-21swapper/011:27:460
2653425510,25sleep30-21swapper/309:13:193
2592425513,22sleep00-21swapper/009:13:040
2321925511,23sleep20-21swapper/207:22:522
2262325517,17sleep00-21swapper/007:18:220
217942557,28sleep00-21swapper/008:58:180
2107925513,18sleep20-21swapper/212:23:122
1972325512,19sleep30-21swapper/308:53:323
1911625513,24sleep319111-21apt-get12:18:003
1731325520,22sleep017401-21cat07:03:190
1634525512,25sleep216334-21apt-get08:43:052
101392557,13sleep20-21swapper/211:48:192
1006725512,20sleep20-21swapper/208:23:102
87025417,18sleep20-21swapper/211:18:202
3078325416,24sleep030574-21/usr/sbin/munin09:28:110
3078325416,24sleep030574-21/usr/sbin/munin09:28:110
24542549,17sleep30-21swapper/311:23:213
1512825412,26sleep015197-21/usr/sbin/munin08:38:190
2616425311,25sleep226193-21irqstats09:13:102
2615225310,32sleep20-21swapper/207:31:442
2497825313,24sleep024972-21apt-get07:28:030
2477825315,21sleep20-21swapper/209:08:132
2288725317,19sleep30-21swapper/309:03:053
833425212,23sleep38331-21apt-get08:18:023
2909725210,27sleep029125-21df09:23:060
275262525,32sleep30-21swapper/311:00:483
2019125113,23sleep020193-21apt-get10:38:000
1390425117,19sleep30-21swapper/312:02:503
856925011,24sleep08646-21grep11:43:210
600725011,15sleep30-21swapper/309:53:093
600725011,15sleep30-21swapper/309:53:093
2972425012,22sleep329730-21diskstats07:43:083
28152509,16sleep00-21swapper/009:43:050
2383025016,19sleep00-21swapper/007:23:140
80362479,15sleep30-21swapper/311:43:073
3096624616,16sleep30-21swapper/309:28:143
3096624616,16sleep30-21swapper/309:28:143
1743124513,18sleep20-21swapper/210:28:132
2066524210,16sleep30-21swapper/312:23:013
205252428,19sleep320564-21memory10:38:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional