You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 10:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Thu Mar 12, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
215122212858,66sleep30-21swapper/319:05:383
21020128859,10sleep20-21swapper/219:05:012
21517128561,11sleep10-21swapper/119:06:161
21538827853,6sleep00-21swapper/019:09:000
215686992520,2cyclictest0-21swapper/323:15:203
21568699241,3cyclictest0-21swapper/322:55:213
215686992319,2cyclictest0-21swapper/321:10:173
215686992218,2cyclictest0-21swapper/322:05:193
21568699221,3cyclictest0-21swapper/322:23:223
21568699220,19cyclictest0-21swapper/321:53:233
215686992117,2cyclictest0-21swapper/300:05:003
21568699211,3cyclictest0-21swapper/323:23:033
21568699211,16cyclictest0-21swapper/300:36:573
21568699211,16cyclictest0-21swapper/300:05:103
215686992015,2cyclictest0-21swapper/323:35:473
215686992011,6cyclictest459190-21taskset22:12:593
215686992011,5cyclictest0-21swapper/323:58:563
21568699201,15cyclictest0-21swapper/321:45:183
215686992010,6cyclictest434170-21turbostat22:00:003
21568699193,13cyclictest0-21swapper/322:37:113
215686991916,2cyclictest0-21swapper/321:40:053
215686991916,2cyclictest0-21swapper/321:40:053
21568699191,15cyclictest0-21swapper/323:45:173
21568699191,15cyclictest0-21swapper/321:39:473
215686991911,4cyclictest6706292sleep300:13:273
21568699191,14cyclictest0-21swapper/323:07:473
21568699191,14cyclictest0-21swapper/322:30:103
21568699191,14cyclictest0-21swapper/322:26:133
21568699190,2cyclictest0-21swapper/322:16:503
6950602181,4sleep0695073-21rm00:25:380
6276662181,12sleep2345742-21diskmemload23:49:492
215686991813,2cyclictest0-21swapper/300:24:503
21568699181,14cyclictest0-21swapper/323:41:253
21568699181,14cyclictest0-21swapper/323:33:093
21568699181,14cyclictest0-21swapper/323:26:283
21568699181,14cyclictest0-21swapper/323:00:233
21568699181,14cyclictest0-21swapper/322:44:433
21568699181,14cyclictest0-21swapper/322:01:393
21568699181,14cyclictest0-21swapper/321:23:143
21568699181,14cyclictest0-21swapper/300:26:363
21568699181,14cyclictest0-21swapper/300:17:273
21568699180,14cyclictest0-21swapper/321:25:583
7048442171,12sleep0704858-21ssh00:32:000
6093172171,3sleep00-21swapper/023:39:040
5462852171,5sleep0171rcu_preempt23:02:000
5297122171,3sleep00-21swapper/022:53:140
5090522171,2sleep00-21swapper/022:40:190
3966632171,5sleep1345742-21diskmemload21:37:071
3779172171,4sleep1345742-21diskmemload21:25:281
21568699177,6cyclictest324456-21turbostat20:50:003
21568699176,8cyclictest292493-21turbostat20:24:563
215686991713,2cyclictest0-21swapper/322:46:073
215686991713,1cyclictest0-21swapper/323:10:183
21568699171,13cyclictest0-21swapper/319:45:233
21568699170,4cyclictest2497732sleep319:38:323
21568699170,14cyclictest0-21swapper/323:50:523
21568699170,14cyclictest0-21swapper/322:50:153
21568699170,13cyclictest0-21swapper/321:15:393
21568699170,13cyclictest0-21swapper/300:30:103
7153322161,4sleep1345742-21diskmemload00:39:181
6613612161,3sleep1345742-21diskmemload00:07:431
5658402161,4sleep1345742-21diskmemload23:14:441
5209802161,4sleep1345742-21diskmemload22:48:381
4345052161,2sleep10-21swapper/122:00:081
4227302161,10sleep20-21swapper/221:51:512
4221962161,3sleep00-21swapper/021:51:090
3812152161,4sleep0345742-21diskmemload21:29:480
21568699161,3cyclictest0-21swapper/321:30:063
215686991611,3cyclictest0-21swapper/321:00:103
21568699161,13cyclictest0-21swapper/319:50:143
215686991611,2cyclictest0-21swapper/321:10:003
21568699161,12cyclictest0-21swapper/320:45:003
21568699160,13cyclictest0-21swapper/320:25:203
21568699160,13cyclictest0-21swapper/319:11:393
7045702151,2sleep20-21swapper/200:31:382
5568632151,3sleep0556877-21ssh23:09:260
5385222151,8sleep1345742-21diskmemload22:58:201
4932522150,3sleep20-21swapper/222:31:402
4673142150,3sleep00-21swapper/022:17:060
215686991513,1cyclictest0-21swapper/319:45:003
215686991513,1cyclictest0-21swapper/319:45:003
21568699151,2cyclictest0-21swapper/319:20:113
215686991512,1cyclictest0-21swapper/320:50:113
215686991512,1cyclictest0-21swapper/320:15:183
21568699150,13cyclictest0-21swapper/320:10:173
21568699150,12cyclictest0-21swapper/320:55:193
21568699150,12cyclictest0-21swapper/320:35:113
21568699150,12cyclictest0-21swapper/320:05:163
21568699150,12cyclictest0-21swapper/319:33:103
6976692141,2sleep10-21swapper/100:29:041
6874102140,1sleep10-21swapper/100:22:071
6680122141,9sleep2668159-21/usr/sbin/munin00:10:252
6634192141,4sleep1663430-21ssh00:10:081
5044262141,8sleep2345742-21diskmemload22:39:532
4686052141,9sleep2345742-21diskmemload22:18:492
4389282141,9sleep0439061-21mailstats22:00:220
4047192141,3sleep0345742-21diskmemload21:41:100
4047192141,3sleep0345742-21diskmemload21:41:100
3535122140,3sleep1345742-21diskmemload21:13:451
21568699142,8cyclictest307476-21taskset20:30:183
215686991412,1cyclictest0-21swapper/319:15:013
215686991411,1cyclictest0-21swapper/320:00:173
215686991411,1cyclictest0-21swapper/319:55:193
21568699140,2cyclictest0-21swapper/319:25:123
6521852131,3sleep1345742-21diskmemload00:02:061
6512472131,8sleep2651261-21ssh00:00:512
6237432131,2sleep10-21swapper/123:45:231
6040382131,7sleep2605669-21grep23:35:172
5740382130,8sleep0345742-21diskmemload23:19:000
4919912131,8sleep0492241-21sed22:30:250
4769162130,2sleep1345742-21diskmemload22:23:131
3215542131,2sleep2323270-21cat20:45:172
2782132131,4sleep1279971-21tr20:05:161
6626622121,3sleep0662676-21ssh00:09:270
6512752120,8sleep0651285-21ssh00:00:530
6424792121,7sleep0642494-21ssh23:55:560
6180462125,3sleep2618057-21sh23:43:392
6090712121,6sleep1345742-21diskmemload23:38:451
5670592122,5sleep20-21swapper/223:15:172
5005042121,7sleep0496156-21/usr/sbin/munin22:35:220
4948042126,1sleep10-21swapper/122:33:431
4821592121,5sleep2482673-21memory22:25:182
4573922121,7sleep1345742-21diskmemload22:10:381
4512322121,6sleep1345742-21diskmemload22:09:021
3960632121,7sleep2396080-21ssh21:36:192
3712112121,7sleep0371222-21ssh21:23:110
3524492121,7sleep0352463-21ssh21:12:200
3300272122,5sleep1330046-21apt-get20:55:101
3074222122,5sleep2307468-21ntpq20:30:182
2997682121,7sleep2301565-21grep20:25:172
2875232121,5sleep2287725-21diskstats20:15:152
2808642121,5sleep0280914-21ntpq20:05:190
2380562121,5sleep0238172-21ntpq19:25:190
6853162110,2sleep00-21swapper/000:20:170
6801252111,4sleep1345742-21diskmemload00:19:261
6787912110,3sleep0678802-21ssh00:17:410
6724882115,2sleep20-21swapper/200:15:142
6546612110,5sleep20-21swapper/200:05:042
6429062111,2sleep20-21swapper/223:56:302
6365672111,5sleep1636649-21munin-run23:55:011
6255312110,5sleep00-21swapper/023:46:580
6175572111,4sleep00-21swapper/023:43:010
6170102110,6sleep1345742-21diskmemload23:42:181
5980212111,3sleep20-21swapper/223:30:402
5891122111,3sleep20-21swapper/223:25:272
5845032111,6sleep0584724-21hddtemp_smartct23:25:130
5820252111,5sleep1345742-21diskmemload23:23:021
5565712111,6sleep1345742-21diskmemload23:09:031
5465982111,5sleep1345742-21diskmemload23:02:261
5371472111,6sleep0537159-21ssh22:56:310
5282102111,4sleep1345742-21diskmemload22:51:151
5048452111,3sleep20-21swapper/222:40:052
4969042111,6sleep1498488-21tr22:35:151
4741242111,5sleep2474180-21ntpq22:20:212
4580572111,3sleep00-21swapper/022:11:290
4476902111,5sleep2447791-21ntpq22:05:182
4107312111,3sleep20-21swapper/221:45:172
4052492111,2sleep20-21swapper/221:41:532
4052492111,2sleep20-21swapper/221:41:532
4042812110,1sleep10-21swapper/121:40:351
4042812110,1sleep10-21swapper/121:40:351
3817382110,4sleep20-21swapper/221:30:102
3648132111,6sleep1365325-21grep21:20:151
3409512111,5sleep0341161-21hddtemp_smartct21:05:150
3368332111,6sleep2338611-21cut21:00:152
3072802111,5sleep1307421-21ntpq20:30:171
2758602111,7sleep0275888-21sed20:00:260
7133112101,2sleep20-21swapper/200:36:392
7081452101,6sleep0709558-21kthreadcore00:35:160
7043012101,3sleep1345742-21diskmemload00:31:171
6899062101,6sleep20-21swapper/200:25:042
6813072101,4sleep2681516-21diskstats00:20:112
6679342100,1sleep00-21swapper/000:10:230
6436502101,5sleep1643662-21rm23:57:291
6283712101,4sleep2628505-21diskstats23:50:132
5996302101,4sleep1345742-21diskmemload23:32:481
5816532101,2sleep00-21swapper/023:22:330
5723782101,2sleep10-21swapper/123:16:481
5644672100,6sleep0345742-21diskmemload23:12:550
5530912101,5sleep2553134-21ntpq23:05:192
5458972101,3sleep20-21swapper/223:01:302
5367092101,4sleep2345742-21diskmemload22:55:572
5287252101,2sleep20-21swapper/222:51:562
5199022101,3sleep00-21swapper/022:47:130
5187992100,6sleep2345742-21diskmemload22:45:472
5098892101,3sleep1345742-21diskmemload22:40:321
4828282101,4sleep0482879-21ntpq22:25:200
4787562101,4sleep1478970-21diskstats22:25:131
4780202101,4sleep0345742-21diskmemload22:24:400
4663982101,3sleep1345742-21diskmemload22:15:551
4600072101,5sleep2345742-21diskmemload22:14:042
4388002101,5sleep2438859-21ntpq22:00:212
4339382100,2sleep10-21swapper/121:59:441
4318432101,3sleep00-21swapper/021:56:580
4241912100,1sleep10-21swapper/121:53:461
4159042101,3sleep1345742-21diskmemload21:49:211
4079112101,5sleep0407925-21ssh21:45:030
3890072101,4sleep1345742-21diskmemload21:33:341
3859372101,4sleep0385997-21ntpq21:30:200
3785362101,2sleep20-21swapper/221:26:172
3641342101,3sleep2363572-21apt-get21:20:042
3604692101,3sleep1345742-21diskmemload21:15:581
3598262101,4sleep2360083-21users21:15:272
3300492101,4sleep2330301-21df20:55:112
2985712101,4sleep0298636-21irqstats20:25:150
2827042101,5sleep10-21swapper/120:10:171
2552372101,5sleep1255256-21munin-run19:45:001
2552372101,5sleep1255256-21munin-run19:44:591
2273632101,5sleep1227488-21/usr/sbin/munin19:15:181
2234372101,4sleep2223456-21apt-get19:15:112
2211522101,5sleep0221460-21kthreadcore19:15:010
633505291,3sleep0345742-21diskmemload23:50:590
598405290,2sleep00-21swapper/023:31:100
592559291,4sleep1592570-21ssh23:29:591
559082291,4sleep2560749-21grep23:10:162
449663291,4sleep0449677-21ssh22:06:560
434148291,4sleep2434261-21latency_hist22:00:012
395421291,2sleep00-21swapper/021:35:290
355084291,2sleep00-21swapper/021:15:090
351507290,1sleep20-21swapper/221:11:062
344711291,4sleep2344763-21ld21:05:212
324726291,4sleep0324981-21df20:50:120
319399292,3sleep1319575-21expr20:45:121
302126291,4sleep1302177-21ntpq20:25:201
296761291,4sleep1296809-21ntpq20:20:191
287438291,4sleep0287457-21apt-get20:15:130
270194291,4sleep2270246-21modprobe19:55:192
260443290,1sleep10-21swapper/119:48:451
250741291,4sleep0252032-21kthreadcore19:40:140
250741291,4sleep0252032-21kthreadcore19:40:140
249687290,2sleep00-21swapper/019:37:260
248963291,4sleep2249012-21/usr/sbin/munin19:35:192
233302291,4sleep2233486-21users19:20:242
216008291,4sleep2215479-21apt-get19:10:022
575188281,2sleep2575217-21apt-get23:20:102
324142280,1sleep00-21swapper/020:46:110
314584281,3sleep0314886-21kthreadcore20:40:140
308702281,2sleep20-21swapper/220:35:112
288007282,3sleep1289170-21kthreadcore20:15:181
287177280,5sleep0287323-21latency_hist20:15:010
265824280,1sleep10-21swapper/119:54:261
265545280,3sleep20-21swapper/219:50:542
255519281,4sleep2255686-21cstates19:45:132
244841281,3sleep1245044-21cstates19:35:101
234676281,3sleep1235395-21kthreadcore19:25:161
345464271,3sleep1345465-21pool-udisksd21:06:401
339382271,3sleep1339429-21sed21:00:191
334893270,1sleep00-21swapper/020:57:250
328901271,3sleep2329033-21mailstats20:50:222
327541271,3sleep1328606-21/usr/sbin/munin20:50:171
318426271,3sleep20-21swapper/220:40:242
314889271,3sleep1316576-21kthreadcore20:40:151
308184270,1sleep00-21swapper/020:31:440
297780270,1sleep00-21swapper/020:24:390
287119271,3sleep2287120-21sshd20:14:192
260824271,2sleep00-21swapper/019:50:120
255791271,2sleep0256008-21smartctl19:45:150
244498270,1sleep00-21swapper/019:34:130
239445271,2sleep2239464-21apt-get19:30:102
233643270,1sleep00-21swapper/019:22:190
335376261,2sleep00-21swapper/021:00:100
293335261,2sleep2294545-21kthreadcore20:20:162
281568260,1sleep20-21swapper/220:06:092
276529260,1sleep20-21swapper/220:04:492
271044260,1sleep00-21swapper/019:58:060
270891260,1sleep10-21swapper/119:56:101
255233260,4sleep20-21swapper/219:44:572
255233260,4sleep20-21swapper/219:44:572
244169260,1sleep10-21swapper/119:30:371
233677260,1sleep10-21swapper/119:22:441
220861260,4sleep00-21swapper/019:11:210
216036260,1sleep10-21swapper/119:10:091
308700250,2sleep10-21swapper/120:35:101
276272250,3sleep10-21swapper/120:01:311
238227250,1sleep20-21swapper/219:25:212
308482240,1sleep00-21swapper/020:35:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional