You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 16:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Wed Nov 19, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
191100211759,53sleep30-21swapper/307:08:093
19117528772,9sleep20-21swapper/207:09:052
19096228656,9sleep10-21swapper/107:06:221
19112228551,6sleep00-21swapper/007:08:250
191390992824,2cyclictest0-21swapper/310:50:003
19139099281,9cyclictest220275-21taskset07:35:013
191390992522,1cyclictest0-21swapper/311:40:003
191390992522,1cyclictest0-21swapper/307:20:263
19139099242,14cyclictest428817-21taskset10:20:273
191390992419,2cyclictest0-21swapper/309:35:183
191390992418,3cyclictest0-21swapper/310:50:193
19139099241,19cyclictest566011-21taskset11:55:123
4265482235,13sleep2428210-21cat10:20:202
191390992319,2cyclictest0-21swapper/311:15:183
191390992312,9cyclictest328124-21chrt09:12:563
19139099227,7cyclictest241835-21taskset07:55:133
191390992219,1cyclictest0-21swapper/311:50:293
191390992210,8cyclictest271073-21taskset08:20:183
19139099213,14cyclictest66550irq/131-enp1s0-TxRx-311:47:323
19139099212,13cyclictest415383-21taskset10:12:493
191390992118,1cyclictest0-21swapper/309:20:283
191390992117,2cyclictest0-21swapper/311:05:163
191390992117,2cyclictest0-21swapper/310:40:283
191390992116,2cyclictest0-21swapper/310:00:263
191390992113,4cyclictest3492042sleep309:27:363
19139099211,15cyclictest433724-21chrt10:25:203
3632732201,2sleep00-21swapper/009:36:070
19139099207,9cyclictest311099-21sleep09:00:013
19139099202,13cyclictest402852-21taskset10:05:153
191390992015,3cyclictest0-21swapper/311:00:003
191390992015,2cyclictest0-21swapper/310:15:173
19139099201,15cyclictest66550irq/131-enp1s0-TxRx-310:39:123
19139099201,15cyclictest445214-21chrt10:33:223
19139099201,13cyclictest321695-21turbostat09:10:013
4978212191,4sleep0321671-21diskmemload11:07:270
4768092191,13sleep2476309-21apt-get10:55:022
19139099192,12cyclictest378090-21taskset09:46:113
19139099192,12cyclictest351199-21taskset09:30:163
191390991917,1cyclictest0-21swapper/308:35:003
191390991916,2cyclictest0-21swapper/311:29:333
191390991916,1cyclictest0-21swapper/311:30:133
191390991916,1cyclictest0-21swapper/309:40:073
191390991916,1cyclictest0-21swapper/309:20:013
19139099191,14cyclictest519298-21chrt11:21:013
19139099191,14cyclictest0-21swapper/312:25:143
19139099191,14cyclictest0-21swapper/312:09:033
4247992185,8sleep0424943-21cat10:20:170
4064852182,4sleep1406542-21ntpq10:05:221
2896892181,11sleep0289892-21diskstats08:40:170
19139099182,12cyclictest549938-21taskset11:44:183
191390991815,1cyclictest0-21swapper/307:50:123
191390991813,2cyclictest0-21swapper/307:45:003
19139099181,15cyclictest0-21swapper/312:00:043
19139099181,14cyclictest0-21swapper/312:20:203
19139099181,14cyclictest0-21swapper/311:01:433
19139099181,13cyclictest593581-21chrt12:11:003
19139099181,13cyclictest0-21swapper/311:10:123
19139099181,13cyclictest0-21swapper/309:56:173
19139099181,13cyclictest0-21swapper/309:50:063
19139099180,14cyclictest0-21swapper/312:30:233
5142252171,3sleep00-21swapper/011:20:140
4810482171,11sleep1481105-21ntpq10:55:211
3853182171,10sleep1321671-21diskmemload09:50:561
3577992171,5sleep1357947-21cat09:35:011
3571632171,2sleep20-21swapper/209:33:452
2860702170,1sleep30-21swapper/308:35:183
19139099175,1cyclictest0-21swapper/308:00:193
19139099172,12cyclictest267479-21chrt08:15:293
19139099171,13cyclictest631907-21chrt12:37:293
19139099171,12cyclictest191568-21chrt07:10:123
19139099171,12cyclictest0-21swapper/312:15:053
19139099171,10cyclictest289595-21chrt08:40:063
6038052161,10sleep2802-21snmpd12:20:142
5641012161,11sleep2564115-21ssh11:52:162
4456072161,3sleep00-21swapper/010:34:120
4445832161,3sleep10-21swapper/110:32:001
4388352161,2sleep20-21swapper/210:30:022
3942592161,7sleep2321671-21diskmemload09:59:222
3431292161,10sleep0343285-21latency_hist09:25:020
191390991613,1cyclictest0-21swapper/308:45:183
191390991612,2cyclictest0-21swapper/308:25:123
19139099161,10cyclictest301923-21taskset08:50:163
19139099160,13cyclictest0-21swapper/308:10:233
19139099160,13cyclictest0-21swapper/307:27:153
6267492151,10sleep0627926-21kthreadcore12:35:180
6155632152,8sleep0615618-21/usr/sbin/munin12:25:240
5880852151,8sleep2321671-21diskmemload12:09:472
5656142150,3sleep0321671-21diskmemload11:54:530
5428172150,3sleep20-21swapper/211:39:362
5215132151,8sleep0321671-21diskmemload11:25:120
4595622151,2sleep20-21swapper/210:42:092
4520222151,2sleep00-21swapper/010:37:160
4494182151,3sleep2450312-21ssh10:35:222
4013602151,2sleep2321671-21diskmemload10:03:442
3721682151,8sleep0321671-21diskmemload09:44:020
3290912150,1sleep20-21swapper/209:14:402
1921032150,3sleep20-21swapper/207:10:152
19139099158,4cyclictest219500-21chrt07:30:283
19139099151,10cyclictest196830-21chrt07:15:023
19139099150,13cyclictest0-21swapper/308:05:163
19139099150,12cyclictest0-21swapper/307:45:113
19139099150,12cyclictest0-21swapper/307:45:113
6119722141,8sleep2612883-21cut12:25:182
5950882141,4sleep2595099-21ssh12:14:212
5633022141,9sleep1563317-21ssh11:50:501
5433822142,6sleep2321671-21diskmemload11:40:132
5393212141,8sleep1540195-21grep11:35:181
5186932141,8sleep2518733-21wc11:20:282
4696892141,8sleep2470778-21cat10:50:182
4679212141,7sleep2321671-21diskmemload10:48:292
4614212141,3sleep1461442-21apt-get10:45:111
4249442141,8sleep1426547-21cut10:20:191
4098172141,7sleep0409891-21tune2fs10:10:150
3556382147,4sleep0355644-21sh09:30:470
3369822141,9sleep2321671-21diskmemload09:20:182
3337242141,9sleep2333774-21ntpq09:15:212
3223082141,7sleep1321671-21diskmemload09:10:141
19139099140,11cyclictest0-21swapper/309:00:313
5885572132,5sleep0588570-21apt-get12:10:130
5716762131,8sleep1571686-21ssh11:57:031
5289262131,6sleep0528950-21ssh11:30:120
4725482131,6sleep0473031-21memory10:50:190
4594122131,6sleep1321671-21diskmemload10:41:501
4386402131,4sleep1438661-21latency_hist10:30:001
4164712131,6sleep1416630-21cut10:15:021
4100942130,1sleep2410514-21cut10:10:182
2251812131,7sleep187-21kswapd007:37:361
6247452121,5sleep1321671-21diskmemload12:32:471
6171712121,4sleep1321671-21diskmemload12:27:341
5802312120,9sleep00-21swapper/012:03:550
5730302121,5sleep00-21swapper/011:59:340
5659352120,1sleep20-21swapper/211:55:032
4911742121,4sleep1321671-21diskmemload11:04:151
4908242121,6sleep2490833-21ssh11:03:332
4843662121,6sleep0321671-21diskmemload11:00:050
4762992121,8sleep00-21swapper/010:55:010
4748782121,4sleep1321671-21diskmemload10:52:211
4529302121,4sleep1321671-21diskmemload10:38:501
4220722121,5sleep0321671-21diskmemload10:16:160
3877392121,6sleep1387768-21apt-get09:55:121
3860892121,5sleep00-21swapper/009:52:200
3782262121,5sleep20-21swapper/209:46:272
3730942121,7sleep1373251-21wc09:45:141
3499722121,7sleep1321671-21diskmemload09:29:231
3490672121,3sleep0349073-21ssh09:27:170
2581282121,7sleep2258329-21kthreadcore08:10:162
2298042120,1sleep20-21swapper/207:40:222
2215422121,7sleep0223289-21kthreadcore07:35:190
6322422110,2sleep2171rcu_preempt12:38:092
6025812111,4sleep1321671-21diskmemload12:18:281
5877162111,3sleep1321671-21diskmemload12:09:061
5797432110,6sleep1579749-21ssh12:03:001
5785082111,2sleep20-21swapper/212:00:342
5575492111,4sleep1321671-21diskmemload11:49:411
5560002110,2sleep2556010-21ssh11:46:342
5500382111,6sleep1321671-21diskmemload11:44:281
5347682111,3sleep20-21swapper/211:32:392
5339442111,6sleep1533948-21ssh11:30:531
5209762111,3sleep1321671-21diskmemload11:24:361
5122482111,3sleep20-21swapper/211:16:232
5058302111,3sleep10-21swapper/111:13:251
5055712111,4sleep2321671-21diskmemload11:12:542
5045342111,4sleep0321671-21diskmemload11:10:560
4925752110,7sleep10-21swapper/111:05:201
4681372111,3sleep00-21swapper/010:48:550
4383052111,4sleep2321671-21diskmemload10:29:212
4231572111,4sleep1321671-21diskmemload10:18:211
4075392111,4sleep2321671-21diskmemload10:06:202
4011632111,4sleep1321671-21diskmemload10:03:231
3990692111,5sleep0399116-21ntpq10:00:200
3784372111,3sleep00-21swapper/009:46:530
3709122111,3sleep10-21swapper/109:41:221
3697612111,5sleep2369812-21ntpq09:40:222
3639222110,4sleep1321671-21diskmemload09:37:281
3631052111,4sleep2321671-21diskmemload09:35:412
3437372111,5sleep2343949-21hddtemp_smartct09:25:152
3421882111,3sleep10-21swapper/109:22:421
3359312111,3sleep10-21swapper/109:19:331
3229782111,6sleep0324124-21kthreadcore09:10:190
3050692111,6sleep0305290-21unixbench_multi08:50:270
2963612111,6sleep0298085-21kthreadcore08:45:170
2721572111,4sleep2272209-21ntpq08:20:222
2408662111,6sleep0241089-21unixbench_multi07:50:260
2102692111,6sleep0210540-21kthreadcore07:25:170
2041592111,5sleep2204411-21df07:20:142
6302872101,4sleep1630339-21ntpq12:35:231
6239262101,3sleep00-21swapper/012:31:050
6239002101,3sleep2321671-21diskmemload12:31:042
6091782100,1sleep00-21swapper/012:21:230
6025702101,3sleep0321671-21diskmemload12:18:270
6014212101,2sleep20-21swapper/212:16:202
5945742101,3sleep1321671-21diskmemload12:13:021
5859592101,4sleep0321671-21diskmemload12:05:360
5564162101,3sleep0321671-21diskmemload11:47:250
5482192101,2sleep00-21swapper/011:40:380
5128062100,2sleep10-21swapper/111:17:381
5121272101,3sleep0321671-21diskmemload11:16:100
4592022101,3sleep0321671-21diskmemload10:41:250
4229132101,3sleep2321671-21diskmemload10:17:542
4083942101,3sleep0321671-21diskmemload10:08:140
3926892101,3sleep0321671-21diskmemload09:55:510
3863402101,3sleep20-21swapper/209:52:542
3333432101,3sleep0333662-21strings09:15:190
3163742100,6sleep0316604-21cat09:05:010
3100532101,5sleep0310099-21ntpq08:55:230
3047392101,4sleep2304785-21ntpq08:50:202
3045972101,4sleep1304738-21ntpq08:50:191
2893182101,5sleep0289354-21perf08:40:000
2784052101,5sleep2278512-21base6408:30:012
2734052101,4sleep0273657-21df08:25:130
2418132101,3sleep2241832-21apt-get07:55:142
2359882101,5sleep2236004-21dump-pmu-power07:50:002
2359882101,5sleep2236004-21dump-pmu-power07:50:002
2256672101,3sleep0225686-21apt-get07:40:130
2041372101,3sleep10-21swapper/107:20:131
2038532101,5sleep1204089-21grep07:20:011
2028052101,4sleep2202862-21ntpq07:15:212
609856291,2sleep10-21swapper/112:22:411
542900291,2sleep00-21swapper/011:39:460
527843291,2sleep20-21swapper/211:28:342
527579290,1sleep10-21swapper/111:28:011
491911291,4sleep2491921-21ssh11:05:092
431527291,2sleep00-21swapper/010:25:060
289893291,4sleep1290079-21tail08:40:171
289594290,1sleep20-21swapper/208:40:062
273123291,4sleep0273177-21latency_hist08:25:010
267159291,4sleep1267249-21sed08:15:291
261908291,4sleep0261950-21cat08:10:260
256382291,4sleep0256430-21ntpq08:05:210
256194291,4sleep2256334-21ntp_192.168.11408:05:202
246775291,2sleep0802-21snmpd07:59:280
236496290,4sleep1235997-21apt-get07:50:031
231029291,4sleep0231213-21expr07:45:130
231029291,4sleep0231213-21expr07:45:130
215019291,3sleep00-21swapper/007:30:150
191931291,4sleep1192102-21cat07:10:151
316635281,2sleep00-21swapper/009:05:120
315151280,3sleep2315324-21strings09:00:222
295421281,4sleep1295512-21grep08:45:141
284057280,1sleep20-21swapper/208:35:012
278947280,3sleep0278966-21apt-get08:30:140
246817280,1sleep10-21swapper/107:59:571
214752281,3sleep20-21swapper/207:30:002
209744280,1sleep10-21swapper/107:25:061
310106270,2sleep1310155-21ntpq08:55:241
299951270,5sleep20-21swapper/208:47:072
278039270,1sleep10-21swapper/108:25:341
268135270,2sleep10-21swapper/108:20:131
267574270,4sleep00-21swapper/008:16:350
266967271,3sleep2267082-21gzip08:15:262
261707271,2sleep10-21swapper/108:10:231
251884270,1sleep10-21swapper/108:02:251
237093271,2sleep20-21swapper/207:50:162
233265271,3sleep1234825-21load07:45:191
233265271,3sleep1234825-21load07:45:181
219708270,1sleep20-21swapper/207:31:002
215681271,2sleep1217349-21tr07:30:191
203835270,1sleep00-21swapper/007:19:480
316657261,2sleep10-21swapper/109:05:141
310893260,1sleep20-21swapper/208:57:532
284271260,2sleep10-21swapper/108:35:121
283028260,1sleep10-21swapper/108:30:231
248884261,2sleep00-21swapper/008:00:200
226397261,2sleep10-21swapper/107:40:181
224403260,3sleep20-21swapper/207:35:252
209217261,2sleep00-21swapper/007:25:020
316889250,1sleep20-21swapper/209:05:152
252629250,1sleep10-21swapper/108:05:141
247107250,1sleep20-21swapper/208:00:142
194568250,1sleep00-21swapper/007:10:180
311064240,1sleep10-21swapper/109:00:001
279124240,1sleep20-21swapper/208:30:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional