You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 00:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Tue Nov 18, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21357549919571,1947cyclictest24309332sleep310:56:523
2135482212955,69sleep30-21swapper/307:08:143
213559229461,10sleep10-21swapper/107:09:361
213559328461,10sleep20-21swapper/207:09:372
213559927857,14sleep00-21swapper/007:09:390
2135754992621,2cyclictest0-21swapper/310:10:133
2135754992520,3cyclictest0-21swapper/311:10:153
213575499251,20cyclictest2175247-21chrt07:44:363
213575499251,11cyclictest2370662-21chrt10:17:203
2135754992418,3cyclictest2201957-21turbostat08:10:003
2135754992418,3cyclictest0-21swapper/311:35:223
2135754992418,3cyclictest0-21swapper/309:55:263
2135754992416,6cyclictest0-21swapper/312:23:533
23554862231,3sleep00-21swapper/010:06:550
2135754992320,1cyclictest0-21swapper/307:35:243
213575499231,18cyclictest2570965-21taskset12:27:043
213575499231,18cyclictest2506966-21taskset11:46:583
213575499231,18cyclictest2416835-21taskset10:48:223
25455242221,2sleep00-21swapper/012:12:350
25455242221,2sleep00-21swapper/012:12:350
21644462222,6sleep12164464-21apt-get07:35:131
213575499222,16cyclictest2407215-21chrt10:40:253
213575499222,13cyclictest2514476-21taskset11:51:523
213575499222,13cyclictest2266415-21taskset09:10:133
213575499221,8cyclictest2288130-21taskset09:24:083
2135754992218,2cyclictest0-21swapper/310:50:183
2135754992216,3cyclictest0-21swapper/309:50:113
213575499221,17cyclictest2294563-21taskset09:26:533
2135754992117,2cyclictest0-21swapper/312:00:183
2135754992117,2cyclictest0-21swapper/311:00:283
213575499211,15cyclictest2296745-21chrt09:30:013
25003522201,8sleep02266040-21diskmemload11:44:130
24602812201,5sleep12266040-21diskmemload11:15:311
213575499202,13cyclictest2462022-21taskset11:18:493
2135754992017,1cyclictest0-21swapper/312:15:203
2135754992017,1cyclictest0-21swapper/309:15:193
2135754992016,2cyclictest0-21swapper/311:33:013
2135754992014,3cyclictest0-21swapper/311:41:353
2135754992013,3cyclictest0-21swapper/312:30:233
213575499201,15cyclictest2537933-21taskset12:08:403
213575499201,14cyclictest2543311-21chrt12:10:203
213575499201,14cyclictest2543311-21chrt12:10:203
213575499201,14cyclictest2316893-21chrt09:41:203
213575499201,14cyclictest2266061-21sleep09:10:003
25289842191,3sleep20-21swapper/212:00:422
24695922191,7sleep02266040-21diskmemload11:23:560
24695922191,7sleep02266040-21diskmemload11:23:560
213575499192,11cyclictest2464418-21taskset11:20:173
213575499192,11cyclictest2464418-21taskset11:20:173
2135754991916,1cyclictest0-21swapper/308:20:123
2135754991916,1cyclictest0-21swapper/307:30:263
213575499191,14cyclictest2476760-21chrt11:28:073
213575499191,14cyclictest2446212-21chrt11:07:233
213575499191,14cyclictest0-21swapper/312:35:113
213575499191,14cyclictest0-21swapper/310:30:313
213575499191,14cyclictest0-21swapper/310:21:403
213575499191,14cyclictest0-21swapper/310:21:403
213575499191,14cyclictest0-21swapper/310:02:103
213575499191,10cyclictest2244797-21taskset08:50:013
213575499190,12cyclictest2401627-21taskset10:38:523
25896592181,3sleep10-21swapper/112:39:021
25637812181,3sleep20-21swapper/212:24:032
24314602181,3sleep00-21swapper/010:57:500
23937072180,3sleep20-21swapper/210:32:582
22961682181,4sleep02266040-21diskmemload09:29:550
213575499182,12cyclictest2522663-21taskset11:58:093
2135754991815,1cyclictest0-21swapper/309:45:183
2135754991815,1cyclictest0-21swapper/307:20:153
2135754991814,2cyclictest0-21swapper/308:26:373
2135754991812,3cyclictest2185794-21taskset07:50:373
213575499181,13cyclictest0-21swapper/310:05:143
213575499181,13cyclictest0-21swapper/309:36:443
25224122171,4sleep02266040-21diskmemload11:57:440
24922282171,3sleep12492238-21ssh11:38:241
24473472170,11sleep20-21swapper/211:09:512
23415722171,2sleep00-21swapper/009:59:460
23256802171,3sleep00-21swapper/009:48:510
22615352171,5sleep02262403-21cut09:05:180
2135754991714,1cyclictest0-21swapper/309:00:203
2135754991714,1cyclictest0-21swapper/308:40:173
213575499171,13cyclictest0-21swapper/310:25:023
213575499171,13cyclictest0-21swapper/308:30:123
213575499170,14cyclictest0-21swapper/308:50:153
213575499170,13cyclictest0-21swapper/307:10:173
25700682161,8sleep02266040-21diskmemload12:25:540
24712072161,3sleep20-21swapper/211:25:182
24322202161,3sleep12266040-21diskmemload10:59:161
24233282161,9sleep02423339-21ssh10:51:530
24231322161,3sleep22266040-21diskmemload10:51:312
23940302160,4sleep12266040-21diskmemload10:33:341
23773242161,4sleep22266040-21diskmemload10:20:362
23773242161,4sleep22266040-21diskmemload10:20:352
23644392161,9sleep12266040-21diskmemload10:14:541
23318272161,4sleep12266040-21diskmemload09:51:131
22948702161,3sleep10-21swapper/109:27:321
2135754991613,1cyclictest0-21swapper/308:35:163
213575499161,12cyclictest2196693-21turbostat08:05:013
213575499160,13cyclictest0-21swapper/308:15:163
213575499160,12cyclictest0-21swapper/307:55:123
213575499160,10cyclictest2254863-21chrt08:56:423
213575499160,10cyclictest2207296-21taskset08:12:343
25615882151,8sleep02266040-21diskmemload12:21:090
24545712150,2sleep00-21swapper/011:14:060
24458772150,3sleep12266040-21diskmemload11:06:411
24383882151,2sleep20-21swapper/211:01:252
23841362151,7sleep22384189-21ntpq10:25:212
23121762155,6sleep02312369-21kthreadcore09:40:170
22800432151,9sleep12266040-21diskmemload09:17:371
22181322156,4sleep22218145-21idleruntime-cro08:25:002
213575499151,9cyclictest2143128-21taskset07:15:023
2135754991512,1cyclictest0-21swapper/307:25:193
2135754991512,1cyclictest0-21swapper/307:25:183
213575499150,13cyclictest0-21swapper/307:45:143
25827592141,7sleep02582841-21irqstats12:35:200
25816422148,2sleep0221rcuc/012:34:480
25451982140,2sleep20-21swapper/212:12:092
25451982140,2sleep20-21swapper/212:12:092
25233022141,8sleep12523312-21ssh11:59:211
25145482141,7sleep00-21swapper/011:52:020
24101912141,8sleep12410239-21munin-run10:45:011
24015162140,1sleep20-21swapper/210:38:322
23107692141,6sleep00-21swapper/009:39:090
23029302141,8sleep02266040-21diskmemload09:33:080
22027542141,9sleep12202878-21perl08:10:161
21816512141,9sleep02182424-21cut07:50:150
25720412131,7sleep12572055-21ssh12:28:311
25627672131,6sleep10-21swapper/112:22:421
24991012131,7sleep20-21swapper/211:42:022
24605812131,5sleep20-21swapper/211:16:042
24480662131,8sleep12448219-21head11:10:141
24259042131,4sleep22425995-21ssh10:55:162
23944312131,5sleep02266040-21diskmemload10:34:170
23645442131,5sleep00-21swapper/010:15:010
23626432130,2sleep00-21swapper/010:10:560
23421422131,7sleep12342404-21df10:00:151
23250002131,5sleep10-21swapper/109:47:321
23180252131,8sleep22318033-21ssh09:43:362
23165342131,6sleep12266040-21diskmemload09:40:381
22885682131,3sleep22288622-21systemd-run09:25:002
22556862131,6sleep02255705-21apt-get09:00:130
21692392130,1sleep20-21swapper/207:35:542
25797702121,5sleep12266040-21diskmemload12:32:161
25541552120,7sleep22266040-21diskmemload12:17:462
25525012121,5sleep12266040-21diskmemload12:15:321
25452792120,6sleep12266040-21diskmemload12:12:161
25452792120,6sleep12266040-21diskmemload12:12:161
25302112120,8sleep12530220-21ssh12:03:261
25012192120,2sleep10-21swapper/111:45:141
24856262121,7sleep22266040-21diskmemload11:35:012
24629042125,3sleep22462940-21cpu11:20:132
24629042125,3sleep22462940-21cpu11:20:132
24146792121,6sleep02414747-21ntp_states10:45:240
23654692121,3sleep12366662-21grep10:15:181
23630102121,7sleep22363017-21ssh10:11:502
23494852120,1sleep00-21swapper/010:04:590
22962772121,5sleep22296739-21packagekitd09:30:012
21691982121,5sleep02168627-31find07:35:320
21369182121,7sleep12138008-21kthreadcore07:10:201
25840192111,6sleep22585713-21kthreadcore12:35:222
25777462111,3sleep20-21swapper/212:30:252
25648852111,3sleep20-21swapper/212:25:122
25536362111,3sleep00-21swapper/012:17:040
25377082111,4sleep02266040-21diskmemload12:08:220
25372372111,4sleep22266040-21diskmemload12:07:362
25297692111,3sleep02266040-21diskmemload12:02:210
25140952111,4sleep22266040-21diskmemload11:51:122
25089282111,6sleep12266040-21diskmemload11:50:111
25018392110,7sleep22503287-21kthreadcore11:45:192
25011732111,4sleep02266040-21diskmemload11:45:090
24986322111,3sleep10-21swapper/111:41:091
24930932111,3sleep20-21swapper/211:39:582
24914952111,4sleep02266040-21diskmemload11:37:030
24844812110,1sleep10-21swapper/111:33:071
24823862111,3sleep00-21swapper/011:30:220
24769002111,4sleep12266040-21diskmemload11:28:261
24619872111,4sleep02266040-21diskmemload11:18:470
24525262111,3sleep20-21swapper/211:10:272
24183562110,7sleep12419744-21cut10:50:181
24145622110,3sleep20-21swapper/210:45:222
24071602111,4sleep22266040-21diskmemload10:40:252
24010902111,4sleep02266040-21diskmemload10:37:300
23867862111,3sleep10-21swapper/110:29:091
23791532111,3sleep00-21swapper/010:24:150
23791532111,3sleep00-21swapper/010:24:150
23787802111,4sleep12266040-21diskmemload10:23:361
23787802111,4sleep12266040-21diskmemload10:23:361
23551712111,3sleep10-21swapper/110:06:241
23381722111,4sleep22266040-21diskmemload09:55:192
23332292111,4sleep22266040-21diskmemload09:53:422
23324492111,3sleep02266040-21diskmemload09:52:200
23253092111,4sleep22266040-21diskmemload09:48:092
23106322111,3sleep10-21swapper/109:38:541
23100022111,4sleep22266040-21diskmemload09:37:472
23018512111,6sleep12301854-21ssh09:30:481
22883442111,3sleep10-21swapper/109:24:311
22870632111,6sleep02266040-21diskmemload09:21:450
22800522111,4sleep22266040-21diskmemload09:17:372
22746432111,5sleep02274858-21hddtemp_smartct09:15:160
22729182111,4sleep22266040-21diskmemload09:13:382
22723262111,4sleep12266040-21diskmemload09:12:321
22020712111,4sleep22201945-21apt-get08:10:012
21969472111,5sleep02196966-21apt-get08:05:140
21798322110,7sleep22179846-21mailstats07:45:232
25371982101,3sleep10-21swapper/112:07:311
25218822101,3sleep22266040-21diskmemload11:56:372
24756592100,3sleep02266040-21diskmemload11:25:450
24680532101,3sleep12266040-21diskmemload11:21:091
24680532101,3sleep12266040-21diskmemload11:21:091
24469852101,4sleep02266040-21diskmemload11:09:090
24385832101,5sleep10-21swapper/111:01:441
24167832100,1sleep10-21swapper/110:48:171
24086252102,4sleep02408635-21ssh10:41:590
23951532101,5sleep12395320-21ls10:35:141
23799242101,2sleep00-21swapper/010:25:070
23690162101,3sleep22266040-21diskmemload10:15:222
23558612101,4sleep22266040-21diskmemload10:07:362
23480902101,3sleep22266040-21diskmemload10:02:282
23395522101,3sleep10-21swapper/109:56:091
22733902101,3sleep00-21swapper/009:14:320
22659282101,4sleep12265934-21fwupd09:09:231
22552942101,3sleep12255148-21apt-get09:00:021
22551252101,4sleep02255218-21munin-run09:00:010
22386212101,5sleep22238706-21ls08:40:272
22292742101,4sleep02229484-21hddtemp_smartct08:35:160
22276762101,4sleep12227724-21ntpq08:30:201
22201062100,2sleep22221907-21grep08:25:202
22188482101,4sleep02218964-21irqstats08:25:180
22161832101,4sleep12216972-21memory08:20:181
22077572100,4sleep22207776-21apt-get08:15:142
21755452101,4sleep12175564-21apt-get07:45:111
21742892101,4sleep22174335-21ntpq07:40:192
21704102101,4sleep02170623-21hddtemp_smartct07:40:130
21617202101,4sleep22163020-21memory07:30:172
21514202101,4sleep02152482-21memory07:20:170
21476882101,4sleep22147932-21users07:15:302
21409882102,4sleep0246-21systemd-journal07:15:000
2438137291,2sleep00-21swapper/011:00:590
2293819291,4sleep22293825-21ssh09:25:312
2244915291,4sleep02245131-21diskstats08:50:150
2244582291,4sleep22244795-21grep08:50:002
2197780291,4sleep12199462-21cut08:05:191
2186130290,6sleep1802-21snmpd07:54:531
2159193291,2sleep00-21swapper/007:30:110
2157907291,4sleep22157954-21awk07:25:202
2157907291,4sleep22157954-21awk07:25:192
2141047291,4sleep12141556-21munin-node07:15:011
2249831281,3sleep12249972-21latency_hist08:55:011
2248785281,2sleep20-21swapper/208:50:212
2234779281,3sleep02234850-21irqstats08:40:170
2232950281,3sleep12232998-21ntpq08:35:211
2218965281,3sleep12220104-21kthreadcore08:25:181
2206529280,4sleep00-21swapper/008:10:230
2195992281,3sleep22196043-21sed08:00:272
2190526281,3sleep22190639-21ps07:55:212
2188086281,3sleep10-21swapper/107:55:171
2159215281,4sleep12159461-21expr07:30:121
2158242281,3sleep02158394-21timerwakeupswit07:25:260
2158242281,3sleep02158394-21timerwakeupswit07:25:260
2154197281,3sleep12154411-21smartctl07:25:151
2154197281,3sleep12154411-21smartctl07:25:151
2261258271,3sleep2421rcuc/209:05:162
2260710270,2sleep20-21swapper/209:05:002
2240146271,3sleep00-21swapper/008:45:180
2234852271,3sleep12236216-21kthreadcore08:40:181
2214432271,2sleep00-21swapper/008:20:170
2212753272,2sleep02135748-21cyclictest08:19:370
2212693270,1sleep10-21swapper/108:18:501
2201809270,1sleep20-21swapper/208:08:322
2190413271,2sleep00-21swapper/007:55:200
2148269270,3sleep02148270-21sshd07:19:370
2254231261,2sleep10-21swapper/108:55:251
2254032261,2sleep20-21swapper/208:55:212
2223733261,2sleep20-21swapper/208:30:132
2192265261,2sleep10-21swapper/108:00:181
2185906260,1sleep20-21swapper/207:52:062
2180246260,1sleep00-21swapper/007:46:060
2170120260,2sleep10-21swapper/107:40:101
2148584260,2sleep20-21swapper/207:20:102
2239739250,1sleep10-21swapper/108:45:151
2191697251,1sleep00-21swapper/008:00:130
2229006240,1sleep20-21swapper/208:35:132
2224001240,1sleep00-21swapper/008:30:150
2148933240,1sleep10-21swapper/107:20:141
2135775240,1sleep20-21swapper/207:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional