You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-19 - 02:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Sun Oct 19, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21797422149101,16sleep30-21swapper/319:09:243
21796452149114,28sleep00-21swapper/019:08:080
21797452147101,16sleep10-21swapper/119:09:261
21795022145100,15sleep20-21swapper/219:06:232
218764921290,2sleep00-21swapper/019:15:160
254309821070,3sleep2411rcuc/223:40:092
2179908993631,3cyclictest720-21tpm2-abrmd20:00:003
2179908993223,6cyclictest2487219-21grep23:05:143
2179908993220,9cyclictest736-21gdbus21:35:003
2179908993021,6cyclictest2448284-21cut22:40:173
24549742270,2sleep20-21swapper/222:45:142
2179908992723,3cyclictest726-21in:imuxsock00:10:003
2179908992715,10cyclictest2191744-21idleruntime-cro19:20:003
2179908992622,3cyclictest2574557-21sh00:00:003
2179908992621,3cyclictest2617918-21needreboot00:25:193
2179908992621,3cyclictest2216841-21turbostat19:45:003
2179908992621,3cyclictest2216841-21turbostat19:45:003
217990899257,17cyclictest0-21swapper/321:10:233
2179908992519,3cyclictest0-21swapper/321:40:133
2179908992517,5cyclictest2367830-21diskmemload21:50:183
2179908992419,3cyclictest717-21systemd-logind21:05:013
217990899241,7cyclictest2413135-21sh22:18:193
2179908992416,6cyclictest2326973-21ps21:25:173
217990899241,21cyclictest2267228-21fschecks_count20:35:123
217990899233,18cyclictest2578977-21ntpq00:00:223
217990899233,18cyclictest2578977-21ntpq00:00:223
2179908992319,3cyclictest511ktimers/322:01:013
2179908992319,2cyclictest2431031-21gltestperf22:30:163
2179908992315,7cyclictest0-21swapper/300:35:193
2179908992315,4cyclictest2565496-21sh23:53:323
217990899231,19cyclictest2394608-21ssh22:05:223
2179908992311,8cyclictest2302291-21wc21:10:013
254087222211,7sleep22540883-21ssh23:37:272
2179908992219,2cyclictest2198795-21kthreadcore19:25:193
2179908992218,2cyclictest0-21swapper/322:50:003
2179908992218,2cyclictest0-21swapper/322:15:003
2179908992216,3cyclictest2595170-21ps00:10:243
217990899221,18cyclictest2527227-21fschecks_count23:30:173
217990899221,17cyclictest2552959-21grep23:45:223
217990899212,13cyclictest2478881-21taskset23:00:113
217990899211,7cyclictest2381415-21ssh21:58:363
2179908992116,3cyclictest2247300-21cut20:15:173
2179908992114,5cyclictest2286077-21ps20:50:263
2179908992112,8cyclictest2359538-21kthreadcore21:45:173
217990899211,17cyclictest2602889-21ld00:15:233
2179908992111,7cyclictest2506698-21ntp_kernel_err23:15:203
2179908992111,7cyclictest2287790-21sed20:55:153
217990899211,16cyclictest2499687-21ssh23:10:493
217990899211,14cyclictest943-21runrttasks21:16:123
2179908992110,7cyclictest2424394-21cut22:25:173
2179908992110,7cyclictest2342523-21expr21:35:153
24533332200,2sleep00-21swapper/022:43:360
217990899209,7cyclictest736-21gdbus20:50:003
217990899201,6cyclictest2547612-21ssh23:40:313
217990899201,6cyclictest2475665-21ssh22:55:473
217990899201,6cyclictest2475665-21ssh22:55:463
2179908992015,3cyclictest2606734-21cpuspeed_turbos00:20:153
217990899201,17cyclictest2323150-21sh21:20:373
217990899200,18cyclictest2627110-21taskset00:32:163
25242232191,3sleep00-21swapper/023:26:300
23886912191,2sleep20-21swapper/222:02:382
2179908991915,2cyclictest0-21swapper/320:25:213
2179908991914,2cyclictest0-21swapper/322:35:293
217990899191,16cyclictest2420623-21ssh22:22:283
217990899191,16cyclictest2262048-21/usr/sbin/munin20:30:163
2179908991911,6cyclictest2234985-21cut20:00:163
2179908991911,6cyclictest0-21swapper/323:35:243
2179908991911,5cyclictest2524677-21sh23:27:093
2179908991910,6cyclictest2468901-21ssh22:52:393
217990899190,17cyclictest2511198-21sh23:20:153
26015322180,2sleep00-21swapper/000:15:200
25561682180,3sleep0221rcuc/023:46:260
2179908991813,3cyclictest2252221-21cpuspeed_turbos20:20:143
217990899181,15cyclictest2272482-21hddtemp_smartct20:40:143
25884142170,3sleep10-21swapper/100:06:401
25489892171,3sleep10-21swapper/123:42:391
24366032171,3sleep20-21swapper/222:32:292
24348542171,12sleep02430656-21/usr/sbin/munin22:30:260
217990899177,7cyclictest2184861-21idleruntime-cro19:14:593
217990899177,6cyclictest703-21dbus-daemon20:10:013
217990899176,7cyclictest2222240-21sh19:50:133
217990899172,13cyclictest0-21swapper/320:15:013
2179908991713,2cyclictest0-21swapper/319:20:163
26116092161,2sleep00-21swapper/000:21:220
24777982161,4sleep12302171-21diskmemload22:59:011
24777982161,4sleep12302171-21diskmemload22:59:011
23417392161,2sleep00-21swapper/021:34:380
23242782161,3sleep20-21swapper/221:22:212
2179908991612,2cyclictest720-21tpm2-abrmd19:35:003
2179908991610,4cyclictest2207014-21/usr/sbin/munin19:35:223
25658772151,2sleep20-21swapper/223:54:062
23202782151,10sleep02322061-21cut21:20:190
217990899155,7cyclictest2220799-21collect219:45:223
22926162141,9sleep22292719-21cut21:00:172
26354502130,1sleep00-21swapper/000:39:020
26233022131,7sleep22625044-21cut00:30:182
25651002131,8sleep02565113-21ssh23:52:560
23260562131,9sleep10-21swapper/121:25:011
26215322121,4sleep10-21swapper/100:30:131
25109172121,7sleep12511029-21expr23:20:131
25014422121,5sleep12302171-21diskmemload23:13:281
24628662121,5sleep22302171-21diskmemload22:50:122
24445672121,7sleep12444577-21ssh22:37:251
24386082120,1sleep00-21swapper/022:35:070
23988662121,7sleep12399080-21/usr/sbin/munin22:10:151
23954812120,1sleep20-21swapper/222:05:502
23746512121,7sleep12374665-21cat21:55:111
23043632121,7sleep22305998-21ssh21:10:182
22635602120,7sleep22265332-21cut20:30:192
21974542121,7sleep22198291-21grep19:25:182
25977072111,6sleep02597715-21sh00:13:370
25905832111,3sleep00-21swapper/000:09:520
25627742111,5sleep12562831-21ntpq23:50:211
25258352111,3sleep10-21swapper/123:28:521
25177942112,4sleep02302171-21diskmemload23:23:540
25098912111,3sleep10-21swapper/123:19:101
25075462111,3sleep00-21swapper/023:15:400
25050832111,7sleep22506532-21ssh23:15:182
24953712111,6sleep02496258-21cut23:10:190
24926472111,3sleep00-21swapper/023:07:180
24688902111,3sleep00-21swapper/022:52:390
24618942111,5sleep12302171-21diskmemload22:49:311
24601622111,3sleep00-21swapper/022:46:540
24429382110,1sleep20-21swapper/222:35:292
24217412111,4sleep10-21swapper/122:24:101
24186312111,3sleep22418696-21ntp_states22:20:222
24112842111,6sleep12411299-21sh22:15:361
23898372111,6sleep02302171-21diskmemload22:04:210
23737392111,5sleep12302171-21diskmemload21:54:111
23323262111,5sleep02302171-21diskmemload21:27:320
22408422111,6sleep22240904-21pmu-power20:05:222
26210022100,1sleep00-21swapper/000:29:440
26045492101,4sleep10-21swapper/100:17:101
26036112101,3sleep20-21swapper/200:15:352
25969842101,4sleep22302171-21diskmemload00:12:302
25966322101,2sleep10-21swapper/100:11:561
25808702101,2sleep10-21swapper/100:02:231
25808702101,2sleep10-21swapper/100:02:231
25726262101,4sleep02302171-21diskmemload23:57:090
25707952101,3sleep10-21swapper/123:55:221
25667462101,3sleep20-21swapper/223:55:032
25513852101,4sleep22551510-21irqstats23:45:202
25328322101,4sleep22302171-21diskmemload23:32:222
25328102101,3sleep00-21swapper/023:32:200
25239322101,4sleep22302171-21diskmemload23:26:042
25168452101,3sleep20-21swapper/223:22:322
25017282101,4sleep22302171-21diskmemload23:13:542
24909562100,7sleep20-21swapper/223:05:242
24844002101,2sleep20-21swapper/223:01:512
24771882101,3sleep20-21swapper/222:58:062
24771882101,3sleep20-21swapper/222:58:062
24700432101,3sleep10-21swapper/122:54:241
24526452101,4sleep12302171-21diskmemload22:42:331
24367222101,3sleep10-21swapper/122:32:391
24283772101,3sleep20-21swapper/222:27:032
24283112101,4sleep02302171-21diskmemload22:27:010
24115452101,4sleep02302171-21diskmemload22:15:590
24056852100,3sleep00-21swapper/022:14:110
23957192101,3sleep00-21swapper/022:06:130
23798772101,3sleep20-21swapper/221:56:202
23741752101,2sleep00-21swapper/021:54:500
23729612101,2sleep20-21swapper/221:53:002
23644892101,2sleep20-21swapper/221:47:432
23574892100,3sleep10-21swapper/121:44:131
23565492101,2sleep20-21swapper/221:42:502
23475532101,3sleep20-21swapper/221:36:242
23474172101,3sleep10-21swapper/121:36:101
23423352101,2sleep00-21swapper/021:35:090
23344652101,2sleep10-21swapper/121:30:141
23152812101,4sleep02302171-21diskmemload21:15:500
23072462100,5sleep00-21swapper/021:10:370
22976052101,4sleep12297814-21hddtemp_smartct21:05:171
22859212101,4sleep02285969-21ntpq20:50:240
22759092101,4sleep22275963-21ntpq20:40:212
22720882101,4sleep12272107-21apt-get20:40:121
22620472101,2sleep10-21swapper/120:30:141
22572472100,6sleep22257313-21fschecks_count20:25:132
22458512101,5sleep12245905-21ntpq20:10:221
22271362100,7sleep12227236-21cat19:55:111
21905732101,4sleep22190698-21ntp_192.168.11419:15:202
2635046291,4sleep12302171-21diskmemload00:38:151
2634628291,2sleep20-21swapper/200:37:282
2626822291,3sleep02302171-21diskmemload00:31:440
2618928291,2sleep10-21swapper/100:25:521
2611337291,4sleep22611351-21ssh00:20:552
2611165290,1sleep10-21swapper/100:20:351
2589302291,2sleep20-21swapper/200:07:572
2579720291,2sleep20-21swapper/200:00:402
2579720291,2sleep20-21swapper/200:00:402
2578932291,4sleep02578985-21ntpq00:00:220
2578932291,4sleep02578985-21ntpq00:00:220
2556312291,2sleep10-21swapper/123:46:401
2547626290,3sleep00-21swapper/023:40:330
2540442291,3sleep02302171-21diskmemload23:36:490
2487078291,3sleep10-21swapper/123:05:151
2484015291,3sleep02302171-21diskmemload23:01:170
2478433291,3sleep02302171-21diskmemload22:59:570
2478433291,3sleep02302171-21diskmemload22:59:570
2451805290,3sleep20-21swapper/222:41:192
2426624291,4sleep12426677-21ntpq22:25:221
2421938291,3sleep00-21swapper/022:24:290
2404719291,2sleep20-21swapper/222:12:442
2395853291,2sleep10-21swapper/122:06:261
2389676290,2sleep10-21swapper/122:04:051
2380180290,3sleep00-21swapper/021:56:470
2364879290,2sleep10-21swapper/121:48:171
2356477291,2sleep00-21swapper/021:42:440
2334353290,3sleep20-21swapper/221:30:112
2331857290,3sleep20-21swapper/221:26:502
2331057290,2sleep10-21swapper/121:25:371
2315921291,2sleep10-21swapper/121:16:471
2314413291,4sleep22314477-21ntpq21:15:242
2302503290,3sleep10-21swapper/121:10:101
2287412291,4sleep22287431-21apt-get20:55:122
2280984291,4sleep22281048-21ls20:45:222
2256662290,2sleep20-21swapper/220:23:022
2232050291,3sleep10-21swapper/120:00:121
2230784291,4sleep02230831-21ntpq19:55:180
2215787291,4sleep22215820-21ntpq19:40:212
2215787291,4sleep22215820-21ntpq19:40:202
2180089291,3sleep02180108-21apt-get19:10:130
2619579281,3sleep22619589-21ssh00:27:012
2540586280,2sleep10-21swapper/123:37:021
2531982281,2sleep10-21swapper/123:31:031
2486169281,3sleep12486178-21ssh23:04:311
2410493281,3sleep22410552-21ntpq22:15:212
2364070281,3sleep02302171-21diskmemload21:47:050
2292463281,3sleep12292615-21df_inode21:00:151
2285971280,1sleep10-21swapper/120:50:261
2267032280,2sleep20-21swapper/220:35:112
2240746281,3sleep02240784-21ntpq20:05:200
2237042281,3sleep12237188-21df20:05:131
2235814280,2sleep20-21swapper/220:00:212
2207013280,2sleep00-21swapper/019:35:140
2202436281,3sleep12202993-21cat19:30:181
2195800281,4sleep02195853-21ntp_states19:20:230
2302150271,2sleep00-21swapper/021:10:010
2301780270,1sleep20-21swapper/221:05:302
2292441270,2sleep00-21swapper/021:00:150
2286597270,1sleep20-21swapper/220:51:312
2280880270,3sleep02280930-21ntpq20:45:200
2276734270,1sleep00-21swapper/020:43:280
2271804271,3sleep12271841-21perf20:40:001
2257079271,2sleep00-21swapper/020:25:120
2256749270,1sleep10-21swapper/120:24:091
2242489271,3sleep00-21swapper/020:10:170
2236441270,1sleep00-21swapper/020:00:590
2226460270,1sleep10-21swapper/119:50:461
2221602270,1sleep00-21swapper/019:47:330
2217069270,2sleep20-21swapper/219:45:142
2206428271,3sleep0945-21snmpd19:30:570
2196562270,2sleep10-21swapper/119:22:491
2180711271,3sleep12181960-21kthreadcore19:10:181
2287381260,2sleep00-21swapper/020:55:020
2281667260,1sleep10-21swapper/120:47:131
2261470260,1sleep10-21swapper/120:26:051
2252094260,1sleep00-21swapper/020:20:110
2251509260,1sleep10-21swapper/120:16:141
2251438260,1sleep00-21swapper/020:15:290
2226754260,1sleep20-21swapper/219:54:322
2222696261,2sleep00-21swapper/019:50:170
2215874261,1sleep00-21swapper/019:40:230
2215874261,1sleep00-21swapper/019:40:230
2207190261,2sleep20-21swapper/219:35:162
2202994261,2sleep20-21swapper/219:30:192
2191369260,1sleep10-21swapper/119:15:401
2182391261,2sleep20-21swapper/219:10:192
2271186250,1sleep00-21swapper/020:35:260
2246798250,1sleep20-21swapper/220:15:002
2211676251,2sleep12179902-21cyclictest19:38:351
2201532250,1sleep10-21swapper/119:27:331
2287400240,1sleep10-21swapper/120:55:031
2261834240,1sleep00-21swapper/020:30:010
2247059240,1sleep20-21swapper/220:15:122
2230617240,1sleep20-21swapper/219:55:162
2216846240,1sleep10-21swapper/119:45:011
2212375240,1sleep10-21swapper/119:40:161
2212375240,1sleep10-21swapper/119:40:151
2197176240,1sleep00-21swapper/019:25:150
2192395240,1sleep20-21swapper/219:20:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional