You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-08 - 10:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Nov 08, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1264927877,0sleep10-21swapper/118:38:431
291350450,0irq/25-eth10-21swapper/318:35:413
1257923531,0sleep00-21swapper/018:37:500
1240923424,0sleep20-21swapper/218:35:512
339970,0migration/35375-21sh23:57:523
8225060,0irq/24-0000:00:3138-21cat20:55:082
8225060,0irq/24-0000:00:0-21swapper/223:50:072
339960,0migration/327001-21munin-node22:30:213
339960,0migration/323765-21sh22:47:133
339960,0migration/319614-21munin-node00:05:253
28415060,0irq/27-eth0-tx-9300-21munin-node19:50:240
28415060,0irq/27-eth0-tx-18107-21munin-node22:25:202
28415060,0irq/27-eth0-tx-0-21swapper/223:43:052
28415060,0irq/27-eth0-tx-0-21swapper/223:07:182
28415060,0irq/27-eth0-tx-0-21swapper/223:00:232
28415060,0irq/27-eth0-tx-0-21swapper/223:00:232
28415060,0irq/27-eth0-tx-0-21swapper/221:41:222
28415060,0irq/27-eth0-tx-0-21swapper/221:39:312
28415060,0irq/27-eth0-tx-0-21swapper/221:28:492
28415060,0irq/27-eth0-tx-0-21swapper/221:24:412
28415060,0irq/27-eth0-tx-0-21swapper/221:15:542
28415060,0irq/27-eth0-tx-0-21swapper/221:14:432
28415060,0irq/27-eth0-tx-0-21swapper/221:01:082
28415060,0irq/27-eth0-tx-0-21swapper/220:55:192
28415060,0irq/27-eth0-tx-0-21swapper/220:48:522
28405060,0irq/26-eth0-rx-26268-21irqstats22:50:253
28405060,0irq/26-eth0-rx-24156-21aten2.4-expect23:10:183
28405060,0irq/26-eth0-rx-16996-21cut23:05:213
249960,0watchdog/29671-21kthreadcore23:20:262
249960,0watchdog/232358-21ssh22:31:102
249960,0watchdog/20-21swapper/222:12:302
249960,0watchdog/20-21swapper/221:55:102
249960,0watchdog/20-21swapper/220:05:262
128439965,0cyclictest27029-21ssh21:29:111
128439960,0cyclictest3622-21ksmtuned22:30:501
128439960,0cyclictest15996-21ssh22:42:301
128439960,0cyclictest0-21swapper/123:31:131
128429960,0cyclictest0-21swapper/022:31:040
35150,0ktimersoftd/30-21swapper/321:30:543
339950,0migration/37782-21kthreadcore20:35:223
339950,0migration/37248-21munin-node00:00:243
339950,0migration/37242-21kthreadcore21:15:233
339950,0migration/35718-21kthreadcore22:15:293
339950,0migration/34044-21fschecks_count20:10:213
339950,0migration/331664-21kthreadcore21:10:263
339950,0migration/330871-21kthreadcore18:50:273
339950,0migration/329303-21munin-node19:40:233
339950,0migration/32641-21irqcore19:20:243
339950,0migration/325877-21munin-node21:05:343
339950,0migration/323541-21kthreadcore19:35:263
339950,0migration/323423-21kthreadcore18:45:273
339950,0migration/319443-21kthreadcore23:45:263
339950,0migration/317533-21timerwakeupswit21:00:313
339950,0migration/314974-21munin-node19:50:293
339950,0migration/314281-21cpuspeed_turbos19:05:193
339950,0migration/314094-21kthreadcore21:20:263
339950,0migration/312474-21irqrtprio21:40:253
339950,0migration/310231-21kthreadcore23:40:263
28415050,0irq/27-eth0-tx-26713-21ssh21:07:412
28415050,0irq/27-eth0-tx-23608-21munin-node20:25:260
28415050,0irq/27-eth0-tx-1752-21tr22:35:212
28415050,0irq/27-eth0-tx-0-21swapper/223:58:272
28415050,0irq/27-eth0-tx-0-21swapper/223:30:422
28415050,0irq/27-eth0-tx-0-21swapper/222:56:432
28415050,0irq/27-eth0-tx-0-21swapper/222:50:162
28415050,0irq/27-eth0-tx-0-21swapper/222:46:262
28415050,0irq/27-eth0-tx-0-21swapper/221:49:332
28415050,0irq/27-eth0-tx-0-21swapper/221:30:142
28415050,0irq/27-eth0-tx-0-21swapper/220:40:332
28415050,0irq/27-eth0-tx-0-21swapper/200:06:302
28415050,0irq/27-eth0-tx-0-21swapper/020:35:170
28415050,0irq/27-eth0-tx-0-21swapper/019:42:350
28415050,0irq/27-eth0-tx-0-21swapper/019:25:260
28415050,0irq/27-eth0-tx-0-21swapper/018:49:160
28405050,0irq/26-eth0-rx-8985-21ntp_kernel_err20:55:273
28405050,0irq/26-eth0-rx-7159-21kthreadcore21:35:273
28405050,0irq/26-eth0-rx-32185-21perl23:35:213
28405050,0irq/26-eth0-rx-29043-21cut21:50:263
28405050,0irq/26-eth0-rx-26968-21df_abs22:10:223
28405050,0irq/26-eth0-rx-24246-21uptime22:25:313
28405050,0irq/26-eth0-rx-2424-21if_eth1.222:35:243
28405050,0irq/26-eth0-rx-11057-21cut23:00:293
28405050,0irq/26-eth0-rx-11057-21cut23:00:293
28405050,0irq/26-eth0-rx-0-21swapper/323:21:003
28405050,0irq/26-eth0-rx-0-21swapper/322:22:073
28405050,0irq/26-eth0-rx-0-21swapper/322:06:193
28405050,0irq/26-eth0-rx-0-21swapper/322:00:533
28405050,0irq/26-eth0-rx-0-21swapper/321:55:263
28405050,0irq/26-eth0-rx-0-21swapper/320:47:073
28405050,0irq/26-eth0-rx-0-21swapper/320:40:303
28405050,0irq/26-eth0-rx-0-21swapper/320:05:273
259950,0migration/230051-21idleruntime-cro20:30:082
259950,0migration/224312-21kthreadcore19:35:262
259950,0migration/221916-21kthreadcore18:45:262
259950,0migration/21621-21switchtime19:40:302
249950,0watchdog/28723-21kthreadcore00:00:262
249950,0watchdog/26140-21ssh23:16:342
249950,0watchdog/227249-21kthreadcore23:50:262
249950,0watchdog/224129-21munin-node22:05:262
249950,0watchdog/2212062sleep219:32:182
249950,0watchdog/20-21swapper/223:36:542
249950,0watchdog/20-21swapper/223:11:262
249950,0watchdog/20-21swapper/222:40:382
249950,0watchdog/20-21swapper/222:20:302
249950,0watchdog/20-21swapper/222:15:382
249950,0watchdog/20-21swapper/222:00:462
249950,0watchdog/20-21swapper/221:50:102
249950,0watchdog/20-21swapper/220:23:342
249950,0watchdog/20-21swapper/220:15:222
249950,0watchdog/20-21swapper/219:45:102
249950,0watchdog/20-21swapper/219:25:112
249950,0watchdog/20-21swapper/219:04:112
249950,0watchdog/20-21swapper/218:52:152
19150,0ktimersoftd/10-21swapper/120:15:341
179950,0migration/17441-21sh22:58:021
179950,0migration/15751-21kthreadcore20:10:251
179950,0migration/130573-21kernelversion20:05:271
179950,0migration/121418-21kthreadcore20:45:251
179950,0migration/11770-21taskset21:10:521
179950,0migration/115235-21irqcore00:05:231
179950,0migration/111705-21idleruntime22:00:251
169950,0watchdog/11450-21cat23:15:221
149950,0migration/08583-21irqrtprio19:00:240
149950,0migration/07011-21kthreadcore21:55:260
149950,0migration/05968-21kthreadcore20:55:250
149950,0migration/05391-21kthreadcore19:45:260
149950,0migration/0512-21latency_hist18:55:080
149950,0migration/04420-21munin-node20:10:230
149950,0migration/04365-21turbostat20:35:080
149950,0migration/04231-21kthreadcore22:35:260
149950,0migration/03561-21kthreadcore23:55:270
149950,0migration/031339-21kthreadcore20:05:280
149950,0migration/027874-21taskset19:37:430
149950,0migration/027611-21sh21:30:130
149950,0migration/026625-21kthreadcore20:00:260
149950,0migration/025792-21sh20:46:230
149950,0migration/024048-21munin-node23:30:210
149950,0migration/023240-21kthreadcore21:45:260
149950,0migration/02091-21kthreadcore23:35:250
149950,0migration/020464-21latency23:45:260
149950,0migration/019179-21kthreadcore23:25:270
149950,0migration/018822-21kthreadcore19:05:250
149950,0migration/018170-21sh21:42:500
149950,0migration/017996-21latency_hist22:25:080
149950,0migration/017883-21kthreadcore18:40:280
149950,0migration/017485-21switchtime21:20:310
149950,0migration/017344-21unixbench-2d20:40:340
149950,0migration/014408-21cpuspeed_turbos00:05:190
149950,0migration/013128-21kthreadcore22:00:270
149950,0migration/012091-21kthreadcore23:40:260
149950,0migration/011334-21kthreadcore23:20:270
128439955,0cyclictest0-21swapper/123:37:121
128439950,0cyclictest0-21swapper/123:28:131
128439950,0cyclictest0-21swapper/121:57:431
128429955,0cyclictest0-21swapper/023:08:260
128429955,0cyclictest0-21swapper/023:03:040
128429955,0cyclictest0-21swapper/023:03:040
128429955,0cyclictest0-21swapper/022:21:460
128429955,0cyclictest0-21swapper/021:26:250
128429954,0cyclictest6445-21snmpd19:15:330
128429950,0cyclictest0-21swapper/022:51:240
128429950,0cyclictest0-21swapper/022:45:250
128429950,0cyclictest0-21swapper/022:14:490
339940,0migration/33781-21irqcore19:45:253
339940,0migration/331223-21kthreadcore19:15:253
339940,0migration/327750-21kthreadcore20:00:263
339940,0migration/326992-21taskset19:13:403
339940,0migration/326449-21kthreadcore20:25:233
339940,0migration/311248-21kthreadcore19:00:253
28415040,0irq/27-eth0-tx-0-21swapper/223:25:212
28415040,0irq/27-eth0-tx-0-21swapper/019:20:190
28405040,0irq/26-eth0-rx-1439-21ssh20:50:573
28405040,0irq/26-eth0-rx-1091-21fschecks_count18:55:203
28405040,0irq/26-eth0-rx-10293-21cut19:25:253
28405040,0irq/26-eth0-rx-0-21swapper/323:50:223
28405040,0irq/26-eth0-rx-0-21swapper/323:30:153
28405040,0irq/26-eth0-rx-0-21swapper/323:25:153
28405040,0irq/26-eth0-rx-0-21swapper/323:16:423
28405040,0irq/26-eth0-rx-0-21swapper/322:55:433
28405040,0irq/26-eth0-rx-0-21swapper/322:42:333
28405040,0irq/26-eth0-rx-0-21swapper/321:46:353
28405040,0irq/26-eth0-rx-0-21swapper/321:25:213
259940,0migration/29275-21munin-node20:10:292
259940,0migration/21670-21aten_r1power_en19:20:192
249940,0watchdog/220848-21cpu19:10:232
249940,0watchdog/216912-21if_eth119:55:222
249940,0watchdog/20-21swapper/220:31:062
249940,0watchdog/20-21swapper/220:01:102
249940,0watchdog/20-21swapper/219:50:142
249940,0watchdog/20-21swapper/219:15:312
249940,0watchdog/20-21swapper/219:05:552
179940,0migration/19516-21kthreadcore00:00:261
179940,0migration/17328-21kthreadcore20:35:221
179940,0migration/132425-21kthreadcore19:40:251
179940,0migration/125666-21kthreadcore20:00:261
179940,0migration/124066-21kthreadcore22:05:261
179940,0migration/120602-21latency_hist19:10:081
179940,0migration/119847-21kthreadcore22:25:241
179940,0migration/118175-21taskset21:42:521
179940,0migration/117037-21sh23:05:211
179940,0migration/114389-21kthreadcore21:20:261
179940,0migration/113846-21hwlatdetect18:40:241
179940,0migration/111183-21kthreadcore19:50:241
179940,0migration/110870-21fschecks_time22:20:231
169940,0watchdog/16547-21runrttasks20:41:301
169940,0watchdog/13988-21kthreadcore22:35:261
169940,0watchdog/132686-21tr21:30:261
169940,0watchdog/127781-21munin-node21:50:221
169940,0watchdog/122293-21perl19:35:221
169940,0watchdog/121295-21ssh23:45:421
169940,0watchdog/119049-21kthreadcore19:30:271
169940,0watchdog/113792-21ssh23:20:381
169940,0watchdog/10-21swapper/122:50:261
169940,0watchdog/10-21swapper/122:10:261
169940,0watchdog/10-21swapper/121:05:221
169940,0watchdog/10-21swapper/120:50:261
169940,0watchdog/10-21swapper/119:45:261
149940,0migration/030384-21kthreadcore21:50:260
149940,0migration/026555-21timerwakeupswit19:10:330
149940,0migration/022598-21kthreadcore22:05:260
149940,0migration/013735-21kthreadcore22:40:260
128449943,0cyclictest3425-21lldpd20:36:372
128449943,0cyclictest3110-21irqbalance18:40:122
128449943,0cyclictest1591-21awk18:55:222
128439944,0cyclictest0-21swapper/123:40:121
128439944,0cyclictest0-21swapper/121:38:291
128439944,0cyclictest0-21swapper/121:01:551
128439943,0cyclictest9185-21kworker/1:223:01:331
128439943,0cyclictest9185-21kworker/1:223:01:331
128439943,0cyclictest6445-21snmpd23:55:181
128439943,0cyclictest6445-21snmpd19:26:181
128439943,0cyclictest3425-21lldpd23:14:541
128439943,0cyclictest3119-21awk22:15:271
128439943,0cyclictest23843-21awk23:50:231
128439943,0cyclictest20639-21awk18:45:251
128439943,0cyclictest20210-21awk21:45:231
128439943,0cyclictest17989-21awk22:45:241
128439943,0cyclictest14892-21awk19:05:221
128439943,0cyclictest1372-21sadc19:20:081
128429944,0cyclictest0-21swapper/023:13:470
128429944,0cyclictest0-21swapper/021:35:430
128429943,0cyclictest502-21awk23:15:210
128429943,0cyclictest0-21swapper/020:22:550
128429940,0cyclictest11369-21basename21:20:080
28405030,0irq/26-eth0-rx-0-21swapper/320:32:183
28405030,0irq/26-eth0-rx-0-21swapper/320:17:023
28405030,0irq/26-eth0-rx-0-21swapper/318:40:213
169930,0watchdog/10-21swapper/121:15:181
128439933,0cyclictest28691-21kworker/1:020:55:221
128439933,0cyclictest28691-21kworker/1:020:30:121
128439933,0cyclictest28691-21kworker/1:020:25:381
128439933,0cyclictest28691-21kworker/1:019:55:161
128439933,0cyclictest28691-21kworker/1:019:20:121
128439933,0cyclictest28691-21kworker/1:019:00:181
128439933,0cyclictest28691-21kworker/1:018:55:201
128439933,0cyclictest28691-21kworker/1:018:50:161
128439932,0cyclictest18777-21cut20:20:241
128429933,0cyclictest28775-21awk21:10:230
128429933,0cyclictest0-21swapper/020:51:540
128429932,0cyclictest23726-21tr21:05:280
128429932,0cyclictest21611-21sensors19:55:290
128429932,0cyclictest13834-21kthreadcore21:00:250
128429932,0cyclictest10393-21basename20:15:200
128429931,0cyclictest0-21swapper/000:00:270
128429930,0cyclictest2858-21perl22:15:250
128429930,0cyclictest15045-21munin-node19:30:250
128429930,0cyclictest0-21swapper/018:50:280
339920,0migration/316047-21iostat_ios19:30:253
28405020,0irq/26-eth0-rx-0-21swapper/320:20:203
28405020,0irq/26-eth0-rx-0-21swapper/319:55:183
128429921,0cyclictest11078-21diskmemload23:50:090
128429920,0cyclictest0-21swapper/022:55:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional