You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 20:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Jun 28, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2673823727,0sleep30-21swapper/306:43:133
2686123523,0sleep00-21swapper/006:44:420
2676223531,0sleep20-21swapper/206:43:322
2671523430,0sleep10-21swapper/106:42:591
35160,0ktimersoftd/30-21swapper/311:26:413
35160,0ktimersoftd/30-21swapper/311:26:413
339960,0migration/331839-21sh08:45:343
339960,0migration/328209-21sh07:30:273
339960,0migration/324398-21sh11:49:463
339960,0migration/322059-21sh09:42:523
339960,0migration/318997-21sh10:21:583
339960,0migration/317879-21sh10:42:243
339960,0migration/317566-21diskmemload12:05:203
339960,0migration/310259-21missed_timers10:15:273
28415060,0irq/27-eth0-tx-28372-21cut09:25:252
28415060,0irq/27-eth0-tx-2211-21telnet10:55:202
28415060,0irq/27-eth0-tx-0-21swapper/212:10:212
28415060,0irq/27-eth0-tx-0-21swapper/212:06:352
28415060,0irq/27-eth0-tx-0-21swapper/211:51:432
28415060,0irq/27-eth0-tx-0-21swapper/211:35:252
28415060,0irq/27-eth0-tx-0-21swapper/211:31:022
28415060,0irq/27-eth0-tx-0-21swapper/211:23:452
28415060,0irq/27-eth0-tx-0-21swapper/211:19:402
28415060,0irq/27-eth0-tx-0-21swapper/210:41:282
28415060,0irq/27-eth0-tx-0-21swapper/210:38:082
28415060,0irq/27-eth0-tx-0-21swapper/210:27:262
28415060,0irq/27-eth0-tx-0-21swapper/210:21:342
28415060,0irq/27-eth0-tx-0-21swapper/210:12:092
28415060,0irq/27-eth0-tx-0-21swapper/210:05:492
28415060,0irq/27-eth0-tx-0-21swapper/209:56:402
28415060,0irq/27-eth0-tx-0-21swapper/209:41:132
28415060,0irq/27-eth0-tx-0-21swapper/209:10:322
28415060,0irq/27-eth0-tx-0-21swapper/209:06:372
28415060,0irq/27-eth0-tx-0-21swapper/208:51:202
28415060,0irq/27-eth0-tx-0-21swapper/007:03:050
28405060,0irq/26-eth0-rx-27117-21diskstats10:50:303
28405060,0irq/26-eth0-rx-25837-21cut09:25:243
28405060,0irq/26-eth0-rx-24084-21users11:05:293
269669964,0cyclictest0-21swapper/209:35:342
269639960,0cyclictest25404-21diskmemload09:45:340
269639960,0cyclictest0-21swapper/011:20:290
249960,0watchdog/26547-21runrttasks07:40:152
249960,0watchdog/20-21swapper/212:00:592
249960,0watchdog/20-21swapper/211:41:392
249960,0watchdog/20-21swapper/209:16:032
149960,0migration/0479-21sh08:49:190
149960,0migration/031919-21latency_hist09:30:040
149960,0migration/015188-21sh12:01:300
339950,0migration/39019-21ntp_offset07:40:273
339950,0migration/37750-21munin-node08:05:233
339950,0migration/37072-21munin-node11:15:273
339950,0migration/37043-21taskset06:50:263
339950,0migration/36968-21runrttasks09:11:253
339950,0migration/35377-21kthreadcore07:15:233
339950,0migration/332431-21runrttasks09:09:323
339950,0migration/331206-21kthreadcore06:45:273
339950,0migration/330960-21fschecks_time08:00:293
339950,0migration/330412-21kthreadcore07:10:253
339950,0migration/327458-21kthreadcore07:55:253
339950,0migration/324723-21latency_hist12:10:043
339950,0migration/324259-21taskset09:03:453
339950,0migration/323181-21aten_r1power_vo09:45:183
339950,0migration/322891-21taskset09:21:303
339950,0migration/321116-21taskset10:04:343
339950,0migration/313849-21kthreadcore08:35:243
28415050,0irq/27-eth0-tx-9121-21munin-node08:55:252
28415050,0irq/27-eth0-tx-6372-21iceccd08:25:240
28415050,0irq/27-eth0-tx-17009-21munin-node09:00:202
28415050,0irq/27-eth0-tx-1447-21munin-node06:50:230
28415050,0irq/27-eth0-tx-0-21swapper/211:29:412
28415050,0irq/27-eth0-tx-0-21swapper/211:29:412
28415050,0irq/27-eth0-tx-0-21swapper/211:06:102
28415050,0irq/27-eth0-tx-0-21swapper/211:00:352
28415050,0irq/27-eth0-tx-0-21swapper/210:45:112
28415050,0irq/27-eth0-tx-0-21swapper/210:34:032
28415050,0irq/27-eth0-tx-0-21swapper/210:00:542
28415050,0irq/27-eth0-tx-0-21swapper/209:30:192
28415050,0irq/27-eth0-tx-0-21swapper/208:45:292
28415050,0irq/27-eth0-tx-0-21swapper/008:08:450
28415050,0irq/27-eth0-tx-0-21swapper/008:00:280
28415050,0irq/27-eth0-tx-0-21swapper/007:55:220
28405050,0irq/26-eth0-rx-9461-21if_err_eth1.309:15:243
28405050,0irq/26-eth0-rx-76622chrt09:35:223
28405050,0irq/26-eth0-rx-7522-21ssh11:37:513
28405050,0irq/26-eth0-rx-5842-21cpu08:30:183
28405050,0irq/26-eth0-rx-4928-21cut11:55:253
28405050,0irq/26-eth0-rx-375-21tr10:10:303
28405050,0irq/26-eth0-rx-32260-21ssh11:11:413
28405050,0irq/26-eth0-rx-26526-21grep07:05:243
28405050,0irq/26-eth0-rx-25303-21cut10:05:233
28405050,0irq/26-eth0-rx-14332-21processes11:40:293
28405050,0irq/26-eth0-rx-12064-21unixbench_multi09:55:303
28405050,0irq/26-eth0-rx-0-21swapper/312:01:403
28405050,0irq/26-eth0-rx-0-21swapper/311:01:253
28405050,0irq/26-eth0-rx-0-21swapper/310:45:283
28405050,0irq/26-eth0-rx-0-21swapper/310:30:203
28405050,0irq/26-eth0-rx-0-21swapper/308:50:573
269649955,0cyclictest0-21swapper/111:33:091
269649955,0cyclictest0-21swapper/108:57:041
269649950,0cyclictest0-21swapper/111:23:421
269649950,0cyclictest0-21swapper/109:45:381
269649950,0cyclictest0-21swapper/109:13:471
269639955,0cyclictest0-21swapper/010:50:220
269639955,0cyclictest0-21swapper/010:18:120
269639955,0cyclictest0-21swapper/007:25:060
269639950,0cyclictest0-21swapper/010:55:150
269639950,0cyclictest0-21swapper/009:51:340
269639950,0cyclictest0-21swapper/008:58:450
259950,0migration/24715-21sh10:15:052
259950,0migration/229520-21cpuspeed_turbos07:35:192
259950,0migration/226606-21kthreadcore07:30:232
259950,0migration/222482-21unixbench_multi09:20:312
259950,0migration/218215-21kthreadcore07:25:302
259950,0migration/216808-21aten_r1power_po11:45:182
259950,0migration/21090-21kthreadcore08:25:242
249950,0watchdog/231698-21if_eth009:50:232
249950,0watchdog/20-21swapper/211:55:352
249950,0watchdog/20-21swapper/210:50:272
249950,0watchdog/20-21swapper/209:48:032
249950,0watchdog/20-21swapper/208:34:472
249950,0watchdog/20-21swapper/208:23:352
249950,0watchdog/20-21swapper/208:15:352
249950,0watchdog/20-21swapper/208:10:552
249950,0watchdog/20-21swapper/208:01:112
249950,0watchdog/20-21swapper/207:52:232
249950,0watchdog/20-21swapper/207:20:192
249950,0watchdog/20-21swapper/207:10:112
19150,0ktimersoftd/10-21swapper/108:53:271
179950,0migration/130500-21munin-node11:50:301
169950,0watchdog/16824-21ssh09:33:431
169950,0watchdog/114055-21missed_timers11:40:271
149950,0migration/09502-21sh11:40:230
149950,0migration/07108-21sh11:56:330
149950,0migration/07096-21timerandwakeup08:50:290
149950,0migration/05721-21kthreadcore09:10:280
149950,0migration/04748-21kthreadcore09:30:240
149950,0migration/04012-21kthreadcore07:15:230
149950,0migration/03992-21fschecks_count07:40:210
149950,0migration/032277-21taskset11:34:210
149950,0migration/03224-21latency_hist10:35:040
149950,0migration/027229-21aten_r1power_en06:45:210
149950,0migration/026987-21sh10:27:360
149950,0migration/026166-21taskset10:49:210
149950,0migration/025727-21taskset11:10:110
149950,0migration/021453-21latency_hist10:05:040
149950,0migration/020620-21kthreadcore07:50:260
149950,0migration/019577-21fschecks_time08:40:210
149950,0migration/018073-21munin-node10:20:280
149950,0migration/017922-21latency_hist08:15:040
149950,0migration/017588-21idleruntime12:05:200
149950,0migration/016773-21aten_r1power_fr11:45:180
149950,0migration/016083-21sh09:20:030
149950,0migration/015688-21latency_hist07:25:040
149950,0migration/014260-21taskset06:56:040
149950,0migration/011903-21fschecks_time10:40:240
149950,0migration/011465-21missed_timers09:55:260
149950,0migration/011350-21aten_r1power_cu08:10:210
149950,0migration/010086-21kthreadcore09:35:280
339940,0migration/35922-21sh09:30:393
339940,0migration/33268-21taskset07:39:503
339940,0migration/325911-21irqcore11:30:213
339940,0migration/320648-21kthreadcore07:50:263
339940,0migration/31714-21kthreadcore08:25:243
28415040,0irq/27-eth0-tx-0-21swapper/008:35:220
28415040,0irq/27-eth0-tx-0-21swapper/007:48:260
28415040,0irq/27-eth0-tx-0-21swapper/007:10:220
28405040,0irq/26-eth0-rx-0-21swapper/311:50:263
28405040,0irq/26-eth0-rx-0-21swapper/311:20:433
28405040,0irq/26-eth0-rx-0-21swapper/310:55:133
28405040,0irq/26-eth0-rx-0-21swapper/310:35:403
28405040,0irq/26-eth0-rx-0-21swapper/310:25:303
28405040,0irq/26-eth0-rx-0-21swapper/309:50:123
28405040,0irq/26-eth0-rx-0-21swapper/308:55:483
28405040,0irq/26-eth0-rx-0-21swapper/308:15:243
28405040,0irq/26-eth0-rx-0-21swapper/307:45:283
28405040,0irq/26-eth0-rx-0-21swapper/307:00:183
269669943,0cyclictest3425-21lldpd06:50:222
269649944,0cyclictest29974-21kworker/1:208:20:241
269649944,0cyclictest25404-21diskmemload08:46:591
269649944,0cyclictest1284-21kthreadcore07:35:241
269649944,0cyclictest0-21swapper/110:50:261
269649943,0cyclictest6445-21snmpd11:11:351
269649943,0cyclictest6445-21snmpd11:06:201
269649943,0cyclictest6445-21snmpd11:01:051
269649943,0cyclictest6445-21snmpd10:06:491
269649943,0cyclictest6445-21snmpd09:50:491
269649943,0cyclictest6445-21snmpd08:31:491
269649943,0cyclictest6445-21snmpd08:16:491
269649943,0cyclictest5473-21awk08:05:211
269649943,0cyclictest3110-21irqbalance12:00:181
269649943,0cyclictest3110-21irqbalance09:35:491
269649943,0cyclictest3110-21irqbalance09:01:291
269649943,0cyclictest3110-21irqbalance07:45:091
269649943,0cyclictest3110-21irqbalance07:40:491
269649943,0cyclictest3110-21irqbalance07:30:091
269649943,0cyclictest3110-21irqbalance07:25:191
269649943,0cyclictest3110-21irqbalance07:20:191
269649943,0cyclictest3110-21irqbalance07:10:491
269649943,0cyclictest3110-21irqbalance07:05:091
269649943,0cyclictest3110-21irqbalance07:00:191
269649943,0cyclictest3110-21irqbalance06:55:491
269649943,0cyclictest3110-21irqbalance06:50:091
269649943,0cyclictest30246-21awk10:10:271
269649943,0cyclictest22736-21sadc09:45:041
269649943,0cyclictest17860-21awk07:50:241
269649943,0cyclictest1357-21awk11:55:211
269639944,0cyclictest0-21swapper/012:11:240
269639944,0cyclictest0-21swapper/011:53:070
269639943,0cyclictest6445-21snmpd11:07:200
269639943,0cyclictest23467-21awk07:30:210
269639943,0cyclictest22381-21awk07:05:210
259940,0migration/28501-21cpuspeed_turbos06:55:202
259940,0migration/224150-21fschecks_time07:55:222
259940,0migration/215911-21sendmail_mailqu07:45:322
249940,0watchdog/24623-21kthreadcore07:15:232
249940,0watchdog/20-21swapper/211:10:112
249940,0watchdog/20-21swapper/208:41:072
249940,0watchdog/20-21swapper/208:35:312
249940,0watchdog/20-21swapper/208:06:032
249940,0watchdog/20-21swapper/207:05:232
249940,0watchdog/20-21swapper/207:00:192
249940,0watchdog/20-21swapper/206:45:272
179940,0migration/16847-21kthreadcore07:15:241
179940,0migration/16284-21kthreadcore11:15:231
179940,0migration/129836-21kthreadcore10:30:241
179940,0migration/129257-21kthreadcore12:10:251
179940,0migration/128762-21kthreadcore06:45:261
179940,0migration/127974-21kthreadcore09:05:271
179940,0migration/126318-21kthreadcore09:25:241
179940,0migration/124959-21kthreadcore07:55:251
179940,0migration/124120-21kthreadcore08:40:251
179940,0migration/119712-21kthreadcore11:45:241
179940,0migration/119245-21kthreadcore09:20:241
179940,0migration/11738-21kthreadcore08:25:241
179940,0migration/116663-21missed_timers08:10:281
179940,0migration/113672-21kthreadcore08:35:241
169940,0watchdog/18964-21ntpq10:35:311
169940,0watchdog/13945-21sed08:00:391
169940,0watchdog/125404-21diskmemload10:15:591
169940,0watchdog/121735-21kthreadcore11:25:271
169940,0watchdog/121735-21kthreadcore11:25:271
169940,0watchdog/119610-21cut12:05:231
169940,0watchdog/117776-21cut10:00:271
169940,0watchdog/111535-21cpuspeed_turbos10:40:231
169940,0watchdog/10-21swapper/110:25:271
169940,0watchdog/10-21swapper/109:15:271
149940,0migration/08206-21kthreadcore08:30:230
149940,0migration/029974-21fschecks_time10:10:260
149940,0migration/02951-21sh10:34:530
149940,0migration/026761-21kthreadcore08:20:240
149940,0migration/021171-21runrttasks10:04:440
149940,0migration/021001-21sensors_temp09:40:280
149940,0migration/018375-21kthreadcore09:00:240
28405030,0irq/26-eth0-rx-0-21swapper/308:41:073
28405030,0irq/26-eth0-rx-0-21swapper/308:24:353
28405030,0irq/26-eth0-rx-0-21swapper/308:10:213
28405030,0irq/26-eth0-rx-0-21swapper/307:22:523
28405030,0irq/26-eth0-rx-0-21swapper/306:55:193
269649930,0cyclictest12797-21diskstats10:20:201
269639933,0cyclictest3425-21lldpd07:36:330
269639932,0cyclictest6445-21snmpd11:29:200
269639932,0cyclictest6445-21snmpd11:29:200
269639932,0cyclictest17251-21awk09:20:220
269639932,0cyclictest1183-21grep11:35:200
269639930,0cyclictest7503-21ssh11:15:390
269639930,0cyclictest15970-21munin-node11:00:310
169930,0watchdog/1327-21ssh11:35:111
169930,0watchdog/122077-21kthreadcore10:45:271
169930,0watchdog/10-21swapper/110:55:111
169930,0watchdog/10-21swapper/109:55:151
28405020,0irq/26-eth0-rx-0-21swapper/307:25:113
269639921,0cyclictest24851-21sort09:05:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional