You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 22:27
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Tue Jul 15, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1774623828,0sleep30-21swapper/306:27:233
1774623828,0sleep30-21swapper/306:27:233
17853993737,0cyclictest3466-21md0_resync10:25:510
1769623725,0sleep20-21swapper/206:26:462
1769623725,0sleep20-21swapper/206:26:462
1752523525,0sleep10-21swapper/106:24:531
1752523525,0sleep10-21swapper/106:24:531
1748322925,0sleep00-21swapper/006:24:240
1748322925,0sleep00-21swapper/006:24:240
8315080,0irq/26-0000:00:0-21swapper/309:28:133
329980,0watchdog/321031-21kthreadcore08:18:073
329980,0watchdog/30-21swapper/307:03:073
8315070,0irq/26-0000:00:26252-21perf08:27:413
8315070,0irq/26-0000:00:18950-21vmstat10:08:153
8315070,0irq/26-0000:00:0-21swapper/310:48:293
8315070,0irq/26-0000:00:0-21swapper/309:23:063
8315070,0irq/26-0000:00:0-21swapper/306:38:073
35170,0ktimersoftd/30-21swapper/308:54:253
329970,0watchdog/30-21swapper/310:13:063
329970,0watchdog/30-21swapper/309:03:113
8315060,0irq/26-0000:00:74132sleep311:56:253
8315060,0irq/26-0000:00:6613-21cpu09:07:583
8315060,0irq/26-0000:00:5570-21memory07:43:093
8315060,0irq/26-0000:00:36212chrt08:03:553
8315060,0irq/26-0000:00:23354-21perl06:53:013
8315060,0irq/26-0000:00:20310-21kthreadcore07:53:113
8315060,0irq/26-0000:00:18569-21awk11:08:053
8315060,0irq/26-0000:00:18489-21taskset08:13:483
8315060,0irq/26-0000:00:15905-21perl09:13:023
8315060,0irq/26-0000:00:0-21swapper/311:48:053
8315060,0irq/26-0000:00:0-21swapper/311:33:073
8315060,0irq/26-0000:00:0-21swapper/311:29:433
8315060,0irq/26-0000:00:0-21swapper/311:03:023
8315060,0irq/26-0000:00:0-21swapper/310:58:053
8315060,0irq/26-0000:00:0-21swapper/310:53:063
8315060,0irq/26-0000:00:0-21swapper/308:40:443
8315060,0irq/26-0000:00:0-21swapper/308:28:053
8315060,0irq/26-0000:00:0-21swapper/307:15:403
8315060,0irq/26-0000:00:0-21swapper/306:47:473
8315060,0irq/26-0000:00:0-21swapper/306:28:053
339960,0migration/38870-21sh09:44:393
329960,0watchdog/383150irq/26-0000:00:08:01:113
329960,0watchdog/310849-21if_err_eth1.307:28:033
329960,0watchdog/30-21swapper/311:43:143
329960,0watchdog/30-21swapper/311:23:463
329960,0watchdog/30-21swapper/311:17:503
329960,0watchdog/30-21swapper/310:42:463
329960,0watchdog/30-21swapper/310:34:423
329960,0watchdog/30-21swapper/310:31:503
329960,0watchdog/30-21swapper/310:23:183
329960,0watchdog/30-21swapper/310:06:313
329960,0watchdog/30-21swapper/309:58:473
329960,0watchdog/30-21swapper/309:32:593
329960,0watchdog/30-21swapper/308:43:033
329960,0watchdog/30-21swapper/308:36:473
329960,0watchdog/30-21swapper/307:36:473
329960,0watchdog/30-21swapper/306:33:473
29935060,0irq/33-eth0-tx-0-21swapper/211:28:082
29925060,0irq/32-eth0-rx-22056-21diskstats08:58:003
29925060,0irq/32-eth0-rx-18146-21taskset09:52:023
29925060,0irq/32-eth0-rx-0-21swapper/311:42:043
29925060,0irq/32-eth0-rx-0-21swapper/311:13:293
29925060,0irq/32-eth0-rx-0-21swapper/310:39:123
29925060,0irq/32-eth0-rx-0-21swapper/310:39:123
29925060,0irq/32-eth0-rx-0-21swapper/310:20:453
29925060,0irq/32-eth0-rx-0-21swapper/309:55:073
29925060,0irq/32-eth0-rx-0-21swapper/309:37:523
29925060,0irq/32-eth0-rx-0-21swapper/309:19:113
29925060,0irq/32-eth0-rx-0-21swapper/308:50:353
29925060,0irq/32-eth0-rx-0-21swapper/306:59:113
259960,0migration/224317-21kthreadcore10:33:042
178559966,0cyclictest6386-21runrttasks08:54:582
178539965,0cyclictest5362-21ps08:48:030
178539960,0cyclictest0-21swapper/010:56:150
178539960,0cyclictest0-21swapper/010:04:420
149960,0migration/03160-21sh08:45:530
149960,0migration/017032-21kthreadcore08:13:070
149960,0migration/013524-21kthreadcore10:43:080
8315050,0irq/26-0000:00:0-21swapper/307:37:593
8315050,0irq/26-0000:00:0-21swapper/307:23:043
8315050,0irq/26-0000:00:0-21swapper/307:18:153
8315050,0irq/26-0000:00:0-21swapper/307:08:093
8315050,0irq/26-0000:00:0-21swapper/306:43:013
4150,0ktimersoftd/00-21swapper/011:12:580
329950,0watchdog/30-21swapper/308:07:593
329950,0watchdog/30-21swapper/307:47:473
29935050,0irq/33-eth0-tx-6315-21iceccd07:18:340
29935050,0irq/33-eth0-tx-30081-21ssh09:18:142
29935050,0irq/33-eth0-tx-17473-21sh08:34:172
29935050,0irq/33-eth0-tx-17171-21munin-node07:33:060
29935050,0irq/33-eth0-tx-14244-21munin-node11:43:032
29935050,0irq/33-eth0-tx-12348-21cat10:08:002
29935050,0irq/33-eth0-tx-11433-21chrt10:06:352
29935050,0irq/33-eth0-tx-10133-21munin-node09:48:162
29935050,0irq/33-eth0-tx-0-21swapper/211:12:152
29935050,0irq/33-eth0-tx-0-21swapper/208:32:252
29935050,0irq/33-eth0-tx-0-21swapper/007:11:420
27150,0ktimersoftd/20-21swapper/210:44:542
27150,0ktimersoftd/20-21swapper/209:41:462
259950,0migration/29407-21sh09:46:212
259950,0migration/27952-21aten_r1power_en06:43:012
259950,0migration/25925-21munin-node10:23:012
259950,0migration/24881-21kthreadcore11:53:032
259950,0migration/23700-21needreboot06:58:062
259950,0migration/226717-21latency10:13:072
259950,0migration/225313-21kthreadcore08:58:052
259950,0migration/220993-21kthreadcore09:13:062
259950,0migration/21963-21timerandwakeup08:43:132
249950,0watchdog/214726-21ssh09:12:392
249950,0watchdog/212470-21ssh08:52:232
249950,0watchdog/20-21swapper/210:53:062
178559955,0cyclictest0-21swapper/211:50:362
178559955,0cyclictest0-21swapper/211:38:092
178559955,0cyclictest0-21swapper/211:27:262
178559950,0cyclictest0-21swapper/211:06:182
178559950,0cyclictest0-21swapper/210:39:302
178559950,0cyclictest0-21swapper/210:39:302
178559950,0cyclictest0-21swapper/210:18:032
178559950,0cyclictest0-21swapper/207:30:562
178549955,0cyclictest0-21swapper/111:09:541
178549955,0cyclictest0-21swapper/109:43:201
178549955,0cyclictest0-21swapper/108:45:191
178549955,0cyclictest0-21swapper/108:36:391
178549955,0cyclictest0-21swapper/106:48:301
178549950,0cyclictest0-21swapper/109:21:351
178549950,0cyclictest0-21swapper/106:37:561
178539955,0cyclictest2092-21md1_raid111:09:390
178539955,0cyclictest0-21swapper/007:39:390
178539955,0cyclictest0-21swapper/006:51:150
178539955,0cyclictest0-21swapper/006:28:330
178539950,0cyclictest0-21swapper/011:37:430
178539950,0cyclictest0-21swapper/010:00:360
178539950,0cyclictest0-21swapper/009:27:010
179950,0migration/16072-21munin-node07:03:051
179950,0migration/130510-21kthreadcore06:33:101
179950,0migration/129543-21kthreadcore09:58:071
179950,0migration/125907-21kthreadcore11:48:051
179950,0migration/122796-21if_eth1.208:58:031
179950,0migration/120678-21kthreadcore09:13:061
179950,0migration/12017-21kthreadcore09:03:071
179950,0migration/119399-21taskset10:09:351
179950,0migration/114777-21munin-plugin-st09:12:461
179950,0migration/113575-21latency10:43:081
179950,0migration/112336-21munin-node10:23:151
179950,0migration/112025-21sh11:38:101
179950,0migration/11045-21switchtime11:13:131
169950,0watchdog/127254-21cut07:17:591
15670250,0sleep218643-21tr10:28:072
149950,0migration/09919-21kthreadcore09:28:050
149950,0migration/09309-21sh09:46:010
149950,0migration/03484-21kthreadcore10:38:090
149950,0migration/03484-21kthreadcore10:38:090
149950,0migration/031191-21kthreadcore10:18:050
149950,0migration/030227-21aten_r1power_en09:03:000
149950,0migration/030019-21switchtime09:18:130
149950,0migration/029167-3199-raid-check08:02:530
149950,0migration/023609-21fschecks_time11:48:000
149950,0migration/021489-21kthreadcore07:13:060
149950,0migration/021488-21kthreadcore08:38:030
149950,0migration/020542-21taskset11:27:550
149950,0migration/019182-21kthreadcore10:48:050
149950,0migration/018953-21taskset11:23:560
149950,0migration/018274-21kthreadcore07:53:100
149950,0migration/018033-21kthreadcore10:08:070
149950,0migration/016612-21taskset07:29:150
149950,0migration/014827-21kthreadcore11:03:020
149950,0migration/014321-21kthreadcore08:33:060
149950,0migration/014136-21kthreadcore09:48:050
149950,0migration/012144-21kthreadcore06:43:080
149950,0migration/01205-21kthreadcore07:43:070
29935040,0irq/33-eth0-tx-30206-21ssh11:33:022
29935040,0irq/33-eth0-tx-23119-21ssh10:48:192
29935040,0irq/33-eth0-tx-2065-21md0_raid108:08:010
29935040,0irq/33-eth0-tx-0-21swapper/209:57:492
29935040,0irq/33-eth0-tx-0-21swapper/209:52:532
29935040,0irq/33-eth0-tx-0-21swapper/209:03:262
29935040,0irq/33-eth0-tx-0-21swapper/008:27:510
29935040,0irq/33-eth0-tx-0-21swapper/007:23:140
29935040,0irq/33-eth0-tx-0-21swapper/006:57:590
259940,0migration/25578-21irqcore08:08:042
259940,0migration/24938-21kthreadcore07:23:102
259940,0migration/229496-21runrttasks06:54:542
259940,0migration/229086-21kthreadcore07:18:062
259940,0migration/227630-21fschecks_time11:13:012
259940,0migration/227206-21fschecks_time08:23:022
259940,0migration/22253-21kthreadcore07:43:082
259940,0migration/217835-21idleruntime09:33:062
259940,0migration/217231-21kthreadcore06:48:052
259940,0migration/215656-21irqcore07:53:092
249940,0watchdog/29266-21hwlatdetect09:28:032
249940,0watchdog/229232-21aten2.4-expect08:02:592
249940,0watchdog/221032-21kthreadcore08:18:072
249940,0watchdog/220630-21perl07:13:032
249940,0watchdog/215243-21kthreadcore08:13:072
249940,0watchdog/20-21swapper/208:37:552
249940,0watchdog/20-21swapper/207:08:152
178559944,0cyclictest0-21swapper/207:48:072
178559943,0cyclictest4293-21awk11:18:022
178559943,0cyclictest25684-21awk07:38:052
178559943,0cyclictest24288-21sadc07:37:462
178559943,0cyclictest23871-21snmpd10:58:262
178559943,0cyclictest23871-21snmpd09:22:552
178559943,0cyclictest23871-21snmpd07:03:252
178559943,0cyclictest23871-21snmpd06:40:102
178559943,0cyclictest23871-21snmpd06:34:252
178559943,0cyclictest23871-21snmpd06:30:252
178549944,0cyclictest0-21swapper/110:57:551
178549944,0cyclictest0-21swapper/109:53:071
178549943,0cyclictest698-21awk09:23:041
178549943,0cyclictest3599-21lldpd09:32:521
178549943,0cyclictest3599-21lldpd07:52:551
178549943,0cyclictest3599-21lldpd07:28:551
178549943,0cyclictest32753-21awk11:53:001
178549943,0cyclictest32323-21awk10:38:061
178549943,0cyclictest32323-21awk10:38:061
178549943,0cyclictest25935-21awk10:53:031
178549943,0cyclictest23871-21snmpd11:44:251
178549943,0cyclictest23871-21snmpd11:24:401
178549943,0cyclictest23871-21snmpd10:34:411
178549943,0cyclictest23871-21snmpd08:38:251
178549943,0cyclictest23871-21snmpd07:59:251
178549943,0cyclictest13824-21irqrtprio07:08:141
178549943,0cyclictest11221-21awk09:48:031
178539944,0cyclictest0-21swapper/011:55:130
178539944,0cyclictest0-21swapper/011:19:160
178539944,0cyclictest0-21swapper/008:52:480
178539943,0cyclictest3599-21lldpd10:29:150
178539943,0cyclictest26926-21sadc09:57:460
178539943,0cyclictest23871-21snmpd09:38:100
178539943,0cyclictest23774-21awk06:53:030
178539943,0cyclictest18014-21awk09:33:070
178539943,0cyclictest15313-21awk11:43:030
179940,0migration/19984-21kthreadcore08:08:061
179940,0migration/19226-21taskset07:24:191
179940,0migration/13417-21timerandwakeup10:18:131
179940,0migration/132625-21fschecks_time07:43:041
179940,0migration/132224-21kthreadcore08:23:061
179940,0migration/131736-21kthreadcore11:33:051
179940,0migration/130546-21irqcore08:03:041
179940,0migration/129703-21kthreadcore09:38:061
179940,0migration/121958-21kthreadcore07:13:071
179940,0migration/120901-21runrttasks11:28:031
179940,0migration/112152-21kthreadcore07:48:061
179940,0migration/110053-21users11:18:121
169940,0watchdog/15358-21iostat_ios08:48:031
169940,0watchdog/14682-21iostat_ios10:03:071
169940,0watchdog/130320-21df06:57:591
169940,0watchdog/124533-21ssh10:52:341
169940,0watchdog/124360-21kthreadcore10:13:061
169940,0watchdog/117680-21kthreadcore10:28:061
169940,0watchdog/116665-21ps11:03:341
169940,0watchdog/10-21swapper/108:53:071
169940,0watchdog/10-21swapper/108:18:071
149940,0migration/08476-21kthreadcore07:03:070
149940,0migration/06760-21kthreadcore10:58:060
149940,0migration/026196-21fschecks_time06:33:050
178559930,0cyclictest22133-21df07:58:012
178549932,0cyclictest25748-21irqcore07:38:061
178549930,0cyclictest17495-21df07:32:591
178549930,0cyclictest12365-21tune2fs08:13:021
178539933,0cyclictest5943-21kworker/0:108:22:490
178539933,0cyclictest5943-21kworker/0:108:17:510
178539933,0cyclictest5943-21kworker/0:107:57:530
178539933,0cyclictest5943-21kworker/0:107:47:490
178539933,0cyclictest4314-21kworker/0:209:12:570
178539933,0cyclictest4314-21kworker/0:209:07:550
178539933,0cyclictest24511-21kworker/0:206:37:530
178539933,0cyclictest2414-21kworker/0:110:32:570
178539933,0cyclictest2414-21kworker/0:110:12:470
178539933,0cyclictest1157-21kworker/0:008:58:030
169930,0watchdog/10-21swapper/109:27:551
169930,0watchdog/10-21swapper/108:27:591
169930,0watchdog/10-21swapper/106:54:271
169930,0watchdog/10-21swapper/106:43:031
169930,0watchdog/10-21swapper/106:28:071
149930,0migration/05540-21runrttasks11:37:550
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional