You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-03 - 00:22
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Tue Sep 02, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
121726261,0sleep00-21swapper/006:19:480
8472994949,0cyclictest8310-21md0_resync10:56:402
820123625,0sleep10-21swapper/106:22:341
801423425,0sleep20-21swapper/206:20:272
827122824,0sleep30-21swapper/306:23:203
8470992020,0cyclictest41ktimersoftd/010:56:400
329980,0watchdog/30-21swapper/310:25:073
84719975,0cyclictest0-21swapper/111:27:531
8315070,0irq/26-0000:00:7017-21munin-run10:34:473
8315070,0irq/26-0000:00:3413-21sh09:06:283
8315070,0irq/26-0000:00:28760-21munin-node07:40:153
8315070,0irq/26-0000:00:0-21swapper/310:55:113
8315070,0irq/26-0000:00:0-21swapper/308:40:143
329970,0watchdog/314466-21taskset07:34:313
329970,0watchdog/30-21swapper/309:30:073
329970,0watchdog/30-21swapper/308:56:193
84729960,0cyclictest0-21swapper/208:42:382
84719960,0cyclictest0-21swapper/110:28:461
84709960,0cyclictest13987-21ssh09:31:230
84709960,0cyclictest0-21swapper/011:32:380
84709960,0cyclictest0-21swapper/009:01:260
84709960,0cyclictest0-21swapper/007:48:450
8315060,0irq/26-0000:00:8945-21tr06:25:003
8315060,0irq/26-0000:00:6027-21grep11:45:043
8315060,0irq/26-0000:00:32724-21entropy07:24:593
8315060,0irq/26-0000:00:32181-21aten2.4-expect08:29:593
8315060,0irq/26-0000:00:30447-21ls06:39:473
8315060,0irq/26-0000:00:28642-21cut10:45:063
8315060,0irq/26-0000:00:27454-21munin-node11:05:053
8315060,0irq/26-0000:00:26769-21yum09:00:163
8315060,0irq/26-0000:00:20934-21port_sshpriv07:35:093
8315060,0irq/26-0000:00:17408-21iostat_ios08:20:043
8315060,0irq/26-0000:00:13783-21grep11:14:583
8315060,0irq/26-0000:00:13783-21grep11:14:583
8315060,0irq/26-0000:00:10318-21pmu-power11:10:073
8315060,0irq/26-0000:00:0-21swapper/311:35:273
8315060,0irq/26-0000:00:0-21swapper/310:40:043
8315060,0irq/26-0000:00:0-21swapper/309:40:043
8315060,0irq/26-0000:00:0-21swapper/309:17:143
8315060,0irq/26-0000:00:0-21swapper/309:14:183
8315060,0irq/26-0000:00:0-21swapper/308:15:103
8315060,0irq/26-0000:00:0-21swapper/307:55:053
8315060,0irq/26-0000:00:0-21swapper/307:20:113
8315060,0irq/26-0000:00:0-21swapper/306:50:043
8315060,0irq/26-0000:00:0-21swapper/306:50:043
8315060,0irq/26-0000:00:0-21swapper/306:45:103
339960,0migration/330955-21sh08:27:163
339960,0migration/325382-21timerwakeupswit10:00:153
329960,0watchdog/383150irq/26-0000:00:06:59:033
329960,0watchdog/3299250irq/32-eth0-rx-10:37:273
329960,0watchdog/30-21swapper/311:50:593
329960,0watchdog/30-21swapper/311:29:473
329960,0watchdog/30-21swapper/311:21:233
329960,0watchdog/30-21swapper/311:00:273
329960,0watchdog/30-21swapper/310:20:033
329960,0watchdog/30-21swapper/310:11:393
329960,0watchdog/30-21swapper/310:06:233
329960,0watchdog/30-21swapper/309:50:233
329960,0watchdog/30-21swapper/309:44:473
329960,0watchdog/30-21swapper/309:24:553
329960,0watchdog/30-21swapper/308:44:593
329960,0watchdog/30-21swapper/308:34:473
329960,0watchdog/30-21swapper/307:10:193
329960,0watchdog/30-21swapper/307:06:113
329960,0watchdog/30-21swapper/306:59:473
329960,0watchdog/30-21swapper/306:44:353
29935060,0irq/33-eth0-tx-31130-21tune2fs11:24:592
29935060,0irq/33-eth0-tx-10065-21rm11:10:052
29925060,0irq/32-eth0-rx-0-21swapper/311:41:443
29925060,0irq/32-eth0-rx-0-21swapper/310:50:193
29925060,0irq/32-eth0-rx-0-21swapper/310:15:223
29925060,0irq/32-eth0-rx-0-21swapper/309:37:443
29925060,0irq/32-eth0-rx-0-21swapper/309:20:083
149960,0migration/023565-21munin-run09:39:460
84729955,0cyclictest0-21swapper/209:27:342
84729955,0cyclictest0-21swapper/207:46:402
84729955,0cyclictest0-21swapper/206:48:492
84729955,0cyclictest0-21swapper/206:44:242
84729955,0cyclictest0-21swapper/206:27:512
84719955,0cyclictest0-21swapper/110:35:231
84719955,0cyclictest0-21swapper/108:50:141
84719955,0cyclictest0-21swapper/108:31:561
84719955,0cyclictest0-21swapper/107:32:491
84719950,0cyclictest0-21swapper/111:06:461
84719950,0cyclictest0-21swapper/106:28:201
84709955,0cyclictest0-21swapper/011:49:580
84709955,0cyclictest0-21swapper/010:34:370
84709955,0cyclictest0-21swapper/009:40:320
84709955,0cyclictest0-21swapper/009:06:290
84709950,0cyclictest0-21swapper/010:17:440
8315050,0irq/26-0000:00:5427-21ls07:50:013
8315050,0irq/26-0000:00:0-21swapper/307:45:063
8315050,0irq/26-0000:00:0-21swapper/307:15:003
4150,0ktimersoftd/00-21swapper/011:01:330
4150,0ktimersoftd/00-21swapper/009:25:140
339950,0migration/327216-21fschecks_time08:05:053
339950,0migration/317918-21kthreadcore06:30:043
329950,0watchdog/30-21swapper/308:49:553
329950,0watchdog/30-21swapper/308:10:193
329950,0watchdog/30-21swapper/308:00:033
29935050,0irq/33-eth0-tx-9566-21kthreadcore10:35:042
29935050,0irq/33-eth0-tx-8685-21munin-node06:25:010
29935050,0irq/33-eth0-tx-6315-21iceccd08:10:540
29935050,0irq/33-eth0-tx-4416-21munin-node07:50:020
29935050,0irq/33-eth0-tx-31981-21ssh11:40:502
29935050,0irq/33-eth0-tx-20098-21ssh11:16:082
29935050,0irq/33-eth0-tx-20098-21ssh11:16:082
29935050,0irq/33-eth0-tx-18776-21munin-node10:00:012
29935050,0irq/33-eth0-tx-18294-21ssh09:59:102
29935050,0irq/33-eth0-tx-15883-21munin-node10:40:072
29935050,0irq/33-eth0-tx-1369-21munin-node11:45:072
29935050,0irq/33-eth0-tx-11820-21munin-node07:55:030
29935050,0irq/33-eth0-tx-0-21swapper/211:35:042
29935050,0irq/33-eth0-tx-0-21swapper/211:30:062
29935050,0irq/33-eth0-tx-0-21swapper/209:35:042
29935050,0irq/33-eth0-tx-0-21swapper/208:45:252
29935050,0irq/33-eth0-tx-0-21swapper/208:36:512
29935050,0irq/33-eth0-tx-0-21swapper/008:20:070
29935050,0irq/33-eth0-tx-0-21swapper/007:41:560
29935050,0irq/33-eth0-tx-0-21swapper/006:41:460
29925050,0irq/32-eth0-rx-0-21swapper/311:24:543
29925050,0irq/32-eth0-rx-0-21swapper/309:54:463
27150,0ktimersoftd/20-21swapper/208:13:192
259950,0migration/29771-21munin-node10:10:072
259950,0migration/2526-21timerwakeupswit08:05:172
259950,0migration/230756-21munin-node10:45:082
259950,0migration/230140-21sh11:24:102
259950,0migration/22943-21unixbench_singl09:05:132
259950,0migration/22741-21cpuspeed_turbos08:50:002
259950,0migration/224088-21kthreadcore10:20:062
259950,0migration/222859-21kthreadcore08:00:052
259950,0migration/216084-21kthreadcore09:15:052
259950,0migration/21479-21munin-node11:05:092
259950,0migration/213421-21kthreadcore08:55:052
249950,0watchdog/219013-21ssh10:59:592
19150,0ktimersoftd/10-21swapper/108:27:131
179950,0migration/125318-21fschecks_time10:45:011
179950,0migration/110166-21kthreadcore09:10:061
179950,0migration/110017-21kthreadcore11:30:041
149950,0migration/09575-21taskset07:05:370
149950,0migration/08422-21latency_hist08:34:460
149950,0migration/08155-21kthreadcore08:50:060
149950,0migration/07501-21kthreadcore09:50:040
149950,0migration/04790-21fschecks_time11:10:000
149950,0migration/03610-21kthreadcore11:45:040
149950,0migration/030397-21munin-node10:25:040
149950,0migration/029782-21taskset09:23:170
149950,0migration/029381-21kthreadcore07:20:090
149950,0migration/029378-21kthreadcore11:05:040
149950,0migration/028700-21kthreadcore08:45:040
149950,0migration/028375-21munin-node11:20:130
149950,0migration/028303-21kthreadcore11:40:040
149950,0migration/026371-21kthreadcore10:45:050
149950,0migration/026201-21kthreadcore08:25:090
149950,0migration/02388-21taskset07:03:220
149950,0migration/021293-21kthreadcore08:00:050
149950,0migration/021210-21kthreadcore08:40:060
149950,0migration/021144-21sh11:18:560
149950,0migration/021144-21sh11:18:560
149950,0migration/020801-21kthreadcore10:00:060
149950,0migration/020070-21kthreadcore10:40:040
149950,0migration/02005-21kthreadcore07:25:030
149950,0migration/019867-21taskset06:51:330
149950,0migration/019867-21taskset06:51:330
149950,0migration/019368-21taskset08:59:020
149950,0migration/018694-21kthreadcore06:30:040
149950,0migration/014537-21latency_hist07:34:460
149950,0migration/013053-21kthreadcore10:35:050
149950,0migration/01274-21taskset09:49:410
149950,0migration/011014-21fschecks_time09:55:000
149950,0migration/010853-21latency09:10:060
149950,0migration/010739-21kthreadcore08:15:060
84729943,0cyclictest3599-21lldpd06:35:212
84729943,0cyclictest30381-21awk10:25:042
84729943,0cyclictest23871-21snmpd07:54:392
84729943,0cyclictest23871-21snmpd06:34:392
84729943,0cyclictest21050-21awk09:00:042
84729943,0cyclictest17242-21awk08:20:022
84719944,0cyclictest0-21swapper/109:49:021
84719944,0cyclictest0-21swapper/109:23:391
84719943,0cyclictest9644-21awk08:35:031
84719943,0cyclictest5424-21sadc06:44:471
84719943,0cyclictest3983-21awk07:05:011
84719943,0cyclictest30593-21awk07:45:041
84719943,0cyclictest28848-21awk07:00:031
84719943,0cyclictest24500-21awk06:35:051
84719943,0cyclictest23871-21snmpd11:54:241
84719943,0cyclictest23871-21snmpd08:47:541
84719943,0cyclictest23871-21snmpd07:16:541
84719943,0cyclictest23871-21snmpd06:50:541
84719943,0cyclictest23871-21snmpd06:50:541
84719943,0cyclictest21451-21awk06:55:021
84719943,0cyclictest20954-21awk09:00:041
84719943,0cyclictest20171-21awk08:00:031
84719943,0cyclictest11443-21awk09:55:021
84719943,0cyclictest10838-21awk10:55:021
84709944,0cyclictest0-21swapper/010:04:580
84709943,0cyclictest695-21awk08:30:020
84709943,0cyclictest3262-21irqbalance09:15:010
4140,0ktimersoftd/08309-21md1_resync11:27:200
29935040,0irq/33-eth0-tx-11396-21ssh09:10:132
29935040,0irq/33-eth0-tx-0-21swapper/211:50:062
29935040,0irq/33-eth0-tx-0-21swapper/210:49:592
29935040,0irq/33-eth0-tx-0-21swapper/210:15:032
29935040,0irq/33-eth0-tx-0-21swapper/209:45:542
29935040,0irq/33-eth0-tx-0-21swapper/209:41:222
29935040,0irq/33-eth0-tx-0-21swapper/209:20:082
29935040,0irq/33-eth0-tx-0-21swapper/208:30:212
29935040,0irq/33-eth0-tx-0-21swapper/208:25:232
259940,0migration/26595-21latency_hist09:29:462
259940,0migration/25178-21kthreadcore07:05:022
259940,0migration/23027-21irqcore09:50:022
259940,0migration/228719-21irqcore10:05:032
259940,0migration/219085-21taskset07:15:042
259940,0migration/216618-21kthreadcore07:35:052
259940,0migration/215426-21kthreadcore07:55:052
249940,0watchdog/213387-21df06:49:592
249940,0watchdog/213387-21df06:49:592
249940,0watchdog/20-21swapper/207:00:032
179940,0migration/16957-21kthreadcore08:10:051
179940,0migration/16804-21kthreadcore11:45:051
179940,0migration/16485-21kthreadcore07:50:041
179940,0migration/131726-21taskset09:43:281
179940,0migration/129835-21kthreadcore10:05:041
179940,0migration/126822-21kthreadcore07:40:071
179940,0migration/125794-21kthreadcore10:20:061
179940,0migration/122830-21taskset10:40:301
179940,0migration/117858-21kthreadcore06:30:041
179940,0migration/117787-21switchtime08:55:141
179940,0migration/11724-21kthreadcore09:05:051
179940,0migration/116451-21cpuspeed_turbos08:19:591
179940,0migration/114659-21kthreadcore10:15:031
179940,0migration/114434-21munin-node11:15:011
179940,0migration/114434-21munin-node11:15:011
179940,0migration/111882-21runrttasks07:54:591
179940,0migration/111574-21kthreadcore08:15:071
169940,0watchdog/132449-21sh11:42:031
169940,0watchdog/127936-21needreboot11:20:071
169940,0watchdog/127565-21if_eth108:05:071
169940,0watchdog/124564-21kthreadcore10:00:071
169940,0watchdog/111826-21awk06:45:071
169940,0watchdog/110963-21ssh10:12:031
169940,0watchdog/10-21swapper/109:50:031
169940,0watchdog/10-21swapper/109:35:311
169940,0watchdog/10-21swapper/109:32:311
169940,0watchdog/10-21swapper/109:24:591
169940,0watchdog/10-21swapper/107:35:031
149940,0migration/08425-21kthreadcore10:10:050
149940,0migration/025228-21kthreadcore06:55:050
84729932,0cyclictest26360-21irqrtprio07:20:082
84729932,0cyclictest21579-21awk06:55:032
84729932,0cyclictest11373-21awk07:10:082
84729932,0cyclictest10249-21sed08:15:062
84719932,0cyclictest30903-21cat07:20:101
84719930,0cyclictest21502-21cut08:40:061
84719930,0cyclictest20523-21cut11:35:021
84719930,0cyclictest12416-21cut07:10:091
84709933,0cyclictest9194-21kworker/0:007:09:580
84709933,0cyclictest3394-21kworker/0:208:04:490
84709933,0cyclictest21546-21kworker/0:206:34:560
84709933,0cyclictest20884-21kworker/0:110:49:590
84709933,0cyclictest20884-21kworker/0:110:20:230
84709932,0cyclictest18330-21grep11:35:010
84709930,0cyclictest17547-21munin-node07:15:070
29935030,0irq/33-eth0-tx-0-21swapper/210:31:092
29935030,0irq/33-eth0-tx-0-21swapper/007:34:590
29935030,0irq/33-eth0-tx-0-21swapper/006:44:580
259930,0migration/227097-21kthreadcore07:40:072
249930,0watchdog/20-21swapper/207:34:312
249930,0watchdog/20-21swapper/207:25:032
19130,0ktimersoftd/15156-1kworker/1:2H11:10:281
169930,0watchdog/1736-21munin-node10:50:031
169930,0watchdog/15873-21ssh10:31:231
169930,0watchdog/123328-21diskmemload09:16:031
169930,0watchdog/10-21swapper/111:00:031
169930,0watchdog/10-21swapper/107:25:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional