You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-22 - 21:25
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2.osadl.org (updated Tue Apr 22, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1262828180,0sleep10-21swapper/106:28:311
1268226357,0sleep00-21swapper/006:29:060
1299526056,0sleep20-21swapper/206:32:372
13085994646,0cyclictest30426-21md0_resync07:53:280
1274023828,0sleep30-21swapper/306:29:463
13086993636,0cyclictest30424-21md1_resync07:53:281
8315070,0irq/26-0000:00:16307-21chrt07:43:123
329970,0watchdog/314330-21cpu10:23:043
8315060,0irq/26-0000:00:8254-21head09:53:093
8315060,0irq/26-0000:00:8208-21head11:23:183
8315060,0irq/26-0000:00:26091-21sed10:53:073
8315060,0irq/26-0000:00:19897-21perl10:03:093
8315060,0irq/26-0000:00:15456-21yum11:53:153
8315060,0irq/26-0000:00:0-21swapper/311:20:143
8315060,0irq/26-0000:00:0-21swapper/311:03:093
8315060,0irq/26-0000:00:0-21swapper/310:43:113
8315060,0irq/26-0000:00:0-21swapper/309:50:163
8315060,0irq/26-0000:00:0-21swapper/309:42:453
8315060,0irq/26-0000:00:0-21swapper/308:59:523
8315060,0irq/26-0000:00:0-21swapper/307:58:083
8315060,0irq/26-0000:00:0-21swapper/307:13:033
8315060,0irq/26-0000:00:0-21swapper/306:33:113
4160,0ktimersoftd/00-21swapper/009:08:330
329960,0watchdog/39605-21kthreadcore08:43:133
329960,0watchdog/3299250irq/32-eth0-rx-09:06:483
329960,0watchdog/325997-21munin-node07:48:133
329960,0watchdog/311126-21perl06:53:173
329960,0watchdog/30-21swapper/311:58:403
329960,0watchdog/30-21swapper/311:37:563
329960,0watchdog/30-21swapper/311:36:323
329960,0watchdog/30-21swapper/311:28:443
329960,0watchdog/30-21swapper/311:08:403
329960,0watchdog/30-21swapper/310:59:483
329960,0watchdog/30-21swapper/310:50:483
329960,0watchdog/30-21swapper/310:38:243
329960,0watchdog/30-21swapper/310:35:043
329960,0watchdog/30-21swapper/310:08:083
329960,0watchdog/30-21swapper/309:59:083
329960,0watchdog/30-21swapper/309:34:003
329960,0watchdog/30-21swapper/309:24:283
329960,0watchdog/30-21swapper/309:13:043
329960,0watchdog/30-21swapper/309:09:483
329960,0watchdog/30-21swapper/308:51:573
329960,0watchdog/30-21swapper/308:34:093
329960,0watchdog/30-21swapper/308:13:573
329960,0watchdog/30-21swapper/307:41:413
329960,0watchdog/30-21swapper/307:36:253
329960,0watchdog/30-21swapper/307:07:533
329960,0watchdog/30-21swapper/307:00:053
329960,0watchdog/30-21swapper/306:47:573
29925060,0irq/32-eth0-rx-23876-21cat09:18:113
29925060,0irq/32-eth0-rx-0-21swapper/311:44:423
29925060,0irq/32-eth0-rx-0-21swapper/311:14:363
29925060,0irq/32-eth0-rx-0-21swapper/310:32:103
29925060,0irq/32-eth0-rx-0-21swapper/310:20:413
29925060,0irq/32-eth0-rx-0-21swapper/310:17:083
29925060,0irq/32-eth0-rx-0-21swapper/309:47:323
29925060,0irq/32-eth0-rx-0-21swapper/308:53:193
149960,0migration/011605-21sh11:31:280
130879960,0cyclictest0-21swapper/209:13:082
130859960,0cyclictest21211-21diskmemload11:51:060
130859960,0cyclictest0-21swapper/010:53:520
130859960,0cyclictest0-21swapper/010:52:080
130859960,0cyclictest0-21swapper/008:58:380
130859960,0cyclictest0-21swapper/008:43:390
8315050,0irq/26-0000:00:4814-21latency_hist09:27:523
8315050,0irq/26-0000:00:27924-21latency_hist06:42:523
8315050,0irq/26-0000:00:26944-21tr07:23:093
8315050,0irq/26-0000:00:0-21swapper/311:47:593
8315050,0irq/26-0000:00:0-21swapper/308:37:563
8315050,0irq/26-0000:00:0-21swapper/308:08:463
8315050,0irq/26-0000:00:0-21swapper/307:53:193
8315050,0irq/26-0000:00:0-21swapper/307:18:183
339950,0migration/332742-21gltestperf08:18:073
329950,0watchdog/30-21swapper/308:28:093
329950,0watchdog/30-21swapper/308:22:573
329950,0watchdog/30-21swapper/307:27:533
329950,0watchdog/30-21swapper/307:03:293
329950,0watchdog/30-21swapper/306:38:333
29935050,0irq/33-eth0-tx-8655-21munin-node11:53:072
29935050,0irq/33-eth0-tx-6315-21iceccd06:57:410
29935050,0irq/33-eth0-tx-29213-21grep11:18:102
29935050,0irq/33-eth0-tx-14135-21munin-node10:23:062
29935050,0irq/33-eth0-tx-10535-21munin-node09:58:152
29935050,0irq/33-eth0-tx-0-21swapper/007:18:110
29925050,0irq/32-eth0-rx-13368-21cut08:03:093
259950,0migration/29450-21kthreadcore09:33:062
259950,0migration/26860-21missed_timers10:38:142
259950,0migration/24143-21kthreadcore09:03:122
259950,0migration/231592-21kthreadcore10:58:092
259950,0migration/226136-21kthreadcore10:33:062
259950,0migration/223422-21kthreadcore08:33:132
259950,0migration/220248-21sh09:17:552
259950,0migration/213083-21taskset11:06:192
249950,0watchdog/225170-21aten2.4-expect07:03:052
19150,0ktimersoftd/10-21swapper/107:26:091
149950,0migration/08792-21kthreadcore08:23:110
149950,0migration/06832-21latency_hist09:32:520
149950,0migration/06120-21taskset08:18:230
149950,0migration/0575-21kthreadcore10:58:100
149950,0migration/05271-21sh08:41:320
149950,0migration/04934-21kthreadcore10:38:120
149950,0migration/029070-21sh08:36:230
149950,0migration/027296-21taskset10:07:480
149950,0migration/02678-21taskset06:47:320
149950,0migration/026358-21kthreadcore08:13:130
149950,0migration/02529-21kthreadcore07:08:100
149950,0migration/025247-21kthreadcore11:38:080
149950,0migration/023083-21kthreadcore09:18:100
149950,0migration/023069-21sh10:29:300
149950,0migration/022004-21kthreadcore06:58:140
149950,0migration/020007-21latency_hist07:47:520
149950,0migration/017930-21kthreadcore07:43:130
149950,0migration/017519-21kthreadcore10:23:100
149950,0migration/016537-21aten2.4-expect08:53:000
149950,0migration/015227-21kthreadcore11:33:060
149950,0migration/015054-21kthreadcore09:58:120
149950,0migration/01425-21aten_r1power_po11:23:030
149950,0migration/013230-21taskset10:20:530
149950,0migration/012111-21sh11:03:450
149950,0migration/011783-21kthreadcore07:13:160
149950,0migration/010324-21fschecks_time10:43:060
130879955,0cyclictest21211-21diskmemload09:08:062
130879955,0cyclictest0-21swapper/209:27:292
130879950,0cyclictest0-21swapper/208:56:542
130869955,0cyclictest0-21swapper/111:35:531
130869955,0cyclictest0-21swapper/111:30:411
130869955,0cyclictest0-21swapper/108:22:071
130869950,0cyclictest0-21swapper/111:19:381
130869950,0cyclictest0-21swapper/110:12:001
130869950,0cyclictest0-21swapper/109:37:191
130869950,0cyclictest0-21swapper/109:25:201
130869950,0cyclictest0-21swapper/109:20:461
130869950,0cyclictest0-21swapper/108:45:271
130859955,0cyclictest0-21swapper/011:43:040
130859955,0cyclictest0-21swapper/011:08:230
130859955,0cyclictest0-21swapper/009:15:200
130859950,0cyclictest0-21swapper/009:38:400
130859950,0cyclictest0-21swapper/007:29:520
130859950,0cyclictest0-21swapper/006:48:050
130859950,0cyclictest0-21swapper/006:39:150
29935040,0irq/33-eth0-tx-22641-21ssh10:28:172
29935040,0irq/33-eth0-tx-2136-21perl11:23:052
29935040,0irq/33-eth0-tx-2065-21md0_raid110:43:112
29935040,0irq/33-eth0-tx-20111-21ssh10:53:002
29935040,0irq/33-eth0-tx-13925-21ssh11:07:562
29935040,0irq/33-eth0-tx-0-21swapper/211:48:242
29935040,0irq/33-eth0-tx-0-21swapper/211:44:262
29935040,0irq/33-eth0-tx-0-21swapper/211:33:132
29935040,0irq/33-eth0-tx-0-21swapper/211:28:392
29935040,0irq/33-eth0-tx-0-21swapper/211:13:062
29935040,0irq/33-eth0-tx-0-21swapper/210:04:152
29935040,0irq/33-eth0-tx-0-21swapper/209:53:042
29935040,0irq/33-eth0-tx-0-21swapper/209:49:052
29935040,0irq/33-eth0-tx-0-21swapper/209:39:332
29935040,0irq/33-eth0-tx-0-21swapper/209:29:342
29935040,0irq/33-eth0-tx-0-21swapper/208:58:392
29935040,0irq/33-eth0-tx-0-21swapper/008:08:040
29935040,0irq/33-eth0-tx-0-21swapper/007:23:070
259940,0migration/28742-21kthreadcore06:48:152
259940,0migration/26355-21sendmail_mailqu07:08:152
259940,0migration/232293-21fschecks_count08:18:062
259940,0migration/230500-21kthreadcore07:53:152
259940,0migration/224178-21kthreadcore06:38:122
259940,0migration/218598-21fschecks_time06:58:092
259940,0migration/218068-21sh10:47:532
259940,0migration/217401-21latency_hist11:57:522
259940,0migration/215918-21kthreadcore06:33:102
259940,0migration/210098-21kthreadcore10:18:112
249940,0watchdog/20-21swapper/210:13:042
249940,0watchdog/20-21swapper/209:42:522
249940,0watchdog/20-21swapper/207:58:132
19140,0ktimersoftd/10-21swapper/112:01:541
179940,0migration/14711-21kthreadcore06:48:131
179940,0migration/131191-21munin-node08:58:121
179940,0migration/130850-21fschecks_time11:43:071
179940,0migration/120489-21irqrtprio10:03:131
169940,0watchdog/16444-21ssh09:32:081
169940,0watchdog/129064-21diskstats10:58:041
169940,0watchdog/12666-21kthreadcore07:28:131
169940,0watchdog/119386-21users07:43:211
169940,0watchdog/11689-21munin-node06:43:131
169940,0watchdog/114301-21ssh10:23:041
169940,0watchdog/10-21swapper/109:57:521
169940,0watchdog/10-21swapper/109:03:131
169940,0watchdog/10-21swapper/107:48:051
149940,0migration/06195-21sh09:31:220
149940,0migration/05062-21kthreadcore09:53:060
149940,0migration/03835-21irqrtprio07:58:110
149940,0migration/029306-21latency_hist10:12:520
149940,0migration/028293-21kthreadcore11:18:090
149940,0migration/018872-21kthreadcore06:33:110
130879944,0cyclictest0-21swapper/208:23:572
130879944,0cyclictest0-21swapper/207:33:492
130879943,0cyclictest7125-21awk08:43:102
130879943,0cyclictest3599-21lldpd10:08:562
130879943,0cyclictest3599-21lldpd08:31:062
130879943,0cyclictest3599-21lldpd08:07:362
130879943,0cyclictest3599-21lldpd07:52:062
130879943,0cyclictest3262-21irqbalance11:37:562
130879943,0cyclictest3262-21irqbalance08:48:562
130879943,0cyclictest3262-21irqbalance08:38:062
130879943,0cyclictest3262-21irqbalance06:52:572
130879943,0cyclictest3262-21irqbalance06:42:572
130879943,0cyclictest25537-21awk08:13:112
130879943,0cyclictest23871-21snmpd08:12:032
130879943,0cyclictest23204-21awk07:23:072
130879943,0cyclictest12427-21sh07:42:522
130869944,0cyclictest0-21swapper/111:12:041
130869944,0cyclictest0-21swapper/109:47:341
130869944,0cyclictest0-21swapper/108:32:411
130869944,0cyclictest0-21swapper/108:23:131
130869944,0cyclictest0-21swapper/108:17:421
130869944,0cyclictest0-21swapper/106:33:391
130869943,0cyclictest6259-21awk11:03:081
130869943,0cyclictest3599-21lldpd11:54:091
130869943,0cyclictest3599-21lldpd07:38:001
130869943,0cyclictest3487-21awk07:58:101
130869943,0cyclictest24846-21sadc07:02:531
130869943,0cyclictest24219-21sadc08:12:521
130869943,0cyclictest23871-21snmpd11:25:191
130869943,0cyclictest23871-21snmpd10:51:341
130869943,0cyclictest23871-21snmpd10:18:041
130869943,0cyclictest23871-21snmpd07:03:171
130869943,0cyclictest21906-21sadc07:22:521
130869943,0cyclictest21117-21awk10:53:041
130869943,0cyclictest17457-21awk08:53:031
130869943,0cyclictest17167-21awk11:13:031
130869943,0cyclictest1163-21awk07:08:091
130869943,0cyclictest11496-21awk06:53:191
130859944,0cyclictest0-21swapper/010:09:440
130859944,0cyclictest0-21swapper/007:04:480
130859943,0cyclictest3599-21lldpd09:49:520
130859943,0cyclictest3599-21lldpd09:45:220
130859943,0cyclictest31927-21kworker/0:109:23:110
130859943,0cyclictest23871-21snmpd11:54:190
130859943,0cyclictest23871-21snmpd10:33:040
130859940,0cyclictest14147-21munin-node08:28:160
249930,0watchdog/20-21swapper/207:18:172
249930,0watchdog/20-21swapper/207:13:132
179930,0migration/131508-21runrttasks10:34:001
179930,0migration/122859-21taskset10:28:541
169930,0watchdog/130426-21md0_resync08:38:011
169930,0watchdog/10-21swapper/111:48:201
169930,0watchdog/10-21swapper/111:37:561
169930,0watchdog/10-21swapper/110:38:081
169930,0watchdog/10-21swapper/109:53:481
169930,0watchdog/10-21swapper/109:09:401
169930,0watchdog/10-21swapper/108:34:251
130879932,0cyclictest3528-21sensors07:28:162
130879932,0cyclictest12693-21stty07:43:052
130869933,0cyclictest3599-21lldpd07:33:001
130869933,0cyclictest0-21swapper/109:49:181
130869932,0cyclictest6386-21runrttasks08:49:091
130869932,0cyclictest3599-21lldpd10:43:061
130869932,0cyclictest3599-21lldpd09:13:191
130869932,0cyclictest3599-21lldpd07:12:541
130869932,0cyclictest3599-21lldpd06:38:041
130869932,0cyclictest30570-21awk10:13:051
130869932,0cyclictest16630-21awk09:38:081
130869932,0cyclictest10873-21awk08:03:071
130859933,0cyclictest4133-21kworker/0:011:13:380
130859933,0cyclictest31927-21kworker/0:109:03:550
130859933,0cyclictest31927-21kworker/0:108:03:030
130859933,0cyclictest31927-21kworker/0:107:38:170
130859933,0cyclictest31347-21kworker/0:211:57:520
130859931,0cyclictest4924-21sed07:33:090
130859930,0cyclictest15304-21ssh08:50:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional