You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-03 - 03:33

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot2.osadl.org (updated Sat Dec 03, 2022 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
267025142,0sleep10-21swapper/118:30:551
265423727,0sleep20-21swapper/218:30:482
271923026,0sleep00-21swapper/018:31:300
2842622824,0sleep30-21swapper/318:28:573
8315060,0irq/26-0000:00:10132-21crond23:33:542
339960,0migration/33330-21sh21:19:493
339960,0migration/322705-21missed_timers23:19:143
339960,0migration/321528-21sh23:02:443
339960,0migration/317073-21sh22:39:453
339960,0migration/314816-21sh23:15:013
339960,0migration/311590-21munin-node20:09:113
329960,0watchdog/30-21swapper/319:34:153
30039960,0cyclictest19028-21ps22:24:191
28015060,0irq/29-eth0-tx-17565-21munin-node18:44:170
28015060,0irq/29-eth0-tx-13924-21munin-node19:04:110
28005060,0irq/28-eth0-rx-31914-21munin-node23:24:293
28005060,0irq/28-eth0-rx-31243-21munin-node20:44:143
28005060,0irq/28-eth0-rx-23330-21users20:54:243
28005060,0irq/28-eth0-rx-22371-21ssh22:24:483
28005060,0irq/28-eth0-rx-21786-21tr21:14:133
149960,0migration/06003-21sh23:10:230
149960,0migration/016281-21smart_sda20:09:170
149960,0migration/010870-21fschecks_time19:24:130
8315050,0irq/26-0000:00:3032-17sedispatch20:23:552
8286250,0sleep211451-21kthreadcore18:59:142
4150,0ktimersoftd/00-21swapper/022:24:060
339950,0migration/39800-21runrttasks18:34:303
339950,0migration/39723-21kthreadcore22:54:183
339950,0migration/38731-21munin-node20:49:113
339950,0migration/37594-21irqcore22:19:123
339950,0migration/36649-21kthreadcore20:04:133
339950,0migration/36190-21fschecks_time21:24:103
339950,0migration/35589-21timerwakeupswit20:24:223
339950,0migration/35480-21proc_pri19:39:173
339950,0migration/330798-21kthreadcore22:49:123
339950,0migration/327233-21kthreadcore18:49:143
339950,0migration/318831-21crond23:38:543
339950,0migration/318218-21fschecks_time19:29:093
339950,0migration/317391-21timerandwakeup21:09:213
339950,0migration/317363-21latency_hist18:43:563
339950,0migration/317297-21latency_hist19:28:553
339950,0migration/31332-21kthreadcore19:59:133
339950,0migration/312599-21munin-node20:29:183
329950,0watchdog/332029-21kthreadcore22:14:183
329950,0watchdog/314929-21irqcore19:49:113
329950,0watchdog/312122-21diskstats22:04:063
329950,0watchdog/30-21swapper/323:29:143
30049955,0cyclictest0-21swapper/223:12:292
30049955,0cyclictest0-21swapper/222:08:572
30049955,0cyclictest0-21swapper/220:03:562
30049954,0cyclictest14648-21cut23:54:132
30049954,0cyclictest12883-21awk22:04:102
30039955,0cyclictest13382-21diskmemload21:59:051
30039955,0cyclictest0-21swapper/123:37:561
30039955,0cyclictest0-21swapper/122:51:261
30039955,0cyclictest0-21swapper/122:32:021
30039955,0cyclictest0-21swapper/122:18:481
30039950,0cyclictest0-21swapper/123:12:331
30039950,0cyclictest0-21swapper/123:03:061
30039950,0cyclictest0-21swapper/122:21:001
30039950,0cyclictest0-21swapper/121:31:391
30029955,0cyclictest0-21swapper/023:43:050
30029955,0cyclictest0-21swapper/022:19:130
30029954,0cyclictest28908-21awk23:44:110
30029950,0cyclictest0-21swapper/021:53:590
28015050,0irq/29-eth0-tx-6596-21munin-node19:44:210
28015050,0irq/29-eth0-tx-3733-21perl21:59:082
28015050,0irq/29-eth0-tx-28101-21munin-node22:49:062
28015050,0irq/29-eth0-tx-24426-21if_eth021:34:112
28015050,0irq/29-eth0-tx-22983-21ssh20:54:222
28015050,0irq/29-eth0-tx-19133-21munin-node22:44:132
28015050,0irq/29-eth0-tx-14567-21sshd19:51:040
28015050,0irq/29-eth0-tx-0-21swapper/223:05:492
28015050,0irq/29-eth0-tx-0-21swapper/221:50:242
28015050,0irq/29-eth0-tx-0-21swapper/221:13:562
28015050,0irq/29-eth0-tx-0-21swapper/220:42:222
28015050,0irq/29-eth0-tx-0-21swapper/020:33:570
28015050,0irq/29-eth0-tx-0-21swapper/019:58:350
28015050,0irq/29-eth0-tx-0-21swapper/019:14:170
28015050,0irq/29-eth0-tx-0-21swapper/019:09:100
28015050,0irq/29-eth0-tx-0-21swapper/018:54:110
28005050,0irq/28-eth0-rx-6363-21runrttasks23:39:073
28005050,0irq/28-eth0-rx-629-21fschecks_time21:39:123
28005050,0irq/28-eth0-rx-3661-21kthreadcore21:04:123
28005050,0irq/28-eth0-rx-306-21cut23:09:123
28005050,0irq/28-eth0-rx-30249-21tr21:54:163
28005050,0irq/28-eth0-rx-28729-21sh21:34:143
28005050,0irq/28-eth0-rx-26581-21awk20:59:103
28005050,0irq/28-eth0-rx-18079-21df_abs21:49:113
28005050,0irq/28-eth0-rx-14609-21expr21:29:073
28005050,0irq/28-eth0-rx-14580-21cut23:54:133
28005050,0irq/28-eth0-rx-0-21swapper/323:59:173
28005050,0irq/28-eth0-rx-0-21swapper/323:45:003
28005050,0irq/28-eth0-rx-0-21swapper/323:04:153
28005050,0irq/28-eth0-rx-0-21swapper/322:31:303
28005050,0irq/28-eth0-rx-0-21swapper/321:59:573
28005050,0irq/28-eth0-rx-0-21swapper/321:44:083
28005050,0irq/28-eth0-rx-0-21swapper/320:14:143
259950,0migration/27090-21sh21:40:072
259950,0migration/24617-21sh22:15:402
259950,0migration/24324-21sh21:22:142
259950,0migration/230540-21munin-node19:34:162
259950,0migration/229950-21kthreadcore23:44:132
259950,0migration/219573-21fschecks_count23:39:082
259950,0migration/216306-21sh22:39:152
259950,0migration/214537-21gltestperf20:34:082
259950,0migration/212196-21latency20:29:142
249950,0watchdog/227259-21kthreadcore23:59:182
249950,0watchdog/226301-21tune2fs23:24:182
249950,0watchdog/220912-21ssh23:01:142
19150,0ktimersoftd/10-21swapper/121:52:281
179950,0migration/129780-21kthreadcore19:59:121
179950,0migration/128940-21munin-node23:44:121
179950,0migration/126378-21latency_hist21:53:551
179950,0migration/120022-21cpuspeed_turbos21:14:071
179950,0migration/119623-21kthreadcore19:29:131
179950,0migration/115895-21kthreadcore20:34:121
179950,0migration/113487-21cpuspeed_turbos23:54:081
169950,0watchdog/15247-21latency_hist21:23:551
149950,0migration/09230-21munin-node18:34:150
149950,0migration/08769-21kthreadcore20:29:130
149950,0migration/08622-21fschecks_time20:49:100
149950,0migration/08265-21sh21:42:550
149950,0migration/07451-21kthreadcore22:54:170
149950,0migration/0697-21taskset22:33:480
149950,0migration/04412-21irqcore19:19:130
149950,0migration/0416-21kthreadcore22:14:190
149950,0migration/032693-21irqrtprio20:44:130
149950,0migration/028946-21gltestperf19:59:090
149950,0migration/028296-21sh22:11:310
149950,0migration/027950-21kthreadcore21:34:140
149950,0migration/027708-21taskset21:17:070
149950,0migration/027169-21kthreadcore19:34:140
149950,0migration/022984-21kthreadcore22:44:140
149950,0migration/020997-21kthreadcore21:49:170
149950,0migration/018894-21kthreadcore20:54:150
149950,0migration/017388-21latency_hist19:28:550
149950,0migration/016355-21ntp_offset18:39:170
149950,0migration/015256-21kthreadcore22:39:130
8315040,0irq/26-0000:00:1525-21latency_hist23:28:542
339940,0migration/318402-21fschecks_time18:44:093
329940,0watchdog/324553-21cpu20:19:073
329940,0watchdog/30-21swapper/319:00:593
30049943,0cyclictest3346-21snmpd22:28:562
30049943,0cyclictest3346-21snmpd21:00:112
30049943,0cyclictest3346-21snmpd19:57:262
30049943,0cyclictest3346-21snmpd19:43:262
30049943,0cyclictest3346-21snmpd19:26:112
30049943,0cyclictest3070-21irqbalance19:19:022
30049943,0cyclictest3070-21irqbalance19:14:222
30049943,0cyclictest3070-21irqbalance19:09:222
30049943,0cyclictest3070-21irqbalance18:54:222
30049943,0cyclictest3070-21irqbalance18:49:322
30049943,0cyclictest3070-21irqbalance18:48:422
30049943,0cyclictest3070-21irqbalance18:39:022
30049943,0cyclictest3070-21irqbalance18:34:322
30049943,0cyclictest29073-21awk19:59:102
30049943,0cyclictest18423-21awk19:29:112
30049943,0cyclictest14780-21awk19:49:102
30049943,0cyclictest12782-21python19:44:202
30039944,0cyclictest0-21swapper/123:39:031
30039944,0cyclictest0-21swapper/123:14:461
30039943,0cyclictest8792-21awk20:49:111
30039943,0cyclictest3346-21snmpd21:41:561
30039943,0cyclictest3346-21snmpd20:54:101
30039943,0cyclictest3346-21snmpd20:39:111
30039943,0cyclictest3346-21snmpd20:32:111
30039943,0cyclictest3346-21snmpd19:40:111
30039943,0cyclictest3346-21snmpd19:03:261
30039943,0cyclictest3346-21snmpd18:53:261
30039943,0cyclictest3346-21snmpd18:39:561
30039943,0cyclictest3111-21awk21:04:101
30039943,0cyclictest3070-21irqbalance22:37:111
30039943,0cyclictest23030-21awk23:59:151
30039943,0cyclictest21724-21awk22:09:161
30039943,0cyclictest18665-21awk18:44:111
30039943,0cyclictest18269-21awk20:14:111
30039943,0cyclictest14819-21awk19:49:101
30029943,0cyclictest356-21awk19:39:110
30029943,0cyclictest3346-21snmpd21:01:410
30029943,0cyclictest3070-21irqbalance23:55:300
30029943,0cyclictest3070-21irqbalance21:19:210
30029943,0cyclictest3070-21irqbalance20:40:510
30029943,0cyclictest3070-21irqbalance20:22:210
30029943,0cyclictest3070-21irqbalance20:14:010
30029943,0cyclictest17789-21awk23:19:100
30029943,0cyclictest12095-21awk21:09:110
30029943,0cyclictest10050-21awk21:44:100
30029943,0cyclictest0-21swapper/022:07:050
28015040,0irq/29-eth0-tx-8495-21fschecks_time23:14:082
28015040,0irq/29-eth0-tx-4694-21ssh23:49:072
28015040,0irq/29-eth0-tx-0-21swapper/223:18:572
28015040,0irq/29-eth0-tx-0-21swapper/222:54:112
28015040,0irq/29-eth0-tx-0-21swapper/222:25:042
28015040,0irq/29-eth0-tx-0-21swapper/221:54:012
28015040,0irq/29-eth0-tx-0-21swapper/221:44:042
28015040,0irq/29-eth0-tx-0-21swapper/221:29:072
28015040,0irq/29-eth0-tx-0-21swapper/221:04:092
28015040,0irq/29-eth0-tx-0-21swapper/220:49:032
28015040,0irq/29-eth0-tx-0-21swapper/220:44:342
28005040,0irq/28-eth0-rx-0-21swapper/323:49:003
28005040,0irq/28-eth0-rx-0-21swapper/322:43:573
28005040,0irq/28-eth0-rx-0-21swapper/322:34:113
28005040,0irq/28-eth0-rx-0-21swapper/322:10:073
28005040,0irq/28-eth0-rx-0-21swapper/320:41:093
28005040,0irq/28-eth0-rx-0-21swapper/320:34:123
259940,0migration/27845-21irqrtprio22:19:122
259940,0migration/25283-21kthreadcore22:34:152
259940,0migration/217348-21kthreadcore19:04:132
259940,0migration/215703-21kthreadcore20:09:132
249940,0watchdog/212054-21rm21:09:112
249940,0watchdog/20-21swapper/221:24:112
249940,0watchdog/20-21swapper/220:19:072
19140,0ktimersoftd/10-21swapper/122:47:261
179940,0migration/18980-21needreboot19:19:161
179940,0migration/17434-21kthreadcore22:54:171
179940,0migration/130441-21kthreadcore19:14:131
179940,0migration/128553-21kthreadcore19:34:141
179940,0migration/128242-21kthreadcore20:59:121
179940,0migration/126543-21kthreadcore23:04:151
179940,0migration/124034-21kthreadcore19:09:151
179940,0migration/12382-21kthreadcore20:24:141
179940,0migration/123817-21fschecks_count21:34:081
179940,0migration/117221-21kthreadcore22:04:121
179940,0migration/112771-21kthreadcore19:24:181
179940,0migration/112735-21rt-features19:44:181
179940,0migration/111972-21kthreadcore20:09:111
169940,0watchdog/13285-21cat23:29:141
169940,0watchdog/129862-21ssh21:19:111
169940,0watchdog/1173792sleep121:48:461
169940,0watchdog/113382-21diskmemload20:44:111
169940,0watchdog/10-21swapper/123:24:221
169940,0watchdog/10-21swapper/120:19:071
149940,0migration/07725-21kthreadcore23:49:130
149940,0migration/06696-21kthreadcore20:04:130
149940,0migration/04414-21kthreadcore20:24:150
149940,0migration/03192-21kthreadcore22:34:140
149940,0migration/025674-21idleruntime18:49:110
149940,0migration/024589-21kthreadcore23:59:170
149940,0migration/019189-21kthreadcore22:59:130
149940,0migration/016358-21kthreadcore21:29:120
149940,0migration/010648-21kthreadcore21:24:150
329930,0watchdog/30-21swapper/319:14:153
329930,0watchdog/30-21swapper/319:09:153
329930,0watchdog/30-21swapper/318:54:193
30049933,0cyclictest30453-21kworker/2:020:13:562
30039933,0cyclictest14023-21lldpd19:07:071
30039932,0cyclictest5407-21if_err_eth023:49:101
30039932,0cyclictest3932-21awk20:04:111
30039932,0cyclictest17848-21awk23:19:101
30029932,0cyclictest9281-21ssh21:05:260
30029932,0cyclictest4029-21awk21:59:100
30029932,0cyclictest2733-21if_eth023:29:120
30029930,0cyclictest8644-21perl23:14:080
30029930,0cyclictest28711-21perl22:49:080
30029930,0cyclictest23084-21perl23:04:110
30029930,0cyclictest11172-21perl23:34:070
28005030,0irq/28-eth0-rx-0-21swapper/319:58:083
28005030,0irq/28-eth0-rx-0-21swapper/319:44:093
28005030,0irq/28-eth0-rx-0-21swapper/319:23:283
169930,0watchdog/10-21swapper/122:39:061
169930,0watchdog/10-21swapper/121:08:591
169930,0watchdog/10-21swapper/119:54:071
169930,0watchdog/10-21swapper/118:54:151
169930,0watchdog/10-21swapper/118:34:071
149930,0migration/032338-21runrttasks23:24:470
329920,0watchdog/30-21swapper/319:03:593
30029921,0cyclictest6799-21aten2_r1power_e18:59:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional