You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-01 - 18:32

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2.osadl.org (updated Fri Jul 01, 2022 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1705123828,0sleep30-21swapper/306:50:093
1702223524,0sleep00-21swapper/006:49:460
1691023526,0sleep20-21swapper/206:48:322
1711423324,0sleep10-21swapper/106:50:511
17355991616,0cyclictest10344-21sensors11:38:091
35160,0ktimersoftd/30-21swapper/311:05:293
339960,0migration/37650-21timerwakeupswit10:43:123
339960,0migration/35581-21sh09:49:323
339960,0migration/331373-21kthreadcore10:58:043
339960,0migration/326355-21sh11:31:273
28005060,0irq/28-eth0-rx-14527-21ntp_states09:18:173
28005060,0irq/28-eth0-rx-0-21swapper/311:42:593
28005060,0irq/28-eth0-rx-0-21swapper/311:08:053
173569966,0cyclictest0-21swapper/211:45:542
173559965,0cyclictest2931-21kthreadcore11:53:041
173559960,0cyclictest0-21swapper/110:36:551
173559960,0cyclictest0-21swapper/109:13:411
173549960,0cyclictest0-21swapper/010:08:060
149960,0migration/05000-21kthreadcore10:43:040
149960,0migration/032675-21df_abs11:17:580
149960,0migration/030767-21pmu-power06:58:090
149960,0migration/025751-21sh09:26:370
149960,0migration/019984-21sh09:40:060
149960,0migration/019746-21kernelversion11:28:040
149960,0migration/01223-21sh09:29:070
8315050,0irq/26-0000:00:8372-21sh09:53:022
8315050,0irq/26-0000:00:20911-21sadc12:22:472
339950,0migration/39931-21kthreadcore08:58:063
339950,0migration/37299-21latency_hist09:52:483
339950,0migration/35776-21kthreadcore08:13:053
339950,0migration/34412-21kthreadcore09:13:073
339950,0migration/330743-21taskset09:11:443
339950,0migration/328758-21kthreadcore06:58:063
339950,0migration/327784-21irqcore09:28:033
339950,0migration/32656-21kthreadcore11:53:043
339950,0migration/325267-21kthreadcore09:43:063
339950,0migration/317576-21aten2_r1power_c06:53:013
339950,0migration/315342-21runrttasks10:32:523
339950,0migration/313712-21kthreadcore08:18:053
339950,0migration/313646-21latency_hist07:57:483
339950,0migration/310250-21taskset07:32:483
329950,0watchdog/36846-21fschecks_count08:38:003
329950,0watchdog/34462-21iostat_ios09:33:043
329950,0watchdog/330746-21sh11:14:513
329950,0watchdog/326771-21iostat_ios10:03:043
329950,0watchdog/325728-21irqcore07:43:043
329950,0watchdog/321244-21diskstats08:03:003
329950,0watchdog/31956-21sed08:08:123
329950,0watchdog/316933-21kthreadcore09:03:083
329950,0watchdog/314234-21kthreadcore12:18:033
329950,0watchdog/30-21swapper/307:03:043
329950,0watchdog/30-21swapper/307:03:043
28015050,0irq/29-eth0-tx-2963-21munin-node12:12:592
28015050,0irq/29-eth0-tx-24460-21munin-node10:37:582
28015050,0irq/29-eth0-tx-21510-21munin-node09:43:002
28015050,0irq/29-eth0-tx-21109-21munin-node07:18:150
28015050,0irq/29-eth0-tx-17565-21munin-node06:53:050
28015050,0irq/29-eth0-tx-10241-21munin-node10:48:112
28015050,0irq/29-eth0-tx-0-21swapper/210:35:342
28015050,0irq/29-eth0-tx-0-21swapper/209:38:052
28015050,0irq/29-eth0-tx-0-21swapper/209:18:312
28015050,0irq/29-eth0-tx-0-21swapper/007:55:080
28015050,0irq/29-eth0-tx-0-21swapper/007:08:060
28005050,0irq/28-eth0-rx-984-21sh12:09:113
28005050,0irq/28-eth0-rx-3723-21tune2fs12:12:593
28005050,0irq/28-eth0-rx-32630-21ssh10:41:323
28005050,0irq/28-eth0-rx-28453-21iostat_ios11:33:023
28005050,0irq/28-eth0-rx-20535-21ssh10:17:543
28005050,0irq/28-eth0-rx-19700-21cut09:23:073
28005050,0irq/28-eth0-rx-12411-21ssh09:37:593
28005050,0irq/28-eth0-rx-0-21swapper/311:58:053
28005050,0irq/28-eth0-rx-0-21swapper/311:49:343
28005050,0irq/28-eth0-rx-0-21swapper/311:42:303
28005050,0irq/28-eth0-rx-0-21swapper/311:20:163
28005050,0irq/28-eth0-rx-0-21swapper/310:53:093
28005050,0irq/28-eth0-rx-0-21swapper/310:48:373
28005050,0irq/28-eth0-rx-0-21swapper/310:09:133
28005050,0irq/28-eth0-rx-0-21swapper/309:58:053
28005050,0irq/28-eth0-rx-0-21swapper/308:56:593
259950,0migration/28677-21fschecks_count11:57:592
259950,0migration/2465-21fschecks_count11:17:592
259950,0migration/231538-21aten2_r1power_e09:12:592
259950,0migration/225710-21kthreadcore09:08:082
259950,0migration/224493-21sh11:28:092
259950,0migration/212832-21timerwakeupswit10:28:152
249950,0watchdog/23985-21iostat_ios08:13:042
249950,0watchdog/214993-21cut07:58:042
249950,0watchdog/213325-21sed09:02:482
249950,0watchdog/211639-21cat10:12:592
249950,0watchdog/20-21swapper/207:18:082
249950,0watchdog/20-21swapper/206:58:122
19150,0ktimersoftd/10-21swapper/109:36:421
173569955,0cyclictest31861-21kworker/2:212:11:302
173569955,0cyclictest31861-21kworker/2:211:12:382
173569955,0cyclictest31861-21kworker/2:210:42:512
173569955,0cyclictest31861-21kworker/2:210:21:052
173569955,0cyclictest31861-21kworker/2:210:11:272
173569955,0cyclictest31861-21kworker/2:210:06:352
173569955,0cyclictest31861-21kworker/2:209:48:152
173569955,0cyclictest31861-21kworker/2:209:29:252
173569955,0cyclictest31861-21kworker/2:208:44:332
173569955,0cyclictest31861-21kworker/2:208:28:072
173569955,0cyclictest31861-21kworker/2:208:11:172
173569955,0cyclictest31861-21kworker/2:207:42:492
173569955,0cyclictest31861-21kworker/2:207:41:252
173569955,0cyclictest31861-21kworker/2:207:34:152
173569955,0cyclictest31861-21kworker/2:207:31:552
173569955,0cyclictest31861-21kworker/2:207:22:502
173569955,0cyclictest31861-21kworker/2:207:13:362
173569955,0cyclictest31861-21kworker/2:207:08:582
173569955,0cyclictest31861-21kworker/2:207:03:242
173569955,0cyclictest31861-21kworker/2:207:03:242
173569955,0cyclictest31861-21kworker/2:206:54:082
173569955,0cyclictest27642-21diskmemload11:37:512
173569950,0cyclictest0-21swapper/212:05:462
173569950,0cyclictest0-21swapper/212:05:462
173569950,0cyclictest0-21swapper/211:26:402
173569950,0cyclictest0-21swapper/211:13:442
173569950,0cyclictest0-21swapper/210:01:322
173559955,0cyclictest0-21swapper/110:58:411
173559955,0cyclictest0-21swapper/107:51:421
173549955,0cyclictest27642-21diskmemload09:19:420
173549954,0cyclictest13837-21cut11:43:010
173549950,0cyclictest0-21swapper/010:13:090
173549950,0cyclictest0-21swapper/008:16:380
179950,0migration/132000-21switchtime10:03:121
179950,0migration/131222-21taskset07:44:231
179950,0migration/127197-21kthreadcore11:13:111
179950,0migration/119863-21kthreadcore06:53:071
179950,0migration/117674-21munin-node09:23:001
179950,0migration/110848-21kthreadcore11:23:051
169950,0watchdog/123398-21open_inodes07:38:081
149950,0migration/08676-21kthreadcore10:28:060
149950,0migration/05257-21aten2_r1power_v11:02:580
149950,0migration/04990-21munin-node11:53:050
149950,0migration/032467-21kthreadcore11:33:040
149950,0migration/029830-21kthreadcore07:23:060
149950,0migration/028751-21taskset09:44:280
149950,0migration/024419-21kthreadcore11:48:050
149950,0migration/02414-21kthreadcore07:03:050
149950,0migration/02414-21kthreadcore07:03:050
149950,0migration/023891-21taskset08:25:210
149950,0migration/023616-21sh10:36:370
149950,0migration/021194-21sh09:05:410
149950,0migration/021165-21kthreadcore10:53:050
149950,0migration/019369-21missed_timers08:43:060
149950,0migration/018983-21diskmemload11:08:050
149950,0migration/016637-21switchtime10:48:110
149950,0migration/016496-21sh09:57:570
149950,0migration/015426-21irqrtprio07:13:110
149950,0migration/014181-21kthreadcore11:23:060
149950,0migration/014132-21kthreadcore07:33:060
149950,0migration/011995-21kthreadcore11:58:040
149950,0migration/011421-21irqrtprio08:18:040
8315040,0irq/26-0000:00:0-21swapper/211:32:532
8315040,0irq/26-0000:00:0-21swapper/207:52:482
339940,0migration/326873-21sensors_fan08:48:133
339940,0migration/322042-21kthreadcore07:38:063
339940,0migration/314508-21fschecks_count07:13:063
339940,0migration/311580-21kthreadcore07:53:063
329940,0watchdog/34217-21iostat_ios07:28:043
329940,0watchdog/327236-21kthreadcore08:28:043
329940,0watchdog/30-21swapper/310:12:593
329940,0watchdog/30-21swapper/308:23:083
329940,0watchdog/30-21swapper/307:23:043
329940,0watchdog/30-21swapper/307:18:083
28015040,0irq/29-eth0-tx-3117-21wget07:28:000
28015040,0irq/29-eth0-tx-22225-21ssh11:47:502
28015040,0irq/29-eth0-tx-0-21swapper/211:53:012
28015040,0irq/29-eth0-tx-0-21swapper/211:02:482
28015040,0irq/29-eth0-tx-0-21swapper/210:52:482
28015040,0irq/29-eth0-tx-0-21swapper/209:32:512
28015040,0irq/29-eth0-tx-0-21swapper/209:22:552
28015040,0irq/29-eth0-tx-0-21swapper/208:57:532
28015040,0irq/29-eth0-tx-0-21swapper/208:53:112
28015040,0irq/29-eth0-tx-0-21swapper/008:28:010
28015040,0irq/29-eth0-tx-0-21swapper/008:08:120
28015040,0irq/29-eth0-tx-0-21swapper/007:37:590
28005040,0irq/28-eth0-rx-0-21swapper/312:02:553
28005040,0irq/28-eth0-rx-0-21swapper/312:02:553
28005040,0irq/28-eth0-rx-0-21swapper/311:23:003
28005040,0irq/28-eth0-rx-0-21swapper/310:28:273
28005040,0irq/28-eth0-rx-0-21swapper/310:23:033
28005040,0irq/28-eth0-rx-0-21swapper/308:32:533
259940,0migration/26302-21latency_hist08:37:482
259940,0migration/2551-21kernelversion08:33:042
259940,0migration/230872-21irqcore10:23:062
259940,0migration/224580-21kthreadcore08:03:062
259940,0migration/212551-21kthreadcore08:18:042
249940,0watchdog/228805-21fschecks_count10:57:592
173569943,0cyclictest21685-21awk08:48:042
173559943,0cyclictest7398-21awk08:38:021
173559943,0cyclictest6192-21awk11:03:011
173559943,0cyclictest4083-21awk07:28:031
173559943,0cyclictest3070-21irqbalance10:54:341
173559943,0cyclictest29263-21awk07:23:041
173559943,0cyclictest28967-21awk08:53:041
173559943,0cyclictest27735-21awk09:28:031
173559943,0cyclictest27365-21awk12:08:011
173559943,0cyclictest25792-21awk06:58:031
173559943,0cyclictest23424-21awk11:48:031
173559943,0cyclictest22123-21awk07:18:061
173559943,0cyclictest21574-21awk10:18:021
173559943,0cyclictest18195-21awk08:23:051
173559943,0cyclictest17662-21awk09:58:041
173549944,0cyclictest0-21swapper/012:17:230
173549944,0cyclictest0-21swapper/010:24:110
173549944,0cyclictest0-21swapper/010:18:120
173549943,0cyclictest4359-21awk09:33:030
173549943,0cyclictest3346-21snmpd10:59:190
173549943,0cyclictest32638-21awk09:13:040
173549943,0cyclictest3070-21irqbalance12:08:230
173549943,0cyclictest3070-21irqbalance10:39:540
173549943,0cyclictest3070-21irqbalance08:52:540
173549943,0cyclictest3070-21irqbalance08:51:540
173549943,0cyclictest23461-21awk09:08:050
173549943,0cyclictest18322-21awk12:03:020
173549943,0cyclictest18322-21awk12:03:020
173549943,0cyclictest13732-21munin-node07:58:020
173549941,0cyclictest0-21swapper/008:32:590
173549940,0cyclictest0-21swapper/009:54:230
173549940,0cyclictest0-21swapper/008:59:500
179940,0migration/18916-21kthreadcore07:08:051
179940,0migration/15437-21munin-node08:58:031
179940,0migration/132037-21kthreadcore08:08:041
179940,0migration/1311-21kthreadcore09:48:051
179940,0migration/128760-21kthreadcore10:38:041
179940,0migration/127452-21munin-node09:43:071
179940,0migration/125280-21taskset11:29:051
179940,0migration/12429-21kthreadcore08:33:051
179940,0migration/120628-21taskset07:15:011
179940,0migration/119705-21runrttasks10:16:341
179940,0migration/117730-21kthreadcore07:58:051
179940,0migration/117350-21kthreadcore08:43:051
179940,0migration/117268-21aten2_r1power_p12:02:581
179940,0migration/117268-21aten2_r1power_p12:02:581
179940,0migration/116574-21timerwakeupswit07:33:131
179940,0migration/114313-21kthreadcore09:38:051
179940,0migration/114026-21sh10:30:281
179940,0migration/112679-21kthreadcore09:18:141
179940,0migration/111357-21kthreadcore11:58:031
179940,0migration/110492-21taskset12:14:531
179940,0migration/110219-21cpuspeed_turbos08:17:591
169940,0watchdog/19368-21ssh10:46:311
169940,0watchdog/1775-21irqcore07:03:041
169940,0watchdog/1775-21irqcore07:03:041
169940,0watchdog/17697-21sh11:20:191
169940,0watchdog/13353-21ntp_states10:23:111
169940,0watchdog/130917-21chrt08:28:441
169940,0watchdog/124906-21kthreadcore08:48:081
169940,0watchdog/115075-21ssh09:55:201
169940,0watchdog/112922-21perl12:17:591
169940,0watchdog/11073-21munin-node11:33:071
169940,0watchdog/110621-21diskstats10:47:591
169940,0watchdog/10-21swapper/110:07:511
169940,0watchdog/10-21swapper/109:08:081
169940,0watchdog/10-21swapper/109:02:481
149940,0migration/06115-21sh09:50:380
329930,0watchdog/30-21swapper/308:43:043
329930,0watchdog/30-21swapper/307:48:003
329930,0watchdog/30-21swapper/307:08:003
249930,0watchdog/20-21swapper/208:23:082
173569933,0cyclictest31861-21kworker/2:207:48:072
173559932,0cyclictest3889-21awk08:13:031
173559932,0cyclictest24221-21kthreadcore08:03:061
173559932,0cyclictest15292-21awk11:08:031
173549933,0cyclictest0-21swapper/011:39:230
173549932,0cyclictest7422-21awk08:38:020
173549932,0cyclictest5260-21missed_timers07:48:070
173549932,0cyclictest30294-21kthreadcore07:43:060
173549930,0cyclictest24231-21kthreadcore08:03:060
173549930,0cyclictest1968-21munin-run10:07:480
173549930,0cyclictest12714-21tune2fs12:17:590
169930,0watchdog/10-21swapper/111:42:551
173559921,0cyclictest6351-21latency_hist07:52:481
173549920,0cyclictest23183-21wc11:12:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional