You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-10-04 - 16:22

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Tue Oct 04, 2022 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2632227473,0sleep20-21swapper/206:38:472
2627025453,0sleep10-21swapper/106:38:101
26608994747,0cyclictest24151-21md0_resync08:49:320
26609994645,0cyclictest24149-21md1_resync08:49:311
1955723828,0sleep00-21swapper/006:36:240
2633922723,0sleep30-21swapper/306:38:563
266109988,0cyclictest0-21swapper/208:41:382
266089974,0cyclictest20330-21awk07:21:100
8315060,0irq/26-0000:00:9000-21port_sshpriv07:31:363
8315060,0irq/26-0000:00:7581-21grep07:11:343
8315060,0irq/26-0000:00:7113-21munin-node07:56:233
8315060,0irq/26-0000:00:30148-21cut09:46:243
8315060,0irq/26-0000:00:27992-21sensors_fan08:06:523
8315060,0irq/26-0000:00:22623-21memory09:21:423
8315060,0irq/26-0000:00:20535-21ssh09:41:023
8315060,0irq/26-0000:00:19084-21cut07:16:333
8315060,0irq/26-0000:00:18054-21munin-node10:11:413
8315060,0irq/26-0000:00:15393-21df10:31:293
8315060,0irq/26-0000:00:15002-21perl08:01:283
8315060,0irq/26-0000:00:14773-21kthreadcore07:36:303
8315060,0irq/26-0000:00:13804-21irqstats09:01:433
8315060,0irq/26-0000:00:11635-21diskstats08:21:353
8315060,0irq/26-0000:00:10213-21grep11:36:383
339960,0migration/37144-21pmu-power10:41:373
339960,0migration/328937-21sh09:08:273
339960,0migration/327838-21munin-node09:41:523
339960,0migration/323936-21sh12:03:213
28005060,0irq/28-eth0-rx-3309-21perl09:31:303
28005060,0irq/28-eth0-rx-32560-21perl11:16:283
28005060,0irq/28-eth0-rx-24941-21perl10:36:253
28005060,0irq/28-eth0-rx-17087-21ssh10:47:433
28005060,0irq/28-eth0-rx-0-21swapper/311:26:343
266109966,0cyclictest0-21swapper/209:10:252
266109960,0cyclictest16338-21ssh11:23:572
266109960,0cyclictest0-21swapper/212:08:152
266099966,0cyclictest0-21swapper/111:38:151
266089960,0cyclictest1917-21md0_raid109:28:280
266089960,0cyclictest0-21swapper/011:53:210
266089960,0cyclictest0-21swapper/010:57:420
266089960,0cyclictest0-21swapper/009:47:320
266089960,0cyclictest0-21swapper/008:20:590
266089960,0cyclictest0-21swapper/008:05:420
259960,0migration/214394-21sh09:53:122
149960,0migration/030145-21sh11:12:560
149960,0migration/027430-21munin-node09:41:450
149960,0migration/027380-21sh10:17:120
8315050,0irq/26-0000:00:7319-21munin-node09:16:303
8315050,0irq/26-0000:00:6363-21runrttasks07:51:253
8315050,0irq/26-0000:00:6209-21kthreadcore11:01:303
8315050,0irq/26-0000:00:4777-21kthreadcore08:56:483
8315050,0irq/26-0000:00:4075-21cpu08:16:323
8315050,0irq/26-0000:00:32722-21ssh09:27:073
8315050,0irq/26-0000:00:30298-21unixbench_singl07:01:383
8315050,0irq/26-0000:00:28801-21wc07:26:283
8315050,0irq/26-0000:00:28560-21perl10:56:243
8315050,0irq/26-0000:00:27288-21sh10:16:583
8315050,0irq/26-0000:00:26160-21df_abs08:31:293
8315050,0irq/26-0000:00:21612-21kthreadcore07:41:323
8315050,0irq/26-0000:00:19266-21diskstats10:51:253
8315050,0irq/26-0000:00:1840-21kthreadcore11:51:323
8315050,0irq/26-0000:00:16634-21cpuspeed_turbos06:56:273
8315050,0irq/26-0000:00:16634-21cpuspeed_turbos06:56:273
8315050,0irq/26-0000:00:10274-21kthreadcore10:26:373
8315050,0irq/26-0000:00:0-21swapper/311:46:093
8315050,0irq/26-0000:00:0-21swapper/309:56:293
8315050,0irq/26-0000:00:0-21swapper/309:56:293
8315050,0irq/26-0000:00:0-21swapper/309:51:343
8315050,0irq/26-0000:00:0-21swapper/309:11:323
8315050,0irq/26-0000:00:0-21swapper/308:11:463
8315050,0irq/26-0000:00:0-21swapper/307:47:263
8315050,0irq/26-0000:00:0-21swapper/307:06:253
339950,0migration/38343-21gltestperf11:56:263
339950,0migration/330147-21fschecks_time10:21:313
339950,0migration/328542-21kthreadcore12:06:363
339950,0migration/327815-21kthreadcore10:01:343
339950,0migration/327677-21latency_hist07:26:103
339950,0migration/326804-21aten2_r1power_e11:31:273
339950,0migration/324622-21netstat08:26:373
339950,0migration/31826-21aten2_r1power_v06:46:303
339950,0migration/31373-21fschecks_time08:36:273
28015050,0irq/29-eth0-tx-2524-21fschecks_count10:06:282
28015050,0irq/29-eth0-tx-23717-21munin-node07:01:290
28015050,0irq/29-eth0-tx-1917-21md0_raid109:21:392
28015050,0irq/29-eth0-tx-1737-21munin-node06:46:460
28015050,0irq/29-eth0-tx-17279-21kthreadcore11:06:302
28015050,0irq/29-eth0-tx-17257-21kthreadcore09:01:442
28015050,0irq/29-eth0-tx-16071-21munin-node09:56:292
28015050,0irq/29-eth0-tx-16071-21munin-node09:56:292
28015050,0irq/29-eth0-tx-0-21swapper/211:47:472
28015050,0irq/29-eth0-tx-0-21swapper/211:41:332
28015050,0irq/29-eth0-tx-0-21swapper/211:35:072
28015050,0irq/29-eth0-tx-0-21swapper/210:56:532
28015050,0irq/29-eth0-tx-0-21swapper/210:11:442
28015050,0irq/29-eth0-tx-0-21swapper/209:14:432
28015050,0irq/29-eth0-tx-0-21swapper/208:51:052
28015050,0irq/29-eth0-tx-0-21swapper/007:53:490
28005050,0irq/28-eth0-rx-4463-21kthreadcore10:06:353
28005050,0irq/28-eth0-rx-31416-21kthreadcore06:41:313
28005050,0irq/28-eth0-rx-19889-21kthreadcore08:46:403
28005050,0irq/28-eth0-rx-13117-21grep11:41:263
28005050,0irq/28-eth0-rx-0-21swapper/311:21:313
28005050,0irq/28-eth0-rx-0-21swapper/311:13:163
28005050,0irq/28-eth0-rx-0-21swapper/311:06:103
28005050,0irq/28-eth0-rx-0-21swapper/308:41:333
27150,0ktimersoftd/20-21swapper/206:57:072
27150,0ktimersoftd/20-21swapper/206:57:072
266109955,0cyclictest1944-21md1_raid108:34:192
266109955,0cyclictest0-21swapper/211:52:292
266109955,0cyclictest0-21swapper/211:12:172
266109955,0cyclictest0-21swapper/210:22:192
266109955,0cyclictest0-21swapper/210:16:292
266109955,0cyclictest0-21swapper/209:41:292
266109955,0cyclictest0-21swapper/208:59:322
266109955,0cyclictest0-21swapper/207:08:232
266109955,0cyclictest0-21swapper/207:01:562
266109950,0cyclictest0-21swapper/210:49:262
266109950,0cyclictest0-21swapper/208:09:432
266099955,0cyclictest0-21swapper/111:57:201
266099955,0cyclictest0-21swapper/111:00:351
266099955,0cyclictest0-21swapper/109:39:421
266099955,0cyclictest0-21swapper/109:26:271
266099955,0cyclictest0-21swapper/108:59:491
266099955,0cyclictest0-21swapper/108:52:051
266099955,0cyclictest0-21swapper/107:23:191
266099955,0cyclictest0-21swapper/106:47:041
266099950,0cyclictest0-21swapper/110:53:381
266099950,0cyclictest0-21swapper/107:13:421
266089955,0cyclictest0-21swapper/011:38:320
266089955,0cyclictest0-21swapper/010:03:280
266089955,0cyclictest0-21swapper/009:16:190
266089955,0cyclictest0-21swapper/009:11:070
266089955,0cyclictest0-21swapper/007:57:490
266089955,0cyclictest0-21swapper/007:29:260
266089950,0cyclictest0-21swapper/010:31:030
259950,0migration/27589-21irqrtprio10:26:362
259950,0migration/226250-21sh08:51:202
259950,0migration/225289-21sh10:01:232
259950,0migration/219885-21kthreadcore07:41:322
259950,0migration/218842-21sh09:37:152
259950,0migration/215430-21kthreadcore08:21:442
19150,0ktimersoftd/10-21swapper/110:32:211
19150,0ktimersoftd/10-21swapper/106:57:071
19150,0ktimersoftd/10-21swapper/106:57:071
169950,0watchdog/13599-21perl07:31:261
149950,0migration/09836-21irqcore11:21:300
149950,0migration/07949-21kthreadcore09:31:390
149950,0migration/07635-21fschecks_time09:51:350
149950,0migration/03880-21if_eth107:31:310
149950,0migration/031709-21latency_hist11:16:090
149950,0migration/030019-21kthreadcore06:41:310
149950,0migration/028970-21sh11:47:510
149950,0migration/028920-21latency_hist08:11:100
149950,0migration/027922-21kthreadcore12:06:360
149950,0migration/026295-21munin-node08:51:250
149950,0migration/025518-21sh11:29:090
149950,0migration/022903-21taskset10:34:000
149950,0migration/022279-21latency06:56:340
149950,0migration/022279-21latency06:56:340
149950,0migration/022237-21fschecks_count08:06:300
149950,0migration/020280-21sh11:43:400
149950,0migration/019597-21taskset09:03:050
149950,0migration/019493-21munin-node08:26:300
149950,0migration/015715-21kthreadcore11:06:290
149950,0migration/015153-21kthreadcore07:36:300
149950,0migration/013119-21latency_hist07:16:110
149950,0migration/012291-21kthreadcore06:51:330
149950,0migration/011736-21taskset09:36:220
149950,0migration/011690-21sh11:02:060
149950,0migration/010964-21munin-node10:46:350
8315040,0irq/26-0000:00:0-21swapper/308:51:253
329940,0watchdog/316148-21sshd06:55:393
28015040,0irq/29-eth0-tx-21974-21ssh10:31:502
28015040,0irq/29-eth0-tx-0-21swapper/212:01:262
28015040,0irq/29-eth0-tx-0-21swapper/211:57:122
28015040,0irq/29-eth0-tx-0-21swapper/211:16:522
28015040,0irq/29-eth0-tx-0-21swapper/211:01:162
28015040,0irq/29-eth0-tx-0-21swapper/210:51:242
28015040,0irq/29-eth0-tx-0-21swapper/209:46:392
266109944,0cyclictest0-21swapper/208:26:352
266109944,0cyclictest0-21swapper/206:55:182
266109943,0cyclictest5016-21awk08:16:382
266109943,0cyclictest3346-21snmpd09:16:272
266109943,0cyclictest3346-21snmpd08:36:122
266109943,0cyclictest3346-21snmpd07:46:272
266109943,0cyclictest3346-21snmpd07:26:272
266109943,0cyclictest3070-21irqbalance11:36:362
266109943,0cyclictest3070-21irqbalance09:31:262
266109943,0cyclictest3070-21irqbalance09:27:162
266109943,0cyclictest3070-21irqbalance08:01:172
266109943,0cyclictest18706-21awk11:26:322
266109943,0cyclictest14260-21awk07:16:302
266109943,0cyclictest14023-21lldpd07:40:522
266109943,0cyclictest14023-21lldpd07:25:232
266109943,0cyclictest14023-21lldpd07:11:222
266099944,0cyclictest0-21swapper/111:51:301
266099944,0cyclictest0-21swapper/109:19:031
266099944,0cyclictest0-21swapper/108:21:301
266099944,0cyclictest0-21swapper/107:27:351
266099944,0cyclictest0-21swapper/107:05:021
266099943,0cyclictest7987-21awk09:51:381
266099943,0cyclictest3346-21snmpd11:32:411
266099943,0cyclictest3346-21snmpd10:45:571
266099943,0cyclictest3346-21snmpd10:21:271
266099943,0cyclictest3346-21snmpd10:03:271
266099943,0cyclictest3346-21snmpd09:47:571
266099943,0cyclictest3346-21snmpd07:17:571
266099943,0cyclictest23959-21awk11:11:311
266099943,0cyclictest17636-21awk09:21:341
266099943,0cyclictest17624-21awk12:01:341
266099943,0cyclictest15366-21awk08:01:311
266099943,0cyclictest14023-21lldpd10:49:011
266099943,0cyclictest14023-21lldpd10:27:281
266099943,0cyclictest14023-21lldpd08:41:531
266099943,0cyclictest14023-21lldpd08:31:231
266099943,0cyclictest14023-21lldpd08:16:221
266099943,0cyclictest14023-21lldpd08:11:221
266099943,0cyclictest14023-21lldpd08:08:221
266099943,0cyclictest14023-21lldpd07:08:521
266099943,0cyclictest14023-21lldpd06:41:231
266099943,0cyclictest13360-21awk09:01:381
266089943,0cyclictest3191-21awk10:06:320
266089943,0cyclictest3070-21irqbalance11:31:360
266089943,0cyclictest3070-21irqbalance09:21:360
266089943,0cyclictest3070-21irqbalance08:36:370
266089943,0cyclictest3070-21irqbalance08:31:270
266089943,0cyclictest3070-21irqbalance07:41:270
266089943,0cyclictest3070-21irqbalance07:21:170
266089943,0cyclictest30569-21awk10:21:350
266089943,0cyclictest17165-21awk09:56:320
266089943,0cyclictest17165-21awk09:56:320
259940,0migration/25682-21latency07:51:412
259940,0migration/230008-21munin-node08:11:392
259940,0migration/229907-21kthreadcore06:41:312
259940,0migration/224969-21fschecks_time10:36:252
259940,0migration/21745-21idleruntime10:41:312
249940,0watchdog/28304-21cut07:56:342
179940,0migration/19988-21taskset10:09:071
179940,0migration/13886-21kthreadcore08:36:331
179940,0migration/128884-21kthreadcore07:46:471
179940,0migration/122486-21kthreadcore11:26:351
179940,0migration/121425-21fschecks_count09:41:291
179940,0migration/118071-21fschecks_time07:41:281
179940,0migration/116147-21kthreadcore11:41:311
179940,0migration/114822-21munin-node11:21:351
169940,0watchdog/18305-21sed07:56:341
169940,0watchdog/13159-21cut09:31:301
169940,0watchdog/130961-21ssh10:36:451
169940,0watchdog/126493-21if_err_eth1.312:06:331
169940,0watchdog/122445-21if_eth109:06:381
169940,0watchdog/119410-21perl08:26:301
169940,0watchdog/118506-21ssh10:12:421
169940,0watchdog/10-21swapper/111:01:251
149940,0migration/09992-21kthreadcore11:56:300
149940,0migration/04106-21kthreadcore09:11:410
149940,0migration/031070-21aten2_r1power_e07:06:240
149940,0migration/016907-21kthreadcore08:21:440
149940,0migration/014500-21kthreadcore10:11:310
28015030,0irq/29-eth0-tx-0-21swapper/008:41:300
266109933,0cyclictest18514-21kworker/2:007:31:242
266109932,0cyclictest3346-21snmpd06:46:272
266099933,0cyclictest14023-21lldpd07:36:231
266099932,0cyclictest17246-21awk09:56:331
266099932,0cyclictest17246-21awk09:56:331
266099932,0cyclictest10127-21awk06:51:311
266099930,0cyclictest14409-21grep11:06:251
266089933,0cyclictest17598-21awk12:01:330
266089932,0cyclictest3346-21snmpd10:54:570
266089932,0cyclictest3213-21cpuspeed_turbos08:56:310
266089932,0cyclictest3070-21irqbalance07:46:170
266089932,0cyclictest25019-21gltestperf10:36:260
266089932,0cyclictest1878-21awk10:41:320
179930,0migration/131677-21latency_hist11:16:091
179930,0migration/126344-21kthreadcore11:46:311
179930,0migration/122099-21kthreadcore10:16:341
169930,0watchdog/10-21swapper/109:11:381
169930,0watchdog/10-21swapper/107:51:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional