You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-08 - 08:06

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Thu Jun 08, 2023 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
563121612,0sleep30-21swapper/318:06:503
259990,0migration/25775-21taskset18:08:332
28015080,0irq/29-eth0-tx-6111-21avahi-daemon18:06:410
19180,0ktimersoftd/16111-21avahi-daemon18:06:271
61009964,0cyclictest31517-21kworker/2:023:32:182
61009960,0cyclictest0-21swapper/222:51:042
60989960,0cyclictest0-21swapper/022:38:210
60989960,0cyclictest0-21swapper/021:28:490
339960,0migration/39928-21sh20:25:353
339960,0migration/39461-21sh23:25:443
339960,0migration/38742-21sh21:01:303
339960,0migration/36180-21munin-node19:41:363
339960,0migration/331036-21sh18:51:173
339960,0migration/329375-21munin-node22:21:503
339960,0migration/329303-21runrttasks22:58:493
339960,0migration/326760-21sh20:33:253
339960,0migration/325154-21sh22:37:493
339960,0migration/320005-21sh21:23:323
339960,0migration/319581-21sh22:52:103
339960,0migration/314515-21sh21:38:193
28015060,0irq/29-eth0-tx-31081-21munin-node19:36:460
28005060,0irq/28-eth0-rx-6462-21vmstat20:56:593
28005060,0irq/28-eth0-rx-5913-21cut22:46:463
28005060,0irq/28-eth0-rx-27434-21perl21:11:363
28005060,0irq/28-eth0-rx-19546-21port_sshpriv18:16:473
149960,0migration/09008-21sh21:53:480
149960,0migration/019051-21ntp_offset19:46:460
149960,0migration/013499-21idleruntime21:21:360
61009955,0cyclictest6111-21avahi-daemon23:03:512
61009955,0cyclictest6111-21avahi-daemon18:51:452
61009955,0cyclictest15985-21kworker/2:019:16:462
61009955,0cyclictest13456-21kworker/2:121:26:512
61009955,0cyclictest0-21swapper/219:46:562
61009954,0cyclictest31517-21kworker/2:023:18:332
61009953,0cyclictest31517-21kworker/2:021:56:312
61009950,0cyclictest6111-21avahi-daemon22:05:312
61009950,0cyclictest6111-21avahi-daemon21:39:012
61009950,0cyclictest6111-21avahi-daemon21:01:302
61009950,0cyclictest0-21swapper/218:21:192
60999955,0cyclictest6111-21avahi-daemon21:53:541
60999955,0cyclictest0-21swapper/121:32:281
60999950,0cyclictest6111-21avahi-daemon22:53:591
60999950,0cyclictest6111-21avahi-daemon20:15:411
60999950,0cyclictest0-21swapper/123:18:441
60989955,0cyclictest6111-21avahi-daemon20:16:180
60989955,0cyclictest0-21swapper/023:14:460
60989955,0cyclictest0-21swapper/022:46:330
60989955,0cyclictest0-21swapper/021:59:530
60989955,0cyclictest0-21swapper/021:08:290
60989955,0cyclictest0-21swapper/020:25:170
60989954,0cyclictest0-21swapper/018:26:440
60989950,0cyclictest0-21swapper/022:56:480
339950,0migration/39067-21kthreadcore18:56:463
339950,0migration/38890-21kthreadcore23:06:443
339950,0migration/35820-21latency22:26:503
339950,0migration/34380-21needreboot23:01:443
339950,0migration/34250-21kthreadcore20:01:423
339950,0migration/3343-21kthreadcore19:36:383
339950,0migration/330128-21kthreadcore18:26:443
339950,0migration/329197-21kthreadcore23:36:423
339950,0migration/327413-21kthreadcore20:16:443
339950,0migration/322794-21kthreadcore18:21:423
339950,0migration/322647-21munin-node21:26:413
339950,0migration/322250-21kthreadcore19:06:423
339950,0migration/321779-21sh20:49:183
339950,0migration/321105-21kthreadcore22:01:403
339950,0migration/317549-21kthreadcore22:16:433
339950,0migration/31552-21taskset19:57:173
339950,0migration/314881-21kthreadcore19:46:423
339950,0migration/311410-21kthreadcore22:31:423
339950,0migration/310616-21fschecks_time18:36:393
329950,0watchdog/39545-21sed19:21:183
329950,0watchdog/37117-21kthreadcore22:11:453
329950,0watchdog/32944-21diskstats18:31:343
329950,0watchdog/32303-21cat19:16:183
329950,0watchdog/320359-21cut23:31:403
329950,0watchdog/30-21swapper/318:41:463
28015050,0irq/29-eth0-tx-22904-21ssh21:42:092
28015050,0irq/29-eth0-tx-21885-21cpu22:56:382
28015050,0irq/29-eth0-tx-20253-21munin-node19:51:400
28015050,0irq/29-eth0-tx-16844-21ssh23:27:142
28015050,0irq/29-eth0-tx-13987-21munin-node20:46:472
28015050,0irq/29-eth0-tx-127082sleep222:13:342
28015050,0irq/29-eth0-tx-0-21swapper/223:06:532
28015050,0irq/29-eth0-tx-0-21swapper/221:53:192
28015050,0irq/29-eth0-tx-0-21swapper/221:15:402
28015050,0irq/29-eth0-tx-0-21swapper/221:07:352
28015050,0irq/29-eth0-tx-0-21swapper/020:11:320
28015050,0irq/29-eth0-tx-0-21swapper/019:11:460
28015050,0irq/29-eth0-tx-0-21swapper/018:31:380
28005050,0irq/28-eth0-rx-32642-21ssh21:49:133
28005050,0irq/28-eth0-rx-29518-21iostat_ios20:36:443
28005050,0irq/28-eth0-rx-26897-21latency_hist22:41:163
28005050,0irq/28-eth0-rx-22776-21kthreadcore21:06:503
28005050,0irq/28-eth0-rx-20837-21kthreadcore23:11:463
28005050,0irq/28-eth0-rx-0-21swapper/323:29:333
28005050,0irq/28-eth0-rx-0-21swapper/322:07:573
28005050,0irq/28-eth0-rx-0-21swapper/321:56:383
28005050,0irq/28-eth0-rx-0-21swapper/321:52:183
28005050,0irq/28-eth0-rx-0-21swapper/321:19:293
28005050,0irq/28-eth0-rx-0-21swapper/320:51:343
28005050,0irq/28-eth0-rx-0-21swapper/320:43:203
28005050,0irq/28-eth0-rx-0-21swapper/320:11:393
27150,0ktimersoftd/26111-21avahi-daemon22:46:112
259950,0migration/28170-21munin-node20:21:512
259950,0migration/226311-21sh22:40:262
259950,0migration/215391-21latency19:21:462
249950,0watchdog/213618-21awk21:21:372
179950,0migration/19806-21aten2_r1power_c18:36:361
179950,0migration/1611-21sh20:19:271
179950,0migration/128989-21kthreadcore19:56:471
179950,0migration/117705-21sh21:06:241
179950,0migration/116206-21munin-node21:41:361
169950,0watchdog/125952-21grep18:46:461
149950,0migration/09988-21munin-node19:21:380
149950,0migration/09860-21latency_hist23:26:160
149950,0migration/09027-21aten2_r1power_e22:31:360
149950,0migration/07720-21sh22:29:140
149950,0migration/06303-21cpuspeed_turbos18:56:390
149950,0migration/0553-21rt-features21:11:450
149950,0migration/04927-21sh20:41:320
149950,0migration/04315-21munin-node21:31:460
149950,0migration/04309-21sh22:10:010
149950,0migration/03210-21kthreadcore23:21:420
149950,0migration/030735-21aten2_r1power_c23:01:350
149950,0migration/029889-21sh19:31:450
149950,0migration/028221-21fschecks_time19:56:420
149950,0migration/026943-21sh22:03:320
149950,0migration/026092-21kthreadcore18:46:460
149950,0migration/024773-21needreboot23:31:430
149950,0migration/023967-21fschecks_time22:21:360
149950,0migration/023412-21sh21:43:230
149950,0migration/022869-21latency_hist20:51:170
149950,0migration/022039-21kthreadcore20:31:450
149950,0migration/021426-21fschecks_time18:21:380
149950,0migration/019352-21kthreadcore18:41:450
149950,0migration/019090-21kthreadcore19:26:450
149950,0migration/01524-21kthreadcore20:56:480
149950,0migration/015147-21kthreadcore18:16:430
149950,0migration/015047-21munin-node20:46:410
149950,0migration/014720-21fschecks_count22:16:380
149950,0migration/014549-21kthreadcore22:51:430
8315040,0irq/26-0000:00:0-21swapper/319:26:393
61009944,0cyclictest6835-21kworker/2:020:36:212
61009944,0cyclictest6835-21kworker/2:020:31:232
61009944,0cyclictest6835-21kworker/2:020:26:392
61009944,0cyclictest6835-21kworker/2:020:02:062
61009944,0cyclictest6835-21kworker/2:019:57:062
61009944,0cyclictest31517-21kworker/2:022:31:352
61009944,0cyclictest31517-21kworker/2:022:21:312
61009944,0cyclictest31517-21kworker/2:022:06:212
61009944,0cyclictest15985-21kworker/2:019:06:382
61009944,0cyclictest15985-21kworker/2:019:02:022
61009944,0cyclictest15985-21kworker/2:018:56:182
61009944,0cyclictest15985-21kworker/2:018:46:182
61009944,0cyclictest10536-21kworker/2:219:36:182
61009944,0cyclictest10536-21kworker/2:219:26:182
61009943,0cyclictest6032-21awk20:41:392
61009943,0cyclictest3644-21awk18:31:362
61009943,0cyclictest3346-21snmpd23:11:272
61009943,0cyclictest3346-21snmpd21:46:422
61009943,0cyclictest28700-21awk19:11:382
61009943,0cyclictest14023-21lldpd22:16:252
61009943,0cyclictest14023-21lldpd20:08:082
61009943,0cyclictest14023-21lldpd18:41:372
61009943,0cyclictest14023-21lldpd18:36:382
61009943,0cyclictest13893-21awk22:51:412
60999944,0cyclictest6111-21avahi-daemon22:12:211
60999944,0cyclictest6111-21avahi-daemon21:15:071
60999944,0cyclictest6111-21avahi-daemon20:50:511
60999943,0cyclictest3346-21snmpd22:16:571
60999943,0cyclictest3346-21snmpd22:09:271
60999943,0cyclictest3346-21snmpd21:57:571
60999943,0cyclictest3346-21snmpd21:26:271
60999943,0cyclictest3346-21snmpd20:08:121
60999943,0cyclictest3346-21snmpd19:11:271
60999943,0cyclictest3346-21snmpd18:33:261
60999943,0cyclictest3346-21snmpd18:28:561
60999943,0cyclictest3346-21snmpd18:22:411
60999943,0cyclictest3346-21snmpd18:16:561
60999943,0cyclictest3070-21irqbalance23:33:531
60999943,0cyclictest3070-21irqbalance23:01:331
60999943,0cyclictest3070-21irqbalance22:01:331
60999943,0cyclictest3070-21irqbalance21:46:431
60999943,0cyclictest3070-21irqbalance20:32:441
60999943,0cyclictest3070-21irqbalance20:03:341
60999943,0cyclictest3070-21irqbalance19:51:341
60999943,0cyclictest3070-21irqbalance19:01:441
60999943,0cyclictest3070-21irqbalance18:41:241
60999943,0cyclictest22815-21awk22:56:431
60999943,0cyclictest17146-21awk23:11:431
60999943,0cyclictest10100-21awk22:31:401
60999942,0cyclictest19259-21sensors19:46:471
60989944,0cyclictest0-21swapper/020:28:170
60989943,0cyclictest3346-21snmpd19:16:420
60989943,0cyclictest3070-21irqbalance23:06:530
60989943,0cyclictest3070-21irqbalance22:41:330
60989943,0cyclictest3070-21irqbalance22:12:030
60989943,0cyclictest3070-21irqbalance21:46:230
60989943,0cyclictest3070-21irqbalance21:36:230
60989943,0cyclictest3070-21irqbalance21:18:530
60989943,0cyclictest3070-21irqbalance21:01:240
60989943,0cyclictest3070-21irqbalance20:07:540
60989943,0cyclictest3070-21irqbalance19:01:340
60989943,0cyclictest3070-21irqbalance18:51:540
60989943,0cyclictest15614-21cat18:36:450
339940,0migration/39495-21kthreadcore18:11:383
339940,0migration/330607-21diskmemload21:31:353
339940,0migration/330204-21kthreadcore19:11:393
339940,0migration/327377-21kthreadcore19:31:423
329940,0watchdog/3280050irq/28-eth0-rx-19:01:423
329940,0watchdog/30-21swapper/321:41:413
329940,0watchdog/30-21swapper/319:51:373
28015040,0irq/29-eth0-tx-30515-21ssh21:31:292
28015040,0irq/29-eth0-tx-16371-21diskmemload21:16:232
28015040,0irq/29-eth0-tx-0-21swapper/223:36:292
28015040,0irq/29-eth0-tx-0-21swapper/223:21:252
28015040,0irq/29-eth0-tx-0-21swapper/222:26:362
28015040,0irq/29-eth0-tx-0-21swapper/220:56:272
28015040,0irq/29-eth0-tx-0-21swapper/220:51:452
28015040,0irq/29-eth0-tx-0-21swapper/220:16:432
28015040,0irq/29-eth0-tx-0-21swapper/019:41:440
28005040,0irq/28-eth0-rx-0-21swapper/323:16:303
28005040,0irq/28-eth0-rx-0-21swapper/320:26:423
259940,0migration/28399-21kthreadcore19:41:412
259940,0migration/230176-21kthreadcore18:26:442
259940,0migration/220501-21cpuspeed_turbos19:51:362
249940,0watchdog/26111-21avahi-daemon20:11:532
249940,0watchdog/26111-21avahi-daemon19:31:372
179940,0migration/19166-21runrttasks19:18:551
179940,0migration/17099-21fschecks_time18:11:331
179940,0migration/13466-21kthreadcore22:26:481
179940,0migration/129259-21missed_timers22:21:481
179940,0migration/127729-21fschecks_time22:41:371
179940,0migration/12303-21fschecks_count20:21:391
179940,0migration/121812-21irqrtprio19:06:411
179940,0migration/117109-21runrttasks21:05:291
179940,0migration/112306-21timerwakeupswit19:41:491
179940,0migration/11014-21mdstat23:36:441
169940,0watchdog/17646-21tune2fs21:36:371
169940,0watchdog/16363-21runrttasks20:27:011
169940,0watchdog/113617-21ethtool21:21:371
169940,0watchdog/10-21swapper/123:06:441
169940,0watchdog/10-21swapper/121:18:211
169940,0watchdog/10-21swapper/120:56:491
169940,0watchdog/10-21swapper/119:31:411
169940,0watchdog/10-21swapper/119:31:171
149940,0migration/06728-21fschecks_count18:11:330
149940,0migration/03406-21sh23:40:390
149940,0migration/03121-21hwlatdetect20:01:390
149940,0migration/03002-21sh20:37:430
61009933,0cyclictest23348-21kworker/2:218:16:362
61009933,0cyclictest23348-21kworker/2:218:11:182
60999932,0cyclictest3070-21irqbalance23:26:531
60999932,0cyclictest3070-21irqbalance22:46:231
60999930,0cyclictest1972-21gltestperf23:21:381
60989932,0cyclictest32502-21ssh23:19:320
60989932,0cyclictest26620-21sensors19:06:470
28005030,0irq/28-eth0-rx-0-21swapper/318:51:423
169930,0watchdog/16111-21avahi-daemon20:51:411
169930,0watchdog/10-21swapper/122:40:171
169930,0watchdog/10-21swapper/120:36:451
169930,0watchdog/10-21swapper/119:21:181
169930,0watchdog/10-21swapper/118:54:461
8315020,0irq/26-0000:00:0-21swapper/320:06:173
60999921,0cyclictest6363-21runrttasks19:36:211
60999920,0cyclictest0-21swapper/118:56:201
169920,0watchdog/16111-21avahi-daemon20:41:331
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional