You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-10 - 09:11

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Sat Jun 10, 2023 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
83150100,0irq/26-0000:00:0-21swapper/318:06:043
28735090,0irq/27-eth10-21swapper/218:10:412
28015090,0irq/29-eth0-tx-0-21swapper/018:10:130
179990,0migration/124238-21cpuspeed_turbos18:06:171
8315060,0irq/26-0000:00:26741-21cat19:36:293
8315060,0irq/26-0000:00:19434-21latency_hist22:41:023
8315060,0irq/26-0000:00:0-21swapper/321:51:023
4160,0ktimersoftd/018850-21ssh21:10:240
35160,0ktimersoftd/30-21swapper/321:30:593
339960,0migration/39818-21timerwakeupswit21:56:413
339960,0migration/39518-21munin-node18:41:193
339960,0migration/330733-21sh20:56:323
339960,0migration/327843-21sh23:39:023
339960,0migration/327321-21sh21:14:163
339960,0migration/312758-21sh22:53:163
339960,0migration/312758-21sh22:53:163
339960,0migration/310286-21crond22:36:023
329960,0watchdog/37617-21ntpdc21:01:343
329960,0watchdog/315602-21crond20:51:023
329960,0watchdog/311052-21tune2fs22:36:223
309699960,0cyclictest6111-21avahi-daemon21:58:282
309699960,0cyclictest6111-21avahi-daemon21:21:252
28005060,0irq/28-eth0-rx-23324-21sleep22:23:063
28005060,0irq/28-eth0-rx-15389-21sendmail_mailst20:11:313
149960,0migration/07523-21fschecks_count22:16:190
149960,0migration/06011-21sh20:44:340
149960,0migration/028346-21sh21:16:030
149960,0migration/025634-21pmu-power21:11:310
149960,0migration/014110-21irqrtprio19:51:330
4150,0ktimersoftd/00-21swapper/021:49:300
339950,0migration/38625-21runrttasks20:08:543
339950,0migration/38567-21kthreadcore19:01:283
339950,0migration/37754-21kthreadcore19:46:283
339950,0migration/36949-21taskset22:16:163
339950,0migration/34388-21kthreadcore18:36:263
339950,0migration/33659-21kthreadcore19:21:253
339950,0migration/332711-21fschecks_time21:36:243
339950,0migration/329432-21kthreadcore18:31:313
339950,0migration/32730-21cpuspeed_turbos23:26:213
339950,0migration/323222-21runrttasks19:34:593
339950,0migration/321544-21kthreadcore19:56:313
339950,0migration/31563-21kthreadcore19:41:383
339950,0migration/313432-21fschecks_count19:06:203
329950,0watchdog/3826-21interrupts23:06:263
329950,0watchdog/34739-21irqcore20:26:273
329950,0watchdog/318973-21cat18:46:273
329950,0watchdog/317542-21cpuspeed_turbos23:16:223
329950,0watchdog/314319-21ssh22:56:143
329950,0watchdog/30-21swapper/323:31:263
309699955,0cyclictest6111-21avahi-daemon21:15:542
309699955,0cyclictest0-21swapper/222:16:072
309699950,0cyclictest0-21swapper/223:32:072
309699950,0cyclictest0-21swapper/223:07:312
309699950,0cyclictest0-21swapper/220:19:372
309689955,0cyclictest2431-21nvme22:11:301
309689950,0cyclictest6111-21avahi-daemon20:41:231
309689950,0cyclictest0-21swapper/123:17:561
309639955,0cyclictest8822-21diskmemload22:57:270
309639954,0cyclictest0-21swapper/019:01:360
309639950,0cyclictest0-21swapper/022:40:160
28015050,0irq/29-eth0-tx-5346-21munin-node22:51:272
28015050,0irq/29-eth0-tx-5346-21munin-node22:51:272
28015050,0irq/29-eth0-tx-32727-21kthreadcore20:41:282
28015050,0irq/29-eth0-tx-28418-21ssh23:40:292
28015050,0irq/29-eth0-tx-27480-21munin-node18:31:300
28015050,0irq/29-eth0-tx-27023-21munin-node20:01:330
28015050,0irq/29-eth0-tx-24966-21munin-node22:26:282
28015050,0irq/29-eth0-tx-16431-21munin-node19:31:230
28015050,0irq/29-eth0-tx-12839-21df_inode20:31:232
28015050,0irq/29-eth0-tx-11009-21diskstats22:36:212
28015050,0irq/29-eth0-tx-0-21swapper/223:12:552
28015050,0irq/29-eth0-tx-0-21swapper/222:32:202
28015050,0irq/29-eth0-tx-0-21swapper/221:46:102
28015050,0irq/29-eth0-tx-0-21swapper/221:38:582
28015050,0irq/29-eth0-tx-0-21swapper/221:33:052
28015050,0irq/29-eth0-tx-0-21swapper/220:54:512
28015050,0irq/29-eth0-tx-0-21swapper/019:00:230
28005050,0irq/28-eth0-rx-9666-21if_eth123:11:283
28005050,0irq/28-eth0-rx-3346-21snmpd20:51:063
28005050,0irq/28-eth0-rx-26150-21cut21:31:263
28005050,0irq/28-eth0-rx-26052-21kthreadcore23:01:273
28005050,0irq/28-eth0-rx-22676-21kthreadcore20:16:263
28005050,0irq/28-eth0-rx-12300-21ssh22:01:193
28005050,0irq/28-eth0-rx-0-21swapper/323:23:313
28005050,0irq/28-eth0-rx-0-21swapper/322:46:243
28005050,0irq/28-eth0-rx-0-21swapper/322:14:133
28005050,0irq/28-eth0-rx-0-21swapper/322:06:273
28005050,0irq/28-eth0-rx-0-21swapper/321:41:213
28005050,0irq/28-eth0-rx-0-21swapper/321:16:213
28005050,0irq/28-eth0-rx-0-21swapper/321:07:533
28005050,0irq/28-eth0-rx-0-21swapper/320:45:493
28005050,0irq/28-eth0-rx-0-21swapper/320:38:233
28005050,0irq/28-eth0-rx-0-21swapper/320:21:263
259950,0migration/29150-21sh21:03:452
259950,0migration/24788-21sh22:11:562
259950,0migration/229442-21munin-node23:01:312
259950,0migration/222592-21irqcore22:06:272
259950,0migration/219366-21sh22:02:262
179950,0migration/19220-21latency21:56:351
179950,0migration/16872-21latency_hist20:46:031
179950,0migration/127217-21turbostat.cron18:30:591
179950,0migration/127217-21turbostat.cron18:30:591
179950,0migration/12285-21kthreadcore21:36:311
179950,0migration/116609-21missed_timers22:36:301
179950,0migration/116207-21kthreadcore19:51:351
179950,0migration/115576-21timerwakeupswit20:11:341
179950,0migration/113999-21kthreadcore22:01:391
149950,0migration/09041-21fschecks_count23:11:220
149950,0migration/06808-21sh21:37:370
149950,0migration/06222-21cpuspeed_turbos18:16:250
149950,0migration/0580-21hwlatdetect23:06:240
149950,0migration/04923-21kthreadcore21:56:330
149950,0migration/04663-21kthreadcore19:21:250
149950,0migration/04235-21sh22:49:210
149950,0migration/031716-21fschecks_count18:11:190
149950,0migration/030833-21sh23:03:420
149950,0migration/030088-21sh21:32:240
149950,0migration/029310-21latency19:36:300
149950,0migration/027951-21irqcore21:51:270
149950,0migration/026337-21sh20:20:380
149950,0migration/02353-21fschecks_time22:31:200
149950,0migration/022984-21kthreadcore20:36:250
149950,0migration/020367-21sh21:26:330
149950,0migration/019363-21sh23:34:470
149950,0migration/018672-21kthreadcore18:46:270
149950,0migration/015343-21sh21:41:340
149950,0migration/013760-21fschecks_count18:21:390
149950,0migration/013303-21hddtemp_smartct22:01:320
149950,0migration/012040-21kthreadcore18:41:250
149950,0migration/01175-21kthreadcore19:41:380
339940,0migration/331615-21fschecks_time18:56:243
339940,0migration/330097-21kthreadcore20:01:293
339940,0migration/312710-21kthreadcore19:26:273
329940,0watchdog/310151-21kthreadcore18:16:353
329940,0watchdog/30-21swapper/320:31:313
329940,0watchdog/30-21swapper/319:51:353
329940,0watchdog/30-21swapper/318:51:193
329940,0watchdog/30-21swapper/318:11:233
309699944,0cyclictest6111-21avahi-daemon21:07:512
309699943,0cyclictest8118-21awk20:46:272
309699943,0cyclictest7177-21awk18:16:332
309699943,0cyclictest3346-21snmpd20:02:212
309699943,0cyclictest3346-21snmpd19:03:062
309699943,0cyclictest3346-21snmpd18:44:212
309699943,0cyclictest3070-21irqbalance23:16:082
309699943,0cyclictest3070-21irqbalance22:41:182
309699943,0cyclictest3070-21irqbalance20:07:292
309699943,0cyclictest3070-21irqbalance19:33:592
309689944,0cyclictest0-21swapper/123:27:041
309689943,0cyclictest3346-21snmpd19:10:361
309689943,0cyclictest3346-21snmpd18:40:211
309689943,0cyclictest3070-21irqbalance23:36:081
309689943,0cyclictest3070-21irqbalance23:15:381
309689943,0cyclictest3070-21irqbalance22:52:481
309689943,0cyclictest3070-21irqbalance22:52:481
309689943,0cyclictest3070-21irqbalance22:31:281
309689943,0cyclictest3070-21irqbalance20:34:091
309689943,0cyclictest3070-21irqbalance19:46:391
309689943,0cyclictest3070-21irqbalance18:47:401
309689943,0cyclictest24818-21awk18:51:231
309689943,0cyclictest24691-21awk19:36:271
309689943,0cyclictest17134-21awk22:21:271
309689943,0cyclictest14023-21lldpd21:18:431
309689943,0cyclictest14023-21lldpd20:02:111
309689940,0cyclictest0-21swapper/121:51:131
309639944,0cyclictest0-21swapper/021:23:390
309639943,0cyclictest8355-21grep20:06:320
309639943,0cyclictest3274-21awk18:36:240
309639943,0cyclictest3070-21irqbalance23:38:480
309639943,0cyclictest3070-21irqbalance23:26:080
309639943,0cyclictest3070-21irqbalance23:21:080
309639943,0cyclictest3070-21irqbalance23:17:280
309639943,0cyclictest3070-21irqbalance22:55:580
309639943,0cyclictest3070-21irqbalance22:55:580
309639943,0cyclictest3070-21irqbalance20:56:090
309639943,0cyclictest3070-21irqbalance20:46:090
309639943,0cyclictest3070-21irqbalance20:22:390
309639943,0cyclictest3070-21irqbalance19:16:090
309639943,0cyclictest3070-21irqbalance19:08:000
309639943,0cyclictest23743-21munin-node18:51:190
309639940,0cyclictest20352-21munin-node18:26:270
309639940,0cyclictest20352-21munin-node18:26:270
309639940,0cyclictest0-21swapper/019:46:280
28015040,0irq/29-eth0-tx-0-21swapper/223:21:212
28015040,0irq/29-eth0-tx-0-21swapper/222:56:042
28015040,0irq/29-eth0-tx-0-21swapper/222:21:502
28015040,0irq/29-eth0-tx-0-21swapper/221:51:252
28015040,0irq/29-eth0-tx-0-21swapper/221:41:102
28015040,0irq/29-eth0-tx-0-21swapper/221:26:052
28015040,0irq/29-eth0-tx-0-21swapper/220:56:142
28015040,0irq/29-eth0-tx-0-21swapper/220:36:112
28015040,0irq/29-eth0-tx-0-21swapper/220:12:502
28015040,0irq/29-eth0-tx-0-21swapper/020:11:270
28015040,0irq/29-eth0-tx-0-21swapper/019:11:180
28005040,0irq/28-eth0-rx-6111-21avahi-daemon21:21:293
28005040,0irq/28-eth0-rx-18368-21fschecks_time21:46:213
28005040,0irq/28-eth0-rx-0-21swapper/322:26:283
259940,0migration/26176-21fschecks_time19:46:242
259940,0migration/23554-21irqcore18:36:252
259940,0migration/231960-21if_eth018:56:262
259940,0migration/230112-21kthreadcore18:31:312
259940,0migration/221500-21kthreadcore19:56:312
259940,0migration/220186-21runrttasks19:11:192
259940,0migration/211281-21kthreadcore19:26:262
249940,0watchdog/28822-21diskmemload20:21:032
249940,0watchdog/24740-21irqcore20:26:272
249940,0watchdog/229489-21ssh22:46:182
249940,0watchdog/2195962sleep219:55:512
249940,0watchdog/21641-21kthreadcore21:16:262
249940,0watchdog/20-21swapper/223:26:262
249940,0watchdog/20-21swapper/219:41:392
179940,0migration/19714-21cpuspeed_turbos18:41:201
179940,0migration/18298-21irqcore22:16:231
179940,0migration/16119-21gltestperf21:21:221
179940,0migration/14027-21kthreadcore19:21:251
179940,0migration/132716-21fschecks_time23:06:231
179940,0migration/130906-21munin-node22:26:301
179940,0migration/130830-21missed_timers20:56:331
179940,0migration/127370-21ntp_kernel_err22:06:301
179940,0migration/126924-21taskset22:42:551
179940,0migration/121920-21kthreadcore19:56:311
179940,0migration/119421-21switchtime18:21:521
179940,0migration/119333-21runrttasks23:34:411
179940,0migration/115353-21if_eth1.221:26:281
169940,0watchdog/126793-21kthreadcore20:36:271
169940,0watchdog/123833-21cat20:16:271
169940,0watchdog/12204-21missed_timers22:46:261
169940,0watchdog/122009-21ntp_states20:51:341
169940,0watchdog/119384-21ssh21:11:061
169940,0watchdog/11497-21cut18:11:231
169940,0watchdog/10-21swapper/123:21:141
169940,0watchdog/10-21swapper/121:31:261
169940,0watchdog/10-21swapper/120:21:031
169940,0watchdog/10-21swapper/119:26:271
169940,0watchdog/10-21swapper/119:16:191
149940,0migration/022011-21fschecks_time22:06:230
149940,0migration/020191-21kthreadcore20:51:310
329930,0watchdog/36111-21avahi-daemon19:16:193
329930,0watchdog/30-21swapper/319:13:513
329930,0watchdog/30-21swapper/318:26:353
329930,0watchdog/30-21swapper/318:26:353
329930,0watchdog/30-21swapper/318:21:433
309699933,0cyclictest6678-21kworker/2:119:21:112
309699933,0cyclictest6678-21kworker/2:119:16:112
309699933,0cyclictest6678-21kworker/2:119:06:232
309699933,0cyclictest30155-21kworker/2:018:51:052
309699933,0cyclictest30155-21kworker/2:018:46:112
309699933,0cyclictest30155-21kworker/2:018:26:292
309699933,0cyclictest30155-21kworker/2:018:26:292
309699933,0cyclictest30155-21kworker/2:018:21:052
309699932,0cyclictest3070-21irqbalance18:11:102
309689933,0cyclictest6111-21avahi-daemon20:08:541
309689932,0cyclictest9764-21sensors20:26:331
309689932,0cyclictest7031-21awk18:16:321
309689932,0cyclictest3070-21irqbalance19:31:191
309689932,0cyclictest2548-21awk21:01:271
309689932,0cyclictest21143-21awk19:11:231
309689932,0cyclictest14023-21lldpd18:59:401
309639932,0cyclictest6005-21kthreadcore20:26:280
309639932,0cyclictest3346-21snmpd21:01:060
309639932,0cyclictest3070-21irqbalance19:26:090
309639932,0cyclictest30220-21ssh22:11:080
309639932,0cyclictest20746-21awk22:41:260
309639932,0cyclictest17058-21awk22:21:270
309639932,0cyclictest13576-21awk20:31:270
28015030,0irq/29-eth0-tx-0-21swapper/019:56:230
249930,0watchdog/20-21swapper/219:36:032
169930,0watchdog/16111-21avahi-daemon22:59:541
169930,0watchdog/16111-21avahi-daemon21:41:221
169930,0watchdog/16111-21avahi-daemon21:06:181
169930,0watchdog/10-21swapper/119:04:271
309689920,0cyclictest0-21swapper/118:31:041
309639920,0cyclictest0-21swapper/022:26:020
179920,0migration/123229-21sh23:01:031
169920,0watchdog/16111-21avahi-daemon21:46:141
169920,0watchdog/16111-21avahi-daemon19:41:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional