You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-08-05 - 06:47

Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack1slot2.osadl.org (updated Thu Aug 05, 2021 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291450560,0irq/31-eth10-21swapper/119:19:151
1678823828,0sleep30-21swapper/319:19:253
1680523524,0sleep00-21swapper/019:19:380
1679223525,0sleep20-21swapper/219:19:282
171979988,0cyclictest2363-21nvme00:03:392
171949970,0cyclictest7296-21ssh21:52:030
4160,0ktimersoftd/00-21swapper/023:43:490
28425060,0irq/33-eth0-tx-22857-21munin-node20:38:350
172079966,0cyclictest0-21swapper/319:55:313
172079960,0cyclictest0-21swapper/323:13:423
172079960,0cyclictest0-21swapper/319:58:403
171979966,0cyclictest2363-21nvme23:31:012
171979966,0cyclictest2363-21nvme23:08:202
171979966,0cyclictest2363-21nvme22:37:292
171979966,0cyclictest2363-21nvme22:29:342
171979966,0cyclictest2363-21nvme21:37:502
171979966,0cyclictest2363-21nvme00:47:342
171979966,0cyclictest2363-21nvme00:23:382
171979966,0cyclictest2363-21nvme00:19:392
171959966,0cyclictest5395-21snmpd23:59:281
171959960,0cyclictest13003-21diskmemload21:28:461
171959960,0cyclictest0-21swapper/100:10:221
171949960,0cyclictest0-21swapper/000:06:030
8305050,0irq/26-0000:00:20815-21if_err_eth1.219:28:392
8305050,0irq/26-0000:00:0-21swapper/220:23:212
35150,0ktimersoftd/30-21swapper/323:43:493
35150,0ktimersoftd/30-21swapper/322:42:233
339950,0migration/37150-21rt-features21:08:443
339950,0migration/36366-21switchtime22:58:493
339950,0migration/327657-21sh23:22:573
339950,0migration/323572-21needreboot21:33:403
339950,0migration/323171-21taskset00:27:113
339950,0migration/321319-21sh23:49:213
339950,0migration/32039-21smart_sda20:58:443
339950,0migration/31824-21sh22:19:453
339950,0migration/314970-21idleruntime23:08:433
339950,0migration/312259-21sh23:05:333
339950,0migration/310075-21latency_hist00:13:223
329950,0watchdog/36794-21load00:08:443
329950,0watchdog/312588-21ssh22:32:033
329950,0watchdog/310931-21sh21:55:033
28425050,0irq/33-eth0-tx-8991-21munin-node00:48:382
28425050,0irq/33-eth0-tx-843-21processes22:18:432
28425050,0irq/33-eth0-tx-8179-21munin-node21:13:400
28425050,0irq/33-eth0-tx-4800-21ssh21:48:372
28425050,0irq/33-eth0-tx-28782-21switchtime22:13:492
28425050,0irq/33-eth0-tx-23409-21httpd21:53:552
28425050,0irq/33-eth0-tx-22682-21cut22:08:372
28425050,0irq/33-eth0-tx-0-21swapper/223:41:032
28425050,0irq/33-eth0-tx-0-21swapper/223:13:392
28425050,0irq/33-eth0-tx-0-21swapper/200:14:072
28425050,0irq/33-eth0-tx-0-21swapper/021:23:320
28425050,0irq/33-eth0-tx-0-21swapper/020:08:390
28415050,0irq/32-eth0-rx-29243-21switchtime22:48:473
28415050,0irq/32-eth0-rx-28484-21munin-node22:13:453
28415050,0irq/32-eth0-rx-16140-21munin-node00:18:433
28415050,0irq/32-eth0-rx-15989-21ssh21:25:353
28415050,0irq/32-eth0-rx-1428-21fschecks_time00:03:343
28415050,0irq/32-eth0-rx-12154-21vmstat20:14:023
28415050,0irq/32-eth0-rx-0-21swapper/322:53:543
27150,0ktimersoftd/20-21swapper/222:42:232
259950,0migration/26710-21sh22:58:552
259950,0migration/231296-21sh21:43:372
259950,0migration/228979-21munin-node22:48:432
259950,0migration/221382-21munin-node20:33:382
259950,0migration/220209-21latency_hist20:33:212
259950,0migration/216091-21sh21:25:502
19150,0ktimersoftd/10-21swapper/123:43:491
172079955,0cyclictest0-21swapper/323:24:163
172079955,0cyclictest0-21swapper/322:08:263
172079950,0cyclictest0-21swapper/323:41:463
172079950,0cyclictest0-21swapper/321:38:573
172079950,0cyclictest0-21swapper/321:28:323
172079950,0cyclictest0-21swapper/300:52:333
171979955,0cyclictest6873-21kworker/2:020:39:572
171979955,0cyclictest6873-21kworker/2:020:15:352
171979955,0cyclictest2363-21nvme23:54:222
171979955,0cyclictest2363-21nvme23:35:382
171979955,0cyclictest2363-21nvme22:57:562
171979955,0cyclictest2363-21nvme22:08:122
171979955,0cyclictest2363-21nvme22:01:442
171979955,0cyclictest2363-21nvme21:39:572
171979955,0cyclictest2363-21nvme20:55:132
171979955,0cyclictest2363-21nvme19:38:412
171979955,0cyclictest2363-21nvme19:23:362
171979955,0cyclictest2363-21nvme00:39:062
171979955,0cyclictest2363-21nvme00:28:292
171979950,0cyclictest0-21swapper/200:37:012
171959955,0cyclictest13003-21diskmemload23:30:571
171959955,0cyclictest0-21swapper/122:28:031
171959955,0cyclictest0-21swapper/122:05:111
171959955,0cyclictest0-21swapper/100:45:311
171959954,0cyclictest5395-21snmpd22:18:151
171959954,0cyclictest22200-21sadc19:33:221
171959950,0cyclictest0-21swapper/122:44:171
171959950,0cyclictest0-21swapper/122:40:491
171959950,0cyclictest0-21swapper/122:12:071
171959950,0cyclictest0-21swapper/121:52:341
171959950,0cyclictest0-21swapper/121:43:161
171949955,0cyclictest13003-21diskmemload00:17:150
171949953,0cyclictest6477-21kworker/0:222:28:060
171949950,0cyclictest13003-21diskmemload23:29:470
171949950,0cyclictest0-21swapper/023:10:430
171949950,0cyclictest0-21swapper/022:50:320
171949950,0cyclictest0-21swapper/019:23:220
179950,0migration/1369-21munin-node21:43:461
179950,0migration/129833-21taskset20:49:311
179950,0migration/119762-21sh23:13:371
179950,0migration/117856-21sh00:20:381
179950,0migration/11390-21ntp_offset19:53:461
179950,0migration/110687-21latency_hist21:18:211
179950,0migration/110398-21if_eth1.223:03:391
149950,0migration/08155-21latency_hist21:53:210
149950,0migration/06763-21munin-node20:03:440
149950,0migration/06171-21fschecks_time21:08:360
149950,0migration/029933-21sh21:41:200
149950,0migration/027854-21latency_hist20:48:210
149950,0migration/025809-21sh22:11:360
149950,0migration/025241-21mdstat00:28:400
149950,0migration/023734-21fschecks_time22:43:370
149950,0migration/023386-21latency_hist23:18:220
149950,0migration/019277-21sendmail_mailtr22:03:440
149950,0migration/019091-21gltestperf22:38:400
149950,0migration/014092-21latency_hist23:43:220
149950,0migration/012599-21taskset21:18:560
149950,0migration/011877-21taskset23:04:500
149950,0migration/010638-21switchtime00:48:500
339940,0migration/332126-21timerwakeupswit20:53:533
339940,0migration/331794-21munin-node21:43:393
339940,0migration/329804-21latency_hist19:48:213
339940,0migration/319059-21munin-node20:28:443
339940,0migration/317566-21ksmtuned20:28:033
339940,0migration/311888-21missed_timers21:18:383
329940,0watchdog/34511-21if_err_eth121:48:353
329940,0watchdog/30-21swapper/323:53:363
329940,0watchdog/30-21swapper/322:23:473
329940,0watchdog/30-21swapper/320:33:513
28425040,0irq/33-eth0-tx-22971-21ssh22:43:302
28425040,0irq/33-eth0-tx-14154-21ssh23:08:392
28425040,0irq/33-eth0-tx-13003-21diskmemload21:29:172
28425040,0irq/33-eth0-tx-0-21swapper/223:58:252
28425040,0irq/33-eth0-tx-0-21swapper/223:43:472
28425040,0irq/33-eth0-tx-0-21swapper/223:23:412
28425040,0irq/33-eth0-tx-0-21swapper/223:18:512
28425040,0irq/33-eth0-tx-0-21swapper/222:23:432
28425040,0irq/33-eth0-tx-0-21swapper/019:53:450
28415040,0irq/32-eth0-rx-13567-21ssh22:33:293
28415040,0irq/32-eth0-rx-0-21swapper/323:28:423
28415040,0irq/32-eth0-rx-0-21swapper/321:58:273
28415040,0irq/32-eth0-rx-0-21swapper/300:43:403
28415040,0irq/32-eth0-rx-0-21swapper/300:33:293
28415040,0irq/32-eth0-rx-0-21swapper/300:28:253
28415040,0irq/32-eth0-rx-0-21swapper/300:00:263
259940,0migration/28749-21if_eth1.220:08:432
259940,0migration/27660-21sh00:09:022
259940,0migration/22955-21fschecks_count19:58:372
259940,0migration/219663-21fschecks_time23:48:352
172079944,0cyclictest0-21swapper/319:42:173
172079943,0cyclictest9171-21awk21:13:363
172079943,0cyclictest5498-21grep23:33:333
172079943,0cyclictest5395-21snmpd22:03:303
172079943,0cyclictest5395-21snmpd20:08:433
172079943,0cyclictest4102-21awk21:03:413
172079943,0cyclictest3111-21irqbalance20:44:013
172079943,0cyclictest25146-21sadc20:43:213
172079943,0cyclictest23934-21awk22:43:393
172079943,0cyclictest23309-21awk19:33:373
172079943,0cyclictest18423-21awk19:23:373
172079941,0cyclictest3560-21mysqld20:18:443
172079940,0cyclictest0-21swapper/300:38:373
171979944,0cyclictest2363-21nvme20:44:502
171979944,0cyclictest2363-21nvme19:50:462
171979944,0cyclictest2363-21nvme19:47:132
171979944,0cyclictest2363-21nvme19:33:522
171979943,0cyclictest6637-21awk21:08:392
171979943,0cyclictest5395-21snmpd21:20:142
171979943,0cyclictest5395-21snmpd20:59:592
171979943,0cyclictest5395-21snmpd20:18:592
171979943,0cyclictest5395-21snmpd19:57:582
171979943,0cyclictest4078-21awk21:03:412
171979943,0cyclictest28778-21awk20:48:352
171959944,0cyclictest0-21swapper/121:57:461
171959943,0cyclictest9985-21awk00:48:421
171959943,0cyclictest5395-21snmpd23:41:461
171959943,0cyclictest5395-21snmpd23:25:461
171959943,0cyclictest5395-21snmpd22:37:151
171959943,0cyclictest5395-21snmpd21:59:001
171959943,0cyclictest5395-21snmpd21:34:151
171959943,0cyclictest5395-21snmpd21:24:141
171959943,0cyclictest5395-21snmpd21:15:441
171959943,0cyclictest5395-21snmpd21:09:141
171959943,0cyclictest5395-21snmpd21:06:441
171959943,0cyclictest5395-21snmpd20:44:441
171959943,0cyclictest5395-21snmpd20:34:441
171959943,0cyclictest5395-21snmpd20:24:441
171959943,0cyclictest5395-21snmpd20:21:441
171959943,0cyclictest5395-21snmpd20:08:581
171959943,0cyclictest5395-21snmpd20:05:431
171959943,0cyclictest5395-21snmpd00:39:461
171959943,0cyclictest5395-21snmpd00:06:161
171959943,0cyclictest32259-21sadc19:53:211
171959943,0cyclictest3111-21irqbalance23:00:111
171959943,0cyclictest3111-21irqbalance22:28:411
171959943,0cyclictest20575-21awk00:23:371
171949944,0cyclictest0-21swapper/023:24:450
171949943,0cyclictest6477-21kworker/0:221:29:260
171949943,0cyclictest5395-21snmpd23:57:460
171949943,0cyclictest5395-21snmpd23:16:040
171949943,0cyclictest5395-21snmpd22:28:300
171949943,0cyclictest5395-21snmpd22:14:300
171949943,0cyclictest5395-21snmpd22:01:150
171949943,0cyclictest5395-21snmpd21:46:300
171949943,0cyclictest5395-21snmpd21:35:300
171949943,0cyclictest5395-21snmpd21:05:290
171949943,0cyclictest5395-21snmpd20:26:590
171949943,0cyclictest5395-21snmpd20:01:430
171949943,0cyclictest5395-21snmpd19:43:580
171949943,0cyclictest5395-21snmpd00:25:160
171949943,0cyclictest5395-21snmpd00:12:310
171949943,0cyclictest453-21munin-node20:58:360
171949943,0cyclictest32714-21awk22:18:390
171949943,0cyclictest3111-21irqbalance23:01:310
171949943,0cyclictest2584-21kworker/0:223:59:040
171949943,0cyclictest21264-21awk20:33:370
171949943,0cyclictest20844-21awk19:28:390
171949943,0cyclictest15904-21awk00:18:410
171949943,0cyclictest11301-21awk20:13:500
171949942,0cyclictest5395-21snmpd20:18:280
171949940,0cyclictest22976-21perl19:33:360
171949940,0cyclictest0-21swapper/020:53:410
171949940,0cyclictest0-21swapper/000:43:250
179940,0migration/131690-21sh22:52:531
179940,0migration/130559-21cpuspeed_turbos20:53:411
179940,0migration/12720-21taskset21:02:191
179940,0migration/123991-21fschecks_count23:18:341
179940,0migration/123576-21fschecks_time20:38:361
179940,0migration/121398-21sh23:49:301
169940,0watchdog/14237-21python19:58:471
169940,0watchdog/130732-21sh00:34:401
169940,0watchdog/128818-21pluginstate19:43:431
169940,0watchdog/126351-21ssh00:29:321
169940,0watchdog/113003-21diskmemload23:12:481
169940,0watchdog/11177-21munin-node22:18:471
149940,0migration/018016-21latency_hist22:38:220
28415030,0irq/32-eth0-rx-0-21swapper/320:05:283
172079932,0cyclictest5395-21snmpd19:48:583
172079930,0cyclictest3561-21mysqld20:48:363
172079930,0cyclictest0-21swapper/319:28:393
171979932,0cyclictest6134-21awk20:03:382
171979932,0cyclictest5395-21snmpd21:15:142
171959932,0cyclictest5395-21snmpd20:31:271
171959932,0cyclictest5395-21snmpd19:38:281
171959932,0cyclictest5395-21snmpd19:26:431
171959930,0cyclictest433-21perl22:53:441
171959930,0cyclictest0-21swapper/123:53:361
171949933,0cyclictest29211-21kworker/0:122:53:340
171949933,0cyclictest2584-21kworker/0:223:48:300
171949933,0cyclictest2584-21kworker/0:223:33:380
171949933,0cyclictest16010-21kworker/0:020:43:210
171949933,0cyclictest16010-21kworker/0:019:48:390
171949933,0cyclictest16010-21kworker/0:019:38:210
171949933,0cyclictest15791-21kworker/0:200:38:260
171949933,0cyclictest15791-21kworker/0:200:33:260
171949932,0cyclictest5395-21snmpd20:28:290
169930,0watchdog/113003-21diskmemload00:13:481
169930,0watchdog/10-21swapper/123:33:361
169930,0watchdog/10-21swapper/120:13:391
169930,0watchdog/10-21swapper/119:33:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional