You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-04-12 - 09:01

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack1slot0.osadl.org (updated Mon Apr 12, 2021 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
176042840,0sleep00-21swapper/021:58:240
243462630,0sleep10-21swapper/121:22:241
90382610,0sleep20-21swapper/200:27:492
312712610,0sleep30-21swapper/300:18:373
59402600,0sleep20-21swapper/221:26:372
59402600,0sleep20-21swapper/221:26:372
47702590,0sleep34768-21df_inode22:00:023
44802590,0sleep30-21swapper/323:06:463
301152590,0sleep30-21swapper/300:26:503
221542590,0sleep20-21swapper/223:30:282
950325734,7sleep30-21swapper/319:07:223
57032570,0sleep35699-21sshd00:21:563
51322560,0sleep20-21swapper/200:30:132
218512560,0sleep30-21swapper/323:13:503
242462540,0sleep1811-21dbus-daemon00:12:251
103672540,0sleep20-21swapper/223:23:592
103672540,0sleep20-21swapper/223:23:582
889525136,9sleep20-21swapper/219:05:122
41852510,0sleep30-21swapper/322:44:263
152542510,0sleep2321ktimersoftd/221:55:222
106312510,0sleep1981799cyclictest23:43:251
948624735,7sleep10-21swapper/119:07:071
942324735,7sleep00-21swapper/019:06:260
67432460,2sleep2982399cyclictest00:10:542
261272450,2sleep2982399cyclictest23:39:132
99392420,0sleep30-21swapper/322:19:543
286802200,0sleep30-21swapper/300:01:423
126442190,0sleep10-21swapper/123:51:541
69382180,0sleep10-21swapper/122:53:021
63242180,1sleep16317-21sshd23:15:131
63242180,1sleep16317-21sshd23:15:121
5052180,0sleep1499-21bash22:08:031
19602180,1sleep0111rcuc/021:14:330
138662180,1sleep313859-21lspci21:41:243
138662180,1sleep313859-21lspci21:41:233
137212180,0sleep013715-21lspci23:40:510
299312170,1sleep029934-21hwlatdetect23:20:050
299312170,1sleep029934-21hwlatdetect23:20:040
983099165,3cyclictest41-21ksoftirqd/321:13:183
981799160,1cyclictest15826-21sshd23:57:481
83402160,0sleep00-21swapper/022:28:040
267592160,0sleep00-21swapper/021:31:240
121242160,1sleep212125-21lspci00:08:382
983099156,4cyclictest41-21ksoftirqd/300:38:083
983099155,1cyclictest41-21ksoftirqd/323:40:563
983099155,1cyclictest41-21ksoftirqd/322:59:393
327462150,0sleep132738-21sshd21:48:401
256852150,0sleep00-21swapper/000:23:430
211472150,0sleep10-21swapper/122:12:351
121642150,0sleep20-21swapper/221:41:152
121642150,0sleep20-21swapper/221:41:142
983099145,3cyclictest41-21ksoftirqd/321:39:553
983099144,3cyclictest41-21ksoftirqd/321:59:013
983099142,3cyclictest41-21ksoftirqd/300:10:003
134572140,0sleep10-21swapper/120:40:061
983099135,4cyclictest41-21ksoftirqd/323:59:343
983099135,4cyclictest41-21ksoftirqd/322:37:003
983099134,2cyclictest41-21ksoftirqd/321:52:433
983099133,2cyclictest41-21ksoftirqd/321:29:293
983099133,2cyclictest41-21ksoftirqd/321:29:283
983099132,3cyclictest41-21ksoftirqd/323:50:153
983099129,2cyclictest41-21ksoftirqd/321:49:323
983099128,1cyclictest41-21ksoftirqd/322:31:083
983099123,4cyclictest41-21ksoftirqd/323:24:553
983099123,4cyclictest41-21ksoftirqd/323:24:543
983099123,3cyclictest41-21ksoftirqd/322:50:403
983099123,3cyclictest41-21ksoftirqd/321:24:193
983099123,1cyclictest41-21ksoftirqd/323:27:143
983099123,1cyclictest41-21ksoftirqd/322:20:083
983099123,1cyclictest41-21ksoftirqd/322:09:353
983099122,3cyclictest41-21ksoftirqd/322:48:033
983099122,3cyclictest41-21ksoftirqd/300:32:353
983099122,2cyclictest41-21ksoftirqd/323:32:363
983099120,3cyclictest41-21ksoftirqd/321:33:353
983099120,1cyclictest6131-21lspci22:14:043
983099120,1cyclictest409-21grep23:39:483
9823991210,1cyclictest12765-21id22:59:082
982399120,1cyclictest23855-21sshd22:01:392
982399120,11cyclictest810-21systemd-logind21:13:432
982399120,11cyclictest21911-21id22:12:382
981799125,3cyclictest25-21ksoftirqd/123:11:371
981799121,3cyclictest15-21kswork22:28:021
9817991210,1cyclictest15426-21id23:02:061
9817991210,1cyclictest15426-21id23:02:061
981799120,1cyclictest20857-21bash23:38:451
980599128,1cyclictest1345-21nfsd22:46:530
9805991211,0cyclictest20251-21lspci22:20:430
9805991211,0cyclictest0-21swapper/000:28:570
980599120,1cyclictest20143-21sshd00:06:310
980599120,11cyclictest811-21dbus-daemon23:57:580
980599120,11cyclictest11097-21bash00:14:060
250792120,2sleep0980599cyclictest23:16:520
250792120,2sleep0980599cyclictest23:16:510
983099113,1cyclictest41-21ksoftirqd/321:17:493
983099112,3cyclictest41-21ksoftirqd/323:18:333
983099112,3cyclictest41-21ksoftirqd/323:18:333
983099112,3cyclictest41-21ksoftirqd/323:01:403
983099112,3cyclictest41-21ksoftirqd/323:01:393
983099112,1cyclictest41-21ksoftirqd/300:11:073
983099110,1cyclictest41-21ksoftirqd/323:46:473
982399110,10cyclictest11004-21bash23:15:372
982399110,10cyclictest11004-21bash23:15:362
982399110,10cyclictest0-21swapper/223:27:332
982399110,10cyclictest0-21swapper/222:25:532
982399110,10cyclictest0-21swapper/221:21:092
981799119,1cyclictest17182-21rm21:15:511
981799112,1cyclictest131rcu_sched00:24:241
981799111,3cyclictest26652-21kworker/1:100:09:351
981799110,1cyclictest17607-21taskset21:50:051
980599119,1cyclictest931-21gdbus22:14:480
980599119,1cyclictest318-21bash21:17:190
9805991110,0cyclictest1447-21snmpd23:13:540
9805991110,0cyclictest0-21swapper/022:09:370
980599110,1cyclictest14853-21sshd21:44:200
980599110,1cyclictest14853-21sshd21:44:190
980599110,10cyclictest0-21swapper/023:49:150
980599110,10cyclictest0-21swapper/023:30:210
980599110,10cyclictest0-21swapper/023:05:090
980599110,10cyclictest0-21swapper/021:38:270
980599110,0cyclictest0-21swapper/022:51:560
983099109,0cyclictest0-21swapper/320:34:563
983099109,0cyclictest0-21swapper/319:35:023
983099101,3cyclictest6304-21sshd22:25:063
983099101,3cyclictest41-21ksoftirqd/319:14:553
982399109,0cyclictest0-21swapper/223:56:192
982399109,0cyclictest0-21swapper/223:47:152
982399109,0cyclictest0-21swapper/222:49:082
982399109,0cyclictest0-21swapper/222:39:172
982399100,0cyclictest0-21swapper/200:20:022
982399100,0cyclictest0-21swapper/200:04:512
981799105,1cyclictest25-21ksoftirqd/123:45:281
981799104,2cyclictest25-21ksoftirqd/100:26:081
981799104,1cyclictest25-21ksoftirqd/121:26:491
981799104,1cyclictest25-21ksoftirqd/121:26:481
981799104,1cyclictest25-21ksoftirqd/100:16:331
981799103,2cyclictest25-21ksoftirqd/123:21:261
981799103,2cyclictest25-21ksoftirqd/123:21:251
981799102,1cyclictest789-21polkitd23:26:501
981799100,5cyclictest26652-21kworker/1:123:33:351
980599109,0cyclictest0-21swapper/022:19:370
980599109,0cyclictest0-21swapper/000:36:130
980599101,1cyclictest131rcu_sched22:05:010
980599100,0cyclictest0-21swapper/022:31:160
98309990,0cyclictest0-21swapper/320:15:353
98239998,0cyclictest0-21swapper/222:17:492
98239998,0cyclictest0-21swapper/222:07:292
98239998,0cyclictest0-21swapper/219:52:292
98179998,0cyclictest0-21swapper/121:56:531
98179997,1cyclictest27428-21sshd22:40:481
98179994,2cyclictest25-21ksoftirqd/100:04:541
98179993,2cyclictest25-21ksoftirqd/122:55:331
98179993,2cyclictest25-21ksoftirqd/122:15:091
98179992,3cyclictest25-21ksoftirqd/123:05:461
98179992,1cyclictest25-21ksoftirqd/122:48:141
98179990,8cyclictest20269-21kworker/1:321:32:011
98179990,7cyclictest24648-21kworker/1:321:10:461
98179990,7cyclictest24571-21kworker/1:200:35:041
98179990,1cyclictest131rcu_sched22:21:121
98179990,0cyclictest0-21swapper/121:43:291
98179990,0cyclictest0-21swapper/121:43:281
98059998,0cyclictest0-21swapper/021:52:020
98059998,0cyclictest0-21swapper/020:36:080
98059998,0cyclictest0-21swapper/019:39:570
98059997,1cyclictest9-21ksoftirqd/023:50:480
98059995,1cyclictest9-21ksoftirqd/022:42:250
98059990,8cyclictest27041-21kworker/0:321:28:210
98059990,8cyclictest27041-21kworker/0:321:28:200
98059990,8cyclictest0-21swapper/021:24:140
98309987,0cyclictest0-21swapper/320:00:143
98239987,0cyclictest0-21swapper/221:35:202
98239986,1cyclictest2529-21lspci00:35:352
98239986,1cyclictest15031-21lspci22:34:112
98239985,2cyclictest33-21ksoftirqd/223:00:052
98239985,2cyclictest33-21ksoftirqd/223:00:042
98239982,5cyclictest13635-21sshd21:52:292
98239980,7cyclictest28655-21sshd21:34:242
98239980,7cyclictest24617-21sshd21:19:312
98239980,7cyclictest20095-21sshd23:52:342
98239980,7cyclictest18794-21sshd23:44:082
98239980,7cyclictest16481-21sshd23:07:492
98239980,7cyclictest11918-21sshd22:22:502
98179982,2cyclictest25-21ksoftirqd/122:35:221
98179982,2cyclictest25-21ksoftirqd/122:30:081
98179982,2cyclictest18656-21kworker/1:422:04:311
98179980,7cyclictest20269-21kworker/1:321:35:281
98179980,1cyclictest121rcu_preempt00:31:421
98059986,1cyclictest931-21gdbus23:00:150
98059986,1cyclictest931-21gdbus23:00:140
98059985,1cyclictest9-21ksoftirqd/023:36:230
98059981,1cyclictest131rcu_sched21:46:110
98059981,1cyclictest131rcu_sched00:16:200
98059980,7cyclictest550-21kworker/0:200:00:270
98059980,7cyclictest2893-21kworker/0:123:27:130
98059980,7cyclictest24864-21kworker/0:300:33:260
98059980,7cyclictest10226-21sshd22:56:040
98059980,7cyclictest10102-21sshd22:36:290
98309970,6cyclictest0-21swapper/320:09:263
98309970,6cyclictest0-21swapper/319:42:193
98309970,0cyclictest0-21swapper/319:19:513
98239975,1cyclictest31739-21lspci00:15:492
98239972,4cyclictest811-21dbus-daemon22:40:042
98239970,6cyclictest6733-21sshd20:23:312
98239970,6cyclictest5921-21sshd22:50:072
98239970,6cyclictest5270-21sshd21:46:152
98239970,6cyclictest11873-21sshd23:10:092
98179974,2cyclictest1448-21snmpd20:05:291
98059976,0cyclictest0-21swapper/019:59:290
98059974,2cyclictest1447-21snmpd20:01:520
98059970,6cyclictest0-21swapper/021:07:370
98059970,6cyclictest0-21swapper/020:08:030
98309965,0cyclictest0-21swapper/321:07:143
98309965,0cyclictest0-21swapper/320:45:273
98309963,2cyclictest1448-21snmpd20:21:083
98309963,2cyclictest1448-21snmpd19:20:043
98309963,2cyclictest1447-21snmpd20:50:033
98309963,2cyclictest1447-21snmpd20:40:013
98309963,2cyclictest1447-21snmpd20:28:153
98309963,2cyclictest1447-21snmpd20:12:563
98309963,2cyclictest1447-21snmpd19:27:483
98309960,3cyclictest24570-21grep19:45:063
98239965,0cyclictest0-21swapper/220:32:322
98239965,0cyclictest0-21swapper/219:40:142
98239965,0cyclictest0-21swapper/219:10:122
98239963,2cyclictest1448-21snmpd20:58:112
98239963,2cyclictest1448-21snmpd20:09:442
98239963,2cyclictest1447-21snmpd20:43:032
98239963,2cyclictest1447-21snmpd19:58:202
98239963,2cyclictest1447-21snmpd19:17:302
98239962,1cyclictest33-21ksoftirqd/219:36:302
98239960,5cyclictest0-21swapper/220:35:272
98179963,2cyclictest1448-21snmpd19:12:251
98179963,2cyclictest1447-21snmpd19:42:551
98179960,1cyclictest852-21ntpd19:15:001
98179960,1cyclictest17673-21sendmail_mailqu20:50:131
98179960,1cyclictest1-21systemd20:00:011
98179960,1cyclictest121rcu_preempt20:40:001
98059965,0cyclictest0-21swapper/020:33:050
98059965,0cyclictest0-21swapper/020:29:590
98059965,0cyclictest0-21swapper/019:15:530
98059964,1cyclictest17102-21head19:25:100
98059963,2cyclictest1447-21snmpd19:46:350
98059961,1cyclictest16043-21munin-plugin-st19:25:000
98059960,5cyclictest22333-21munin-run21:05:000
98309953,1cyclictest21029-21interrupts21:00:053
98309953,1cyclictest1448-21snmpd20:55:003
98309953,1cyclictest1448-21snmpd19:55:353
98309953,1cyclictest1447-21snmpd19:51:113
98309953,1cyclictest13313-21cat20:40:043
98309950,2cyclictest121rcu_preempt19:30:163
98239953,1cyclictest7471-21cat20:25:052
98239953,1cyclictest321ktimersoftd/220:10:122
98239953,1cyclictest19353-21unixbench_multi19:30:162
98239953,1cyclictest14870-21cpuspeed20:45:032
98239953,1cyclictest14729-21cat19:20:052
98239953,1cyclictest1448-21snmpd21:06:182
98239953,1cyclictest1448-21snmpd20:50:032
98239953,1cyclictest1448-21snmpd20:17:482
98239953,1cyclictest1448-21snmpd20:00:552
98239953,1cyclictest1448-21snmpd19:45:142
98239953,1cyclictest1238-21runrttasks19:25:042
98239952,2cyclictest1448-21snmpd21:01:212
98179953,1cyclictest9426-21cat20:30:061
98179953,1cyclictest3648-21cat20:15:061
98179953,1cyclictest24060-21cpuspeed_turbos19:45:031
98179953,1cyclictest23731-21kworker/1:020:00:011
98179953,1cyclictest22842-21df_inode21:05:041
98179953,1cyclictest20505-21cat19:35:051
98179953,1cyclictest16686-21irqstats19:25:071
98179953,1cyclictest14824-21latency19:20:061
98179953,1cyclictest1448-21snmpd20:25:271
98179953,1cyclictest1448-21snmpd20:10:521
98179953,1cyclictest1448-21snmpd19:50:371
98179953,1cyclictest1448-21snmpd19:31:191
98179953,1cyclictest1447-21snmpd21:00:101
98179953,1cyclictest1447-21snmpd20:55:061
98179951,2cyclictest15469-21grep20:45:061
98179950,1cyclictest5310-21cpuspeed_turbos20:20:031
98059953,1cyclictest22134-21cpuspeed_turbos19:40:020
98059953,1cyclictest18182-21/usr/sbin/munin19:30:040
98059953,1cyclictest1447-21snmpd20:55:000
98059953,1cyclictest1447-21snmpd20:20:450
98059953,1cyclictest1447-21snmpd20:15:450
98059953,1cyclictest13315-21cat20:40:050
98059953,1cyclictest10110-21cpuspeed_turbos19:10:030
98059952,2cyclictest856-21gdbus20:50:000
98059952,2cyclictest1447-21snmpd20:10:240
98059952,2cyclictest1447-21snmpd19:51:160
98059952,1cyclictest10733-21kworker/0:320:45:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional