You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-06-15 - 08:50

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack1slot0.osadl.org (updated Tue Jun 15, 2021 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
277922970,0sleep21-21systemd23:09:372
297612960,0sleep029760-21grepconf.sh22:29:210
252342850,0sleep20-21swapper/223:18:032
110192670,0sleep011012-21sshd21:20:300
300182630,0sleep00-21swapper/022:06:130
174832610,1sleep3391rcuc/323:11:303
172352610,0sleep30-21swapper/323:49:163
1891526037,7sleep20-21swapper/219:05:052
95972590,0sleep00-21swapper/021:17:220
90572590,0sleep29055-21lspci21:44:152
240642590,0sleep10-21swapper/123:55:391
322992580,1sleep2820-21dbus-daemon21:49:132
314672580,0sleep00-21swapper/022:20:430
275512580,0sleep10-21swapper/121:45:511
141372580,0sleep20-21swapper/222:07:432
276932570,1sleep11-21systemd00:04:491
178492570,1sleep31-21systemd23:20:153
47002560,0sleep1231rcuc/121:29:171
212022560,0sleep00-21swapper/021:59:440
2039925633,7sleep00-21swapper/019:07:240
2026025634,7sleep10-21swapper/119:06:091
12252560,0sleep30-21swapper/321:52:103
109432560,0sleep00-21swapper/023:34:150
63542550,0sleep16356-21bash00:05:451
318352550,0sleep00-21swapper/000:02:150
226302550,0sleep10-21swapper/100:15:581
152882550,0sleep015284-21sshd23:14:160
295242540,0sleep129526-21id22:26:221
122622540,0sleep20-21swapper/223:43:022
244752530,0sleep324470-21bash23:55:403
224382530,0sleep2321ktimersoftd/222:40:132
81972520,0sleep28195-21lspci22:21:342
2025225139,7sleep30-21swapper/319:06:023
86902450,0sleep10-21swapper/100:26:241
25542450,0sleep30-21swapper/321:19:503
26262430,0sleep10-21swapper/121:22:471
48522190,0sleep04850-21lspci22:30:000
104862190,0sleep20-21swapper/200:35:182
5792160,0sleep10-21swapper/122:20:531
307622160,0sleep20-21swapper/220:55:222
20745991610,1cyclictest24285-21bash22:17:153
20745991610,1cyclictest24148-21bash22:43:153
191732160,0sleep00-21swapper/022:57:130
120742160,1sleep01-21systemd00:32:310
32822150,0sleep0820-21dbus-daemon21:49:300
20738991514,0cyclictest25832-21lspci23:29:492
2073899151,11cyclictest801-21systemd-logind23:53:382
192702140,0sleep30-21swapper/300:04:033
41652130,0sleep30-21swapper/322:26:593
294372130,0sleep30-21swapper/323:35:503
20738991312,0cyclictest19831-21lspci22:31:172
20738991211,0cyclictest25488-21lspci22:14:322
2073399120,1cyclictest11127-21lspci22:53:381
2073399120,1cyclictest0-21swapper/121:10:281
2072699120,1cyclictest10215-21sshd00:20:420
2074599119,1cyclictest1-21systemd21:33:203
2074599110,10cyclictest15289-21bash22:04:593
2074599110,10cyclictest0-21swapper/323:16:123
2074599110,10cyclictest0-21swapper/321:22:403
2073899111,1cyclictest33-21ksoftirqd/200:14:532
2073899111,0cyclictest33-21ksoftirqd/221:12:332
20738991110,0cyclictest0-21swapper/222:57:562
2073399110,1cyclictest30992-21sshd21:34:411
2073399110,10cyclictest1203-21sshd22:36:221
2073399110,10cyclictest0-21swapper/122:58:291
2073399110,10cyclictest0-21swapper/122:46:331
2073399110,10cyclictest0-21swapper/121:19:021
2073399110,10cyclictest0-21swapper/100:37:471
2073399110,10cyclictest0-21swapper/100:11:091
2072699119,1cyclictest6386-21sshd23:04:490
2072699119,1cyclictest0-21swapper/022:02:020
127132110,1sleep3401ktimersoftd/300:26:463
2074599109,0cyclictest0-21swapper/323:42:023
2074599109,0cyclictest0-21swapper/322:23:533
2074599109,0cyclictest0-21swapper/322:06:063
2074599108,1cyclictest18324-21lspci22:31:093
2074599108,1cyclictest0-21swapper/322:39:513
2074599100,0cyclictest0-21swapper/300:24:103
2074599100,0cyclictest0-21swapper/300:18:013
2073899109,0cyclictest20649-21lspci23:37:592
2073899109,0cyclictest0-21swapper/223:12:362
2073899109,0cyclictest0-21swapper/222:26:162
2073899109,0cyclictest0-21swapper/221:54:082
2073899109,0cyclictest0-21swapper/221:23:082
2073899109,0cyclictest0-21swapper/219:44:482
2073899108,1cyclictest27123-21kworker/2:022:39:122
2073899101,1cyclictest33-21ksoftirqd/223:00:162
2073899101,1cyclictest33-21ksoftirqd/221:34:402
2073899101,1cyclictest131rcu_sched23:22:082
2073899100,9cyclictest29458-21kworker/2:222:50:522
2073899100,9cyclictest0-21swapper/219:46:352
2073899100,9cyclictest0-21swapper/200:03:002
2073899100,8cyclictest7685-21kworker/2:021:35:162
2073899100,0cyclictest131rcu_sched21:19:102
2073899100,0cyclictest0-21swapper/223:47:092
2073399109,0cyclictest0-21swapper/123:40:281
2073399109,0cyclictest0-21swapper/119:25:211
2073399109,0cyclictest0-21swapper/100:24:041
2073399100,9cyclictest0-21swapper/122:15:411
2073399100,9cyclictest0-21swapper/120:55:081
2073399100,0cyclictest0-21swapper/123:24:291
2073399100,0cyclictest0-21swapper/121:54:431
2072699109,0cyclictest24791-21lspci00:36:350
2072699109,0cyclictest21369-21lspci23:09:020
2072699109,0cyclictest0-21swapper/023:56:590
2072699109,0cyclictest0-21swapper/023:47:470
2072699109,0cyclictest0-21swapper/023:25:390
2072699109,0cyclictest0-21swapper/000:26:320
2072699108,1cyclictest9-21ksoftirqd/022:20:000
2072699100,9cyclictest0-21swapper/020:08:500
2072699100,0cyclictest13801-21sshd21:53:180
2072699100,0cyclictest0-21swapper/023:17:150
2072699100,0cyclictest0-21swapper/022:45:080
2072699100,0cyclictest0-21swapper/022:37:130
2072699100,0cyclictest0-21swapper/021:30:130
207459998,0cyclictest0-21swapper/323:05:133
207459998,0cyclictest0-21swapper/323:01:103
207459998,0cyclictest0-21swapper/322:45:303
207459998,0cyclictest0-21swapper/321:57:053
207459998,0cyclictest0-21swapper/321:27:423
207459998,0cyclictest0-21swapper/319:39:513
207459998,0cyclictest0-21swapper/300:32:243
207459997,1cyclictest26964-21lspci22:14:413
207459997,1cyclictest1-21systemd21:48:023
207459990,0cyclictest0-21swapper/300:35:093
207389998,0cyclictest0-21swapper/200:25:162
207389997,1cyclictest33-21ksoftirqd/223:55:262
207389997,1cyclictest33-21ksoftirqd/222:03:072
207389997,1cyclictest19268-21lspci22:45:382
207389992,2cyclictest33-21ksoftirqd/200:09:042
207389990,8cyclictest5784-21kworker/2:122:19:302
207389990,1cyclictest131rcu_sched21:29:022
207389990,1cyclictest131rcu_sched00:33:272
207389990,1cyclictest1-21systemd23:32:542
207389990,1cyclictest121rcu_preempt21:58:062
207339998,0cyclictest0-21swapper/120:46:451
207339997,1cyclictest9806-21sshd22:13:071
207339990,0cyclictest0-21swapper/123:49:301
207269997,1cyclictest9-21ksoftirqd/000:07:140
207269991,3cyclictest9-21ksoftirqd/021:25:440
207269990,1cyclictest131rcu_sched00:12:580
207269990,0cyclictest0-21swapper/021:37:570
207269990,0cyclictest0-21swapper/020:14:480
2651280,0sleep30-21swapper/321:13:363
207459987,0cyclictest0-21swapper/323:51:043
207459987,0cyclictest0-21swapper/322:58:363
207459987,0cyclictest0-21swapper/321:35:073
207459987,0cyclictest0-21swapper/320:00:103
207459987,0cyclictest0-21swapper/300:14:393
207459986,1cyclictest18550-21lspci23:31:593
207459986,1cyclictest12286-21lspci21:44:333
207459980,7cyclictest5263-21sshd22:50:143
207459980,7cyclictest28509-21sshd23:27:033
207459980,7cyclictest14678-21sshd00:09:263
207389987,0cyclictest0-21swapper/220:16:372
207389986,1cyclictest33-21ksoftirqd/200:15:582
207389980,7cyclictest1469-21sshd00:22:512
207339987,0cyclictest0-21swapper/123:09:091
207339986,1cyclictest7349-21lspci23:25:091
207339986,1cyclictest7168-21lspci21:55:301
207339986,1cyclictest6573-21lspci22:07:011
207339986,1cyclictest0-21swapper/123:50:581
207339983,4cyclictest820-21dbus-daemon22:42:091
207339980,7cyclictest32739-21sshd21:37:401
207269987,0cyclictest0-21swapper/023:54:340
207269987,0cyclictest0-21swapper/023:24:120
207269987,0cyclictest0-21swapper/019:53:500
207269987,0cyclictest0-21swapper/019:41:330
207269984,0cyclictest9-21ksoftirqd/022:50:210
207269983,1cyclictest9-21ksoftirqd/023:40:070
207269981,4cyclictest9-21ksoftirqd/023:36:130
207269980,7cyclictest5559-21sshd22:41:330
207269980,1cyclictest131rcu_sched22:10:160
207269980,0cyclictest131rcu_sched21:42:280
20304280,6sleep020469-21grepconf.sh00:15:460
207459976,0cyclictest0-21swapper/321:08:183
207459970,6cyclictest0-21swapper/320:55:333
207389970,6cyclictest6134-21sshd19:52:502
207339976,0cyclictest0-21swapper/119:53:371
207339975,1cyclictest27149-21sshd22:00:151
207339975,1cyclictest1542-21lspci23:10:091
207339975,1cyclictest12842-21lspci23:37:161
207339975,1cyclictest10100-21lspci23:31:141
207339970,6cyclictest3422-21sshd23:01:361
207339970,6cyclictest2958-21sshd21:06:491
207339970,6cyclictest23686-21sshd23:15:011
207339970,6cyclictest20181-21sshd00:30:191
207339970,6cyclictest15762-21sshd21:41:561
207339970,6cyclictest10664-21sshd22:30:251
207339970,6cyclictest0-21swapper/120:02:401
207269976,0cyclictest0-21swapper/019:20:520
207269970,6cyclictest838-21sshd21:00:500
207269970,1cyclictest121rcu_preempt21:10:120
207459964,1cyclictest24429-21df_inode20:40:143
207459963,2cyclictest1398-21snmpd20:49:443
207459963,2cyclictest1398-21snmpd20:09:043
207389964,1cyclictest909-21NetworkManager19:16:432
207389962,1cyclictest0-21swapper/220:31:512
207389961,2cyclictest131rcu_sched20:49:552
207389960,5cyclictest9860-21kworker/2:220:01:352
207389960,5cyclictest955-21sshd21:01:442
207389960,5cyclictest0-21swapper/220:10:392
207389960,5cyclictest0-21swapper/219:22:222
207389960,0cyclictest0-21swapper/219:35:562
207339965,0cyclictest0-21swapper/119:32:081
207339964,1cyclictest26232-21smart_sda19:20:211
207339963,2cyclictest1398-21snmpd20:39:561
207339963,2cyclictest1397-21snmpd19:10:111
207339960,5cyclictest0-21swapper/120:17:011
207339960,0cyclictest0-21swapper/120:54:481
207269965,0cyclictest0-21swapper/019:58:200
207269965,0cyclictest0-21swapper/019:12:380
207269963,2cyclictest1397-21snmpd20:50:400
207269963,2cyclictest1397-21snmpd19:39:150
207269960,5cyclictest15968-21sshd20:18:500
207459954,0cyclictest0-21swapper/319:16:353
207459953,1cyclictest32730-21open_files21:00:183
207459953,1cyclictest3100-21cat19:45:183
207459953,1cyclictest29578-21cat19:30:163
207459953,1cyclictest22505-21hwlatdetect20:35:153
207459953,1cyclictest21311-21cat19:10:123
207459953,1cyclictest1398-21snmpd20:51:153
207459953,1cyclictest1398-21snmpd20:31:033
207459953,1cyclictest1398-21snmpd20:10:043
207459953,1cyclictest1398-21snmpd19:55:103
207459953,1cyclictest1397-21snmpd20:25:193
207459953,1cyclictest1397-21snmpd20:20:113
207459953,1cyclictest1397-21snmpd19:22:173
207459952,2cyclictest5467-21grep19:50:203
207459952,2cyclictest15129-21ntp_kernel_err20:15:183
207459952,2cyclictest1132-21if_err_eno219:40:173
207459950,4cyclictest27668-21tr19:25:153
207389953,1cyclictest25005-21grep20:40:212
207389953,1cyclictest20922-21/usr/sbin/munin19:10:132
207389953,1cyclictest16757-21cat20:20:172
207389953,1cyclictest1398-21snmpd21:06:242
207389953,1cyclictest1398-21snmpd20:50:112
207389953,1cyclictest1398-21snmpd20:06:432
207389953,1cyclictest1397-21snmpd20:35:432
207389953,1cyclictest1397-21snmpd20:25:012
207389953,1cyclictest1397-21snmpd19:30:292
207389951,0cyclictest121rcu_preempt19:25:212
207389950,4cyclictest6575-21/usr/sbin/munin19:55:112
207339953,1cyclictest6575-21/usr/sbin/munin19:55:121
207339953,1cyclictest32343-21cat21:00:151
207339953,1cyclictest2689-21/usr/sbin/munin19:45:151
207339953,1cyclictest1546-21grep19:40:191
207339953,1cyclictest1398-21snmpd20:40:051
207339953,1cyclictest1398-21snmpd20:30:291
207339953,1cyclictest1398-21snmpd19:35:511
207339953,1cyclictest1397-21snmpd20:25:431
207339953,1cyclictest1397-21snmpd20:20:351
207339953,1cyclictest1397-21snmpd19:15:061
207339953,1cyclictest12850-21cat20:10:171
207339953,1cyclictest11249-21sed20:05:201
207269953,1cyclictest8945-21cat20:00:170
207269953,1cyclictest3099-21cat19:45:180
207269953,1cyclictest29655-21sort19:30:170
207269953,1cyclictest27640-21cat19:25:130
207269953,1cyclictest23763-21interrupts19:15:150
207269953,1cyclictest1398-21snmpd20:55:030
207269953,1cyclictest1398-21snmpd20:36:120
207269953,1cyclictest1397-21snmpd21:05:180
207269953,1cyclictest1397-21snmpd20:40:010
207269953,1cyclictest1397-21snmpd20:30:580
207269953,1cyclictest1397-21snmpd20:25:400
207269952,2cyclictest16497-21tr20:20:140
207269951,1cyclictest26432-21idleruntime20:45:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional