You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-22 - 05:40

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack1slot0.osadl.org (updated Fri Oct 22, 2021 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
87072940,1sleep1820-21dbus-daemon22:22:221
233042640,0sleep10-21swapper/100:05:501
134032620,0sleep30-21swapper/323:18:053
22642610,0sleep00-21swapper/021:11:500
171132610,1sleep317111-21bash22:53:373
58402600,1sleep15836-21sshd21:18:041
118442600,0sleep11364-21nfsd23:20:401
280352590,0sleep10-21swapper/100:14:411
230112590,1sleep323010-21sshd00:19:473
54862580,0sleep00-21swapper/000:29:250
320782580,0sleep20-21swapper/221:31:512
112062580,1sleep2321ktimersoftd/221:49:262
80822570,0sleep20-21swapper/223:45:122
63842570,1sleep36378-21sshd22:05:343
481225746,7sleep10-21swapper/119:08:221
267122570,1sleep2321ktimersoftd/200:28:272
252582570,0sleep10-21swapper/121:50:331
215932570,1sleep021590-21lspci00:33:340
58072560,0sleep20-21swapper/222:22:072
490925633,7sleep00-21swapper/019:09:280
320542560,0sleep00-21swapper/022:40:560
30062560,0sleep00-21swapper/021:20:480
171362560,1sleep0505799cyclictest22:36:560
303342550,0sleep130336-21tty23:52:361
244882550,0sleep30-21swapper/300:05:563
9812540,1sleep21-21systemd22:02:212
289302540,0sleep00-21swapper/021:50:520
257672530,0sleep20-21swapper/200:06:032
230922530,0sleep10-21swapper/122:01:271
11112530,0sleep10-21swapper/123:39:061
325725240,7sleep30-21swapper/319:05:033
490625039,7sleep20-21swapper/219:09:272
36772460,0sleep30-21swapper/321:26:403
119502240,1sleep01-21systemd23:40:010
153612220,0sleep30-21swapper/322:45:083
186042210,0sleep10-21swapper/123:07:291
138342210,0sleep313835-21bash22:33:543
32382200,0sleep10-21swapper/122:13:371
285252200,2sleep3507599cyclictest23:11:043
247792200,0sleep00-21swapper/023:52:070
47302190,0sleep00-21swapper/021:40:310
305802190,0sleep30-21swapper/323:22:183
278512190,0sleep30-21swapper/300:39:423
265152190,1sleep126514-21sshd23:57:491
223802190,0sleep10-21swapper/122:56:481
108942190,1sleep310898-21bash23:04:073
96082180,0sleep00-21swapper/023:15:010
78062180,0sleep07805-21lspci21:49:080
308142180,1sleep1231rcuc/123:00:161
235372180,1sleep323540-21if_eno222:40:153
172212180,1sleep3820-21dbus-daemon21:55:243
15252180,2sleep3507599cyclictest21:48:353
60552170,0sleep1231rcuc/123:25:401
225682170,1sleep21-21systemd23:21:362
96882160,0sleep30-21swapper/322:14:113
188082160,0sleep0828-21rngd21:39:020
5172150,1sleep1506-21sshd23:19:471
506999131,1cyclictest131rcu_sched23:10:322
5075991210,1cyclictest16729-21lspci00:24:493
507599120,11cyclictest13386-21sshd21:21:473
506999122,3cyclictest33-21ksoftirqd/221:11:592
5069991211,0cyclictest14589-21lspci22:11:482
506999120,11cyclictest30295-21sshd00:17:362
506299120,1cyclictest14758-21cp00:30:141
506299120,11cyclictest1-21systemd21:10:201
505799120,11cyclictest21562-21packagekitd22:04:270
507599119,1cyclictest28323-21sshd23:08:213
507599119,1cyclictest15612-21lspci23:45:483
5075991110,0cyclictest0-21swapper/323:34:103
5075991110,0cyclictest0-21swapper/322:38:283
507599110,10cyclictest0-21swapper/323:54:513
507599110,10cyclictest0-21swapper/323:39:593
507599110,10cyclictest0-21swapper/322:56:043
507599110,10cyclictest0-21swapper/300:33:363
507599110,10cyclictest0-21swapper/300:28:353
506999119,1cyclictest32436-21sshd23:03:102
506999119,1cyclictest0-21swapper/222:54:272
506999110,1cyclictest7661-21lspci21:43:352
506999110,10cyclictest0-21swapper/222:43:192
506299119,1cyclictest31807-21sshd21:56:401
506299119,1cyclictest0-21swapper/122:38:401
506299110,1cyclictest31684-21sshd22:49:191
506299110,1cyclictest0-21swapper/121:25:011
506299110,10cyclictest27397-21bash22:18:281
506299110,10cyclictest0-21swapper/122:53:491
505799119,1cyclictest25090-21bash22:32:030
505799110,1cyclictest26245-21sshd21:31:210
505799110,10cyclictest0-21swapper/000:04:550
505799110,10cyclictest0-21swapper/000:04:540
507599108,1cyclictest25341-21bash22:18:173
507599100,0cyclictest0-21swapper/321:31:213
506999109,0cyclictest0-21swapper/223:07:042
506999109,0cyclictest0-21swapper/220:20:242
506999109,0cyclictest0-21swapper/200:00:002
506999108,1cyclictest27226-21sshd21:28:432
506999108,1cyclictest1-21systemd23:28:022
506999108,1cyclictest0-21swapper/223:17:132
506999103,3cyclictest33-21ksoftirqd/221:54:292
506999101,1cyclictest131rcu_sched23:37:262
506999100,9cyclictest0-21swapper/220:15:442
506999100,3cyclictest13110-21sshd21:15:472
506999100,1cyclictest33-21ksoftirqd/200:34:412
506999100,1cyclictest131rcu_sched22:46:372
506999100,0cyclictest131rcu_sched00:03:352
506999100,0cyclictest131rcu_sched00:03:352
506999100,0cyclictest0-21swapper/222:30:532
506999100,0cyclictest0-21swapper/221:02:342
506999100,0cyclictest0-21swapper/219:10:212
506299109,0cyclictest0-21swapper/119:40:141
506299109,0cyclictest0-21swapper/100:04:231
506299109,0cyclictest0-21swapper/100:04:231
506299100,0cyclictest0-21swapper/123:12:231
506299100,0cyclictest0-21swapper/122:30:151
506299100,0cyclictest0-21swapper/122:26:131
506299100,0cyclictest0-21swapper/121:10:001
506299100,0cyclictest0-21swapper/120:15:261
505799109,0cyclictest0-21swapper/023:37:460
505799109,0cyclictest0-21swapper/022:59:300
505799109,0cyclictest0-21swapper/022:45:570
505799100,9cyclictest0-21swapper/019:40:550
505799100,0cyclictest0-21swapper/023:28:150
505799100,0cyclictest0-21swapper/022:52:340
505799100,0cyclictest0-21swapper/022:09:080
505799100,0cyclictest0-21swapper/021:26:590
505799100,0cyclictest0-21swapper/019:49:570
505799100,0cyclictest0-21swapper/019:22:300
260082100,0sleep2311rcuc/222:15:302
8997290,0sleep00-21swapper/022:19:410
50759998,0cyclictest19150-21lspci00:02:463
50759998,0cyclictest19150-21lspci00:02:453
50759998,0cyclictest0-21swapper/323:42:533
50759998,0cyclictest0-21swapper/323:25:083
50759998,0cyclictest0-21swapper/322:26:023
50759998,0cyclictest0-21swapper/321:15:003
50759998,0cyclictest0-21swapper/321:07:533
50759990,0cyclictest0-21swapper/321:50:503
50699998,0cyclictest33-21ksoftirqd/200:21:012
50699998,0cyclictest0-21swapper/223:32:332
50699998,0cyclictest0-21swapper/222:35:142
50699998,0cyclictest0-21swapper/219:19:332
50699997,1cyclictest33-21ksoftirqd/223:42:382
50699991,2cyclictest33-21ksoftirqd/223:50:132
50699991,1cyclictest33-21ksoftirqd/222:05:072
50699991,1cyclictest131rcu_sched22:58:082
50699990,8cyclictest5186-21kworker/2:300:10:072
50699990,8cyclictest1982-21kworker/2:100:37:372
50699990,1cyclictest33-21ksoftirqd/222:25:582
50699990,1cyclictest33-21ksoftirqd/221:38:412
50699990,1cyclictest33-21ksoftirqd/221:21:132
50699990,1cyclictest131rcu_sched21:56:352
50699990,0cyclictest0-21swapper/220:50:082
50629998,0cyclictest0-21swapper/100:26:591
50579998,0cyclictest24572-21sshd23:49:240
50579998,0cyclictest0-21swapper/022:28:020
50579998,0cyclictest0-21swapper/021:16:100
50579998,0cyclictest0-21swapper/000:35:440
50579998,0cyclictest0-21swapper/000:14:360
50579990,8cyclictest7759-21sshd00:15:370
50579990,8cyclictest3625-21sshd23:09:000
50759986,1cyclictest23176-21lspci21:19:423
50759986,1cyclictest20909-21lspci23:57:203
50759985,1cyclictest3845-21sshd00:12:353
50759980,7cyclictest28347-21sshd21:37:033
50759980,7cyclictest22936-21sshd21:42:063
50759980,7cyclictest20184-21sshd22:01:123
50699980,0cyclictest0-21swapper/221:09:402
50629987,0cyclictest0-21swapper/120:54:301
50629987,0cyclictest0-21swapper/100:37:001
50629986,1cyclictest5139-21lspci00:15:231
50629986,1cyclictest2943-21lspci21:45:541
50629986,1cyclictest18802-21lspci21:36:131
50629980,7cyclictest7163-21sshd23:34:061
50629980,7cyclictest3934-21sshd21:20:531
50629980,7cyclictest18143-21sshd23:43:171
50629980,7cyclictest14867-21sshd21:41:241
50629980,7cyclictest13435-21sshd21:30:171
50579987,0cyclictest0-21swapper/020:42:550
50579986,1cyclictest8593-21lspci22:11:150
50579980,7cyclictest8162-21sshd21:57:250
50579980,7cyclictest5784-21sshd23:22:580
50579980,7cyclictest5343-21sshd23:00:480
50579980,7cyclictest3397-21sshd23:30:580
50579980,7cyclictest26290-21sshd22:23:550
50579980,7cyclictest23053-21sshd00:08:390
50579980,7cyclictest17677-21sshd00:24:530
50759970,6cyclictest24572-21sshd19:55:543
50759970,6cyclictest22927-21sshd22:20:483
50699976,0cyclictest0-21swapper/220:09:262
50699976,0cyclictest0-21swapper/219:58:082
50699976,0cyclictest0-21swapper/219:52:492
50699970,0cyclictest0-21swapper/220:43:502
50629976,0cyclictest0-21swapper/100:20:331
50629975,1cyclictest14715-21cat20:55:181
50629970,6cyclictest5067-21sshd22:41:251
50629970,6cyclictest341-21sshd22:05:051
50629970,6cyclictest22626-21sshd19:50:531
50629970,6cyclictest1225-21sshd23:47:221
50579976,0cyclictest0-21swapper/019:50:160
50579970,6cyclictest30345-21sshd23:55:230
50579970,6cyclictest15946-21sshd23:10:040
50579970,6cyclictest0-21swapper/020:09:390
50759964,1cyclictest1398-21snmpd20:23:083
50759963,2cyclictest1398-21snmpd20:55:393
50759963,2cyclictest1398-21snmpd20:05:163
50759963,2cyclictest1398-21snmpd19:45:293
50759963,2cyclictest1397-21snmpd20:37:363
50759963,2cyclictest1397-21snmpd20:30:243
50759963,2cyclictest1397-21snmpd19:30:163
50759963,2cyclictest1397-21snmpd19:24:313
50759963,2cyclictest1397-21snmpd19:18:073
50759963,2cyclictest1397-21snmpd19:12:013
50699965,0cyclictest0-21swapper/220:31:342
50699965,0cyclictest0-21swapper/220:02:302
50699964,1cyclictest13838-21hddtemp_smartct19:30:182
50699960,5cyclictest1938-21kworker/2:120:25:542
50629963,2cyclictest26923-21latency_hist20:04:591
50629963,2cyclictest1398-21snmpd20:26:081
50629960,5cyclictest0-21swapper/119:10:411
50579965,0cyclictest0-21swapper/020:15:180
50579964,1cyclictest4006-21sshd20:26:300
50579963,2cyclictest1398-21snmpd20:50:390
50579963,2cyclictest1397-21snmpd19:59:110
50579963,2cyclictest1397-21snmpd19:39:380
50579963,2cyclictest1397-21snmpd19:33:280
50579960,5cyclictest19619-21sshd21:06:380
50579960,5cyclictest0-21swapper/019:28:290
50579960,1cyclictest15126-21ntp_states20:55:220
50759953,1cyclictest9317-21pmu-power20:40:203
50759953,1cyclictest31248-21cstates20:15:143
50759953,1cyclictest29517-21cat20:10:143
50759953,1cyclictest16326-21cpuspeed21:00:143
50759953,1cyclictest1398-21snmpd20:50:423
50759953,1cyclictest1398-21snmpd20:25:473
50759953,1cyclictest1397-21snmpd20:45:573
50759953,1cyclictest1397-21snmpd20:00:383
50759953,1cyclictest1397-21snmpd19:35:343
50759953,1cyclictest1397-21snmpd19:26:043
50759953,1cyclictest1219-21runrttasks19:40:133
50759952,2cyclictest22866-21taskset19:53:303
50699953,1cyclictest29108-21/usr/sbin/munin20:10:182
50699953,1cyclictest19350-21/usr/sbin/munin19:45:162
50699953,1cyclictest1398-21snmpd20:35:142
50699953,1cyclictest1398-21snmpd19:20:032
50699953,1cyclictest1397-21snmpd20:55:042
50699953,1cyclictest1397-21snmpd20:45:062
50699953,1cyclictest1397-21snmpd19:40:322
50699953,1cyclictest11948-21cat19:25:142
50699952,2cyclictest1398-21snmpd19:35:172
50629953,1cyclictest4982-21cat20:30:151
50629953,1cyclictest16276-21/usr/sbin/munin21:00:201
50629953,1cyclictest1398-21snmpd20:44:111
50629953,1cyclictest1398-21snmpd20:36:021
50629953,1cyclictest1398-21snmpd20:20:381
50629953,1cyclictest1398-21snmpd19:55:221
50629953,1cyclictest1398-21snmpd19:46:061
50629953,1cyclictest1398-21snmpd19:35:571
50629953,1cyclictest1398-21snmpd19:25:051
50629953,1cyclictest1397-21snmpd20:45:031
50629953,1cyclictest1397-21snmpd20:10:101
50629953,1cyclictest1397-21snmpd20:06:201
50629953,1cyclictest1397-21snmpd19:30:071
50629953,1cyclictest1397-21snmpd19:15:511
50629953,1cyclictest10033-21cat19:20:171
50579953,1cyclictest6936-21cat20:35:180
50579953,1cyclictest1398-21snmpd20:45:110
50579953,1cyclictest1398-21snmpd20:10:220
50579953,1cyclictest1397-21snmpd20:21:270
50579953,1cyclictest1397-21snmpd20:00:460
50579953,1cyclictest1397-21snmpd19:15:360
50579953,1cyclictest1397-21snmpd19:10:460
50579952,2cyclictest5234-21grep20:30:190
50579950,4cyclictest0-21swapper/021:00:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional