You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-17 - 05:12

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack1slot0.osadl.org (updated Sun Oct 17, 2021 00:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
78892630,7sleep37887-21lspci00:14:453
108052620,0sleep110799-21bash22:38:241
218832610,0sleep10-21swapper/122:50:191
97902600,2sleep0804799cyclictest21:21:070
792625834,7sleep30-21swapper/319:09:353
185252580,1sleep2913-21gdbus00:37:402
325972570,0sleep032599-21grep23:21:380
273232570,0sleep30-21swapper/322:39:503
222392570,0sleep20-21swapper/200:29:462
195632570,0sleep10-21swapper/121:00:131
175342570,0sleep30-21swapper/321:30:153
87732560,1sleep01364-21nfsd22:02:210
50252560,0sleep1231rcuc/121:23:401
258522560,0sleep10-21swapper/122:01:021
148422560,1sleep214813-21sshd22:35:572
134462560,0sleep0101ktimersoftd/023:17:140
706225533,4sleep20-21swapper/219:05:212
35442550,0sleep10-21swapper/120:20:011
277622550,0sleep30-21swapper/322:56:173
107332550,0sleep00-21swapper/000:09:290
21162540,0sleep02115-21systemd-cgroups21:37:090
107282540,1sleep010725-21sshd00:17:420
93632530,0sleep29358-21sshd22:30:002
232762530,0sleep10-21swapper/122:25:341
125212520,0sleep10-21swapper/100:20:371
18832510,0sleep00-21swapper/000:28:000
780225039,7sleep00-21swapper/019:08:080
771325039,7sleep10-21swapper/119:07:131
22092490,0sleep00-21swapper/021:42:390
106802420,1sleep0867-21gdbus22:16:160
207342410,0sleep00-21swapper/021:55:120
217372360,0sleep30-21swapper/322:22:453
317112240,0sleep20-21swapper/200:22:172
105132200,1sleep310512-21bash00:17:423
270092190,0sleep00-21swapper/021:33:490
258872190,0sleep30-21swapper/321:50:103
117532190,0sleep20-21swapper/221:38:002
34052180,0sleep30-21swapper/322:12:553
277422180,0sleep10-21swapper/123:54:241
19362180,0sleep30-21swapper/323:38:233
74672170,0sleep17465-21lspci21:56:461
72192170,1sleep0111rcuc/022:24:170
301442170,1sleep230142-21sshd23:07:342
252312170,0sleep10-21swapper/100:05:131
64592160,0sleep20-21swapper/223:35:582
200292160,0sleep0111rcuc/023:59:160
174092160,0sleep2311rcuc/220:55:142
804799150,1cyclictest8250-21lspci21:48:390
804799150,10cyclictest20213-21sshd22:47:260
253292150,1sleep125330-21sshd22:12:001
89212140,0sleep18909-21sshd21:26:451
804799140,10cyclictest30758-21sshd00:38:430
804799140,10cyclictest29582-21sshd23:01:590
66622140,0sleep10-21swapper/122:24:141
167792140,0sleep00-21swapper/023:53:270
8060991312,0cyclictest11810-21lspci00:01:152
8060991312,0cyclictest11112-21lspci23:55:412
804799138,4cyclictest19730-21sshd22:50:100
804799138,4cyclictest18814-21sshd22:36:170
804799132,1cyclictest0-21swapper/023:31:540
804799130,8cyclictest6251-21sshd23:13:480
804799130,0cyclictest0-21swapper/000:33:510
8067991210,1cyclictest913-21gdbus23:08:103
806799120,1cyclictest27218-21sshd23:54:213
806799120,1cyclictest26701-21sshd23:59:513
806799120,11cyclictest6699-21sshd23:44:203
806099120,1cyclictest5632-21/usr/sbin/munin20:25:252
806099120,1cyclictest1364-21nfsd23:47:152
805599120,1cyclictest820-21dbus-daemon00:37:531
805599120,11cyclictest8949-21sshd22:43:441
804799120,11cyclictest3097-21sshd22:12:530
804799120,11cyclictest22294-21sshd21:25:120
804799120,11cyclictest21564-21gdbus22:58:050
806799119,1cyclictest801-21systemd-logind00:26:013
806799119,1cyclictest1-21systemd21:12:533
806799110,1cyclictest3386-21sshd21:37:163
806799110,10cyclictest0-21swapper/322:50:443
806799110,10cyclictest0-21swapper/300:22:073
806799110,0cyclictest0-21swapper/323:25:073
806099119,1cyclictest20480-21sshd21:19:102
8060991110,0cyclictest1527-21lspci00:14:122
806099110,10cyclictest0-21swapper/223:44:492
806099110,10cyclictest0-21swapper/222:43:562
806099110,10cyclictest0-21swapper/222:23:452
806099110,10cyclictest0-21swapper/221:24:592
806099110,0cyclictest0-21swapper/221:45:202
805599119,1cyclictest29845-21bash00:00:061
805599119,1cyclictest11876-21sshd23:25:241
805599110,10cyclictest0-21swapper/123:49:281
805599110,10cyclictest0-21swapper/100:28:021
804799119,1cyclictest27722-21sshd22:25:570
804799118,2cyclictest0-21swapper/021:10:140
804799112,1cyclictest31637-21bash23:29:570
8047991110,0cyclictest0-21swapper/023:38:380
804799110,2cyclictest22286-21sshd23:48:230
804799110,10cyclictest29183-21sshd21:17:000
806799109,0cyclictest0-21swapper/323:18:013
806799109,0cyclictest0-21swapper/321:29:423
806799109,0cyclictest0-21swapper/321:15:573
806799108,1cyclictest0-21swapper/323:30:373
806799100,0cyclictest0-21swapper/323:46:463
806799100,0cyclictest0-21swapper/323:14:393
806799100,0cyclictest0-21swapper/322:43:283
806099109,0cyclictest33-21ksoftirqd/223:54:592
806099109,0cyclictest31798-21lspci21:31:262
806099109,0cyclictest23275-21lspci23:04:162
806099109,0cyclictest0-21swapper/222:51:332
806099109,0cyclictest0-21swapper/219:18:442
806099109,0cyclictest0-21swapper/200:32:262
806099108,1cyclictest9072-21lspci21:54:132
806099103,1cyclictest33-21ksoftirqd/222:57:432
806099101,3cyclictest33-21ksoftirqd/200:18:212
806099101,2cyclictest33-21ksoftirqd/223:12:292
806099101,1cyclictest33-21ksoftirqd/222:31:022
806099101,1cyclictest33-21ksoftirqd/222:31:022
806099100,1cyclictest33-21ksoftirqd/221:11:542
806099100,0cyclictest131rcu_sched23:23:152
806099100,0cyclictest0-21swapper/200:09:322
805599109,0cyclictest0-21swapper/123:57:221
805599109,0cyclictest0-21swapper/123:18:171
805599109,0cyclictest0-21swapper/123:14:541
805599109,0cyclictest0-21swapper/121:15:541
805599100,0cyclictest0-21swapper/123:21:401
804799108,1cyclictest0-21swapper/019:31:340
804799104,1cyclictest32504-21sshd21:50:390
804799104,1cyclictest29303-21kworker/u8:123:43:070
804799101,8cyclictest9839-21lspci00:20:220
804799101,8cyclictest833-21polkitd22:08:020
804799100,9cyclictest3911-21sshd23:08:060
804799100,7cyclictest0-21swapper/020:33:150
804799100,0cyclictest0-21swapper/000:11:340
80679998,0cyclictest0-21swapper/321:57:473
80679998,0cyclictest0-21swapper/321:48:153
80679997,1cyclictest6716-21cat20:25:253
80679990,0cyclictest0-21swapper/300:05:403
80609998,0cyclictest6065-21sshd22:10:212
80609998,0cyclictest3515-21lspci21:29:052
80609998,0cyclictest1700-21lspci23:30:072
80609998,0cyclictest0-21swapper/220:44:082
80609997,1cyclictest33-21ksoftirqd/223:28:332
80609992,2cyclictest33-21ksoftirqd/222:05:152
80609992,1cyclictest33-21ksoftirqd/221:56:252
80609990,8cyclictest0-21swapper/221:00:272
80559998,0cyclictest0-21swapper/123:08:261
80559998,0cyclictest0-21swapper/100:15:071
80559990,8cyclictest3791-21sshd22:57:001
80479998,0cyclictest0-21swapper/019:50:280
80479996,2cyclictest0-21swapper/020:41:490
80479993,2cyclictest1397-21snmpd19:26:530
80479993,1cyclictest574-21cat22:40:180
80479993,1cyclictest1398-21snmpd20:37:030
80479993,1cyclictest1398-21snmpd19:36:160
80479993,1cyclictest1397-21snmpd20:53:330
80479992,2cyclictest0-21swapper/020:23:240
80479990,8cyclictest29375-21sshd00:00:040
80479990,8cyclictest12788-21sshd22:30:160
80479990,8cyclictest12788-21sshd22:30:150
80479990,4cyclictest7290-21taskset20:29:440
80479990,2cyclictest30161-21/usr/sbin/munin20:05:250
80479990,2cyclictest30161-21/usr/sbin/munin20:05:250
80479990,0cyclictest0-21swapper/019:14:470
80679987,0cyclictest0-21swapper/321:42:263
80679986,1cyclictest27127-21lspci22:48:023
80679986,1cyclictest14909-21lspci22:19:263
80679986,1cyclictest1478-21systemd-cgroups23:21:453
80679980,7cyclictest3934-21sshd21:23:343
80679980,7cyclictest3588-21sshd00:36:223
80679980,7cyclictest336-21sshd22:29:133
80679980,7cyclictest15923-21sshd23:00:483
80679980,7cyclictest14330-21sshd00:34:363
80679980,7cyclictest10605-21sshd22:30:063
80679980,7cyclictest10605-21sshd22:30:063
80609987,0cyclictest15678-21lspci22:00:162
80609981,1cyclictest33-21ksoftirqd/223:15:372
80609980,7cyclictest9821-21kworker/2:122:45:032
80609980,7cyclictest14853-21sshd21:40:562
80609980,3cyclictest131rcu_sched22:17:242
80559986,1cyclictest25-21ksoftirqd/121:51:031
80559986,1cyclictest16115-21lspci22:05:401
80559980,7cyclictest23640-21sshd23:42:591
80559980,7cyclictest20706-21sshd00:10:191
80559980,7cyclictest19982-21kworker/1:421:34:241
80559980,1cyclictest13138-21lspci21:38:061
80479983,1cyclictest28920-21cut20:00:190
80479983,1cyclictest21672-21cat21:05:180
80479983,1cyclictest1397-21snmpd20:45:310
80479982,1cyclictest20839-21cat19:40:180
80479982,1cyclictest17522-21cstates20:55:130
80479982,1cyclictest1398-21snmpd19:22:270
80479982,1cyclictest1397-21snmpd20:15:090
80479982,1cyclictest1397-21snmpd19:45:430
80479981,1cyclictest9-21ksoftirqd/019:16:410
80479980,4cyclictest867-21gdbus20:14:590
80479980,2cyclictest27202-21sensors_volt19:55:230
80679970,6cyclictest5702-21sshd00:00:433
80679970,6cyclictest19673-21sshd22:00:303
80679970,6cyclictest15039-21sshd22:05:343
80609976,0cyclictest0-21swapper/220:08:372
80609976,0cyclictest0-21swapper/220:08:372
80609970,6cyclictest0-21swapper/220:37:122
80559976,0cyclictest0-21swapper/100:30:091
80559975,1cyclictest9054-21ntp_states19:10:181
80559971,1cyclictest131rcu_sched23:01:351
80559970,6cyclictest28298-21kworker/1:223:35:311
80559970,6cyclictest26424-21sshd21:10:221
80559970,6cyclictest26355-21kworker/1:322:15:371
80559970,6cyclictest20798-21kworker/1:421:45:061
80559970,6cyclictest20758-21kworker/1:122:45:101
80559970,6cyclictest16474-21sshd21:41:051
80559970,6cyclictest16355-21sshd22:30:311
80559970,6cyclictest16355-21sshd22:30:311
80559970,6cyclictest1459-21sshd23:30:061
80479970,2cyclictest19593-21hddtemp_smartct21:00:150
80679964,1cyclictest1398-21snmpd19:13:013
80679963,2cyclictest1397-21snmpd21:02:413
80679963,2cyclictest1397-21snmpd20:45:343
80679963,2cyclictest1397-21snmpd20:30:553
80679963,2cyclictest1397-21snmpd19:23:383
80679960,0cyclictest0-21swapper/319:59:353
80609966,0cyclictest1397-21snmpd19:55:452
80609965,0cyclictest0-21swapper/220:50:262
80609965,0cyclictest0-21swapper/219:10:102
80609964,1cyclictest3980-21df20:20:142
80609964,1cyclictest33-21ksoftirqd/219:44:592
80609960,5cyclictest31859-21kworker/2:120:10:432
80609960,5cyclictest0-21swapper/220:48:202
80609960,0cyclictest0-21swapper/219:37:572
80559965,0cyclictest1397-21snmpd20:42:311
80559963,2cyclictest1398-21snmpd20:08:201
80559963,2cyclictest1398-21snmpd20:08:201
80559960,5cyclictest0-21swapper/120:46:121
80559960,5cyclictest0-21swapper/120:20:501
80559960,5cyclictest0-21swapper/119:49:121
80559960,0cyclictest0-21swapper/120:11:481
80679953,1cyclictest854-21ksmtuned19:52:053
80679953,1cyclictest17468-21smicount19:30:243
80679953,1cyclictest1398-21snmpd21:05:043
80679953,1cyclictest1398-21snmpd20:50:363
80679953,1cyclictest1398-21snmpd20:21:303
80679953,1cyclictest1398-21snmpd20:10:233
80679953,1cyclictest1398-21snmpd19:25:583
80679953,1cyclictest1397-21snmpd20:55:113
80679953,1cyclictest1397-21snmpd20:35:043
80679953,1cyclictest1397-21snmpd20:05:543
80679953,1cyclictest1397-21snmpd20:05:543
80679953,1cyclictest1397-21snmpd20:00:113
80679953,1cyclictest1397-21snmpd19:47:303
80679953,1cyclictest1397-21snmpd19:40:023
80679953,1cyclictest1397-21snmpd19:35:353
80679953,1cyclictest1397-21snmpd19:16:433
80679952,2cyclictest2000-21cat20:15:133
80679952,2cyclictest11520-21grep20:40:133
80609953,1cyclictest24360-21cpuspeed19:50:142
80609953,1cyclictest21669-21cat21:05:182
80609953,1cyclictest1398-21snmpd20:32:172
80609953,1cyclictest1398-21snmpd20:15:412
80609953,1cyclictest1398-21snmpd19:25:132
80609953,1cyclictest1397-21snmpd20:01:212
80609953,1cyclictest1397-21snmpd19:30:142
80609952,2cyclictest22694-21sed19:45:182
80609952,2cyclictest12788-21tr19:20:162
80559953,1cyclictest9950-21cat20:35:181
80559953,1cyclictest8000-21cat20:30:171
80559953,1cyclictest28701-21latency20:00:181
80559953,1cyclictest1398-21snmpd20:25:151
80559953,1cyclictest1398-21snmpd19:50:501
80559953,1cyclictest1398-21snmpd19:40:121
80559953,1cyclictest1398-21snmpd19:25:281
80559953,1cyclictest1398-21snmpd19:15:011
80559953,1cyclictest1397-21snmpd21:06:091
80559953,1cyclictest1397-21snmpd20:56:081
80559953,1cyclictest1397-21snmpd20:50:251
80559953,1cyclictest1397-21snmpd19:35:021
80559953,1cyclictest1397-21snmpd19:21:171
80559952,2cyclictest26685-21interrupts19:55:181
80559952,2cyclictest16502-21taskset19:30:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional