You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-27 - 04:46

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack1slot0.osadl.org (updated Mon Jun 27, 2022 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1201921150,1sleep212017-21lspci21:39:242
1201921150,1sleep212017-21lspci21:39:242
63532840,0sleep20-21swapper/223:37:232
153212690,1sleep125-21ksoftirqd/121:27:211
14502670,0sleep20-21swapper/222:55:082
20142660,1sleep22010-21lspci23:49:252
1256226543,7sleep20-21swapper/219:09:032
34052600,2sleep31275699cyclictest23:02:393
116942600,0sleep011685-21sshd22:31:110
41702590,0sleep00-21swapper/000:37:270
1264825949,6sleep10-21swapper/119:09:581
280332580,0sleep20-21swapper/222:47:152
235032580,0sleep20-21swapper/200:11:012
232992580,0sleep00-21swapper/023:23:560
222732580,0sleep322272-21sshd21:12:493
165852580,0sleep00-21swapper/000:13:030
75912570,0sleep00-21swapper/021:46:230
309892570,0sleep10-21swapper/100:04:061
240452570,0sleep30-21swapper/322:46:563
37502560,0sleep00-21swapper/023:27:190
283552560,0sleep028350-21sshd21:10:060
235262560,0sleep20-21swapper/221:23:012
115672560,0sleep00-21swapper/000:02:330
206942540,0sleep20-21swapper/222:31:532
98302500,0sleep10-21swapper/100:35:191
1252724938,7sleep00-21swapper/019:08:410
168022480,2sleep31275699cyclictest23:20:533
1236024837,7sleep30-21swapper/319:06:463
314032430,0sleep231401-21lspci23:26:572
9362360,1sleep1937-21tty23:49:201
272752240,0sleep10-21swapper/123:51:161
236262240,0sleep023624-21awk21:00:010
65752230,0sleep00-21swapper/022:01:060
212492210,0sleep20-21swapper/223:43:282
153262210,0sleep3391rcuc/322:41:193
54142200,0sleep00-21swapper/022:15:520
54142200,0sleep00-21swapper/022:15:520
36112200,0sleep00-21swapper/021:18:550
166912200,0sleep30-21swapper/321:39:453
166912200,0sleep30-21swapper/321:39:453
61742190,0sleep16176-21bash21:58:381
318122190,0sleep00-21swapper/019:55:160
40862180,0sleep30-21swapper/300:19:433
40722180,0sleep20-21swapper/221:55:592
313172180,0sleep30-21swapper/322:03:013
277392180,2sleep31275699cyclictest23:29:113
298292170,0sleep00-21swapper/023:16:580
298292170,0sleep00-21swapper/023:16:580
212422170,0sleep10-21swapper/123:11:221
151262170,2sleep31275699cyclictest23:33:073
315932160,0sleep331594-21grep00:34:333
266202160,1sleep2321ktimersoftd/223:11:472
234252160,1sleep0111rcuc/022:09:550
151562160,0sleep30-21swapper/321:59:213
31652150,0sleep00-21swapper/000:09:310
279172150,0sleep11-21systemd21:10:021
12743991513,1cyclictest20552-21lspci22:14:381
109392150,0sleep20-21swapper/222:21:152
232542130,2sleep01273799cyclictest00:18:400
12756991311,1cyclictest41-21ksoftirqd/323:00:003
12750991312,0cyclictest5536-21lspci21:26:362
1275099130,12cyclictest867-21gdbus23:03:352
1274399130,1cyclictest29860-21sshd22:57:151
1273799130,12cyclictest29679-21bash21:25:580
98822120,0sleep19835-21sshd22:21:101
12756991210,1cyclictest9012-21sshd22:33:293
1275699120,1cyclictest820-21dbus-daemon00:06:393
1275699120,11cyclictest12887-21bash00:35:343
1275699120,11cyclictest11268-21grep21:16:573
12750991210,1cyclictest31689-21lspci22:42:362
1275099120,11cyclictest1-21systemd22:52:282
1275099120,11cyclictest0-21swapper/222:01:262
12743991210,1cyclictest1-21systemd22:00:151
1273799120,1cyclictest21562-21packagekitd23:40:180
1273799120,1cyclictest13277-21bash22:58:290
1273799120,1cyclictest1-21systemd23:34:200
1273799120,11cyclictest19542-21bash21:30:100
254972110,0sleep20-21swapper/221:45:172
201542110,1sleep120152-21sshd21:49:531
1275699119,1cyclictest24626-21bash21:52:383
1275699111,0cyclictest131rcu_sched23:06:503
12756991110,0cyclictest0-21swapper/323:10:573
12756991110,0cyclictest0-21swapper/300:23:263
1275699110,10cyclictest0-21swapper/322:37:223
1275699110,10cyclictest0-21swapper/322:10:043
1275699110,0cyclictest24143-21lspci21:32:553
1275099119,1cyclictest6853-21latency_hist00:25:002
1275099119,1cyclictest33-21ksoftirqd/221:50:152
1275099119,1cyclictest0-21swapper/223:52:112
1275099119,1cyclictest0-21swapper/222:14:302
1275099113,1cyclictest33-21ksoftirqd/223:22:032
1275099111,1cyclictest131rcu_sched22:06:592
12750991110,0cyclictest30547-21lspci23:19:332
12750991110,0cyclictest30547-21lspci23:19:332
12750991110,0cyclictest0-21swapper/223:06:392
12750991110,0cyclictest0-21swapper/200:04:372
1275099110,10cyclictest0-21swapper/220:05:192
1274399119,1cyclictest31777-21sshd21:31:021
1274399112,1cyclictest25-21ksoftirqd/122:26:371
12743991110,0cyclictest0-21swapper/100:08:091
1274399110,1cyclictest11621-21lspci22:33:411
1274399110,10cyclictest2226-21lspci21:41:031
1274399110,10cyclictest0-21swapper/122:42:431
1273799119,1cyclictest820-21dbus-daemon21:54:330
1273799119,1cyclictest7964-21sshd21:21:460
1273799113,4cyclictest19828-21munin-run21:40:000
1273799113,4cyclictest19828-21munin-run21:40:000
1273799110,10cyclictest0-21swapper/022:25:520
302702100,0sleep10-21swapper/121:38:191
302702100,0sleep10-21swapper/121:38:191
1275699109,0cyclictest0-21swapper/323:15:183
1275699109,0cyclictest0-21swapper/323:15:183
1275699109,0cyclictest0-21swapper/300:26:023
1275699100,9cyclictest29993-21sshd22:25:123
1275699100,9cyclictest0-21swapper/322:20:053
1275099109,0cyclictest0-21swapper/200:36:062
1275099109,0cyclictest0-21swapper/200:30:272
1275099108,1cyclictest33-21ksoftirqd/200:09:532
1275099105,2cyclictest33-21ksoftirqd/223:30:312
1275099101,0cyclictest131rcu_sched00:28:592
1275099100,9cyclictest0-21swapper/219:56:472
1275099100,1cyclictest131rcu_sched22:37:152
1275099100,1cyclictest131rcu_sched21:13:212
1274399109,0cyclictest0-21swapper/123:03:431
1274399108,1cyclictest0-21swapper/123:25:501
1274399100,9cyclictest7482-21sshd23:57:131
1274399100,1cyclictest801-21systemd-logind22:17:341
1274399100,1cyclictest801-21systemd-logind22:17:341
1274399100,1cyclictest131rcu_sched00:22:471
1274399100,0cyclictest7747-21sshd21:19:151
1274399100,0cyclictest131rcu_sched23:31:201
1274399100,0cyclictest0-21swapper/121:05:021
1273799109,0cyclictest0-21swapper/023:14:090
1273799109,0cyclictest0-21swapper/020:10:250
1273799108,1cyclictest21524-21lspci22:24:350
1273799108,1cyclictest16133-21grepconf.sh23:35:370
1273799100,9cyclictest0-21swapper/019:52:290
1273799100,0cyclictest0-21swapper/023:50:580
1273799100,0cyclictest0-21swapper/023:46:170
1273799100,0cyclictest0-21swapper/022:46:320
127569998,0cyclictest0-21swapper/323:49:323
127569998,0cyclictest0-21swapper/323:38:523
127569998,0cyclictest0-21swapper/321:45:033
127569997,1cyclictest41-21ksoftirqd/321:21:563
127569990,8cyclictest25884-21kworker/3:322:05:233
127509998,0cyclictest18461-21lspci21:17:312
127509998,0cyclictest0-21swapper/221:40:552
127509998,0cyclictest0-21swapper/220:24:532
127509991,0cyclictest131rcu_sched22:16:272
127509991,0cyclictest131rcu_sched22:16:272
127509990,8cyclictest13781-21kworker/2:323:58:562
127509990,8cyclictest13335-21kworker/2:022:25:562
127509990,0cyclictest0-21swapper/220:13:542
127439998,0cyclictest0-21swapper/123:15:301
127439998,0cyclictest0-21swapper/123:15:301
127439998,0cyclictest0-21swapper/119:54:391
127439997,1cyclictest25492-21sshd23:06:471
127439997,1cyclictest25-21ksoftirqd/122:51:591
127439997,1cyclictest25-21ksoftirqd/121:20:201
127439997,1cyclictest25-21ksoftirqd/100:14:121
127439991,1cyclictest131rcu_sched22:45:341
127439990,8cyclictest30268-21kworker/1:000:28:021
127439990,8cyclictest2867-21kworker/1:423:22:391
127379998,0cyclictest0-21swapper/022:11:590
127379992,1cyclictest121rcu_preempt22:53:340
127569987,0cyclictest0-21swapper/321:06:573
127569987,0cyclictest0-21swapper/300:10:313
127569986,1cyclictest41-21ksoftirqd/323:41:323
127569986,1cyclictest41-21ksoftirqd/322:15:563
127569986,1cyclictest41-21ksoftirqd/322:15:563
127569986,1cyclictest20957-21lspci21:40:053
127569983,2cyclictest820-21dbus-daemon22:50:013
127569980,7cyclictest22446-21sshd23:50:553
127569980,7cyclictest11381-21sshd23:55:043
127509986,1cyclictest33-21ksoftirqd/221:30:162
127509986,1cyclictest33-21ksoftirqd/200:16:352
127439987,0cyclictest0-21swapper/119:30:161
127439987,0cyclictest0-21swapper/119:10:331
127439987,0cyclictest0-21swapper/100:30:051
127439986,1cyclictest25-21ksoftirqd/122:05:101
127439986,1cyclictest25-21ksoftirqd/100:15:571
127439980,7cyclictest29551-21kworker/1:422:35:141
127439980,7cyclictest12460-21sshd23:35:211
127439980,0cyclictest131rcu_sched23:40:171
127379987,0cyclictest0-21swapper/023:09:470
127379987,0cyclictest0-21swapper/021:03:220
127379986,1cyclictest23101-21lspci21:55:020
127379986,1cyclictest17320-21lspci00:33:260
127379986,1cyclictest13581-21lspci23:00:560
127379980,7cyclictest4053-21sshd22:42:580
127379980,7cyclictest28303-21sshd00:24:080
127379980,7cyclictest24813-21sshd00:26:210
127569975,1cyclictest41-21ksoftirqd/300:00:323
127569970,6cyclictest30139-21sshd21:26:013
127569970,6cyclictest23322-21sshd20:56:543
127509976,0cyclictest0-21swapper/219:39:212
127509970,6cyclictest0-21swapper/220:36:302
127509970,0cyclictest0-21swapper/221:05:432
127509970,0cyclictest0-21swapper/220:59:552
127439976,0cyclictest0-21swapper/120:50:101
127439976,0cyclictest0-21swapper/120:46:281
127439975,1cyclictest25-21ksoftirqd/121:50:321
127439970,6cyclictest0-21swapper/120:30:391
127439970,6cyclictest0-21swapper/119:18:191
127379976,0cyclictest0-21swapper/019:19:330
127379970,6cyclictest24693-21sshd21:40:180
127379970,6cyclictest23781-21sshd22:37:030
127379970,6cyclictest16707-21sshd23:55:250
127379970,6cyclictest0-21swapper/020:35:450
127379970,6cyclictest0-21swapper/019:38:060
127569965,0cyclictest0-21swapper/321:00:193
127569965,0cyclictest0-21swapper/319:56:523
127569965,0cyclictest0-21swapper/319:18:363
127569963,2cyclictest21343-21cpuspeed_turbos19:30:113
127569963,2cyclictest1398-21snmpd20:17:343
127569963,2cyclictest1397-21snmpd20:33:273
127569963,2cyclictest1397-21snmpd20:07:333
127569963,2cyclictest1397-21snmpd19:47:413
127569963,2cyclictest1397-21snmpd19:23:313
127569963,2cyclictest1397-21snmpd19:11:043
127569960,5cyclictest20591-21sshd19:26:273
127569960,5cyclictest18793-21sshd20:45:203
127569960,5cyclictest0-21swapper/320:27:423
127569960,5cyclictest0-21swapper/319:35:343
127509963,2cyclictest1398-21snmpd20:44:202
127509963,2cyclictest1398-21snmpd20:32:492
127509963,2cyclictest1398-21snmpd19:50:032
127509963,2cyclictest1397-21snmpd20:17:292
127509960,5cyclictest0-21swapper/219:13:302
127439963,2cyclictest1398-21snmpd20:44:021
127439960,5cyclictest0-21swapper/119:23:181
127379965,0cyclictest0-21swapper/021:06:160
127379965,0cyclictest0-21swapper/019:21:290
127379963,2cyclictest1663-21sed20:00:190
127379963,2cyclictest1398-21snmpd20:27:190
127379963,2cyclictest1397-21snmpd20:51:090
127379963,2cyclictest1397-21snmpd19:12:300
127379960,0cyclictest0-21swapper/019:32:180
127569953,1cyclictest29334-21cat19:50:133
127569953,1cyclictest1398-21snmpd20:35:033
127569953,1cyclictest1398-21snmpd20:10:193
127569953,1cyclictest1397-21snmpd20:22:183
127569953,1cyclictest1397-21snmpd19:40:373
127569953,1cyclictest1219-21runrttasks20:50:223
127569953,1cyclictest1219-21runrttasks20:00:193
127569952,2cyclictest16208-21grep20:40:113
127509953,1cyclictest25453-21cat19:40:122
127509953,1cyclictest21986-21open_files19:30:172
127509953,1cyclictest17473-21cstates19:20:112
127509953,1cyclictest15734-21if_eno219:15:142
127509953,1cyclictest1398-21snmpd21:01:172
127509953,1cyclictest1398-21snmpd19:25:062
127509953,1cyclictest1397-21snmpd20:50:072
127509953,1cyclictest1397-21snmpd20:45:252
127509953,1cyclictest1397-21snmpd19:45:052
127509953,1cyclictest10648-21cat20:25:132
127509952,2cyclictest601-21cat20:00:102
127439954,0cyclictest0-21swapper/119:25:161
127439953,1cyclictest8707-21cat20:20:161
127439953,1cyclictest23545-21cat19:35:131
127439953,1cyclictest14510-21cat20:35:131
127439953,1cyclictest1398-21snmpd20:05:301
127439953,1cyclictest1398-21snmpd19:45:141
127439953,1cyclictest1397-21snmpd21:00:061
127439953,1cyclictest1397-21snmpd19:58:311
127439953,1cyclictest1397-21snmpd19:40:111
127439953,1cyclictest1219-21runrttasks20:15:321
127439953,1cyclictest1103-21cut20:00:151
127439953,1cyclictest10633-21cat20:25:141
127439952,2cyclictest1398-21snmpd20:11:011
127439952,2cyclictest1397-21snmpd20:55:071
127379953,1cyclictest6085-21kworker/0:220:15:170
127379953,1cyclictest3523-21sshd19:25:210
127379953,1cyclictest27413-21cat19:45:130
127379953,1cyclictest18386-21cat20:45:150
127379953,1cyclictest17331-21sshd20:40:260
127379953,1cyclictest1398-21snmpd20:20:100
127379953,1cyclictest1398-21snmpd20:08:160
127379953,1cyclictest1397-21snmpd20:30:030
127379953,1cyclictest1397-21snmpd19:40:530
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional