You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-07 - 15:50

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #4, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack4slot0.osadl.org (updated Thu Jul 07, 2022 12:44:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3066721510,0sleep00-21swapper/011:11:580
1376021430,0sleep10-21swapper/109:46:111
2123321180,0sleep00-21swapper/011:25:250
27628998885,1cyclictest24917-21ipmitool09:40:130
274992870,30sleep3102550irq/24-eno111:37:143
141452840,1sleep014146-21bash10:59:110
27644998381,1cyclictest6775-21ipmitool10:00:152
27644998381,1cyclictest6775-21ipmitool10:00:142
230422810,0sleep30-21swapper/310:34:453
247282780,0sleep10-21swapper/110:42:131
226922780,0sleep00-21swapper/011:44:010
173942760,0sleep00-21swapper/010:01:240
173942760,0sleep00-21swapper/010:01:240
299562740,0sleep329952-21lspci10:17:193
289782730,0sleep10-21swapper/112:13:391
212762730,0sleep00-21swapper/009:25:200
214262720,0sleep2311rcuc/209:58:132
214262720,0sleep2311rcuc/209:58:132
27628997169,1cyclictest19308-21ipmitool10:05:150
39612690,0sleep1231rcuc/112:36:121
20572690,3sleep22764499cyclictest09:23:142
72062680,0sleep10-21swapper/111:45:541
289022680,0sleep30-21swapper/311:15:223
204852680,0sleep00-21swapper/009:36:030
188002670,0sleep30-21swapper/311:03:193
17262660,0sleep00-21swapper/011:38:000
48442640,3sleep22764499cyclictest11:09:042
29032640,0sleep22904-21grepconf.sh11:52:442
297072630,1sleep329705-21sshd12:17:183
27644996158,1cyclictest16012-21ipmitool09:50:192
238712610,0sleep125838-21nfsd12:27:341
277742590,0sleep20-21swapper/212:17:052
277912580,1sleep12763699cyclictest10:46:111
2689525727,9sleep10-21swapper/107:05:221
27628995553,1cyclictest5248-21ipmitool09:45:130
188942540,0sleep30-21swapper/310:56:063
2701525126,18sleep00-21swapper/007:05:330
215722490,2sleep12763699cyclictest10:12:401
2729024529,9sleep20-21swapper/207:08:442
2717024326,10sleep30-21swapper/307:07:213
27644993230,1cyclictest19331-21ipmitool10:05:142
161412290,0sleep00-21swapper/012:30:190
127122270,1sleep125-21ksoftirqd/110:22:411
2764499268,2cyclictest593-21systemd-journal09:25:322
8612250,1sleep1860-21sshd11:23:101
302522250,0sleep30-21swapper/311:08:163
216192250,1sleep321620-21sshd09:18:113
174812250,1sleep217484-21grepconf.sh11:03:102
2762899244,2cyclictest24815-21lspci09:18:320
187492240,0sleep10-21swapper/111:06:591
98742230,0sleep20-21swapper/210:11:202
80882230,1sleep3391rcuc/307:40:103
296882230,1sleep10-21swapper/110:50:071
2764499233,2cyclictest0-21swapper/210:18:262
2764499230,4cyclictest1428-21sshd12:39:352
273342230,0sleep00-21swapper/011:07:570
269382230,0sleep20-21swapper/210:38:492
209682230,1sleep233-21ksoftirqd/209:43:222
188592230,0sleep00-21swapper/012:08:580
2764499220,4cyclictest0-21swapper/211:16:232
2764499220,4cyclictest0-21swapper/211:11:052
2764499220,4cyclictest0-21swapper/208:58:252
2764499220,4cyclictest0-21swapper/208:58:252
2764499220,4cyclictest0-21swapper/208:54:312
2764499220,4cyclictest0-21swapper/207:12:342
2763699220,4cyclictest27456-21sshd09:51:331
2762899220,4cyclictest0-21swapper/009:07:210
2764499210,3cyclictest21544-21sshd11:40:152
2764499210,3cyclictest0-21swapper/211:45:462
2764499210,3cyclictest0-21swapper/208:06:572
2763699210,3cyclictest0-21swapper/110:58:221
2763699210,3cyclictest0-21swapper/110:39:431
2763699210,1cyclictest0-21swapper/109:11:081
2762899212,1cyclictest0-21swapper/008:40:430
2762899210,3cyclictest31827-21runrttasks10:53:580
2762899210,3cyclictest0-21swapper/012:11:290
2762899210,2cyclictest2851-21sshd10:32:280
2762899210,2cyclictest0-21swapper/011:34:240
2762899210,1cyclictest0-21swapper/008:46:190
259072210,0sleep00-21swapper/010:35:070
185782210,0sleep30-21swapper/310:45:123
41132200,0sleep20-21swapper/209:37:532
2764499203,2cyclictest0-21swapper/208:46:162
2764499202,2cyclictest0-21swapper/208:42:502
2764499202,0cyclictest0-21swapper/207:49:182
2764499200,2cyclictest28490-21lspci10:24:282
2764499200,2cyclictest0-21swapper/208:00:122
2764499200,2cyclictest0-21swapper/207:36:472
2762899203,2cyclictest0-21swapper/007:54:140
2762899202,2cyclictest2500-21sshd12:25:130
2762899200,2cyclictest0-21swapper/010:41:070
2762899200,2cyclictest0-21swapper/009:04:410
2762899200,2cyclictest0-21swapper/008:36:350
2762899200,2cyclictest0-21swapper/008:06:330
2762899200,2cyclictest0-21swapper/007:45:140
253652200,1sleep125363-21sshd12:31:201
2765099190,17cyclictest23166-21sshd11:29:243
2764499192,2cyclictest23950-21bash12:20:192
2764499190,4cyclictest0-21swapper/211:58:572
2764499190,4cyclictest0-21swapper/209:05:352
2764499190,3cyclictest0-21swapper/208:34:462
2764499190,1cyclictest0-21swapper/212:00:402
2764499190,1cyclictest0-21swapper/210:58:172
2764499190,1cyclictest0-21swapper/210:51:262
2764499190,1cyclictest0-21swapper/210:42:552
2764499190,1cyclictest0-21swapper/210:30:592
2764499190,1cyclictest0-21swapper/210:28:432
2764499190,1cyclictest0-21swapper/209:04:172
2764499190,1cyclictest0-21swapper/208:39:252
2764499190,1cyclictest0-21swapper/208:22:162
2764499190,1cyclictest0-21swapper/207:57:392
2764499190,1cyclictest0-21swapper/207:54:032
2764499190,1cyclictest0-21swapper/207:44:572
2764499190,1cyclictest0-21swapper/207:29:142
2764499190,1cyclictest0-21swapper/207:29:142
2764499190,1cyclictest0-21swapper/207:23:562
2763699190,4cyclictest27976-21sshd10:31:401
2763699190,4cyclictest0-21swapper/110:07:511
2763699190,4cyclictest0-21swapper/108:52:521
2763699190,4cyclictest0-21swapper/108:12:571
2763699190,4cyclictest0-21swapper/107:24:051
2763699190,3cyclictest0-21swapper/112:18:381
2763699190,1cyclictest0-21swapper/112:22:131
2763699190,1cyclictest0-21swapper/111:00:151
2763699190,1cyclictest0-21swapper/109:42:331
2763699190,1cyclictest0-21swapper/108:59:561
2763699190,1cyclictest0-21swapper/108:59:551
2763699190,1cyclictest0-21swapper/108:33:101
2763699190,1cyclictest0-21swapper/108:17:361
2763699190,1cyclictest0-21swapper/108:06:061
2763699190,1cyclictest0-21swapper/107:44:051
2763699190,1cyclictest0-21swapper/107:31:471
2763699190,1cyclictest0-21swapper/107:28:591
2763699190,1cyclictest0-21swapper/107:28:591
2762899192,2cyclictest5073-21lspci12:39:580
2762899192,1cyclictest0-21swapper/011:55:140
2762899190,1cyclictest26098-21sshd11:48:030
2762899190,1cyclictest19607-21sshd11:21:360
2762899190,1cyclictest0-21swapper/012:20:550
2762899190,1cyclictest0-21swapper/011:52:360
2762899190,1cyclictest0-21swapper/011:01:030
2762899190,1cyclictest0-21swapper/010:12:360
2762899190,1cyclictest0-21swapper/009:30:020
2762899190,1cyclictest0-21swapper/009:12:290
2762899190,1cyclictest0-21swapper/008:30:190
2762899190,1cyclictest0-21swapper/008:15:140
2762899190,1cyclictest0-21swapper/008:12:570
2762899190,1cyclictest0-21swapper/008:00:370
2762899190,1cyclictest0-21swapper/007:58:360
2762899190,1cyclictest0-21swapper/007:43:280
2762899190,1cyclictest0-21swapper/007:37:480
2762899190,1cyclictest0-21swapper/007:32:170
2762899190,1cyclictest0-21swapper/007:17:350
2764499184,1cyclictest33-21ksoftirqd/211:30:012
2764499182,1cyclictest0-21swapper/212:34:082
2764499180,3cyclictest0-21swapper/207:18:432
2764499180,2cyclictest3212-21bash11:30:442
2764499180,1cyclictest0-21swapper/210:49:222
2764499180,1cyclictest0-21swapper/209:45:122
2764499180,1cyclictest0-21swapper/209:10:272
2764499180,0cyclictest0-21swapper/212:28:102
2764499180,0cyclictest0-21swapper/209:18:202
2763699180,3cyclictest15946-21lspci11:43:161
2763699180,3cyclictest0-21swapper/109:03:291
2763699180,3cyclictest0-21swapper/108:38:011
2763699180,3cyclictest0-21swapper/108:01:001
2763699180,3cyclictest0-21swapper/107:13:381
2763699180,2cyclictest0-21swapper/111:10:321
2763699180,1cyclictest0-21swapper/112:07:431
2763699180,1cyclictest0-21swapper/109:19:561
2762899180,3cyclictest0-21swapper/007:10:070
2762899180,2cyclictest12347-1kworker/0:2H10:30:010
2762899180,1cyclictest0-21swapper/010:48:250
2765099175,2cyclictest29870-21sshd11:11:543
2765099175,10cyclictest23250-21bash11:44:043
2764499170,2cyclictest0-21swapper/211:23:052
2764499170,2cyclictest0-21swapper/207:34:062
2763699170,2cyclictest0-21swapper/111:31:351
2763699170,1cyclictest0-21swapper/108:21:391
2763699170,16cyclictest8066-21sshd11:16:401
2762899173,1cyclictest0-21swapper/007:29:020
2762899173,1cyclictest0-21swapper/007:29:020
2762899170,1cyclictest0-21swapper/008:50:010
2762899170,11cyclictest27195-21sshd10:24:190
2765099164,10cyclictest7545-21sshd10:29:223
2764499160,1cyclictest0-21swapper/208:25:102
2764499160,15cyclictest12589-21kworker/2:012:07:152
2763699163,10cyclictest0-21swapper/109:28:261
2763699160,4cyclictest10468-21lspci10:04:151
2763699160,4cyclictest10468-21lspci10:04:151
2763699160,3cyclictest9797-21sshd09:38:331
2763699160,3cyclictest22316-21sshd11:25:321
2763699160,1cyclictest25839-21nfsd09:56:181
2763699160,1cyclictest25839-21nfsd09:56:181
2763699160,1cyclictest0-21swapper/111:57:241
2763699160,1cyclictest0-21swapper/108:49:281
2763699160,1cyclictest0-21swapper/107:48:061
2762899160,1cyclictest0-21swapper/012:16:310
2762899160,1cyclictest0-21swapper/008:56:250
2762899160,1cyclictest0-21swapper/008:56:250
2762899160,1cyclictest0-21swapper/008:20:520
2765099154,10cyclictest102550irq/24-eno110:50:143
27650991513,1cyclictest7677-21bash11:45:593
27644991513,1cyclictest33-21ksoftirqd/209:35:012
27644991512,2cyclictest0-21swapper/212:13:282
27636991513,1cyclictest0-21swapper/112:01:051
2763699150,14cyclictest0-21swapper/108:40:131
2763699150,0cyclictest0-21swapper/109:30:201
2763699150,0cyclictest0-21swapper/109:21:411
27628991512,2cyclictest31790-21nscd09:24:050
2762899150,1cyclictest0-21swapper/011:15:350
2762899150,1cyclictest0-21swapper/009:58:250
2762899150,1cyclictest0-21swapper/009:58:240
2765099149,4cyclictest0-21swapper/310:10:103
2765099148,5cyclictest2650-21cp12:36:043
2765099144,1cyclictest23236-21sshd09:29:103
2765099142,2cyclictest7166-21bash12:14:513
27650991412,1cyclictest0-21swapper/308:04:333
2765099140,2cyclictest8024-21lspci11:31:203
2765099140,10cyclictest26501-21sshd12:31:273
2765099140,10cyclictest15769-21sshd09:53:533
27636991410,3cyclictest3172-21sshd10:25:161
2763699140,0cyclictest0-21swapper/107:50:061
97032130,0sleep00-21swapper/012:04:210
2765099133,9cyclictest21231-21getsmi08:10:193
2765099132,10cyclictest29222-21grepconf.sh12:24:363
2765099132,10cyclictest16542-21sshd09:46:293
27650991311,1cyclictest401ktimersoftd/307:15:113
27650991311,1cyclictest31943-21bash12:06:473
27650991311,1cyclictest25193-21bash10:20:253
27650991311,1cyclictest14555-21sshd09:39:053
27650991311,1cyclictest0-21swapper/310:01:093
27650991311,1cyclictest0-21swapper/310:01:083
2765099130,9cyclictest23050-21sshd11:58:353
2765099130,9cyclictest20538-21smart_sda10:05:193
2765099130,3cyclictest0-21swapper/312:25:153
2765099130,1cyclictest593-21systemd-journal09:30:123
2765099130,1cyclictest0-21swapper/310:36:123
2765099130,1cyclictest0-21swapper/309:14:563
2765099130,12cyclictest593-21systemd-journal09:56:203
2765099130,12cyclictest593-21systemd-journal09:56:203
2765099130,12cyclictest3693-21bash12:00:023
2765099130,12cyclictest0-21swapper/311:20:133
2765099130,10cyclictest593-21systemd-journal11:51:343
2765099130,10cyclictest15192-21rs:main3
2764499131,2cyclictest23138-21sshd11:36:442
2763699130,1cyclictest21336-21lspci11:54:481
2763699130,1cyclictest0-21swapper/109:07:141
27628991311,1cyclictest16244-21sshd09:53:470
2762899130,1cyclictest0-21swapper/010:15:060
2762899130,1cyclictest0-21swapper/007:23:170
2765099129,2cyclictest0-21swapper/307:10:113
27650991210,1cyclictest0-21swapper/308:30:243
2765099120,9cyclictest0-21swapper/307:30:273
2765099120,1cyclictest0-21swapper/308:05:093
2765099120,10cyclictest6443-21kernelversion07:35:153
2765099120,10cyclictest593-21systemd-journal09:20:023
2765099120,10cyclictest5676-21sshd10:40:043
2765099120,10cyclictest32644-21idleruntime07:20:153
2765099120,10cyclictest25840-21nfsd08:25:223
2765099120,10cyclictest25248-21awk08:20:193
2765099120,10cyclictest22608-21hddtemp_smartct08:15:123
2765099120,10cyclictest0-21swapper/309:00:173
2765099120,10cyclictest0-21swapper/308:55:213
2765099120,10cyclictest0-21swapper/308:55:203
2764499120,0cyclictest0-21swapper/208:10:132
2763699120,3cyclictest0-21swapper/107:39:301
2763699120,2cyclictest22147-21sshd10:16:211
2763699120,1cyclictest0-21swapper/107:55:421
2763699120,10cyclictest0-21swapper/111:35:021
2762899120,0cyclictest0-21swapper/008:26:070
2765099118,2cyclictest11985-21latency_hist07:50:003
2765099110,9cyclictest0-21swapper/309:05:133
2765099110,2cyclictest2024-21cpuspeed_turbos07:25:103
2765099110,2cyclictest2024-21cpuspeed_turbos07:25:103
2765099110,10cyclictest4460-21df_inode08:50:123
2765099110,10cyclictest2031-21latency_hist08:45:003
2765099110,10cyclictest15191-21in:imjournal08:35:003
2765099110,10cyclictest0-21swapper/308:45:163
2763699110,1cyclictest0-21swapper/108:25:041
2763699110,10cyclictest30171-21cstates07:15:021
2765099100,1cyclictest12402-21diskstats07:50:143
2765099100,1cyclictest0-21swapper/307:55:143
2764499100,9cyclictest0-21swapper/208:15:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional