You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-28 - 07:28

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #a, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rackaslot0.osadl.org (updated Tue Mar 28, 2023 00:44:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
132342830,0sleep00-21swapper/022:00:520
40992640,1sleep14096-21sshd21:10:011
62662620,1sleep16261-21sshd00:04:191
23182600,1sleep30-21swapper/322:39:583
23182600,1sleep30-21swapper/322:39:573
189392590,0sleep118913-21sshd00:26:161
192752580,1sleep010-21rcuc/000:22:570
213932560,1sleep121385-21sshd21:24:521
197552560,1sleep319753-21bash21:34:313
79752540,0sleep07976-21bash23:32:090
66312530,1sleep26629-21sshd00:07:492
325522520,1sleep332550-21sshd21:35:393
83612510,0sleep30-21swapper/321:40:043
155022510,0sleep3381ktimersoftd/321:47:293
1076325124,9sleep30-21swapper/319:09:283
103882510,0sleep00-21swapper/000:01:150
163212500,0sleep31718-21runrttasks20:25:223
1066925024,8sleep00-21swapper/019:08:180
1018825026,19sleep10-21swapper/119:05:261
877924926,9sleep20-21swapper/219:05:142
324122490,0sleep30-21swapper/300:31:053
181862490,0sleep30-21swapper/321:27:463
28502480,0sleep02851-21sshd23:38:140
251222480,0sleep10-21swapper/121:55:201
19202480,0sleep30-21swapper/321:16:123
227762460,0sleep20-21swapper/222:38:442
227762460,0sleep20-21swapper/222:38:442
8742450,0sleep21718-21runrttasks22:16:322
292352450,0sleep00-21swapper/000:17:040
183922420,1sleep318393-21bash22:01:273
96602410,0sleep20-21swapper/223:12:492
11068992220,1cyclictest0-21swapper/300:06:163
1106799222,1cyclictest14523-21cat00:15:302
11066992220,1cyclictest30917-21sshd00:20:391
11067992119,1cyclictest0-21swapper/221:26:572
1106599205,13cyclictest0-21swapper/021:16:530
11065992010,5cyclictest4999-21sshd21:23:090
230432190,1sleep122997-21sshd22:15:231
1106599199,5cyclictest27523-21cp23:04:530
1106599199,5cyclictest27523-21cp23:04:530
1106599195,8cyclictest13947-21sshd21:44:010
11065991913,4cyclictest0-21swapper/021:49:250
1106599190,13cyclictest21481-21sshd21:51:290
114652180,0sleep00-21swapper/022:47:150
1106599183,3cyclictest0-21swapper/000:31:560
1106599180,13cyclictest0-21swapper/000:37:060
1106599180,10cyclictest14523-21sshd21:10:570
1106599180,10cyclictest0-21swapper/023:55:080
28722170,1sleep22871-21bash22:13:242
246872170,1sleep224690-21bash21:21:482
22292170,1sleep00-21swapper/021:59:520
1106599179,5cyclictest20648-21sshd21:37:530
1106599179,4cyclictest0-21swapper/023:23:510
1106599177,8cyclictest0-21swapper/022:09:570
1106599176,4cyclictest0-21swapper/023:09:380
1106599173,4cyclictest18498-21sshd00:29:440
11065991716,1cyclictest18103-21sshd00:12:280
11065991714,2cyclictest30539-21munin-plugin-st19:49:590
11065991714,2cyclictest2705-21bash22:10:030
1106599170,12cyclictest9621-21sshd22:27:360
261172160,0sleep126104-21sshd23:30:381
126572160,0sleep20-21swapper/223:32:412
11068991612,1cyclictest39-21ksoftirqd/323:43:063
11067991614,1cyclictest0-21swapper/221:51:132
1106599169,3cyclictest0-21swapper/023:42:240
1106599165,8cyclictest20228-21tune2fs20:35:170
1106599165,5cyclictest25832-21sshd22:35:410
1106599165,5cyclictest25832-21sshd22:35:410
11065991614,2cyclictest25735-21bash21:28:350
11065991611,4cyclictest17210-21bash22:21:420
1106599161,11cyclictest0-21swapper/022:18:190
1106599160,15cyclictest0-21swapper/019:55:130
128942150,1sleep10-21swapper/122:47:241
11068991513,1cyclictest39-21ksoftirqd/323:55:133
11067991514,1cyclictest0-21swapper/222:34:112
1106699159,2cyclictest0-21swapper/123:57:551
1106699150,3cyclictest0-21swapper/121:44:091
1106699150,2cyclictest0-21swapper/122:04:031
1106599152,7cyclictest30108-21ntp_states22:55:290
1106599151,5cyclictest3789-21diskmemload22:43:410
1106599151,10cyclictest7121-21sshd23:45:030
1106599150,8cyclictest0-21swapper/000:06:500
1106599150,11cyclictest26748-21sshd22:51:590
1106599150,11cyclictest0-21swapper/023:10:100
1106599150,11cyclictest0-21swapper/021:00:130
1106599150,10cyclictest0-21swapper/023:54:010
67092140,1sleep36708-21bash21:56:463
62482140,0sleep10-21swapper/122:20:281
52332140,0sleep25230-21bash22:53:062
287452140,1sleep10-21swapper/122:29:371
1106899149,1cyclictest39-21ksoftirqd/300:38:353
1106899148,2cyclictest22217-21sshd23:27:003
1106899144,9cyclictest39-21ksoftirqd/323:50:093
1106899144,9cyclictest39-21ksoftirqd/322:58:243
1106899144,1cyclictest39-21ksoftirqd/323:15:553
1106899144,1cyclictest39-21ksoftirqd/322:24:553
1106899142,9cyclictest39-21ksoftirqd/323:20:273
1106899141,2cyclictest0-21swapper/323:37:393
11068991412,1cyclictest39-21ksoftirqd/323:03:513
11068991412,1cyclictest39-21ksoftirqd/323:03:513
11068991412,1cyclictest39-21ksoftirqd/322:51:513
11068991412,1cyclictest39-21ksoftirqd/322:34:253
11068991412,1cyclictest39-21ksoftirqd/321:22:123
11068991412,1cyclictest39-21ksoftirqd/321:11:143
11068991412,1cyclictest39-21ksoftirqd/300:02:143
11068991412,1cyclictest20356-21sshd00:23:043
11068991410,3cyclictest39-21ksoftirqd/321:51:593
11068991410,1cyclictest39-21ksoftirqd/300:27:283
1106899140,13cyclictest0-21swapper/323:45:313
11067991412,1cyclictest10026-21sshd21:57:102
11067991410,2cyclictest0-21swapper/200:11:332
1106799140,11cyclictest0-21swapper/223:28:012
1106699149,1cyclictest114750irq/25-eth022:08:491
1106699149,1cyclictest114750irq/25-eth000:11:461
1106699149,1cyclictest0-21swapper/123:29:411
1106699149,1cyclictest0-21swapper/122:54:471
1106699148,2cyclictest23063-21sshd23:07:431
1106699148,2cyclictest1579-21sshd00:07:561
1106699148,2cyclictest1221-21sshd23:50:411
1106699148,1cyclictest0-21swapper/121:30:521
1106699141,1cyclictest114750irq/25-eth023:38:221
1106699141,1cyclictest114750irq/25-eth023:02:461
1106699141,1cyclictest114750irq/25-eth023:02:461
1106699140,2cyclictest26766-21sshd21:25:181
1106699140,1cyclictest6470-21sshd22:13:451
1106699140,1cyclictest0-21swapper/123:17:491
1106699140,1cyclictest0-21swapper/121:54:121
1106599144,4cyclictest10248-21apache_processe20:15:130
1106599143,9cyclictest0-21swapper/019:39:040
1106599143,7cyclictest0-21swapper/019:25:130
11065991410,4cyclictest114750irq/25-eth019:30:190
1106599140,9cyclictest0-21swapper/023:16:580
1106599140,13cyclictest8266-21bash22:30:410
1106599140,11cyclictest0-21swapper/023:27:010
319732130,0sleep21718-21runrttasks23:53:472
160242130,0sleep20-21swapper/222:41:132
141142130,1sleep20-21swapper/200:29:172
1106899138,4cyclictest1707-21sshd22:43:053
1106899133,9cyclictest39-21ksoftirqd/323:11:533
11068991312,1cyclictest39-21ksoftirqd/300:10:513
11068991311,1cyclictest39-21ksoftirqd/322:45:233
11068991311,1cyclictest39-21ksoftirqd/300:16:213
11068991310,1cyclictest39-21ksoftirqd/323:30:313
11068991310,1cyclictest114750irq/25-eth022:05:563
1106899130,12cyclictest0-21swapper/322:14:293
1106899130,11cyclictest0-21swapper/323:07:253
1106799139,3cyclictest1579-21sshd22:26:412
1106799139,1cyclictest0-21swapper/222:59:292
11067991311,1cyclictest28047-21sshd23:56:312
11067991311,1cyclictest20634-21bash23:20:182
1106799131,11cyclictest0-21swapper/223:48:422
1106799130,4cyclictest27755-21sshd00:23:532
1106799130,4cyclictest0-21swapper/223:16:172
1106799130,1cyclictest0-21swapper/222:20:152
1106799130,12cyclictest28685-21bash21:48:552
1106799130,12cyclictest26092-21sshd21:35:082
1106799130,12cyclictest23149-21sshd22:48:272
1106799130,12cyclictest11462-21sshd22:07:312
1106799130,12cyclictest0-21swapper/223:42:102
1106799130,12cyclictest0-21swapper/223:36:172
1106799130,12cyclictest0-21swapper/223:08:062
1106799130,12cyclictest0-21swapper/200:02:232
1106699139,3cyclictest0-21swapper/100:35:421
1106699139,1cyclictest114750irq/25-eth023:44:261
1106699138,1cyclictest1351-21sshd21:49:291
1106699138,1cyclictest0-21swapper/121:16:511
1106699130,9cyclictest0-21swapper/122:30:061
1106699130,2cyclictest0-21swapper/123:14:461
1106699130,2cyclictest0-21swapper/122:56:121
1106699130,1cyclictest3832-21sshd21:39:351
1106699130,1cyclictest29326-21bash00:34:181
1106699130,1cyclictest0-21swapper/123:22:471
1106699130,10cyclictest27373-21sshd22:39:121
1106699130,10cyclictest27373-21sshd22:39:121
1106599133,7cyclictest122650irq/26-eth1-rx-20:59:250
1106599130,8cyclictest0-21swapper/021:30:090
1106599130,13cyclictest0-21swapper/019:21:290
1106899129,2cyclictest0-21swapper/322:25:173
1106899120,9cyclictest0-21swapper/322:15:383
1106899120,11cyclictest0-21swapper/320:50:153
1106899120,11cyclictest0-21swapper/319:32:333
1106799129,2cyclictest0-21swapper/222:03:542
1106799129,2cyclictest0-21swapper/221:15:472
1106799129,2cyclictest0-21swapper/221:10:492
1106799128,3cyclictest31594-21sshd21:42:242
1106799121,2cyclictest0-21swapper/221:30:112
1106799120,12cyclictest0-21swapper/200:31:412
1106799120,11cyclictest0-21swapper/223:00:532
1106799120,11cyclictest0-21swapper/223:00:532
1106799120,11cyclictest0-21swapper/200:35:062
1106699128,3cyclictest9254-21bash23:45:151
1106699128,1cyclictest0-21swapper/122:41:071
1106699127,4cyclictest0-21swapper/100:15:151
1106599125,3cyclictest0-21swapper/020:44:370
1106599124,5cyclictest0-21swapper/020:11:250
1106599122,8cyclictest122650irq/26-eth1-rx-20:45:040
1106599122,8cyclictest122650irq/26-eth1-rx-20:45:040
1106599120,11cyclictest0-21swapper/019:14:590
1106599120,10cyclictest0-21swapper/020:31:410
11068991110,1cyclictest0-21swapper/319:57:173
1106899110,1cyclictest0-21swapper/321:02:133
1106899110,1cyclictest0-21swapper/320:45:513
1106899110,1cyclictest0-21swapper/320:45:513
1106899110,1cyclictest0-21swapper/320:41:543
1106899110,11cyclictest0-21swapper/320:08:553
1106899110,10cyclictest0-21swapper/320:16:143
1106899110,10cyclictest0-21swapper/319:23:573
1106699110,2cyclictest0-21swapper/119:15:001
1106599118,1cyclictest0-21swapper/019:50:160
1106599117,3cyclictest0-21swapper/019:15:020
1106599116,2cyclictest0-21swapper/021:05:130
1106599114,5cyclictest0-21swapper/020:06:450
1106599110,9cyclictest0-21swapper/019:44:560
1106599110,1cyclictest0-21swapper/020:52:570
1106899109,1cyclictest0-21swapper/320:00:533
1106899109,1cyclictest0-21swapper/319:49:593
1106899108,1cyclictest0-21swapper/320:35:173
1106899105,4cyclictest18872-21sed20:30:233
1106899104,5cyclictest0-21swapper/320:23:373
1106799108,1cyclictest0-21swapper/219:40:192
1106799100,9cyclictest32374-21perl21:00:152
1106799100,9cyclictest0-21swapper/220:35:172
1106799100,9cyclictest0-21swapper/220:10:162
1106799100,1cyclictest0-21swapper/221:05:162
1106699109,1cyclictest0-21swapper/119:56:191
1106699109,1cyclictest0-21swapper/119:47:151
1106699100,1cyclictest0-21swapper/120:34:381
1106699100,10cyclictest0-21swapper/121:04:231
1106699100,10cyclictest0-21swapper/120:44:311
1106699100,10cyclictest0-21swapper/120:20:431
1106699100,10cyclictest0-21swapper/120:03:381
1106599104,3cyclictest0-21swapper/020:00:210
1106599100,5cyclictest0-21swapper/020:28:250
110689999,0cyclictest0-21swapper/319:18:513
110689998,1cyclictest0-21swapper/319:25:353
110689998,1cyclictest0-21swapper/319:10:573
110689990,8cyclictest0-21swapper/319:50:223
110679997,1cyclictest25272-21perl20:45:212
110679997,1cyclictest25272-21perl20:45:212
110679990,1cyclictest0-21swapper/220:25:152
110669998,1cyclictest0-21swapper/120:57:181
110669998,1cyclictest0-21swapper/120:52:111
110669998,1cyclictest0-21swapper/120:27:341
110669998,1cyclictest0-21swapper/120:09:401
110669990,5cyclictest0-21swapper/120:49:251
110669990,5cyclictest0-21swapper/120:49:251
110659990,5cyclictest0-21swapper/020:20:140
110689987,1cyclictest0-21swapper/319:35:553
110689980,7cyclictest0-21swapper/321:05:203
110679982,2cyclictest0-21swapper/220:40:332
110669987,1cyclictest0-21swapper/119:54:341
110669987,1cyclictest0-21swapper/119:33:251
110669981,6cyclictest0-21swapper/119:10:131
110689975,1cyclictest8666-21fw_packets20:10:183
110689971,5cyclictest0-21swapper/319:40:153
110689970,6cyclictest0-21swapper/320:55:123
110679977,0cyclictest0-21swapper/220:55:162
110679975,1cyclictest25948-21latency_hist19:40:002
110679975,1cyclictest2503-21taskset19:55:332
110679975,1cyclictest19590-21perl19:25:172
110679971,5cyclictest0-21swapper/219:45:182
110679970,6cyclictest0-21swapper/220:50:202
110679970,6cyclictest0-21swapper/220:05:162
110669976,1cyclictest0-21swapper/119:40:021
110669973,2cyclictest0-21swapper/120:15:191
110669971,5cyclictest0-21swapper/119:20:211
110669970,6cyclictest0-21swapper/120:10:591
110669970,6cyclictest0-21swapper/119:35:231
110669970,4cyclictest0-21swapper/121:05:191
110679962,2cyclictest0-21swapper/220:30:182
110679962,2cyclictest0-21swapper/220:00:262
110679962,2cyclictest0-21swapper/219:50:232
110679961,4cyclictest0-21swapper/220:20:142
110679961,3cyclictest0-21swapper/219:30:192
110679960,4cyclictest0-21swapper/219:15:002
110669961,1cyclictest114750irq/25-eth019:25:121
110669960,6cyclictest0-21swapper/120:35:101
110679951,3cyclictest0-21swapper/219:20:192
110679950,4cyclictest0-21swapper/220:15:122
110679950,2cyclictest11092-21apache_volume19:10:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional