You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-05-20 - 06:31

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack5slot0.osadl.org (updated Fri May 20, 2022 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3170721600,0sleep231705-21sshd23:05:192
345721460,3sleep32914799cyclictest22:26:313
1438821400,1sleep214366-21sshd00:06:032
3078821390,0sleep130759-21sshd23:36:121
247321150,0sleep12471-21bash00:13:551
3201521050,1sleep10-21swapper/121:25:061
1749421050,2sleep117478-21sshd23:04:091
2520321020,3sleep02914499cyclictest21:54:590
2881529833,45sleep30-21swapper/319:09:063
112012960,2sleep211184-21sshd23:31:442
89132900,2sleep10-21swapper/122:49:231
60372890,2sleep21875-21nfsd22:35:092
181812890,0sleep00-21swapper/023:15:230
51922880,2sleep35176-21sshd22:01:353
280412880,2sleep128037-21sshd21:19:131
200472880,3sleep22914699cyclictest22:33:312
287342850,3sleep30-21swapper/323:05:073
260082850,2sleep20-21swapper/222:56:292
102692850,1sleep30-21swapper/321:39:553
100262850,2sleep20-21swapper/223:54:082
2868028433,39sleep10-21swapper/119:07:231
25182840,1sleep02504-21sshd22:37:390
15802840,2sleep11578-21sshd00:02:011
284862830,2sleep228481-21sshd00:10:192
125342820,1sleep012530-21sshd22:05:040
21092810,3sleep30-21swapper/321:11:203
157712810,1sleep015758-21sshd22:24:530
85942800,2sleep28593-21bash22:24:152
225402800,3sleep10-21swapper/123:27:061
260092790,1sleep00-21swapper/021:38:280
71202780,2sleep10-21swapper/121:47:551
225892780,2sleep30-21swapper/322:17:023
181232770,2sleep20-21swapper/221:23:552
28062750,2sleep30-21swapper/300:13:573
271912750,1sleep20-21swapper/221:38:332
189072750,3sleep30-21swapper/321:54:253
181062740,4sleep22914699cyclictest22:41:462
101662720,3sleep02914499cyclictest22:49:300
301612700,1sleep030163-21nfsd419:10:180
98742690,2sleep32914799cyclictest23:20:193
98742690,2sleep32914799cyclictest23:20:193
237082690,1sleep20-21swapper/221:29:562
219452680,4sleep32914799cyclictest23:01:433
27782650,0sleep00-21swapper/022:32:020
160392630,2sleep316038-21bash23:51:493
2886926031,16sleep20-21swapper/219:09:482
211212600,2sleep20-21swapper/222:11:192
2856424033,4sleep00-21swapper/019:06:070
234462380,0sleep323447-21id21:43:483
244312370,0sleep224432-21bash22:50:392
2914799350,7cyclictest0-21swapper/321:20:233
278812340,0sleep327882-21proc_pri23:30:213
2914799320,7cyclictest0-21swapper/300:07:403
152832310,4sleep12914599cyclictest22:13:401
82202300,4sleep12914599cyclictest23:51:071
154802301,27sleep2138550irq/27-eth023:23:432
154802301,27sleep2138550irq/27-eth023:23:432
142602300,3sleep114256-21sshd00:06:011
104272300,2sleep110411-21sshd23:11:551
191012290,1sleep30-21swapper/323:38:013
2914799277,2cyclictest0-21swapper/300:29:323
29145992722,3cyclictest1987-21sshd23:05:451
18802270,3sleep30-21swapper/322:12:293
323522260,24sleep1138550irq/27-eth021:52:481
2914799260,3cyclictest0-21swapper/322:23:173
2914799250,3cyclictest0-21swapper/323:11:373
2914799250,3cyclictest0-21swapper/321:32:113
268252250,7sleep30-21swapper/322:56:343
2914799248,2cyclictest0-21swapper/323:17:293
29147992411,2cyclictest0-21swapper/323:41:553
2914799240,10cyclictest0-21swapper/321:25:513
2914499240,22cyclictest15892-21sshd23:23:440
2914499240,22cyclictest15892-21sshd23:23:440
233922240,22sleep1138550irq/27-eth000:24:501
2914799238,3cyclictest0-21swapper/323:25:283
2914799238,3cyclictest0-21swapper/322:08:013
2914799238,2cyclictest0-21swapper/322:36:443
2914799238,2cyclictest0-21swapper/321:46:143
29147992311,1cyclictest0-21swapper/300:39:383
2914799230,21cyclictest24663-21sshd22:53:373
2914799230,21cyclictest0-21swapper/320:25:163
2914799230,10cyclictest0-21swapper/323:49:283
2914799230,10cyclictest0-21swapper/322:46:203
2914799230,10cyclictest0-21swapper/321:16:033
2914799230,10cyclictest0-21swapper/300:31:093
2914799230,10cyclictest0-21swapper/300:22:243
29145992316,4cyclictest0-21swapper/123:23:051
29145992316,4cyclictest0-21swapper/123:23:051
2914799220,9cyclictest0-21swapper/300:00:223
2914799220,3cyclictest0-21swapper/323:56:353
2914799220,2cyclictest0-21swapper/322:41:483
2914799220,10cyclictest0-21swapper/322:31:553
2914799220,10cyclictest0-21swapper/321:58:123
2914799220,10cyclictest0-21swapper/300:15:203
2914599224,3cyclictest27009-21bash22:20:121
2914599221,18cyclictest22-21ksoftirqd/123:55:121
2914599220,5cyclictest0-21swapper/122:52:591
2914499223,14cyclictest6570-21sshd21:28:270
2914499220,21cyclictest4938-21sshd23:14:190
2914499220,18cyclictest27215-21cp22:53:500
183182220,1sleep00-21swapper/023:35:100
29146992116,3cyclictest7927-21sshd21:17:272
29146992114,4cyclictest11238-21grep23:40:102
29146992111,7cyclictest9100-21bash23:00:332
2914599219,4cyclictest9246-21sshd22:40:591
2914599218,4cyclictest30817-21bash22:37:171
29145992116,3cyclictest0-21swapper/121:37:181
29145992114,3cyclictest138550irq/27-eth022:19:141
2914599211,3cyclictest0-21swapper/121:14:201
29145992112,5cyclictest11227-21bash23:40:111
211972210,1sleep00-21swapper/022:16:550
150522210,2sleep215047-21bash21:34:452
297292200,2sleep129709-21sshd23:16:241
29146992015,2cyclictest17891-21sshd22:02:442
29146992014,4cyclictest13810-21sshd22:05:112
29146992012,5cyclictest2616-21bash23:36:352
29146992010,8cyclictest1955-21snmpd23:49:402
2914599205,4cyclictest14208-21sshd22:02:231
2914599204,9cyclictest16568-21sshd22:58:301
29145992011,6cyclictest138550irq/27-eth023:45:171
2914599201,11cyclictest8026-21sshd22:07:261
2914599200,18cyclictest0-21swapper/100:36:271
2914499204,14cyclictest18387-21kworker/0:021:24:520
2914499201,15cyclictest0-21swapper/021:49:310
2914499201,10cyclictest9699-21bash22:10:210
19852200,0sleep00-21swapper/019:20:150
2914799199,2cyclictest0-21swapper/320:16:343
29146991914,3cyclictest17791-21sshd23:18:122
29146991914,3cyclictest17117-21bash22:27:452
2914599194,10cyclictest0-21swapper/121:00:211
2914599191,8cyclictest0-21swapper/121:57:171
2914599190,3cyclictest0-21swapper/123:30:331
2914599190,13cyclictest14355-21rm21:40:151
2914699186,3cyclictest11965-21sshd23:29:022
2914699185,5cyclictest14940-21bash23:57:232
2914699183,14cyclictest138550irq/27-eth021:45:332
29146991814,3cyclictest4761-21sshd00:31:582
29146991814,3cyclictest4559-21bash21:42:092
29146991814,2cyclictest11749-21sshd21:53:482
29146991813,3cyclictest8649-21cp23:14:372
29146991813,3cyclictest6196-21bash00:20:112
29146991813,3cyclictest21271-21sshd22:47:372
29146991813,3cyclictest1987-21sshd21:56:292
2914699180,16cyclictest2294-21sshd22:15:182
2914599188,4cyclictest14651-21sshd22:30:171
2914599187,4cyclictest3864-21sshd21:22:401
2914599182,14cyclictest23676-21sshd00:30:401
2914599181,1cyclictest13429-21sshd21:31:471
2914599181,10cyclictest0-21swapper/100:19:551
2914599180,17cyclictest22-21ksoftirqd/100:25:131
2914599180,11cyclictest20474-21id22:25:151
239162180,4sleep22914699cyclictest00:39:452
2914699179,2cyclictest10280-21munin-node19:40:142
2914699172,6cyclictest0-21swapper/221:12:252
29146991714,2cyclictest19358-21sshd00:15:232
29146991712,3cyclictest26840-21sshd00:01:202
2914499171,8cyclictest0-21swapper/021:10:120
2914499171,15cyclictest31801-21bash00:01:490
2914499171,14cyclictest31508-21sshd23:58:520
2914499170,9cyclictest16405-21sshd23:43:250
2914499170,9cyclictest10160-21bash23:48:310
2914499170,13cyclictest0-21swapper/000:37:590
29147991612,2cyclictest3395-21ipmitool20:35:153
2914799160,14cyclictest0-21swapper/319:57:493
29146991612,3cyclictest5698-21id00:26:032
2914599168,5cyclictest23560-21munin-run20:10:001
2914499164,1cyclictest0-21swapper/023:01:100
2914499161,5cyclictest0-21swapper/000:08:480
2914499161,2cyclictest0-21swapper/023:09:480
29144991612,3cyclictest0-21swapper/000:17:210
29144991611,4cyclictest0-21swapper/023:26:170
2914499161,12cyclictest0-21swapper/022:56:170
2914499160,15cyclictest0-21swapper/000:29:140
2914499160,14cyclictest0-21swapper/021:32:320
2914499160,12cyclictest4770-21sshd22:29:280
2914499160,12cyclictest0-21swapper/022:41:040
55532150,0sleep00-21swapper/020:40:130
2914699155,5cyclictest0-21swapper/220:46:182
2914699152,12cyclictest138550irq/27-eth020:50:152
29146991510,1cyclictest0-21swapper/219:25:382
2914599158,5cyclictest0-21swapper/120:30:001
2914599157,5cyclictest0-21swapper/120:51:031
2914499155,2cyclictest3587-21atenall_r5power19:25:110
2914499150,6cyclictest8656-21bash21:17:320
2914499150,2cyclictest0-21swapper/000:12:010
2914499150,14cyclictest6759-21sshd00:32:090
2914499150,13cyclictest31829-21sshd21:55:260
2914499150,11cyclictest23690-21sshd22:00:210
2914799140,13cyclictest25-21rcuop/121:02:533
2914699143,7cyclictest53-21khungtaskd20:58:012
2914699142,11cyclictest138550irq/27-eth019:20:022
2914599149,4cyclictest138550irq/27-eth019:25:501
2914599144,6cyclictest28066-21idleruntime-cro20:20:011
2914599141,12cyclictest138550irq/27-eth020:30:491
2914599140,13cyclictest22-21ksoftirqd/119:55:181
2914599140,13cyclictest0-21swapper/120:37:591
2914599140,13cyclictest0-21swapper/120:10:421
2914499141,8cyclictest0-21swapper/000:20:580
2914499141,2cyclictest0-21swapper/023:50:210
29144991412,1cyclictest28761-21sshd23:30:240
2914499140,12cyclictest0-21swapper/021:40:030
2914799138,3cyclictest23775-21munin-node20:10:213
2914799137,5cyclictest0-21swapper/320:52:073
2914799131,11cyclictest138550irq/27-eth020:30:343
2914799130,4cyclictest11394-21ntp_offset19:40:173
2914799130,1cyclictest138550irq/27-eth019:10:223
2914799130,12cyclictest2121-21runrttasks20:41:363
2914799130,12cyclictest0-21swapper/320:59:203
2914799130,12cyclictest0-21swapper/320:47:243
2914799130,12cyclictest0-21swapper/320:20:083
2914799130,12cyclictest0-21swapper/320:06:283
2914799130,12cyclictest0-21swapper/320:00:023
2914799130,12cyclictest0-21swapper/319:53:363
2914799130,12cyclictest0-21swapper/319:48:383
2914799130,12cyclictest0-21swapper/319:37:073
2914799130,12cyclictest0-21swapper/319:15:033
2914799130,11cyclictest5951-21cpuspeed_turbos19:30:123
2914799130,11cyclictest2010-21iostat19:20:163
2914799130,10cyclictest138550irq/27-eth021:05:583
2914699138,4cyclictest29160-21munin-node19:10:242
2914699138,4cyclictest1877-21nfsd19:15:492
2914699133,5cyclictest0-21swapper/219:45:452
29146991311,1cyclictest0-21swapper/221:00:132
29146991311,1cyclictest0-21swapper/220:11:562
29146991311,1cyclictest0-21swapper/219:50:142
2914699130,5cyclictest0-21swapper/220:20:502
2914699130,5cyclictest0-21swapper/219:35:212
2914699130,4cyclictest1943-21snmpd21:05:382
2914699130,12cyclictest1876-21nfsd20:25:112
2914699130,12cyclictest0-21swapper/220:35:042
2914699130,12cyclictest0-21swapper/220:30:552
2914699130,12cyclictest0-21swapper/220:18:022
2914699130,12cyclictest0-21swapper/219:30:482
2914599139,3cyclictest138550irq/27-eth020:00:221
2914599137,5cyclictest0-21swapper/119:50:441
2914599133,1cyclictest0-21swapper/119:10:051
29145991311,1cyclictest0-21swapper/119:47:411
29145991310,2cyclictest0-21swapper/120:21:041
2914599130,5cyclictest0-21swapper/120:48:481
2914599130,4cyclictest4900-21munin-node20:40:201
2914599130,12cyclictest0-21swapper/121:05:081
2914599130,12cyclictest0-21swapper/120:58:471
2914599130,12cyclictest0-21swapper/119:44:171
2914599130,12cyclictest0-21swapper/119:20:251
2914599130,12cyclictest0-21swapper/119:19:351
2914499132,1cyclictest0-21swapper/020:56:300
29144991311,1cyclictest0-21swapper/020:01:090
2914499130,1cyclictest0-21swapper/020:39:040
2914499130,10cyclictest0-21swapper/020:46:540
2914799128,3cyclictest138550irq/27-eth019:25:303
2914699128,3cyclictest138550irq/27-eth020:40:112
29146991210,1cyclictest0-21swapper/220:06:332
2914699120,11cyclictest0-21swapper/220:03:402
2914699120,10cyclictest17321-21df_abs19:55:132
2914599128,3cyclictest1163-21jbd2/dm-2-819:30:141
2914599120,11cyclictest138550irq/27-eth019:37:481
2914499128,3cyclictest0-21swapper/021:00:050
2914499128,3cyclictest0-21swapper/020:30:030
2914499128,3cyclictest0-21swapper/020:05:110
2914499128,2cyclictest0-21swapper/020:50:460
2914499127,4cyclictest0-21swapper/019:18:260
2914499122,9cyclictest0-21swapper/020:16:350
2914499122,9cyclictest0-21swapper/019:40:160
2914499121,1cyclictest0-21swapper/019:45:570
29144991210,1cyclictest0-21swapper/021:09:000
2914499120,11cyclictest0-21swapper/020:13:050
2914499119,1cyclictest0-21swapper/019:30:060
2914499117,3cyclictest0-21swapper/019:55:430
2914499117,3cyclictest0-21swapper/019:35:100
2914499112,8cyclictest31393-21kworker/0:220:20:040
2914499110,9cyclictest0-21swapper/020:25:160
2914499110,10cyclictest0-21swapper/019:50:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional