You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-05 - 18:08

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack5slot0.osadl.org (updated Tue Jul 05, 2022 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2103121530,4sleep3345799cyclictest09:48:543
350121430,3sleep13498-21sshd11:02:211
2825721430,3sleep3345799cyclictest10:34:413
1079821230,1sleep010781-21sshd09:32:490
3001121210,2sleep230007-21sshd09:28:342
511321130,0sleep00-21swapper/012:17:430
511321130,0sleep00-21swapper/012:17:420
1814221120,2sleep2345099cyclictest09:54:442
434521060,1sleep041ktimersoftd/012:26:410
190721010,2sleep21893-21sshd10:02:122
3074221000,2sleep1344699cyclictest12:20:091
195802970,1sleep019577-21bash10:51:490
128622960,1sleep10-21swapper/110:54:141
93672950,4sleep00-21swapper/009:17:340
285229526,41sleep10-21swapper/107:05:251
46402940,1sleep30-21swapper/311:29:323
46402940,1sleep30-21swapper/311:29:313
325262940,1sleep00-21swapper/009:50:000
323962930,3sleep0344199cyclictest09:13:410
231792930,3sleep0344199cyclictest11:04:110
208042930,3sleep320803-21bash11:49:003
131742910,4sleep00-21swapper/010:18:130
260882890,2sleep20-21swapper/211:07:272
59292880,3sleep20-21swapper/211:38:362
51282880,2sleep15119-21cp09:38:231
307812880,2sleep230771-21sshd10:37:522
247052880,2sleep224697-21sshd09:58:192
222192880,2sleep322215-21sshd12:01:073
115582880,2sleep311557-21munin-node12:30:253
320032850,2sleep132000-21sshd11:44:001
314292850,2sleep20-21swapper/211:34:592
141012850,0sleep00-21swapper/010:36:170
125522850,2sleep10-21swapper/112:36:361
56102840,0sleep05612-21bash12:05:330
304428432,48sleep30-21swapper/307:07:463
48292830,5sleep3345799cyclictest09:44:263
161092830,0sleep00-21swapper/010:30:290
58742820,2sleep10-21swapper/112:29:541
297202820,2sleep30-21swapper/309:31:313
26882820,2sleep00-21swapper/009:41:070
41842790,2sleep34172-21sshd12:29:453
308162790,2sleep10-21swapper/112:32:151
74462780,3sleep27445-21sshd09:11:182
180812780,0sleep00-21swapper/011:54:450
272152770,1sleep00-21swapper/011:07:320
178192770,4sleep2345099cyclictest10:57:412
166862760,1sleep10-21swapper/109:42:271
116132760,2sleep30-21swapper/309:50:593
61262740,2sleep26123-21sshd10:14:362
119262720,1sleep1344699cyclictest10:57:061
109112720,4sleep2345099cyclictest09:47:582
77482700,3sleep1344699cyclictest12:05:461
219482700,2sleep20-21swapper/212:07:072
260022690,1sleep30-21swapper/310:22:253
170502690,1sleep117049-21sshd12:00:371
65252680,1sleep20-21swapper/208:25:432
289322670,3sleep10-21swapper/110:43:411
28632660,3sleep1344699cyclictest10:29:221
165732660,2sleep116543-21sshd12:12:421
304326432,11sleep20-21swapper/207:07:452
264742540,1sleep026472-21consoletype09:55:260
112922490,0sleep00-21swapper/011:21:030
19842480,0sleep01986-21id12:20:240
305124632,11sleep00-21swapper/007:07:510
294432450,2sleep30-21swapper/310:58:463
47282440,5sleep30-21swapper/311:38:303
97662350,2sleep10-21swapper/110:17:551
276412340,1sleep10-21swapper/111:34:391
224822331,30sleep328750irq/26-ahci[00010:49:093
3457993227,3cyclictest3864-21sshd09:26:043
3441993114,5cyclictest3-21ksoftirqd/011:32:090
249922300,28sleep128750irq/26-ahci[00011:58:251
133662300,24sleep1138550irq/27-eth011:54:201
30982290,3sleep1344699cyclictest11:35:191
189362290,2sleep118934-21sshd10:06:441
3441992814,11cyclictest3-21ksoftirqd/011:58:110
22622280,2sleep32259-21bash11:11:113
44872272,23sleep2138550irq/27-eth010:26:262
3441992711,13cyclictest3-21ksoftirqd/012:36:540
319372270,3sleep231936-21sshd09:37:512
219392270,2sleep30-21swapper/312:13:133
3457992623,2cyclictest0-21swapper/309:36:053
3457992610,6cyclictest138550irq/27-eth011:31:453
344199268,11cyclictest3-21ksoftirqd/010:07:540
3441992613,11cyclictest3-21ksoftirqd/009:36:530
344199261,23cyclictest0-21swapper/010:13:020
3441992611,12cyclictest3-21ksoftirqd/011:43:370
3441992610,7cyclictest26125-21sshd12:04:330
17632260,1sleep00-21swapper/012:32:340
147832260,4sleep20-21swapper/212:00:252
345099252,9cyclictest0-21swapper/209:40:532
3441992512,10cyclictest3-21ksoftirqd/011:45:480
234602250,3sleep20-21swapper/210:40:132
211192250,0sleep20-21swapper/211:15:552
345099248,9cyclictest0-21swapper/207:38:012
345099243,20cyclictest21338-21sshd10:49:002
345099242,9cyclictest0-21swapper/210:07:532
344199240,10cyclictest0-21swapper/009:52:540
171002240,1sleep01876-21nfsd10:00:350
13382240,2sleep10-21swapper/110:11:051
345099230,9cyclictest0-21swapper/209:17:492
344199238,2cyclictest0-21swapper/010:58:470
3441992318,3cyclictest138550irq/27-eth011:13:030
3441992314,3cyclictest3-21ksoftirqd/009:21:570
206682230,2sleep20-21swapper/212:16:052
206682230,2sleep20-21swapper/212:16:052
118472230,3sleep011849-21unixbench_singl07:25:220
3457992219,2cyclictest0-21swapper/309:20:023
3457992210,6cyclictest138550irq/27-eth009:58:183
345099223,9cyclictest14867-21sshd12:12:332
344699228,2cyclictest138550irq/27-eth010:00:521
344699220,18cyclictest0-21swapper/110:45:271
3441992214,6cyclictest3-21ksoftirqd/011:36:470
344199221,12cyclictest29358-21sshd10:28:480
3441992210,11cyclictest3-21ksoftirqd/011:18:340
312122220,2sleep32121-21runrttasks10:16:523
204802220,2sleep320470-21sshd11:18:583
201242220,20sleep3138550irq/27-eth011:00:513
200822220,2sleep10-21swapper/112:19:081
200822220,2sleep10-21swapper/112:19:081
345799219,5cyclictest138550irq/27-eth010:14:493
3457992110,1cyclictest27645-21sshd09:13:153
345099213,13cyclictest5406-21sshd12:23:462
3450992114,6cyclictest0-21swapper/208:05:172
345099211,14cyclictest0-21swapper/210:24:252
344699218,10cyclictest27650-21sshd09:19:171
344199218,10cyclictest3-21ksoftirqd/010:41:360
3441992117,3cyclictest3-21ksoftirqd/012:12:450
3441992114,5cyclictest3-21ksoftirqd/010:23:200
3441992112,7cyclictest3-21ksoftirqd/010:48:590
345799206,8cyclictest138550irq/27-eth010:36:133
345099205,2cyclictest0-21swapper/208:58:422
345099204,8cyclictest0-21swapper/208:00:372
345099204,14cyclictest4389-21sshd12:29:462
345099203,11cyclictest11819-21sshd11:27:092
345099203,11cyclictest11819-21sshd11:27:092
3450992015,3cyclictest5707-21sshd10:17:332
3450992012,2cyclictest138550irq/27-eth009:30:472
3450992010,5cyclictest0-21swapper/207:28:232
344699209,3cyclictest25821-21cut09:55:231
3441992015,3cyclictest3-21ksoftirqd/009:26:380
158512200,1sleep30-21swapper/307:35:193
35542190,14sleep3138550irq/27-eth011:56:233
345799196,5cyclictest138550irq/27-eth010:07:543
3457991910,4cyclictest138550irq/27-eth010:00:313
345099199,4cyclictest138550irq/27-eth008:45:232
345099196,5cyclictest138550irq/27-eth011:10:132
345099193,11cyclictest18406-21sshd11:57:452
345099192,13cyclictest0-21swapper/209:24:562
3450991911,5cyclictest2854-21sshd11:23:212
344699199,1cyclictest13505-21sshd10:30:181
344699197,5cyclictest6720-21sshd10:23:421
344699192,7cyclictest6337-21sshd09:20:171
3446991914,3cyclictest0-21swapper/109:26:051
344699190,12cyclictest138550irq/27-eth009:33:211
344699190,10cyclictest11394-21munin-run08:40:011
3441991915,2cyclictest3-21ksoftirqd/011:27:030
3441991915,2cyclictest3-21ksoftirqd/011:27:030
345799183,1cyclictest0-21swapper/312:05:053
3457991812,5cyclictest47-21rcuop/310:41:123
345799180,7cyclictest0-21swapper/312:19:113
345799180,7cyclictest0-21swapper/312:19:103
345099185,2cyclictest0-21swapper/207:45:062
345099185,2cyclictest0-21swapper/207:41:062
345099183,2cyclictest0-21swapper/207:21:262
345099182,2cyclictest0-21swapper/208:11:022
345099182,14cyclictest10746-21sshd12:33:252
345099182,11cyclictest10311-21sshd12:39:272
345099182,11cyclictest0-21swapper/207:15:572
345099181,13cyclictest0-21swapper/210:32:192
345099180,2cyclictest0-21swapper/209:06:532
344699186,6cyclictest138550irq/27-eth009:47:451
3446991811,4cyclictest4449-21sshd11:47:281
344699180,13cyclictest32101-21sshd09:10:331
344199181,14cyclictest0-21swapper/007:15:100
291212180,2sleep30-21swapper/312:38:093
88932170,15sleep1138550irq/27-eth009:53:511
345799178,4cyclictest3898-21sshd11:23:273
345799178,2cyclictest7528-21sshd10:53:453
345799178,2cyclictest1722-21bash12:20:223
345799177,1cyclictest0-21swapper/310:28:523
345799176,4cyclictest32114-21sshd09:19:443
345799170,6cyclictest11941-21cp11:06:053
345799170,13cyclictest0-21swapper/309:02:583
345799170,11cyclictest1210-21bash11:41:073
345099173,11cyclictest30356-21id11:52:532
345099173,10cyclictest0-21swapper/207:56:232
345099172,8cyclictest0-21swapper/211:40:052
345099171,8cyclictest0-21swapper/209:00:452
345099171,8cyclictest0-21swapper/208:40:372
345099171,2cyclictest0-21swapper/207:10:452
345099171,11cyclictest3306-21sshd11:02:192
344699178,2cyclictest138550irq/27-eth011:07:461
344699177,5cyclictest138550irq/27-eth011:28:401
344699177,5cyclictest138550irq/27-eth011:28:401
344699176,5cyclictest138550irq/27-eth010:35:021
344699170,13cyclictest32601-21sshd11:17:031
344699170,13cyclictest0-21swapper/107:35:581
345799167,7cyclictest0-21swapper/307:50:133
345799167,5cyclictest138550irq/27-eth007:10:173
345799165,1cyclictest0-21swapper/311:50:173
345099163,10cyclictest138550irq/27-eth011:45:062
345099161,13cyclictest138550irq/27-eth007:52:562
345099160,11cyclictest36-21rcuop/210:51:082
344699160,11cyclictest138550irq/27-eth011:11:371
344199168,2cyclictest0-21swapper/008:35:210
3441991613,2cyclictest3-21ksoftirqd/008:05:210
344199160,2cyclictest0-21swapper/008:15:110
345099152,7cyclictest138550irq/27-eth008:54:322
345099152,12cyclictest147450irq/28-eth1-rx-08:37:202
345099150,13cyclictest0-21swapper/207:30:232
344699157,5cyclictest0-21swapper/108:06:341
344699152,12cyclictest138550irq/27-eth007:16:001
344699150,3cyclictest147450irq/28-eth1-rx-07:54:511
344699150,14cyclictest7970-21sshd11:20:431
344699150,12cyclictest14509-21grep08:45:141
344199159,5cyclictest3-21ksoftirqd/008:40:220
344199157,2cyclictest0-21swapper/007:25:000
344199154,2cyclictest0-21swapper/007:55:190
3441991511,3cyclictest3-21ksoftirqd/008:10:220
344199150,13cyclictest4159-21grep08:20:230
344199150,13cyclictest4159-21grep08:20:230
345799149,3cyclictest8238-21cpuspeed_turbos07:20:123
345799142,11cyclictest138550irq/27-eth008:15:543
345799141,12cyclictest25-21rcuop/108:34:193
345799140,8cyclictest0-21swapper/308:20:363
345799140,8cyclictest0-21swapper/308:20:363
345799140,7cyclictest15887-21munin-run08:50:013
345799140,6cyclictest14095-21unixbench_singl07:30:223
345799140,6cyclictest11621-21munin-node08:40:223
345799140,13cyclictest0-21swapper/307:26:273
345799140,12cyclictest2121-21runrttasks07:17:563
345099143,2cyclictest0-21swapper/208:15:092
344699140,9cyclictest6905-21idleruntime-cro08:30:001
344699140,13cyclictest22333-21sshd09:01:281
344699140,13cyclictest0-21swapper/108:04:201
344199142,3cyclictest0-21swapper/008:30:260
344199141,9cyclictest0-21swapper/007:48:030
344199140,12cyclictest3771-21cpuspeed_turbos07:10:130
344199140,12cyclictest0-21swapper/009:01:180
344199140,11cyclictest0-21swapper/007:30:230
345799139,3cyclictest29646-21timerandwakeup08:05:223
345799139,3cyclictest138550irq/27-eth008:10:033
345799138,4cyclictest20151-21sshd08:57:283
345799130,8cyclictest1877-21nfsd08:35:533
345799130,12cyclictest0-21swapper/309:07:163
345799130,12cyclictest0-21swapper/307:55:433
345799130,12cyclictest0-21swapper/307:48:353
345799130,12cyclictest0-21swapper/307:41:543
345799130,11cyclictest5198-21munin-node08:25:123
345799130,11cyclictest17466-21munin-node08:50:223
345099132,10cyclictest25-21rcuop/108:30:202
3450991311,1cyclictest0-21swapper/208:23:482
3450991311,1cyclictest0-21swapper/208:23:482
344699132,3cyclictest1877-21nfsd08:31:481
344699132,10cyclictest138550irq/27-eth008:40:571
344699131,11cyclictest138550irq/27-eth009:05:511
3446991311,1cyclictest0-21swapper/107:58:061
3446991311,1cyclictest0-21swapper/107:46:151
344699130,9cyclictest12286-21perf07:30:011
344699130,1cyclictest0-21swapper/108:15:181
344699130,12cyclictest138550irq/27-eth007:40:101
344699130,12cyclictest0-21swapper/108:56:291
344699130,12cyclictest0-21swapper/108:10:421
344699130,12cyclictest0-21swapper/107:30:151
344699130,12cyclictest0-21swapper/107:22:541
344699130,12cyclictest0-21swapper/107:11:401
344699130,10cyclictest138550irq/27-eth008:20:201
344699130,10cyclictest138550irq/27-eth008:20:201
344199139,3cyclictest3-21ksoftirqd/007:50:220
344199139,3cyclictest0-21swapper/008:51:070
344199138,3cyclictest3-21ksoftirqd/007:40:120
344199130,12cyclictest0-21swapper/008:45:260
344199130,12cyclictest0-21swapper/008:27:480
344199130,10cyclictest0-21swapper/009:07:340
344199130,10cyclictest0-21swapper/008:00:110
345799120,11cyclictest0-21swapper/308:00:283
344699120,11cyclictest0-21swapper/108:50:021
344199120,11cyclictest0-21swapper/007:35:150
344199120,10cyclictest0-21swapper/008:55:520
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional