You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-04 - 01:55

x86 Intel Xeon E5620 @2400 MHz, Linux 4.19.127-rt55 (Profile)

Latency plot of system in rack #c, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackcslot0.osadl.org (updated Sat Dec 03, 2022 00:43:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
142352139118,15sleep00-21swapper/019:14:570
142352139118,15sleep00-21swapper/019:14:570
2646021260,0sleep120-21swapper/1221:20:174
147442126103,18sleep40-21swapper/419:19:1810
147442126103,18sleep40-21swapper/419:19:1810
14620212593,27sleep70-21swapper/719:18:1713
14620212593,27sleep70-21swapper/719:18:1713
145192125103,17sleep10-21swapper/119:16:501
145192125103,17sleep10-21swapper/119:16:501
2371221230,2sleep60-21swapper/622:06:4412
13502212099,14sleep90-21swapper/919:14:5015
13502212099,14sleep90-21swapper/919:14:5015
146922116103,9sleep80-21swapper/819:18:3414
146922116103,9sleep80-21swapper/819:18:3414
2009821130,1sleep60-21swapper/621:28:2912
14550211389,19sleep110-21swapper/1119:17:173
14550211389,19sleep110-21swapper/1119:17:173
2534421110,0sleep30-21swapper/323:23:119
14497210881,22sleep140-21swapper/1419:16:316
14497210881,22sleep140-21swapper/1419:16:316
1229821070,0sleep8811ktimersoftd/800:34:4214
2433321050,0sleep140-21swapper/1423:52:166
2433321050,0sleep140-21swapper/1423:52:166
1277121050,6sleep131527399cyclictest22:14:205
2306121020,6sleep61526499cyclictest23:47:0312
2306121020,6sleep61526499cyclictest23:47:0312
866621000,6sleep91526999cyclictest21:40:5015
866621000,6sleep91526999cyclictest21:40:5015
277821000,1sleep110-21swapper/1123:39:063
277821000,1sleep110-21swapper/1123:39:063
266622990,1sleep130-21swapper/1323:57:205
266622990,1sleep130-21swapper/1323:57:205
234792990,1sleep40-21swapper/422:06:4210
1450329876,17sleep20-21swapper/219:16:358
1450329876,17sleep20-21swapper/219:16:358
1447729876,17sleep130-21swapper/1319:16:145
1447729876,17sleep130-21swapper/1319:16:145
212492960,1sleep111051ktimersoftd/1121:24:173
311592950,1sleep140-21swapper/1422:50:126
253822950,0sleep00-21swapper/023:37:350
253822950,0sleep00-21swapper/023:37:350
1472329568,22sleep30-21swapper/319:19:019
1472329568,22sleep30-21swapper/319:19:019
90122920,1sleep111041rcuc/1121:49:583
206092920,1sleep140-21swapper/1421:51:376
189522920,0sleep110-21swapper/1100:24:033
323602910,0sleep130-21swapper/1321:53:285
288062910,1sleep3391rcuc/323:18:559
218452910,3sleep71526599cyclictest22:20:0813
1470028975,4sleep150-21swapper/1519:18:417
1470028975,4sleep150-21swapper/1519:18:417
1449528865,18sleep120-21swapper/1219:16:294
1449528865,18sleep120-21swapper/1219:16:294
1448728566,15sleep60-21swapper/619:16:2312
1448728566,15sleep60-21swapper/619:16:2312
64322840,0sleep40-21swapper/422:13:2310
276172830,0sleep140-21swapper/1423:28:446
1454428375,4sleep50-21swapper/519:17:1111
1454428375,4sleep50-21swapper/519:17:1111
299942810,5sleep11525899cyclictest22:16:331
130962810,5sleep141527599cyclictest22:43:156
1449328072,4sleep100-21swapper/1019:16:272
1449328072,4sleep100-21swapper/1019:16:272
305602780,0sleep130-21swapper/1323:29:155
280562720,1sleep40-21swapper/421:43:5810
280562720,1sleep40-21swapper/421:43:5810
311902710,6sleep01525599cyclictest00:18:470
311902710,6sleep01525599cyclictest00:18:470
20952640,1sleep151371ktimersoftd/1521:21:247
248882630,0sleep70-21swapper/721:52:1813
195902630,2sleep15138-21ksoftirqd/1523:41:197
195902630,2sleep15138-21ksoftirqd/1523:41:197
88642600,1sleep90-21swapper/922:32:4415
205852590,1sleep80-21swapper/821:37:5314
183302590,0sleep100-21swapper/1000:43:182
23602560,1sleep70-21swapper/723:39:0213
23602560,1sleep70-21swapper/723:39:0213
1527599460,44cyclictest0-21swapper/1422:03:036
15275994414,28cyclictest0-21swapper/1422:48:166
1527599440,3cyclictest6472-21gdbus00:27:546
1527599440,3cyclictest6472-21gdbus00:27:546
1527599430,12cyclictest0-21swapper/1400:32:016
1527599428,3cyclictest17561-21systemd23:46:086
1527599428,3cyclictest17561-21systemd23:46:086
1527599421,10cyclictest8569-21sshd22:13:426
1527599420,39cyclictest23645-21systemd23:18:096
119942420,0sleep10-21swapper/122:37:571
1527599419,3cyclictest2156-21sshd22:06:466
15275994111,1cyclictest0-21swapper/1423:33:026
15275994111,1cyclictest0-21swapper/1423:33:026
1527599410,24cyclictest8181-21sshd23:06:266
219312400,1sleep10-21swapper/100:29:461
15275994014,23cyclictest15226-21systemd22:33:476
15275994013,25cyclictest0-21swapper/1421:46:356
15275994012,1cyclictest0-21swapper/1400:49:136
1527599400,11cyclictest26515-21sshd23:23:246
15275993913,23cyclictest2868-21systemd-cgroups22:55:336
15275993910,3cyclictest2156-21sshd00:43:576
1527599390,1cyclictest0-21swapper/1423:37:136
1527599390,1cyclictest0-21swapper/1423:37:136
87452380,0sleep40-21swapper/422:28:0610
208652380,2sleep150-21swapper/1522:20:007
1527599388,4cyclictest1769-21systemd22:22:236
1527599381,4cyclictest0-21swapper/1419:29:386
15275993813,22cyclictest24471-21sshd23:13:416
1527599377,3cyclictest9385-21sshd23:01:356
15275993712,23cyclictest6456-21packagekitd21:57:326
15275993710,3cyclictest12356-21run-parts22:38:006
15259993713,4cyclictest6947-21systemd22:27:498
1527599369,3cyclictest20130-2110-uname00:37:096
1527599368,3cyclictest5701-21sh22:27:386
1527599365,3cyclictest24018-21polkitd23:57:206
1527599365,3cyclictest24018-21polkitd23:57:206
1527599362,10cyclictest1-21systemd23:42:256
1527599362,10cyclictest1-21systemd23:42:256
15275993614,20cyclictest0-21swapper/1400:24:076
156182350,1sleep120-21swapper/1223:45:494
156182350,1sleep120-21swapper/1223:45:494
1527599358,4cyclictest26901-21sh00:06:496
1527599358,3cyclictest22329-21sshd00:11:006
1527599356,4cyclictest21853-21systemd00:16:016
1527599356,4cyclictest21853-21systemd00:16:016
1527599351,12cyclictest11595-21rm21:27:096
1527599350,1cyclictest0-21swapper/1421:22:446
1527599350,1cyclictest0-21swapper/1400:04:096
322212340,0sleep100-21swapper/1022:22:032
186402340,1sleep80-21swapper/800:24:0014
4422330,8sleep11525899cyclictest22:22:101
256712330,23sleep60-21swapper/623:18:2612
1527599335,2cyclictest0-21swapper/1421:35:556
15259993312,4cyclictest30707-21systemd21:25:128
308982320,1sleep130-21swapper/1321:34:295
155692320,3sleep315782-21needreboot21:14:489
1527599327,3cyclictest24907-21sshd22:15:426
15265993217,8cyclictest0-21swapper/721:34:5913
66182310,1sleep150-21swapper/1523:30:067
66182310,1sleep150-21swapper/1523:30:067
31782310,1sleep120-21swapper/1200:08:174
240152310,2sleep90-21swapper/922:06:4815
158792310,1sleep20-21swapper/200:14:488
158792310,1sleep20-21swapper/200:14:488
1527599318,3cyclictest24551-21uname21:43:256
1527599318,3cyclictest24551-21uname21:43:256
1527599317,3cyclictest30077-21latency_hist21:34:276
1527599317,3cyclictest30077-21latency_hist21:34:276
1527599312,5cyclictest0-21swapper/1419:54:406
1526599312,27cyclictest0-21swapper/722:05:5013
1526599310,29cyclictest0-21swapper/722:42:1713
1526599310,28cyclictest23005-21sshd23:52:0213
1526599310,28cyclictest23005-21sshd23:52:0213
1526599310,23cyclictest148241sshd22:52:5513
111112310,24sleep50-21swapper/521:36:1411
33412300,1sleep20-21swapper/222:04:008
323982300,1sleep40-21swapper/423:19:2810
301652300,26sleep130-21swapper/1300:25:265
301652300,26sleep130-21swapper/1300:25:265
237052300,27sleep110-21swapper/1122:02:013
230872300,0sleep150-21swapper/1500:24:397
230872300,0sleep150-21swapper/1500:24:397
187512300,6sleep61526499cyclictest00:00:4212
182732300,2sleep318271-21systemd-cgroups22:34:149
1527599307,3cyclictest18968-21latency_hist19:24:276
1527599307,3cyclictest18968-21latency_hist19:24:276
1526599303,24cyclictest1-21systemd23:40:2713
1526599303,24cyclictest1-21systemd23:40:2713
106092300,25sleep60-21swapper/622:56:4612
79962290,1sleep60-21swapper/621:35:4212
225992290,1sleep110-21swapper/1122:06:343
1526599291,25cyclictest15374-21sshd22:29:1013
15265992911,16cyclictest0-21swapper/723:59:4213
15265992911,15cyclictest18333-21sshd21:46:5613
137922290,6sleep11525899cyclictest00:34:511
67222280,0sleep130-21swapper/1321:26:275
5072280,23sleep110-21swapper/1100:45:443
232952280,2sleep223279-21sshd22:06:408
225912280,1sleep10-21swapper/100:11:031
1526899287,10cyclictest0-21swapper/823:49:2714
1526899287,10cyclictest0-21swapper/823:49:2714
15268992810,15cyclictest5796-21run-parts23:15:2114
1526599289,15cyclictest6472-21gdbus23:31:2413
1526599289,15cyclictest6472-21gdbus23:31:2413
1526599284,18cyclictest969-21dbus-daemon22:45:3413
1526599282,24cyclictest0-21swapper/723:13:1213
1526599282,23cyclictest969-21dbus-daemon22:10:3713
1526599281,25cyclictest0-21swapper/700:34:2413
1526599281,24cyclictest968-21systemd-logind23:56:1213
1526599281,24cyclictest968-21systemd-logind23:56:1213
15265992812,13cyclictest0-21swapper/700:05:0013
15265992810,15cyclictest11318-21systemd00:49:0213
1526599280,26cyclictest0-21swapper/723:02:4713
1526599280,25cyclictest0-21swapper/721:27:5613
15259992812,3cyclictest20519-21systemd21:24:118
127292280,1sleep130-21swapper/1322:04:595
111252280,2sleep111127-21sshd21:41:141
111252280,2sleep111127-21sshd21:41:141
315122273,20sleep50-21swapper/523:53:2811
315122273,20sleep50-21swapper/523:53:2811
247612270,3sleep13969-21dbus-daemon23:13:455
1527599277,3cyclictest20196-21cstates19:24:416
1526899278,17cyclictest0-21swapper/823:41:1014
1526899278,17cyclictest0-21swapper/823:41:1014
1526899277,18cyclictest969-21dbus-daemon22:27:0914
1526899276,10cyclictest0-21swapper/822:05:1214
15268992719,6cyclictest97950irq/42-eth0-TxR00:39:4914
1526599277,18cyclictest467-21jbd2/md125p1-800:16:4713
1526599277,18cyclictest467-21jbd2/md125p1-800:16:4713
1526599273,21cyclictest14502-21sshd21:55:2013
1526599273,18cyclictest101461sshd22:32:5613
1526599271,24cyclictest9890-21in:imuxsock00:35:0413
1526599270,24cyclictest969-21dbus-daemon00:13:1513
1526599270,24cyclictest0-21swapper/700:27:0413
1526599270,24cyclictest0-21swapper/700:27:0413
149612270,0sleep150-21swapper/1500:42:357
118652270,1sleep90-21swapper/900:22:0915
103222270,1sleep150-21swapper/1520:29:497
170372260,2sleep90-21swapper/900:10:0715
155142260,1sleep50-21swapper/523:55:3711
155142260,1sleep50-21swapper/523:55:3711
15270992623,2cyclictest961rcuc/1022:06:232
1526899268,15cyclictest17360-21nfsd21:56:5414
1526899266,8cyclictest82-21ksoftirqd/821:48:4614
1526899265,9cyclictest23516-21sshd23:08:5214
1526899261,14cyclictest32621-2110-uname22:03:2914
1526599269,14cyclictest1-21systemd00:41:1713
1526599266,12cyclictest0-21swapper/723:47:5713
1526599266,12cyclictest0-21swapper/723:47:5713
1526599265,18cyclictest0-21swapper/700:22:1413
1526599263,21cyclictest0-21swapper/723:24:4413
1526599263,17cyclictest512-21systemd-journal21:30:5213
1526599263,17cyclictest512-21systemd-journal21:30:5213
1526599262,21cyclictest6264-21sshd23:06:0513
1526599262,21cyclictest452-21sshd23:19:2913
1526599262,19cyclictest512-21systemd-journal23:14:3613
1526599261,20cyclictest131201cpuspeed_turbos20:34:3913
106132260,23sleep50-21swapper/522:52:1311
70972250,21sleep30-21swapper/300:20:279
6912250,1sleep100-21swapper/1021:34:422
303412250,21sleep60-21swapper/622:31:0612
262582250,21sleep100-21swapper/1022:39:592
262092250,21sleep130-21swapper/1321:48:085
217092250,0sleep90-21swapper/921:51:4815
180912250,21sleep150-21swapper/1521:37:287
172672250,21sleep60-21swapper/623:36:2212
172672250,21sleep60-21swapper/623:36:2212
1527599257,2cyclictest11213-21sshd19:54:046
1527599250,10cyclictest11857-21sshd20:33:046
1527099259,12cyclictest98-21ksoftirqd/1022:47:232
1526899258,7cyclictest0-21swapper/823:24:3314
1526899257,9cyclictest0-21swapper/800:30:1314
1526899256,9cyclictest0-21swapper/821:21:2514
1526899256,8cyclictest0-21swapper/823:37:2414
1526899256,8cyclictest0-21swapper/823:37:2414
1526899256,10cyclictest0-21swapper/821:53:3414
1526899255,12cyclictest0-21swapper/821:30:1614
1526899255,12cyclictest0-21swapper/821:30:1614
1526899255,12cyclictest0-21swapper/821:28:1414
15268992519,4cyclictest811ktimersoftd/822:14:4214
15268992513,5cyclictest1735950nfsd23:49:5114
15268992513,5cyclictest1735950nfsd23:49:5114
1526599256,17cyclictest0-21swapper/721:22:0513
1526599255,17cyclictest25691-21cp22:54:3013
1526599251,15cyclictest159361expr21:14:4913
1526599250,23cyclictest0-21swapper/720:29:4813
1526599250,22cyclictest0-21swapper/721:59:5313
1526399253,12cyclictest131rcu_preempt23:03:5911
1526099250,22cyclictest1-21systemd00:06:549
1525599250,2cyclictest0-21swapper/021:54:380
101472250,21sleep50-21swapper/523:35:1711
101472250,21sleep50-21swapper/523:35:1711
43402240,2sleep150-21swapper/1523:05:457
278182240,2sleep30-21swapper/322:35:199
274562240,1sleep100-21swapper/1000:06:562
254022240,1sleep3401ktimersoftd/321:20:089
253182240,1sleep90-21swapper/921:38:4315
1527099246,10cyclictest98-21ksoftirqd/1021:49:472
15270992419,3cyclictest0-21swapper/1022:56:112
15270992410,8cyclictest131rcu_preempt00:16:432
15270992410,8cyclictest131rcu_preempt00:16:432
1527099240,18cyclictest21989-21sshd23:13:182
1526899248,9cyclictest0-21swapper/800:12:5114
1526899248,8cyclictest0-21swapper/800:45:3114
1526899247,9cyclictest12809-21sshd21:41:3014
1526899247,9cyclictest12809-21sshd21:41:3014
1526899247,10cyclictest3506-21sshd00:26:1914
1526899247,10cyclictest3506-21sshd00:26:1914
1526899246,8cyclictest82-21ksoftirqd/823:22:1714
1526899245,8cyclictest82-21ksoftirqd/822:21:5614
1526899245,8cyclictest82-21ksoftirqd/800:18:2214
1526899245,8cyclictest82-21ksoftirqd/800:18:2214
1526899245,8cyclictest0-21swapper/800:09:2414
1526899245,17cyclictest2156-21sshd23:02:1614
1526899244,9cyclictest28505-21sshd22:54:4714
15268992417,3cyclictest0-21swapper/822:44:3614
1526599247,8cyclictest0-21swapper/722:34:4613
1526599246,16cyclictest0-21swapper/721:39:3713
1526599246,16cyclictest0-21swapper/721:39:3713
1526599242,13cyclictest170071sshd22:14:4213
1526599241,21cyclictest0-21swapper/720:02:3613
1526599240,22cyclictest0-21swapper/720:54:3713
1526599240,21cyclictest0-21swapper/720:24:3913
1526499241,2cyclictest0-21swapper/600:09:4112
1526399248,11cyclictest131rcu_preempt22:07:0111
15263992414,5cyclictest131rcu_preempt00:06:2011
1526399240,18cyclictest131rcu_preempt23:33:1411
1526399240,18cyclictest131rcu_preempt23:33:1411
15261992419,3cyclictest6322-21systemd22:23:0510
1526099244,14cyclictest0-21swapper/323:47:549
1526099244,14cyclictest0-21swapper/323:47:549
1526099242,12cyclictest0-21swapper/300:37:129
15260992418,4cyclictest41-21ksoftirqd/323:08:379
15260992411,7cyclictest131rcu_preempt22:14:389
15260992410,9cyclictest141rcu_sched21:39:129
15255992415,7cyclictest13809-21sshd23:21:130
1525599241,20cyclictest17359-21nfsd00:22:270
149042240,0sleep50-21swapper/522:57:3311
54822230,0sleep130-21swapper/1323:44:295
54822230,0sleep130-21swapper/1323:44:295
316392230,1sleep60-21swapper/620:15:1312
305382230,1sleep120-21swapper/1221:44:234
305382230,1sleep120-21swapper/1221:44:234
298232230,0sleep60-21swapper/621:53:0512
207952230,20sleep10-21swapper/122:58:341
191562230,1sleep110-21swapper/1123:41:163
191562230,1sleep110-21swapper/1123:41:163
156662230,0sleep60-21swapper/623:31:3212
156662230,0sleep60-21swapper/623:31:3212
1527599232,4cyclictest0-21swapper/1420:24:386
1527099236,16cyclictest971ktimersoftd/1023:20:202
1527099233,13cyclictest98-21ksoftirqd/1023:47:152
1527099233,13cyclictest98-21ksoftirqd/1023:47:152
15270992313,5cyclictest24018-21polkitd22:27:382
1527099231,11cyclictest98-21ksoftirqd/1021:45:342
15270992310,6cyclictest131rcu_preempt21:21:202
1526899236,9cyclictest82-21ksoftirqd/822:09:5814
1526899236,9cyclictest0-21swapper/823:31:2414
1526899236,9cyclictest0-21swapper/823:31:2414
1526899236,7cyclictest0-21swapper/822:54:0714
1526899236,6cyclictest0-21swapper/822:38:2214
1526899235,8cyclictest0-21swapper/822:31:0114
1526899231,11cyclictest0-21swapper/823:10:0714
1526599235,11cyclictest0-21swapper/719:44:4713
1526599234,17cyclictest0-21swapper/719:54:3213
1526599233,18cyclictest0-21swapper/720:19:3613
1526599232,19cyclictest0-21swapper/719:24:3813
1526599232,18cyclictest0-21swapper/720:44:2913
1526599231,20cyclictest0-21swapper/720:49:4613
1526599230,20cyclictest0-21swapper/721:14:2713
1526599230,20cyclictest0-21swapper/721:04:4713
15264992316,4cyclictest13375-21sd-pam21:46:1312
15263992313,6cyclictest26478-21systemd22:30:2911
1526099231,2cyclictest0-21swapper/321:09:419
15260992312,6cyclictest131rcu_preempt23:50:249
15260992312,6cyclictest131rcu_preempt23:50:249
15260992312,6cyclictest131rcu_preempt22:45:089
15260992311,5cyclictest141rcu_sched00:01:409
1526099230,21cyclictest512-21systemd-journal00:10:189
1526099230,17cyclictest41-21ksoftirqd/322:54:379
99142220,1sleep60-21swapper/623:25:4012
82042220,1sleep10-21swapper/100:09:111
77262220,1sleep10-21swapper/122:09:041
31902220,2sleep43281-21systemd-cgroups00:13:1510
309642220,19sleep40-21swapper/421:29:4810
309642220,19sleep40-21swapper/421:29:4810
269792220,1sleep100-21swapper/1019:31:582
1527899221,18cyclictest31804-21kworker/15:122:56:347
1527299222,4cyclictest145-17kauditd22:05:314
15271992216,4cyclictest32486-21run-parts22:45:543
1527099225,9cyclictest0-21swapper/1023:15:172
1527099223,14cyclictest98-21ksoftirqd/1000:26:462
1527099223,14cyclictest98-21ksoftirqd/1000:26:462
1527099223,10cyclictest0-21swapper/1021:56:302
15270992218,2cyclictest97250irq/34-eth0-TxR21:59:342
15270992211,6cyclictest98-21ksoftirqd/1023:29:172
1526999227,7cyclictest131rcu_preempt23:02:0215
15269992216,3cyclictest1-21systemd23:54:3815
15269992216,3cyclictest1-21systemd23:54:3815
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional