You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-21 - 23:18
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackcslot0.osadl.org (updated Thu Sep 21, 2023 00:43:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2129621760,7sleep21028199cyclictest22:53:138
729421330,0sleep70-21swapper/700:08:4413
96512122103,14sleep00-21swapper/019:22:430
2251921180,8sleep51028599cyclictest00:10:3511
2251921180,8sleep51028599cyclictest00:10:3511
95682117102,10sleep80-21swapper/819:21:3114
95982116103,8sleep40-21swapper/419:21:5910
386721160,6sleep121029599cyclictest00:39:194
258721150,1sleep40-21swapper/423:31:3010
9541211390,17sleep30-21swapper/319:21:109
2156621110,1sleep221570-21nfsd423:29:208
9466210991,14sleep20-21swapper/219:20:048
1381221090,0sleep10-21swapper/100:23:051
1381221090,0sleep10-21swapper/100:23:051
3175021080,2sleep20-21swapper/222:27:168
9773210793,10sleep120-21swapper/1219:23:434
419321020,2sleep104192-21sshd23:44:592
1194421000,1sleep90-21swapper/923:59:4115
221992970,5sleep111029499cyclictest20:54:203
138172960,6sleep151029999cyclictest21:49:307
138172960,6sleep151029999cyclictest21:49:307
87372950,0sleep60-21swapper/600:29:0312
316182950,6sleep01027899cyclictest23:30:540
222962950,4sleep81029199cyclictest21:54:4414
222962950,4sleep81029199cyclictest21:54:4414
303902940,1sleep140-21swapper/1400:02:266
111022940,1sleep120-21swapper/1223:32:594
223852930,0sleep30-21swapper/323:52:379
72512920,0sleep140-21swapper/1422:14:546
157262920,1sleep50-21swapper/519:29:1211
157262920,1sleep50-21swapper/519:29:1211
133842920,6sleep41028399cyclictest21:31:5610
19862900,0sleep10-21swapper/123:22:051
181932900,1sleep20-21swapper/223:42:428
972928972,4sleep60-21swapper/619:23:0512
946528972,4sleep10-21swapper/119:20:031
945228974,11sleep70-21swapper/719:19:5313
185612890,1sleep70-21swapper/700:50:4413
139212870,0sleep110-21swapper/1123:59:593
964828675,6sleep140-21swapper/1419:22:416
949828672,9sleep130-21swapper/1319:20:315
103412860,1sleep00-21swapper/023:04:570
946228572,9sleep150-21swapper/1519:20:017
281372850,0sleep150-21swapper/1522:58:477
971528375,4sleep110-21swapper/1119:22:543
958728372,6sleep100-21swapper/1019:21:492
27562820,0sleep100-21swapper/1022:14:182
944928173,4sleep50-21swapper/519:19:5111
629528072,4sleep90-21swapper/919:18:5115
110032760,0sleep10-21swapper/123:59:331
210442750,1sleep20-21swapper/200:31:248
66842700,0sleep20-21swapper/223:14:058
82072690,3sleep341-21ksoftirqd/300:13:139
82072690,3sleep341-21ksoftirqd/300:13:139
111842610,1sleep80-21swapper/823:05:0514
215802460,1sleep00-21swapper/021:37:130
53112440,1sleep50-21swapper/523:09:0311
1029199423,36cyclictest17360-21nfsd00:39:5514
110682410,1sleep1211070-21systemd-cgroups22:42:414
10291994013,25cyclictest0-21swapper/821:52:4314
10291994013,25cyclictest0-21swapper/821:52:4314
88062390,0sleep50-21swapper/522:38:1711
1029299398,4cyclictest1676-21sshd22:37:1215
1029199391,36cyclictest0-21swapper/822:44:3214
10292993812,2cyclictest2156-21sshd00:39:2515
10291993810,26cyclictest0-21swapper/821:32:3114
28537-21370,1sshd0-21swapper/421:25:2510
221152370,1sleep110-21swapper/1121:24:333
134442370,4sleep9881rcuc/919:24:2715
134442370,4sleep9881rcuc/919:24:2715
10292993710,3cyclictest24018-21polkitd21:23:5315
1029199377,23cyclictest103041updatedb.mlocat00:22:5014
1029199377,23cyclictest103041updatedb.mlocat00:22:5014
1029199373,31cyclictest15786-21cp23:55:5714
10291993720,15cyclictest0-21swapper/822:43:1414
10291993715,20cyclictest0-21swapper/822:06:5814
1029199371,33cyclictest26085-21sshd00:51:5214
10291993710,20cyclictest0-21swapper/821:45:5214
10291993710,19cyclictest0-21swapper/823:52:5814
10291993710,18cyclictest173621nfsd23:21:0714
75532360,1sleep90-21swapper/922:55:4515
50682360,1sleep20-21swapper/200:21:148
50682360,1sleep20-21swapper/200:21:148
10292993611,2cyclictest26748-21sshd00:38:1315
1029199369,25cyclictest0-21swapper/800:13:2914
1029199369,25cyclictest0-21swapper/800:13:2914
1029199368,20cyclictest0-21swapper/800:36:1914
1029199367,27cyclictest0-21swapper/822:36:5014
1029199367,26cyclictest0-21swapper/820:53:5014
1029199366,28cyclictest28944-21sshd21:38:2414
1029199365,29cyclictest24561-21systemd-cgroups22:13:0314
1029199364,23cyclictest0-21swapper/823:58:5714
10291993615,18cyclictest2962-21sshd23:18:0114
10291993612,21cyclictest969-21dbus-daemon22:17:3314
10291993612,21cyclictest0-21swapper/823:23:5214
10291993611,23cyclictest9595-21sshd00:44:4914
10291993611,22cyclictest1061-21gdbus23:03:1814
10291993610,22cyclictest0-21swapper/823:46:2114
265932350,0sleep60-21swapper/623:43:5112
1029199359,23cyclictest24610-21kworker/8:222:32:3714
1029199359,23cyclictest0-21swapper/820:04:1114
1029199358,24cyclictest969-21dbus-daemon21:59:3214
1029199358,24cyclictest969-21dbus-daemon21:59:3214
1029199357,25cyclictest0-21swapper/800:29:4814
1029199356,27cyclictest0-21swapper/822:21:2814
10291993528,5cyclictest0-21swapper/822:24:3114
10291993520,11cyclictest1082-21gdbus21:24:2214
10291993513,20cyclictest0-21swapper/820:14:1314
10291993512,21cyclictest0-21swapper/821:41:1414
10291993512,20cyclictest0-21swapper/819:24:1214
10291993512,20cyclictest0-21swapper/819:24:1214
10291993511,22cyclictest0-21swapper/820:44:2914
10291993511,20cyclictest696-21sshd23:31:1114
10291993510,23cyclictest0-21swapper/822:49:3114
92482340,1sleep150-21swapper/1500:04:037
136052342,3sleep1222104-21systemd-logind22:47:154
1029199347,25cyclictest0-21swapper/800:14:2814
1029199347,25cyclictest0-21swapper/800:14:2814
1029199347,22cyclictest31020-21sshd23:35:3214
1029199345,27cyclictest0-21swapper/820:54:2114
10291993415,17cyclictest97450irq/37-eth0-TxR00:24:3714
10291993414,18cyclictest0-21swapper/822:54:3414
10291993412,19cyclictest1-21systemd00:04:5714
10291993411,17cyclictest0-21swapper/823:40:4414
10291993410,22cyclictest0-21swapper/820:59:5914
59432330,1sleep20-21swapper/221:48:458
318672330,28sleep60-21swapper/622:49:3312
291902330,7sleep31028299cyclictest21:34:039
284312330,0sleep60-21swapper/600:06:5112
106792330,0sleep10-21swapper/100:09:101
106792330,0sleep10-21swapper/100:09:101
1029199339,22cyclictest21125-21sshd23:11:1514
1029199339,21cyclictest0-21swapper/821:08:5414
10291993320,11cyclictest0-21swapper/820:29:1414
10291993313,18cyclictest0-21swapper/820:39:2714
10291993313,18cyclictest0-21swapper/820:25:0714
10291993312,19cyclictest0-21swapper/819:29:2714
10291993312,19cyclictest0-21swapper/819:29:2714
10281993311,19cyclictest18941-21uname21:54:198
10281993311,19cyclictest18941-21uname21:54:198
116802320,1sleep40-21swapper/400:40:1610
1029199322,28cyclictest0-21swapper/819:48:5214
10291993214,16cyclictest0-21swapper/819:34:2114
10291993214,16cyclictest0-21swapper/819:34:2114
10291993214,15cyclictest0-21swapper/819:59:1414
10291993212,18cyclictest0-21swapper/819:44:2914
10291993211,19cyclictest0-21swapper/821:04:0714
10291993211,19cyclictest0-21swapper/820:19:5914
307792310,0sleep120-21swapper/1200:20:194
307792310,0sleep120-21swapper/1200:20:194
29922312,25sleep422104-21systemd-logind22:32:2410
246142310,1sleep50-21swapper/500:25:0211
105152310,1sleep100-21swapper/1021:31:312
1029299310,13cyclictest22104-21systemd-logind22:16:3415
1029199317,22cyclictest0-21swapper/820:09:0714
10291993110,19cyclictest0-21swapper/820:34:1514
10291993110,19cyclictest0-21swapper/819:39:1114
10291993110,19cyclictest0-21swapper/819:39:1114
319972300,2sleep120-21swapper/1221:43:354
282932300,2sleep1128287-21sshd22:40:323
173682300,1sleep0111ktimersoftd/020:09:140
158352300,25sleep130-21swapper/1300:00:185
1029199306,22cyclictest0-21swapper/821:19:0114
1029199305,23cyclictest0-21swapper/819:54:3414
10291993010,18cyclictest0-21swapper/821:14:3014
1028199300,23cyclictest14803-21sshd22:34:108
1028199300,22cyclictest0-21swapper/200:34:358
78262290,1sleep110-21swapper/1122:55:483
246412290,2sleep100-21swapper/1023:02:442
241902290,0sleep60-21swapper/623:57:1312
231742290,0sleep10-21swapper/121:37:301
10292992911,3cyclictest4668-21run-parts21:34:4915
10292992910,3cyclictest23941-21sshd23:48:1415
1028599296,5cyclictest131rcu_preempt21:34:0311
1028199298,18cyclictest0-21swapper/221:42:338
1028199296,5cyclictest33-21ksoftirqd/223:04:388
1028199295,6cyclictest0-21swapper/200:01:068
1028199290,22cyclictest1-21systemd22:43:098
1028199290,17cyclictest11486110-uname23:55:198
50362280,1sleep10-21swapper/121:39:221
48292280,25sleep60-21swapper/623:04:1612
300562280,24sleep100-21swapper/1021:34:112
248272280,2sleep15138-21ksoftirqd/1523:16:267
198952280,1sleep140-21swapper/1421:36:566
1029299280,13cyclictest0-21swapper/923:50:5915
10285992811,8cyclictest131rcu_preempt23:17:3511
1028199288,10cyclictest321ktimersoftd/221:50:238
1028199288,10cyclictest321ktimersoftd/221:50:238
1028199286,10cyclictest0-21swapper/222:45:598
10281992810,16cyclictest0-21swapper/223:18:518
8842270,2sleep70-21swapper/723:35:5013
6685-21270,1sshd0-21swapper/722:59:5613
249292270,1sleep30-21swapper/323:07:109
227102270,2sleep922709-21sshd23:11:3215
215152270,2sleep721565-21sshd22:35:0813
136172270,1sleep70-21swapper/723:42:0013
1029299279,4cyclictest19329-21run-parts21:54:2115
1029299279,4cyclictest19329-21run-parts21:54:2115
10287992721,4cyclictest18316-21sshd00:24:0813
1028199277,11cyclictest33-21ksoftirqd/223:03:198
1028199273,22cyclictest0-21swapper/222:14:088
10281992722,3cyclictest311rcuc/223:47:438
1028199272,22cyclictest17781-21sshd22:07:518
1028199271,24cyclictest0-21swapper/200:09:368
1028199271,24cyclictest0-21swapper/200:09:368
1028199271,15cyclictest1-21systemd23:09:328
10281992710,14cyclictest15810-21run-parts21:32:178
76002260,2sleep57692-21sshd22:33:1311
47442260,22sleep50-21swapper/522:28:0811
273832260,1sleep40-21swapper/400:02:0110
270292260,22sleep30-21swapper/300:52:019
257452260,2sleep12969-21dbus-daemon22:13:174
253302260,0sleep00-21swapper/022:22:090
247212260,1sleep100-21swapper/1023:43:392
161382260,22sleep50-21swapper/522:43:2311
137302260,1sleep1241ktimersoftd/121:27:521
13452260,1sleep100-21swapper/1021:26:092
1029999268,9cyclictest19064-21sshd22:21:187
1029999264,13cyclictest138-21ksoftirqd/1500:53:447
10285992613,6cyclictest131rcu_preempt21:29:2911
10285992610,6cyclictest131rcu_preempt23:05:2811
1028399261,22cyclictest17157-2110-uname21:41:1810
1028399261,20cyclictest0-21swapper/419:29:1110
1028399261,20cyclictest0-21swapper/419:29:1110
1028199267,10cyclictest0-21swapper/223:34:048
1028199266,11cyclictest33-21ksoftirqd/222:20:458
1028199265,18cyclictest25688-21run-parts22:30:508
1028199264,18cyclictest3489-21bash22:01:398
1028199264,18cyclictest3489-21bash22:01:398
1028199263,6cyclictest0-21swapper/200:42:428
1028199263,20cyclictest26538-21sshd22:54:018
10281992612,5cyclictest969-21dbus-daemon21:34:048
1028199260,23cyclictest16105-21sshd23:51:438
1028099260,23cyclictest0-21swapper/100:04:091
59422250,1sleep130-21swapper/1322:19:275
50392250,1sleep10961rcuc/1000:49:112
49312250,0sleep30-21swapper/323:59:029
324212250,2sleep30-21swapper/300:16:119
324212250,2sleep30-21swapper/300:16:119
281532250,21sleep130-21swapper/1300:52:115
265852250,1sleep30-21swapper/321:33:479
189732250,1sleep130-21swapper/1320:49:175
162052250,1sleep130-21swapper/1321:49:445
162052250,1sleep130-21swapper/1321:49:445
135082250,1sleep60-21swapper/623:41:5912
1029999257,12cyclictest138-21ksoftirqd/1500:37:177
1029999256,12cyclictest138-21ksoftirqd/1522:59:007
1029999256,10cyclictest0-21swapper/1523:21:267
1029999255,18cyclictest1371ktimersoftd/1500:18:557
1029999255,18cyclictest1371ktimersoftd/1500:18:557
1029999255,12cyclictest138-21ksoftirqd/1522:43:427
10299992514,5cyclictest131rcu_preempt23:35:167
1029999251,18cyclictest138-21ksoftirqd/1522:17:557
1029899251,22cyclictest0-21swapper/1400:41:596
1028599259,8cyclictest131rcu_preempt00:21:0711
1028599259,8cyclictest131rcu_preempt00:21:0711
1028599254,6cyclictest58-21ksoftirqd/522:19:5611
10285992512,7cyclictest141rcu_sched00:32:0111
10285992511,8cyclictest131rcu_preempt22:50:2511
1028199256,11cyclictest0-21swapper/222:08:578
1028199253,19cyclictest13385-21run-parts00:45:268
1028199252,21cyclictest0-21swapper/200:24:298
1028199252,20cyclictest2556-21needreboot20:29:168
10281992516,7cyclictest321ktimersoftd/223:23:538
1028199251,21cyclictest0-21swapper/200:16:038
1028199251,21cyclictest0-21swapper/200:16:038
1028199251,20cyclictest20198-21run-parts00:05:278
1028199250,22cyclictest512-21systemd-journal00:50:308
1028199250,22cyclictest31757-1kworker/2:1H21:24:498
1027899251,21cyclictest12089-21avahi-daemon21:41:380
89812240,20sleep40-21swapper/422:46:3510
55482240,1sleep40-21swapper/423:18:2510
320122240,1sleep40-21swapper/422:27:1910
232802240,2sleep150-21swapper/1500:01:247
210082240,1sleep70-21swapper/700:46:4413
184152240,1sleep30-21swapper/323:56:239
150102240,1sleep60-21swapper/600:23:2712
150102240,1sleep60-21swapper/600:23:2712
1029999247,8cyclictest138-21ksoftirqd/1500:13:307
1029999247,8cyclictest138-21ksoftirqd/1500:13:307
1029999246,9cyclictest0-21swapper/1521:27:527
1029999246,8cyclictest138-21ksoftirqd/1522:46:417
1029999246,13cyclictest138-21ksoftirqd/1523:46:457
1029999245,14cyclictest138-21ksoftirqd/1522:08:337
1029999245,10cyclictest138-21ksoftirqd/1521:48:327
10299992421,2cyclictest1361rcuc/1500:43:577
10299992417,4cyclictest1-21systemd22:50:567
10299992412,7cyclictest131rcu_preempt23:33:097
10299992412,6cyclictest141rcu_sched22:02:067
10299992412,6cyclictest141rcu_sched22:02:067
10299992412,6cyclictest131rcu_preempt23:50:507
1029899241,20cyclictest1-21systemd22:49:066
10295992419,3cyclictest24022-21gdbus22:23:424
1029299249,4cyclictest857-21sshd23:03:5115
1028599249,9cyclictest131rcu_preempt21:43:1111
1028599248,9cyclictest25224-21systemd-cgroups00:37:5811
1028599246,9cyclictest131rcu_preempt23:54:0011
1028599246,4cyclictest141rcu_sched22:06:3811
1028599243,4cyclictest0-21swapper/523:20:2111
10285992411,8cyclictest131rcu_preempt21:59:3111
10285992411,8cyclictest131rcu_preempt21:59:3111
10285992411,7cyclictest141rcu_sched23:02:2911
10285992411,7cyclictest131rcu_preempt23:25:5111
10285992411,6cyclictest141rcu_sched00:39:3911
10285992410,8cyclictest141rcu_sched23:58:5711
10285992410,6cyclictest141rcu_sched00:44:4411
1028299249,8cyclictest131rcu_preempt00:21:199
1028299249,8cyclictest131rcu_preempt00:21:199
1028199245,16cyclictest2599-21latency_hist19:53:498
1028199244,18cyclictest0-21swapper/219:23:508
1028199244,18cyclictest0-21swapper/219:23:508
1028199244,17cyclictest0-21swapper/221:01:598
1028199242,20cyclictest0-21swapper/221:06:438
100662240,1sleep30-21swapper/319:23:509
100662240,1sleep30-21swapper/319:23:509
318732230,19sleep130-21swapper/1300:43:385
303502230,1sleep140-21swapper/1422:59:036
286862230,0sleep140-21swapper/1422:40:356
266822230,1sleep150-21swapper/1519:43:497
266822230,1sleep150-21swapper/1519:43:497
192442230,0sleep100-21swapper/1000:18:572
192442230,0sleep100-21swapper/1000:18:572
176352230,1sleep00-21swapper/021:59:100
176352230,1sleep00-21swapper/021:59:100
175832230,1sleep017671-21sshd22:57:130
174552230,19sleep30-21swapper/320:48:529
172572230,0sleep40-21swapper/400:04:5710
133172230,2sleep70-21swapper/721:45:0613
117972230,1sleep00-21swapper/021:27:350
1029999239,7cyclictest131rcu_preempt21:42:477
1029999237,9cyclictest138-21ksoftirqd/1523:41:107
1029999237,8cyclictest138-21ksoftirqd/1521:34:017
1029999237,7cyclictest138-21ksoftirqd/1522:09:057
1029999236,10cyclictest138-21ksoftirqd/1521:56:067
1029999236,10cyclictest138-21ksoftirqd/1521:56:067
1029999235,10cyclictest138-21ksoftirqd/1523:56:597
10299992311,7cyclictest131rcu_preempt23:11:227
10298992317,3cyclictest969-21dbus-daemon23:38:596
10298992317,3cyclictest1082-21gdbus23:09:516
10297992310,7cyclictest131rcu_preempt23:57:315
1029799230,6cyclictest141rcu_sched23:02:345
10295992318,3cyclictest24022-21gdbus22:30:104
10295992316,4cyclictest9705-21sh22:15:174
10287992318,3cyclictest9885-21systemd-cgroups22:24:1513
1028599239,9cyclictest131rcu_preempt00:15:5111
1028599239,9cyclictest131rcu_preempt00:15:5111
1028599239,8cyclictest131rcu_preempt22:46:5811
1028599239,7cyclictest141rcu_sched23:50:0911
1028599239,7cyclictest141rcu_sched21:47:4411
1028599239,7cyclictest131rcu_preempt21:51:0711
1028599239,7cyclictest131rcu_preempt21:51:0711
1028599239,7cyclictest131rcu_preempt21:27:5211
1028599239,6cyclictest131rcu_preempt23:42:0511
1028599238,9cyclictest131rcu_preempt00:51:5811
10285992310,7cyclictest131rcu_preempt23:34:5611
10285992310,7cyclictest131rcu_preempt23:30:2611
1028299239,7cyclictest41-21ksoftirqd/323:34:299
1028199239,11cyclictest6743-21nfsd420:34:218
1028199235,16cyclictest0-21swapper/220:49:188
1027899231,20cyclictest17360-21nfsd00:50:490
6752220,1sleep110-21swapper/1100:52:583
37582220,20sleep130-21swapper/1320:29:315
301472220,1sleep140-21swapper/1423:17:156
263112220,1sleep40-21swapper/400:11:0710
263112220,1sleep40-21swapper/400:11:0710
250032220,1sleep30-21swapper/323:20:489
238292220,1sleep90-21swapper/923:20:3815
23712220,18sleep40-21swapper/422:41:3010
236742220,2sleep60-21swapper/621:37:3512
1029999227,8cyclictest138-21ksoftirqd/1500:16:397
1029999227,8cyclictest138-21ksoftirqd/1500:16:397
1029999227,14cyclictest0-21swapper/1522:25:587
1029999225,10cyclictest0-21swapper/1521:32:117
1029999224,9cyclictest138-21ksoftirqd/1522:35:587
1029999224,10cyclictest138-21ksoftirqd/1500:25:517
1029999223,9cyclictest969-21dbus-daemon00:32:467
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional