You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-11 - 04:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack0slot0.osadl.org (updated Mon Dec 11, 2023 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2303221830,3sleep6560999cyclictest00:38:246
2963921810,2sleep3559299cyclictest23:58:383
2634721800,2sleep6560999cyclictest00:24:296
3224121760,1sleep4559799cyclictest00:06:154
3224121760,1sleep4559799cyclictest00:06:154
365021730,3sleep3559299cyclictest00:03:103
365021730,3sleep3559299cyclictest00:03:103
672721620,1sleep71-21systemd22:18:527
709021600,5sleep1558099cyclictest23:06:091
709021600,5sleep1558099cyclictest23:06:081
2697521570,4sleep0557399cyclictest23:05:430
2697521570,4sleep0557399cyclictest23:05:430
2245321550,3sleep6560999cyclictest22:18:196
1864321550,1sleep618640-21sshd22:54:446
1662221370,1sleep2311rcuc/222:54:392
238621340,0sleep10-21swapper/100:06:221
238621340,0sleep10-21swapper/100:06:211
488721230,0sleep50-21swapper/521:45:325
2770621230,0sleep40-21swapper/421:48:274
328721220,1sleep13283-21sshd00:16:081
1335021210,2sleep2558599cyclictest23:54:512
3060821150,1sleep430611-21systemd-cgroups00:20:194
2840921140,4sleep0557399cyclictest00:00:430
2840921140,4sleep0557399cyclictest00:00:420
2880421130,1sleep778-21ksoftirqd/722:02:267
2880421130,1sleep778-21ksoftirqd/722:02:257
907021100,1sleep29065-21sshd00:11:592
972521080,4sleep2558599cyclictest23:13:482
937421070,1sleep00-21swapper/023:36:210
2406121060,2sleep024065-21bash00:23:190
1211521060,1sleep60-21swapper/600:33:436
2441821050,5sleep5560399cyclictest23:49:495
1583621050,1sleep1221rcuc/123:02:081
1583621050,1sleep1221rcuc/123:02:071
2880421040,1sleep07-21ksoftirqd/023:34:530
2706521020,0sleep30-21swapper/323:08:593
2706521020,0sleep30-21swapper/323:08:593
280592990,1sleep328025-21sshd00:16:573
212582990,0sleep70-21swapper/723:10:557
156032990,3sleep215588-21sshd23:03:122
156032990,3sleep215588-21sshd23:03:122
146032990,2sleep6666-21dbus-daemon23:19:216
245512980,1sleep024537-21sshd00:35:140
186162980,1sleep618590-21sshd23:25:546
70302960,1sleep17017-21sshd23:48:091
126982960,1sleep60-21swapper/622:34:076
64622930,0sleep70-21swapper/700:27:047
321902920,1sleep532186-21sshd22:34:465
293912920,1sleep629392-21systemd-cgroups21:50:386
176262920,1sleep317618-21sshd00:38:143
105532920,0sleep70-21swapper/723:43:587
270602910,2sleep50-21swapper/521:10:545
250862890,1sleep5581rcuc/523:25:045
184852890,1sleep618487-21grepconf.sh22:20:206
99402880,1sleep69941-21sshd00:25:026
264042880,1sleep60-21swapper/621:36:356
70872850,1sleep37078-21sshd22:38:123
239782850,0sleep10-21swapper/121:35:261
194792850,0sleep719480-21sshd21:26:427
59302840,0sleep50-21swapper/500:02:095
59302840,0sleep50-21swapper/500:02:085
298942840,0sleep2321ktimersoftd/223:55:232
290052840,0sleep10-21swapper/122:49:411
270492840,0sleep4491rcuc/422:45:194
191962840,1sleep319193-21sshd22:42:533
70762830,0sleep40-21swapper/423:12:364
64242830,1sleep10-21swapper/122:37:071
214242830,3sleep7561599cyclictest23:29:167
197012830,2sleep00-21swapper/022:25:440
131582830,1sleep313152-21bash21:12:353
87792820,0sleep10-21swapper/122:05:001
87792820,0sleep10-21swapper/122:05:001
96812810,1sleep2702-21gdbus22:07:092
96812810,1sleep2702-21gdbus22:07:092
77242810,1sleep70-21swapper/722:50:037
266662810,0sleep10-21swapper/123:31:331
204332810,0sleep10-21swapper/123:21:391
96962800,1sleep10-21swapper/121:53:131
311182800,0sleep30-21swapper/322:01:253
311182800,0sleep30-21swapper/322:01:253
26972800,1sleep52642-21sshd22:53:065
322662780,1sleep00-21swapper/022:20:460
266002780,2sleep7561599cyclictest00:15:497
122832780,0sleep312282-21systemd-cgroups21:38:163
321252770,0sleep1221rcuc/121:28:121
26332770,0sleep20-21swapper/221:51:542
160722770,1sleep30-21swapper/322:32:043
68092760,0sleep50-21swapper/522:50:015
299382760,1sleep229904-21sshd23:15:312
256292760,0sleep30-21swapper/323:41:133
98672750,0sleep50-21swapper/523:01:555
98672750,0sleep50-21swapper/523:01:555
84562750,0sleep50-21swapper/521:51:015
196332750,1sleep619632-21grep00:19:576
177432750,0sleep00-21swapper/022:34:170
8172740,0sleep70-21swapper/723:50:087
77042740,4sleep4559799cyclictest23:50:204
69602740,0sleep70-21swapper/721:35:567
286142740,1sleep628615-21sshd22:38:566
237582740,1sleep2321ktimersoftd/221:46:092
176542740,0sleep40-21swapper/421:12:444
242292730,0sleep30-21swapper/300:30:523
15722730,2sleep4559799cyclictest23:39:224
58322720,7sleep45835-21bash23:41:384
297152720,0sleep60-21swapper/622:29:186
263692720,1sleep2311rcuc/222:39:562
209182720,0sleep70-21swapper/700:21:027
245072710,0sleep20-21swapper/221:35:272
234782710,1sleep669-21ksoftirqd/623:20:406
229222710,0sleep30-21swapper/321:56:523
112992710,1sleep311289-21sshd21:47:543
53092700,1sleep07-21ksoftirqd/000:18:220
39332700,5sleep1558099cyclictest21:30:301
39122700,1sleep53874-21sshd23:11:255
68242690,0sleep00-21swapper/021:33:490
284722690,0sleep60-21swapper/622:47:296
160952690,0sleep10-21swapper/123:19:241
141842690,1sleep0557399cyclictest00:31:380
533826847,6sleep30-21swapper/319:09:393
259252680,0sleep70-21swapper/723:06:477
259252680,0sleep70-21swapper/723:06:467
15572680,1sleep778-21ksoftirqd/722:32:397
74542670,1sleep30-21swapper/321:51:003
106442630,3sleep5560399cyclictest00:37:595
240252620,0sleep50-21swapper/500:07:065
240252620,0sleep50-21swapper/500:07:055
204232620,3sleep220428-21id21:40:412
64542610,0sleep20-21swapper/223:29:502
310802610,1sleep6666-21dbus-daemon21:12:066
26752600,1sleep01107-21nfsd23:24:190
162522600,1sleep216251-21sshd22:14:552
25132590,1sleep32517-21grepconf.sh23:12:283
224742590,2sleep4711-21systemd-logind23:20:384
45072530,1sleep51-21systemd23:07:095
45072530,1sleep51-21systemd23:07:085
36032530,10sleep280550irq/25-eno1-rx-23:06:022
36032530,10sleep280550irq/25-eno1-rx-23:06:012
516425242,6sleep70-21swapper/719:07:267
184882520,0sleep50-21swapper/522:12:495
215742510,0sleep10-21swapper/123:56:101
532125040,6sleep40-21swapper/419:09:244
509525028,7sleep60-21swapper/619:06:376
535424939,6sleep10-21swapper/119:09:451
532624837,7sleep00-21swapper/019:09:280
148022480,2sleep20-21swapper/200:06:472
148022480,2sleep20-21swapper/200:06:462
173832470,0sleep60-21swapper/600:05:466
173832470,0sleep60-21swapper/600:05:456
111582470,1sleep3401rcuc/300:20:433
250502460,0sleep3559299cyclictest23:19:413
112202460,1sleep611221-21tty23:38:366
333124535,6sleep50-21swapper/519:05:085
225992450,1sleep10-21swapper/100:31:541
108932450,1sleep610895-21kworker/u16:022:09:226
108932450,1sleep610895-21kworker/u16:022:09:226
43552440,1sleep0557399cyclictest23:54:340
290202440,0sleep70-21swapper/700:05:067
290202440,0sleep70-21swapper/700:05:067
205122440,0sleep60-21swapper/623:30:186
194712440,0sleep719469-21bash23:18:257
10802440,0sleep40-21swapper/423:59:514
58132430,0sleep30-21swapper/323:54:363
248672430,3sleep4559799cyclictest21:15:094
230972430,1sleep3401rcuc/300:07:043
230972430,1sleep3401rcuc/300:07:033
174442430,2sleep517442-21bash21:41:395
57642420,0sleep60-21swapper/600:14:036
522624231,7sleep20-21swapper/219:08:092
324802410,0sleep10-21swapper/100:38:431
291122410,1sleep4491rcuc/400:13:464
18152410,0sleep20-21swapper/223:37:122
73212400,0sleep40-21swapper/422:01:434
73212400,0sleep40-21swapper/422:01:424
249112390,0sleep70-21swapper/722:05:307
249112390,0sleep70-21swapper/722:05:307
314022380,0sleep4559799cyclictest00:36:304
205542380,0sleep00-21swapper/022:04:190
205542380,0sleep00-21swapper/022:04:190
110192370,0sleep30-21swapper/323:29:593
215192360,1sleep421474-21sshd23:19:344
103932360,4sleep1558099cyclictest00:29:221
130432320,1sleep50-21swapper/521:35:065
26942300,0sleep10-21swapper/100:03:091
26942300,0sleep10-21swapper/100:03:081
307192290,1sleep030715-21bash00:27:520
111632290,0sleep40-21swapper/422:51:124
160132280,0sleep20-21swapper/200:25:142
39582270,1sleep03949-21sshd21:20:510
28982260,0sleep70-21swapper/723:01:427
28982260,0sleep70-21swapper/723:01:417
160192260,1sleep716013-21sshd00:34:577
147742250,1sleep7761rcuc/721:33:007
134502250,1sleep713444-21sshd22:37:217
7392240,1sleep3672-21sshd21:15:243
38202240,0sleep00-21swapper/022:16:370
280442240,0sleep1221rcuc/122:22:471
216852240,1sleep1221rcuc/121:57:551
167292240,1sleep20-21swapper/222:57:512
40162230,1sleep24000-21rm23:46:582
156862230,1sleep560-21ksoftirqd/522:06:175
156862230,1sleep560-21ksoftirqd/522:06:165
116682230,1sleep10-21swapper/122:19:021
192232220,3sleep6560999cyclictest23:41:016
558599211,2cyclictest666-21dbus-daemon22:04:582
558599211,2cyclictest666-21dbus-daemon22:04:572
558599210,19cyclictest0-21swapper/200:20:182
201222210,1sleep60-21swapper/623:56:076
561599205,4cyclictest9116-21sshd22:56:317
560999204,4cyclictest666-21dbus-daemon21:31:546
560999201,8cyclictest31553-21bash22:40:066
560999200,1cyclictest0-21swapper/623:50:566
560999200,1cyclictest0-21swapper/623:48:036
560399200,1cyclictest1110-21nfsd22:41:405
5585992010,7cyclictest10815-21sshd22:45:502
558599200,2cyclictest0-21swapper/200:35:542
558599200,1cyclictest15722-21sshd22:30:572
260082200,1sleep40-21swapper/420:55:384
22722200,0sleep00-21swapper/022:35:540
200382200,1sleep020015-21sshd22:12:520
197662200,0sleep30-21swapper/323:31:193
151682200,0sleep50-21swapper/500:19:475
561599192,5cyclictest11117-21gdbus21:50:237
560399193,13cyclictest0-21swapper/520:59:245
558099195,3cyclictest24-21ksoftirqd/123:14:081
45952190,2sleep44566-21sshd22:24:114
326542190,0sleep4491rcuc/423:25:194
269772190,0sleep10-21swapper/122:32:251
261822190,1sleep126172-21sshd23:52:021
245162190,1sleep424510-21sshd23:03:314
245162190,1sleep424510-21sshd23:03:304
187842190,1sleep618782-21sshd21:17:036
173732190,1sleep4491rcuc/421:36:174
143792190,1sleep014380-21sshd21:46:560
101872190,0sleep20-21swapper/221:57:312
77512180,1sleep0121rcuc/022:57:330
561599184,3cyclictest0-21swapper/721:18:047
561599183,4cyclictest1937-21grepconf.sh22:40:127
561599183,2cyclictest728-21NetworkManager21:55:417
561599182,4cyclictest23108-21sshd23:45:277
561599180,16cyclictest29433-21sshd22:12:077
560999183,7cyclictest0-21swapper/600:00:296
560999183,7cyclictest0-21swapper/600:00:286
560399180,16cyclictest20539-21sshd23:21:395
5592991816,1cyclictest30516-21sshd00:27:523
558599180,9cyclictest13532-21grep00:30:312
558599180,1cyclictest1671-21bash21:33:382
558599180,1cyclictest16336-21sshd21:16:592
558599180,1cyclictest0-21swapper/221:23:252
558099180,16cyclictest666-21dbus-daemon00:21:581
42352180,1sleep3401rcuc/300:14:013
231122180,1sleep4711-21systemd-logind21:53:404
226992180,1sleep522644-21sshd21:16:065
150892180,0sleep10-21swapper/121:42:391
123072180,0sleep10-21swapper/121:16:501
8112170,0sleep60-21swapper/621:47:346
561599175,5cyclictest664-21polkitd00:35:587
561599173,4cyclictest12164-21bash22:26:327
561599172,3cyclictest26953-21sshd22:20:357
561599170,15cyclictest8148-21sshd23:30:567
560999173,7cyclictest0-21swapper/622:11:446
558599179,7cyclictest15700-21bash21:10:312
5585991710,6cyclictest0-21swapper/221:28:322
558599170,2cyclictest21019-21grepconf.sh23:24:562
557399170,14cyclictest679-21gdbus23:00:520
557399170,14cyclictest679-21gdbus23:00:510
304632170,1sleep530452-21bash21:21:435
234942170,0sleep30-21swapper/323:04:343
234942170,0sleep30-21swapper/323:04:343
156942170,1sleep715655-21sshd23:58:097
155082170,0sleep10-21swapper/122:59:591
126352170,0sleep00-21swapper/023:56:580
111362170,0sleep50-21swapper/520:25:275
561599165,4cyclictest8179-21nscd21:45:107
561599164,5cyclictest718-21avahi-daemon21:10:377
561599164,4cyclictest78-21ksoftirqd/721:40:067
561599163,3cyclictest14584-21sshd00:10:027
561599162,5cyclictest78-21ksoftirqd/700:00:417
561599162,5cyclictest78-21ksoftirqd/700:00:407
561599162,4cyclictest78-21ksoftirqd/722:48:187
561599161,2cyclictest78-21ksoftirqd/721:20:167
560999162,4cyclictest27165-21sshd21:23:476
560999160,14cyclictest0-21swapper/622:03:296
560999160,14cyclictest0-21swapper/622:03:296
5603991612,3cyclictest7068-21bash23:42:475
5603991612,3cyclictest6362-21bash23:38:265
5603991612,2cyclictest0-21swapper/523:50:115
5603991612,2cyclictest0-21swapper/521:25:025
560399160,15cyclictest0-21swapper/520:11:205
560399160,13cyclictest0-21swapper/521:01:285
559799160,1cyclictest0-21swapper/422:35:014
559799160,15cyclictest27068-21bash00:19:064
559299161,13cyclictest666-21dbus-daemon22:56:023
558599164,10cyclictest666-21dbus-daemon19:20:012
558599160,14cyclictest32740-21/usr/sbin/munin22:15:262
558099167,3cyclictest91rcu_preempt22:29:411
558099167,1cyclictest24-21ksoftirqd/123:38:561
558099163,2cyclictest24-21ksoftirqd/122:41:381
558099162,4cyclictest24-21ksoftirqd/121:45:301
5580991614,1cyclictest23085-21kworker/1:220:38:021
557399160,15cyclictest453-21sshd23:41:290
306802160,0sleep70-21swapper/720:00:217
210012160,1sleep30-21swapper/322:26:503
114762160,0sleep7771ktimersoftd/723:21:217
8202150,1sleep0816-21bash23:15:380
561599152,4cyclictest21194-21sshd23:35:417
560999154,7cyclictest0-21swapper/623:05:456
560999154,7cyclictest0-21swapper/623:05:456
560999153,5cyclictest0-21swapper/621:56:126
560999152,6cyclictest0-21swapper/622:56:236
5609991512,1cyclictest22166-21sshd21:26:486
560999151,13cyclictest728-21NetworkManager21:41:436
560999150,13cyclictest0-21swapper/623:03:196
560999150,13cyclictest0-21swapper/623:03:186
560399152,7cyclictest0-21swapper/520:42:485
560399152,12cyclictest0-21swapper/519:53:545
560399152,12cyclictest0-21swapper/519:39:365
5603991513,1cyclictest6361-21tty22:25:195
560399150,2cyclictest22400-21sshd23:59:285
560399150,13cyclictest24678-21sshd22:17:195
560399150,13cyclictest0-21swapper/522:01:195
560399150,13cyclictest0-21swapper/522:01:195
560399150,13cyclictest0-21swapper/519:19:245
559799150,1cyclictest0-21swapper/400:26:574
559799150,14cyclictest0-21swapper/422:31:384
559299151,2cyclictest0-21swapper/322:10:123
559299150,2cyclictest12360-21sshd22:06:103
559299150,2cyclictest12360-21sshd22:06:093
559299150,2cyclictest0-21swapper/321:29:293
558599155,8cyclictest8331-21sshd22:22:092
558599154,9cyclictest702-21gdbus00:15:092
558599154,9cyclictest3629-21sshd23:32:572
558599152,2cyclictest711-21systemd-logind23:40:372
558599152,11cyclictest666-21dbus-daemon22:40:222
558599150,2cyclictest664-21polkitd00:02:492
558599150,2cyclictest664-21polkitd00:02:482
558599150,2cyclictest16227-21sshd22:25:362
558099159,3cyclictest24-21ksoftirqd/100:11:581
558099158,2cyclictest24-21ksoftirqd/122:52:311
558099156,1cyclictest24-21ksoftirqd/122:09:311
558099156,1cyclictest24-21ksoftirqd/122:09:311
558099154,1cyclictest101rcu_sched23:29:361
558099151,1cyclictest101rcu_sched23:43:381
5580991511,2cyclictest666-21dbus-daemon21:22:411
5580991510,1cyclictest24-21ksoftirqd/122:14:351
5573991512,2cyclictest702-21gdbus23:46:110
557399150,2cyclictest0-21swapper/021:43:280
557399150,14cyclictest25731-21cp00:11:270
557399150,14cyclictest0-21swapper/021:39:060
316042150,0sleep00-21swapper/021:18:340
10722150,1sleep31068-21sshd21:31:273
84282140,0sleep60-21swapper/623:10:306
5603991411,2cyclictest29494-21sshd22:38:575
560399140,8cyclictest0-21swapper/521:06:285
560399140,8cyclictest0-21swapper/519:10:485
560399140,7cyclictest10418-21sshd23:15:575
560399140,1cyclictest0-21swapper/500:33:455
560399140,13cyclictest0-21swapper/520:51:365
560399140,13cyclictest0-21swapper/520:02:325
560399140,12cyclictest27648-21sshd21:55:575
559799140,2cyclictest0-21swapper/423:49:114
559799140,1cyclictest29320-21sshd21:58:104
559799140,1cyclictest19731-21sshd22:25:434
559799140,1cyclictest0-21swapper/421:31:254
559799140,13cyclictest6915-21bash21:22:014
559799140,13cyclictest29402-21sshd00:34:194
5592991412,1cyclictest0-21swapper/323:22:153
559299140,2cyclictest666-21dbus-daemon22:51:353
559299140,13cyclictest0-21swapper/322:22:253
558099148,2cyclictest24-21ksoftirqd/121:10:381
5580991412,1cyclictest23085-21kworker/1:219:14:301
557399149,4cyclictest7-21ksoftirqd/022:05:200
557399149,4cyclictest7-21ksoftirqd/022:05:200
5573991412,1cyclictest0-21swapper/023:14:360
5573991411,2cyclictest31431-21sshd22:51:540
5573991410,2cyclictest0-21swapper/000:09:350
5573991410,2cyclictest0-21swapper/000:09:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional