You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-24 - 10:12
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackcslot0.osadl.org (updated Fri Oct 24, 2025 00:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
733821340,2sleep1585299cyclictest23:44:501
5197212192,25sleep140-21swapper/1419:09:536
51152117102,11sleep00-21swapper/019:08:510
5166211694,13sleep10-21swapper/119:09:241
418721140,4sleep110-21swapper/1119:07:243
5193211291,15sleep100-21swapper/1019:09:492
2779721110,1sleep10-21swapper/123:00:421
5047210983,21sleep20-21swapper/219:08:048
5369210777,25sleep60-21swapper/619:11:3712
35321070,6sleep3585599cyclictest22:03:309
43721040,1sleep70-21swapper/723:37:1113
43721040,1sleep70-21swapper/723:37:1113
750521010,0sleep50-21swapper/500:06:1811
5140210078,17sleep30-21swapper/319:09:109
320932980,1sleep120-21swapper/1221:20:514
316392960,1sleep140-21swapper/1423:37:016
316392960,1sleep140-21swapper/1423:37:016
203072950,1sleep60-21swapper/623:13:5312
203072950,1sleep60-21swapper/623:13:5312
132832950,1sleep140-21swapper/1423:56:336
256042940,0sleep50-21swapper/521:58:5911
258972930,0sleep20-21swapper/223:00:278
103992920,1sleep910397-21systemd-cgroups23:23:5015
103992920,1sleep910397-21systemd-cgroups23:23:5015
536829175,4sleep50-21swapper/519:11:3611
304932910,1sleep10-21swapper/122:14:101
507129075,11sleep40-21swapper/419:08:2310
286792900,7sleep7586099cyclictest23:04:1713
537628972,4sleep130-21swapper/1319:11:445
509228975,9sleep80-21swapper/819:08:4314
239752890,0sleep90-21swapper/921:44:3615
80032880,6sleep6585999cyclictest22:43:5612
99702870,5sleep15587399cyclictest22:08:157
93322870,0sleep140-21swapper/1422:29:376
528228667,14sleep70-21swapper/719:11:0613
503728673,9sleep90-21swapper/919:07:5515
192128672,9sleep120-21swapper/1219:07:104
91282850,1sleep50-21swapper/521:29:0211
93492840,0sleep150-21swapper/1522:40:367
506428375,4sleep150-21swapper/1519:08:187
224552820,1sleep111051ktimersoftd/1100:12:013
161822820,1sleep10-21swapper/121:57:511
307862810,1sleep3401ktimersoftd/300:28:289
249872810,1sleep00-21swapper/023:07:220
103892810,0sleep10-21swapper/123:09:161
261752800,0sleep20-21swapper/221:16:568
212192780,0sleep100-21swapper/1021:55:032
115592780,6sleep0585199cyclictest23:30:540
115592780,6sleep0585199cyclictest23:30:540
297392770,0sleep100-21swapper/1023:40:262
297392770,0sleep100-21swapper/1023:40:262
234602750,5sleep0585199cyclictest21:16:380
176872670,0sleep50-21swapper/500:11:0711
234492640,5sleep6585999cyclictest22:27:3712
141862640,6sleep13587099cyclictest00:26:455
233112580,0sleep40-21swapper/400:12:1010
5862995315,36cyclictest0-21swapper/822:02:1514
1139099480,2rtkit-daemon1069-21NetworkManager21:43:474
586699449,33cyclictest7656-21env23:05:362
1139099440,23rtkit-daemon0-21swapper/1321:59:215
1139099440,16rtkit-daemon0-21swapper/421:36:514
1139099430,2rtkit-daemon11389-21rtkit-daemon23:06:424
586699420,39cyclictest24882-21sshd00:38:382
61732410,1sleep50-21swapper/522:15:0211
586699415,3cyclictest1068-21dbus-daemon23:08:442
586699411,8cyclictest26634-21kworker/10:100:02:512
586699410,3cyclictest1233-21gdbus22:24:592
1139099410,1rtkit-daemon19628-21sshd00:11:285
586699405,3cyclictest1233-21gdbus22:46:552
586699400,37cyclictest10345-21sshd22:15:302
586699400,37cyclictest0-21swapper/1023:51:072
586699399,27cyclictest11948-21systemd23:16:522
586699399,27cyclictest11948-21systemd23:16:522
586699398,4cyclictest24188-21uname23:28:382
586699398,4cyclictest24188-21uname23:28:382
586699398,29cyclictest450-21sshd23:01:192
586699397,3cyclictest14454-21systemd21:40:012
586699397,3cyclictest1-21systemd22:32:332
586699395,6cyclictest0-21swapper/1022:55:192
586699394,4cyclictest4902-21sshd21:49:472
586699394,3cyclictest32387-21missed_timers00:32:272
5866993913,23cyclictest18484-21sshd23:42:312
5866993913,23cyclictest14553-21systemd21:43:262
5866993910,27cyclictest0-21swapper/1022:41:222
586699390,1cyclictest0-21swapper/1000:22:502
36092390,0sleep110-21swapper/1123:15:513
36092390,0sleep110-21swapper/1123:15:513
85002380,33sleep13525-21systemd-journal23:52:165
586699389,26cyclictest1297-21sshd22:18:352
586699388,3cyclictest30612-21sshd23:32:522
586699388,3cyclictest30612-21sshd23:32:522
586699387,3cyclictest27794-21sshd21:23:372
586699385,3cyclictest5467-21gdbus23:57:542
586699385,3cyclictest1-21systemd00:30:592
586699385,2cyclictest0-21swapper/1023:22:562
586699385,2cyclictest0-21swapper/1023:22:562
586699382,5cyclictest0-21swapper/1000:12:592
586699380,10cyclictest11087-21systemd21:29:162
586699378,3cyclictest1-21systemd21:32:332
586699376,3cyclictest1068-21dbus-daemon22:31:042
586699374,3cyclictest5467-21gdbus23:20:382
586699374,3cyclictest5467-21gdbus23:20:382
5866993712,22cyclictest0-21swapper/1021:18:092
5866993710,24cyclictest17963-21sshd00:07:192
586699370,9cyclictest22414-21sshd00:20:412
586699367,2cyclictest1297-21sshd21:59:452
586699366,3cyclictest525-21systemd-journal22:05:382
586699364,4cyclictest1068-21dbus-daemon22:47:572
326182360,1sleep60-21swapper/623:51:2012
24542360,2sleep00-21swapper/022:28:510
1139099360,1rtkit-daemon0-21swapper/420:39:244
87582350,2sleep14587199cyclictest00:06:276
5866993510,23cyclictest30163-2110-uname21:13:562
586699350,11cyclictest0-21swapper/1023:52:592
1139099350,0rtkit-daemon0-21swapper/1323:51:435
586699346,3cyclictest8110-21sh22:11:542
586699336,3cyclictest9710-21sh19:52:292
273772330,1sleep30-21swapper/300:21:149
1139099330,1rtkit-daemon0-21swapper/1221:46:005
310662320,1sleep50-21swapper/522:03:1611
280072320,7sleep11106-21ksoftirqd/1123:18:313
280072320,7sleep11106-21ksoftirqd/1123:18:313
177892320,2sleep141068-21dbus-daemon00:16:536
134172320,27sleep21-21systemd22:55:078
1139099320,24rtkit-daemon1078-21ptp4l20:17:414
586699316,2cyclictest22522-21latency_hist19:32:012
233602310,27sleep923489-21kill21:48:0615
129992310,6sleep4585799cyclictest00:19:3910
1139099310,2rtkit-daemon1078-21ptp4l00:18:4411
263152300,0sleep30-21swapper/322:56:439
1139099300,1rtkit-daemon0-21swapper/1322:13:005
69212290,1sleep130-21swapper/1322:32:495
46422290,25sleep20-21swapper/223:19:438
46422290,25sleep20-21swapper/223:19:438
40672290,2sleep140-21swapper/1423:23:086
40672290,2sleep140-21swapper/1423:23:086
311162290,27sleep20-21swapper/220:17:368
293302290,1sleep120-21swapper/1223:29:154
293302290,1sleep120-21swapper/1223:29:154
280322290,2sleep151068-21dbus-daemon21:13:417
183312290,1sleep80-21swapper/800:30:5814
175852290,2sleep15138-21ksoftirqd/1500:03:477
127152290,1sleep120-21swapper/1221:15:314
586699289,17cyclictest9860-21latency_hist19:17:012
586699288,2cyclictest0-21swapper/1019:42:312
586699284,3cyclictest1032-17auditd20:47:012
48192280,2sleep21068-21dbus-daemon00:14:168
317572280,1sleep15587399cyclictest23:22:367
317572280,1sleep15587399cyclictest23:22:367
251902280,23sleep50-21swapper/522:17:1111
1139099280,0rtkit-daemon0-21swapper/522:38:2311
586699277,3cyclictest6779-21latency_hist19:52:012
586699274,2cyclictest28930-21sh20:17:152
260712270,0sleep10-21swapper/122:56:411
235412270,1sleep30-21swapper/321:19:559
214292270,24sleep021580-21systemd-cgroups23:39:290
214292270,24sleep021580-21systemd-cgroups23:39:290
199052270,24sleep2525-21systemd-journal00:38:058
148662270,1sleep30-21swapper/300:07:049
68782260,0sleep90-21swapper/922:07:5315
66922260,2sleep80-21swapper/823:34:0014
66922260,2sleep80-21swapper/823:34:0014
586699266,3cyclictest23280-21latency_hist20:12:012
586699266,2cyclictest1361-21runrttasks20:06:422
586699260,1cyclictest0-21swapper/1019:57:152
585799268,9cyclictest1068-21dbus-daemon00:25:2510
585799263,5cyclictest0-21swapper/422:32:3110
585799260,24cyclictest481ktimersoftd/422:08:0610
43312260,1sleep130-21swapper/1323:05:125
41382260,1sleep150-21swapper/1523:15:557
41382260,1sleep150-21swapper/1523:15:557
320602260,0sleep120-21swapper/1223:47:254
304982260,1sleep9881rcuc/920:57:1615
304982260,1sleep9881rcuc/920:57:1615
280922260,1sleep028090-21systemd-cgroups00:28:090
166872260,21sleep00-21swapper/022:33:570
123372260,1sleep50-21swapper/523:02:2311
1139099260,0rtkit-daemon0-21swapper/1220:03:074
9252250,0sleep00-21swapper/000:25:120
75052250,1sleep90-21swapper/900:02:3815
586699254,3cyclictest29023-21awk19:37:222
585799256,4cyclictest481ktimersoftd/423:13:4010
585799256,4cyclictest481ktimersoftd/423:13:4010
585799255,14cyclictest49-21ksoftirqd/423:34:0910
585799255,14cyclictest49-21ksoftirqd/423:34:0910
5857992521,2cyclictest481ktimersoftd/422:06:4410
5857992521,2cyclictest471rcuc/422:39:2810
5857992518,5cyclictest49-21ksoftirqd/421:21:0610
274092250,1sleep00-21swapper/020:17:010
271722250,1sleep150-21swapper/1523:43:297
244032250,21sleep131-21systemd23:43:115
242902250,2sleep00-21swapper/021:48:130
235752251,2sleep110-21swapper/1122:24:143
210012250,1sleep80-21swapper/821:47:4814
204972250,0sleep10-21swapper/121:22:481
1139099250,0rtkit-daemon0-21swapper/1300:31:105
101662250,0sleep90-21swapper/921:25:2115
101512250,22sleep150-21swapper/1522:51:197
88612241,19sleep10-21swapper/100:09:591
586699240,2cyclictest0-21swapper/1020:27:012
586099240,6cyclictest0-21swapper/723:44:4013
585799247,10cyclictest49-21ksoftirqd/423:51:1110
585799246,9cyclictest1068-21dbus-daemon21:53:0910
585799246,13cyclictest131rcu_preempt21:44:4910
585799245,9cyclictest4741-21sshd00:40:0410
585799245,10cyclictest0-21swapper/423:21:0010
585799245,10cyclictest0-21swapper/423:21:0010
5857992420,2cyclictest471rcuc/421:47:0810
5857992418,4cyclictest0-21swapper/421:14:2510
290862240,1sleep90-21swapper/921:41:4115
278512240,1sleep70-21swapper/721:45:0713
269612240,1sleep90-21swapper/922:21:0015
260392241,2sleep31068-21dbus-daemon23:00:299
254252240,19sleep120-21swapper/1200:24:204
246202240,2sleep110-21swapper/1123:28:423
246202240,2sleep110-21swapper/1123:28:423
196072240,20sleep319752-21sshd22:48:439
178542240,1sleep90-21swapper/921:19:1315
167402240,1sleep70-21swapper/723:56:5913
1139099240,0rtkit-daemon0-21swapper/420:53:5410
1139099240,0rtkit-daemon0-21swapper/420:53:5410
88702230,19sleep00-21swapper/022:12:010
5871992316,4cyclictest26318-21df_abs21:27:166
587099231,20cyclictest477-21jbd2/md124p1-821:41:365
587099231,20cyclictest0-21swapper/1322:52:145
586999231,20cyclictest0-21swapper/1219:12:164
586699235,2cyclictest4126-21timerandwakeup21:02:322
586699235,2cyclictest4126-21timerandwakeup21:02:322
586699233,3cyclictest4652-21cpuspeed_turbos20:27:152
586699231,3cyclictest0-21swapper/1020:42:192
5863992318,2cyclictest0-21swapper/923:37:1915
5863992318,2cyclictest0-21swapper/923:37:1915
585799239,8cyclictest24979-21systemd-cgroups00:35:2310
585799237,9cyclictest49-21ksoftirqd/423:11:1810
585799237,10cyclictest49-21ksoftirqd/423:24:2710
585799237,10cyclictest49-21ksoftirqd/423:24:2710
585799236,4cyclictest49-21ksoftirqd/421:41:0510
585799235,9cyclictest49-21ksoftirqd/422:00:3210
585799235,8cyclictest0-21swapper/423:42:3110
585799234,4cyclictest49-21ksoftirqd/422:21:1710
585799234,17cyclictest481ktimersoftd/422:47:2510
5851992318,3cyclictest26625-21sshd21:52:130
5851992317,4cyclictest29593-21systemd00:32:180
5851992316,5cyclictest0-21swapper/023:43:070
291272230,19sleep10-21swapper/122:39:031
262832230,1sleep8801rcuc/822:52:4814
260512230,1sleep30-21swapper/300:01:289
223602230,1sleep60-21swapper/623:39:3512
223602230,1sleep60-21swapper/623:39:3512
222372230,19sleep140-21swapper/1422:38:126
21992231,2sleep02347-21sshd23:58:440
134392230,2sleep913544-21sshd23:49:0015
126712230,1sleep20-21swapper/221:47:118
100262230,2sleep510136-21sshd23:30:4411
100262230,2sleep510136-21sshd23:30:4411
99082220,1sleep30-21swapper/322:12:039
90872220,1sleep140-21swapper/1422:44:056
81212220,1sleep60-21swapper/623:55:5412
77182220,3sleep117785-21sshd00:14:583
586699225,2cyclictest259532sleep1019:32:492
5860992217,3cyclictest9295-2110-uname21:36:0113
585799228,8cyclictest22385-21sshd21:23:0210
585799227,9cyclictest49-21ksoftirqd/422:24:4610
585799227,9cyclictest49-21ksoftirqd/400:09:2910
585799227,9cyclictest22431-21sshd22:42:0910
585799227,8cyclictest18473-21sshd23:57:0610
585799226,8cyclictest13947-21kill23:38:3210
585799226,8cyclictest13947-21kill23:38:3210
585799225,9cyclictest49-21ksoftirqd/423:02:5610
585799225,9cyclictest49-21ksoftirqd/421:33:0910
585799225,9cyclictest1068-21dbus-daemon22:12:2510
585799225,7cyclictest49-21ksoftirqd/422:28:5110
5857992219,2cyclictest481ktimersoftd/400:27:5010
5857992218,3cyclictest471rcuc/400:06:1510
5857992215,4cyclictest1068-21dbus-daemon21:27:2310
585199229,5cyclictest131rcu_preempt22:23:250
585199227,6cyclictest0-21swapper/021:19:570
585199226,8cyclictest1-21systemd22:41:280
57462220,19sleep60-21swapper/600:33:0512
56422220,1sleep70-21swapper/723:12:1913
56422220,1sleep70-21swapper/723:12:1913
315372220,18sleep140-21swapper/1400:36:096
309432220,18sleep80-21swapper/823:29:2714
309432220,18sleep80-21swapper/823:29:2714
301402220,2sleep20-21swapper/223:29:218
301402220,2sleep20-21swapper/223:29:218
278302220,0sleep20-21swapper/223:25:518
278302220,0sleep20-21swapper/223:25:518
201822220,20sleep50-21swapper/520:45:2711
178092220,18sleep110-21swapper/1123:49:343
170752220,18sleep90-21swapper/922:37:3515
133112220,19sleep30-21swapper/323:38:289
133112220,19sleep30-21swapper/323:38:289
1139099220,5rtkit-daemon0-21swapper/1522:55:2213
5873992117,2cyclictest0-21swapper/1521:27:457
5871992116,3cyclictest0-21swapper/1421:47:436
587199211,3cyclictest5467-21gdbus00:28:276
587199211,3cyclictest0-21swapper/1400:21:166
587199211,17cyclictest0-21swapper/1400:41:116
587199210,5cyclictest1-21systemd00:25:506
587199210,19cyclictest10538-21kworker/14:321:12:226
587199210,18cyclictest21319-21sshd23:17:386
587199210,18cyclictest21319-21sshd23:17:386
5870992115,3cyclictest0-21swapper/1300:21:245
587099211,18cyclictest15257-21sshd23:24:245
587099211,18cyclictest15257-21sshd23:24:245
5869992116,3cyclictest525-21systemd-journal23:58:164
586299210,20cyclictest6794-21kworker/8:300:41:4914
5860992114,3cyclictest15096-21sshd00:30:3213
5858992116,3cyclictest10480-21sh23:56:1111
585799217,5cyclictest5453-21sshd23:01:5210
585799216,8cyclictest0-21swapper/423:29:0410
585799216,8cyclictest0-21swapper/423:29:0410
585799216,7cyclictest1349-21nfsd23:52:2610
585499211,18cyclictest25147-21sshd21:27:148
5852992114,4cyclictest2723-21systemd23:15:441
5852992114,4cyclictest2723-21systemd23:15:441
5852992114,4cyclictest18258-21sshd00:23:341
5852992114,4cyclictest13275-21bash21:47:131
585199211,5cyclictest0-21swapper/023:23:270
585199211,5cyclictest0-21swapper/023:23:270
5851992112,2cyclictest10-21ksoftirqd/023:14:420
5851992112,2cyclictest10-21ksoftirqd/023:14:420
56932210,18sleep20-21swapper/222:15:008
43312210,0sleep10-21swapper/123:59:001
32842216,3sleep143457-21sshd21:32:086
310052210,1sleep50-21swapper/500:13:1611
28372210,1sleep131201rcuc/1319:47:015
244472210,1sleep120-21swapper/1221:37:274
161422210,2sleep140-21swapper/1421:43:386
131752210,2sleep20-21swapper/223:38:278
131752210,2sleep20-21swapper/223:38:278
124882210,0sleep80-21swapper/821:32:4914
117252210,19sleep40-21swapper/420:32:5210
1139099210,2rtkit-daemon1233-21gdbus00:23:1211
1139099210,0rtkit-daemon0-21swapper/1219:37:244
77902200,1sleep140-21swapper/1422:00:446
5873992015,3cyclictest12274-21sshd00:19:347
5871992017,2cyclictest28547-21kworker/14:021:37:186
5871992014,4cyclictest21489-21grep20:07:246
5871992014,2cyclictest0-21swapper/1422:08:476
587199200,17cyclictest1975-21kworker/14:023:34:506
587199200,17cyclictest1975-21kworker/14:023:34:506
587199200,17cyclictest0-21swapper/1421:55:236
587199200,17cyclictest0-21swapper/1421:20:116
5870992015,3cyclictest525-21systemd-journal22:19:035
5870992015,3cyclictest145-21kauditd23:28:075
5870992015,3cyclictest145-21kauditd23:28:075
5870992014,4cyclictest12545-21sh21:22:115
5870992013,4cyclictest8213-21sshd21:32:255
587099200,2cyclictest0-21swapper/1321:47:025
586999201,4cyclictest26953-21sshd00:32:014
586999200,17cyclictest12027-21kill00:40:584
586999200,17cyclictest0-21swapper/1222:07:474
5868992013,4cyclictest27513-21sh21:52:143
586899200,18cyclictest21469-21kworker/11:421:34:423
5862992015,3cyclictest31377-21cups-browsed22:25:2714
586299201,16cyclictest18541-21updatedb.mlocat00:15:0214
586099201,4cyclictest21420-21bash00:23:5413
586099201,17cyclictest0-21swapper/721:40:2513
586099200,2cyclictest0-21swapper/722:32:2213
5859992015,3cyclictest1238-21gdbus00:31:4812
5858992015,3cyclictest1-21systemd21:46:2211
585899200,3cyclictest1233-21gdbus22:08:5011
585899200,17cyclictest14855-21systemd-cgroups23:42:1611
585799203,15cyclictest481ktimersoftd/419:12:1910
5857992010,6cyclictest131rcu_preempt22:52:4710
5855992013,4cyclictest1408-21systemd23:15:339
5855992013,4cyclictest1408-21systemd23:15:339
585199209,6cyclictest131rcu_preempt00:19:510
585199208,6cyclictest10-21ksoftirqd/000:10:230
585199207,6cyclictest131rcu_preempt22:42:440
585199202,9cyclictest10-21ksoftirqd/021:23:360
5851992017,2cyclictest111ktimersoftd/021:57:220
5851992016,2cyclictest111ktimersoftd/021:41:320
5851992010,2cyclictest10-21ksoftirqd/023:21:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional