You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 19:27
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackcslot0.osadl.org (updated Fri Nov 28, 2025 00:43:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2655921710,1sleep3574999cyclictest22:27:499
1249621310,1sleep150-21swapper/1522:29:537
2525621240,1sleep80-21swapper/821:35:4514
51502122103,15sleep50-21swapper/519:11:1911
1783211784,28sleep30-21swapper/319:07:329
5151211591,20sleep60-21swapper/619:11:2012
5001211492,17sleep100-21swapper/1019:09:312
5239210893,10sleep40-21swapper/419:11:5010
5000210792,9sleep90-21swapper/919:09:3015
2673121040,1sleep90-21swapper/922:35:0815
5056210383,15sleep20-21swapper/219:09:568
505529983,11sleep10-21swapper/119:09:551
207262980,0sleep90-21swapper/900:34:1015
80642970,1sleep20-21swapper/222:36:548
195352960,1sleep50-21swapper/520:07:4311
514529472,4sleep00-21swapper/019:11:140
281982930,0sleep20-21swapper/223:06:038
281462930,0sleep70-21swapper/723:36:0513
170812920,1sleep00-21swapper/023:42:390
252229081,4sleep150-21swapper/1519:07:447
194492900,2sleep121032-17auditd22:11:554
180228981,4sleep110-21swapper/1119:07:403
5242880,1sleep100-21swapper/1021:25:212
500528772,11sleep140-21swapper/1419:09:356
502028675,7sleep70-21swapper/719:09:4513
312032860,2sleep101-21systemd21:14:392
25012860,1sleep130-21swapper/1321:36:545
106262860,0sleep120-21swapper/1200:14:094
106262860,0sleep120-21swapper/1200:14:094
515828574,7sleep130-21swapper/1319:11:275
233062850,1sleep150-21swapper/1520:12:427
11982850,7sleep2574899cyclictest22:54:498
216032840,0sleep150-21swapper/1500:34:167
515728359,20sleep120-21swapper/1219:11:264
266322830,0sleep10-21swapper/121:35:541
179928274,4sleep80-21swapper/819:07:3714
145272800,0sleep150-21swapper/1521:12:517
61442780,1sleep20-21swapper/223:18:058
151632780,7sleep9575999cyclictest22:22:3415
158432750,0sleep20-21swapper/200:05:258
158432750,0sleep20-21swapper/200:05:258
76302720,5sleep15576699cyclictest21:55:397
76302720,5sleep15576699cyclictest21:55:397
47692720,7sleep7575699cyclictest00:28:3113
1139099520,29rtkit-daemon0-21swapper/1320:18:2611
128802510,0sleep40-21swapper/421:41:3410
128802510,0sleep40-21swapper/421:41:3410
225272490,0sleep60-21swapper/622:31:0512
1139099490,22rtkit-daemon1234-21polkitd23:58:3813
574999452,8cyclictest0-21swapper/321:52:449
574999452,8cyclictest0-21swapper/321:52:449
5749994517,21cyclictest525-21systemd-journal23:20:469
77412440,2sleep67744-21sshd00:25:1712
77412440,2sleep67744-21sshd00:25:1712
5749994413,24cyclictest23836-21sshd23:35:339
5749994412,24cyclictest0-21swapper/322:16:059
574999432,35cyclictest0-21swapper/300:16:229
574999432,35cyclictest0-21swapper/300:16:229
5749994316,24cyclictest14015-21sshd23:53:169
5749994313,21cyclictest20942-21systemd-cgroups22:12:079
5749994311,29cyclictest0-21swapper/323:28:009
5749994212,24cyclictest0-21swapper/300:29:489
5749994210,30cyclictest0-21swapper/321:27:169
5749994210,30cyclictest0-21swapper/300:04:349
5749994210,30cyclictest0-21swapper/300:04:349
39382420,0sleep10-21swapper/122:06:231
39382420,0sleep10-21swapper/122:06:231
574999418,23cyclictest26390-21sshd23:24:359
5749994120,18cyclictest25502-21kill21:35:469
5749994114,24cyclictest1-21systemd21:15:319
5749994113,25cyclictest0-21swapper/300:07:589
5749994113,25cyclictest0-21swapper/300:07:589
5749994111,28cyclictest0-21swapper/300:35:049
574999410,39cyclictest0-21swapper/323:16:229
1139099410,3rtkit-daemon15595-21munin-node23:27:537
574999409,24cyclictest0-21swapper/322:06:569
574999409,24cyclictest0-21swapper/322:06:569
574999405,29cyclictest0-21swapper/300:22:449
574999405,29cyclictest0-21swapper/300:22:449
5749994021,17cyclictest0-21swapper/323:38:569
5749994016,22cyclictest0-21swapper/321:41:519
5749994016,22cyclictest0-21swapper/321:41:519
5749994015,17cyclictest1098-21in:imuxsock23:02:499
5749994014,24cyclictest20635-21uname22:34:249
5749994014,24cyclictest0-21swapper/322:20:179
5749994014,24cyclictest0-21swapper/300:18:069
5749994014,24cyclictest0-21swapper/300:18:069
5749994014,19cyclictest0-21swapper/322:26:429
5749994011,25cyclictest26269-21sshd22:42:469
5749994011,19cyclictest5529-21systemd-cgroups00:40:059
5749994010,28cyclictest0-21swapper/323:02:069
1139099400,0rtkit-daemon0-21swapper/1523:06:567
574999394,27cyclictest0-21swapper/322:55:079
5749993915,21cyclictest29195-21systemd21:43:039
5749993915,21cyclictest29195-21systemd21:43:039
5749993915,21cyclictest1-21systemd23:57:549
5749993915,16cyclictest26656-21systemd-cgroups23:47:359
5749993914,22cyclictest21613-21cp21:27:529
5749993913,20cyclictest0-21swapper/322:51:239
5749993911,26cyclictest0-21swapper/321:48:469
5749993911,26cyclictest0-21swapper/321:48:469
5749993910,26cyclictest0-21swapper/323:43:079
5749993714,21cyclictest0-21swapper/322:38:309
5749993714,20cyclictest17521-21sshd23:08:169
5749993712,22cyclictest0-21swapper/321:17:399
5749993711,24cyclictest0-21swapper/321:57:419
5749993711,24cyclictest0-21swapper/321:57:419
5749993618,16cyclictest14793-21sed20:02:309
5749993617,17cyclictest0-21swapper/319:27:589
5749993617,17cyclictest0-21swapper/319:27:589
5749993615,19cyclictest0-21swapper/320:28:419
5749993614,20cyclictest0-21swapper/319:39:099
1139099360,1rtkit-daemon0-21swapper/1321:37:415
1139099360,1rtkit-daemon0-21swapper/1321:37:415
5749993517,16cyclictest0-21swapper/320:16:419
5749993516,17cyclictest0-21swapper/321:02:299
5749993516,17cyclictest0-21swapper/319:50:489
5749993516,16cyclictest27636-21sed20:53:019
5749993514,19cyclictest22458-21apt-get19:32:409
5749993514,19cyclictest0-21swapper/319:12:469
5749993514,19cyclictest0-21swapper/319:12:469
5749993514,18cyclictest0-21swapper/320:37:589
5749993513,19cyclictest0-21swapper/320:33:019
5749993511,21cyclictest0-21swapper/321:02:479
82432340,1sleep150-21swapper/1521:33:437
80982340,2sleep78094-21sshd22:44:2813
5749993417,15cyclictest27954-21cstates20:17:429
5749993417,15cyclictest16915-21sed19:23:019
5749993417,15cyclictest16915-21sed19:23:019
5749993416,17cyclictest0-21swapper/320:48:159
5749993416,16cyclictest0-21swapper/319:53:219
5749993415,17cyclictest0-21swapper/321:10:599
5749993415,17cyclictest0-21swapper/320:22:599
5749993415,17cyclictest0-21swapper/320:07:469
5749993415,17cyclictest0-21swapper/320:02:349
5749993413,19cyclictest0-21swapper/320:42:459
5749993413,19cyclictest0-21swapper/319:42:399
261982340,1sleep15138-21ksoftirqd/1522:23:447
247112340,1sleep150-21swapper/1500:11:167
247112340,1sleep150-21swapper/1500:11:167
220722340,0sleep40-21swapper/400:38:0510
192282340,1sleep80-21swapper/821:20:1414
186702340,6sleep7575699cyclictest23:31:1613
1139099340,3rtkit-daemon19736-21env21:31:3911
67992330,2sleep61073-21systemd-logind23:10:5712
5749993313,18cyclictest0-21swapper/319:17:429
5749993313,18cyclictest0-21swapper/319:17:429
1139099330,16rtkit-daemon1069-21NetworkManager21:52:0613
1139099330,16rtkit-daemon1069-21NetworkManager21:52:0613
242592320,27sleep7832-21jbd2/md125-800:02:3513
242592320,27sleep7832-21jbd2/md125-800:02:3513
308882310,1sleep50-21swapper/521:32:4511
307522310,2sleep01347-21nfsd00:07:170
307522310,2sleep01347-21nfsd00:07:170
252812310,6sleep12576399cyclictest23:05:414
175812310,1sleep00-21swapper/023:04:410
42222300,6sleep1574799cyclictest22:13:361
322482300,2sleep70-21swapper/722:20:4113
315902300,4sleep15576699cyclictest00:31:517
146302301,26sleep151069-21NetworkManager22:48:547
1139099300,2rtkit-daemon1068-21dbus-daemon23:37:137
1139099300,1rtkit-daemon0-21swapper/1321:44:135
1139099300,1rtkit-daemon0-21swapper/1321:44:135
1139099300,0rtkit-daemon0-21swapper/723:51:1513
1139099300,0rtkit-daemon0-21swapper/1320:50:285
75932290,1sleep10-21swapper/121:41:001
75932290,1sleep10-21swapper/121:41:001
66762290,25sleep150-21swapper/1521:44:107
66762290,25sleep150-21swapper/1521:44:107
576099297,16cyclictest98-21ksoftirqd/1021:55:552
576099297,16cyclictest98-21ksoftirqd/1021:55:552
273582290,25sleep627484-21sshd22:20:0812
268162290,2sleep1026788-21sshd00:31:142
19082290,25sleep130-21swapper/1323:02:465
161932290,1sleep60-21swapper/600:33:3612
135812290,22sleep13122-21ksoftirqd/1323:15:195
1139099290,2rtkit-daemon1078-21ptp4l19:42:415
576099286,6cyclictest14840-21sshd22:07:432
5759992824,2cyclictest90-21ksoftirqd/919:47:3915
5751992810,6cyclictest49-21ksoftirqd/423:38:4910
318792280,25sleep130-21swapper/1322:39:305
228572280,1sleep00-21swapper/021:46:200
228572280,1sleep00-21swapper/021:46:200
180992280,25sleep140-21swapper/1420:03:046
161132280,24sleep130-21swapper/1323:08:075
129872280,24sleep10-21swapper/121:30:501
66932270,0sleep60-21swapper/622:06:4412
66932270,0sleep60-21swapper/622:06:4412
66512270,2sleep76762-21systemd-cgroups22:25:3013
576099276,4cyclictest971ktimersoftd/1021:57:412
576099276,4cyclictest971ktimersoftd/1021:57:412
574899270,25cyclictest0-21swapper/220:17:408
45322270,22sleep50-21swapper/500:32:3011
252442270,6sleep11576199cyclictest20:52:443
249402270,23sleep1125066-21systemctl23:09:093
247302270,1sleep20-21swapper/223:16:468
220422270,1sleep60-21swapper/621:57:2212
220422270,1sleep60-21swapper/621:57:2212
214442270,2sleep421547-21sshd22:00:5010
214442270,2sleep421547-21sshd22:00:5010
144322270,24sleep00-21swapper/000:41:150
123292270,1sleep110-21swapper/1100:33:083
116412270,0sleep150-21swapper/1523:57:057
1139099270,3rtkit-daemon0-21swapper/1319:55:045
1139099270,0rtkit-daemon0-21swapper/1320:08:295
91682260,2sleep141068-21dbus-daemon23:07:366
77662260,23sleep77899-21sshd22:10:3113
64692260,0sleep70-21swapper/723:56:2513
576099269,14cyclictest2484-21sshd23:02:482
576099264,20cyclictest971ktimersoftd/1023:07:522
575899260,24cyclictest0-21swapper/819:17:4114
575899260,24cyclictest0-21swapper/819:17:4114
5751992615,6cyclictest131rcu_preempt00:31:4510
31212260,1sleep10-21swapper/122:43:511
255052260,3sleep71233-21gdbus22:53:5113
24852260,1sleep20-21swapper/222:39:508
223662260,1sleep130-21swapper/1323:35:235
161122260,23sleep50-21swapper/500:09:3011
161122260,23sleep50-21swapper/500:09:3011
13892260,1sleep40-21swapper/423:13:5010
12472260,1sleep80-21swapper/800:24:3114
12472260,1sleep80-21swapper/800:24:3114
118852260,1sleep4481ktimersoftd/422:22:1310
116122260,2sleep121073-21systemd-logind21:48:394
116122260,2sleep121073-21systemd-logind21:48:394
115222260,2sleep90-21swapper/919:17:4715
115222260,2sleep90-21swapper/919:17:4715
5764992514,6cyclictest122-21ksoftirqd/1300:21:525
5764992514,6cyclictest122-21ksoftirqd/1300:21:525
576099258,8cyclictest30565-21sshd22:43:102
5760992522,2cyclictest961rcuc/1000:37:252
5751992512,6cyclictest49-21ksoftirqd/423:00:0610
5751992511,9cyclictest131rcu_preempt21:29:0510
5751992511,8cyclictest141rcu_sched22:43:5010
5751992511,7cyclictest49-21ksoftirqd/421:13:0310
5751992510,8cyclictest131rcu_preempt21:20:1110
251112250,1sleep70-21swapper/720:12:5013
222042250,1sleep90-21swapper/923:08:5015
185132251,20sleep121233-21gdbus21:56:574
185132251,20sleep121233-21gdbus21:56:574
151062250,1sleep70-21swapper/700:18:2213
151062250,1sleep70-21swapper/700:18:2213
147212250,1sleep100-21swapper/1021:19:442
134702250,2sleep120-21swapper/1222:18:204
1139099250,0rtkit-daemon0-21swapper/1321:03:475
78702240,20sleep00-21swapper/023:18:200
576699240,6cyclictest0-21swapper/1523:52:247
576499247,9cyclictest0-21swapper/1323:20:335
576499242,6cyclictest122-21ksoftirqd/1321:57:445
576499242,6cyclictest122-21ksoftirqd/1321:57:445
5764992411,8cyclictest141rcu_sched00:34:075
576099249,7cyclictest13291-21sshd22:03:202
576099249,7cyclictest13291-21sshd22:03:202
576099248,8cyclictest909-21systemd-cgroups23:29:062
576099247,11cyclictest98-21ksoftirqd/1022:48:532
576099247,10cyclictest98-21ksoftirqd/1023:00:552
576099247,10cyclictest98-21ksoftirqd/1000:19:362
576099247,10cyclictest98-21ksoftirqd/1000:19:362
576099246,9cyclictest98-21ksoftirqd/1022:40:482
576099246,9cyclictest0-21swapper/1021:28:372
576099246,8cyclictest98-21ksoftirqd/1023:40:062
576099246,7cyclictest0-21swapper/1000:25:202
576099246,7cyclictest0-21swapper/1000:25:202
576099246,12cyclictest98-21ksoftirqd/1021:43:052
576099246,12cyclictest98-21ksoftirqd/1021:43:052
576099246,10cyclictest98-21ksoftirqd/1022:35:032
576099245,11cyclictest98-21ksoftirqd/1022:28:122
576099245,10cyclictest0-21swapper/1000:02:522
576099245,10cyclictest0-21swapper/1000:02:522
576099244,9cyclictest0-21swapper/1000:14:232
576099244,9cyclictest0-21swapper/1000:14:232
576099244,10cyclictest0-21swapper/1021:40:092
576099244,10cyclictest0-21swapper/1021:40:092
5760992420,3cyclictest961rcuc/1023:36:312
576099240,17cyclictest10028-21sshd22:25:552
575199249,8cyclictest131rcu_preempt22:35:4410
5751992413,6cyclictest131rcu_preempt00:12:0910
5751992413,6cyclictest131rcu_preempt00:12:0910
5751992410,7cyclictest131rcu_preempt22:53:0410
28992240,2sleep00-21swapper/000:20:550
28992240,2sleep00-21swapper/000:20:550
247602240,19sleep130-21swapper/1323:47:265
239712240,20sleep10-21swapper/100:34:351
226612240,0sleep60-21swapper/600:15:3912
226612240,0sleep60-21swapper/600:15:3912
215872240,21sleep00-21swapper/021:31:530
206312240,20sleep8525-21systemd-journal21:42:2714
206312240,20sleep8525-21systemd-journal21:42:2714
173402240,1sleep60-21swapper/622:34:0212
164142240,2sleep150-21swapper/1522:15:127
151072240,20sleep40-21swapper/423:34:2710
150662240,2sleep50-21swapper/500:41:2011
13982240,20sleep111521-21systemd-cgroups23:59:423
117732240,20sleep130-21swapper/1321:48:415
117732240,20sleep130-21swapper/1321:48:415
1139099240,0rtkit-daemon0-21swapper/519:40:3111
100332240,2sleep2525-21systemd-journal23:56:538
9272230,1sleep00-21swapper/022:54:470
576499238,10cyclictest131rcu_preempt21:31:345
5764992312,7cyclictest131rcu_preempt22:26:335
5764992311,6cyclictest141rcu_sched00:30:555
5761992317,4cyclictest13866-21run-parts21:41:403
5761992317,4cyclictest13866-21run-parts21:41:403
576099237,9cyclictest98-21ksoftirqd/1022:19:012
576099237,8cyclictest98-21ksoftirqd/1021:33:272
576099237,11cyclictest141rcu_sched23:18:072
576099237,10cyclictest98-21ksoftirqd/1022:53:342
576099237,10cyclictest21427-21sshd23:23:572
576099235,7cyclictest98-21ksoftirqd/1021:48:262
576099235,7cyclictest98-21ksoftirqd/1021:48:262
576099235,11cyclictest98-21ksoftirqd/1000:39:012
576099233,6cyclictest0-21swapper/1023:50:462
5760992318,3cyclictest971ktimersoftd/1023:57:432
5759992319,2cyclictest0-21swapper/923:23:0015
575999230,3cyclictest525-21systemd-journal23:36:3815
575999230,3cyclictest22735-21systemd23:39:1215
5756992315,2cyclictest8616-21sshd23:22:3613
575299234,3cyclictest1245-21gdbus21:13:5611
575199234,10cyclictest49-21ksoftirqd/423:57:2110
5751992313,6cyclictest49-21ksoftirqd/421:55:0310
5751992313,6cyclictest49-21ksoftirqd/421:55:0310
5751992312,6cyclictest141rcu_sched23:50:4010
5751992312,5cyclictest131rcu_preempt00:18:4110
5751992312,5cyclictest131rcu_preempt00:18:4110
5751992311,6cyclictest49-21ksoftirqd/423:19:1310
5751992310,8cyclictest131rcu_preempt23:05:0810
574799231,19cyclictest0-21swapper/120:42:411
42422230,18sleep00-21swapper/022:40:050
38212230,0sleep0111ktimersoftd/022:06:220
38212230,0sleep0111ktimersoftd/022:06:220
34602230,2sleep60-21swapper/622:17:2712
312932230,1sleep1531428-21systemd-cgroups21:25:117
292042230,1sleep9881rcuc/923:06:1015
276452230,2sleep140-21swapper/1423:47:426
270852230,1sleep00-21swapper/021:17:410
244562230,19sleep60-21swapper/623:43:1912
229262230,1sleep140-21swapper/1422:01:006
229262230,1sleep140-21swapper/1422:01:006
211622230,19sleep110-21swapper/1123:20:073
191272230,1sleep150-21swapper/1500:22:477
191272230,1sleep150-21swapper/1500:22:477
172742230,1sleep130-21swapper/1300:26:285
172742230,1sleep130-21swapper/1300:26:285
166872230,1sleep116686-21systemctl23:08:101
115402230,1sleep151371ktimersoftd/1523:37:507
1139099230,2rtkit-daemon1078-21ptp4l21:08:575
1139099230,2rtkit-daemon1069-21NetworkManager22:00:497
1139099230,2rtkit-daemon1069-21NetworkManager22:00:497
57742220,19sleep120-21swapper/1220:27:544
576499228,8cyclictest131rcu_preempt22:29:055
5764992213,4cyclictest141rcu_sched22:16:005
576499221,18cyclictest0-21swapper/1300:09:135
576499221,18cyclictest0-21swapper/1300:09:135
5764992211,6cyclictest131rcu_preempt23:42:235
5764992211,6cyclictest131rcu_preempt21:19:295
5763992216,4cyclictest114-21ksoftirqd/1222:24:544
576099226,8cyclictest98-21ksoftirqd/1022:12:352
576099224,9cyclictest98-21ksoftirqd/1000:08:042
576099224,9cyclictest98-21ksoftirqd/1000:08:042
5760992218,2cyclictest98-21ksoftirqd/1020:27:432
5756992218,2cyclictest0-21swapper/721:36:1913
5756992216,3cyclictest1068-21dbus-daemon22:30:2913
5756992216,3cyclictest0-21swapper/721:23:5113
575699221,18cyclictest11062-21gdbus23:03:5213
575699221,18cyclictest0-21swapper/722:17:0413
575499220,3cyclictest28497-21systemd-cgroups00:38:5812
575199229,8cyclictest49-21ksoftirqd/422:39:5210
575199229,7cyclictest49-21ksoftirqd/423:45:0910
575199229,7cyclictest49-21ksoftirqd/400:25:1110
575199229,7cyclictest49-21ksoftirqd/400:25:1110
575199229,7cyclictest131rcu_preempt21:34:5610
575199227,6cyclictest131rcu_preempt21:49:2810
575199227,6cyclictest131rcu_preempt21:49:2810
575199226,6cyclictest49-21ksoftirqd/422:26:3710
575199226,10cyclictest49-21ksoftirqd/400:35:3910
575199225,9cyclictest9360-21upowerd22:08:4810
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional