You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 19:53
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Fri Nov 28, 2025 12:43:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
137042930,0sleep30-21swapper/311:44:293
305182710,0sleep50-21swapper/509:39:225
94042650,0sleep10-21swapper/111:39:071
255212510,0sleep20-21swapper/209:43:432
318332490,1sleep5763-21runrttasks10:55:225
58762480,0sleep00-21swapper/009:14:260
280462480,0sleep30-21swapper/310:24:203
227452470,0sleep10-21swapper/110:33:471
198342470,0sleep20-21swapper/211:45:242
215372430,0sleep10-21swapper/109:53:031
247042420,0sleep20-21swapper/209:28:052
101042420,0sleep40-21swapper/411:08:504
101042420,0sleep40-21swapper/411:08:504
362124129,9sleep20-21swapper/206:54:082
348924129,8sleep60-21swapper/606:52:286
339624128,9sleep70-21swapper/706:51:087
277512410,0sleep20-21swapper/208:29:282
171452410,0sleep40-21swapper/410:02:274
171452410,0sleep40-21swapper/410:02:274
135462410,0sleep7763-21runrttasks11:24:157
351424027,9sleep00-21swapper/006:52:460
344224028,9sleep40-21swapper/406:51:464
339424029,8sleep50-21swapper/506:51:065
331424028,9sleep10-21swapper/106:49:581
293942400,0sleep30-21swapper/309:39:123
250952400,0sleep40-21swapper/411:10:504
140132400,0sleep70-21swapper/710:11:517
82152390,1sleep341-21ksoftirqd/307:54:293
29832390,0sleep10-21swapper/109:34:451
113523926,9sleep30-21swapper/306:49:123
27552380,0sleep50-21swapper/510:40:245
3821992710,12cyclictest0-21swapper/510:11:045
382199213,11cyclictest26341-21sshd11:06:005
382199213,11cyclictest26341-21sshd11:06:005
382199210,9cyclictest0-21swapper/508:43:185
3823992016,4cyclictest73-21ksoftirqd/709:44:547
382199203,5cyclictest0-21swapper/508:07:145
3821992015,5cyclictest9913-21kworker/u16:312:16:525
382199200,8cyclictest0-21swapper/508:36:285
3823991914,5cyclictest73-21ksoftirqd/709:54:227
177312190,0sleep60-21swapper/609:26:416
382199187,0cyclictest0-21swapper/511:34:525
382199176,0cyclictest0-21swapper/511:30:545
382199170,10cyclictest0-21swapper/507:49:225
382099170,16cyclictest19828-21sshd09:57:484
382399160,2cyclictest121rcu_preempt10:59:247
382199165,5cyclictest0-21swapper/507:39:145
382199165,11cyclictest0-21swapper/512:02:205
3823991513,1cyclictest73-21ksoftirqd/712:23:157
382199159,5cyclictest1-21systemd10:30:405
382199155,9cyclictest31849-21sshd11:42:325
382199154,5cyclictest0-21swapper/508:12:205
382199154,0cyclictest0-21swapper/508:15:225
382199154,0cyclictest0-21swapper/508:00:185
382199150,9cyclictest0-21swapper/511:14:545
382199150,9cyclictest0-21swapper/510:21:325
382199150,9cyclictest0-21swapper/507:35:285
382199150,13cyclictest0-21swapper/509:09:385
3809991512,3cyclictest25-21ksoftirqd/110:12:241
3823991413,1cyclictest73-21ksoftirqd/709:19:177
3823991413,1cyclictest73-21ksoftirqd/709:04:257
3823991412,1cyclictest73-21ksoftirqd/709:33:557
3823991412,1cyclictest73-21ksoftirqd/709:29:077
3823991410,1cyclictest73-21ksoftirqd/712:01:067
382199149,5cyclictest0-21swapper/511:21:385
382199149,0cyclictest0-21swapper/508:47:025
382199148,6cyclictest0-21swapper/510:54:045
382199148,6cyclictest0-21swapper/509:20:505
382199148,5cyclictest20449-21grepconf.sh08:56:025
382199148,5cyclictest0-21swapper/509:33:405
382199148,1cyclictest6082-21systemd-cgroups11:13:025
382199147,5cyclictest0-21swapper/507:22:365
382199147,0cyclictest0-21swapper/509:49:025
382199147,0cyclictest0-21swapper/507:02:405
382199146,5cyclictest0-21swapper/512:11:325
382199145,5cyclictest0-21swapper/509:28:205
382199145,4cyclictest0-21swapper/507:25:025
382199143,5cyclictest0-21swapper/507:12:125
382199143,5cyclictest0-21swapper/507:12:125
382199143,5cyclictest0-21swapper/507:05:145
382199143,0cyclictest0-21swapper/511:56:565
382199143,0cyclictest0-21swapper/510:01:385
382199143,0cyclictest0-21swapper/510:01:385
382199143,0cyclictest0-21swapper/508:20:265
382199142,9cyclictest0-21swapper/507:45:085
382199142,11cyclictest0-21swapper/510:24:545
382199142,0cyclictest0-21swapper/508:59:345
382199142,0cyclictest0-21swapper/508:33:165
382199140,8cyclictest0-21swapper/508:50:545
382199140,11cyclictest0-21swapper/507:56:205
3819991413,1cyclictest41-21ksoftirqd/309:06:043
3809991413,1cyclictest6733-21kworker/1:112:03:241
3809991412,2cyclictest25-21ksoftirqd/110:35:301
3809991412,1cyclictest12519-21kworker/1:009:44:541
3802991412,1cyclictest9-21ksoftirqd/011:54:260
46042130,0sleep20-21swapper/210:56:232
3823991313,0cyclictest0-21swapper/710:02:247
3823991313,0cyclictest0-21swapper/710:02:247
3823991312,1cyclictest73-21ksoftirqd/711:55:537
3823991312,1cyclictest73-21ksoftirqd/709:40:287
3823991311,1cyclictest73-21ksoftirqd/712:16:027
3823991311,1cyclictest73-21ksoftirqd/710:40:187
3823991311,1cyclictest73-21ksoftirqd/709:09:227
3823991311,1cyclictest32371-21bash08:58:087
382399130,1cyclictest131rcu_sched11:05:507
382399130,1cyclictest131rcu_sched11:05:507
3822991311,1cyclictest85350irq/25-em1-rx-010:36:526
382199138,5cyclictest0-21swapper/511:51:055
382199138,5cyclictest0-21swapper/511:46:305
382199138,5cyclictest0-21swapper/511:25:545
382199138,5cyclictest0-21swapper/510:45:585
382199138,5cyclictest0-21swapper/510:34:365
382199138,5cyclictest0-21swapper/510:14:485
382199138,5cyclictest0-21swapper/510:14:485
382199138,5cyclictest0-21swapper/509:54:305
382199138,5cyclictest0-21swapper/509:52:425
382199138,5cyclictest0-21swapper/509:34:485
382199138,5cyclictest0-21swapper/509:16:085
382199138,5cyclictest0-21swapper/507:14:525
382199138,0cyclictest0-21swapper/512:07:345
382199138,0cyclictest0-21swapper/511:00:105
382199138,0cyclictest0-21swapper/507:31:365
382199138,0cyclictest0-21swapper/506:55:145
382199137,5cyclictest5671-21bash10:05:345
382199136,0cyclictest0-21swapper/509:04:145
382199132,6cyclictest0-21swapper/508:24:185
382199132,5cyclictest0-21swapper/512:20:075
3819991312,1cyclictest41-21ksoftirqd/309:49:453
3819991312,1cyclictest41-21ksoftirqd/309:13:393
3819991311,1cyclictest41-21ksoftirqd/312:17:313
3819991311,1cyclictest41-21ksoftirqd/312:04:273
3819991310,2cyclictest41-21ksoftirqd/310:06:383
3809991312,1cyclictest25-21ksoftirqd/112:07:361
3809991312,1cyclictest25-21ksoftirqd/111:29:231
3809991312,1cyclictest25-21ksoftirqd/111:12:221
3809991312,1cyclictest25-21ksoftirqd/110:47:021
3809991312,1cyclictest25-21ksoftirqd/109:04:251
301792130,0sleep20-21swapper/212:17:042
382399129,2cyclictest73-21ksoftirqd/712:07:307
3823991212,0cyclictest0-21swapper/710:09:007
3823991211,1cyclictest73-21ksoftirqd/711:39:207
3823991211,1cyclictest73-21ksoftirqd/711:32:007
3823991211,1cyclictest73-21ksoftirqd/711:19:127
3823991211,1cyclictest73-21ksoftirqd/711:14:317
3823991211,1cyclictest73-21ksoftirqd/710:51:467
3823991211,1cyclictest73-21ksoftirqd/710:44:247
3823991211,1cyclictest73-21ksoftirqd/710:29:237
3823991211,1cyclictest73-21ksoftirqd/710:19:187
3823991211,1cyclictest73-21ksoftirqd/709:02:597
3823991211,0cyclictest73-21ksoftirqd/710:54:207
3823991210,1cyclictest861-21gdbus11:10:407
3823991210,1cyclictest73-21ksoftirqd/712:10:287
3823991210,1cyclictest73-21ksoftirqd/711:52:487
3823991210,1cyclictest73-21ksoftirqd/711:45:367
3823991210,1cyclictest73-21ksoftirqd/710:34:277
3823991210,1cyclictest73-21ksoftirqd/710:18:437
3823991210,1cyclictest73-21ksoftirqd/710:18:437
3823991210,1cyclictest73-21ksoftirqd/709:54:087
382399121,11cyclictest0-21swapper/710:25:067
382399120,11cyclictest10865-21sshd09:15:117
3822991210,1cyclictest65-21ksoftirqd/608:56:006
382299120,11cyclictest24636-21bash12:00:396
3819991210,2cyclictest41-21ksoftirqd/310:57:293
3819991210,1cyclictest41-21ksoftirqd/311:39:293
3819991210,1cyclictest41-21ksoftirqd/311:29:093
3819991210,1cyclictest41-21ksoftirqd/309:29:093
381999120,11cyclictest1-21systemd11:49:263
381799129,2cyclictest711-21dbus-daemon10:46:222
380999129,2cyclictest25-21ksoftirqd/110:50:291
380999128,1cyclictest25-21ksoftirqd/110:57:391
3809991211,1cyclictest25-21ksoftirqd/112:14:551
3809991211,1cyclictest25-21ksoftirqd/111:53:461
3809991211,1cyclictest25-21ksoftirqd/110:43:441
3809991211,1cyclictest25-21ksoftirqd/110:04:091
3809991211,1cyclictest25-21ksoftirqd/110:04:091
3809991211,1cyclictest25-21ksoftirqd/109:24:461
3809991211,1cyclictest25-21ksoftirqd/109:17:171
3809991210,1cyclictest25-21ksoftirqd/111:48:451
3809991210,1cyclictest25-21ksoftirqd/111:43:371
3809991210,1cyclictest25-21ksoftirqd/110:24:071
3809991210,1cyclictest25-21ksoftirqd/110:14:311
3809991210,1cyclictest25-21ksoftirqd/110:14:311
3809991210,1cyclictest25-21ksoftirqd/109:02:351
3809991210,1cyclictest11609-21kworker/1:109:57:121
380299123,4cyclictest3834-21sshd10:35:420
274432120,0sleep40-21swapper/411:46:384
169102120,0sleep00-21swapper/009:06:010
382399116,5cyclictest73-21ksoftirqd/708:26:467
382399116,5cyclictest73-21ksoftirqd/707:28:587
382399116,5cyclictest15151-21kworker/7:107:01:487
382399116,5cyclictest0-21swapper/708:34:307
382399116,5cyclictest0-21swapper/707:32:287
382399116,1cyclictest15151-21kworker/7:107:07:007
382399116,0cyclictest0-21swapper/707:18:107
382399115,6cyclictest73-21ksoftirqd/708:15:127
3823991111,0cyclictest0-21swapper/707:59:207
3823991111,0cyclictest0-21swapper/707:45:247
3823991110,1cyclictest73-21ksoftirqd/711:34:377
382399111,10cyclictest0-21swapper/708:49:267
382399110,5cyclictest0-21swapper/708:07:247
3822991110,1cyclictest65-21ksoftirqd/611:31:326
3822991110,1cyclictest65-21ksoftirqd/609:38:476
382299110,11cyclictest0-21swapper/609:49:596
381999110,10cyclictest20201-21sshd10:43:323
381799119,1cyclictest13851-21sshd10:06:572
381799119,1cyclictest0-21swapper/210:27:552
381799116,3cyclictest10606-21sshd10:51:582
381799110,10cyclictest0-21swapper/211:12:412
380999119,2cyclictest25-21ksoftirqd/111:04:191
380999119,2cyclictest25-21ksoftirqd/111:04:191
380999119,1cyclictest5629-21kworker/1:211:20:041
380999119,1cyclictest25-21ksoftirqd/112:21:521
380999119,1cyclictest25-21ksoftirqd/110:28:321
380999119,1cyclictest25-21ksoftirqd/109:41:331
380999118,1cyclictest25-21ksoftirqd/109:09:231
3809991110,1cyclictest31382-21kworker/1:108:56:121
3809991110,1cyclictest25-21ksoftirqd/111:03:071
3809991110,1cyclictest25-21ksoftirqd/110:04:111
380299113,4cyclictest23748-21/usr/sbin/munin07:29:090
380299110,10cyclictest711-21dbus-daemon12:03:570
382399106,0cyclictest0-21swapper/708:44:227
382399105,5cyclictest0-21swapper/707:35:467
382399105,5cyclictest0-21swapper/707:20:347
382399105,0cyclictest0-21swapper/708:39:407
382399105,0cyclictest0-21swapper/708:20:127
382399105,0cyclictest0-21swapper/708:11:147
382399105,0cyclictest0-21swapper/707:55:287
382399105,0cyclictest0-21swapper/707:49:387
382399105,0cyclictest0-21swapper/707:39:167
382399105,0cyclictest0-21swapper/707:10:067
382399105,0cyclictest0-21swapper/707:10:067
382399105,0cyclictest0-21swapper/706:55:007
382399100,4cyclictest0-21swapper/708:31:207
382299109,1cyclictest65-21ksoftirqd/611:38:086
382299107,3cyclictest65-21ksoftirqd/610:16:356
382299107,3cyclictest65-21ksoftirqd/610:16:356
382299100,6cyclictest28355-21grepconf.sh09:18:116
382299100,10cyclictest0-21swapper/609:08:576
382299100,1cyclictest711-21dbus-daemon09:03:366
382099100,2cyclictest758-21sshd09:30:044
382099100,1cyclictest0-21swapper/410:39:364
382099100,1cyclictest0-21swapper/410:37:514
381999109,1cyclictest4261-21kworker/3:209:59:063
381999109,1cyclictest41-21ksoftirqd/311:58:213
381999109,1cyclictest41-21ksoftirqd/311:34:563
381999109,1cyclictest41-21ksoftirqd/311:34:083
381999109,1cyclictest41-21ksoftirqd/311:09:183
381999109,1cyclictest41-21ksoftirqd/310:52:213
381999109,1cyclictest41-21ksoftirqd/309:15:283
381999108,1cyclictest41-21ksoftirqd/312:24:093
381999108,1cyclictest41-21ksoftirqd/311:18:033
381999108,1cyclictest41-21ksoftirqd/310:22:153
381999107,2cyclictest41-21ksoftirqd/310:59:343
381799100,10cyclictest0-21swapper/210:18:182
381799100,10cyclictest0-21swapper/210:18:182
380999109,1cyclictest25-21ksoftirqd/112:13:301
380999109,1cyclictest25-21ksoftirqd/111:54:221
380999109,1cyclictest25-21ksoftirqd/111:26:361
380999109,1cyclictest25-21ksoftirqd/111:16:161
380999109,1cyclictest25-21ksoftirqd/109:19:291
380999108,1cyclictest25-21ksoftirqd/109:32:081
380999108,1cyclictest25-21ksoftirqd/107:19:091
380299109,1cyclictest0-21swapper/011:09:120
380299104,3cyclictest0-21swapper/007:19:090
380299100,10cyclictest0-21swapper/012:08:220
380299100,10cyclictest0-21swapper/009:12:250
169992100,0sleep00-21swapper/009:47:170
38239997,1cyclictest31067-21sshd09:34:207
38229998,1cyclictest65-21ksoftirqd/609:31:566
38229997,1cyclictest65-21ksoftirqd/611:44:316
38229997,1cyclictest65-21ksoftirqd/611:06:596
38229997,1cyclictest65-21ksoftirqd/611:06:596
38229997,1cyclictest65-21ksoftirqd/610:25:396
38229997,1cyclictest1-21systemd10:00:476
38229997,1cyclictest1-21systemd10:00:476
38229994,3cyclictest12322-21sshd10:11:346
38229993,6cyclictest0-21swapper/610:30:206
38229993,5cyclictest30475-21bash10:55:056
38229993,3cyclictest0-21swapper/610:08:136
38229990,9cyclictest0-21swapper/611:20:556
38229990,9cyclictest0-21swapper/611:10:496
38229990,9cyclictest0-21swapper/609:46:576
38229990,0cyclictest0-21swapper/611:14:466
38209998,1cyclictest49-21ksoftirqd/412:04:204
38209998,1cyclictest0-21swapper/409:39:324
38209997,1cyclictest49-21ksoftirqd/407:07:374
38209993,5cyclictest5912-21sshd10:20:474
38209993,5cyclictest26587-21sshd09:53:544
38209993,3cyclictest8357-21sshd10:46:334
38209991,4cyclictest1602-21id11:22:204
38209990,8cyclictest12104-21grepconf.sh10:11:324
38209990,1cyclictest0-21swapper/412:09:574
38199999,0cyclictest0-21swapper/312:11:303
38199998,1cyclictest41-21ksoftirqd/310:30:053
38199998,1cyclictest41-21ksoftirqd/309:29:253
38199998,1cyclictest41-21ksoftirqd/309:19:393
38199996,2cyclictest9970-21grepconf.sh10:36:433
38199996,2cyclictest41-21ksoftirqd/311:05:503
38199996,2cyclictest41-21ksoftirqd/311:05:503
38199994,3cyclictest15417-21sshd10:47:453
38199994,3cyclictest13264-21sshd08:54:423
38199994,3cyclictest0-21swapper/309:46:363
38179999,0cyclictest0-21swapper/209:54:012
38179994,3cyclictest4217-21kworker/2:112:02:402
38179990,9cyclictest0-21swapper/211:49:352
38179990,9cyclictest0-21swapper/211:04:482
38179990,9cyclictest0-21swapper/211:04:482
38179990,9cyclictest0-21swapper/209:00:412
38179990,8cyclictest22935-21grepconf.sh09:38:082
38029999,0cyclictest0-21swapper/012:19:240
38029999,0cyclictest0-21swapper/010:39:360
38029997,1cyclictest16795-21sshd10:02:240
38029997,1cyclictest16795-21sshd10:02:240
38029992,6cyclictest9825-21sshd10:46:480
38029992,4cyclictest0-21swapper/010:09:090
38029990,9cyclictest0-21swapper/012:12:470
38029990,9cyclictest0-21swapper/011:38:250
38029990,9cyclictest0-21swapper/011:16:020
38029990,9cyclictest0-21swapper/010:15:580
38029990,9cyclictest0-21swapper/010:15:580
38029990,0cyclictest0-21swapper/011:32:160
38029990,0cyclictest0-21swapper/011:00:440
38029990,0cyclictest0-21swapper/009:30:140
38229988,0cyclictest641ktimersoftd/612:04:256
38229988,0cyclictest0-21swapper/611:00:446
38229988,0cyclictest0-21swapper/609:41:076
38229987,1cyclictest65-21ksoftirqd/612:11:426
38229987,1cyclictest65-21ksoftirqd/611:55:186
38229987,1cyclictest65-21ksoftirqd/611:52:116
38229987,1cyclictest65-21ksoftirqd/611:42:336
38229987,1cyclictest65-21ksoftirqd/610:52:536
38229986,1cyclictest65-21ksoftirqd/611:25:436
38229986,1cyclictest65-21ksoftirqd/609:57:546
38229986,1cyclictest65-21ksoftirqd/609:20:226
38229986,1cyclictest65-21ksoftirqd/609:11:326
38229983,3cyclictest0-21swapper/610:47:346
38229982,0cyclictest0-21swapper/607:09:266
38229982,0cyclictest0-21swapper/607:09:266
38229980,8cyclictest8860-21kworker/6:210:39:356
38229980,8cyclictest0-21swapper/608:41:406
38229980,2cyclictest0-21swapper/608:33:076
38209987,1cyclictest49-21ksoftirqd/409:04:444
38209985,2cyclictest31582-21grep08:57:594
38209984,3cyclictest22806-21bash12:20:404
38209983,5cyclictest0-21swapper/410:50:464
38209983,3cyclictest12383-21sshd11:39:224
38209983,2cyclictest28788-21sshd09:23:214
38209983,1cyclictest17519-21sshd11:14:334
38209982,5cyclictest18720-21sshd09:16:324
38209982,4cyclictest14993-21sshd10:27:244
38209982,3cyclictest732-21NetworkManager10:06:134
38209982,3cyclictest20127-21sshd11:50:164
38209982,3cyclictest0-21swapper/412:18:204
38209982,2cyclictest11419-21sshd09:36:134
38209981,4cyclictest25706-21bash09:28:154
38209981,4cyclictest11347-21sshd12:03:424
38209980,8cyclictest18141-21kworker/4:311:30:244
38209980,8cyclictest0-21swapper/411:26:064
38209980,8cyclictest0-21swapper/407:44:264
38209980,7cyclictest1148-21kworker/4:210:32:524
38209980,7cyclictest10494-21kworker/4:009:48:254
38209980,5cyclictest14507-21sshd09:00:264
38209980,3cyclictest740-21gdbus09:11:054
38199988,0cyclictest0-21swapper/307:24:253
38199986,1cyclictest41-21ksoftirqd/310:15:133
38199986,1cyclictest41-21ksoftirqd/310:15:133
38199986,1cyclictest41-21ksoftirqd/310:10:113
38199986,1cyclictest41-21ksoftirqd/309:59:213
38199986,1cyclictest41-21ksoftirqd/309:59:213
38199984,2cyclictest5210-21sshd09:35:083
38199984,2cyclictest17101-21sshd09:00:553
38199984,2cyclictest13949-21sshd11:19:193
38199980,7cyclictest20468-21kworker/3:111:59:193
38199980,1cyclictest0-21swapper/308:50:313
38179988,0cyclictest0-21swapper/211:32:522
38179988,0cyclictest0-21swapper/211:24:312
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional