You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 20:29
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Sat Nov 22, 2025 12:43:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
230412570,7sleep233-21ksoftirqd/210:02:152
5592480,0sleep20-21swapper/209:05:132
265132480,0sleep00-21swapper/010:48:580
163132470,0sleep00-21swapper/011:38:240
2547224525,16sleep10-21swapper/106:50:041
50632440,0sleep40-21swapper/411:36:364
43012440,0sleep00-21swapper/008:09:220
262902440,1sleep125-21ksoftirqd/109:14:491
88852420,0sleep10-21swapper/112:16:131
309582420,0sleep00-21swapper/010:54:430
2565524129,9sleep20-21swapper/206:52:382
2564924128,9sleep50-21swapper/506:52:335
2331024128,9sleep60-21swapper/606:49:296
214052410,0sleep10-21swapper/110:37:541
188972410,0sleep60-21swapper/609:24:396
179862410,0sleep10-21swapper/109:56:061
91112400,0sleep60-21swapper/610:40:486
2572524027,9sleep40-21swapper/406:53:364
2553624027,9sleep00-21swapper/006:50:590
2533524027,8sleep70-21swapper/706:49:467
233182400,0sleep50-21swapper/507:44:375
2330724027,9sleep30-21swapper/306:49:263
213152400,0sleep40-21swapper/409:03:314
52332390,0sleep50-21swapper/509:27:515
296502390,0sleep40-21swapper/406:59:364
179332300,0sleep10-21swapper/111:28:481
25973992419,5cyclictest57-21ksoftirqd/510:21:005
2597499223,18cyclictest9262-21grepconf.sh10:20:496
25973992014,6cyclictest57-21ksoftirqd/510:58:485
2597099202,18cyclictest99-21kswapd007:28:594
25952992017,3cyclictest9-21ksoftirqd/010:00:180
178382200,0sleep717831-21bash10:11:497
2597399198,6cyclictest22317-21kworker/5:310:38:105
25973991915,0cyclictest0-21swapper/510:28:045
25973991914,5cyclictest57-21ksoftirqd/509:30:185
25973991913,5cyclictest2255-21sshd11:56:305
2595299191,17cyclictest15513-21sshd10:41:550
2597399186,6cyclictest0-21swapper/508:55:585
25973991813,5cyclictest9980-21kworker/5:111:30:105
25973991813,5cyclictest57-21ksoftirqd/510:08:205
25973991813,0cyclictest57-21ksoftirqd/511:52:125
2597399177,9cyclictest1561-21kworker/5:410:10:365
25973991713,4cyclictest57-21ksoftirqd/509:14:485
25973991712,5cyclictest8004-21kworker/5:109:09:365
25973991712,2cyclictest57-21ksoftirqd/512:22:005
25973991712,0cyclictest0-21swapper/508:43:345
2596999170,5cyclictest711-21dbus-daemon11:29:513
2597599168,5cyclictest0-21swapper/707:36:137
2597599168,0cyclictest0-21swapper/708:37:217
2597599168,0cyclictest0-21swapper/708:37:217
2597599168,0cyclictest0-21swapper/707:33:137
2597599162,6cyclictest0-21swapper/709:36:447
2597599162,6cyclictest0-21swapper/709:36:447
25975991614,1cyclictest25582-21sshd11:55:067
2597399166,5cyclictest57-21ksoftirqd/507:24:235
2597399165,0cyclictest0-21swapper/507:42:035
25973991612,0cyclictest0-21swapper/511:18:285
25973991612,0cyclictest0-21swapper/511:18:285
25973991611,5cyclictest57-21ksoftirqd/507:15:235
25973991611,1cyclictest6771-21kworker/5:311:06:545
25973991610,6cyclictest57-21ksoftirqd/511:12:385
2596999161,14cyclictest18259-21sshd09:08:063
2596999160,5cyclictest31717-21id10:39:303
2595299163,1cyclictest9-21ksoftirqd/011:56:310
25952991611,3cyclictest9-21ksoftirqd/011:12:120
2595299160,13cyclictest1-21systemd12:16:190
15592160,0sleep50-21swapper/509:38:165
15592160,0sleep50-21swapper/509:38:165
306142150,0sleep00-21swapper/011:50:490
295022150,0sleep20-21swapper/210:49:242
2597599158,5cyclictest0-21swapper/708:29:457
2597599158,5cyclictest0-21swapper/708:29:457
2597599156,8cyclictest754-21sshd11:05:267
2597599154,5cyclictest31037-21sshd09:04:527
2597499150,13cyclictest32474-21bash09:58:336
2597399159,1cyclictest15828-21kworker/5:311:49:145
2597399157,5cyclictest57-21ksoftirqd/511:00:275
25973991512,3cyclictest57-21ksoftirqd/509:49:305
25973991511,0cyclictest0-21swapper/511:59:385
25973991511,0cyclictest0-21swapper/508:07:435
25973991510,5cyclictest57-21ksoftirqd/507:06:535
25973991510,1cyclictest8587-21kworker/5:112:08:085
2597399150,9cyclictest0-21swapper/509:55:245
2597399150,2cyclictest0-21swapper/509:21:505
2596999151,5cyclictest12803-21grep11:42:553
2596999151,1cyclictest0-21swapper/309:20:313
2596999150,6cyclictest12970-21id11:12:363
2596999150,4cyclictest5264-21sshd09:59:283
2596199150,14cyclictest11794-21sshd11:22:232
25952991514,1cyclictest32383-21kworker/0:007:22:090
25952991513,1cyclictest9-21ksoftirqd/010:49:470
25952991513,1cyclictest9-21ksoftirqd/010:36:570
25952991513,1cyclictest9-21ksoftirqd/010:07:290
25952991512,3cyclictest9-21ksoftirqd/011:26:360
25952991511,1cyclictest9-21ksoftirqd/011:43:380
240612150,0sleep30-21swapper/311:54:533
2597599149,4cyclictest8511-21sshd10:45:527
2597599148,5cyclictest8185-21sshd10:20:387
2597599148,5cyclictest0-21swapper/712:15:367
2597599148,5cyclictest0-21swapper/709:48:087
2597599148,5cyclictest0-21swapper/708:46:497
2597599148,5cyclictest0-21swapper/708:46:497
2597599148,5cyclictest0-21swapper/708:12:257
2597599148,5cyclictest0-21swapper/708:01:257
2597599148,5cyclictest0-21swapper/707:55:537
2597599148,5cyclictest0-21swapper/707:55:537
2597599148,5cyclictest0-21swapper/707:43:057
2597599148,5cyclictest0-21swapper/707:14:497
2597599148,5cyclictest0-21swapper/707:02:317
2597599148,5cyclictest0-21swapper/706:56:357
2597599147,5cyclictest0-21swapper/710:34:407
2597599147,5cyclictest0-21swapper/709:22:087
2597599146,5cyclictest0-21swapper/708:14:497
2597599146,5cyclictest0-21swapper/707:05:197
2597599146,0cyclictest0-21swapper/711:44:007
2597599146,0cyclictest0-21swapper/710:30:487
2597599143,5cyclictest0-21swapper/710:18:247
2597599142,6cyclictest0-21swapper/712:12:367
2597599142,6cyclictest0-21swapper/709:13:007
25975991412,2cyclictest0-21swapper/708:21:037
2597599140,9cyclictest0-21swapper/707:25:577
2597399149,4cyclictest57-21ksoftirqd/510:18:345
2597399148,6cyclictest0-21swapper/508:34:295
2597399148,6cyclictest0-21swapper/508:34:295
2597399148,0cyclictest0-21swapper/508:52:255
2597399143,5cyclictest0-21swapper/510:42:485
25973991410,4cyclictest57-21ksoftirqd/511:39:565
2596999140,5cyclictest4778-21sshd10:20:063
2596999140,5cyclictest14194-21grep09:18:403
2596999140,4cyclictest8464-21grep10:56:243
2596999140,4cyclictest668-21systemd-logind10:28:043
2596999140,2cyclictest839-21bash09:43:043
2596999140,2cyclictest839-21bash09:43:043
2595299149,3cyclictest9-21ksoftirqd/010:29:120
2595299149,2cyclictest9-21ksoftirqd/010:59:450
25952991413,1cyclictest9-21ksoftirqd/010:17:030
25952991413,1cyclictest9-21ksoftirqd/010:10:520
25952991413,1cyclictest9-21ksoftirqd/009:25:030
25952991412,1cyclictest9-21ksoftirqd/011:05:020
25952991412,1cyclictest9-21ksoftirqd/010:29:410
25952991411,3cyclictest9-21ksoftirqd/008:59:400
93552130,0sleep10-21swapper/110:10:231
308372130,0sleep20-21swapper/211:50:512
2597599138,5cyclictest0-21swapper/712:04:387
2597599138,5cyclictest0-21swapper/711:02:047
2597599138,5cyclictest0-21swapper/709:40:147
2597599138,5cyclictest0-21swapper/709:40:147
2597599138,5cyclictest0-21swapper/709:31:147
2597599138,5cyclictest0-21swapper/709:15:367
2597599138,5cyclictest0-21swapper/708:53:027
2597599138,5cyclictest0-21swapper/708:25:137
2597599138,5cyclictest0-21swapper/707:49:297
2597599138,5cyclictest0-21swapper/707:46:357
2597599138,1cyclictest8466-21bash11:11:527
2597599138,1cyclictest6451-21systemd-cgroups11:36:487
2597599138,1cyclictest2330-21sshd09:27:227
2597599138,0cyclictest0-21swapper/710:01:247
2597599138,0cyclictest0-21swapper/708:41:457
2597599138,0cyclictest0-21swapper/708:05:117
2597599138,0cyclictest0-21swapper/707:23:237
2597599137,5cyclictest0-21swapper/711:28:027
2597599137,5cyclictest0-21swapper/710:24:307
2597599137,5cyclictest0-21swapper/707:11:277
2597599135,0cyclictest0-21swapper/710:43:347
2597599132,5cyclictest1-21systemd08:58:187
2597599132,5cyclictest0-21swapper/711:15:127
2597599132,5cyclictest0-21swapper/711:15:127
2597599130,12cyclictest1-21systemd12:20:507
2597499137,6cyclictest0-21swapper/609:53:506
2597399138,5cyclictest0-21swapper/511:36:245
2597399138,5cyclictest0-21swapper/510:44:245
2597399138,5cyclictest0-21swapper/509:49:065
2597399138,5cyclictest0-21swapper/509:07:505
2597399138,5cyclictest0-21swapper/508:31:275
2597399138,5cyclictest0-21swapper/508:31:275
2597399138,5cyclictest0-21swapper/508:00:555
2597399138,5cyclictest0-21swapper/507:01:195
2597399138,1cyclictest9980-21kworker/5:111:21:305
2597399138,0cyclictest0-21swapper/512:18:285
2597399138,0cyclictest0-21swapper/508:27:415
2597399138,0cyclictest0-21swapper/508:11:315
2597399138,0cyclictest0-21swapper/507:50:575
2597399138,0cyclictest0-21swapper/507:37:095
2597399137,0cyclictest0-21swapper/507:12:095
2597399132,6cyclictest57-21ksoftirqd/510:00:345
2597399132,5cyclictest0-21swapper/509:40:225
2597399132,5cyclictest0-21swapper/509:40:225
2597399132,5cyclictest0-21swapper/506:55:175
2597399132,0cyclictest0-21swapper/508:44:275
2597399132,0cyclictest0-21swapper/508:44:275
2597399130,7cyclictest22317-21kworker/5:310:52:125
2597399130,7cyclictest0-21swapper/507:57:355
2597399130,7cyclictest0-21swapper/507:57:355
2596999131,6cyclictest2768-21grepconf.sh11:25:513
2596999131,5cyclictest6808-21sshd08:55:313
2596999131,4cyclictest7450-21grepconf.sh10:30:263
2596999131,1cyclictest0-21swapper/311:51:243
2596999130,5cyclictest9938-21grepconf.sh11:01:463
2596999130,5cyclictest711-21dbus-daemon10:07:073
2596999130,5cyclictest5943-21bash10:45:273
2596999130,5cyclictest21270-21grep10:12:243
2596999130,5cyclictest17078-21bash09:02:503
2596999130,5cyclictest13982-21bash09:50:003
2596999130,5cyclictest12929-21grepconf.sh10:36:293
2596999130,5cyclictest11615-21id10:16:183
2596999130,4cyclictest29883-21sshd11:35:113
2596999130,4cyclictest28897-21bash11:45:123
2596999130,4cyclictest21996-21grepconf.sh09:56:473
2596999130,4cyclictest10529-21rm09:28:433
2596999130,2cyclictest22715-21bash09:46:313
2596999130,2cyclictest0-21swapper/312:07:173
2595299132,1cyclictest9-21ksoftirqd/011:31:170
25952991312,1cyclictest9-21ksoftirqd/009:42:400
25952991312,1cyclictest9-21ksoftirqd/009:42:400
25952991312,1cyclictest9-21ksoftirqd/009:29:230
25952991312,1cyclictest9-21ksoftirqd/009:20:390
25952991311,1cyclictest9-21ksoftirqd/012:19:330
25952991311,1cyclictest9-21ksoftirqd/012:09:350
25952991311,1cyclictest9-21ksoftirqd/012:07:120
25952991311,1cyclictest9-21ksoftirqd/011:21:460
25952991311,1cyclictest9-21ksoftirqd/011:16:010
25952991311,1cyclictest9-21ksoftirqd/011:16:010
25952991311,1cyclictest9-21ksoftirqd/009:49:430
2595299131,1cyclictest131rcu_sched08:58:280
120202130,0sleep40-21swapper/409:18:134
2597599127,5cyclictest0-21swapper/711:47:227
2597599127,5cyclictest0-21swapper/710:55:007
2597599127,5cyclictest0-21swapper/710:51:247
2597599127,5cyclictest0-21swapper/710:05:427
2597599127,5cyclictest0-21swapper/709:49:467
2597599127,5cyclictest0-21swapper/709:01:107
2597599127,0cyclictest0-21swapper/712:00:187
2597599127,0cyclictest0-21swapper/711:31:527
2597599127,0cyclictest0-21swapper/711:23:367
2597599120,0cyclictest0-21swapper/711:50:007
2597399127,5cyclictest0-21swapper/508:21:595
2597399127,5cyclictest0-21swapper/508:16:455
2597399127,4cyclictest1-21systemd08:59:325
2597399127,0cyclictest0-21swapper/507:32:115
2597399126,1cyclictest8587-21kworker/5:112:09:385
2597399126,1cyclictest57-21ksoftirqd/511:26:525
25969991211,1cyclictest41-21ksoftirqd/309:10:123
2596999121,2cyclictest1884-21grepconf.sh12:10:223
2596999120,5cyclictest28523-21bash11:14:483
2596999120,5cyclictest28523-21bash11:14:483
2596999120,5cyclictest15035-21bash09:29:233
2596999120,4cyclictest955-21sshd10:49:443
2596999120,4cyclictest5736-21bash12:20:253
2596999120,4cyclictest2215-21sshd11:05:413
2596999120,4cyclictest1812-21grep12:01:073
2596999120,4cyclictest15399-21bash09:34:563
2596999120,4cyclictest15399-21bash09:34:563
2596999120,4cyclictest12331-21awk07:24:403
2596999120,3cyclictest3443-21sshd12:15:213
2596999120,2cyclictest24396-21bash11:19:243
2595899120,11cyclictest19861-21bash11:08:461
2595299129,1cyclictest9-21ksoftirqd/011:59:240
25952991211,1cyclictest9-21ksoftirqd/009:44:330
25952991210,1cyclictest9-21ksoftirqd/010:19:310
25952991210,1cyclictest9-21ksoftirqd/009:37:390
25952991210,1cyclictest9-21ksoftirqd/009:37:390
25952991210,1cyclictest9-21ksoftirqd/009:10:500
25952991210,1cyclictest9-21ksoftirqd/009:05:370
2595299120,11cyclictest28923-21kworker/0:309:16:460
141492120,0sleep40-21swapper/411:28:114
5142110,0sleep60-21swapper/612:00:566
325872110,0sleep40-21swapper/411:40:524
274402110,0sleep12595899cyclictest09:47:171
272992110,0sleep60-21swapper/610:18:536
2597599116,5cyclictest0-21swapper/709:55:107
2597499119,2cyclictest85350irq/25-em1-rx-009:35:116
2597499119,2cyclictest85350irq/25-em1-rx-009:35:116
2597499115,6cyclictest0-21swapper/610:12:526
2597499110,1cyclictest0-21swapper/611:27:166
2597399110,0cyclictest0-21swapper/507:20:195
2597099110,2cyclictest1252-21bash10:29:374
2596999110,5cyclictest23982-21ntp_states07:44:413
2596999110,5cyclictest13520-21df_inode08:24:353
2596999110,2cyclictest18271-21ntp_states07:34:443
2596199119,1cyclictest33-21ksoftirqd/209:00:202
2596199110,10cyclictest7546-21grepconf.sh11:06:332
2595899118,2cyclictest8884-21sshd09:34:011
2595299119,1cyclictest9-21ksoftirqd/011:44:330
25952991110,1cyclictest9-21ksoftirqd/009:56:300
127862110,0sleep40-21swapper/410:10:584
64962100,0sleep40-21swapper/411:16:334
64962100,0sleep40-21swapper/411:16:334
2597499109,1cyclictest65-21ksoftirqd/610:36:556
2597499109,1cyclictest0-21swapper/609:47:226
2597499107,1cyclictest85350irq/25-em1-rx-012:10:076
2597499103,6cyclictest13331-21sshd10:31:276
2597499100,10cyclictest0-21swapper/609:31:336
2597399105,5cyclictest14191-21kworker/5:010:29:545
2597099109,1cyclictest49-21ksoftirqd/409:23:384
2597099109,1cyclictest0-21swapper/412:05:004
2597099107,2cyclictest4202-21grep09:38:424
2597099107,2cyclictest4202-21grep09:38:424
2597099103,6cyclictest26537-21sshd11:55:144
2597099103,1cyclictest7827-21bash11:01:254
2596999100,5cyclictest2089-21diskstats08:04:363
2596199109,1cyclictest33-21ksoftirqd/211:55:002
2596199109,1cyclictest33-21ksoftirqd/210:17:012
2596199109,1cyclictest33-21ksoftirqd/209:20:382
2596199109,1cyclictest0-21swapper/209:52:092
2596199108,1cyclictest33-21ksoftirqd/212:03:592
2596199108,1cyclictest1-21systemd11:13:012
2596199108,1cyclictest1-21systemd09:10:342
2596199100,9cyclictest27278-21bash10:08:232
2595899106,1cyclictest17404-21kworker/1:011:40:401
2595899103,6cyclictest26153-21sshd12:09:261
2595899100,4cyclictest15885-21sshd09:35:011
2595899100,4cyclictest15885-21sshd09:35:011
2595899100,4cyclictest1-21systemd11:17:101
2595899100,4cyclictest1-21systemd11:17:101
105692100,0sleep60-21swapper/610:51:376
4822290,0sleep10-21swapper/111:46:341
4406290,0sleep40-21swapper/410:35:014
259749999,0cyclictest0-21swapper/610:03:046
259749999,0cyclictest0-21swapper/609:19:536
259749999,0cyclictest0-21swapper/608:56:146
259749998,1cyclictest0-21swapper/610:44:426
259749995,1cyclictest32093-21id10:24:326
259749990,9cyclictest17899-21kworker/6:111:18:246
259749990,9cyclictest17899-21kworker/6:111:18:246
259749990,9cyclictest0-21swapper/611:05:406
259749990,9cyclictest0-21swapper/608:39:406
259749990,0cyclictest0-21swapper/610:56:036
259749990,0cyclictest0-21swapper/609:14:316
259709997,1cyclictest49-21ksoftirqd/409:53:314
259709997,1cyclictest49-21ksoftirqd/409:39:594
259709997,1cyclictest49-21ksoftirqd/409:39:594
259709992,2cyclictest5317-21bash12:15:384
259709991,3cyclictest0-21swapper/409:09:374
259709990,9cyclictest9884-21kworker/4:210:21:424
259709990,9cyclictest0-21swapper/410:52:574
259699998,1cyclictest30558-21kworker/3:107:59:373
259699991,1cyclictest41-21ksoftirqd/307:24:073
259699991,1cyclictest0-21swapper/308:40:353
259699991,1cyclictest0-21swapper/308:37:283
259699991,1cyclictest0-21swapper/308:37:283
259699991,1cyclictest0-21swapper/308:09:423
259699991,1cyclictest0-21swapper/307:14:423
259699990,1cyclictest41-21ksoftirqd/306:59:343
259619998,1cyclictest33-21ksoftirqd/209:44:442
259619997,1cyclictest33-21ksoftirqd/212:13:302
259619996,2cyclictest33-21ksoftirqd/211:30:092
259619993,3cyclictest0-21swapper/211:41:042
259619990,9cyclictest0-21swapper/209:17:372
259619990,0cyclictest0-21swapper/211:14:452
259619990,0cyclictest0-21swapper/211:14:452
259619990,0cyclictest0-21swapper/210:56:562
259619990,0cyclictest0-21swapper/209:26:082
259589999,0cyclictest0-21swapper/110:41:391
259589999,0cyclictest0-21swapper/110:21:281
259589997,1cyclictest8230-21sshd12:20:491
259589994,3cyclictest24026-21sshd11:59:401
259589993,6cyclictest14515-21sshd10:57:251
259589993,3cyclictest668-21systemd-logind11:09:381
259589990,4cyclictest8496-21sshd09:49:261
259589990,4cyclictest0-21swapper/110:29:571
259589990,0cyclictest0-21swapper/111:34:471
259589990,0cyclictest0-21swapper/110:17:031
259529999,0cyclictest0-21swapper/007:44:350
259529998,1cyclictest9-21ksoftirqd/008:29:210
259529998,1cyclictest32383-21kworker/0:007:50:030
259529997,1cyclictest9-21ksoftirqd/008:04:220
8057280,1sleep28168-21grepconf.sh10:45:492
5893280,1sleep40-21swapper/410:45:274
27611280,0sleep20-21swapper/209:57:432
259749988,0cyclictest0-21swapper/611:01:436
259749986,1cyclictest861-21gdbus11:49:536
259749986,1cyclictest65-21ksoftirqd/612:23:156
259749984,2cyclictest711-21dbus-daemon09:01:226
259749983,4cyclictest19852-21grep11:59:166
259749983,3cyclictest711-21dbus-daemon11:20:326
259749983,2cyclictest711-21dbus-daemon09:40:226
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional