You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 20:59
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Wed Dec 03, 2025 12:43:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
57402850,0sleep30-21swapper/309:55:373
280402660,0sleep50-21swapper/510:50:575
110862590,0sleep50-21swapper/509:40:415
209682550,0sleep00-21swapper/009:53:020
1312625226,9sleep60-21swapper/606:49:236
90492470,0sleep40-21swapper/412:21:224
278632440,2sleep31372699cyclictest11:11:293
315472420,0sleep30-21swapper/307:24:103
308572420,0sleep10-21swapper/110:46:121
12572420,0sleep60-21swapper/611:54:256
295542410,0sleep10-21swapper/109:22:271
215242410,0sleep10-21swapper/110:14:091
178842400,0sleep00-21swapper/009:20:100
1337624028,9sleep30-21swapper/306:52:403
1317424027,9sleep50-21swapper/506:49:545
1314924028,8sleep70-21swapper/706:49:327
17892390,0sleep50-21swapper/511:17:405
1320723926,9sleep10-21swapper/106:50:221
1319123928,8sleep40-21swapper/406:50:094
1314223926,9sleep00-21swapper/006:49:250
197912380,0sleep70-21swapper/712:02:437
1330623827,8sleep20-21swapper/206:51:432
219232190,0sleep20-21swapper/209:42:312
39312180,0sleep40-21swapper/411:12:584
210202150,0sleep40-21swapper/411:47:264
1372199150,12cyclictest28783-21sshd09:59:210
230312140,0sleep40-21swapper/410:24:294
13728991410,2cyclictest28104-21nfsd11:33:315
116832140,1sleep773-21ksoftirqd/712:06:277
233702130,0sleep70-21swapper/710:34:337
206782130,0sleep10-21swapper/109:58:151
13733991312,1cyclictest65-21ksoftirqd/609:46:226
13728991311,2cyclictest57-21ksoftirqd/511:40:325
13728991311,1cyclictest15653-21kworker/u16:012:18:195
13725991312,1cyclictest33-21ksoftirqd/210:07:212
1372199131,11cyclictest21368-21sshd09:05:100
108562130,0sleep50-21swapper/511:56:005
34362120,1sleep449-21ksoftirqd/410:52:164
13733991210,1cyclictest65-21ksoftirqd/612:15:486
13733991210,1cyclictest65-21ksoftirqd/612:13:586
1372899128,3cyclictest25060-21cp09:32:455
1372899120,1cyclictest0-21swapper/510:45:355
13725991210,1cyclictest33-21ksoftirqd/211:39:162
1372499122,4cyclictest26066-21sshd10:55:461
13721991211,1cyclictest9-21ksoftirqd/012:10:020
13721991211,1cyclictest9-21ksoftirqd/010:42:440
1373599119,1cyclictest711-21dbus-daemon09:19:457
1373599111,4cyclictest9089-21sshd09:56:167
1373599110,10cyclictest23852-21sshd11:15:577
1373399110,5cyclictest27436-21sshd11:06:136
1373399110,5cyclictest27436-21sshd11:06:136
1372899119,2cyclictest8887-21kworker/u16:008:59:175
1372899119,2cyclictest28105-21nfsd10:07:335
1372899119,1cyclictest57-21ksoftirqd/509:04:455
1372899116,0cyclictest0-21swapper/508:21:295
1372899110,10cyclictest27365-21kworker/5:010:15:025
1372899110,1cyclictest0-21swapper/509:55:005
1372799119,1cyclictest49-21ksoftirqd/411:59:384
1372799114,4cyclictest10501-21sshd10:17:384
13727991110,1cyclictest0-21swapper/409:00:374
1372799110,11cyclictest0-21swapper/411:52:154
1372799110,11cyclictest0-21swapper/411:52:154
1372699110,1cyclictest31775-21sshd09:17:213
1372599119,1cyclictest33-21ksoftirqd/212:09:442
1372599110,10cyclictest24510-21kworker/2:109:29:192
1372599110,10cyclictest10914-21sshd09:14:032
1372599110,10cyclictest0-21swapper/211:48:232
1372499118,2cyclictest15508-21bash12:01:511
1372499110,1cyclictest4046-21sshd11:49:331
1372499110,1cyclictest4046-21sshd11:49:331
13721991110,1cyclictest9-21ksoftirqd/012:01:170
13721991110,1cyclictest9-21ksoftirqd/009:13:400
1372199110,10cyclictest12581-21sshd09:30:330
112172110,0sleep70-21swapper/709:30:187
104022110,0sleep10-21swapper/109:45:501
1373599109,1cyclictest0-21swapper/711:32:307
1373599109,1cyclictest0-21swapper/708:55:387
1373599100,4cyclictest668-21systemd-logind10:39:447
1373599100,4cyclictest31739-21sshd11:12:107
1373599100,10cyclictest0-21swapper/710:50:057
1373399109,1cyclictest65-21ksoftirqd/611:16:206
1373399109,1cyclictest65-21ksoftirqd/610:49:196
1373399104,3cyclictest8428-21sshd10:27:296
1373399104,3cyclictest13372-21sshd09:19:236
1372899109,1cyclictest57-21ksoftirqd/511:24:545
1372899109,1cyclictest0-21swapper/509:16:425
1372899107,1cyclictest22059-21kworker/u16:107:08:505
1372899106,3cyclictest57-21ksoftirqd/510:00:275
1372799109,1cyclictest49-21ksoftirqd/409:09:174
1372799108,1cyclictest49-21ksoftirqd/411:54:374
1372799107,1cyclictest49-21ksoftirqd/409:21:204
1372799106,1cyclictest0-21swapper/410:44:164
1372799100,10cyclictest0-21swapper/412:05:584
1372699100,10cyclictest0-21swapper/309:52:403
1372699100,1cyclictest0-21swapper/310:07:043
1372599109,1cyclictest33-21ksoftirqd/211:02:242
1372599108,1cyclictest33-21ksoftirqd/209:54:152
1372499109,1cyclictest0-21swapper/111:39:471
1372499108,1cyclictest23863-21kworker/1:211:56:011
1372499102,2cyclictest2032-21grepconf.sh11:07:211
1372499102,2cyclictest2032-21grepconf.sh11:07:211
1372499101,4cyclictest0-21swapper/109:00:351
1372199109,1cyclictest9-21ksoftirqd/012:04:370
1372199109,1cyclictest9-21ksoftirqd/011:28:390
1372199109,1cyclictest9-21ksoftirqd/010:43:590
1372199109,1cyclictest9-21ksoftirqd/009:17:420
1372199109,1cyclictest9-21ksoftirqd/009:03:580
1372199108,1cyclictest9-21ksoftirqd/011:49:180
1372199108,1cyclictest9-21ksoftirqd/011:49:180
1372199108,1cyclictest9-21ksoftirqd/009:56:540
1372199105,3cyclictest9-21ksoftirqd/011:54:300
1372199104,4cyclictest0-21swapper/010:25:180
1372199104,3cyclictest28821-21sshd11:01:280
1372199104,3cyclictest23927-21iostat_ios10:04:130
23843290,2sleep523968-21grepconf.sh11:21:015
137359998,1cyclictest0-21swapper/709:15:437
137359997,1cyclictest73-21ksoftirqd/710:57:417
137359993,1cyclictest13747-21sshd10:02:377
137359992,6cyclictest25397-21sshd11:42:517
137359992,6cyclictest0-21swapper/709:28:197
137359991,4cyclictest3114-21sshd11:49:257
137359991,4cyclictest3114-21sshd11:49:257
137359991,3cyclictest0-21swapper/711:54:087
137359990,9cyclictest0-21swapper/711:03:397
137359990,9cyclictest0-21swapper/710:06:067
137359990,9cyclictest0-21swapper/709:52:047
137359990,4cyclictest0-21swapper/709:10:437
137359990,0cyclictest0-21swapper/712:14:357
137339998,1cyclictest85350irq/25-em1-rx-012:19:056
137339998,1cyclictest65-21ksoftirqd/610:54:016
137339998,1cyclictest65-21ksoftirqd/609:54:396
137339997,1cyclictest65-21ksoftirqd/611:44:326
137339996,1cyclictest65-21ksoftirqd/610:38:226
137339994,3cyclictest12111-21sshd10:23:016
137339993,3cyclictest21286-21proc_pri08:54:196
137339992,4cyclictest8140-21cp09:50:516
137339990,1cyclictest0-21swapper/612:06:036
137289999,0cyclictest0-21swapper/512:06:485
137289997,1cyclictest57-21ksoftirqd/510:57:425
137289997,1cyclictest57-21ksoftirqd/510:25:275
137289997,1cyclictest17755-21cp11:35:555
137289992,4cyclictest4176-21sshd11:02:455
137289990,3cyclictest0-21swapper/511:10:415
137289990,3cyclictest0-21swapper/507:16:125
137289990,1cyclictest0-21swapper/512:19:365
137279999,0cyclictest0-21swapper/409:47:534
137279993,3cyclictest24559-21sshd09:16:024
137279992,4cyclictest30661-21rm11:43:544
137279992,3cyclictest22552-21systemd-cgroups08:54:264
137279990,9cyclictest3656-21kworker/4:409:07:004
137279990,9cyclictest0-21swapper/411:04:024
137279990,9cyclictest0-21swapper/411:04:024
137279990,9cyclictest0-21swapper/409:39:504
137279990,0cyclictest0-21swapper/409:30:324
137269999,0cyclictest0-21swapper/309:00:573
137269992,3cyclictest11749-21sshd11:40:173
137269990,9cyclictest0-21swapper/311:16:323
137269990,9cyclictest0-21swapper/310:51:573
137269990,9cyclictest0-21swapper/310:16:573
137269990,9cyclictest0-21swapper/309:40:223
137269990,4cyclictest9229-21sshd10:01:293
137269990,0cyclictest0-21swapper/311:44:083
137259999,0cyclictest0-21swapper/210:00:332
137259998,1cyclictest33-21ksoftirqd/211:18:432
137259998,1cyclictest33-21ksoftirqd/210:46:132
137259998,1cyclictest22407-21sshd11:25:502
137259998,1cyclictest21359-21sshd09:47:472
137259994,3cyclictest4736-21sshd10:11:272
137259994,3cyclictest28321-21bash10:35:242
137259994,3cyclictest1035-21sshd11:49:182
137259994,3cyclictest1035-21sshd11:49:182
137259994,3cyclictest0-21swapper/209:53:532
137259990,9cyclictest7712-21sshd10:27:212
137259990,9cyclictest0-21swapper/210:55:152
137259990,9cyclictest0-21swapper/210:49:442
137259990,9cyclictest0-21swapper/210:19:472
137259990,9cyclictest0-21swapper/209:04:072
137249999,0cyclictest0-21swapper/110:32:001
137249999,0cyclictest0-21swapper/110:24:251
137249993,6cyclictest15734-21sshd11:35:331
137249993,3cyclictest10011-21nscd11:29:161
137249992,6cyclictest19168-21sshd10:34:091
137249992,6cyclictest16339-21sshd12:07:181
137249990,9cyclictest0-21swapper/112:13:221
137249990,9cyclictest0-21swapper/110:59:541
137249990,9cyclictest0-21swapper/109:59:211
137249990,4cyclictest8914-21sshd11:45:221
137249990,4cyclictest0-21swapper/109:14:291
137249990,0cyclictest0-21swapper/110:10:521
137219999,0cyclictest0-21swapper/011:40:170
137219998,1cyclictest9-21ksoftirqd/012:20:300
137219998,1cyclictest9-21ksoftirqd/012:15:380
137219998,1cyclictest28104-21nfsd11:19:290
137219998,1cyclictest0-21swapper/010:49:220
137219997,1cyclictest9-21ksoftirqd/009:44:060
137219997,1cyclictest9-21ksoftirqd/009:39:010
137219994,3cyclictest25211-21sshd09:27:330
137219994,2cyclictest28861-21sshd10:35:300
137219993,5cyclictest24396-21sshd11:05:420
137219993,5cyclictest24396-21sshd11:05:420
137219992,3cyclictest6759-21sshd09:34:510
137219990,9cyclictest0-21swapper/011:44:330
137219990,9cyclictest0-21swapper/011:34:460
4343280,0sleep30-21swapper/308:29:123
3626280,0sleep30-21swapper/310:36:533
137359986,1cyclictest73-21ksoftirqd/711:05:447
137359986,1cyclictest73-21ksoftirqd/711:05:447
137359986,1cyclictest73-21ksoftirqd/709:48:587
137359986,1cyclictest73-21ksoftirqd/709:04:107
137359986,1cyclictest30597-21grepconf.sh10:10:227
137359985,2cyclictest711-21dbus-daemon09:36:237
137359985,2cyclictest21316-21sshd10:24:187
137359985,2cyclictest17193-21grepconf.sh11:24:537
137359983,1cyclictest29165-21grepconf.sh11:48:447
137359982,3cyclictest9787-21perf07:43:587
137359982,3cyclictest0-21swapper/711:23:447
137359980,7cyclictest669-21avahi-daemon12:19:287
137359980,2cyclictest4197-21sshd10:16:387
137339987,1cyclictest85350irq/25-em1-rx-008:04:156
137339987,1cyclictest65-21ksoftirqd/611:35:346
137339987,1cyclictest65-21ksoftirqd/610:04:166
137339987,1cyclictest0-21swapper/611:00:566
137339986,1cyclictest65-21ksoftirqd/611:20:036
137339986,1cyclictest65-21ksoftirqd/610:13:236
137339986,1cyclictest65-21ksoftirqd/609:36:106
137339986,1cyclictest65-21ksoftirqd/609:27:296
137339985,2cyclictest25574-21grep09:32:496
137339983,2cyclictest0-21swapper/610:29:586
137339982,3cyclictest577-21bash09:17:336
137339982,3cyclictest17496-21munin-run08:53:586
137339982,3cyclictest0-21swapper/610:17:116
137339981,3cyclictest14668-21processes09:09:186
137339980,5cyclictest5017-21bash10:42:176
137289987,1cyclictest57-21ksoftirqd/509:09:205
137289987,1cyclictest0-21swapper/510:38:125
137289986,1cyclictest57-21ksoftirqd/510:40:025
137289986,1cyclictest57-21ksoftirqd/509:35:335
137289985,2cyclictest6089-21grep09:45:065
137289985,2cyclictest26439-21bash10:09:415
137289985,1cyclictest57-21ksoftirqd/508:54:015
137289984,2cyclictest2485950sshd11:05:485
137289984,2cyclictest2485950sshd11:05:485
137289983,2cyclictest0-21swapper/507:39:065
137289983,0cyclictest0-21swapper/508:18:045
137289982,4cyclictest711-21dbus-daemon11:49:075
137289982,4cyclictest711-21dbus-daemon11:49:075
137289982,3cyclictest9037-21sshd09:50:595
137289982,3cyclictest5545-21bash11:44:465
137289980,5cyclictest711-21dbus-daemon12:00:205
137289980,5cyclictest18979-21sshd09:26:075
137289980,2cyclictest0-21swapper/510:19:555
137289980,2cyclictest0-21swapper/508:40:415
137279988,0cyclictest0-21swapper/410:57:154
137279986,1cyclictest49-21ksoftirqd/411:24:444
137279985,2cyclictest711-21dbus-daemon12:13:574
137279985,2cyclictest24997-21bash10:34:524
137279984,1cyclictest49-21ksoftirqd/411:34:214
137279984,1cyclictest20414-21sshd10:03:514
137279983,4cyclictest17670-21sshd10:59:274
137279983,2cyclictest31181-21sshd09:49:144
137279983,2cyclictest0-21swapper/408:21:314
137279983,1cyclictest3790-21sshd10:42:054
137279982,4cyclictest31974-21sshd09:34:004
137279982,4cyclictest2611-21sshd10:21:244
137279982,3cyclictest27920-21sshd12:14:134
137279982,3cyclictest0-21swapper/410:10:474
137269988,0cyclictest0-21swapper/312:21:583
137269988,0cyclictest0-21swapper/310:58:433
137269988,0cyclictest0-21swapper/310:43:543
137269988,0cyclictest0-21swapper/310:13:573
137269987,1cyclictest0-21swapper/311:28:503
137269987,1cyclictest0-21swapper/311:23:193
137269986,1cyclictest41-21ksoftirqd/312:18:303
137269986,1cyclictest41-21ksoftirqd/309:12:093
137269986,1cyclictest0-21swapper/311:50:543
137269986,1cyclictest0-21swapper/311:50:543
137269985,2cyclictest16362-21grep09:25:253
137269985,2cyclictest14340-21grep11:56:323
137269984,1cyclictest19627-21sshd09:20:273
137269983,2cyclictest0-21swapper/306:59:343
137269981,3cyclictest8811-21nscd09:29:223
137259988,0cyclictest0-21swapper/210:15:322
137259988,0cyclictest0-21swapper/209:12:182
137259987,1cyclictest33-21ksoftirqd/209:00:482
137259986,1cyclictest33-21ksoftirqd/211:37:022
137259983,3cyclictest25325-21grepconf.sh09:21:402
137259982,3cyclictest30150-21grepconf.sh12:19:252
137259982,3cyclictest24763-21rm11:31:272
137259982,3cyclictest24671-21grepconf.sh11:05:452
137259982,3cyclictest24671-21grepconf.sh11:05:452
137249988,0cyclictest0-21swapper/111:24:291
137249988,0cyclictest0-21swapper/110:20:381
137249988,0cyclictest0-21swapper/110:07:521
137249988,0cyclictest0-21swapper/109:10:201
137249985,2cyclictest21521-21bash08:54:201
137249984,3cyclictest4084-21sshd09:34:261
137249982,1cyclictest15430-21sshd12:22:251
137249981,6cyclictest0-21swapper/109:42:261
137249980,0cyclictest0-21swapper/109:26:431
137219988,0cyclictest0-21swapper/010:16:540
137219988,0cyclictest0-21swapper/010:13:540
137219988,0cyclictest0-21swapper/008:04:180
137219987,1cyclictest0-21swapper/008:53:580
137219986,1cyclictest9-21ksoftirqd/011:30:320
137219986,1cyclictest9-21ksoftirqd/011:09:500
137219986,1cyclictest9-21ksoftirqd/008:54:310
137219984,2cyclictest21181-21sshd10:19:140
137219983,4cyclictest16427-21perf06:58:580
137219983,3cyclictest5962-21sshd11:18:210
137219983,1cyclictest0-21swapper/008:09:240
137219980,0cyclictest0-21swapper/010:54:530
137359977,0cyclictest0-21swapper/707:04:207
137359975,1cyclictest711-21dbus-daemon10:45:157
137359974,2cyclictest27828-21grepconf.sh12:09:077
137359974,2cyclictest16653-21grepconf.sh08:59:147
137359970,6cyclictest31897-21bash09:39:017
137359970,4cyclictest0-21swapper/711:34:337
137359970,1cyclictest18992-21grepconf.sh10:19:007
137359970,1cyclictest0-21swapper/710:30:167
137339977,0cyclictest85350irq/25-em1-rx-007:24:136
137339975,1cyclictest65-21ksoftirqd/611:59:126
137339975,1cyclictest65-21ksoftirqd/607:37:216
137339975,1cyclictest0-21swapper/611:39:436
137339974,2cyclictest8624-21bash11:29:006
137339974,2cyclictest31770-21rm09:39:006
137339974,2cyclictest31362-21grep11:49:046
137339974,2cyclictest31362-21grep11:49:046
137339974,2cyclictest17971-21bash10:44:146
137339974,2cyclictest16122-21grepconf.sh11:24:416
137339972,3cyclictest0-21swapper/611:09:156
137339971,3cyclictest32487-21bash09:59:536
137339971,3cyclictest15528-21perf07:53:586
137339971,3cyclictest13289-21cstates09:04:096
137339971,2cyclictest15622-21bash08:59:106
137339970,4cyclictest19259-21munin-run07:03:586
137289975,2cyclictest99-21kswapd008:29:525
137289975,1cyclictest57-21ksoftirqd/507:30:135
137289974,2cyclictest9754-21grepconf.sh09:19:065
137289974,2cyclictest763-21runrttasks07:37:165
137289974,2cyclictest24947-21bash10:29:515
137289972,4cyclictest27823-21bash12:09:075
137289972,2cyclictest0-21swapper/507:26:145
137279977,0cyclictest0-21swapper/408:39:184
137279975,1cyclictest49-21ksoftirqd/408:04:114
137279974,2cyclictest31244-21grepconf.sh09:54:264
137279974,2cyclictest25694-21bash10:29:594
137279974,2cyclictest16625-21bash11:19:464
137279974,2cyclictest11239-21bash09:24:324
137279974,2cyclictest10625-21grepconf.sh11:14:034
137279972,3cyclictest8599-21sshd11:29:004
137279972,2cyclictest711-21dbus-daemon10:03:594
137269977,0cyclictest0-21swapper/311:34:213
137269976,1cyclictest41-21ksoftirqd/310:59:113
137269976,1cyclictest41-21ksoftirqd/309:04:293
137269975,1cyclictest41-21ksoftirqd/307:21:433
137269974,2cyclictest9257-21bash11:29:063
137269974,2cyclictest861-21gdbus12:01:393
137269974,2cyclictest5152-21bash09:44:553
137269974,2cyclictest30487-21bash10:46:093
137269974,2cyclictest27685-21bash12:09:063
137269974,2cyclictest24991-21bash08:54:543
137269974,2cyclictest21460-21sshd11:05:123
137269974,2cyclictest21460-21sshd11:05:123
137269974,2cyclictest19767-21bash10:29:123
137269974,2cyclictest10958-21sshd09:35:353
137269974,2cyclictest10934-21grepconf.sh12:06:193
137269972,2cyclictest0-21swapper/308:16:433
137269970,1cyclictest23212-21sshd10:19:223
137269970,1cyclictest0-21swapper/310:24:573
137259976,1cyclictest33-21ksoftirqd/212:14:302
137259976,1cyclictest33-21ksoftirqd/212:00:152
137259976,1cyclictest33-21ksoftirqd/209:24:592
137259976,1cyclictest0-21swapper/212:05:252
137259975,1cyclictest33-21ksoftirqd/208:57:432
137259974,2cyclictest9480-21bash10:43:022
137259974,2cyclictest6963-21bash09:34:542
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional