You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-09 - 21:07
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Sun Nov 09, 2025 12:43:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
97882920,0sleep3391rcuc/311:09:233
3236125550,2sleep10-21swapper/106:50:401
257282500,1sleep09-21ksoftirqd/011:38:350
92112490,0sleep10-21swapper/112:07:121
22782490,0sleep60-21swapper/609:53:586
83782480,0sleep30-21swapper/311:47:113
254772480,0sleep40-21swapper/412:05:544
155332480,0sleep40-21swapper/412:15:074
132432480,0sleep60-21swapper/612:12:296
3257424635,8sleep40-21swapper/406:53:394
91802440,0sleep70-21swapper/710:39:247
236462440,1sleep523644-21sshd11:40:495
220292440,1sleep141699cyclictest10:31:251
129482440,0sleep212946-21sshd09:17:582
121892440,0sleep50-21swapper/511:47:295
107202430,0sleep50-21swapper/511:32:245
6552420,0sleep10-21swapper/107:50:041
64572420,0sleep70-21swapper/711:39:387
298242420,0sleep20-21swapper/211:51:192
145602420,2sleep646099cyclictest12:17:346
70182410,0sleep10-21swapper/111:17:541
45202410,0sleep70-21swapper/710:06:087
3262924128,9sleep60-21swapper/606:54:216
3259324128,9sleep50-21swapper/506:53:565
3246124130,8sleep70-21swapper/706:52:067
3239024129,9sleep30-21swapper/306:51:063
183732410,0sleep30-21swapper/310:31:033
73372400,1sleep665-21ksoftirqd/612:02:046
3248724029,8sleep20-21swapper/206:52:252
3232924028,8sleep00-21swapper/006:50:150
312932400,1sleep431291-21grepconf.sh10:56:194
290812400,0sleep30-21swapper/309:46:583
230392400,0sleep10-21swapper/111:35:481
45382380,0sleep50-21swapper/509:01:515
204972360,2sleep646099cyclictest10:34:186
153162350,0sleep70-21swapper/710:52:057
45299260,10cyclictest711-21dbus-daemon11:18:075
46099250,2cyclictest14711-21grepconf.sh10:46:016
45299250,14cyclictest18443-21sshd10:37:145
45299220,11cyclictest31555-21grep09:56:385
46099210,2cyclictest12130-21grepconf.sh11:30:026
45299210,7cyclictest27999-21grepconf.sh10:47:185
45299210,7cyclictest0-21swapper/511:38:165
45299210,6cyclictest0-21swapper/512:04:535
45299210,10cyclictest12307-21bash10:57:465
46699200,19cyclictest7966-21sshd12:22:027
45299200,7cyclictest1428-21bash11:26:205
45299200,6cyclictest6765-21grepconf.sh10:27:055
45299200,11cyclictest31155-21grep10:50:265
45299200,10cyclictest5867-21bash12:01:575
45299200,10cyclictest19042-21bash09:45:595
44499204,8cyclictest16001-21grepconf.sh10:30:504
46699190,18cyclictest29081-21systemd-cgroups11:53:507
46099190,2cyclictest15733-21id11:47:466
46099190,11cyclictest6457-21sshd09:20:106
46099190,1cyclictest0-21swapper/611:07:326
45299190,10cyclictest8543-21rm09:08:515
45299190,10cyclictest8231-21bash11:59:445
45299190,10cyclictest8231-21bash11:59:445
44499190,2cyclictest10961-21sshd11:15:154
44499190,18cyclictest711-21dbus-daemon09:47:104
46699188,5cyclictest0-21swapper/710:47:207
46699180,15cyclictest7503-21bash10:27:107
46699180,14cyclictest17964-21sshd11:30:227
45299180,9cyclictest2828-21sshd11:14:425
45299180,9cyclictest18390-21sshd10:22:145
45299180,9cyclictest16370-21sshd11:52:505
45299180,8cyclictest4987-21id11:08:575
45299180,7cyclictest31373-21cp09:53:345
45299180,7cyclictest13209-21grepconf.sh10:13:045
45299180,6cyclictest25429-21sed10:05:065
45299180,6cyclictest23274-21sshd09:10:095
45299180,5cyclictest0-21swapper/512:18:475
45299180,10cyclictest29901-21sshd09:19:305
408991814,3cyclictest9-21ksoftirqd/010:08:170
40899180,13cyclictest12813-21bash09:58:010
46699176,5cyclictest0-21swapper/711:20:427
46699176,5cyclictest0-21swapper/707:14:547
46699175,12cyclictest0-21swapper/707:23:587
46699175,11cyclictest13150-21bash12:10:027
46699175,0cyclictest0-21swapper/711:27:407
46699170,16cyclictest669-21avahi-daemon12:19:427
46699170,16cyclictest668-21systemd-logind09:11:267
46099170,2cyclictest32620-21bash09:41:006
46099170,2cyclictest30642-21grep09:04:276
45299170,9cyclictest711-21dbus-daemon09:42:165
45299170,8cyclictest21872-21bash12:20:335
45299170,8cyclictest1671-21bash09:19:535
45299170,8cyclictest10107-21grepconf.sh10:03:405
45299170,7cyclictest0-21swapper/508:59:175
45299170,6cyclictest758-21sshd10:14:555
45299170,6cyclictest0-21swapper/509:37:585
46699166,9cyclictest14571-21sshd09:45:307
46699166,10cyclictest0-21swapper/711:06:047
46699165,5cyclictest0-21swapper/707:31:007
46699165,11cyclictest0-21swapper/708:21:367
46699165,11cyclictest0-21swapper/706:56:227
46699165,11cyclictest0-21swapper/706:56:227
46699165,10cyclictest17144-21grep10:58:147
46699164,9cyclictest10191-21sshd09:05:227
46699164,5cyclictest0-21swapper/708:41:067
46699164,12cyclictest0-21swapper/708:38:127
46699164,12cyclictest0-21swapper/708:10:267
46699164,12cyclictest0-21swapper/707:55:307
46699160,16cyclictest0-21swapper/707:00:227
46699160,15cyclictest30970-21sshd11:02:327
46699160,10cyclictest0-21swapper/708:05:307
46099160,7cyclictest0-21swapper/612:05:006
46099160,2cyclictest9282-21sshd09:38:596
46099160,1cyclictest0-21swapper/611:22:156
45299160,7cyclictest8749-21grepconf.sh11:21:165
45299160,7cyclictest23801-21sshd09:31:075
45299160,6cyclictest0-21swapper/511:00:545
44499160,2cyclictest13275-21grepconf.sh10:15:494
44499160,11cyclictest711-21dbus-daemon09:40:074
408991613,1cyclictest9-21ksoftirqd/012:13:580
26342160,2sleep040899cyclictest09:38:220
146612160,0sleep20-21swapper/210:52:002
105122160,0sleep60-21swapper/612:22:146
46699157,7cyclictest711-21dbus-daemon11:43:207
46699156,9cyclictest0-21swapper/711:55:427
46699156,9cyclictest0-21swapper/711:55:427
46699156,5cyclictest0-21swapper/707:35:007
46699155,9cyclictest668-21systemd-logind11:15:387
46699155,9cyclictest10309-21bash08:59:147
46699155,10cyclictest0-21swapper/712:04:287
46699154,5cyclictest0-21swapper/707:07:567
46699153,8cyclictest861-21gdbus09:24:407
46699153,12cyclictest0-21swapper/708:51:447
46699153,11cyclictest30322-21systemd-cgroups09:16:327
46699153,11cyclictest23038-21grepconf.sh09:03:427
46699150,9cyclictest0-21swapper/707:26:347
46699150,9cyclictest0-21swapper/707:12:327
46699150,14cyclictest28747-21bash09:28:427
46699150,11cyclictest1791-21sshd09:59:587
46699150,11cyclictest0-21swapper/707:51:527
46099150,14cyclictest17466-21sshd11:50:196
45299150,6cyclictest7358-21grep09:29:465
45299150,6cyclictest0-21swapper/512:10:585
45299150,2cyclictest11950-21bash10:33:305
425991513,1cyclictest33-21ksoftirqd/211:47:052
416991514,1cyclictest25-21ksoftirqd/110:04:451
416991513,1cyclictest25-21ksoftirqd/109:16:431
416991510,3cyclictest25-21ksoftirqd/112:15:401
408991514,1cyclictest9-21ksoftirqd/011:30:200
408991513,1cyclictest9-21ksoftirqd/012:19:500
408991513,1cyclictest9-21ksoftirqd/012:05:070
408991513,1cyclictest9-21ksoftirqd/012:01:010
408991513,1cyclictest9-21ksoftirqd/011:04:110
408991513,1cyclictest9-21ksoftirqd/009:33:400
252252150,0sleep60-21swapper/611:55:556
252252150,0sleep60-21swapper/611:55:556
189232150,0sleep10-21swapper/109:42:551
46699146,8cyclictest28105-21nfsd10:32:007
46699143,4cyclictest17455-21bash09:30:307
46699143,11cyclictest0-21swapper/708:28:427
46699141,13cyclictest0-21swapper/707:40:367
46699141,12cyclictest0-21swapper/712:07:587
46699140,8cyclictest711-21dbus-daemon11:45:077
46699140,5cyclictest0-21swapper/709:52:347
46099141,3cyclictest19054-21sshd09:49:136
46099140,0cyclictest0-21swapper/609:09:216
45299140,5cyclictest0-21swapper/510:42:205
416991413,1cyclictest25-21ksoftirqd/111:27:041
416991411,1cyclictest25-21ksoftirqd/112:03:531
408991413,1cyclictest9-21ksoftirqd/011:25:230
408991412,1cyclictest9-21ksoftirqd/012:15:030
408991412,1cyclictest9-21ksoftirqd/011:20:000
408991412,1cyclictest9-21ksoftirqd/011:19:460
408991412,1cyclictest9-21ksoftirqd/010:56:190
408991412,1cyclictest9-21ksoftirqd/010:45:290
408991412,1cyclictest9-21ksoftirqd/010:20:090
408991412,1cyclictest9-21ksoftirqd/010:00:590
408991412,1cyclictest9-21ksoftirqd/009:41:410
408991412,1cyclictest9-21ksoftirqd/009:17:480
408991412,1cyclictest9-21ksoftirqd/009:10:290
46699135,2cyclictest938-21sshd10:41:507
46699135,0cyclictest0-21swapper/708:45:267
46699134,2cyclictest859-21polkitd10:22:087
46699133,9cyclictest1-21systemd10:15:447
46699133,3cyclictest11724-21sshd10:09:567
46699131,11cyclictest11640-21sshd09:57:547
46699130,4cyclictest22270-21sshd09:40:067
46699130,12cyclictest22771-21bash11:13:327
44499134,8cyclictest19124-21sshd09:27:364
44499133,6cyclictest18863-21sshd10:37:164
44499130,2cyclictest31970-21cpuspeed_turbos09:35:004
44499130,10cyclictest14237-21sshd10:49:054
416991312,1cyclictest25-21ksoftirqd/110:47:501
416991312,1cyclictest25-21ksoftirqd/110:17:231
416991312,1cyclictest25-21ksoftirqd/109:45:321
416991312,1cyclictest25-21ksoftirqd/109:13:071
416991311,1cyclictest8345-21latency_hist09:29:501
416991311,1cyclictest25-21ksoftirqd/111:59:491
416991311,1cyclictest25-21ksoftirqd/111:59:491
416991311,1cyclictest25-21ksoftirqd/111:54:221
416991311,1cyclictest25-21ksoftirqd/110:27:221
416991311,1cyclictest25-21ksoftirqd/109:30:221
416991311,1cyclictest25-21ksoftirqd/109:01:081
408991312,1cyclictest9-21ksoftirqd/010:42:130
408991312,1cyclictest9-21ksoftirqd/009:08:280
408991312,1cyclictest9-21ksoftirqd/009:00:060
408991311,1cyclictest9-21ksoftirqd/011:55:110
408991311,1cyclictest9-21ksoftirqd/011:55:110
408991311,1cyclictest9-21ksoftirqd/011:52:000
408991311,1cyclictest9-21ksoftirqd/011:45:230
408991311,1cyclictest9-21ksoftirqd/010:30:140
408991311,1cyclictest9-21ksoftirqd/010:15:030
408991311,1cyclictest9-21ksoftirqd/010:11:200
408991311,1cyclictest9-21ksoftirqd/009:20:000
40899131,1cyclictest131rcu_sched09:51:040
40899130,1cyclictest131rcu_sched11:44:280
27022130,0sleep40-21swapper/410:09:094
212932130,0sleep10-21swapper/109:55:371
124152130,0sleep40-21swapper/411:00:354
46699124,2cyclictest2044-21bash09:35:097
46699124,0cyclictest0-21swapper/708:14:567
46699120,12cyclictest0-21swapper/708:29:587
46699120,12cyclictest0-21swapper/707:46:107
46099122,7cyclictest7282-21sshd11:00:106
46099120,2cyclictest8801-21sshd10:12:396
46099120,2cyclictest8319-21grep10:00:236
46099120,2cyclictest711-21dbus-daemon10:19:566
46099120,2cyclictest32720-21sshd10:29:366
46099120,2cyclictest3248-21sshd10:42:046
46099120,2cyclictest32342-21bash10:35:136
46099120,2cyclictest31263-21grepconf.sh09:10:526
46099120,2cyclictest31089-21bash09:28:566
46099120,2cyclictest30024-21grepconf.sh10:05:286
46099120,2cyclictest29203-21grep10:59:226
46099120,2cyclictest20874-21bash11:10:126
46099120,2cyclictest15773-21grepconf.sh11:37:486
46099120,2cyclictest14882-21id11:24:526
46099120,11cyclictest30947-21sshd11:41:246
46099120,1cyclictest0-21swapper/609:16:246
44499124,7cyclictest5665-21id09:20:074
44499124,7cyclictest3534-21bash11:44:244
44499124,7cyclictest32263-21sshd10:02:444
44499124,7cyclictest12191-21sshd11:21:364
44499124,7cyclictest10276-21rm09:05:234
44499123,8cyclictest711-21dbus-daemon11:47:334
44499120,3cyclictest19234-21sshd10:25:124
44499120,2cyclictest7531-21grepconf.sh09:02:084
44499120,2cyclictest711-21dbus-daemon12:11:124
44499120,2cyclictest31881-21sshd11:54:044
44499120,2cyclictest16496-21grepconf.sh10:10:124
44499120,2cyclictest16016-21systemd-cgroups09:55:084
44499120,2cyclictest15684-21grepconf.sh10:22:004
44499120,2cyclictest12495-21sshd12:22:244
43699129,2cyclictest23317-21grep12:18:173
43699122,4cyclictest28829-21sshd09:04:163
43699120,12cyclictest0-21swapper/309:27:213
43699120,11cyclictest668-21systemd-logind09:32:443
425991211,1cyclictest33-21ksoftirqd/209:27:582
425991211,1cyclictest33-21ksoftirqd/209:22:162
425991210,1cyclictest33-21ksoftirqd/212:10:482
42599121,10cyclictest859-21polkitd12:20:352
42599120,2cyclictest14210-21bash10:21:522
416991211,1cyclictest25-21ksoftirqd/112:23:351
416991211,1cyclictest25-21ksoftirqd/111:42:571
416991211,1cyclictest25-21ksoftirqd/110:20:031
416991211,1cyclictest25-21ksoftirqd/110:07:031
416991210,1cyclictest25-21ksoftirqd/111:34:331
416991210,1cyclictest25-21ksoftirqd/111:11:431
416991210,1cyclictest25-21ksoftirqd/110:52:541
416991210,1cyclictest25-21ksoftirqd/107:59:491
40899129,3cyclictest9-21ksoftirqd/008:59:040
408991211,1cyclictest9-21ksoftirqd/010:39:090
408991211,1cyclictest9-21ksoftirqd/009:45:350
408991211,1cyclictest9-21ksoftirqd/009:25:090
408991210,1cyclictest9-21ksoftirqd/011:05:080
408991210,1cyclictest9-21ksoftirqd/010:52:000
408991210,1cyclictest9-21ksoftirqd/010:27:250
40899120,12cyclictest0-21swapper/008:02:550
40899120,1cyclictest131rcu_sched11:11:120
309192120,2sleep444499cyclictest12:01:224
160502120,0sleep40-21swapper/409:18:164
120412120,0sleep20-21swapper/208:55:312
65712110,0sleep10-21swapper/110:39:101
46699116,5cyclictest0-21swapper/708:01:187
46099115,5cyclictest10553-21bash09:30:026
46099114,6cyclictest7361-21grep11:15:046
46099113,7cyclictest5015-21sshd08:54:526
46099113,7cyclictest4206-21grepconf.sh10:15:056
46099111,9cyclictest668-21systemd-logind09:55:076
46099110,1cyclictest26977-21latency10:50:086
46099110,1cyclictest191692chrt08:20:126
45299118,0cyclictest0-21swapper/507:00:385
45299110,10cyclictest27585-21missed_timers07:40:065
44499119,1cyclictest0-21swapper/408:59:274
44499114,6cyclictest24279-21grepconf.sh10:49:584
44499114,6cyclictest10683-21bash11:34:584
44499113,7cyclictest16855-21grepconf.sh11:30:174
44499113,7cyclictest1268-21grep10:41:524
44499113,2cyclictest668-21systemd-logind09:50:024
44499112,2cyclictest711-21dbus-daemon11:57:024
44499112,2cyclictest711-21dbus-daemon11:57:024
44499110,2cyclictest24401-21gltestperf07:35:044
44499110,10cyclictest15399-21sshd09:30:194
44499110,1cyclictest21975-21iostat09:10:044
44499110,1cyclictest0-21swapper/411:06:244
43699119,1cyclictest711-21dbus-daemon09:41:343
43699119,1cyclictest1-21systemd10:35:483
43699112,4cyclictest0-21swapper/311:37:543
43699111,4cyclictest30745-21sshd10:02:363
43699110,5cyclictest2807-21grep11:51:453
43699110,5cyclictest26649-21sshd09:19:113
43699110,11cyclictest0-21swapper/309:12:523
42599119,1cyclictest33-21ksoftirqd/211:20:182
42599119,1cyclictest33-21ksoftirqd/211:15:092
42599119,1cyclictest33-21ksoftirqd/209:14:012
42599119,1cyclictest33-21ksoftirqd/207:04:502
41699119,1cyclictest711-21dbus-daemon11:45:041
41699119,1cyclictest25-21ksoftirqd/111:08:311
41699119,1cyclictest25-21ksoftirqd/110:41:341
41699119,1cyclictest25-21ksoftirqd/109:35:491
416991110,1cyclictest25-21ksoftirqd/111:22:421
416991110,1cyclictest25-21ksoftirqd/111:03:101
416991110,1cyclictest25-21ksoftirqd/110:58:591
309212110,0sleep10-21swapper/109:07:401
218112110,0sleep40-21swapper/411:25:184
209642110,0sleep40-21swapper/411:10:124
200382110,0sleep30-21swapper/310:19:253
46099103,6cyclictest757-21snmpd07:32:216
46099103,6cyclictest26744-21sort08:35:056
46099102,7cyclictest24362-21gltestperf07:35:046
46099102,2cyclictest31339-21egrep08:45:016
46099100,9cyclictest6147-21fschecks_count08:00:036
46099100,9cyclictest23174-21cstates08:30:016
45299105,5cyclictest0-21swapper/508:17:265
45299103,6cyclictest757-21snmpd07:56:575
45299103,6cyclictest757-21snmpd07:21:085
45299103,6cyclictest25927-21expr08:35:015
45299100,1cyclictest0-21swapper/508:45:035
44499103,6cyclictest0-21swapper/407:14:564
44499103,1cyclictest757-21snmpd07:41:424
44499102,2cyclictest16547-21ntp_offset07:20:074
44499100,9cyclictest0-21swapper/407:30:064
44499100,10cyclictest0-21swapper/407:45:494
43699108,1cyclictest16173-21sshd11:42:483
43699107,1cyclictest16603-21kworker/3:411:23:063
43699103,6cyclictest8711-21sshd10:48:343
43699103,6cyclictest0-21swapper/312:11:143
43699101,4cyclictest7447-21sshd10:21:143
43699101,4cyclictest0-21swapper/312:24:233
43699101,3cyclictest0-21swapper/311:34:063
43699100,4cyclictest9070-21sshd11:15:093
43699100,4cyclictest20160-21sshd10:58:313
43699100,4cyclictest0-21swapper/310:39:493
43699100,10cyclictest0-21swapper/309:56:523
42599109,1cyclictest33-21ksoftirqd/212:15:122
42599109,1cyclictest33-21ksoftirqd/209:33:432
42599109,1cyclictest0-21swapper/209:58:232
42599108,1cyclictest33-21ksoftirqd/211:32:372
42599108,1cyclictest33-21ksoftirqd/209:45:142
42599107,1cyclictest33-21ksoftirqd/211:40:322
42599103,4cyclictest711-21dbus-daemon09:00:292
42599100,9cyclictest27596-21grep10:34:572
42599100,9cyclictest0-21swapper/210:56:332
42599100,10cyclictest15276-21kworker/2:310:48:522
41699109,1cyclictest25-21ksoftirqd/110:12:001
41699109,1cyclictest25-21ksoftirqd/108:56:371
41699109,1cyclictest25-21ksoftirqd/108:54:491
41699108,1cyclictest25-21ksoftirqd/112:10:031
41699108,1cyclictest25-21ksoftirqd/109:50:561
41699100,1cyclictest0-21swapper/109:22:521
4609993,5cyclictest757-21snmpd08:27:556
4609993,5cyclictest2087-21cstates08:50:056
4609992,6cyclictest14461-21cat08:15:006
4609992,2cyclictest4363-21gltestperf07:00:026
4609992,1cyclictest6597-21cpuspeed_turbos07:05:036
4609992,1cyclictest28661-21cpuspeed_turbos08:40:006
4609992,1cyclictest17774-21sort07:24:496
4609991,1cyclictest0-21swapper/607:15:116
4609990,8cyclictest0-21swapper/607:50:076
4609990,8cyclictest0-21swapper/607:45:026
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional