You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 19:54
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot2.osadl.org (updated Thu Sep 18, 2025 12:43:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
51875225645,7sleep60-21swapper/615:26:276
51881925544,8sleep50-21swapper/515:27:225
51876125139,8sleep40-21swapper/415:26:334
51859924511,29sleep70-21swapper/715:24:197
519126994341,2cyclictest561181-21kworker/u16:4+events_unbound16:07:454
519126994240,2cyclictest621045-21kworker/u16:0+events_unbound17:13:174
519131994139,2cyclictest470432-21kworker/u16:1+flush-8:015:59:456
519131994038,2cyclictest621193-21kworker/u16:1+flush-8:016:26:466
519131994038,2cyclictest621193-21kworker/u16:1+flush-8:016:26:466
519128993937,2cyclictest729659-21kworker/u16:1+events_unbound17:48:265
519131993836,2cyclictest523379-21kworker/u16:3+events_unbound15:31:456
519131993836,2cyclictest470432-21kworker/u16:1+events_unbound16:03:456
519126993836,2cyclictest1053373-21kworker/u16:0+flush-8:020:08:164
519126993735,2cyclictest621045-21kworker/u16:0+events_unbound17:06:564
519126993735,2cyclictest581837-21kworker/u16:2+events_unbound16:22:444
519126993735,2cyclictest1141226-21kworker/u16:5+events_unbound20:47:264
519126993634,2cyclictest523379-21kworker/u16:3+flush-8:015:33:024
519131993432,2cyclictest621193-21kworker/u16:1+events_unbound16:20:026
519131993432,2cyclictest1102408-21kworker/u16:0+events_unbound20:37:446
519126993432,2cyclictest945010-21kworker/u16:3+events_unbound19:23:204
519131993331,2cyclictest945010-21kworker/u16:3+flush-8:019:23:276
519131993331,2cyclictest797964-21kworker/u16:5+events_unbound18:18:296
519131993331,2cyclictest777393-21kworker/u16:3+flush-8:017:45:286
519131993331,2cyclictest1122139-21kworker/u16:4+events_unbound20:55:336
519131993331,2cyclictest1053373-21kworker/u16:0+events_unbound20:08:006
519126993331,2cyclictest1141226-21kworker/u16:5+events_unbound20:50:014
519131993230,2cyclictest680517-21kworker/u16:2+flush-8:017:15:016
519131993230,2cyclictest680517-21kworker/u16:2+flush-8:016:51:016
519131993230,2cyclictest1141226-21kworker/u16:5+events_unbound20:45:006
519126993230,2cyclictest797964-21kworker/u16:5+events_unbound18:54:274
519126993230,2cyclictest1132273-21kworker/u16:2+events_unbound20:39:264
519131993129,2cyclictest906062-21kworker/u16:1+flush-8:019:39:136
519131993129,2cyclictest797964-21kworker/u16:5+flush-8:018:30:276
519131993129,2cyclictest621045-21kworker/u16:0+flush-8:017:10:556
519131993129,2cyclictest222843-21kworker/u16:5+events_unbound15:37:456
519126993129,2cyclictest846893-21kworker/u16:0+events_unbound18:37:094
519126993129,2cyclictest1063790-21kworker/u16:1+events_unbound20:31:014
519126993128,2cyclictest846893-21kworker/u16:0+events_unbound19:43:014
519126993128,2cyclictest777393-21kworker/u16:3+flush-8:017:37:534
519126993128,2cyclictest621193-21kworker/u16:1+events_unbound16:57:554
519131993028,2cyclictest906062-21kworker/u16:1+flush-8:018:59:266
519131993028,2cyclictest818093-21kworker/u16:3+events_unbound18:05:596
519131993028,2cyclictest797965-21kworker/u16:6+flush-8:018:08:236
519131993028,2cyclictest797964-21kworker/u16:5+flush-8:018:50:336
519131993028,2cyclictest797964-21kworker/u16:5+flush-8:018:50:336
519131993028,2cyclictest797952-21kworker/u16:2+events_unbound17:48:066
519131993028,2cyclictest621193-21kworker/u16:1+events_unbound16:53:546
519131993028,2cyclictest1083048-21kworker/u16:2+events_unbound20:13:306
519128993028,2cyclictest1043408-21kworker/u16:4+flush-8:020:18:205
519126993028,2cyclictest846893-21kworker/u16:0+events_unbound19:21:274
519131992927,2cyclictest945010-21kworker/u16:3+events_unbound19:31:256
519131992927,2cyclictest581837-21kworker/u16:2+flush-8:016:17:406
519131992927,2cyclictest572025-21kworker/u16:0+events_unbound15:55:466
519131992927,2cyclictest523379-21kworker/u16:3+events_unbound16:11:106
519131992927,2cyclictest1132273-21kworker/u16:2+events_unbound20:51:276
519131992927,2cyclictest1063790-21kworker/u16:1+events_unbound20:31:056
519131992927,2cyclictest1063790-21kworker/u16:1+events_unbound20:04:056
519128992927,2cyclictest621045-21kworker/u16:0+flush-8:016:28:055
519128992927,2cyclictest621045-21kworker/u16:0+flush-8:016:28:055
519128992926,3cyclictest985066-21kworker/u16:2+flush-8:019:34:365
519128992926,2cyclictest945010-21kworker/u16:3+events_unbound20:08:045
519126992927,2cyclictest488584-21kworker/u16:2+events_unbound15:40:024
519126992926,2cyclictest906062-21kworker/u16:1+events_unbound19:53:014
519126992926,2cyclictest1102408-21kworker/u16:0+flush-8:020:23:014
519131992826,2cyclictest945010-21kworker/u16:3+flush-8:019:08:266
519131992826,2cyclictest945010-21kworker/u16:3+events_unbound19:56:266
519131992826,2cyclictest846936-21kworker/u16:1+events_unbound18:15:086
519131992826,2cyclictest640600-21kworker/u16:3+events_unbound16:33:296
519131992826,2cyclictest621045-21kworker/u16:0+events_unbound17:29:346
519126992826,2cyclictest846893-21kworker/u16:0+events_unbound19:34:444
519126992826,2cyclictest846893-21kworker/u16:0+events_unbound18:40:274
519126992826,2cyclictest621045-21kworker/u16:0+events_unbound17:28:374
519126992826,2cyclictest552897-21kworker/u16:0+events_unbound15:45:014
519126992826,2cyclictest1032777-21kworker/u16:2+events_unbound20:00:014
519131992725,2cyclictest945010-21kworker/u16:3+events_unbound19:17:176
519131992725,2cyclictest906062-21kworker/u16:1+flush-8:019:33:256
519131992725,2cyclictest846893-21kworker/u16:0+flush-8:018:39:296
519131992725,2cyclictest797964-21kworker/u16:5+flush-8:018:34:536
519131992725,2cyclictest729659-21kworker/u16:1+events_unbound17:26:056
519131992725,2cyclictest621045-21kworker/u16:0+flush-8:017:37:066
519131992725,2cyclictest222843-21kworker/u16:5+events_unbound15:41:476
519131992725,2cyclictest1102408-21kworker/u16:0+events_unbound20:27:306
519128992725,2cyclictest797964-21kworker/u16:5+events_unbound18:38:205
519128992725,2cyclictest222843-21kworker/u16:5+events_unbound15:32:565
519126992725,2cyclictest945010-21kworker/u16:3+flush-8:019:41:264
519126992725,2cyclictest906062-21kworker/u16:1+flush-8:019:52:014
519126992725,2cyclictest906062-21kworker/u16:1+flush-8:019:05:054
519126992725,2cyclictest856873-21kworker/u16:2+flush-8:018:17:454
519126992725,2cyclictest808837-21kworker/u16:0+events_unbound17:57:374
519133992618,7cyclictest589235-21systemd-journal20:21:267
519131992624,2cyclictest846893-21kworker/u16:0+events_unbound19:21:116
519131992624,2cyclictest797964-21kworker/u16:5+events_unbound18:57:486
519131992624,2cyclictest797964-21kworker/u16:5+events_unbound18:45:246
519131992624,2cyclictest797964-21kworker/u16:5+events_unbound18:45:246
519131992624,2cyclictest640600-21kworker/u16:3+events_unbound16:30:256
519131992624,2cyclictest640600-21kworker/u16:3+events_unbound16:30:256
519131992624,2cyclictest621193-21kworker/u16:1+events_unbound16:39:386
519131992624,2cyclictest1063790-21kworker/u16:1+events_unbound20:39:066
519131992624,2cyclictest1032777-21kworker/u16:2+flush-8:019:50:096
519131992624,2cyclictest1032777-21kworker/u16:2+events_unbound19:46:406
519131992623,2cyclictest729659-21kworker/u16:1+events_unbound17:53:016
519128992623,2cyclictest797964-21kworker/u16:5+events_unbound18:18:085
519126992624,2cyclictest797964-21kworker/u16:5+flush-8:018:49:094
519126992624,2cyclictest797964-21kworker/u16:5+flush-8:018:49:094
519126992624,2cyclictest621193-21kworker/u16:1+flush-8:016:38:074
519131992523,2cyclictest621045-21kworker/u16:0+events_unbound17:42:056
519131992523,2cyclictest621045-21kworker/u16:0+events_unbound17:04:256
519131992523,2cyclictest1102408-21kworker/u16:0+events_unbound20:19:536
519128992523,2cyclictest572025-21kworker/u16:0+flush-8:015:53:185
519126992523,2cyclictest945010-21kworker/u16:3+events_unbound20:05:294
519126992523,2cyclictest906062-21kworker/u16:1+events_unbound19:31:174
519126992523,2cyclictest581837-21kworker/u16:2+events_unbound16:10:494
519126992523,2cyclictest523379-21kworker/u16:3+events_unbound15:49:254
519131992422,2cyclictest680517-21kworker/u16:2+events_unbound16:59:266
519131992422,2cyclictest523379-21kworker/u16:3+events_unbound15:49:586
519126992422,2cyclictest945010-21kworker/u16:3+events_unbound19:11:274
519126992422,2cyclictest846893-21kworker/u16:0+flush-8:018:31:014
519126992422,2cyclictest797964-21kworker/u16:5+flush-8:018:21:414
519126992422,2cyclictest523379-21kworker/u16:3+flush-8:015:28:464
519126992422,2cyclictest470432-21kworker/u16:1+flush-8:015:57:014
519133992323,0cyclictest0-21swapper/716:52:527
519131992321,2cyclictest797964-21kworker/u16:5+flush-8:018:26:176
519131992321,2cyclictest470432-21kworker/u16:1+flush-8:015:44:026
519128992321,2cyclictest985066-21kworker/u16:2+events_unbound19:23:055
519126992321,2cyclictest797965-21kworker/u16:6+flush-8:018:08:214
519126992321,2cyclictest523379-21kworker/u16:3+flush-8:016:13:104
519126992321,2cyclictest1063790-21kworker/u16:1+events_unbound20:15:294
519126992320,2cyclictest640600-21kworker/u16:3+flush-8:016:32:574
519133992216,5cyclictest589235-21systemd-journal18:07:517
51913399220,21cyclictest589235-21systemd-journal20:12:507
51913399220,20cyclictest768556-21latency_hist17:32:527
519131992220,2cyclictest915846-21kworker/u16:2+events_unbound19:07:216
519131992220,2cyclictest808837-21kworker/u16:0+events_unbound18:01:516
519131992220,2cyclictest1032777-21kworker/u16:2+events_unbound20:01:136
519128992220,2cyclictest640600-21kworker/u16:3+flush-8:016:33:105
519126992220,2cyclictest945010-21kworker/u16:3+events_unbound20:19:144
519126992220,2cyclictest719184-21kworker/u16:4+events_unbound17:27:374
519133992118,2cyclictest886509-21turbostat18:32:517
519128992119,2cyclictest561181-21kworker/u16:4+events_unbound16:13:175
519128992119,2cyclictest1063790-21kworker/u16:1+flush-8:020:06:565
519128992118,2cyclictest640600-21kworker/u16:3+events_unbound16:58:085
519126992119,2cyclictest906062-21kworker/u16:1+flush-8:018:59:094
519126992119,2cyclictest846893-21kworker/u16:0+flush-8:018:43:524
519126992119,2cyclictest846893-21kworker/u16:0+flush-8:018:43:524
519126992119,2cyclictest729659-21kworker/u16:1+events_unbound17:51:354
519131992018,2cyclictest640600-21kworker/u16:3+events_unbound16:45:086
519128992018,2cyclictest906062-21kworker/u16:1+events_unbound19:38:155
519128992018,2cyclictest808837-21kworker/u16:0+flush-8:018:01:285
519128992018,2cyclictest808837-21kworker/u16:0+events_unbound17:53:095
519126992018,2cyclictest945010-21kworker/u16:3+events_unbound19:14:524
519126992018,2cyclictest621193-21kworker/u16:1+flush-8:016:53:174
519126992018,2cyclictest621045-21kworker/u16:0+flush-8:017:08:524
519126992018,2cyclictest621045-21kworker/u16:0+events_unbound16:45:264
519126992018,2cyclictest1063790-21kworker/u16:1+events_unbound20:56:454
519126992018,2cyclictest1063790-21kworker/u16:1+events_unbound20:34:564
51913399190,1cyclictest0-21swapper/718:13:157
51913399190,18cyclictest699826-21turbostat16:57:517
519128991917,2cyclictest856873-21kworker/u16:2+flush-8:018:28:065
519128991917,2cyclictest797964-21kworker/u16:5+flush-8:018:51:025
519128991917,2cyclictest797964-21kworker/u16:5+flush-8:018:51:025
519126991917,2cyclictest729659-21kworker/u16:1+events_unbound17:19:304
519133991815,2cyclictest1313-21snmpd17:40:487
519131991816,2cyclictest729659-21kworker/u16:1+events_unbound17:18:556
519128991816,2cyclictest915846-21kworker/u16:2+events_unbound19:13:225
519128991816,2cyclictest719184-21kworker/u16:4+events_unbound17:18:125
519128991816,2cyclictest621193-21kworker/u16:1+flush-8:016:48:205
519128991816,2cyclictest470432-21kworker/u16:1+flush-8:016:03:225
519128991816,2cyclictest1053373-21kworker/u16:0+flush-8:019:59:265
519128991814,3cyclictest797965-21kworker/u16:6+flush-8:018:08:225
519128991814,3cyclictest719184-21kworker/u16:4+flush-8:017:43:165
519133991716,0cyclictest0-21swapper/717:08:097
51913399170,15cyclictest589235-21systemd-journal17:49:257
519128991715,2cyclictest906062-21kworker/u16:1+events_unbound18:43:095
519128991715,2cyclictest906062-21kworker/u16:1+events_unbound18:43:095
519128991713,3cyclictest561181-21kworker/u16:4+flush-8:016:08:075
519128991712,4cyclictest779318-21diskstats17:38:055
519126991715,2cyclictest797964-21kworker/u16:5+events_unbound18:02:064
519126991714,2cyclictest470432-21kworker/u16:1+flush-8:015:58:074
519133991613,2cyclictest1313-21snmpd15:54:367
519133991612,3cyclictest1313-21snmpd16:24:377
519133991612,3cyclictest1313-21snmpd16:24:377
519133991610,5cyclictest589235-21systemd-journal19:07:517
519128991614,2cyclictest729659-21kworker/u16:1+flush-8:017:33:105
519128991613,2cyclictest856873-21kworker/u16:2+flush-8:018:16:565
519128991613,2cyclictest680517-21kworker/u16:2+events_unbound17:28:065
519128991613,2cyclictest621193-21kworker/u16:1+flush-8:016:23:105
519128991613,2cyclictest621193-21kworker/u16:1+flush-8:016:23:105
519128991612,3cyclictest935513-21latency_hist18:57:515
519126991614,2cyclictest797965-21kworker/u16:6+flush-8:018:03:184
519126991614,2cyclictest621193-21kworker/u16:1+flush-8:016:30:494
519126991614,2cyclictest621193-21kworker/u16:1+flush-8:016:30:494
519126991614,2cyclictest621193-21kworker/u16:1+events_unbound16:52:264
519126991614,2cyclictest621045-21kworker/u16:0+events_unbound17:35:534
519133991512,2cyclictest1313-21snmpd18:53:077
519133991512,2cyclictest1313-21snmpd16:31:497
519133991512,2cyclictest1313-21snmpd16:31:497
51913399150,13cyclictest0-21swapper/720:53:217
51913399150,13cyclictest0-21swapper/720:48:137
519128991513,2cyclictest906062-21kworker/u16:1+flush-8:019:53:175
519128991513,2cyclictest797964-21kworker/u16:5+events_unbound18:03:035
519128991513,2cyclictest488584-21kworker/u16:2+flush-8:015:27:535
519126991513,2cyclictest867097-21kworker/u16:4+events_unbound18:24:044
519126991513,2cyclictest621193-21kworker/u16:1+flush-8:016:26:234
519126991513,2cyclictest621193-21kworker/u16:1+flush-8:016:26:234
51913399147,6cyclictest27904-21gmain19:38:047
51913399147,6cyclictest27394-21gmain18:38:127
51913399147,6cyclictest0-21swapper/716:04:447
519133991413,1cyclictest0-21swapper/719:34:517
51913399140,2cyclictest1091994-21ntpq20:13:187
519128991413,1cyclictest0-21swapper/520:38:195
519128991412,2cyclictest945010-21kworker/u16:3+flush-8:020:33:175
519128991412,2cyclictest906062-21kworker/u16:1+events_unbound19:43:225
519128991412,2cyclictest846893-21kworker/u16:0+events_unbound19:28:215
519128991412,2cyclictest680517-21kworker/u16:2+events_unbound17:07:515
519128991411,2cyclictest846893-21kworker/u16:0+flush-8:018:33:195
51913399135,7cyclictest0-21swapper/716:58:357
519133991312,0cyclictest0-21swapper/719:13:447
51912899139,3cyclictest846893-21kworker/u16:0+flush-8:018:23:155
519128991311,2cyclictest846893-21kworker/u16:0+flush-8:018:58:145
519128991311,2cyclictest680517-21kworker/u16:2+events_unbound16:53:095
519128991311,2cyclictest1063790-21kworker/u16:1+flush-8:020:45:565
519126991311,2cyclictest777393-21kworker/u16:3+events_unbound17:45:574
51913399129,3cyclictest96-21ksoftirqd/720:38:097
51913399129,2cyclictest985079-21/usr/sbin/munin19:23:137
51913399129,2cyclictest1313-21snmpd19:00:357
51913399129,2cyclictest1313-21snmpd17:58:277
51913399129,2cyclictest1313-21snmpd17:47:457
51913399129,2cyclictest1313-21snmpd17:33:097
51913399129,2cyclictest1313-21snmpd17:19:117
51913399129,2cyclictest1313-21snmpd16:39:557
51913399129,2cyclictest1313-21snmpd15:41:147
51913399129,2cyclictest1313-21snmpd15:33:117
51913399128,3cyclictest589235-21systemd-journal20:44:267
51913399128,3cyclictest1313-21snmpd16:35:287
51913399127,4cyclictest582369-21/usr/sbin/munin15:58:127
51913399124,7cyclictest0-21swapper/715:29:437
51913399120,2cyclictest0-21swapper/720:23:127
51913399120,2cyclictest0-21swapper/716:08:067
51913399120,11cyclictest1313-21snmpd18:12:097
51913399120,11cyclictest0-21swapper/720:28:147
51913399120,11cyclictest0-21swapper/719:58:157
51913399120,10cyclictest1313-21snmpd15:42:537
51912899127,4cyclictest552509-21latency_hist15:42:515
519128991210,2cyclictest748210-21kworker/u16:3+flush-8:017:23:255
519128991210,2cyclictest561181-21kworker/u16:4+flush-8:016:02:205
519128991210,2cyclictest1122139-21kworker/u16:4+events_unbound20:48:055
51912899120,1cyclictest0-21swapper/520:23:305
51912899120,1cyclictest0-21swapper/517:14:295
51912899120,12cyclictest0-21swapper/516:44:325
51913399118,2cyclictest1313-21snmpd20:35:227
51913399118,2cyclictest1313-21snmpd19:28:267
51913399118,2cyclictest1313-21snmpd19:19:567
51913399118,2cyclictest1313-21snmpd18:32:527
51913399118,2cyclictest1313-21snmpd18:25:197
51913399118,2cyclictest0-21swapper/718:48:157
51913399118,2cyclictest0-21swapper/718:48:157
51913399117,3cyclictest1127-21ntpd17:53:237
51913399116,4cyclictest27763-21gvfs-afc-volume19:43:107
51913399111,8cyclictest562710-21/usr/sbin/munin15:48:127
519133991111,0cyclictest0-21swapper/719:56:377
51913399110,6cyclictest948-21NetworkManager16:46:187
51913399110,4cyclictest619594-21sshd16:13:167
51913399110,10cyclictest1313-21snmpd18:46:287
51913399110,10cyclictest1313-21snmpd18:46:287
51912899119,2cyclictest470432-21kworker/u16:1+events_unbound15:43:215
51912899117,3cyclictest1063790-21kworker/u16:1+flush-8:020:53:185
51912899113,5cyclictest729283-21awk17:12:515
519128991110,1cyclictest0-21swapper/520:13:535
519128991110,1cyclictest0-21swapper/519:06:535
51912899110,11cyclictest0-21swapper/519:49:465
51913399107,3cyclictest27763-21gvfs-afc-volume18:18:167
51913399107,2cyclictest1313-21snmpd19:51:217
51913399107,2cyclictest0-21swapper/717:13:067
51913399106,3cyclictest0-21swapper/717:03:147
51913399105,4cyclictest921-21dbus-daemon19:12:517
51913399102,7cyclictest948-21NetworkManager17:23:067
51913399100,9cyclictest0-21swapper/720:03:217
51913399100,1cyclictest0-21swapper/716:18:117
51912899106,3cyclictest523379-21kworker/u16:3+flush-8:015:48:155
5191289998,1cyclictest0-21swapper/519:18:335
5191289997,2cyclictest561181-21kworker/u16:4+flush-8:016:18:185
5191289987,1cyclictest0-21swapper/516:41:515
5191289983,4cyclictest1121319-21ntp_kernel_err20:28:135
5191289972,4cyclictest956174-21grep19:08:045
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional