You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-25 - 07:50
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot2.osadl.org (updated Sat Jan 25, 2025 00:43:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
176686325519,31sleep70-21swapper/719:06:337
176687025039,7sleep40-21swapper/419:06:384
1767327994239,2cyclictest2147229-21sed22:39:037
176732799400,38cyclictest2095867-21latency_hist22:09:037
176681323726,8sleep50-21swapper/519:05:515
1767327993128,2cyclictest1901337-21perf20:14:037
1767317993129,2cyclictest1983294-21kworker/u16:4+events_unbound21:29:144
176680522717,7sleep60-21swapper/619:05:446
1767322992624,2cyclictest1856705-21kworker/u16:3+flush-8:019:54:206
1767322992523,2cyclictest1890741-21kworker/u16:0+events_unbound20:49:136
1767322992523,2cyclictest1890741-21kworker/u16:0+events_unbound20:49:136
1767317992522,2cyclictest1921029-21kworker/u16:2+events_unbound20:44:044
176732799240,22cyclictest2178230-21latency_hist22:59:037
1767322992422,2cyclictest2056149-21kworker/u16:1+events_unbound21:39:306
1767317992422,2cyclictest2096287-21kworker/u16:3+flush-8:023:39:324
1767317992422,2cyclictest2096287-21kworker/u16:3+events_unbound22:59:164
1767317992422,2cyclictest1681941-21kworker/u16:0+events_unbound19:29:054
1767317992421,2cyclictest1856705-21kworker/u16:3+flush-8:020:19:314
1767317992321,2cyclictest2200959-21kworker/u16:5+flush-8:000:34:234
1767327992220,1cyclictest2323976-21munin-run00:34:027
1767322992220,2cyclictest2200959-21kworker/u16:5+flush-8:000:33:576
1767317992119,2cyclictest2056149-21kworker/u16:1+flush-8:021:59:324
19918452200,1chrt1991844-21sed20:59:307
19342732200,1chrt1934272-21grep20:29:257
1767322992018,2cyclictest2200959-21kworker/u16:5+events_unbound23:59:006
1767322992018,2cyclictest2200959-21kworker/u16:5+events_unbound23:59:006
1767322992018,2cyclictest2064499-21kworker/u16:2+events_unbound23:01:316
1767322992018,2cyclictest1890741-21kworker/u16:0+events_unbound20:16:466
1767322992018,2cyclictest1757905-21kworker/u16:4+events_unbound19:27:166
1767320992017,2cyclictest1983294-21kworker/u16:4+events_unbound21:14:175
1767317992016,3cyclictest1757905-21kworker/u16:4+flush-8:020:04:244
1767322991917,2cyclictest2282479-21kworker/u16:2+events_unbound00:15:086
1767322991917,2cyclictest2200959-21kworker/u16:5+events_unbound00:06:126
1767322991917,2cyclictest2096287-21kworker/u16:3+events_unbound23:43:266
1767322991917,2cyclictest2064499-21kworker/u16:2+flush-8:022:15:366
1767322991917,2cyclictest2002637-21kworker/u16:0+events_unbound21:23:116
1767322991917,2cyclictest1983294-21kworker/u16:4+events_unbound21:16:386
1767322991917,2cyclictest1737192-21kworker/u16:1+events_unbound19:13:456
1767322991916,2cyclictest1983294-21kworker/u16:4+events_unbound21:09:056
1767317991917,2cyclictest1983294-21kworker/u16:4+flush-8:021:04:334
1767317991917,2cyclictest1757905-21kworker/u16:4+flush-8:019:49:294
1767317991917,2cyclictest1493113-21kworker/u16:2+events_unbound19:54:064
1767327991817,0cyclictest0-21swapper/723:43:317
1767322991816,2cyclictest2282479-21kworker/u16:2+events_unbound00:12:436
1767322991816,2cyclictest2200959-21kworker/u16:5+events_unbound23:38:516
1767322991816,2cyclictest2200959-21kworker/u16:5+events_unbound00:23:286
1767322991816,2cyclictest2199075-21kworker/u16:4+events_unbound23:53:146
1767322991816,2cyclictest2064499-21kworker/u16:2+flush-8:022:34:286
1767322991816,2cyclictest1890741-21kworker/u16:0+flush-8:020:09:476
1767322991816,2cyclictest1856705-21kworker/u16:3+events_unbound19:50:376
1767322991816,2cyclictest1757905-21kworker/u16:4+events_unbound20:06:196
1767322991815,2cyclictest2033402-21kworker/u16:3+events_unbound21:54:066
1767317991815,2cyclictest2064499-21kworker/u16:2+events_unbound22:24:054
1767322991715,2cyclictest2200959-21kworker/u16:5+events_unbound23:16:186
1767322991715,2cyclictest2200959-21kworker/u16:5+events_unbound00:37:406
1767322991715,2cyclictest2199075-21kworker/u16:4+events_unbound00:01:376
1767322991715,2cyclictest2096287-21kworker/u16:3+events_unbound22:12:386
1767322991715,2cyclictest2094620-21kworker/u16:0+events_unbound22:21:296
1767322991715,2cyclictest2056149-21kworker/u16:1+events_unbound22:46:486
1767322991715,2cyclictest1961222-21kworker/u16:1+events_unbound21:05:556
1767322991715,2cyclictest1961222-21kworker/u16:1+events_unbound21:01:276
1767322991715,2cyclictest1931495-21kworker/u16:3+events_unbound20:39:056
1767322991715,2cyclictest1931495-21kworker/u16:3+events_unbound20:35:426
1767322991714,2cyclictest2312502-21kworker/u16:3+events_unbound00:24:446
1767322991714,2cyclictest2096287-21kworker/u16:3+events_unbound23:27:566
1767317991715,2cyclictest2096287-21kworker/u16:3+events_unbound22:54:174
1767317991715,2cyclictest2065521-21kworker/u16:4+flush-8:021:54:184
1767317991715,2cyclictest1972330-21kworker/u16:3+flush-8:020:59:144
1767317991715,2cyclictest1493113-21kworker/u16:2+events_unbound19:34:164
1767327991615,0cyclictest0-21swapper/720:54:347
1767327991614,1cyclictest1851260-21awk19:49:037
176732799160,14cyclictest1302-21snmpd19:17:007
176732799160,14cyclictest1302-21snmpd19:17:007
1767322991614,2cyclictest2094620-21kworker/u16:0+events_unbound22:33:486
1767322991614,2cyclictest2065521-21kworker/u16:4+events_unbound22:07:016
1767322991614,2cyclictest2064499-21kworker/u16:2+flush-8:023:08:506
1767322991614,2cyclictest2064499-21kworker/u16:2+flush-8:022:56:136
1767322991614,2cyclictest2064499-21kworker/u16:2+events_unbound22:02:246
1767322991614,2cyclictest2056149-21kworker/u16:1+events_unbound22:50:176
1767322991614,2cyclictest2056149-21kworker/u16:1+events_unbound21:52:436
1767322991614,2cyclictest2044431-21kworker/u16:2+events_unbound21:35:226
1767322991614,2cyclictest2044431-21kworker/u16:2+events_unbound21:35:226
1767322991614,2cyclictest1983294-21kworker/u16:4+flush-8:021:24:306
1767322991614,2cyclictest1961222-21kworker/u16:1+events_unbound21:29:406
1767322991614,2cyclictest1961222-21kworker/u16:1+events_unbound20:55:496
1767322991614,2cyclictest1800860-21kworker/u16:3+events_unbound19:40:196
1767322991614,2cyclictest1757905-21kworker/u16:4+events_unbound20:00:316
1767322991614,2cyclictest1681941-21kworker/u16:0+flush-8:019:17:056
1767322991614,2cyclictest1681941-21kworker/u16:0+flush-8:019:17:056
1767322991613,2cyclictest2064499-21kworker/u16:2+events_unbound22:25:376
176732299160,15cyclictest1815993-21grep19:29:276
1767317991614,2cyclictest2292601-21kworker/u16:1+events_unbound00:17:074
1767317991614,2cyclictest2094620-21kworker/u16:0+events_unbound22:21:354
1767317991614,2cyclictest1830191-21kworker/u16:1+events_unbound19:39:234
1767317991614,2cyclictest1681941-21kworker/u16:0+events_unbound19:14:104
1767317991614,2cyclictest1681941-21kworker/u16:0+events_unbound19:14:104
1767327991511,3cyclictest1861380-21cat19:54:037
176732799150,13cyclictest1302-21snmpd20:25:327
1767322991513,2cyclictest2200959-21kworker/u16:5+events_unbound23:31:096
1767322991513,2cyclictest2199075-21kworker/u16:4+events_unbound23:11:276
1767322991513,2cyclictest2199075-21kworker/u16:4+events_unbound23:11:276
1767322991513,2cyclictest1890741-21kworker/u16:0+flush-8:020:30:596
1767322991513,2cyclictest1890741-21kworker/u16:0+flush-8:020:29:046
1767322991512,2cyclictest2094620-21kworker/u16:0+flush-8:023:19:136
1767320991512,2cyclictest2044431-21kworker/u16:2+events_unbound21:34:175
1767320991512,2cyclictest2044431-21kworker/u16:2+events_unbound21:34:175
1767317991513,2cyclictest2200959-21kworker/u16:5+events_unbound23:29:224
1767317991513,2cyclictest2094620-21kworker/u16:0+flush-8:022:34:184
1767317991513,2cyclictest2065521-21kworker/u16:4+flush-8:021:49:174
1767317991513,2cyclictest1890741-21kworker/u16:0+events_unbound20:28:224
1767317991513,2cyclictest1800860-21kworker/u16:3+events_unbound19:24:224
1767317991513,2cyclictest1493113-21kworker/u16:2+flush-8:019:09:144
1767317991512,2cyclictest2096287-21kworker/u16:3+events_unbound22:48:434
1767317991512,2cyclictest2056149-21kworker/u16:1+events_unbound22:06:244
1767327991412,1cyclictest0-21swapper/722:49:137
1767327991411,1cyclictest1841249-21awk19:44:047
1767327991410,3cyclictest2292848-21awk00:14:037
1767327991410,2cyclictest2012637-21awk21:14:037
176732799140,12cyclictest2189373-21latency_hist23:09:037
176732799140,12cyclictest1302-21snmpd21:06:577
1767322991412,2cyclictest2096287-21kworker/u16:3+events_unbound23:47:016
1767322991412,2cyclictest2096287-21kworker/u16:3+events_unbound22:42:416
1767322991412,2cyclictest2002637-21kworker/u16:0+events_unbound21:45:536
1767322991412,2cyclictest1961222-21kworker/u16:1+events_unbound20:45:516
1767322991412,2cyclictest1921029-21kworker/u16:2+events_unbound20:23:216
1767322991412,2cyclictest1800860-21kworker/u16:3+events_unbound19:34:516
1767322991412,2cyclictest1757905-21kworker/u16:4+events_unbound19:23:506
176732099146,7cyclictest0-21swapper/500:34:175
1767320991413,1cyclictest0-21swapper/519:34:195
176732099140,1cyclictest0-21swapper/521:58:525
176732099140,14cyclictest0-21swapper/500:19:245
1767317991412,2cyclictest2312502-21kworker/u16:3+events_unbound00:23:574
1767317991412,2cyclictest2200959-21kworker/u16:5+events_unbound00:33:144
1767317991412,2cyclictest2199075-21kworker/u16:4+events_unbound00:00:134
1767317991412,2cyclictest2094620-21kworker/u16:0+flush-8:022:29:184
1767317991412,2cyclictest2094620-21kworker/u16:0+events_unbound22:40:424
1767317991412,2cyclictest1757905-21kworker/u16:4+events_unbound19:59:444
1767317991412,1cyclictest0-21swapper/420:39:264
1767317991411,2cyclictest2096287-21kworker/u16:3+events_unbound23:24:134
1767317991411,2cyclictest2064499-21kworker/u16:2+flush-8:023:09:204
1767317991411,2cyclictest2064499-21kworker/u16:2+flush-8:023:09:204
1767317991411,2cyclictest1830191-21kworker/u16:1+events_unbound19:44:064
176732799138,3cyclictest2064797-21perf21:49:037
176732799130,1cyclictest0-21swapper/719:29:317
176732799130,11cyclictest1302-21snmpd22:22:087
176732799130,11cyclictest1302-21snmpd20:53:367
176732799130,11cyclictest1302-21snmpd20:53:367
1767322991311,2cyclictest1493113-21kworker/u16:2+events_unbound19:46:016
1767320991312,1cyclictest0-21swapper/522:44:455
1767320991312,1cyclictest0-21swapper/522:10:045
176732099130,1cyclictest0-21swapper/523:35:025
176732099130,1cyclictest0-21swapper/521:50:125
176732099130,1cyclictest0-21swapper/521:08:355
176732099130,13cyclictest0-21swapper/522:24:065
176732099130,13cyclictest0-21swapper/521:11:065
1767317991311,2cyclictest2282479-21kworker/u16:2+events_unbound00:06:024
1767317991311,2cyclictest2002637-21kworker/u16:0+events_unbound21:43:424
1767317991311,2cyclictest2002637-21kworker/u16:0+events_unbound21:09:424
1767317991311,2cyclictest1890741-21kworker/u16:0+flush-8:020:34:344
1767327991211,0cyclictest0-21swapper/723:29:287
1767327991211,0cyclictest0-21swapper/723:27:457
1767327991211,0cyclictest0-21swapper/723:17:117
1767327991211,0cyclictest0-21swapper/722:12:047
1767327991211,0cyclictest0-21swapper/721:42:177
1767327991211,0cyclictest0-21swapper/720:01:517
1767327991210,1cyclictest2312830-21awk00:24:037
1767327991210,1cyclictest2302819-21awk00:19:037
176732799120,11cyclictest2219504-21latency_hist23:24:037
176732799120,11cyclictest1302-21snmpd21:51:587
176732799120,10cyclictest2157124-21systemd-run22:44:037
176732799120,10cyclictest2032615-21latency_hist21:24:037
176732799120,10cyclictest1302-21snmpd22:59:537
176732799120,10cyclictest1302-21snmpd22:44:117
176732799120,10cyclictest1302-21snmpd22:27:597
176732799120,10cyclictest1302-21snmpd22:02:167
176732799120,10cyclictest1302-21snmpd21:28:257
176732799120,10cyclictest1302-21snmpd20:22:357
176732799120,10cyclictest1302-21snmpd19:20:367
176732799120,10cyclictest1302-21snmpd00:04:167
1767320991211,1cyclictest0-21swapper/523:18:395
1767320991211,1cyclictest0-21swapper/520:20:395
1767320991211,1cyclictest0-21swapper/519:19:395
1767320991211,1cyclictest0-21swapper/519:09:525
1767320991211,1cyclictest0-21swapper/500:04:415
176732099120,1cyclictest0-21swapper/522:53:165
176732099120,1cyclictest0-21swapper/522:08:485
176732099120,12cyclictest0-21swapper/523:12:305
176732099120,12cyclictest0-21swapper/523:12:305
176732099120,12cyclictest0-21swapper/521:41:325
1767317991210,2cyclictest2282479-21kworker/u16:2+flush-8:000:09:234
1767317991210,2cyclictest2096287-21kworker/u16:3+events_unbound23:49:084
1767317991210,2cyclictest2002637-21kworker/u16:0+flush-8:021:19:274
1767317991210,2cyclictest2002637-21kworker/u16:0+events_unbound21:44:334
1767317991210,2cyclictest1983294-21kworker/u16:4+flush-8:021:14:244
176732799116,4cyclictest777618-21gmain23:34:267
1767327991110,0cyclictest0-21swapper/720:07:037
1767327991110,0cyclictest0-21swapper/719:54:467
1767327991110,0cyclictest0-21swapper/700:27:307
176732799110,9cyclictest1302-21snmpd20:14:147
176732799110,9cyclictest1302-21snmpd19:36:017
176732799110,10cyclictest1302-21snmpd23:58:257
176732799110,10cyclictest1302-21snmpd23:58:257
176732799110,10cyclictest1302-21snmpd23:50:157
176732799110,10cyclictest1302-21snmpd23:13:487
176732799110,10cyclictest1302-21snmpd23:13:487
176732799110,10cyclictest1302-21snmpd22:16:537
176732799110,10cyclictest1302-21snmpd21:57:437
176732799110,10cyclictest1302-21snmpd20:34:597
176732099119,1cyclictest2301863-21ntp_states00:14:275
1767320991110,1cyclictest0-21swapper/521:46:535
1767320991110,1cyclictest0-21swapper/521:24:595
1767320991110,1cyclictest0-21swapper/520:54:245
1767320991110,1cyclictest0-21swapper/520:35:575
1767320991110,1cyclictest0-21swapper/520:12:475
1767320991110,1cyclictest0-21swapper/520:04:545
1767320991110,1cyclictest0-21swapper/519:24:325
176732099110,1cyclictest0-21swapper/522:00:085
176732099110,1cyclictest0-21swapper/521:21:125
176732099110,11cyclictest0-21swapper/520:50:375
176732099110,11cyclictest0-21swapper/520:50:375
176732099110,11cyclictest0-21swapper/520:32:385
176732099110,11cyclictest0-21swapper/519:51:495
176732099110,11cyclictest0-21swapper/519:32:165
176732099110,11cyclictest0-21swapper/519:14:465
176732099110,11cyclictest0-21swapper/519:14:465
176731799119,2cyclictest2200959-21kworker/u16:5+flush-8:023:54:374
176731799119,2cyclictest2200959-21kworker/u16:5+flush-8:023:54:374
176731799119,2cyclictest2200959-21kworker/u16:5+events_unbound23:19:494
176731799119,2cyclictest2094620-21kworker/u16:0+flush-8:023:34:554
176731799119,2cyclictest1890741-21kworker/u16:0+events_unbound20:16:364
176731799119,2cyclictest1870301-21kworker/u16:5+events_unbound20:29:154
176731799119,2cyclictest1757905-21kworker/u16:4+events_unbound19:19:134
176731799119,1cyclictest1971001-21kthreadcore20:49:204
176731799119,1cyclictest1971001-21kthreadcore20:49:204
1767317991110,1cyclictest0-21swapper/422:16:294
1767317991110,1cyclictest0-21swapper/421:27:094
176731799110,1cyclictest0-21swapper/423:07:064
176731799110,11cyclictest0-21swapper/423:48:334
176732799109,0cyclictest0-21swapper/720:44:287
176732799107,2cyclictest1811226-21munin-run19:29:037
176732799107,2cyclictest1777299-21munin-run19:14:037
176732799100,9cyclictest1302-21snmpd21:37:047
176732799100,9cyclictest1302-21snmpd21:37:047
176732799100,9cyclictest1302-21snmpd21:14:577
176732799100,8cyclictest1302-21snmpd22:30:207
176732099109,1cyclictest0-21swapper/523:52:045
176732099109,1cyclictest0-21swapper/523:08:245
176732099109,1cyclictest0-21swapper/522:36:145
176732099109,1cyclictest0-21swapper/521:02:455
176732099109,1cyclictest0-21swapper/500:09:345
176732099107,2cyclictest0-21swapper/523:29:175
176731799109,1cyclictest0-21swapper/421:38:394
176731799109,1cyclictest0-21swapper/421:38:394
176731799108,2cyclictest2312502-21kworker/u16:3+events_unbound00:24:054
176731799108,2cyclictest2200959-21kworker/u16:5+events_unbound23:15:414
176731799108,2cyclictest2056149-21kworker/u16:1+flush-8:022:49:144
176731799108,2cyclictest2056149-21kworker/u16:1+events_unbound22:10:154
176731799108,2cyclictest1961222-21kworker/u16:1+events_unbound20:54:534
176731799108,2cyclictest1870301-21kworker/u16:5+flush-8:020:11:324
17673279999,0cyclictest0-21swapper/723:47:467
17673279994,4cyclictest1767654-21cyclictest21:29:247
17673279991,7cyclictest777647-21gmain23:59:307
17673279990,8cyclictest1302-21snmpd00:37:477
17673209998,1cyclictest0-21swapper/519:56:425
17673209998,1cyclictest0-21swapper/500:33:325
17673209994,4cyclictest2169774-21ip_192.168.114.22:54:195
17673209994,4cyclictest2136322-21cat22:29:265
17673209990,4cyclictest0-21swapper/522:59:165
17673279982,5cyclictest1302-21snmpd20:39:267
17673209984,3cyclictest2212032-21grep23:19:205
17673209984,3cyclictest1952940-21if_err_p4p120:39:185
17673209984,3cyclictest1834861-21cat19:39:275
17673209983,4cyclictest2156617-21unixbench-2d22:39:305
17673209983,4cyclictest2117449-21interrupts22:19:225
17673209983,4cyclictest2115120-21sensors22:14:305
17673209983,4cyclictest1872116-21cat19:59:205
17673209981,4cyclictest2251635-21taskset23:48:265
17673209980,7cyclictest0-21swapper/520:24:205
17673209980,1cyclictest0-21swapper/500:25:475
17673209975,1cyclictest0-21swapper/523:39:315
17673209975,1cyclictest0-21swapper/520:14:175
17673209973,3cyclictest2264450-21cut23:54:195
17673209973,3cyclictest2264450-21cut23:54:195
17673209973,3cyclictest2039429-21kthreadcore21:29:195
17673209972,4cyclictest2276775-21kthreadcore23:59:245
17673209965,1cyclictest2096287-21kworker/u16:3+flush-8:023:24:285
17673209965,1cyclictest0-21swapper/519:46:085
17673209964,2cyclictest0-21swapper/520:45:185
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional