You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-13 - 11:55
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8.osadl.org (updated Sun Jul 13, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
216207521020,1sleep20-21swapper/222:31:182
1874393992522,2cyclictest2236798-21munin-run23:19:591
1874393992421,2cyclictest0-21swapper/100:40:011
1874390992422,1cyclictest1988396-21idleruntime-cro20:40:010
1874390992421,2cyclictest0-21swapper/020:55:000
1874390992420,2cyclictest0-21swapper/023:09:590
1874399992320,1cyclictest0-21swapper/323:00:003
1874399992315,5cyclictest2285811-21turbostat23:50:013
1874397992320,2cyclictest0-21swapper/221:00:002
1874397992319,2cyclictest2245377-21cpuspeed_turbos23:25:152
1874397992216,5cyclictest2025687-21diskmemload22:50:222
1874399992118,1cyclictest0-21swapper/322:50:013
1874393992118,2cyclictest522-21in:imuxsock23:30:011
1874393992118,2cyclictest522-21in:imuxsock23:30:001
1874393992115,4cyclictest0-21swapper/122:40:011
1874390992118,2cyclictest2301995-21idleruntime-cro00:00:000
1874390992118,2cyclictest0-21swapper/019:40:010
1874390992118,2cyclictest0-21swapper/019:40:000
1874390992117,2cyclictest0-21swapper/020:59:590
1874390992117,2cyclictest0-21swapper/019:45:000
1874399992017,1cyclictest0-21swapper/300:15:013
1874397992017,2cyclictest2293886-21systemd-run23:55:002
1874397992017,2cyclictest1874384-21cyclictest20:20:002
1874397992015,3cyclictest0-21swapper/219:35:002
1874393992017,2cyclictest0-21swapper/122:59:591
1874393992017,2cyclictest0-21swapper/120:40:011
1874397991916,2cyclictest3564018-21systemctl00:30:002
1874397991916,2cyclictest0-21swapper/222:10:242
1874393991916,2cyclictest3004-21cron21:55:001
1874393991916,2cyclictest2144991-21sendmail_mailqu22:20:251
1874393991916,2cyclictest2107493-21gltestperf22:00:171
1874393991916,2cyclictest0-21swapper/123:30:001
1874393991915,3cyclictest2123986-21irqcore22:10:161
1874393991914,4cyclictest2055987-21unixbench_singl21:25:251
1874390991916,2cyclictest2204788-21cpuspeed_turbos23:00:150
1874390991916,2cyclictest2171892-21cron22:40:000
1874390991916,2cyclictest1957892-21irqcore20:15:190
1874390991916,2cyclictest0-21swapper/019:25:010
1874390991913,5cyclictest2310027-21cut00:05:010
20759132180,1sleep20-21swapper/221:40:202
1874399991815,1cyclictest0-21swapper/323:04:593
1874399991815,1cyclictest0-21swapper/321:00:013
1874397991816,1cyclictest0-21swapper/221:30:002
1874397991816,1cyclictest0-21swapper/220:10:002
1874397991816,1cyclictest0-21swapper/200:15:012
1874397991815,2cyclictest3564018-21systemctl21:35:012
1874397991815,2cyclictest2221068-21cpuspeed_turbos23:10:132
1874397991815,2cyclictest2212968-21fschecks_count23:05:152
1874397991815,2cyclictest2098774-21latency_hist21:55:012
1874397991814,2cyclictest3307926-21kworker/u8:2-events_unbound21:04:592
1874393991815,2cyclictest2358375-21cron00:35:011
1874393991815,2cyclictest2025687-21diskmemload00:20:241
1874393991815,2cyclictest1963977-21gltestperf20:20:161
187439099188,3cyclictest0-21swapper/022:20:010
1874390991815,2cyclictest2326062-21cron00:15:010
1874390991815,2cyclictest2230720-21kthreadcore23:15:170
1874390991815,2cyclictest2116310-21irqrtprio22:05:200
1874390991815,2cyclictest2058096-21idleruntime-cro21:30:000
1874390991814,2cyclictest0-21swapper/021:15:000
1874399991714,1cyclictest0-21swapper/321:40:003
1874399991714,1cyclictest0-21swapper/320:25:223
1874399991714,1cyclictest0-21swapper/300:07:303
1874397991714,2cyclictest2172120-21latency_hist22:40:022
1874397991714,2cyclictest0-21swapper/223:20:002
1874397991714,2cyclictest0-21swapper/219:25:002
1874397991713,2cyclictest2275022-21strings23:40:212
1874393991714,2cyclictest2277686-21cron23:45:001
1874393991714,2cyclictest2262078-21fschecks_time23:35:181
1874393991714,2cyclictest2013061-21cut21:00:001
1874393991714,2cyclictest0-21swapper/121:50:001
1874390991715,1cyclictest0-21swapper/023:30:000
1874390991714,2cyclictest2350967-21gltestperf00:30:170
1874390991714,2cyclictest2342826-21fschecks_time00:25:150
1874390991714,2cyclictest2334947-21irqcore00:20:190
1874390991714,2cyclictest2228721-21cron23:15:000
1874390991714,2cyclictest2129110-21unixbench_singl22:10:240
1874390991714,2cyclictest1974605-21latency20:25:210
1874390991714,2cyclictest1932467-21munin-plugin-st19:55:010
1874390991714,2cyclictest0-21swapper/021:40:000
1874390991713,2cyclictest0-21swapper/021:50:010
1874390991713,2cyclictest0-21swapper/021:35:010
1874390991713,2cyclictest0-21swapper/000:40:010
1874390991711,5cyclictest744-21snmpd21:50:380
1874399991613,1cyclictest0-21swapper/322:40:023
1874399991613,1cyclictest0-21swapper/321:55:153
1874399991613,1cyclictest0-21swapper/321:51:173
1874399991613,1cyclictest0-21swapper/320:40:243
1874399991613,1cyclictest0-21swapper/319:55:203
1874399991613,1cyclictest0-21swapper/319:20:013
1874399991612,2cyclictest0-21swapper/322:20:003
1874399991612,1cyclictest0-21swapper/323:28:313
187439799164,6cyclictest2180102-21awk22:44:592
1874397991614,1cyclictest2028068-21kthreadcore21:10:182
1874397991613,2cyclictest500-21avahi-daemon23:45:272
1874397991613,2cyclictest2301958-21latency_hist00:00:002
1874397991613,2cyclictest2134848-21kthreadcore22:15:222
1874397991613,2cyclictest1-21systemd22:23:152
1874397991612,3cyclictest2150857-21sed22:25:232
1874397991612,3cyclictest1994115-21unixbench_singl20:40:262
1874397991612,2cyclictest0-21swapper/219:15:002
1874397991611,4cyclictest3561480-21systemd-journal23:30:322
1874397991611,4cyclictest3561480-21systemd-journal23:30:312
1874397991611,3cyclictest2114990-21sh22:05:012
1874397991611,3cyclictest2104213-21ntp_kernel_pll_21:55:212
1874393991614,1cyclictest2342864-21fschecks_time00:25:141
1874393991613,2cyclictest2319795-21kernelversion00:10:191
1874393991613,2cyclictest2020234-21ps21:05:171
1874393991613,2cyclictest1981908-21cron20:35:001
1874393991613,2cyclictest1957150-21cron20:15:001
1874393991613,2cyclictest1925829-21unixbench_multi19:45:261
1874393991613,2cyclictest1925829-21unixbench_multi19:45:251
1874393991613,2cyclictest0-21swapper/120:20:011
1874393991612,3cyclictest2313370-21kthreadcore00:05:231
1874393991612,3cyclictest2080417-21unixbench-2d21:40:281
1874393991611,4cyclictest1932036-21unixbench_singl19:50:271
187439399160,14cyclictest2074216-21idleruntime-cro21:40:001
1874390991613,2cyclictest2237451-21fschecks_time23:20:150
1874390991613,2cyclictest2196711-21cpuspeed_turbos22:55:150
1874390991613,2cyclictest2183218-21kthreadcore22:45:170
1874390991613,2cyclictest1976532-21irqcore20:30:160
1874390991613,2cyclictest0-21swapper/023:40:000
1874390991612,2cyclictest0-21swapper/019:14:590
1874399991512,1cyclictest0-21swapper/323:55:203
1874399991512,1cyclictest0-21swapper/323:10:003
1874399991512,1cyclictest0-21swapper/322:35:003
1874399991512,1cyclictest0-21swapper/321:44:493
1874399991512,1cyclictest0-21swapper/321:24:543
1874399991512,1cyclictest0-21swapper/321:10:003
1874399991512,1cyclictest0-21swapper/320:50:013
1874399991512,1cyclictest0-21swapper/320:30:333
1874399991511,2cyclictest2350318-21turbostat00:30:003
1874399991511,1cyclictest0-21swapper/323:10:203
1874399991511,1cyclictest0-21swapper/319:45:003
187439799156,6cyclictest2334078-21systemctl00:20:002
187439799151,4cyclictest2180450-21apt-get22:45:132
1874397991513,1cyclictest1988355-21idleruntime-cro20:40:012
1874397991513,1cyclictest0-21swapper/223:00:002
1874397991512,2cyclictest2338295-21kthreadcore00:20:232
1874397991512,2cyclictest2262284-21irqcore23:35:192
1874397991512,2cyclictest2241693-21kthreadcore23:20:192
1874397991512,2cyclictest0-21swapper/221:20:192
1874397991512,2cyclictest0-21swapper/220:35:002
1874397991512,2cyclictest0-21swapper/220:30:012
1874397991511,2cyclictest3564018-21systemctl20:15:012
1874397991511,2cyclictest2116234-21irqcore22:05:192
187439399159,4cyclictest1-21systemd21:25:011
1874393991512,2cyclictest3564018-21systemctl19:25:001
1874393991512,2cyclictest2307454-21ntp_kernel_pll_00:00:221
1874393991512,2cyclictest2283206-21ntp_kernel_pll_23:45:241
1874393991512,2cyclictest2209883-21proc_pri23:00:241
1874393991512,2cyclictest2148160-21gltestperf22:25:201
1874393991512,2cyclictest2025687-21diskmemload21:15:201
1874393991512,2cyclictest2019500-21cron21:05:001
1874393991512,2cyclictest0-21swapper/123:55:171
1874393991512,2cyclictest0-21swapper/119:40:011
1874393991512,2cyclictest0-21swapper/119:40:001
1874393991512,2cyclictest0-21swapper/100:20:001
1874393991511,3cyclictest2218331-21unixbench-2d23:05:251
1874393991511,3cyclictest0-21swapper/120:50:011
1874393991511,2cyclictest0-21swapper/122:50:161
187439399150,4cyclictest2161657-21unixbench_singl22:30:281
187439399150,2cyclictest2220989-21grep23:10:131
1874390991513,1cyclictest0-21swapper/023:55:010
1874390991513,1cyclictest0-21swapper/020:00:010
1874390991512,2cyclictest2139850-21cpuspeed_turbos22:20:150
1874390991512,2cyclictest2078810-21kthreadcore21:40:210
1874390991512,2cyclictest1968645-21ntp_kernel_pll_20:20:210
1874390991512,2cyclictest1951595-21gltestperf20:10:170
1874390991512,2cyclictest0-21swapper/000:05:270
1874390991511,3cyclictest2024807-21ntp_kernel_pll_21:05:220
1874390991511,2cyclictest2256272-21kthreadcore23:30:220
1874390991511,2cyclictest2256272-21kthreadcore23:30:210
1874399991411,1cyclictest0-21swapper/323:42:163
1874399991411,1cyclictest0-21swapper/322:25:263
1874399991411,1cyclictest0-21swapper/322:11:043
1874399991411,1cyclictest0-21swapper/322:05:213
1874399991411,1cyclictest0-21swapper/322:00:203
1874399991411,1cyclictest0-21swapper/321:25:183
1874399991411,1cyclictest0-21swapper/321:15:133
1874399991411,1cyclictest0-21swapper/321:14:593
1874399991411,1cyclictest0-21swapper/321:04:593
1874399991411,1cyclictest0-21swapper/320:35:193
1874399991411,1cyclictest0-21swapper/320:20:193
1874399991411,1cyclictest0-21swapper/319:30:013
1874399991411,1cyclictest0-21swapper/319:20:213
1874399991411,1cyclictest0-21swapper/300:15:213
1874399991411,1cyclictest0-21swapper/300:00:243
1874397991411,2cyclictest277-21plymouthd20:50:012
1874397991411,2cyclictest2304297-21kthreadcore00:00:182
1874397991411,2cyclictest2086559-21kthreadcore21:45:182
1874397991411,2cyclictest1965962-21kthreadcore20:20:182
1874397991411,2cyclictest1932409-21sshd19:54:352
1874397991411,2cyclictest1921101-21irqcore19:45:182
1874397991411,2cyclictest1921101-21irqcore19:45:172
1874397991411,2cyclictest1911066-21kthreadcore19:35:212
1874397991411,2cyclictest1911066-21kthreadcore19:35:202
187439799141,10cyclictest1901297-21ls19:30:012
1874397991410,3cyclictest1941771-21kthreadcore20:00:212
1874397991410,2cyclictest2301799-21kworker/u8:1-nfsiod00:30:172
1874397991410,2cyclictest2204656-21/usr/sbin/munin23:00:172
187439399143,5cyclictest11systemd19:35:011
187439399141,2cyclictest2139391-21sendmail22:20:011
1874393991411,2cyclictest2291059-21latency23:50:211
1874393991411,2cyclictest2185195-21kthreadcore22:45:191
1874393991411,2cyclictest2116768-21kthreadcore22:05:211
1874393991411,2cyclictest2090225-21kworker/u8:3-writeback22:40:161
1874393991411,2cyclictest0-21swapper/121:15:011
1874393991410,2cyclictest0-21swapper/121:30:251
187439099145,8cyclictest0-21swapper/021:15:230
1874390991412,1cyclictest2122255-21kworker/u8:0+rpciod22:35:010
1874390991412,1cyclictest2081739-21kworker/0:1+kacpid21:55:300
1874390991411,2cyclictest2018605-21ntp_kernel_err21:00:240
1874390991410,3cyclictest2045209-21kthreadcore21:20:200
1874399991311,1cyclictest0-21swapper/321:32:293
1874399991310,1cyclictest0-21swapper/323:35:213
1874399991310,1cyclictest0-21swapper/323:15:173
1874399991310,1cyclictest0-21swapper/322:50:203
1874399991310,1cyclictest0-21swapper/322:20:203
1874399991310,1cyclictest0-21swapper/320:50:193
1874399991310,1cyclictest0-21swapper/320:08:323
1874399991310,1cyclictest0-21swapper/319:35:003
1874399991310,1cyclictest0-21swapper/319:10:193
1874399991310,1cyclictest0-21swapper/300:35:193
1874399991310,1cyclictest0-21swapper/300:25:013
187439799139,3cyclictest2359041-21gltestperf00:35:152
187439799139,3cyclictest1917164-21kthreadcore19:40:212
187439799138,3cyclictest2072015-21grep21:35:252
187439799138,3cyclictest0-21swapper/219:15:032
187439799137,4cyclictest2310558-21grep00:05:192
1874397991310,2cyclictest2024727-21needreboot21:05:222
1874397991310,2cyclictest2005562-21kthreadcore20:50:192
1874397991310,2cyclictest0-21swapper/219:55:182
187439399139,3cyclictest501-21dbus-daemon20:00:001
187439399137,5cyclictest0-21swapper/120:05:011
187439399137,4cyclictest2237224-21awk23:20:141
187439399137,4cyclictest1879711-21ntpq19:10:241
187439399132,2cyclictest0-21swapper/120:50:201
1874393991311,1cyclictest0-21swapper/120:10:001
1874393991310,2cyclictest2103732-21kthreadcore21:55:201
1874393991310,2cyclictest1881648-21kthreadcore19:15:021
1874393991310,2cyclictest0-21swapper/120:25:241
187439099131,4cyclictest1950650-21sshd20:05:350
1874390991312,1cyclictest1867862-21kworker/0:1+kacpid19:18:450
1874390991311,1cyclictest2219767-21kworker/0:1+kacpid23:47:160
1874390991311,1cyclictest2131233-21kworker/0:2+kacpid22:26:200
1874390991311,1cyclictest1981878-21kworker/0:2+kacpid20:42:450
1874390991310,2cyclictest1944798-21cron20:05:000
1874390991310,2cyclictest0-21swapper/023:40:170
1874390991310,2cyclictest0-21swapper/022:45:010
1874390991310,2cyclictest0-21swapper/000:15:260
187439099130,2cyclictest19028311irqrtprio19:30:180
187439999129,1cyclictest0-21swapper/323:20:183
187439999129,1cyclictest0-21swapper/320:15:203
187439999129,1cyclictest0-21swapper/320:10:223
187439999129,1cyclictest0-21swapper/300:30:203
1874399991210,1cyclictest0-21swapper/323:30:233
1874399991210,1cyclictest0-21swapper/323:30:223
1874399991210,1cyclictest0-21swapper/322:40:153
187439799129,2cyclictest2038652-21kthreadcore21:15:182
187439399129,2cyclictest0-21swapper/120:40:161
1874393991210,1cyclictest0-21swapper/119:25:181
187439099128,3cyclictest1895108-21cron19:29:550
1874390991210,1cyclictest2131233-21kworker/0:2+kacpid22:51:320
1874390991210,1cyclictest2098802-21kworker/0:0+kacpid22:00:150
1874390991210,1cyclictest1981878-21kworker/0:2+kacpid20:47:050
1874390991210,1cyclictest0-21swapper/019:45:240
1874390991210,1cyclictest0-21swapper/019:45:230
187439999118,1cyclictest0-21swapper/323:50:183
187439999118,1cyclictest0-21swapper/321:45:173
187439999118,1cyclictest0-21swapper/320:00:203
187439999118,1cyclictest0-21swapper/319:35:173
187439999118,1cyclictest0-21swapper/319:35:163
187439399110,1cyclictest0-21swapper/119:42:221
187439999108,1cyclictest0-21swapper/319:50:203
187439999108,1cyclictest0-21swapper/319:45:233
187439999108,1cyclictest0-21swapper/319:45:223
187439799107,1cyclictest0-21swapper/219:10:012
18739512100,1sleep30-21swapper/319:05:473
18743939974,2cyclictest19850irq/126-ahci[0000:00:17.0]19:10:011
18743909971,4cyclictest1874392-21latency_hist19:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional