You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-22 - 13:58
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8.osadl.org (updated Wed Oct 22, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31312942840,1sleep20-21swapper/219:50:212
33003412320,1chrt3300339-21python321:55:250
31717092280,1chrt3171708-21df_inode20:25:161
3076732992813,11cyclictest557-21gmain23:15:242
35141882270,2sleep2421rcuc/200:10:192
3076732992618,7cyclictest500-21avahi-daemon00:35:002
3076732992618,6cyclictest0-21swapper/200:00:002
3076732992617,8cyclictest0-21swapper/221:45:002
307673299259,11cyclictest3473610-21irqcore23:45:172
307673299259,11cyclictest3473610-21irqcore23:45:172
3076729992521,2cyclictest0-21swapper/100:00:001
3076729992422,1cyclictest0-21swapper/123:20:001
31119572230,1sleep30-21swapper/319:35:193
3076732992318,3cyclictest0-21swapper/221:00:002
3076732992313,8cyclictest3359321-21http22:35:002
307673299231,12cyclictest769-21runrttasks22:05:162
307673299230,12cyclictest3215660-21grep21:05:152
3076735992219,1cyclictest0-21swapper/321:00:003
3076732992218,2cyclictest3247330-21irqrtprio21:25:172
3076732992217,4cyclictest3177431-21cron20:30:012
3076732992211,10cyclictest3254782-21gltestperf21:30:182
307673299221,10cyclictest3818356-21systemd-journal22:24:022
307673299220,10cyclictest3102318-21/usr/sbin/munin19:25:202
3076735992118,1cyclictest0-21swapper/320:40:013
3076732992113,7cyclictest3202043-21ntp_kernel_pll_20:50:222
3076732992113,7cyclictest3195622-21ntp_kernel_err20:45:182
3076732992113,7cyclictest3093647-21kthreadcore19:20:172
307673299210,11cyclictest3369430-21grep22:40:172
307673299210,10cyclictest3456201-21taskset23:34:292
307673299210,10cyclictest3408384-21sh23:05:122
307673299209,8cyclictest3109930-21date19:35:002
307673299201,9cyclictest3236539-21taskset21:17:262
3076732992017,2cyclictest3365091-21runrttasks22:35:222
3076732992013,6cyclictest3324888-21tail22:10:272
3076732992013,6cyclictest3305465-21kthreadcore22:00:182
3076732992013,6cyclictest3226935-21netstat21:10:222
3076732992013,6cyclictest3164280-21proc_pri20:15:242
3076732992012,6cyclictest0-21swapper/219:20:002
3076732992012,6cyclictest0-21swapper/200:10:002
3076732992011,8cyclictest3440345-21sh23:25:012
307673299201,10cyclictest3116193-21/usr/sbin/munin19:40:172
3076732992010,8cyclictest3559586-21ntpq00:35:212
307673299200,9cyclictest3503529-21sh00:01:532
307673299200,11cyclictest3528254-21taskset00:17:332
307673299200,10cyclictest3535767-21sh00:20:392
3076729992017,2cyclictest3335072-21idleruntime-cro22:20:011
3076729992016,2cyclictest0-21swapper/100:40:001
3076729992016,2cyclictest0-21swapper/100:20:001
3076735991916,1cyclictest0-21swapper/321:05:013
3076735991916,1cyclictest0-21swapper/320:30:003
3076735991916,1cyclictest0-21swapper/319:30:003
307673299199,9cyclictest501-21dbus-daemon23:00:012
307673299191,9cyclictest769-21runrttasks21:02:272
307673299191,10cyclictest3383575-21taskset22:49:222
3076732991910,8cyclictest3295375-21gltestperf21:55:162
307673299190,9cyclictest3400485-21runrttasks23:00:152
307673299190,10cyclictest769-21runrttasks20:40:432
3076729991916,2cyclictest3505150-21latency_hist00:05:011
3076729991916,2cyclictest3326985-21cron22:15:001
3076729991913,4cyclictest0-21swapper/123:45:001
34511392180,1sleep10-21swapper/123:30:171
3076735991815,1cyclictest0-21swapper/322:05:003
307673299189,7cyclictest3818356-21systemd-journal22:55:002
307673299187,9cyclictest3110199-21grep19:35:162
307673299185,12cyclictest3458226-21cut23:35:172
307673299185,12cyclictest0-21swapper/222:15:202
3076732991815,2cyclictest3189746-21munin-run20:40:012
3076732991814,3cyclictest3239746-21kthreadcore21:20:222
3076732991812,5cyclictest3343984-21interrupts22:25:172
3076732991811,6cyclictest0-21swapper/220:10:002
3076732991810,7cyclictest3183076-21unixbench-2d20:30:262
3076732991810,6cyclictest0-21swapper/219:15:002
307673299180,9cyclictest3269861-21sh21:39:122
307673299180,9cyclictest3134624-21/usr/sbin/munin19:55:172
307673299180,9cyclictest3134624-21/usr/sbin/munin19:55:172
307672999186,10cyclictest501-21dbus-daemon19:40:001
3076729991816,1cyclictest3076720-21cyclictest21:45:201
3076729991815,2cyclictest3400447-21cpuspeed_turbos23:00:151
3076729991815,2cyclictest3270319-21latency_hist21:40:001
3076729991813,1cyclictest191rcu_preempt21:41:011
3076726991816,1cyclictest514-21systemd-logind00:40:010
3076726991816,1cyclictest508-21polkitd21:40:000
3076726991816,1cyclictest3408079-21systemd-run23:05:010
3076726991815,2cyclictest3335076-21awk22:20:000
3076726991814,2cyclictest3457151-21gltestperf23:35:150
3076726991814,2cyclictest0-21swapper/020:55:020
3076726991812,5cyclictest3275701-21ntp_states21:40:210
3076735991714,1cyclictest3343171-21turbostat22:25:003
3076735991714,1cyclictest0-21swapper/319:50:013
3076735991714,1cyclictest0-21swapper/300:20:003
3076735991713,1cyclictest0-21swapper/323:40:003
307673299173,9cyclictest3294740-21basename21:55:002
3076732991710,6cyclictest0-21swapper/223:50:242
3076732991710,5cyclictest0-21swapper/200:25:232
307673299170,9cyclictest3171003-21chrt20:21:472
307673299170,8cyclictest769-21runrttasks23:12:552
307673299170,10cyclictest769-21runrttasks23:44:412
3076729991714,2cyclictest3457209-21cut23:35:151
3076729991714,2cyclictest0-21swapper/122:40:001
3076729991713,3cyclictest3289938-21kthreadcore21:50:241
3076729991713,3cyclictest3243420-21ntp_kernel_pll_21:20:251
3076729991711,4cyclictest3416048-21cron23:09:591
3076726991714,2cyclictest3345975-21kthreadcore22:25:180
3076726991714,2cyclictest3183657-21latency_hist20:35:010
3076726991714,2cyclictest3109799-21gpgconf19:35:000
3076735991614,1cyclictest0-21swapper/320:50:173
3076735991613,1cyclictest0-21swapper/323:25:143
3076735991613,1cyclictest0-21swapper/323:13:543
3076735991613,1cyclictest0-21swapper/323:05:213
3076735991613,1cyclictest0-21swapper/322:44:223
3076735991613,1cyclictest0-21swapper/322:30:233
3076735991613,1cyclictest0-21swapper/321:45:243
3076735991613,1cyclictest0-21swapper/321:30:213
3076735991613,1cyclictest0-21swapper/321:30:003
3076735991613,1cyclictest0-21swapper/321:14:023
3076735991613,1cyclictest0-21swapper/321:05:153
3076735991613,1cyclictest0-21swapper/319:55:183
3076735991613,1cyclictest0-21swapper/319:55:173
307673299169,5cyclictest0-21swapper/219:50:012
307673299168,7cyclictest3284440-21unixbench_multi21:45:242
307673299166,9cyclictest3157531-21cut20:10:202
3076732991613,2cyclictest3441182-21grep23:25:162
307672999163,2cyclictest3334128-21kworker/u8:1+nfsiod22:59:091
3076729991613,2cyclictest3533169-21kthreadcore00:20:201
3076729991613,2cyclictest3348637-21ntp_kernel_pll_22:25:211
3076729991612,3cyclictest7550irq/9-acpi21:00:171
3076726991613,2cyclictest3318967-21sh22:10:000
3076726991613,2cyclictest3251529-21/usr/sbin/munin21:25:230
3076726991613,2cyclictest0-21swapper/019:20:010
33099802150,1sleep03309988-21ssh22:03:110
3076735991512,1cyclictest0-21swapper/323:50:003
3076735991512,1cyclictest0-21swapper/323:50:003
3076735991512,1cyclictest0-21swapper/323:30:173
3076735991512,1cyclictest0-21swapper/323:16:513
3076735991512,1cyclictest0-21swapper/322:54:573
3076735991512,1cyclictest0-21swapper/320:31:453
3076735991512,1cyclictest0-21swapper/319:35:003
3076735991512,1cyclictest0-21swapper/319:15:003
3076735991512,1cyclictest0-21swapper/300:05:263
3076735991512,1cyclictest0-21swapper/300:05:003
307673299155,9cyclictest3141475-21cut20:00:192
307672999159,5cyclictest3197020-21sshd20:50:021
307672999159,1cyclictest191rcu_preempt22:50:001
3076729991513,1cyclictest0-21swapper/122:35:001
3076729991512,2cyclictest3473005-21/usr/sbin/munin23:45:211
3076729991512,2cyclictest3473005-21/usr/sbin/munin23:45:211
3076729991512,2cyclictest3416990-21irqcore23:10:191
3076729991512,2cyclictest3375979-21latency_hist22:45:011
3076729991512,2cyclictest0-21swapper/120:40:001
3076729991512,2cyclictest0-21swapper/119:35:001
3076729991511,3cyclictest3315039-21kthreadcore22:05:221
3076729991511,3cyclictest1-21systemd00:34:081
3076729991510,4cyclictest3448008-21sh23:29:061
3076726991513,1cyclictest3520706-21kworker/0:2+kacpid00:24:130
3076726991512,2cyclictest524-21rs:main0
3076726991512,2cyclictest3465106-21fschecks_count23:40:150
3076726991512,2cyclictest3384181-21/usr/sbin/munin22:50:290
3076726991512,2cyclictest3261264-21kworker/0:0+kacpi_notify22:45:050
3076726991512,2cyclictest0-21swapper/021:20:000
3076726991511,3cyclictest3505113-21apt-key00:05:000
3076726991511,2cyclictest0-21swapper/023:55:180
3076726991510,4cyclictest3425962-21kthreadcore23:15:180
3076735991412,1cyclictest0-21swapper/320:00:163
3076735991412,0cyclictest0-21swapper/320:19:453
3076735991411,1cyclictest0-21swapper/323:59:103
3076735991411,1cyclictest0-21swapper/323:50:143
3076735991411,1cyclictest0-21swapper/323:42:243
3076735991411,1cyclictest0-21swapper/323:01:173
3076735991411,1cyclictest0-21swapper/322:45:203
3076735991411,1cyclictest0-21swapper/322:39:563
3076735991411,1cyclictest0-21swapper/322:25:193
3076735991411,1cyclictest0-21swapper/322:12:383
3076735991411,1cyclictest0-21swapper/322:05:563
3076735991411,1cyclictest0-21swapper/321:55:173
3076735991411,1cyclictest0-21swapper/321:54:413
3076735991411,1cyclictest0-21swapper/321:40:203
3076735991411,1cyclictest0-21swapper/321:35:243
3076735991411,1cyclictest0-21swapper/320:50:023
3076735991411,1cyclictest0-21swapper/319:50:523
3076735991411,1cyclictest0-21swapper/319:25:003
3076735991411,1cyclictest0-21swapper/300:30:213
3076735991411,1cyclictest0-21swapper/300:25:193
3076735991411,1cyclictest0-21swapper/300:20:243
3076735991411,1cyclictest0-21swapper/300:10:043
3076735991410,1cyclictest0-21swapper/300:35:243
307672999143,5cyclictest34-21ksoftirqd/122:50:221
307672999141,6cyclictest3310819-21munin-run22:05:001
3076729991411,2cyclictest750-21sshd19:24:461
3076729991411,2cyclictest3423855-21kworker/u8:2-xprtiod00:25:251
3076729991411,2cyclictest3221860-21apt-get21:10:151
3076729991411,2cyclictest3209398-21kworker/u8:2+rpciod22:21:161
3076729991411,2cyclictest3203551-21gltestperf20:55:131
3076729991411,2cyclictest3183571-21cron20:35:001
3076729991411,2cyclictest0-21swapper/121:30:211
3076729991410,3cyclictest761-21cups-browsed19:50:001
3076729991410,3cyclictest1-21systemd20:13:471
307672699148,4cyclictest501-21dbus-daemon21:00:000
307672699143,4cyclictest581rcuop/320:25:000
307672699141,5cyclictest3226795-21ssh21:10:200
3076726991412,1cyclictest3351771-21cpuspeed_turbos22:30:130
3076726991412,1cyclictest3261264-21kworker/0:0+kacpid23:05:190
3076726991412,1cyclictest3097271-21kworker/0:0+kacpid19:47:180
3076726991412,1cyclictest0-21swapper/022:15:000
3076726991411,2cyclictest3118314-21kthreadcore19:40:190
3076726991411,2cyclictest3004-21cron22:45:010
3076726991411,2cyclictest0-21swapper/020:45:110
3076726991411,2cyclictest0-21swapper/019:30:000
3076726991410,2cyclictest3164505-21timerandwakeup20:15:250
307672699140,2cyclictest16-21ksoftirqd/000:09:550
3076735991311,1cyclictest0-21swapper/322:55:013
3076735991311,1cyclictest0-21swapper/321:15:183
3076735991310,1cyclictest0-21swapper/323:20:113
3076735991310,1cyclictest0-21swapper/322:15:193
3076735991310,1cyclictest0-21swapper/321:20:223
3076735991310,1cyclictest0-21swapper/320:20:173
3076735991310,1cyclictest0-21swapper/319:40:193
3076735991310,1cyclictest0-21swapper/319:20:003
307673299130,10cyclictest3076809-21latency_hist19:10:002
307672999139,2cyclictest0-21swapper/119:50:281
307672999139,2cyclictest0-21swapper/119:10:001
307672999131,3cyclictest3141158-21head20:00:171
3076729991310,2cyclictest3482816-21kthreadcore23:50:191
3076729991310,2cyclictest3295176-21cpuspeed_turbos21:55:151
3076729991310,2cyclictest3247340-21irqrtprio21:25:171
3076729991310,2cyclictest3196533-21latency_hist20:50:011
3076729991310,2cyclictest3122171-21latency_hist19:45:001
3076729991310,2cyclictest3082379-21unixbench_multi19:10:251
3076729991310,2cyclictest0-21swapper/100:05:241
307672699138,3cyclictest3818356-21systemd-journal19:54:550
307672699138,3cyclictest3437709-21ntp_kernel_pll_23:20:220
307672699138,3cyclictest3138858-21tr19:55:200
307672699138,3cyclictest3138858-21tr19:55:190
307672699138,1cyclictest0-21swapper/021:20:150
3076726991311,1cyclictest3444758-21kthreadcore23:25:180
3076726991311,1cyclictest3395276-21kthreadcore22:55:200
3076726991311,1cyclictest3097271-21kworker/0:0+kacpid21:00:230
3076726991310,2cyclictest3547269-21irqcore00:30:200
3076726991310,2cyclictest3450163-21kthreadcore23:30:170
3076726991310,2cyclictest3417375-21irqrtprio23:10:190
3076726991310,2cyclictest3360647-21irqcore22:35:160
3076726991310,2cyclictest3261264-21kworker/0:0+kacpid22:23:490
3076726991310,2cyclictest3254770-21gltestperf21:30:170
3076726991310,2cyclictest3215686-21cpuspeed_turbos21:05:150
3076726991310,2cyclictest3077009-21cpuspeed_turbos19:10:140
307672699130,4cyclictest3486118-21grep23:50:220
307673599129,1cyclictest0-21swapper/320:45:013
3076735991210,1cyclictest0-21swapper/320:05:153
307672999129,2cyclictest3221478-21diskmemload23:20:151
307672999129,2cyclictest3165498-21tr20:20:151
307672999129,2cyclictest3159354-21cpuspeed_turbos20:15:161
307672999129,2cyclictest0-21swapper/120:05:161
307672999129,2cyclictest0-21swapper/100:10:181
3076729991210,1cyclictest0-21swapper/121:15:001
3076729991210,1cyclictest0-21swapper/119:15:161
307672999120,1cyclictest3221513-21/usr/sbin/munin21:10:001
307672699129,2cyclictest3141071-21fschecks_time20:00:170
307672699129,1cyclictest0-21swapper/019:35:150
307672699128,3cyclictest3526690-21needreboot00:15:200
307672699128,3cyclictest3515501-21kthreadcore00:10:200
307672699127,3cyclictest3818356-21systemd-journal20:28:080
307672699127,3cyclictest3154807-21tr20:10:180
3076726991210,1cyclictest3261264-21kworker/0:0+kacpid21:45:150
3076726991210,1cyclictest3094288-21kthreadcore19:20:180
3076726991210,1cyclictest3076640-21latency_hist19:10:010
3076726991210,1cyclictest0-21swapper/020:40:000
3076726991210,1cyclictest0-21swapper/020:10:010
307672699120,4cyclictest3543092-21grep00:25:200
307673599119,1cyclictest0-21swapper/320:10:193
307672999118,2cyclictest3102900-21unixbench_multi19:25:251
307672999116,4cyclictest3135907-21sed19:55:171
307672999116,4cyclictest3135907-21sed19:55:171
307672999110,1cyclictest0-21swapper/120:40:531
307672699118,2cyclictest3473145-21cpuspeed_turbos23:45:150
307672699118,2cyclictest3473145-21cpuspeed_turbos23:45:140
307672699106,3cyclictest3190216-21sshd20:43:020
30767359975,1cyclictest0-21swapper/319:10:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional