You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 16:29
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8.osadl.org (updated Tue Sep 16, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
38535662410,1sleep30-21swapper/311:35:173
37265202370,1chrt0-21swapper/110:15:191
37265202370,1chrt0-21swapper/110:15:191
39141542340,1chrt3914157-21awk12:10:260
37394462340,1chrt3739447-21smartctl10:25:171
3467675992923,5cyclictest3714794-21turbostat.cron10:10:000
3467675992923,5cyclictest3658398-21grep09:35:000
3467675992518,6cyclictest3587097-21cron08:45:000
3467675992417,6cyclictest3722868-21cron10:15:000
3467675992416,6cyclictest3859595-21cron11:40:000
3467680992218,3cyclictest3706763-21sh10:05:002
3467677992218,3cyclictest514-21systemd-logind10:50:001
3467675992216,5cyclictest547-21gdbus09:15:010
3467675992216,5cyclictest3881552-21mailstats11:50:220
3467675992216,5cyclictest3772792-21kthreadcore10:45:180
346768099217,9cyclictest44-21ksoftirqd/211:40:002
3467677992118,2cyclictest3867965-21latency_hist11:44:591
3467677992118,2cyclictest3762916-21cron10:40:001
3467675992115,5cyclictest3611579-21unixbench_singl09:00:260
3467675992110,9cyclictest0-21swapper/012:15:250
3467682992017,1cyclictest0-21swapper/307:39:593
3467680992017,2cyclictest3879317-21kthreadcore11:50:162
3467680992017,2cyclictest3544041-21sh08:10:002
3467680992017,2cyclictest0-21swapper/210:15:002
3467680992016,3cyclictest761-21cups-browsed11:25:012
3467680992013,2cyclictest191rcu_preempt10:40:142
3467675992014,5cyclictest3563231-21irqcore08:25:160
3467675992014,5cyclictest3563231-21irqcore08:25:160
3467675992013,6cyclictest522-21in:imuxsock07:25:010
3467675992013,6cyclictest3787569-21cpuspeed_turbos10:55:130
3467675992012,6cyclictest3666411-21cron09:39:590
346767599200,3cyclictest3940447-21turbostat12:29:590
3467682991916,1cyclictest0-21swapper/311:20:153
3467682991916,1cyclictest0-21swapper/310:55:203
346768099199,4cyclictest44-21ksoftirqd/207:20:002
3467680991911,3cyclictest44-21ksoftirqd/210:20:002
3467680991911,3cyclictest44-21ksoftirqd/210:19:592
3467680991910,1cyclictest191rcu_preempt11:15:012
3467677991915,3cyclictest514-21systemd-logind07:35:011
3467675991913,5cyclictest3731282-21gltestperf10:20:160
3467675991913,5cyclictest3696071-21ntp_kernel_pll_09:55:200
3467675991913,5cyclictest1-21systemd07:15:010
3467682991815,1cyclictest0-21swapper/312:35:233
3467682991815,1cyclictest0-21swapper/310:10:163
3467682991815,1cyclictest0-21swapper/309:34:593
3467682991815,1cyclictest0-21swapper/307:45:003
3467682991814,1cyclictest0-21swapper/309:00:163
3467680991815,2cyclictest522-21in:imuxsock11:44:592
3467680991815,2cyclictest3634230-21idleruntime-cro09:20:012
3467677991815,2cyclictest3859672-21sh11:40:001
3467677991815,2cyclictest3677407-21kthreadcore09:45:181
3467677991815,2cyclictest3604710-21latency08:55:201
3467677991815,2cyclictest3574801-21sh08:35:011
3467677991814,2cyclictest3787967-21tr10:55:151
3467677991814,2cyclictest0-21swapper/112:15:001
3467677991814,2cyclictest0-21swapper/108:55:001
346767599183,13cyclictest3671449-21ssh09:40:220
3467675991813,4cyclictest0-21swapper/010:50:250
3467675991812,5cyclictest3889348-21needreboot11:55:200
3467675991812,5cyclictest3804906-21kthreadcore11:05:180
3467675991812,5cyclictest3795985-21irqcore11:00:170
3467675991812,5cyclictest3756261-21irqrtprio10:35:160
3467675991812,5cyclictest3724567-21irqrtprio10:15:170
3467675991812,5cyclictest3724567-21irqrtprio10:15:160
3467675991812,5cyclictest3702010-21kthreadcore10:00:200
3467675991812,5cyclictest3685870-21kthreadcore09:50:180
3467675991812,5cyclictest3677894-21kthreadcore09:45:200
3467675991812,5cyclictest3489936-21kthreadcore07:25:160
3467675991812,5cyclictest0-21swapper/008:00:110
3467675991812,4cyclictest0-21swapper/012:30:200
3467675991811,6cyclictest3626842-21gltestperf09:15:160
3467675991811,5cyclictest3868339-21cpuspeed_turbos11:45:130
3467675991811,5cyclictest3860753-21irqcore11:40:140
3467675991811,5cyclictest3819292-21turbostat11:15:000
3467675991811,5cyclictest0-21swapper/012:35:220
3467682991714,1cyclictest0-21swapper/312:30:013
3467682991714,1cyclictest0-21swapper/311:30:003
3467682991714,1cyclictest0-21swapper/310:20:013
3467682991714,1cyclictest0-21swapper/310:20:003
3467682991714,1cyclictest0-21swapper/310:09:593
3467682991714,1cyclictest0-21swapper/309:35:133
3467682991714,1cyclictest0-21swapper/309:05:213
3467682991714,1cyclictest0-21swapper/308:45:223
3467682991714,1cyclictest0-21swapper/308:05:123
346768099178,4cyclictest191rcu_preempt11:05:012
346768099172,4cyclictest44-21ksoftirqd/212:35:012
3467680991714,2cyclictest3651011-21gltestperf09:30:152
3467680991711,1cyclictest471rcuog/210:20:342
3467680991710,2cyclictest191rcu_preempt09:35:182
3467680991710,1cyclictest191rcu_preempt09:28:282
346768099170,6cyclictest3762958-21sendmail10:40:012
3467677991715,1cyclictest0-21swapper/111:30:271
3467677991713,3cyclictest3873277-21needreboot11:45:201
346767599175,5cyclictest3818356-21systemd-journal08:25:000
346767599172,3cyclictest3544316-21apt-get08:10:130
346767599171,9cyclictest3818356-21systemd-journal08:50:010
346767599171,6cyclictest3900113-21idleruntime-cro12:05:000
3467675991712,4cyclictest3599395-21cron08:55:000
3467675991712,4cyclictest3487954-21kworker/0:0+kacpid07:48:250
3467675991712,4cyclictest3487954-21kworker/0:0+kacpid07:37:020
3467675991711,5cyclictest3845719-21kthreadcore11:30:210
3467675991711,5cyclictest3827964-21fschecks_time11:20:150
3467675991711,5cyclictest3821001-21kthreadcore11:15:180
3467675991711,5cyclictest3748550-21sed10:30:170
3467675991711,5cyclictest3614993-21kthreadcore09:05:190
3467675991711,5cyclictest3551839-21kthreadcore08:15:180
346767599171,14cyclictest3765132-21ssh10:40:190
346767599171,13cyclictest3474044-21diskstats07:15:010
3467675991710,6cyclictest3604716-21latency08:55:200
346768299162,1cyclictest0-21swapper/308:14:553
3467682991614,1cyclictest0-21swapper/312:20:173
3467682991614,1cyclictest0-21swapper/312:00:163
3467682991614,1cyclictest0-21swapper/311:30:233
3467682991614,1cyclictest0-21swapper/310:25:203
3467682991614,1cyclictest0-21swapper/310:25:003
3467682991613,1cyclictest0-21swapper/312:35:013
3467682991613,1cyclictest0-21swapper/312:15:193
3467682991613,1cyclictest0-21swapper/312:05:173
3467682991613,1cyclictest0-21swapper/311:50:163
3467682991613,1cyclictest0-21swapper/311:45:133
3467682991613,1cyclictest0-21swapper/311:40:173
3467682991613,1cyclictest0-21swapper/311:15:233
3467682991613,1cyclictest0-21swapper/311:00:173
3467682991613,1cyclictest0-21swapper/310:35:193
3467682991613,1cyclictest0-21swapper/310:30:143
3467682991613,1cyclictest0-21swapper/310:00:173
3467682991613,1cyclictest0-21swapper/309:55:173
3467682991613,1cyclictest0-21swapper/309:50:143
3467682991613,1cyclictest0-21swapper/309:45:203
3467682991613,1cyclictest0-21swapper/309:40:203
3467682991613,1cyclictest0-21swapper/309:25:183
3467682991613,1cyclictest0-21swapper/309:20:173
3467682991613,1cyclictest0-21swapper/308:50:193
3467682991613,1cyclictest0-21swapper/308:30:203
3467682991613,1cyclictest0-21swapper/308:20:173
3467682991613,1cyclictest0-21swapper/307:55:183
3467682991613,1cyclictest0-21swapper/307:20:193
3467682991613,1cyclictest0-21swapper/307:10:183
3467682991612,1cyclictest0-21swapper/308:25:193
3467682991612,1cyclictest0-21swapper/308:25:183
346768099169,2cyclictest191rcu_preempt09:55:162
346768099169,2cyclictest191rcu_preempt08:55:012
346768099166,3cyclictest0-21swapper/211:55:182
346768099163,7cyclictest3869047-21irqcore11:45:162
346768099162,6cyclictest3519418-21munin-run07:49:592
3467680991613,2cyclictest3617674-21unixbench_multi09:05:382
3467680991613,2cyclictest3513295-21latency_hist07:45:012
3467680991613,2cyclictest3488277-21cron07:25:012
3467680991612,3cyclictest3626141-21cron09:15:002
3467680991612,2cyclictest421rcuc/212:39:592
346768099160,5cyclictest3795182-21apt11:00:002
346768099160,4cyclictest3600142-21cut08:55:152
3467677991614,1cyclictest3956583-21cron12:39:591
3467677991613,2cyclictest500-21avahi-daemon08:15:231
3467677991613,2cyclictest3811198-21cron11:10:011
3467677991613,2cyclictest3641406-21kworker/u8:3-rpciod09:35:141
3467677991613,2cyclictest1-21systemd09:16:321
3467677991613,2cyclictest0-21swapper/111:05:011
3467677991613,2cyclictest0-21swapper/110:30:231
3467677991613,2cyclictest0-21swapper/109:35:001
3467677991613,2cyclictest0-21swapper/108:00:001
3467677991612,2cyclictest0-21swapper/107:40:001
3467677991612,2cyclictest0-21swapper/107:15:011
3467677991610,4cyclictest3549778-21grep08:10:241
346767599169,6cyclictest3744658-21unixbench_singl10:25:260
346767599168,6cyclictest3544028-21cron08:10:000
346767599165,5cyclictest3818356-21systemd-journal08:35:010
346767599161,3cyclictest3836083-21grep11:25:150
3467675991610,5cyclictest3902817-21kthreadcore12:05:190
3467675991610,5cyclictest3643906-21irqrtprio09:25:160
3467675991610,5cyclictest3640158-21unixbench_singl09:20:230
3467675991610,5cyclictest3495047-21gltestperf07:30:170
3467682991512,1cyclictest0-21swapper/312:14:593
3467682991512,1cyclictest0-21swapper/311:10:173
3467682991512,1cyclictest0-21swapper/310:50:183
3467682991512,1cyclictest0-21swapper/310:40:163
3467682991512,1cyclictest0-21swapper/309:15:193
3467682991512,1cyclictest0-21swapper/308:40:133
3467682991512,1cyclictest0-21swapper/308:35:153
3467682991512,1cyclictest0-21swapper/308:00:163
3467682991512,1cyclictest0-21swapper/307:50:243
3467682991512,1cyclictest0-21swapper/307:30:153
3467682991512,1cyclictest0-21swapper/307:15:223
346768099159,2cyclictest191rcu_preempt08:25:132
346768099159,2cyclictest191rcu_preempt08:25:122
346768099158,2cyclictest191rcu_preempt11:25:232
346768099156,4cyclictest191rcu_preempt07:55:192
346768099156,4cyclictest0-21swapper/209:52:012
346768099156,2cyclictest191rcu_preempt10:50:132
346768099152,6cyclictest3841452-21kworker/2:2-events11:30:182
3467680991513,1cyclictest3908211-21munin-run12:10:002
346768099151,2cyclictest191rcu_preempt11:06:262
3467680991512,2cyclictest3642325-21latency_hist09:25:002
3467680991512,2cyclictest3561683-21proc_pri08:20:212
3467680991512,2cyclictest0-21swapper/207:30:002
3467680991511,3cyclictest3739321-21cut10:25:162
3467680991511,3cyclictest3672211-21timerandwakeup09:40:272
346768099150,6cyclictest3827349-21munin-run11:20:002
346768099150,4cyclictest44-21ksoftirqd/212:20:142
346768099150,3cyclictest3544763-21ps08:10:162
346767799159,4cyclictest3587098-21cron08:45:011
346767799153,2cyclictest37649572tr10:40:191
3467677991512,2cyclictest3836045-21gltestperf11:25:151
3467677991512,2cyclictest0-21swapper/112:19:591
3467677991510,4cyclictest3818356-21systemd-journal11:20:011
3467677991510,4cyclictest3811893-21grep11:10:151
3467677991510,4cyclictest3482087-21cron07:20:001
346767599159,5cyclictest3513231-21latency_hist07:45:000
346767599158,5cyclictest3531946-21ls08:00:010
346767599156,4cyclictest3519909-21cut07:50:140
346767599151,2cyclictest3575081-21apt-get08:35:130
346767599151,12cyclictest3924692-21apt-get12:20:130
3467682991412,1cyclictest0-21swapper/307:49:593
3467682991411,1cyclictest0-21swapper/311:05:123
3467682991411,1cyclictest0-21swapper/310:45:183
346768099148,2cyclictest191rcu_preempt10:30:242
346768099147,2cyclictest191rcu_preempt07:35:192
346768099144,6cyclictest3935705-21kthreadcore12:25:162
346768099143,6cyclictest44-21ksoftirqd/208:45:212
346768099143,3cyclictest0-21swapper/207:50:162
3467680991412,1cyclictest3618126-21diskmemload12:15:202
3467680991411,2cyclictest3909988-21kthreadcore12:10:182
3467680991411,2cyclictest3582473-21irqrtprio08:40:152
346768099140,9cyclictest1700052-21kworker/u8:2+rpciod10:05:212
346768099140,4cyclictest3894008-21kthreadcore12:00:172
346768099140,4cyclictest3680449-21grep09:45:262
346768099140,4cyclictest3556345-21chrt08:20:002
346768099140,3cyclictest3532483-21irqcore08:00:152
346768099140,3cyclictest3500578-21awk07:35:012
346768099140,3cyclictest0-21swapper/207:10:232
346767799149,3cyclictest3818356-21systemd-journal09:04:521
346767799144,2cyclictest39083492date12:10:011
3467677991411,2cyclictest3898012-21unixbench_multi12:00:261
3467677991411,2cyclictest3736562-21unixbench_multi10:20:261
3467677991411,2cyclictest3715308-21sh10:10:141
3467677991411,2cyclictest3578779-21kthreadcore08:35:181
3467677991411,2cyclictest0-21swapper/110:05:191
3467677991411,2cyclictest0-21swapper/109:55:141
346767799140,2cyclictest3512525-21unixbench-2d07:40:261
3467682991311,1cyclictest0-21swapper/308:55:183
3467682991311,1cyclictest0-21swapper/308:15:153
3467682991311,1cyclictest0-21swapper/307:25:163
3467682991310,1cyclictest0-21swapper/311:55:173
3467682991310,1cyclictest0-21swapper/309:10:163
3467682991310,1cyclictest0-21swapper/307:10:013
346768099133,1cyclictest0-21swapper/208:35:162
346768099132,6cyclictest0-21swapper/210:45:192
3467680991310,2cyclictest0-21swapper/209:00:142
346768099130,8cyclictest3569581-21sed08:30:182
346767799139,3cyclictest3817337-21kworker/u8:1+rpciod12:25:131
346767799139,3cyclictest3592765-21unixbench-2d08:45:251
346767799139,3cyclictest1-21systemd09:05:171
346767799137,4cyclictest3818356-21systemd-journal11:54:591
346767799136,5cyclictest3818356-21systemd-journal08:24:591
3467677991310,2cyclictest3946342-21unixbench_multi12:30:241
3467677991310,2cyclictest3683798-21irqcore09:50:151
3467677991310,2cyclictest3666927-21cpuspeed_turbos09:40:171
3467677991310,2cyclictest3642958-21gltestperf09:25:151
3467677991310,2cyclictest3618126-21diskmemload10:00:201
3467677991310,2cyclictest3532593-21irqcore08:00:141
3467677991310,2cyclictest3482568-21fschecks_count07:20:151
3467677991310,2cyclictest3434094-21kworker/u8:1+nfsiod09:20:521
3467677991310,2cyclictest0-21swapper/111:25:011
3467677991310,2cyclictest0-21swapper/107:25:131
346767799129,2cyclictest3884454-21tr11:55:141
346767799129,2cyclictest3567833-21proc_pri08:25:221
346767799129,2cyclictest3567833-21proc_pri08:25:211
346767799129,2cyclictest3525576-21cron07:55:001
346767799129,2cyclictest3513678-21tr07:45:161
346767799129,2cyclictest0-21swapper/109:10:161
346767799128,3cyclictest3924831-21cpuspeed_turbos12:20:141
346768099115,2cyclictest191rcu_preempt07:10:012
346767799118,2cyclictest3618126-21diskmemload10:50:211
346767799116,4cyclictest3538366-21tune2fs08:05:151
346767599110,5cyclictest3467694-21latency_hist07:10:000
346767799106,2cyclictest3467583-21latency_hist07:10:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional