You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-04 - 16:37
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8.osadl.org (updated Tue Nov 04, 2025 12:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3242392480,0sleep2324242-21grep11:55:242
974022300,1chrt0-21swapper/309:35:243
4093920992624,1cyclictest91484-21systemd-run09:35:001
4093917992623,2cyclictest99812-21sendmail09:40:010
4093917992522,2cyclictest787520-21kworker/u8:1+events_unbound09:00:010
4093917992421,2cyclictest500-21avahi-daemon10:35:000
4093917992421,2cyclictest164435-21sendmail10:20:010
4093917992418,4cyclictest358843-21sendmail12:20:000
4093917992320,2cyclictest245792-21systemd-run11:10:000
4093917992318,3cyclictest4182639-21munin-plugin-st08:20:000
4093917992317,4cyclictest4133385-21cron07:40:000
409392099228,5cyclictest3818356-21systemd-journal12:35:001
4093920992219,2cyclictest4108469-21sendmail07:20:001
4093929992118,1cyclictest0-21swapper/311:45:003
4093929992118,1cyclictest0-21swapper/311:10:013
4093923992118,2cyclictest0-21swapper/208:40:002
4093920992118,2cyclictest522-21in:imuxsock09:50:011
4093920992118,2cyclictest0-21swapper/107:40:001
409391799211,17cyclictest342713-21idleruntime-cro12:10:000
4093929992017,1cyclictest0-21swapper/309:45:173
4093929992017,1cyclictest0-21swapper/309:10:003
4093923992017,2cyclictest4093909-21cyclictest07:15:002
4093923992017,2cyclictest270255-21latency_hist11:25:002
4093920992017,2cyclictest0-21swapper/109:00:011
4093929991916,1cyclictest0-21swapper/310:20:003
4093929991916,1cyclictest0-21swapper/309:55:003
4093929991916,1cyclictest0-21swapper/309:55:003
4093929991916,1cyclictest0-21swapper/309:45:003
4093929991916,1cyclictest0-21swapper/309:00:013
4093923991916,2cyclictest91734-21latency_hist09:35:002
4093923991916,2cyclictest14615-21irqcore08:40:172
4093923991914,3cyclictest334838-21latency_hist12:05:012
4093923991914,3cyclictest1-21systemd07:54:062
4093923991914,3cyclictest1-21systemd07:54:062
409392399190,16cyclictest4158444-21cpuspeed_turbos08:00:142
4093920991916,2cyclictest4120973-21cron07:30:001
4093920991916,2cyclictest0-21swapper/111:20:001
4093920991915,2cyclictest4177210-21irqcore08:15:191
4093917991914,3cyclictest14479-21gltestperf08:40:170
4093917991912,5cyclictest501-21dbus-daemon10:59:590
4093929991815,1cyclictest0-21swapper/312:10:173
4093923991815,2cyclictest75796-21cpuspeed_turbos09:25:142
4093923991815,2cyclictest4183258-21gltestperf08:20:152
4093923991815,2cyclictest4093909-21cyclictest10:15:142
4093920991815,2cyclictest44636-21idleruntime-cro09:05:001
4093920991815,2cyclictest180559-21idleruntime-cro10:30:011
4093920991815,2cyclictest0-21swapper/111:00:001
4093920991814,2cyclictest0-21swapper/110:25:001
4093917991815,2cyclictest83525-21latency_hist09:30:010
4093917991815,2cyclictest4170335-21munin-plugin-st08:10:000
4093917991815,2cyclictest0-21swapper/008:40:010
4093917991814,2cyclictest7477-21apt-key08:35:020
4093929991714,1cyclictest0-21swapper/311:25:183
4093929991714,1cyclictest0-21swapper/310:30:003
4093929991714,1cyclictest0-21swapper/309:20:003
4093929991714,1cyclictest0-21swapper/308:35:333
4093929991714,1cyclictest0-21swapper/307:20:003
4093923991715,1cyclictest205379-21cut10:45:172
4093923991714,2cyclictest132130-21sendmail10:00:002
4093923991714,2cyclictest0-21swapper/211:05:002
4093923991713,3cyclictest1-21systemd10:04:182
4093920991714,2cyclictest500-21avahi-daemon12:10:001
4093920991714,2cyclictest243638-21unixbench_singl11:05:251
4093920991714,1cyclictest0-21swapper/110:50:011
4093920991713,3cyclictest52544-21irqrtprio09:10:181
4093917991713,3cyclictest215549-21kthreadcore10:50:180
4093917991713,3cyclictest1-21systemd10:12:080
4093917991713,2cyclictest0-21swapper/010:10:000
4093917991712,3cyclictest383629-21grep12:35:140
4093929991613,1cyclictest0-21swapper/312:35:143
4093929991613,1cyclictest0-21swapper/312:20:083
4093929991613,1cyclictest0-21swapper/311:00:013
4093929991613,1cyclictest0-21swapper/310:50:183
409392399167,6cyclictest7181-21dump-pmu-power08:35:002
4093923991613,2cyclictest50765-21diskmemload10:05:262
4093923991613,2cyclictest500-21avahi-daemon11:25:242
4093923991612,3cyclictest49871-21ntp_kernel_pll_09:05:242
4093923991612,3cyclictest299932-21sendmail_mailqu11:40:252
4093920991613,2cyclictest235236-21unixbench_singl11:00:251
4093920991613,2cyclictest140223-21apt10:05:001
4093920991613,2cyclictest0-21swapper/110:35:001
4093920991612,3cyclictest266323-21kthreadcore11:20:191
4093920991612,2cyclictest0-21swapper/111:30:001
409391799169,5cyclictest0-21swapper/010:35:200
4093917991613,2cyclictest44213-21unixbench_singl09:00:240
4093917991613,2cyclictest4115230-21tr07:25:130
4093917991613,2cyclictest262068-21kworker/0:1+kacpid11:52:150
4093917991613,2cyclictest205555-21gltestperf10:45:160
4093917991613,2cyclictest0-21swapper/010:45:010
4093917991613,1cyclictest0-21swapper/011:20:240
4093917991612,3cyclictest85114-21irqrtprio09:30:180
4093917991612,3cyclictest4163247-21needreboot08:00:200
4093917991612,3cyclictest287821-21irqrtprio11:35:180
4093917991612,2cyclictest259394-21kworker/u8:3-xprtiod11:55:170
4093917991612,2cyclictest0-21swapper/012:30:000
4093917991612,2cyclictest0-21swapper/009:55:260
4093917991612,2cyclictest0-21swapper/009:40:160
4093917991612,2cyclictest0-21swapper/009:25:000
4093917991610,4cyclictest3818356-21systemd-journal09:12:330
4093929991513,1cyclictest0-21swapper/310:00:003
4093929991512,1cyclictest0-21swapper/311:05:003
4093929991512,1cyclictest0-21swapper/310:10:003
4093929991512,1cyclictest0-21swapper/308:34:393
4093929991512,1cyclictest0-21swapper/308:00:183
4093929991512,1cyclictest0-21swapper/307:43:183
4093929991512,1cyclictest0-21swapper/307:10:003
4093923991512,2cyclictest514-21systemd-logind11:15:002
4093923991512,2cyclictest39213-21irqcore09:00:172
4093923991512,2cyclictest343018-21/usr/sbin/munin12:10:232
4093923991512,2cyclictest25672-21unixbench-2d08:45:262
4093923991512,2cyclictest221041-21latency_hist10:55:012
4093923991512,2cyclictest103176-21kthreadcore09:40:182
4093923991512,2cyclictest0-21swapper/210:30:002
4093923991512,2cyclictest0-21swapper/210:20:212
4093923991512,2cyclictest0-21swapper/210:15:002
4093923991511,3cyclictest4142796-21kthreadcore07:45:172
4093923991511,3cyclictest4138655-21ntp_kernel_err07:40:212
4093923991511,3cyclictest204810-21sh10:45:002
4093920991512,2cyclictest71145-21kthreadcore09:20:171
4093920991512,2cyclictest4158028-21sh08:00:011
4093920991512,2cyclictest4133943-21fschecks_time07:40:161
4093920991512,2cyclictest3818356-21systemd-journal10:55:011
4093920991512,2cyclictest286861-21fschecks_time11:35:171
4093920991512,2cyclictest281568-21kthreadcore11:30:181
4093920991512,2cyclictest156494-21latency_hist10:15:011
4093920991512,2cyclictest0-21swapper/111:40:161
4093920991512,2cyclictest0-21swapper/109:20:001
4093920991511,3cyclictest145010-21kthreadcore10:05:201
4093920991511,2cyclictest0-21swapper/112:30:001
4093920991510,3cyclictest749-21lldpd12:15:141
409391799156,2cyclictest50772-21cron09:10:000
4093917991512,2cyclictest50765-21diskmemload12:30:220
4093917991512,2cyclictest255700-21kthreadcore11:15:170
4093917991512,2cyclictest165491-21irqcore10:20:200
4093917991511,3cyclictest0-21swapper/011:45:220
4093917991511,3cyclictest0-21swapper/011:45:220
4093917991511,3cyclictest0-21swapper/011:35:000
4093917991511,2cyclictest0-21swapper/009:15:150
4093917991511,2cyclictest0-21swapper/007:55:130
409391799150,2cyclictest4151898-21idleruntime-cro07:55:010
409391799150,2cyclictest4151898-21idleruntime-cro07:55:000
409392999142,9cyclictest262017-21turbostat11:24:553
4093929991411,1cyclictest0-21swapper/312:31:213
4093929991411,1cyclictest0-21swapper/312:19:293
4093929991411,1cyclictest0-21swapper/312:07:223
4093929991411,1cyclictest0-21swapper/312:00:173
4093929991411,1cyclictest0-21swapper/311:46:273
4093929991411,1cyclictest0-21swapper/311:46:263
4093929991411,1cyclictest0-21swapper/311:35:263
4093929991411,1cyclictest0-21swapper/311:10:153
4093929991411,1cyclictest0-21swapper/310:45:223
4093929991411,1cyclictest0-21swapper/310:45:013
4093929991411,1cyclictest0-21swapper/310:36:103
4093929991411,1cyclictest0-21swapper/310:31:503
4093929991411,1cyclictest0-21swapper/310:22:273
4093929991411,1cyclictest0-21swapper/310:01:003
4093929991411,1cyclictest0-21swapper/309:26:023
4093929991411,1cyclictest0-21swapper/309:22:533
4093929991411,1cyclictest0-21swapper/308:43:063
4093929991411,1cyclictest0-21swapper/308:19:003
4093929991411,1cyclictest0-21swapper/308:11:143
4093929991411,1cyclictest0-21swapper/307:55:213
4093929991411,1cyclictest0-21swapper/307:35:253
4093929991411,1cyclictest0-21swapper/307:32:463
4093929991411,1cyclictest0-21swapper/307:20:333
4093929991411,1cyclictest0-21swapper/307:11:083
4093929991410,1cyclictest0-21swapper/312:25:193
409392399148,4cyclictest3818356-21systemd-journal07:25:002
4093923991412,1cyclictest382955-21turbostat12:35:012
4093923991411,2cyclictest4191437-21kthreadcore08:25:182
4093923991411,2cyclictest4108441-21munin-plugin-st07:20:012
4093923991411,2cyclictest37964-21unixbench-2d08:55:252
4093923991411,2cyclictest221424-21tr10:55:132
4093923991411,2cyclictest146265-21kworker/u8:1-xprtiod10:30:212
4093923991411,2cyclictest0-21swapper/211:05:182
4093923991410,3cyclictest279998-21kthreadcore11:30:162
4093923991410,2cyclictest0-21swapper/209:45:262
409392099149,4cyclictest3818356-21systemd-journal07:10:021
409392099148,4cyclictest4163701-21grep08:00:241
4093920991411,2cyclictest4166945-21kthreadcore08:05:201
4093920991411,2cyclictest360152-21irqrtprio12:20:201
4093920991411,2cyclictest29395-21kthreadcore08:50:211
4093920991411,2cyclictest12984-21timerandwakeup08:35:251
4093920991411,2cyclictest0-21swapper/112:35:181
4093920991411,2cyclictest0-21swapper/110:45:011
4093920991411,2cyclictest0-21swapper/110:35:231
4093920991410,3cyclictest4142964-21kthreadcore07:45:171
4093920991410,3cyclictest315925-21ntp_kernel_err11:50:241
4093920991410,3cyclictest308352-21unixbench_singl11:45:241
4093920991410,3cyclictest308352-21unixbench_singl11:45:241
4093920991410,2cyclictest92351-21gltestperf09:35:171
409391799147,5cyclictest3818356-21systemd-journal08:55:000
409391799147,5cyclictest3818356-21systemd-journal07:34:330
409391799142,10cyclictest252598-21ssh11:12:020
409391799141,2cyclictest343054-21apt-get12:10:130
4093917991412,1cyclictest4087365-21kworker/0:0+kacpid07:23:290
4093917991412,1cyclictest4087365-21kworker/0:0+kacpid07:15:010
4093917991411,2cyclictest4175845-21timerandwakeup08:10:240
4093917991411,2cyclictest278354-21latency_hist11:30:000
4093917991411,2cyclictest26191-21/usr/sbin/munin08:50:010
4093917991411,2cyclictest229997-21gltestperf11:00:150
4093917991411,2cyclictest170864-21kworker/0:0+kacpid10:26:360
4093917991411,2cyclictest110589-21kthreadcore09:45:180
4093917991410,3cyclictest4133886-21fschecks_count07:40:160
4093917991410,3cyclictest329920-21kthreadcore12:00:190
4093917991410,2cyclictest0-21swapper/011:40:260
4093917991410,2cyclictest0-21swapper/007:15:000
4093929991311,1cyclictest0-21swapper/309:30:063
4093929991310,1cyclictest0-21swapper/311:30:433
4093929991310,1cyclictest0-21swapper/311:16:033
4093929991310,1cyclictest0-21swapper/310:10:403
4093929991310,1cyclictest0-21swapper/309:00:593
4093929991310,1cyclictest0-21swapper/308:27:523
4093929991310,1cyclictest0-21swapper/308:23:033
4093929991310,1cyclictest0-21swapper/308:07:133
4093929991310,1cyclictest0-21swapper/307:25:063
409392399137,4cyclictest3818356-21systemd-journal08:20:002
409392399135,2cyclictest318485-21awk11:55:002
4093923991310,2cyclictest97036-21needreboot09:35:202
4093923991310,2cyclictest60142-21irqcore09:15:162
4093923991310,2cyclictest51499-21cpuspeed_turbos09:10:152
4093923991310,2cyclictest50765-21diskmemload12:05:182
4093923991310,2cyclictest374905-21cron12:30:002
4093923991310,2cyclictest359923-21irqcore12:20:202
4093923991310,2cyclictest351467-21head12:15:162
4093923991310,2cyclictest290974-21kthreadcore11:35:212
4093923991310,2cyclictest189263-21fschecks_time10:35:192
4093923991310,2cyclictest116487-21cpuspeed_turbos09:50:172
4093923991310,2cyclictest116487-21cpuspeed_turbos09:50:162
409392099139,3cyclictest4171057-21irqcore08:10:171
409392099139,3cyclictest251366-21needreboot11:10:211
409392099139,2cyclictest0-21swapper/109:05:181
409392099133,3cyclictest4147722-21cut07:50:211
409392099133,3cyclictest4147722-21cut07:50:201
409392099132,2cyclictest191rcu_preempt10:15:171
4093920991311,1cyclictest23250-21kthreadcore08:45:201
4093920991310,2cyclictest80769-21ntp_kernel_pll_09:25:211
4093920991310,2cyclictest332071-21proc_pri12:00:211
409392099130,1cyclictest0-21swapper/109:40:191
409391799139,3cyclictest4140749-21irqrtprio07:45:150
409391799139,2cyclictest0-21swapper/008:30:010
4093917991310,2cyclictest4120752-21kworker/0:2+kacpid10:00:310
4093917991310,2cyclictest0-21swapper/012:25:000
4093917991310,2cyclictest0-21swapper/007:10:000
409392999129,1cyclictest0-21swapper/311:55:163
409392999129,1cyclictest0-21swapper/308:50:133
4093929991210,1cyclictest0-21swapper/311:50:123
4093929991210,1cyclictest0-21swapper/309:10:033
4093929991210,1cyclictest0-21swapper/308:49:573
4093929991210,1cyclictest0-21swapper/307:50:163
4093929991210,1cyclictest0-21swapper/307:50:163
409392399129,2cyclictest4174196-21kthreadcore08:10:192
409392399129,2cyclictest4157548-21unixbench_multi07:55:222
409392399129,2cyclictest0-21swapper/208:10:012
409392399129,2cyclictest0-21swapper/207:35:162
409392099129,2cyclictest484-21unixbench_multi08:25:241
409392099129,2cyclictest4109336-21gltestperf07:20:171
409392099129,2cyclictest318906-21cpuspeed_turbos11:55:131
409392099129,2cyclictest0-21swapper/108:30:251
409392099129,1cyclictest0-21swapper/112:15:011
409392099123,8cyclictest132167-21awk10:00:001
409392099121,9cyclictest4127159-21munin-run07:35:001
4093920991210,1cyclictest13866-21apt-key08:40:021
4093920991210,1cyclictest117021-21irqcore09:50:191
4093920991210,1cyclictest117021-21irqcore09:50:191
409391799129,2cyclictest51319-21kworker/u8:0-nfsiod09:50:190
409391799129,2cyclictest51319-21kworker/u8:0-nfsiod09:50:190
4093917991210,1cyclictest4185860-21kthreadcore08:20:190
409392999119,1cyclictest0-21swapper/307:45:023
409392399118,2cyclictest524-21rs:main2
409392399118,2cyclictest0-21swapper/208:50:182
409392399117,3cyclictest4126170-21needreboot07:30:202
409392399116,4cyclictest303372-21sed11:45:162
409392399116,4cyclictest303372-21sed11:45:162
409392399116,4cyclictest259739-21sed11:15:242
409392399115,4cyclictest71153-21cut09:20:172
409392399115,4cyclictest4117790-21cut07:25:172
409392399106,2cyclictest4093952-21cat07:10:002
409392099107,2cyclictest4183216-21gltestperf08:20:151
40878892101,6sleep14088038-21df_inode07:05:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional