You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-04 - 12:20
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8.osadl.org (updated Fri Jul 04, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9369772360,0chrt936979-21find00:35:120
452799992926,2cyclictest815338-21idleruntime-cro23:20:001
452799992724,2cyclictest644850-21munin-run21:35:001
452799992522,2cyclictest0-21swapper/120:09:591
9097742240,1chrt0-21swapper/300:15:243
452806992421,1cyclictest0-21swapper/322:00:003
452806992421,1cyclictest0-21swapper/319:55:003
452804992421,2cyclictest847975-21idleruntime-cro23:40:002
452794992421,2cyclictest578817-21systemd-run20:50:010
452794992320,2cyclictest710088-21idleruntime-cro22:15:000
452794992320,2cyclictest0-21swapper/021:00:010
452804992219,2cyclictest685598-21sendmail22:00:002
452804992218,3cyclictest642178-21netstat21:30:202
452799992218,2cyclictest0-21swapper/122:29:591
452794992219,2cyclictest636468-21sh21:30:010
452794992218,3cyclictest3564018-21systemctl23:25:010
452806992118,1cyclictest0-21swapper/321:50:013
452806992118,1cyclictest0-21swapper/319:25:013
452804992118,2cyclictest501-21dbus-daemon21:55:002
452799992117,2cyclictest0-21swapper/119:40:011
452799992116,3cyclictest805027-21grep23:10:251
452806992017,1cyclictest0-21swapper/320:05:173
452806992016,1cyclictest0-21swapper/319:45:003
452804992017,2cyclictest823390-21cron23:25:012
452804992017,2cyclictest710066-21munin-plugin-st22:15:002
45279999206,4cyclictest651471-21kworker/u8:3+xprtiod22:15:161
452799992018,1cyclictest791027-21cron23:05:001
452794992017,2cyclictest783054-21latency_hist23:00:000
452794992017,2cyclictest547-21gdbus23:05:010
452794992016,2cyclictest0-21swapper/020:55:000
452794992015,3cyclictest3004-21cron00:14:590
452804991916,2cyclictest864730-21gltestperf23:50:142
452804991915,3cyclictest884896-21kthreadcore00:00:212
452804991915,2cyclictest0-21swapper/223:35:012
452804991915,2cyclictest0-21swapper/220:55:012
452799991915,2cyclictest702730-21irqcore22:10:181
452794991916,2cyclictest504497-21latency_hist19:50:000
452794991916,2cyclictest3561480-21systemd-journal22:30:190
452794991916,2cyclictest0-21swapper/023:20:010
452794991916,2cyclictest0-21swapper/000:20:000
452794991914,4cyclictest880786-21sort00:00:150
4534712180,1sleep0231rcuc/019:10:170
452806991815,1cyclictest0-21swapper/321:10:013
452806991814,2cyclictest677369-21turbostat21:55:003
452806991814,1cyclictest0-21swapper/323:45:013
452806991814,1cyclictest0-21swapper/322:15:183
45280699180,15cyclictest485832-21turbostat19:39:563
45280499187,4cyclictest473450-21cron19:25:002
452804991815,2cyclictest719471-21grep22:20:172
452804991815,2cyclictest0-21swapper/220:15:002
452804991812,4cyclictest453110-21tr19:10:142
452799991816,1cyclictest856073-21cut23:45:011
452799991815,2cyclictest759971-21ps22:45:181
452799991815,1cyclictest0-21swapper/122:00:181
452794991816,1cyclictest529396-21apt-key20:10:000
452794991815,2cyclictest676087-21kworker/0:2+kacpi_notify23:12:450
452794991815,2cyclictest620878-21cut21:20:170
452794991814,3cyclictest669770-21fschecks_time21:50:150
452794991814,3cyclictest514-21systemd-logind20:15:000
452806991714,1cyclictest0-21swapper/321:40:213
452806991714,1cyclictest0-21swapper/320:50:273
452806991712,2cyclictest628416-21turbostat21:25:013
45280499175,1cyclictest471rcuog/221:18:312
452804991715,1cyclictest0-21swapper/221:45:002
452804991714,2cyclictest824493-21irqcore23:25:132
452804991714,2cyclictest783620-21gltestperf23:00:162
452804991714,2cyclictest766754-21latency_hist22:50:012
452804991714,2cyclictest748261-21sort22:35:242
452804991714,2cyclictest605401-21irqrtprio21:10:192
452804991713,3cyclictest853932-21unixbench-2d23:40:262
452804991713,3cyclictest686696-21irqcore22:00:182
452804991713,3cyclictest560642-21cpuspeed_turbos20:35:142
452799991715,1cyclictest636542-21latency_hist21:30:011
452799991714,2cyclictest880135-21munin-plugin-st00:00:001
452799991714,2cyclictest620329-21systemd-run21:20:011
452799991714,2cyclictest561065-21irqcore20:35:181
452799991714,2cyclictest554055-21idleruntime-cro20:29:591
452799991714,2cyclictest522161-21systemctl20:00:261
452799991713,3cyclictest497713-21grep19:40:261
452799991713,2cyclictest0-21swapper/100:20:131
452799991711,4cyclictest605410-21sed21:10:181
45279999170,5cyclictest7676182cut22:50:191
45279499179,2cyclictest3561480-21systemd-journal00:35:000
452794991714,2cyclictest612631-21cpuspeed_turbos21:15:160
452794991713,3cyclictest566469-21latency_hist20:40:010
452794991710,5cyclictest547871-21cron20:25:000
452806991613,1cyclictest0-21swapper/323:46:393
452806991613,1cyclictest0-21swapper/323:30:183
452806991613,1cyclictest0-21swapper/322:01:103
452806991613,1cyclictest0-21swapper/321:10:223
452806991613,1cyclictest0-21swapper/320:35:243
452806991613,1cyclictest0-21swapper/320:15:183
452806991613,1cyclictest0-21swapper/320:15:003
452806991613,1cyclictest0-21swapper/300:30:203
452806991613,1cyclictest0-21swapper/300:27:343
452806991613,1cyclictest0-21swapper/300:20:153
452806991613,1cyclictest0-21swapper/300:06:283
45280499169,5cyclictest926981-21sshd00:26:192
45280499164,5cyclictest44-21ksoftirqd/220:10:002
45280499163,8cyclictest677694-21kworker/u8:0+xprtiod22:25:132
452804991614,1cyclictest912443-21munin-plugin-st00:20:002
452804991613,2cyclictest753918-21kthreadcore22:40:232
452804991613,2cyclictest753918-21kthreadcore22:40:222
452804991613,2cyclictest604079-21diskmemload22:50:202
452804991613,2cyclictest522-21in:imuxsock00:09:552
452804991613,2cyclictest452790-21cyclictest22:06:192
452804991613,2cyclictest1-21systemd23:12:082
452804991613,2cyclictest0-21swapper/223:50:012
452804991613,2cyclictest0-21swapper/220:35:002
452804991611,4cyclictest782903-21cron23:00:002
452799991614,1cyclictest669229-21latency_hist21:50:011
452799991613,2cyclictest897993-21kthreadcore00:10:171
452799991613,2cyclictest864584-21fschecks_count23:50:141
452799991613,2cyclictest742674-21latency_hist22:35:011
452799991613,2cyclictest466424-21proc_pri19:15:221
452799991613,2cyclictest459019-21grep19:15:001
452799991613,2cyclictest0-21swapper/123:25:211
452799991613,2cyclictest0-21swapper/123:20:141
452799991612,3cyclictest571568-21missed_timers20:40:221
45279499165,10cyclictest829114-21ntp_kernel_pll_23:25:190
452794991613,2cyclictest923907-21kthreadcore00:25:190
452794991613,2cyclictest657501-21kthreadcore21:40:210
452794991613,2cyclictest458962-21kworker/0:1+kacpi_notify20:01:130
452794991613,2cyclictest0-21swapper/023:40:010
452794991612,2cyclictest0-21swapper/000:25:000
452806991513,1cyclictest0-21swapper/323:06:533
452806991512,1cyclictest0-21swapper/323:55:143
452806991512,1cyclictest0-21swapper/323:50:123
452806991512,1cyclictest0-21swapper/323:30:003
452806991512,1cyclictest0-21swapper/323:00:193
452806991512,1cyclictest0-21swapper/322:55:213
452806991512,1cyclictest0-21swapper/322:05:183
452806991512,1cyclictest0-21swapper/321:33:193
452806991512,1cyclictest0-21swapper/320:55:193
452806991512,1cyclictest0-21swapper/320:00:013
452806991512,1cyclictest0-21swapper/300:35:143
452806991512,1cyclictest0-21swapper/300:10:133
452806991510,2cyclictest819716-21taskset23:20:203
45280499153,5cyclictest874582-21kthreadcore23:55:172
45280499153,2cyclictest5004642tr19:45:212
452804991512,2cyclictest931990-21kthreadcore00:30:202
452804991512,2cyclictest810528-21kthreadcore23:15:172
452804991512,2cyclictest796637-21sendmail_mailqu23:05:232
452804991512,2cyclictest750-21sshd00:11:172
452804991512,2cyclictest739267-21kthreadcore22:30:202
452804991512,2cyclictest634292-21unixbench-2d21:25:242
452804991512,2cyclictest620964-21grep21:20:172
452804991512,2cyclictest572978-21cpuspeed_turbos20:45:192
452804991512,2cyclictest3564018-21systemctl19:40:012
452804991512,2cyclictest0-21swapper/222:15:142
452804991511,3cyclictest913189-21gltestperf00:20:152
452804991511,2cyclictest0-21swapper/200:35:242
452804991510,4cyclictest479251-21unixbench_singl19:25:282
45279999156,7cyclictest3561480-21systemd-journal00:09:551
45279999156,1cyclictest797083-21ssh23:05:291
452799991513,1cyclictest933726-21kthreadcore00:30:211
452799991513,1cyclictest694374-21gltestperf22:05:141
452799991513,1cyclictest0-21swapper/100:40:001
452799991512,2cyclictest856342-21/usr/sbin/munin23:45:171
452799991512,2cyclictest473725-21apt-get19:25:151
452799991512,2cyclictest0-21swapper/122:20:191
452799991511,2cyclictest541268-21head20:15:281
452799991510,4cyclictest1-21systemd20:55:011
45279499154,5cyclictest231rcuc/022:19:590
452794991513,1cyclictest604079-21diskmemload22:29:300
452794991512,2cyclictest869054-21kthreadcore23:50:170
452794991512,2cyclictest604608-21cpuspeed_turbos21:10:150
452794991512,2cyclictest601621-21kthreadcore21:05:200
452794991512,2cyclictest514-21systemd-logind20:35:010
452794991512,2cyclictest467290-21idleruntime-cro19:20:010
452794991512,2cyclictest0-21swapper/022:40:000
452794991511,3cyclictest832279-21sh23:30:130
452794991511,2cyclictest774813-21turbostat22:55:000
452794991511,2cyclictest604079-21diskmemload21:35:190
452806991412,1cyclictest0-21swapper/321:25:213
452806991412,1cyclictest0-21swapper/320:40:193
452806991411,1cyclictest0-21swapper/323:19:383
452806991411,1cyclictest0-21swapper/323:10:173
452806991411,1cyclictest0-21swapper/322:50:153
452806991411,1cyclictest0-21swapper/322:45:283
452806991411,1cyclictest0-21swapper/322:35:133
452806991411,1cyclictest0-21swapper/322:25:173
452806991411,1cyclictest0-21swapper/320:45:313
45280499148,4cyclictest572599-21cron20:45:002
452804991412,1cyclictest539927-21kthreadcore20:15:202
452804991412,1cyclictest467237-21cron19:20:012
452804991411,2cyclictest665531-21kthreadcore21:45:192
452804991411,2cyclictest546972-21ntp_kernel_err20:20:222
452804991411,2cyclictest524-21rs:main2
452804991411,2cyclictest0-21swapper/221:00:242
45279999144,2cyclictest581rcuop/320:45:191
452799991412,1cyclictest742951-21tr22:35:131
452799991412,1cyclictest0-21swapper/119:20:141
452799991411,2cyclictest922302-21kthreadcore00:25:191
452799991411,2cyclictest883784-21kthreadcore00:00:191
452799991411,2cyclictest679375-21kthreadcore21:55:191
452799991410,3cyclictest604079-21diskmemload00:15:201
452799991410,2cyclictest677694-21kworker/u8:0-xprtiod23:30:181
45279999140,13cyclictest503449-21ntpq19:45:241
45279499144,2cyclictest0-21swapper/020:43:160
452794991412,1cyclictest769-21runrttasks23:09:140
452794991412,1cyclictest676087-21kworker/0:2+kacpid22:44:000
452794991412,1cyclictest676087-21kworker/0:2+kacpid22:44:000
452794991411,2cyclictest849299-21irqrtprio23:40:170
452794991411,2cyclictest689863-21kthreadcore22:00:200
452794991411,2cyclictest541224-21unixbench_multi20:15:280
45279499141,12cyclictest277-21plymouthd19:55:220
452794991411,2cyclictest0-21swapper/019:20:280
452794991410,3cyclictest694863-21irqcore22:05:150
452794991410,3cyclictest485320-21unixbench-2d19:30:270
452806991310,1cyclictest0-21swapper/323:35:163
452806991310,1cyclictest0-21swapper/322:40:393
452806991310,1cyclictest0-21swapper/322:40:383
452806991310,1cyclictest0-21swapper/322:30:203
452806991310,1cyclictest0-21swapper/322:20:183
452806991310,1cyclictest0-21swapper/321:36:093
452806991310,1cyclictest0-21swapper/321:16:133
452806991310,1cyclictest0-21swapper/321:00:213
452806991310,1cyclictest0-21swapper/320:30:173
452806991310,1cyclictest0-21swapper/320:25:183
452806991310,1cyclictest0-21swapper/320:20:163
452806991310,1cyclictest0-21swapper/320:02:183
452806991310,1cyclictest0-21swapper/319:45:193
452806991310,1cyclictest0-21swapper/319:25:153
452806991310,1cyclictest0-21swapper/319:17:253
452806991310,1cyclictest0-21swapper/319:10:153
452806991310,1cyclictest0-21swapper/300:00:183
45280499139,2cyclictest0-21swapper/220:55:152
452804991310,2cyclictest645184-21/usr/sbin/munin21:35:242
452804991310,2cyclictest493667-21kthreadcore19:40:192
452804991310,2cyclictest0-21swapper/220:00:202
45279999139,3cyclictest840397-21fschecks_time23:35:141
45279999139,3cyclictest511783-21irqrtprio19:55:161
45279999137,4cyclictest509665-21ntp_states19:50:231
45279999134,2cyclictest581rcuop/319:30:201
452799991310,2cyclictest780712-21timerwakeupswit22:55:271
452799991310,2cyclictest751803-21irqcore22:40:211
452799991310,2cyclictest751803-21irqcore22:40:211
452799991310,2cyclictest645218-21apt-get21:35:151
452799991310,2cyclictest603166-21ntp_offset21:05:231
452799991310,2cyclictest592249-21gltestperf21:00:191
452799991310,2cyclictest529907-21gltestperf20:10:181
452799991310,2cyclictest0-21swapper/121:50:151
452799991310,2cyclictest0-21swapper/121:40:171
452799991310,2cyclictest0-21swapper/120:25:001
45279499139,3cyclictest890993-21kthreadcore00:05:190
45279499139,3cyclictest644754-21kworker/0:1+kacpi_notify21:35:010
45279499139,2cyclictest0-21swapper/022:45:230
45279499137,4cyclictest3561480-21systemd-journal23:59:590
45279499133,9cyclictest663200-21kthreadcore21:45:170
45280699128,1cyclictest0-21swapper/319:30:143
45280499129,2cyclictest552365-21kthreadcore20:25:192
45280499129,2cyclictest0-21swapper/219:50:162
45279999129,2cyclictest624802-21kthreadcore21:20:211
45279999129,2cyclictest586852-21irqrtprio20:55:181
45279499129,2cyclictest651471-21kworker/u8:3-rpciod21:55:160
45279499129,2cyclictest595318-21kthreadcore21:00:230
45279499129,2cyclictest510089-21unixbench_singl19:50:250
45279499128,3cyclictest486372-21gltestperf19:35:160
45279499126,4cyclictest478789-21ntp_states19:25:250
45279499125,5cyclictest64450irq/159-eno2:rx-023:45:180
452794991210,1cyclictest452790-21cyclictest22:20:180
45280699119,1cyclictest0-21swapper/322:10:053
45280499117,3cyclictest480362-21interrupts19:30:172
45280499115,4cyclictest513887-21cut19:55:182
45279999118,2cyclictest0-21swapper/120:30:181
45279499118,2cyclictest552431-21kthreadcore20:25:200
45279499110,2cyclictest1-21systemd19:45:010
4528049995,2cyclictest0-21swapper/219:10:012
4527949991,6cyclictest452795-21latency_hist19:10:010
4527999981,4cyclictest452801-21latency_hist19:10:011
4528069964,1cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional