You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-06 - 08:51
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8.osadl.org (updated Sun Jul 06, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
266188021680,1sleep30-21swapper/300:22:493
22832342790,2sleep12283238-21needreboot20:15:211
2194803992925,3cyclictest2597933-21systemd-run23:45:000
2194806992822,4cyclictest2370341-21systemd-run21:25:001
2194806992724,2cyclictest522-21in:imuxsock23:59:591
2194816992623,1cyclictest0-21swapper/300:35:003
2194810992622,2cyclictest2247059-21cat19:50:002
2194810992622,2cyclictest0-21swapper/219:40:002
2194806992622,3cyclictest2606079-21apt-key23:50:001
2194803992617,7cyclictest2260244-21irqcore20:00:180
2194816992522,1cyclictest0-21swapper/322:35:003
2194816992522,1cyclictest0-21swapper/322:00:013
2194803992522,2cyclictest2259526-21idleruntime-cro20:00:000
2194803992515,8cyclictest2435574-21idleruntime-cro22:05:000
2194816992421,1cyclictest0-21swapper/321:35:003
2194810992419,3cyclictest0-21swapper/222:25:002
2194803992420,3cyclictest3564018-21systemctl21:20:020
2194803992413,9cyclictest2577405-21kthreadcore23:30:200
2194803992411,12cyclictest2520370-21kthreadcore22:55:190
219480399240,15cyclictest2533222-21turbostat23:05:000
2194810992319,2cyclictest3564018-21systemctl20:50:002
2194806992321,1cyclictest2259486-21apt20:00:001
2194806992320,2cyclictest2484510-21cron22:35:011
2194806992319,2cyclictest3561480-21systemd-journal21:45:001
2194803992314,8cyclictest2362677-21tr21:20:130
2194816992218,2cyclictest2278040-21turbostat20:15:013
2194816992218,2cyclictest2278040-21turbostat20:15:003
2194810992217,3cyclictest0-21swapper/222:55:012
2194806992218,3cyclictest514-21systemd-logind21:35:001
2194803992219,2cyclictest2247016-21latency_hist19:50:000
2194803992212,9cyclictest2314171-21ntp_kernel_pll_20:40:230
2194816992118,1cyclictest0-21swapper/323:15:003
2194816992118,1cyclictest0-21swapper/321:45:003
2194810992118,2cyclictest501-21dbus-daemon22:32:142
2194810992118,2cyclictest0-21swapper/219:15:012
2194806992117,3cyclictest2321264-21systemd-run20:50:001
2194806992116,3cyclictest2303080-21cpuspeed_turbos20:35:131
219480399218,11cyclictest2509353-21ssh22:50:130
219480399216,5cyclictest2378905-21grep21:30:130
219480399216,3cyclictest2498373-21/usr/sbin/munin22:40:240
219480399215,4cyclictest2487909-21cut22:35:190
219480399214,5cyclictest1-21systemd20:35:010
2194803992118,2cyclictest3561480-21systemd-journal19:33:080
2194803992118,2cyclictest2427494-21sendmail22:00:000
2194803992118,2cyclictest0-21swapper/000:40:000
2194803992113,7cyclictest2342992-21kthreadcore21:05:190
2194803992112,8cyclictest2375811-21ntp_offset21:25:230
2194803992112,8cyclictest0-21swapper/022:10:280
219480399210,4cyclictest2573268-21runrttasks23:29:220
2194816992018,1cyclictest0-21swapper/319:50:003
2194810992016,2cyclictest0-21swapper/219:45:012
219480399208,10cyclictest2622518-21munin-plugin-st00:00:000
219480399207,12cyclictest2546748-21ntp_kernel_pll_23:10:210
219480399206,12cyclictest501-21dbus-daemon00:35:000
219480399205,6cyclictest2436095-21df_inode22:05:140
2194803992018,1cyclictest0-21swapper/020:25:010
2194803992017,2cyclictest0-21swapper/019:20:010
2194803992016,2cyclictest2643296-21kthreadcore00:10:220
2194803992016,2cyclictest2606030-21apt-key23:50:000
2194803992013,6cyclictest2247743-21gltestperf19:50:150
2194803992012,7cyclictest2214601-21latency19:20:200
2194803992012,7cyclictest0-21swapper/021:45:170
2194803992011,7cyclictest0-21swapper/000:25:240
2194803992010,8cyclictest2457084-21kthreadcore22:15:200
2194803992010,8cyclictest0-21swapper/019:30:000
219480399200,4cyclictest2294072-21tr20:25:230
219481699190,15cyclictest2451723-21sleep22:15:003
2194810991917,1cyclictest0-21swapper/200:00:002
2194810991916,2cyclictest0-21swapper/221:20:012
2194810991915,2cyclictest437876-21kworker/u8:2-events_unbound20:05:012
2194810991915,2cyclictest2550252-21irqcore23:15:162
2194810991915,2cyclictest2411071-21latency_hist21:50:012
2194806991916,2cyclictest761-21cups-browsed22:50:011
2194806991915,2cyclictest0-21swapper/122:20:001
219480399199,9cyclictest2465507-21kthreadcore22:20:200
219480399199,9cyclictest0-21swapper/000:20:150
219480399199,8cyclictest2394747-21cron21:40:000
219480399196,5cyclictest2329055-21cut20:55:180
219480399192,5cyclictest501-21dbus-daemon23:25:000
219480399192,15cyclictest0-21swapper/000:00:560
219480399191,8cyclictest1-21systemd23:40:000
219480399191,5cyclictest2632108-21cut00:05:160
219480399191,5cyclictest1-21systemd19:15:020
219480399191,4cyclictest2194705-21latency_hist19:10:010
2194803991911,7cyclictest2469864-21irqrtprio22:25:170
2194803991910,8cyclictest0-21swapper/021:00:220
219480399190,4cyclictest744-21snmpd23:15:160
2194816991815,1cyclictest0-21swapper/323:50:013
2194816991815,1cyclictest0-21swapper/322:50:213
2194816991815,1cyclictest0-21swapper/322:40:003
2194816991815,1cyclictest0-21swapper/322:10:013
2194816991815,1cyclictest0-21swapper/321:30:003
2194816991815,1cyclictest0-21swapper/320:20:003
2194816991812,3cyclictest2630554-21sleep00:05:003
2194810991815,2cyclictest2254522-21irqrtprio19:55:172
2194810991815,2cyclictest0-21swapper/200:00:272
2194810991814,3cyclictest2452690-21gltestperf22:15:162
2194806991815,2cyclictest2354465-21ssh21:15:131
2194806991814,3cyclictest2644500-21timerandwakeup00:10:261
2194806991814,2cyclictest277-21plymouthd00:05:121
219480399188,9cyclictest2326454-21needreboot20:50:200
219480399186,11cyclictest0-21swapper/021:40:190
219480399182,2cyclictest2351291-21missed_timers21:10:220
219480399181,4cyclictest1-21systemd20:20:000
219480399181,3cyclictest2501258-21apt-get22:45:150
219480399180,6cyclictest2607056-21irqcore23:50:170
219480399180,6cyclictest2419104-21ssh21:55:000
219480399180,5cyclictest2537096-21kthreadcore23:05:190
219480399180,4cyclictest2647106-21sh00:15:130
219480399180,3cyclictest2484548-21/usr/sbin/munin22:35:000
219480399180,2cyclictest2303142-21df20:35:130
24258172170,1sleep10-21swapper/121:56:211
2194816991715,1cyclictest0-21swapper/300:40:003
2194816991714,1cyclictest0-21swapper/323:19:593
2194816991714,1cyclictest0-21swapper/320:20:163
219481699171,2cyclictest2638725-21turbostat00:10:003
2194810991714,2cyclictest2489927-21grep22:35:212
2194810991714,2cyclictest2428989-21irqrtprio22:00:182
2194810991714,2cyclictest2209553-21sendmail19:20:012
2194810991714,2cyclictest0-21swapper/222:15:002
2194810991714,2cyclictest0-21swapper/219:25:002
2194810991713,3cyclictest1-21systemd20:39:252
2194806991714,2cyclictest2451746-21systemd-run22:15:001
2194806991714,2cyclictest2246892-21apt-get19:50:001
2194806991714,2cyclictest0-21swapper/122:30:001
2194806991714,2cyclictest0-21swapper/121:35:181
219480399179,7cyclictest0-21swapper/019:45:000
219480399174,1cyclictest2315516-21grep20:45:150
219480399171,5cyclictest2271851-21latency_hist20:10:000
219480399170,5cyclictest2229255-21sed19:35:160
219480399170,3cyclictest501-21dbus-daemon20:15:010
219480399170,3cyclictest501-21dbus-daemon20:15:000
2194816991613,1cyclictest0-21swapper/321:35:183
2194816991613,1cyclictest0-21swapper/321:15:023
2194816991613,1cyclictest0-21swapper/321:00:133
2194816991613,1cyclictest0-21swapper/320:50:163
2194816991613,1cyclictest0-21swapper/320:05:163
2194816991613,1cyclictest0-21swapper/319:45:003
2194816991613,1cyclictest0-21swapper/319:22:303
219481099163,3cyclictest0-21swapper/220:25:212
2194810991613,2cyclictest501-21dbus-daemon19:55:012
2194810991613,2cyclictest2679101-21latency_hist00:35:012
2194810991613,2cyclictest2660183-21needreboot00:20:202
2194810991613,2cyclictest2644042-21missed_timers00:10:222
2194810991613,2cyclictest2636049-21ntp_kernel_pll_00:05:202
2194810991613,2cyclictest2557911-21tr23:20:132
2194810991613,2cyclictest0-21swapper/223:15:012
2194810991613,2cyclictest0-21swapper/222:50:002
2194810991612,3cyclictest2592573-21diskmemload23:40:222
2194810991612,3cyclictest2351555-21processes21:10:242
2194810991612,2cyclictest2636839-21kworker/u8:0-nfsiod00:25:142
2194806991614,1cyclictest0-21swapper/100:30:001
2194806991613,2cyclictest547-21gdbus19:15:061
2194806991613,2cyclictest2518302-21irqrtprio22:55:161
2194806991613,2cyclictest2377691-21runrttasks21:28:251
2194806991613,2cyclictest2201043-21latency_hist19:15:011
2194806991612,3cyclictest2585209-21kthreadcore23:35:201
2194806991612,3cyclictest2527644-21kthreadcore23:00:191
2194806991612,3cyclictest2346617-21gltestperf21:10:171
2194806991611,4cyclictest2687001-21cron00:40:001
219481699159,3cyclictest2509005-21turbostat22:50:003
2194816991513,1cyclictest0-21swapper/322:30:013
2194816991512,1cyclictest0-21swapper/323:35:003
2194816991512,1cyclictest0-21swapper/323:25:203
2194816991512,1cyclictest0-21swapper/323:05:203
2194816991512,1cyclictest0-21swapper/322:56:273
2194816991512,1cyclictest0-21swapper/322:15:163
2194816991512,1cyclictest0-21swapper/322:05:003
2194816991512,1cyclictest0-21swapper/320:00:203
2194816991512,1cyclictest0-21swapper/319:35:173
2194816991512,1cyclictest0-21swapper/319:10:213
2194816991512,1cyclictest0-21swapper/300:28:143
2194816991512,1cyclictest0-21swapper/300:15:193
2194816991512,1cyclictest0-21swapper/300:11:223
219481099159,4cyclictest2426474-21sshd21:57:532
2194810991513,1cyclictest437876-21kworker/u8:2-rpciod21:25:002
2194810991513,1cyclictest2383862-21needreboot21:30:202
2194810991513,1cyclictest0-21swapper/222:45:022
2194810991512,2cyclictest2373264-21kthreadcore21:25:192
2194810991512,2cyclictest2310570-21kthreadcore20:40:192
2194810991512,2cyclictest2217365-21kthreadcore19:25:182
2194810991512,2cyclictest0-21swapper/221:10:012
2194810991512,1cyclictest0-21swapper/223:50:012
2194810991511,3cyclictest2387278-21gltestperf21:35:152
2194810991511,2cyclictest0-21swapper/221:40:262
2194810991510,4cyclictest3561480-21systemd-journal22:58:272
219481099150,4cyclictest2419141-21idleruntime-cro21:55:002
2194806991513,1cyclictest0-21swapper/120:10:001
2194806991512,2cyclictest2650961-21kthreadcore00:15:191
2194806991512,2cyclictest2541338-21latency_hist23:10:011
2194806991512,2cyclictest2514376-21needreboot22:50:201
2194806991512,2cyclictest2461059-21gltestperf22:20:151
2194806991512,2cyclictest2216055-21cpuspeed_turbos19:25:141
2194806991512,2cyclictest0-21swapper/122:05:151
2194806991511,3cyclictest761-21cups-browsed19:45:011
2194806991511,3cyclictest2609158-21kthreadcore23:50:191
2194806991511,2cyclictest0-21swapper/119:30:201
2194816991411,1cyclictest0-21swapper/323:50:203
2194816991411,1cyclictest0-21swapper/323:41:143
2194816991411,1cyclictest0-21swapper/323:35:193
2194816991411,1cyclictest0-21swapper/323:00:193
2194816991411,1cyclictest0-21swapper/321:50:203
2194816991411,1cyclictest0-21swapper/321:20:233
2194816991411,1cyclictest0-21swapper/321:05:173
2194816991411,1cyclictest0-21swapper/320:25:253
2194816991410,1cyclictest0-21swapper/322:44:333
219481099148,4cyclictest501-21dbus-daemon20:20:002
219481099144,2cyclictest191rcu_preempt00:40:002
2194810991411,2cyclictest386-21systemd-udevd20:12:082
2194810991411,2cyclictest386-21systemd-udevd20:12:082
2194810991411,2cyclictest2526985-21kthreadcore23:00:192
2194810991411,2cyclictest2440167-21kthreadcore22:05:182
2194810991411,2cyclictest2327887-21cut20:55:152
2194810991411,2cyclictest2289650-21systemctl20:20:222
2194810991410,3cyclictest2652095-21cat00:15:202
2194810991410,2cyclictest0-21swapper/223:25:152
2194810991410,2cyclictest0-21swapper/220:35:002
219481099140,3cyclictest2589764-21munin-run23:40:012
219481099140,13cyclictest0-21swapper/223:34:552
2194806991411,2cyclictest2590119-21/usr/sbin/munin23:40:251
2194806991411,2cyclictest2558772-21tr23:20:161
2194806991411,2cyclictest2494587-21kthreadcore22:40:211
2194806991411,2cyclictest2492896-21apt-config22:40:011
2194806991411,2cyclictest2407000-21kthreadcore21:45:181
2194806991411,2cyclictest2339138-21timerandwakeup21:00:221
2194806991411,2cyclictest2264777-21ntp_kernel_err20:00:231
2194806991410,2cyclictest0-21swapper/120:55:011
2194806991410,2cyclictest0-21swapper/120:15:001
2194806991410,2cyclictest0-21swapper/120:15:001
219480699140,1cyclictest0-21swapper/121:05:131
2194816991311,1cyclictest0-21swapper/322:20:173
2194816991311,1cyclictest0-21swapper/321:15:233
2194816991310,1cyclictest0-21swapper/323:55:203
2194816991310,1cyclictest0-21swapper/323:20:173
2194816991310,1cyclictest0-21swapper/320:36:163
2194816991310,1cyclictest0-21swapper/319:50:193
2194816991310,1cyclictest0-21swapper/319:30:183
2194816991310,1cyclictest0-21swapper/319:15:033
219481099138,3cyclictest0-21swapper/223:50:152
2194810991311,1cyclictest2473845-21grep22:25:202
2194810991310,2cyclictest2536564-21kthreadcore23:05:182
219480699139,3cyclictest331ktimers/120:30:131
219480699139,3cyclictest2312116-21kthreadcore20:40:211
219480699139,2cyclictest0-21swapper/122:05:001
2194806991310,2cyclictest2578995-21needreboot23:30:211
2194806991310,2cyclictest2566999-21irqrtprio23:25:181
2194806991310,2cyclictest2546917-21mailstats23:10:231
2194806991310,2cyclictest2345938-21diskmemload00:30:181
2194806991310,2cyclictest0-21swapper/120:55:241
219480699130,12cyclictest2550164-21ssh23:15:151
219481699129,1cyclictest0-21swapper/321:45:173
219481699129,1cyclictest0-21swapper/320:55:153
219481699129,1cyclictest0-21swapper/320:35:003
219481699129,1cyclictest0-21swapper/319:55:173
219481699129,1cyclictest0-21swapper/319:30:003
219481099129,2cyclictest2267373-21kthreadcore20:05:182
219481099129,2cyclictest0-21swapper/219:30:192
219481099123,2cyclictest471rcuog/220:50:152
219480699129,2cyclictest2411725-21gltestperf21:50:151
219480699129,2cyclictest2345938-21diskmemload00:20:181
219480699129,2cyclictest2250658-21kthreadcore19:50:181
219480699128,3cyclictest2215087-21timerandwakeup19:20:231
2194806991210,1cyclictest2626744-21kthreadcore00:00:221
21944912120,2sleep30-21swapper/319:07:313
219481699118,1cyclictest0-21swapper/320:46:203
219480699118,2cyclictest0-21swapper/120:25:191
219480699116,4cyclictest2288870-21tr20:20:181
219480699116,4cyclictest2228476-21grep19:35:131
219481099101,2cyclictest191rcu_preempt21:00:112
21948169997,1cyclictest0-21swapper/320:40:203
21948109990,1cyclictest2194797-21cyclictest19:10:012
21948069991,6cyclictest2194705-21latency_hist19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional