You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 09:55
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8.osadl.org (updated Wed Sep 17, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
299170921010,1sleep10-21swapper/120:35:201
31691282290,1chrt0-21swapper/222:39:582
2879663992620,5cyclictest3110362-21sendmail22:00:000
2879667992319,3cyclictest514-21systemd-logind19:25:011
2879667992218,3cyclictest524-21rs:main1
2879663992219,2cyclictest3282229-21kworker/u8:3-rpciod00:00:000
2879663992216,5cyclictest0-21swapper/020:45:000
2879663992215,6cyclictest2918876-21latency_hist19:40:010
2879663992215,5cyclictest3153500-21cpuspeed_turbos22:30:120
2879663992215,5cyclictest3153500-21cpuspeed_turbos22:30:120
2879663992215,5cyclictest2924906-21latency_hist19:45:000
2879667992117,3cyclictest2967898-21cron20:20:001
2879663992114,6cyclictest3363770-21munin-plugin-st00:40:000
2879663992114,6cyclictest3363770-21munin-plugin-st00:40:000
29437162200,1chrt0-21swapper/120:00:131
2879669992017,2cyclictest0-21swapper/221:44:592
2879669992016,3cyclictest522-21in:imuxsock19:20:002
2879667992017,2cyclictest3011404-21latency_hist20:55:011
2879667992017,2cyclictest0-21swapper/121:40:001
2879667992016,2cyclictest0-21swapper/121:10:001
2879663992017,2cyclictest3234550-21sh23:20:000
2879663992015,4cyclictest0-21swapper/020:14:550
2879663992013,6cyclictest3099750-21sendmail_mailqu21:50:230
2879663992013,6cyclictest3065362-21kthreadcore21:30:190
2879663992012,6cyclictest2981855-21kernelversion20:30:160
2879663992012,6cyclictest2981855-21kernelversion20:30:160
2879673991916,1cyclictest0-21swapper/322:00:013
2879667991915,3cyclictest2979319-21ntp_kernel_err20:25:221
2879667991914,3cyclictest3274782-21idleruntime-cro23:44:591
287966399197,5cyclictest0-21swapper/023:00:010
2879663991914,3cyclictest3355611-21apt-key00:35:010
2879663991914,3cyclictest3355611-21apt-key00:35:000
2879663991913,5cyclictest3161841-21tail22:35:170
2879663991913,5cyclictest3110907-21fschecks_time22:00:160
2879663991912,6cyclictest3059332-21ntp_kernel_err21:25:240
2879663991911,7cyclictest3267549-21gltestperf23:40:180
2879663991911,6cyclictest3226218-21turbostat23:15:000
2879669991814,2cyclictest0-21swapper/223:00:002
2879667991815,2cyclictest3029685-21diskmemload22:35:241
2879667991815,2cyclictest2942324-21needreboot19:55:231
2879667991814,3cyclictest3032798-21kthreadcore21:10:181
2879663991813,4cyclictest3085944-21kworker/0:2+kacpid22:18:460
2879663991812,5cyclictest3315166-21turbostat.cron00:14:550
2879663991812,5cyclictest3215096-21kworker/0:0+kacpid00:25:300
2879663991812,5cyclictest3005600-21fschecks_time20:50:150
2879663991812,5cyclictest0-21swapper/022:45:190
2879663991812,5cyclictest0-21swapper/022:45:010
2879663991811,6cyclictest3215096-21kworker/0:0+kacpid00:05:550
2879663991811,6cyclictest2855575-21kworker/0:0+kacpid20:25:180
2879663991811,5cyclictest2943760-21fschecks_count20:00:140
2879663991811,5cyclictest2943330-21cat20:00:000
2879663991811,5cyclictest0-21swapper/000:15:150
2879663991810,7cyclictest3045822-21turbostat21:20:010
287967399175,9cyclictest32506142sleep323:29:513
2879673991711,3cyclictest2998713-21turbostat20:45:013
2879673991711,3cyclictest28943292sleep319:20:133
2879669991714,2cyclictest41487-21kworker/u8:1-xprtiod21:10:222
2879669991714,2cyclictest3818356-21systemd-journal00:25:002
2879669991714,2cyclictest3315162-21latency_hist00:10:012
2879669991714,2cyclictest3252026-21sed23:30:182
2879669991714,2cyclictest0-21swapper/220:45:012
2879669991714,2cyclictest0-21swapper/220:25:002
287966999170,4cyclictest31308321kthreadcore22:15:182
2879667991714,2cyclictest500-21avahi-daemon20:25:001
2879667991714,2cyclictest3307126-21latency_hist00:05:001
2879667991714,2cyclictest3266840-21sendmail23:39:591
2879667991714,2cyclictest0-21swapper/123:10:191
2879667991714,2cyclictest0-21swapper/122:55:011
2879667991713,2cyclictest0-21swapper/122:00:131
2879667991713,2cyclictest0-21swapper/119:50:241
287966799170,3cyclictest3299021-21dump-pmu-power00:00:011
287966399174,5cyclictest3000486-21cut20:45:160
2879663991714,2cyclictest3138651-21kthreadcore22:20:160
2879663991714,2cyclictest2967939-21sendmail20:20:010
2879663991712,4cyclictest3086161-21latency_hist21:45:000
2879663991711,5cyclictest3286275-21kthreadcore23:50:190
2879663991711,5cyclictest3190968-21needreboot22:50:210
287966399171,15cyclictest2888116-21/usr/sbin/munin19:15:260
2879663991711,5cyclictest2855575-21kworker/0:0+kacpid21:45:050
2879663991711,5cyclictest2855575-21kworker/0:0+kacpid19:25:540
2879663991711,5cyclictest0-21swapper/023:09:380
2879663991711,5cyclictest0-21swapper/023:05:010
2879663991711,5cyclictest0-21swapper/000:00:210
2879663991711,4cyclictest3215096-21kworker/0:0+kacpid23:25:110
2879663991710,6cyclictest3334218-21kthreadcore00:20:200
2879663991710,6cyclictest3264382-21switchtime23:35:240
287966399170,5cyclictest3235030-21tr23:20:130
287967399166,7cyclictest30371632sleep321:13:433
287967399165,8cyclictest29609712sleep320:10:233
2879669991613,2cyclictest3331260-21idleruntime-cro00:20:002
2879669991613,2cyclictest3276538-21kthreadcore23:45:152
2879669991613,2cyclictest2944012-21irqcore20:00:172
2879669991612,3cyclictest3188922-21kthreadcore22:50:182
2879669991612,3cyclictest3075895-21unixbench_multi21:35:252
2879669991611,2cyclictest3257652-21kworker/u8:2-xprtiod00:15:002
287966999160,4cyclictest29410951kthreadcore19:55:222
287966799167,6cyclictest547-21gdbus23:00:001
2879667991613,2cyclictest3178855-21irqrtprio22:45:201
2879667991613,2cyclictest3103094-21sed21:55:151
2879667991613,2cyclictest3029685-21diskmemload23:45:221
2879667991613,2cyclictest2889475-21irqrtprio19:15:181
2879667991613,2cyclictest0-21swapper/122:45:011
2879667991612,3cyclictest3344737-21ntp_kernel_err00:25:221
2879667991612,3cyclictest1-21systemd00:33:461
2879667991612,3cyclictest1-21systemd00:33:451
2879667991611,3cyclictest3142770-21grep22:20:241
287966799160,3cyclictest3288825-21grep23:50:251
287966799160,14cyclictest2961797-21awk20:15:001
287966399169,6cyclictest3078123-21sendmail21:40:000
287966399164,10cyclictest3023802-21sort21:05:130
2879663991613,2cyclictest3252236-21irqrtprio23:30:180
2879663991610,5cyclictest3031179-21irqrtprio21:10:160
2879663991610,5cyclictest2925322-21grep19:45:140
2879663991610,5cyclictest2855575-21kworker/0:0+kacpi_notify20:20:170
287966399160,4cyclictest3047399-21sed21:20:170
287966399160,3cyclictest0-21swapper/019:20:140
287967399155,7cyclictest31362492sleep322:18:363
287967399155,7cyclictest30233252sleep321:02:253
287967399154,8cyclictest30171762sleep320:57:393
2879673991512,1cyclictest0-21swapper/322:30:133
2879673991512,1cyclictest0-21swapper/322:30:123
287966999159,4cyclictest2918627-21sshd19:39:452
2879669991512,2cyclictest3284212-21irqrtprio23:50:182
2879669991512,2cyclictest3267295-21tr23:40:162
2879669991512,2cyclictest3264652-21unixbench_singl23:35:252
2879669991512,2cyclictest3246139-21kthreadcore23:25:182
2879669991512,2cyclictest3242598-21sh23:25:012
2879669991512,2cyclictest0-21swapper/221:55:232
2879669991512,2cyclictest0-21swapper/219:45:012
2879669991512,2cyclictest0-21swapper/219:35:012
2879669991511,3cyclictest3059859-21unixbench_singl21:25:292
287966799158,5cyclictest2998705-21munin-run20:45:001
287966799152,7cyclictest191rcu_preempt19:30:001
2879667991512,2cyclictest514-21systemd-logind23:35:001
2879667991512,2cyclictest3328878-21switchtime00:15:241
2879667991512,2cyclictest3043902-21kworker/u8:2-rpciod21:50:001
2879667991512,2cyclictest2986340-21sshd20:34:461
2879667991512,2cyclictest2986340-21sshd20:34:451
2879667991512,2cyclictest0-21swapper/123:15:171
2879667991511,3cyclictest3236199-21irqrtprio23:20:171
2879667991511,3cyclictest3099655-21ntp_offset21:50:201
2879667991511,3cyclictest3029685-21diskmemload21:20:171
2879667991511,2cyclictest3018499-21irqcore21:00:181
2879667991511,2cyclictest0-21swapper/122:25:231
2879667991511,2cyclictest0-21swapper/122:15:181
2879667991510,3cyclictest0-21swapper/119:15:001
287966399157,7cyclictest0-21swapper/020:05:150
287966399153,5cyclictest769-21runrttasks23:47:270
2879663991510,4cyclictest2855575-21kworker/0:0+kacpid20:38:020
287967399147,4cyclictest2986392-21turbostat20:39:553
287967399146,5cyclictest30522442sleep321:21:243
287967399145,6cyclictest33459062sleep300:26:513
287967399145,6cyclictest29061712sleep319:27:463
287967399144,7cyclictest33308282sleep300:19:103
287967399141,9cyclictest29101882sleep319:30:233
2879673991412,1cyclictest0-21swapper/319:40:013
287967399141,10cyclictest30859022sleep321:44:293
287966999149,4cyclictest3818356-21systemd-journal20:49:552
287966999149,4cyclictest2992414-21sshd20:38:452
2879669991411,2cyclictest3296768-21timerwakeupswit23:55:242
2879669991411,2cyclictest3202487-21cpuspeed_turbos23:00:122
2879669991411,2cyclictest3158112-21kthreadcore22:30:192
2879669991411,2cyclictest3158112-21kthreadcore22:30:182
2879669991411,2cyclictest2964998-21tr20:15:202
2879669991410,3cyclictest3047579-21kernelversion21:20:172
2879669991410,3cyclictest2974808-21tr20:25:172
2879669991410,2cyclictest0-21swapper/222:20:242
287966799149,3cyclictest0-21swapper/119:40:011
287966799148,4cyclictest3363747-21cron00:40:011
287966799148,4cyclictest3363747-21cron00:40:001
2879667991411,2cyclictest3058097-21kthreadcore21:25:231
2879667991411,2cyclictest3029685-21diskmemload22:30:201
2879667991411,2cyclictest3029685-21diskmemload22:30:201
2879667991411,2cyclictest0-21swapper/100:05:451
2879667991410,2cyclictest3242626-21kworker/u8:3-xprtiod23:30:011
2879667991410,2cyclictest0-21swapper/121:30:231
287966399149,4cyclictest3017869-21fschecks_count21:00:160
287966399148,5cyclictest3146765-21kthreadcore22:25:200
287966399148,5cyclictest2882813-21kthreadcore19:10:190
287966399148,4cyclictest0-21swapper/019:10:010
287966399147,6cyclictest64450irq/159-eno2:rx-020:55:220
287966399147,6cyclictest3085944-21kworker/0:2+kacpi_notify22:05:410
287966399146,6cyclictest0-21swapper/019:50:150
287966399145,7cyclictest3818356-21systemd-journal19:30:020
287966399141,11cyclictest3121178-21apt-get22:10:100
287967399139,2cyclictest30452502sleep321:18:443
287967399137,3cyclictest31518002sleep322:27:143
287967399137,2cyclictest0-21swapper/300:00:213
287967399135,5cyclictest3266802-21turbostat23:44:543
287967399134,2cyclictest31926342sleep322:52:473
287967399133,8cyclictest2943288-21turbostat20:04:563
287967399131,9cyclictest2885782-21sleep19:15:003
287967399131,8cyclictest30542602sleep321:25:173
2879673991311,1cyclictest0-21swapper/319:45:003
2879673991310,1cyclictest0-21swapper/322:39:053
2879673991310,1cyclictest0-21swapper/322:00:153
2879673991310,1cyclictest0-21swapper/321:30:193
2879673991310,1cyclictest0-21swapper/300:40:003
2879673991310,1cyclictest0-21swapper/300:40:003
287966999139,3cyclictest3124152-21kthreadcore22:10:152
287966999138,3cyclictest0-21swapper/223:15:172
287966999137,4cyclictest547-21gdbus00:30:002
2879669991311,1cyclictest3040221-21kthreadcore21:15:182
2879669991311,1cyclictest2897292-21kthreadcore19:20:192
2879669991310,2cyclictest3350210-21kthreadcore00:30:182
2879669991310,2cyclictest3350210-21kthreadcore00:30:182
2879669991310,2cyclictest3223726-21processes23:10:262
2879669991310,2cyclictest3150440-21netstat22:25:222
2879669991310,2cyclictest3095957-21kthreadcore21:50:172
2879669991310,2cyclictest3065655-21kthreadcore21:30:192
2879669991310,2cyclictest3029685-21diskmemload23:06:302
2879669991310,2cyclictest2930082-21ntp_kernel_pll_19:45:202
2879669991310,2cyclictest2900758-21gltestperf19:25:122
2879669991310,2cyclictest2881191-21irqrtprio19:10:172
2879669991310,2cyclictest0-21swapper/222:45:012
287966799139,3cyclictest3214634-21kthreadcore23:05:161
287966799137,4cyclictest3818356-21systemd-journal00:10:081
287966799137,4cyclictest3044379-21sshd21:16:461
287966799137,4cyclictest2928338-21tr19:45:171
287966799134,8cyclictest3005105-21awk20:50:001
287966799132,2cyclictest471rcuog/220:07:101
287966799131,3cyclictest2924842-21munin-run19:45:011
2879667991310,2cyclictest3207367-21needreboot23:00:201
2879667991310,2cyclictest0-21swapper/122:10:131
2879667991310,2cyclictest0-21swapper/121:40:161
287966799130,4cyclictest3331937-21gltestperf00:20:161
287967399129,1cyclictest0-21swapper/323:58:303
287967399129,1cyclictest0-21swapper/322:45:003
287967399129,1cyclictest0-21swapper/321:50:223
287967399128,2cyclictest3336718-21taskset00:20:243
287967399128,2cyclictest0-21swapper/323:15:003
287967399128,1cyclictest0-21swapper/323:35:233
287967399128,1cyclictest0-21swapper/320:30:163
287967399128,1cyclictest0-21swapper/320:30:163
287967399127,2cyclictest32329522sleep323:16:283
287967399127,2cyclictest32093332sleep323:03:213
287967399127,2cyclictest31778542sleep322:45:143
287967399127,2cyclictest29368022sleep319:51:073
287967399126,3cyclictest29308412sleep319:48:263
287967399125,4cyclictest32399022sleep323:20:223
287967399124,5cyclictest2937677-21taskset19:55:183
287967399122,2cyclictest29681742sleep320:20:123
287967399121,8cyclictest30922942sleep321:45:503
287966999129,2cyclictest3144794-21kworker/u8:0-rpciod22:40:162
287966999129,2cyclictest3114975-21kthreadcore22:00:192
287966999129,2cyclictest3086580-21cpuspeed_turbos21:45:142
287966999129,2cyclictest3023788-21/usr/sbin/munin21:05:182
287966999129,2cyclictest2980613-21cpuspeed_turbos20:30:132
287966999129,2cyclictest2980613-21cpuspeed_turbos20:30:122
287966999129,2cyclictest2955976-21tr20:10:132
287966999129,2cyclictest0-21swapper/220:55:252
287966999128,2cyclictest0-21swapper/220:55:012
287966999124,2cyclictest471rcuog/219:50:232
2879669991210,1cyclictest2951226-21kthreadcore20:05:212
2879669991210,1cyclictest0-21swapper/200:35:162
2879669991210,1cyclictest0-21swapper/200:35:152
287966799121,3cyclictest3017372-21idleruntime-cro21:00:001
31187362110,1sleep30-21swapper/322:05:213
287967399118,1cyclictest0-21swapper/323:50:193
287967399118,1cyclictest0-21swapper/320:15:183
287967399118,1cyclictest0-21swapper/300:14:553
287967399117,2cyclictest3193917-21taskset22:55:033
287967399117,2cyclictest31444952sleep322:23:533
287967399117,2cyclictest2955388-21taskset20:07:173
287967399116,2cyclictest32104192sleep323:05:103
287967399114,5cyclictest3313860-21taskset00:07:113
287967399113,5cyclictest3005382-21chrt20:50:143
287967399110,8cyclictest33476282sleep300:30:113
287967399110,8cyclictest33476282sleep300:30:113
287966999118,2cyclictest3302057-21kthreadcore00:00:182
287966999118,2cyclictest3017977-21gltestperf21:00:172
287966799110,10cyclictest0-21swapper/122:06:391
287967399107,1cyclictest0-21swapper/323:45:173
287967399107,1cyclictest0-21swapper/322:10:123
287967399107,1cyclictest0-21swapper/320:25:173
287967399106,2cyclictest30295072sleep321:07:493
287967399106,2cyclictest3004759-21taskset20:45:483
287967399105,2cyclictest32571562sleep323:31:363
287967399104,4cyclictest3074234-21chrt21:35:193
287966999108,1cyclictest0-21swapper/222:06:242
28796739995,1cyclictest0-21swapper/319:15:163
28796699984,3cyclictest2879787-21cat19:10:022
28796679981,5cyclictest2879555-21latency_hist19:10:011
2879221280,3sleep30-21swapper/319:06:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional