You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-14 - 10:07
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6.osadl.org (updated Sun Sep 14, 2025 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
394456699121114,5cyclictest4041960-21kworker/u8:0+events_unbound00:15:011
394456699112100,9cyclictest3968933-21latency_hist20:25:001
39445639910594,9cyclictest3969187-21apt-get20:25:120
39445639910190,8cyclictest4014090-21kworker/u8:3+flush-179:023:20:000
3944566999989,7cyclictest3950881-21latency_hist19:30:021
3944566999892,4cyclictest4014090-21kworker/u8:3+events_unbound23:00:261
3944566999885,10cyclictest4038230-40mandb00:00:001
3944563999890,6cyclictest4014034-21kworker/u8:1+flush-179:022:45:220
3944563999890,6cyclictest4014034-21kworker/u8:1+flush-179:022:45:210
3944566999786,8cyclictest3946095-21latency_hist19:15:021
3944563999785,9cyclictest3985108-21latency_hist21:15:010
3944566999584,8cyclictest3968354-21kworker/u8:1+flush-179:021:30:251
3944563999584,9cyclictest3962300-21latency_hist20:05:010
3944563999487,5cyclictest3968354-21kworker/u8:1+flush-179:021:40:010
3944563999487,5cyclictest3954045-21kworker/u8:0+events_unbound19:45:000
3944566999384,6cyclictest3999715-21kworker/u8:3+flush-179:022:10:231
3944563999385,6cyclictest4031571-21apt-get23:40:000
3944566999180,8cyclictest3959082-21kworker/u8:1+flush-179:020:05:011
3944566998882,4cyclictest4014090-21kworker/u8:3+flush-179:023:25:231
3944566998878,7cyclictest4002898-21latency_hist22:10:001
3944563998679,5cyclictest4041960-21kworker/u8:0+flush-179:000:15:010
3944566998476,6cyclictest4038267-21kworker/u8:2+flush-179:000:05:261
3944563998474,8cyclictest3972110-21latency_hist20:35:010
3944566998276,4cyclictest3939978-21kworker/u8:1+events_unbound19:10:011
3944563998273,7cyclictest3968354-21kworker/u8:1+flush-179:021:15:220
3944566998175,5cyclictest3968354-21kworker/u8:1+flush-179:021:40:231
3944566998175,4cyclictest3986668-21kworker/u8:0+flush-179:021:35:231
3944566998170,8cyclictest3999710-21latency_hist22:00:021
3944566998170,8cyclictest3999710-21latency_hist22:00:021
3944566997972,5cyclictest4007718-21kworker/u8:3+flush-179:022:30:241
3944563997971,6cyclictest3991484-21latency_hist21:35:010
3944566997871,5cyclictest3959082-21kworker/u8:1+flush-179:020:05:301
3944563997870,5cyclictest4014090-21kworker/u8:3+events_unbound23:20:270
3944563997866,9cyclictest4020427-21latency_hist23:05:010
3944563997768,7cyclictest3999715-21kworker/u8:3+flush-179:022:10:000
3944566997668,6cyclictest3998087-21kworker/u8:0+flush-179:022:40:231
3944563997668,6cyclictest3968354-21kworker/u8:1+events_unbound21:45:240
3944563997666,7cyclictest3944524-21kworker/u8:3+flush-179:019:15:180
3944566997569,5cyclictest3968354-21kworker/u8:1+flush-179:020:54:551
3944563997568,5cyclictest3954045-21kworker/u8:0+events_unbound19:50:220
3944563997564,9cyclictest4051551-21latency_hist00:40:000
3944566997468,4cyclictest4014090-21kworker/u8:3+flush-179:023:15:191
3944566997467,5cyclictest3944521-21kworker/u8:0+flush-179:019:20:131
3944563997468,4cyclictest3959082-21kworker/u8:1+flush-179:020:10:190
3944563997466,6cyclictest3968354-21kworker/u8:1+events_unbound21:25:240
3944566997367,4cyclictest3968354-21kworker/u8:1+flush-179:020:35:001
3944563997367,4cyclictest3956479-21kworker/u8:2+flush-179:019:55:230
3944563997367,4cyclictest3956479-21kworker/u8:2+flush-179:019:55:230
3944563997363,7cyclictest3988308-21latency_hist21:25:010
3944563997363,4cyclictest3959135-21kworker/u8:3+events_unbound20:05:180
3944563997362,9cyclictest3999958-21/usr/sbin/munin22:00:260
3944566997266,4cyclictest4041960-21kworker/u8:0+flush-179:000:15:211
3944563997268,2cyclictest3963108-21kworker/u8:0+flush-179:020:20:210
3944563997265,5cyclictest4041960-21kworker/u8:0+flush-179:000:15:180
3944563997265,5cyclictest4014090-21kworker/u8:3+events_unbound23:25:150
3944563997166,4cyclictest4028422-21kworker/u8:2+events_unbound23:34:590
3944563997165,4cyclictest3956479-21kworker/u8:2+flush-179:019:45:160
3944563997161,8cyclictest601706-21systemd-journal23:43:130
3944566997063,5cyclictest3998087-21kworker/u8:0+flush-179:022:15:341
3944563997063,6cyclictest3968354-21kworker/u8:1+flush-179:021:40:210
3944563997063,5cyclictest4014090-21kworker/u8:3+flush-179:023:50:240
3944563996964,4cyclictest4014090-21kworker/u8:3+flush-179:023:09:560
3944563996962,5cyclictest601706-21systemd-journal20:37:480
3944563996961,6cyclictest4009272-21latency_hist22:30:000
3944566996863,4cyclictest4048399-21kworker/u8:0+flush-179:000:35:001
3944566996861,5cyclictest3996763-21/usr/sbin/munin21:50:261
3944563996857,9cyclictest3949530-21/usr/sbin/munin19:25:260
3944563996855,10cyclictest3978994-21apt-get20:55:140
3944566996761,5cyclictest4014090-21kworker/u8:3+flush-179:023:34:561
3944566996761,5cyclictest3965768-21kworker/u8:3+events_unbound20:15:171
3944566996761,4cyclictest4045341-21kworker/u8:3+flush-179:000:25:191
3944566996761,4cyclictest3968354-21kworker/u8:1+flush-179:021:00:271
3944566996760,5cyclictest3968354-21kworker/u8:1+flush-179:020:55:191
3944566996759,6cyclictest3944524-21kworker/u8:3+flush-179:019:35:241
3944566996757,8cyclictest601706-21systemd-journal22:05:001
3944563996762,4cyclictest4014090-21kworker/u8:3+flush-179:022:58:090
3944563996756,8cyclictest4046769-21latency_hist00:25:000
3944566996660,5cyclictest3986668-21kworker/u8:0+flush-179:021:24:551
3944566996657,7cyclictest4015644-21latency_hist22:50:021
3944566996657,7cyclictest4015644-21latency_hist22:50:021
3944566996656,8cyclictest4038482-21/usr/sbin/munin00:00:171
3944566996556,7cyclictest4007683-21latency_hist22:25:001
3944563996559,4cyclictest3968354-21kworker/u8:1+flush-179:020:40:240
3944563996555,6cyclictest3999715-21kworker/u8:3+flush-179:022:10:200
3944566996454,7cyclictest3989896-21latency_hist21:30:011
3944563996457,5cyclictest4034995-21kworker/u8:0+flush-179:000:00:220
3944566996359,2cyclictest4025468-21apt-get23:20:141
3944566996356,5cyclictest4044620-21kworker/u8:1+flush-179:000:20:231
3944563996358,4cyclictest3954045-21kworker/u8:0+flush-179:019:38:350
3944563996257,4cyclictest3968354-21kworker/u8:1+flush-179:021:50:240
3944563996256,4cyclictest4014090-21kworker/u8:3+flush-179:022:50:240
3944566996154,5cyclictest4014090-21kworker/u8:3+events_unbound22:55:211
3944566996152,7cyclictest3968354-21kworker/u8:1+events_unbound20:40:001
3944563996156,4cyclictest4045341-21kworker/u8:3+flush-179:000:25:310
3944566995952,5cyclictest3968354-21kworker/u8:1+flush-179:020:40:251
3944566995945,12cyclictest4014090-21kworker/u8:3+events_unbound23:40:021
3944563995953,4cyclictest4028363-21kworker/u8:1+flush-179:023:45:240
3944566995852,5cyclictest3959082-21kworker/u8:1+flush-179:020:10:141
3944563995852,5cyclictest4007718-21kworker/u8:3+events_unbound22:30:250
3944563995751,4cyclictest4007659-21kworker/u8:2+flush-179:022:35:170
3944563995750,5cyclictest4034995-21kworker/u8:0+flush-179:023:55:230
3944566995650,5cyclictest4007659-21kworker/u8:2+flush-179:022:25:281
3944566995650,4cyclictest3954045-21kworker/u8:0+flush-179:019:40:211
3944563995650,5cyclictest3977124-21kworker/u8:0+events_unbound21:00:240
3944566995550,4cyclictest4050192-21kworker/u8:1+flush-179:000:35:251
3944566995550,4cyclictest3970565-21kworker/u8:3+events_unbound20:45:281
3944566995550,4cyclictest3944521-21kworker/u8:0+flush-179:019:19:561
3944563995545,8cyclictest4048619-21apt-get00:30:140
3944566995448,5cyclictest4007659-21kworker/u8:2+flush-179:022:40:011
3944563995449,4cyclictest4005538-21kworker/u8:1+events_unbound22:20:250
3944563995448,4cyclictest3998087-21kworker/u8:0+flush-179:021:55:190
3944563995448,4cyclictest3998087-21kworker/u8:0+flush-179:021:55:190
3944566995348,4cyclictest4028363-21kworker/u8:1+flush-179:023:45:201
3944566995346,5cyclictest3950917-21kworker/u8:1+flush-179:019:45:261
3944563995347,4cyclictest3970565-21kworker/u8:3+events_unbound20:45:250
3944566995247,4cyclictest3977124-21kworker/u8:0+flush-179:021:05:351
3944566995247,4cyclictest3956479-21kworker/u8:2+flush-179:019:55:191
3944566995247,4cyclictest3956479-21kworker/u8:2+flush-179:019:55:181
3944566995246,5cyclictest3980392-21kworker/u8:3+events_unbound21:10:151
3944566995246,4cyclictest4014090-21kworker/u8:3+events_unbound23:35:251
3944566995044,5cyclictest3944524-21kworker/u8:3+flush-179:019:30:251
3944563995045,4cyclictest4014090-21kworker/u8:3+flush-179:023:10:260
4505291490,13ptp4l441ktimers/319:10:013
3944566994942,6cyclictest3968354-21kworker/u8:1+events_unbound20:25:231
3944566994830,16cyclictest4017246-21apt22:54:591
3944563994843,4cyclictest3965768-21kworker/u8:3+events_unbound20:15:230
3944563994742,4cyclictest3944521-21kworker/u8:0+events_unbound19:10:130
3944569994641,2cyclictest0-21swapper/220:56:592
3944563994637,7cyclictest601706-21systemd-journal22:41:340
3944566994536,7cyclictest601706-21systemd-journal23:50:021
3944563994539,5cyclictest3977124-21kworker/u8:0+events_unbound21:05:190
3944566994439,4cyclictest3954045-21kworker/u8:0+flush-179:019:50:261
3944563994437,5cyclictest3998087-21kworker/u8:0+flush-179:022:15:150
394457199430,40cyclictest3981943-21turbostat21:05:003
3944571994234,6cyclictest3983539-21turbostat21:10:013
3944566994233,7cyclictest4022015-21latency_hist23:10:011
3944563994237,4cyclictest3977124-21kworker/u8:0+events_unbound20:50:220
3944571994139,1cyclictest0-21swapper/322:29:593
394457199410,38cyclictest4014061-21turbostat22:45:013
3944566994138,2cyclictest4014034-21kworker/u8:1+flush-179:023:10:161
394457199400,4cyclictest4002910-21turbostat22:10:003
394457199400,4cyclictest4001325-21turbostat22:05:003
394457199400,38cyclictest4028397-21turbostat23:30:013
394456999400,35cyclictest0-21swapper/220:29:552
394457199390,4cyclictest3978762-21turbostat20:55:003
3944569993935,2cyclictest0-21swapper/222:37:592
3944566993934,4cyclictest3968354-21kworker/u8:1+flush-179:021:45:201
3944569993834,2cyclictest0-21swapper/220:30:592
394456999380,36cyclictest0-21swapper/223:46:002
394456999380,36cyclictest0-21swapper/222:19:582
3944563993833,4cyclictest4038267-21kworker/u8:2+events_unbound00:05:240
3944563993827,9cyclictest3951138-21apt-get19:30:160
3944563993826,10cyclictest3949290-21cron19:25:000
394457199370,36cyclictest0-21swapper/321:26:593
3944569993734,2cyclictest0-21swapper/200:35:012
3944569993733,2cyclictest0-21swapper/219:36:592
394456999370,2cyclictest0-21swapper/220:42:592
3944566993731,5cyclictest3985542-21cpuspeed_turbos21:15:141
3944571993633,2cyclictest0-21swapper/300:39:553
394457199360,35cyclictest0-21swapper/321:40:593
394456999360,1cyclictest0-21swapper/222:55:002
3944571993533,1cyclictest0-21swapper/323:16:593
3944571993533,1cyclictest0-21swapper/300:14:003
3944571993531,2cyclictest0-21swapper/320:56:593
3944571993529,5cyclictest4012456-21turbostat22:40:003
3944571993432,1cyclictest0-21swapper/323:21:593
3944571993432,1cyclictest0-21swapper/320:25:593
3944571993429,4cyclictest4017247-21turbostat22:55:003
3944569993431,2cyclictest0-21swapper/223:26:592
3944569993431,2cyclictest0-21swapper/219:15:592
3944569993430,2cyclictest0-21swapper/220:19:592
3944571993331,1cyclictest0-21swapper/320:42:593
394456999330,31cyclictest0-21swapper/200:19:592
3944571993229,2cyclictest0-21swapper/322:19:593
3944569993229,2cyclictest0-21swapper/221:39:562
3944569993228,2cyclictest0-21swapper/222:23:002
3944569993228,2cyclictest0-21swapper/200:24:592
3944571993128,1cyclictest0-21swapper/323:45:593
394456999310,1cyclictest0-21swapper/222:29:592
3944571993027,2cyclictest0-21swapper/320:11:593
3944571993025,4cyclictest4049964-21turbostat00:35:013
394457199300,26cyclictest0-21swapper/320:05:023
3944569993027,2cyclictest0-21swapper/223:09:592
394456999300,28cyclictest0-21swapper/219:58:002
394456999300,28cyclictest0-21swapper/219:57:592
394456999300,22cyclictest0-21swapper/220:35:592
3944571992927,1cyclictest0-21swapper/323:50:593
3944571992927,1cyclictest0-21swapper/320:03:003
3944571992926,2cyclictest0-21swapper/320:30:593
3944571992925,2cyclictest0-21swapper/323:05:153
394457199290,28cyclictest0-21swapper/323:40:133
394457199290,28cyclictest0-21swapper/321:15:143
394457199290,28cyclictest0-21swapper/319:45:013
394457199290,27cyclictest0-21swapper/321:35:033
394457199290,27cyclictest0-21swapper/300:00:133
394457199290,26cyclictest0-21swapper/319:50:143
394457199290,1cyclictest0-21swapper/300:20:153
3944569992926,2cyclictest0-21swapper/221:04:592
3944569992925,2cyclictest0-21swapper/223:40:132
3944569992925,2cyclictest0-21swapper/200:25:592
3944569992923,2cyclictest0-21swapper/221:45:172
394456999290,2cyclictest0-21swapper/200:10:592
394456999290,27cyclictest0-21swapper/220:05:032
394456999290,27cyclictest0-21swapper/219:50:132
394456999290,27cyclictest0-21swapper/219:45:012
394456999290,27cyclictest0-21swapper/200:00:142
4505291280,7ptp4l441ktimers/323:39:563
4505291280,6ptp4l61790irq/134-eno1-TxRx-000:30:013
4505291280,4ptp4l441ktimers/321:34:563
3944571992825,2cyclictest0-21swapper/322:20:143
3944571992825,2cyclictest0-21swapper/300:06:583
3944569992825,2cyclictest0-21swapper/223:20:152
3944569992825,2cyclictest0-21swapper/223:03:592
3944569992825,2cyclictest0-21swapper/221:15:142
3944569992823,3cyclictest0-21swapper/222:30:592
394456999280,26cyclictest0-21swapper/221:05:142
4505291270,6ptp4l61790irq/134-eno1-TxRx-023:35:003
4505291270,6ptp4l61790irq/134-eno1-TxRx-022:49:563
4505291270,6ptp4l61790irq/134-eno1-TxRx-022:49:553
4505291270,6ptp4l441ktimers/321:55:013
4505291270,5ptp4l61790irq/134-eno1-TxRx-000:00:013
4505291270,5ptp4l441ktimers/323:05:013
3944571992725,1cyclictest0-21swapper/300:19:583
394457199270,26cyclictest0-21swapper/319:37:003
3944569992723,2cyclictest0-21swapper/222:41:592
3944569992723,2cyclictest0-21swapper/221:20:162
394456999270,25cyclictest0-21swapper/221:54:002
394456999270,24cyclictest0-21swapper/220:46:592
4505291260,7ptp4l61790irq/134-eno1-TxRx-021:15:003
4505291260,6ptp4l441ktimers/321:49:563
4505291260,5ptp4l61790irq/134-eno1-TxRx-022:59:543
4505291260,5ptp4l441ktimers/322:35:013
4505291260,5ptp4l441ktimers/322:00:013
4505291260,5ptp4l441ktimers/322:00:003
4505291260,5ptp4l441ktimers/320:50:003
4505291260,5ptp4l441ktimers/319:59:553
4505291260,5ptp4l441ktimers/319:59:553
4505291260,4ptp4l441ktimers/323:15:003
3944571992623,2cyclictest0-21swapper/321:25:003
394457199260,25cyclictest0-21swapper/320:18:593
394457199260,19cyclictest0-21swapper/319:43:593
3944569992623,2cyclictest0-21swapper/223:54:582
3944569992622,2cyclictest0-21swapper/200:06:582
394456999260,24cyclictest0-21swapper/221:59:592
394456999260,24cyclictest0-21swapper/221:59:592
394456999260,24cyclictest0-21swapper/220:00:582
4505291250,6ptp4l441ktimers/319:24:553
4505291250,5ptp4l61790irq/134-eno1-TxRx-019:34:553
4505291250,5ptp4l441ktimers/322:14:543
3944571992523,1cyclictest0-21swapper/319:12:593
3944571992522,1cyclictest0-21swapper/320:35:593
394457199250,24cyclictest0-21swapper/319:26:593
394457199250,1cyclictest0-21swapper/320:24:553
3944569992523,1cyclictest0-21swapper/223:59:592
3944569992522,2cyclictest0-21swapper/219:23:002
3944569992522,2cyclictest0-21swapper/219:10:022
3944569992521,2cyclictest0-21swapper/220:51:592
394456999250,2cyclictest0-21swapper/222:55:592
394456999250,2cyclictest0-21swapper/221:41:592
394456999250,1cyclictest0-21swapper/222:14:552
3944571992421,2cyclictest0-21swapper/319:19:593
3944569992420,2cyclictest0-21swapper/219:42:002
394456999240,1cyclictest0-21swapper/223:39:582
3944569992321,1cyclictest0-21swapper/223:14:592
3944569992320,2cyclictest0-21swapper/222:49:002
3944569992320,2cyclictest0-21swapper/222:48:592
3944569992320,2cyclictest0-21swapper/222:07:002
3944569992219,2cyclictest0-21swapper/223:18:592
3944569992219,2cyclictest0-21swapper/200:39:002
3944569992118,2cyclictest0-21swapper/220:23:592
3944569992117,2cyclictest0-21swapper/223:34:562
3944569992018,1cyclictest0-21swapper/219:35:002
3944569992016,2cyclictest0-21swapper/221:33:582
3944569991916,2cyclictest0-21swapper/221:13:592
3944563991914,4cyclictest0-21swapper/019:10:010
3944569991814,2cyclictest3950893-21turbostat19:30:002
3944569991512,2cyclictest0-21swapper/222:03:582
394456999117,2cyclictest0-21swapper/221:26:002
39445699980,7cyclictest0-21swapper/220:14:312
39445699930,2cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional