You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 05:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Sun Nov 23, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3156536999083,5cyclictest3190200-21kworker/u8:3+events_unbound21:00:000
3156536998978,8cyclictest3242176-21latency_hist23:35:010
3156540998881,5cyclictest3193618-21kworker/u8:0+flush-179:021:25:251
3156540998680,4cyclictest3230775-21latency_hist23:00:001
3156540998573,9cyclictest3224666-21/usr/sbin/munin22:40:241
3156536998478,4cyclictest3213059-21kworker/u8:3+events_unbound23:10:000
3156536998376,4cyclictest3180363-21kworker/u8:1+flush-179:020:25:280
3156536998174,5cyclictest3154994-21kworker/u8:1+events_unbound20:05:000
3156536998174,5cyclictest3154994-21kworker/u8:1+events_unbound20:05:000
3156536998171,7cyclictest3203209-21latency_hist21:35:000
3156540998073,5cyclictest3206390-21kworker/u8:0+flush-179:022:05:001
3156536997767,7cyclictest3175591-21latency_hist20:10:010
3156536997363,8cyclictest3156452-21latency_hist19:10:020
3156536997165,4cyclictest3193618-21kworker/u8:0+flush-179:021:05:230
3156540997060,7cyclictest3261862-21latency_hist00:35:001
3156536997063,5cyclictest3216210-21apt-get22:14:590
3156536996964,4cyclictest3175574-21kworker/u8:2+flush-179:021:10:160
3156536996962,5cyclictest3213003-21kworker/u8:2+events_unbound22:25:010
3156536996959,8cyclictest3164688-21/usr/sbin/munin19:35:310
3156536996754,10cyclictest271-21systemd-journal00:40:000
3156536996656,7cyclictest3154994-21kworker/u8:1+flush-179:019:55:010
3156536996653,9cyclictest3198127-21apt-get21:19:590
3156540996455,7cyclictest3178766-21latency_hist20:20:001
3156536996456,6cyclictest3193618-21kworker/u8:0+flush-179:021:25:490
3156536996356,4cyclictest3222802-21kworker/u8:1+flush-179:022:45:160
3156536996254,5cyclictest3187032-21kworker/u8:1+flush-179:020:50:240
3156536996254,5cyclictest3187032-21kworker/u8:1+flush-179:020:50:240
3156536996252,7cyclictest3226015-21latency_hist22:45:020
3156540996153,6cyclictest3232635-21apt-get23:05:111
3156540996049,8cyclictest3239004-21latency_hist23:25:001
3156540996042,17cyclictest3245414-21kworker/u8:1+flush-179:000:20:001
3156540995953,4cyclictest3222802-21kworker/u8:1+flush-179:023:30:241
3156536995952,5cyclictest3206390-21kworker/u8:0+events_unbound21:50:010
3156536995948,8cyclictest3216709-21apt-get22:15:130
3156540995853,3cyclictest3222802-21kworker/u8:1+flush-179:023:10:261
3156536995853,4cyclictest3245414-21kworker/u8:1+flush-179:000:00:230
3156540995748,7cyclictest3255228-21latency_hist00:15:011
3156540995748,6cyclictest3232425-21kworker/u8:2+events_unbound23:15:251
3156540995647,6cyclictest3193360-21latency_hist21:05:011
3156536995650,5cyclictest3154994-21kworker/u8:1+flush-179:019:20:220
3156540995551,3cyclictest723-21Xorg19:54:591
3156540995547,6cyclictest3240589-21latency_hist23:30:011
3156536995547,6cyclictest3222866-21kworker/u8:2+flush-179:022:35:260
3156540995450,2cyclictest3250576-21apt-config00:00:021
3156536995441,10cyclictest3250130-21latency_hist00:00:020
3156540995349,3cyclictest3153433-21kworker/u8:3+flush-179:019:30:261
3156540995348,4cyclictest3213059-21kworker/u8:3+events_unbound22:50:201
3156540995345,6cyclictest271-21systemd-journal20:26:421
3156540995236,14cyclictest3232425-21kworker/u8:2+flush-179:023:45:131
3156536995246,5cyclictest3242217-21kworker/u8:0+events_unbound23:35:230
3156536995243,6cyclictest3159693-21latency_hist19:20:010
3156536995242,7cyclictest3162863-21latency_hist19:30:010
3156536995145,4cyclictest3175574-21kworker/u8:2+flush-179:020:20:010
3156536995143,5cyclictest3222802-21kworker/u8:1+events_unbound23:10:200
3156540995042,5cyclictest3188581-21latency_hist20:50:011
3156540995039,10cyclictest3222802-21kworker/u8:1+events_unbound22:50:001
3156536995046,3cyclictest3239052-21kworker/u8:3+flush-179:023:50:180
3156540994943,5cyclictest3193618-21kworker/u8:0+flush-179:021:25:001
3156540994941,6cyclictest3213059-21kworker/u8:3+flush-179:022:20:011
4670791480,9ptp4l441ktimers/319:05:273
3156540994837,9cyclictest3262117-21apt-get00:35:121
3156540994834,12cyclictest723-21Xorg23:44:591
3156540994832,14cyclictest723-21Xorg23:54:001
3156536994843,4cyclictest3256532-21kworker/u8:0+flush-179:000:20:250
3156536994840,6cyclictest3214609-21latency_hist22:10:010
3156536994838,6cyclictest3222802-21kworker/u8:1+events_unbound22:50:170
3156540994738,6cyclictest3177160-21kworker/u8:0+flush-179:020:30:011
3156536994742,4cyclictest3239052-21kworker/u8:3+events_unbound23:40:250
3156536994741,4cyclictest3213059-21kworker/u8:3+events_unbound22:25:230
3156536994739,5cyclictest3154994-21kworker/u8:1+flush-179:019:35:010
3156540994638,6cyclictest3158320-21/usr/sbin/munin19:15:231
3156536994641,4cyclictest3206390-21kworker/u8:0+flush-179:021:50:160
3156536994639,5cyclictest3232425-21kworker/u8:2+flush-179:023:20:210
3156536994638,5cyclictest3239052-21kworker/u8:3+flush-179:000:15:150
3156540994537,6cyclictest3187032-21kworker/u8:1+events_unbound20:55:001
3156540994537,6cyclictest3187032-21kworker/u8:1+events_unbound20:54:591
3156540994441,2cyclictest0-21swapper/100:00:021
3156540994428,14cyclictest3206658-21apt-get21:45:141
3156536994439,4cyclictest3175574-21kworker/u8:2+flush-179:021:00:200
3156536994438,4cyclictest3206390-21kworker/u8:0+flush-179:022:00:210
3156536994438,4cyclictest3175574-21kworker/u8:2+flush-179:020:40:150
3156536994434,7cyclictest3175574-21kworker/u8:2+events_unbound20:10:210
315653699441,41cyclictest3255238-21basename00:15:000
3156540994340,2cyclictest3175574-21kworker/u8:2+flush-179:021:20:011
3156536994340,2cyclictest0-21swapper/023:47:580
3156536994338,4cyclictest3256532-21kworker/u8:0+flush-179:000:30:160
3156536994337,4cyclictest3245414-21kworker/u8:1+flush-179:000:25:230
315653699431,38cyclictest3221491-21apt-get22:30:140
315653699430,41cyclictest723-21Xorg21:56:590
3156541994239,2cyclictest0-21swapper/222:44:592
3156541994239,2cyclictest0-21swapper/222:02:592
3156540994237,3cyclictest3187028-21systemd-run20:45:011
3156540994222,4cyclictest3194458-21phc2sys-jitter21:05:231
3156540994222,4cyclictest3165123-21gltestperf19:35:171
315654099420,2cyclictest723-21Xorg20:35:591
315654099420,27cyclictest0-21swapper/122:25:131
3156536994235,5cyclictest3175574-21kworker/u8:2+flush-179:020:20:180
315654699410,39cyclictest3204831-21turbostat21:40:003
3156540994126,1cyclictest0-21swapper/120:07:591
315654099410,39cyclictest723-21Xorg19:47:591
3156536994132,6cyclictest3162912-21kworker/u8:2+flush-179:019:55:230
3156546994035,4cyclictest3175600-21turbostat20:10:003
315654699400,2cyclictest0-21swapper/323:33:583
3156540994035,3cyclictest3158115-21systemd-run19:15:001
3156536994032,6cyclictest3206390-21kworker/u8:0+flush-179:021:45:010
315653699400,2cyclictest723-21Xorg19:42:590
3156546993934,4cyclictest3211450-21turbostat22:00:003
3156546993933,4cyclictest3248550-21turbostat23:55:003
315654699390,4cyclictest3207991-21turbostat21:50:003
315654699390,38cyclictest3250149-21turbostat00:00:013
315654699390,37cyclictest3203221-21turbostat21:34:593
315654199390,36cyclictest0-21swapper/221:32:002
3156540993935,2cyclictest723-21Xorg23:37:591
3156540993924,1cyclictest0-21swapper/121:35:591
315654099390,2cyclictest723-21Xorg22:31:001
3156536993937,1cyclictest0-21swapper/020:50:000
3156536993933,4cyclictest3239052-21kworker/u8:3+flush-179:000:05:180
3156536993930,7cyclictest3237421-21sendmail23:20:000
3156541993834,2cyclictest0-21swapper/222:28:002
315654199380,36cyclictest0-21swapper/221:42:582
315654199380,36cyclictest0-21swapper/219:31:592
3156540993833,4cyclictest3194952-21turbostat21:14:561
3156540993831,5cyclictest3224423-21latency_hist22:40:011
3156541993733,2cyclictest0-21swapper/222:21:582
315654199370,2cyclictest0-21swapper/222:09:562
3156540993734,2cyclictest0-21swapper/123:02:591
3156540993731,4cyclictest3162912-21kworker/u8:2+flush-179:019:55:281
3156536993734,2cyclictest0-21swapper/020:39:590
315653699370,1cyclictest0-21swapper/023:27:590
3156546993634,1cyclictest0-21swapper/322:34:553
3156546993633,1cyclictest0-21swapper/321:50:583
315654699360,35cyclictest0-21swapper/300:04:593
315654199360,1cyclictest0-21swapper/223:26:002
315654199360,1cyclictest0-21swapper/221:54:592
3156540993630,5cyclictest3213059-21kworker/u8:3+flush-179:022:20:221
3156536993631,3cyclictest3169265-21systemd-run19:50:000
3156541993531,2cyclictest0-21swapper/220:05:002
3156541993531,2cyclictest0-21swapper/220:05:002
315654199350,1cyclictest0-21swapper/220:35:002
3156540993533,1cyclictest0-21swapper/122:09:551
3156540993532,2cyclictest0-21swapper/120:14:591
3156540993532,2cyclictest0-21swapper/119:40:591
3156540993532,2cyclictest0-21swapper/100:05:591
3156536993530,3cyclictest1-21systemd21:40:000
3156546993431,2cyclictest0-21swapper/300:05:593
3156541993431,2cyclictest0-21swapper/223:59:552
3156541993431,1cyclictest0-21swapper/200:19:592
3156540993430,2cyclictest723-21Xorg21:54:591
3156540993430,2cyclictest723-21Xorg21:42:591
3156540993429,3cyclictest3162865-21munin-run19:30:001
3156540993426,5cyclictest3190404-21/usr/sbin/munin20:55:261
3156540993423,9cyclictest3216195-21latency_hist22:15:011
315654699330,4cyclictest3246963-21turbostat23:49:593
3156541993331,1cyclictest0-21swapper/223:15:002
3156541993331,1cyclictest0-21swapper/219:40:002
3156541993330,2cyclictest0-21swapper/220:11:592
3156541993329,2cyclictest0-21swapper/220:28:592
315654099332,3cyclictest3201878-21apt-get21:30:121
315654099330,32cyclictest723-21Xorg00:28:591
3156546993230,1cyclictest0-21swapper/320:24:553
315654699320,1cyclictest0-21swapper/322:13:593
3156541993229,2cyclictest0-21swapper/221:55:582
3156541993229,2cyclictest0-21swapper/220:07:592
315654199320,29cyclictest0-21swapper/200:35:592
3156536993225,6cyclictest2126-10pipewire-media-21:20:250
3156546993129,1cyclictest0-21swapper/323:27:593
3156546993128,2cyclictest0-21swapper/322:51:583
3156541993129,1cyclictest0-21swapper/222:15:002
3156540993127,2cyclictest3209583-21apt-get21:55:021
3156536993126,3cyclictest0-21swapper/019:10:150
3156546993028,1cyclictest0-21swapper/322:44:003
315654699300,29cyclictest0-21swapper/320:39:593
315654699300,28cyclictest0-21swapper/300:15:043
3156541993026,2cyclictest0-21swapper/219:54:552
3156536993026,3cyclictest3213059-21kworker/u8:3+events_unbound23:00:220
4670791290,3ptp4l441ktimers/300:30:003
3156546992927,1cyclictest0-21swapper/320:15:163
3156546992927,1cyclictest0-21swapper/319:50:143
3156546992926,2cyclictest0-21swapper/323:40:133
3156546992926,2cyclictest0-21swapper/321:05:143
315654699290,2cyclictest0-21swapper/319:45:133
315654699290,28cyclictest0-21swapper/322:55:143
315654699290,27cyclictest0-21swapper/323:10:133
315654699290,27cyclictest0-21swapper/320:50:143
315654699290,27cyclictest0-21swapper/320:50:143
3156541992927,1cyclictest0-21swapper/222:59:592
3156541992926,2cyclictest0-21swapper/221:45:152
3156541992926,2cyclictest0-21swapper/221:25:152
3156541992926,2cyclictest0-21swapper/221:19:002
3156541992926,2cyclictest0-21swapper/220:45:142
3156541992926,2cyclictest0-21swapper/219:15:132
3156541992926,2cyclictest0-21swapper/200:25:142
315654199290,27cyclictest0-21swapper/223:15:022
315654199290,27cyclictest0-21swapper/223:01:592
315654199290,27cyclictest0-21swapper/221:20:132
315654199290,27cyclictest0-21swapper/221:00:132
315654199290,27cyclictest0-21swapper/220:35:152
315654199290,27cyclictest0-21swapper/219:10:152
315654199290,1cyclictest0-21swapper/221:35:172
315654199290,1cyclictest0-21swapper/200:00:022
315654099291,3cyclictest3161285-21ls19:25:001
3156546992826,1cyclictest0-21swapper/321:20:023
3156546992826,1cyclictest0-21swapper/319:10:153
3156546992825,2cyclictest0-21swapper/323:20:133
315654699280,2cyclictest0-21swapper/320:30:013
315654699280,27cyclictest0-21swapper/322:45:133
3156541992825,2cyclictest0-21swapper/223:45:142
3156541992825,2cyclictest0-21swapper/223:31:592
3156541992825,2cyclictest0-21swapper/222:15:012
3156541992825,2cyclictest0-21swapper/221:10:132
3156541992825,2cyclictest0-21swapper/200:05:592
315654199280,26cyclictest0-21swapper/221:05:592
3156540992824,3cyclictest3180565-21latency_hist20:25:021
3156536992822,4cyclictest3183300-21phc2sys20:30:230
4670791270,6ptp4l441ktimers/321:30:013
3156546992725,1cyclictest0-21swapper/323:15:023
3156546992725,1cyclictest0-21swapper/322:09:593
3156546992725,1cyclictest0-21swapper/320:49:003
3156546992725,1cyclictest0-21swapper/320:25:173
3156546992724,2cyclictest0-21swapper/321:14:593
315654699270,26cyclictest0-21swapper/319:35:143
315654699270,26cyclictest0-21swapper/300:20:593
315654699270,25cyclictest3216200-21turbostat22:19:543
315654699270,25cyclictest0-21swapper/319:15:193
3156541992724,2cyclictest0-21swapper/219:55:582
3156541992723,2cyclictest0-21swapper/219:27:592
3156541992723,2cyclictest0-21swapper/200:13:592
3156540992724,2cyclictest723-21Xorg00:22:591
3156536992722,4cyclictest3222802-21kworker/u8:1+flush-179:022:55:170
4670791260,6ptp4l441ktimers/300:39:553
3156546992623,2cyclictest0-21swapper/319:24:593
315654699260,25cyclictest0-21swapper/323:04:593
315654699260,24cyclictest0-21swapper/321:15:143
315654699260,24cyclictest0-21swapper/319:30:373
315654699260,23cyclictest0-21swapper/320:55:583
315654699260,1cyclictest0-21swapper/323:05:123
3156541992623,2cyclictest0-21swapper/220:51:002
3156541992623,2cyclictest0-21swapper/220:51:002
315654199260,24cyclictest0-21swapper/220:40:592
315654199260,24cyclictest0-21swapper/219:45:092
315654199260,24cyclictest0-21swapper/219:41:132
3156540992621,3cyclictest3170799-21kworker/u8:0+flush-179:020:00:191
3156540992621,3cyclictest3170799-21kworker/u8:0+flush-179:020:00:191
4670791250,5ptp4l61690irq/136-eno1-TxRx-020:45:003
4670791250,5ptp4l441ktimers/322:39:553
4670791250,3ptp4l441ktimers/300:15:003
3156546992523,1cyclictest0-21swapper/322:25:593
3156546992523,1cyclictest0-21swapper/319:29:023
3156541992522,2cyclictest0-21swapper/223:06:592
3156541992520,2cyclictest0-21swapper/223:50:132
4670791240,5ptp4l61690irq/136-eno1-TxRx-020:05:003
4670791240,5ptp4l61690irq/136-eno1-TxRx-020:05:003
4670791240,3ptp4l441ktimers/320:15:003
3156546992422,1cyclictest0-21swapper/319:42:593
3156546992421,2cyclictest0-21swapper/321:44:003
315654699240,1cyclictest0-21swapper/321:04:003
3156541992421,2cyclictest0-21swapper/223:21:582
4670791230,3ptp4l61690irq/136-eno1-TxRx-000:34:563
4670791230,3ptp4l441ktimers/320:00:013
3156541992321,1cyclictest0-21swapper/222:39:542
3156541992321,1cyclictest0-21swapper/220:59:592
3156541992320,2cyclictest0-21swapper/222:32:002
4670791220,3ptp4l61690irq/136-eno1-TxRx-022:04:563
4670791220,3ptp4l441ktimers/323:39:573
4670791220,3ptp4l441ktimers/322:24:563
3156541992219,2cyclictest0-21swapper/223:40:592
3156541992219,2cyclictest0-21swapper/222:50:582
3156541992219,2cyclictest0-21swapper/222:48:592
3156541992219,2cyclictest0-21swapper/219:22:592
3156540992115,5cyclictest3156701-21latency_hist19:10:031
3156541991815,2cyclictest0-21swapper/223:35:122
3156541991714,2cyclictest0-21swapper/220:24:592
3156541991713,2cyclictest3261872-21turbostat00:35:002
3156541991613,2cyclictest0-21swapper/200:24:592
3156541991311,1cyclictest0-21swapper/220:20:002
31565419930,2cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional