You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 12:00
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot6.osadl.org (updated Tue Nov 25, 2025 00:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
376925999381,10cyclictest391125-21latency_hist19:55:010
376925999286,4cyclictest381575-21kworker/u8:1+events_unbound19:55:000
376928999181,7cyclictest429017-21latency_hist21:50:001
376925998983,4cyclictest478102-21kworker/u8:0+flush-179:000:30:260
376928998574,8cyclictest425597-21latency_hist21:40:021
376928998356,16cyclictest468484-21/usr/sbin/munin23:50:241
376925998373,6cyclictest461604-21kworker/u8:1+events_unbound23:30:250
376925998274,5cyclictest381625-21kworker/u8:2+flush-179:019:40:260
376928998143,27cyclictest467988-21latency_hist23:50:001
376925997871,5cyclictest381625-21kworker/u8:2+flush-179:019:45:240
376928997766,9cyclictest432201-21latency_hist22:00:011
376925997768,6cyclictest417360-21latency_hist21:15:000
376925997666,7cyclictest477861-21latency_hist00:20:010
37692599761,74cyclictest1-21systemd23:20:000
376925997566,6cyclictest442030-21kworker/u8:4+events_unbound23:05:200
376925997460,10cyclictest271-21systemd-journal21:35:210
376925997265,5cyclictest396466-21kworker/u8:2+flush-179:020:25:000
376928997061,7cyclictest407567-21latency_hist20:45:011
376925997062,6cyclictest434296-21kworker/u8:0+events_unbound22:25:240
376928996964,4cyclictest373755-21kworker/u8:0+flush-179:019:20:011
376928996959,7cyclictest484226-21latency_hist00:40:011
376925996959,7cyclictest446789-21latency_hist22:45:000
376925996957,9cyclictest409151-21latency_hist20:50:010
4670791680,18ptp4l441ktimers/319:06:183
376925996863,4cyclictest411030-21kworker/u8:3+flush-179:020:55:230
376925996862,4cyclictest463192-21kworker/u8:0+flush-179:000:05:010
376925996858,8cyclictest438833-21latency_hist22:20:010
376925996760,5cyclictest381575-21kworker/u8:1+flush-179:020:00:250
376925996661,3cyclictest402822-21kworker/u8:3+events_unbound20:35:230
376925996657,6cyclictest478102-21kworker/u8:0+flush-179:000:25:160
376925996657,6cyclictest434296-21kworker/u8:0+flush-179:022:05:200
376925996657,5cyclictest415752-21kworker/u8:1+events_unbound21:45:020
376928996556,6cyclictest433813-21apt-get22:05:001
376928996553,9cyclictest418954-21latency_hist21:20:021
376925996559,5cyclictest442030-21kworker/u8:4+flush-179:023:00:240
376925996558,5cyclictest464853-21kworker/u8:2+events_unbound23:45:250
376925996556,7cyclictest448374-21latency_hist22:50:010
376925996555,7cyclictest466405-21latency_hist23:45:010
37692599650,3cyclictest191rcuc/021:53:590
376925996453,9cyclictest380261-21apt-get19:20:110
376928996357,4cyclictest398261-21apt-get20:15:141
376925996357,4cyclictest381575-21kworker/u8:1+events_unbound20:15:010
376928996253,7cyclictest381575-21kworker/u8:1+flush-179:020:25:251
376925996257,4cyclictest723-21Xorg20:27:590
376928996050,8cyclictest479450-21latency_hist00:25:011
376928996050,7cyclictest445183-21latency_hist22:40:001
376928996047,10cyclictest271-21systemd-journal23:25:011
376925996053,5cyclictest399601-21kworker/u8:0+events_unbound21:05:010
376925996053,4cyclictest415752-21kworker/u8:1+flush-179:021:30:150
376928995949,7cyclictest430603-21latency_hist21:55:011
376925995953,4cyclictest469812-21kworker/u8:1+flush-179:000:05:160
376925995952,5cyclictest442030-21kworker/u8:4+flush-179:023:15:000
376929995855,2cyclictest0-21swapper/220:27:592
376925995852,4cyclictest469812-21kworker/u8:1+flush-179:023:55:200
376925995852,4cyclictest469812-21kworker/u8:1+flush-179:000:35:160
376925995851,5cyclictest442030-21kworker/u8:4+events_unbound22:30:230
376925995848,7cyclictest380009-21latency_hist19:20:010
37692599581,55cyclictest1-21systemd21:10:010
376928995739,16cyclictest723-21Xorg22:09:591
376929995653,2cyclictest0-21swapper/221:36:592
376928995648,6cyclictest420537-21kworker/u8:0+flush-179:021:35:011
376928995648,5cyclictest384764-21latency_hist19:35:011
376925995650,5cyclictest434296-21kworker/u8:0+flush-179:022:10:210
376925995540,11cyclictest433930-21store22:05:010
376925995444,8cyclictest399601-21kworker/u8:0+flush-179:021:15:200
376925995347,4cyclictest434296-21kworker/u8:0+events_unbound22:25:000
376928995243,7cyclictest415752-21kworker/u8:1+flush-179:021:20:241
376928995242,8cyclictest452083-21apt-get23:00:141
376928995242,7cyclictest449715-21kworker/u8:0+flush-179:022:55:001
376928995242,7cyclictest449715-21kworker/u8:0+flush-179:022:55:001
376928995240,9cyclictest404374-21latency_hist20:35:021
376925995245,5cyclictest427467-21kworker/u8:3+events_unbound21:55:250
376925995244,5cyclictest415752-21kworker/u8:1+events_unbound21:20:160
376928995147,3cyclictest723-21Xorg00:06:591
376925995146,4cyclictest427467-21kworker/u8:3+flush-179:022:50:160
376925995146,4cyclictest427467-21kworker/u8:3+flush-179:022:50:150
376925995143,6cyclictest376784-21latency_hist19:10:010
376925995142,6cyclictest378397-21latency_hist19:15:010
376925995045,4cyclictest381575-21kworker/u8:1+flush-179:020:30:220
376925995040,7cyclictest463192-21kworker/u8:0+flush-179:023:50:210
376929994945,2cyclictest0-21swapper/221:52:592
376928994940,7cyclictest481036-21latency_hist00:30:011
376925994944,4cyclictest381625-21kworker/u8:2+flush-179:019:30:240
376925994943,4cyclictest427467-21kworker/u8:3+flush-179:022:35:180
376925994941,5cyclictest381575-21kworker/u8:1+events_unbound19:35:180
376929994744,2cyclictest0-21swapper/222:32:592
376928994738,7cyclictest396420-21latency_hist20:10:021
376928994734,10cyclictest394579-21latency_hist20:05:011
376925994740,5cyclictest453395-21kworker/u8:0+events_unbound23:25:250
376928994638,6cyclictest474928-21apt-get00:10:131
376928994635,8cyclictest442030-21kworker/u8:4+flush-179:023:10:211
37693199450,5cyclictest409159-21turbostat20:50:003
376928994539,5cyclictest396677-21apt-get20:10:111
376928994536,7cyclictest271-21systemd-journal22:23:201
376928994535,8cyclictest463468-21apt-get23:35:141
376928994535,8cyclictest463192-21kworker/u8:0+flush-179:000:00:001
376928994337,4cyclictest440677-21apt-get22:25:131
376928994333,7cyclictest463213-21latency_hist23:35:011
376925994339,3cyclictest407601-21kworker/u8:2+flush-179:020:50:200
37692599430,41cyclictest723-21Xorg20:44:590
37693199420,40cyclictest467992-21turbostat23:50:003
376928994239,2cyclictest723-21Xorg19:54:591
376928994234,6cyclictest422137-21latency_hist21:30:001
376928994228,12cyclictest409779-21cpuspeed_turbos20:50:141
37692899421,1cyclictest391640-21apt-get19:55:131
376925994234,5cyclictest396466-21kworker/u8:2+flush-179:020:19:590
37692599420,40cyclictest723-21Xorg00:22:000
376928994133,6cyclictest456823-21/usr/sbin/munin23:15:181
376928994133,5cyclictest0-21swapper/120:49:581
376928994126,13cyclictest723-21Xorg19:20:591
376925994136,4cyclictest407601-21kworker/u8:2+flush-179:021:25:250
376931994035,4cyclictest445200-21turbostat22:40:003
37693199400,5cyclictest442017-21turbostat22:30:003
37693199400,38cyclictest396436-21turbostat20:10:013
376928994035,3cyclictest475-21dbus-daemon00:04:011
376928994032,6cyclictest271-21systemd-journal23:10:001
376928994031,7cyclictest469812-21kworker/u8:1+flush-179:000:15:251
37693199390,37cyclictest425591-21turbostat21:40:013
376929993936,2cyclictest0-21swapper/222:24:002
37692999390,36cyclictest0-21swapper/222:39:552
376925993935,2cyclictest0-21swapper/020:05:590
376929993835,2cyclictest0-21swapper/219:24:562
37692999380,36cyclictest0-21swapper/200:02:592
376928993834,3cyclictest405960-21latency_hist20:40:001
37692899380,37cyclictest0-21swapper/119:37:591
376931993734,2cyclictest0-21swapper/319:29:593
37692999370,35cyclictest0-21swapper/222:50:592
37692999370,35cyclictest0-21swapper/222:50:592
37692999370,35cyclictest0-21swapper/219:51:592
37692999370,2cyclictest0-21swapper/221:19:552
37692899378,5cyclictest0-21swapper/121:11:001
376928993734,2cyclictest0-21swapper/121:05:001
376928993723,12cyclictest463196-21kworker/u8:3+flush-179:023:40:261
4670791360,3ptp4l441ktimers/319:40:143
376931993634,1cyclictest0-21swapper/322:54:593
376931993634,1cyclictest0-21swapper/322:54:583
376931993634,1cyclictest0-21swapper/321:14:543
376931993633,2cyclictest0-21swapper/319:48:593
37693199360,35cyclictest0-21swapper/319:18:593
376929993634,1cyclictest0-21swapper/219:50:002
376929993633,2cyclictest0-21swapper/222:44:002
37692999360,2cyclictest0-21swapper/220:53:002
376928993633,2cyclictest723-21Xorg19:46:591
37692899363,30cyclictest381848-21apt-get19:25:141
376928993632,3cyclictest442030-21kworker/u8:4+flush-179:022:33:201
376928993631,4cyclictest412614-21apt21:00:011
376925993632,3cyclictest723-21Xorg21:46:590
376931993533,1cyclictest0-21swapper/322:16:003
376931993533,1cyclictest0-21swapper/321:58:593
376931993533,1cyclictest0-21swapper/300:20:593
376929993532,2cyclictest0-21swapper/223:04:552
376929993532,2cyclictest0-21swapper/220:05:592
37692899352,3cyclictest481295-21apt-get00:30:151
376928993516,18cyclictest0-21swapper/121:40:191
376925993526,7cyclictest442030-21kworker/u8:4+events_unbound22:55:020
376931993432,1cyclictest0-21swapper/321:30:583
376931993432,1cyclictest0-21swapper/321:29:553
376931993431,2cyclictest0-21swapper/322:12:583
376931993428,4cyclictest427448-21turbostat21:45:013
376929993432,1cyclictest0-21swapper/200:00:002
376929993431,2cyclictest0-21swapper/223:29:552
376929993431,2cyclictest0-21swapper/221:47:002
376928993427,6cyclictest449965-21apt-get22:55:031
37692599341,4cyclictest0-21swapper/023:20:030
376929993330,2cyclictest0-21swapper/220:57:592
376929993330,2cyclictest0-21swapper/220:47:592
376929993330,2cyclictest0-21swapper/200:37:002
376928993329,3cyclictest723-21Xorg21:09:591
376928993328,4cyclictest442030-21kworker/u8:4+flush-179:022:40:251
376928993325,6cyclictest436440-21kernelversion22:10:191
376928993311,5cyclictest439069-21sed22:20:011
37692599331,4cyclictest474928-21apt-get00:10:140
37693199320,31cyclictest0-21swapper/322:23:593
376929993230,1cyclictest0-21swapper/222:14:592
376929993230,1cyclictest0-21swapper/221:04:592
376929993229,2cyclictest0-21swapper/221:42:582
376929993229,2cyclictest0-21swapper/200:20:592
376929993228,2cyclictest0-21swapper/220:04:592
37692999320,2cyclictest0-21swapper/219:55:032
376928993225,5cyclictest381575-21kworker/u8:1+flush-179:020:20:181
376928993216,15cyclictest373755-21kworker/u8:0+flush-179:019:10:321
37692899320,30cyclictest723-21Xorg23:25:591
376925993225,5cyclictest373755-21kworker/u8:0+events_unbound19:25:150
376929993128,2cyclictest0-21swapper/223:52:592
376929993128,2cyclictest0-21swapper/222:49:552
37692999310,29cyclictest0-21swapper/219:15:132
37692999310,27cyclictest0-21swapper/223:05:122
37692999310,21cyclictest0-21swapper/200:10:142
376928993129,1cyclictest0-21swapper/119:44:551
376931993028,1cyclictest0-21swapper/320:36:593
376931993028,1cyclictest0-21swapper/300:29:553
376929993025,2cyclictest0-21swapper/220:41:582
37692999300,29cyclictest0-21swapper/223:20:032
37692999300,27cyclictest0-21swapper/223:30:162
37692599301,2cyclictest723-21Xorg23:38:580
376931992927,1cyclictest0-21swapper/320:50:133
37693199290,28cyclictest0-21swapper/323:50:593
37693199290,28cyclictest0-21swapper/320:10:143
37693199290,27cyclictest0-21swapper/322:55:023
37693199290,27cyclictest0-21swapper/322:45:143
37693199290,27cyclictest0-21swapper/300:05:143
37693199290,1cyclictest0-21swapper/322:05:123
37693199290,1cyclictest0-21swapper/321:20:143
37693199290,1cyclictest0-21swapper/320:25:143
37693199290,1cyclictest0-21swapper/319:10:143
37693199290,1cyclictest0-21swapper/300:15:143
376929992926,2cyclictest0-21swapper/223:15:592
376929992926,2cyclictest0-21swapper/220:22:592
37692999290,2cyclictest0-21swapper/221:30:152
37692999290,2cyclictest0-21swapper/221:05:142
37692999290,26cyclictest0-21swapper/219:30:152
376928992924,4cyclictest448457-21latency_hist22:50:021
376931992826,1cyclictest0-21swapper/323:25:153
376931992826,1cyclictest0-21swapper/319:54:563
376931992826,1cyclictest0-21swapper/319:20:143
37693199280,2cyclictest0-21swapper/320:20:153
37693199280,27cyclictest0-21swapper/320:15:153
376929992825,2cyclictest0-21swapper/222:03:002
376929992825,2cyclictest0-21swapper/220:17:592
37692999280,2cyclictest0-21swapper/222:15:132
37692999280,25cyclictest0-21swapper/223:36:592
4670791270,7ptp4l441ktimers/323:25:013
4670791270,6ptp4l441ktimers/323:45:013
4670791270,6ptp4l441ktimers/300:00:013
376931992725,1cyclictest0-21swapper/321:00:123
37693199270,26cyclictest0-21swapper/321:07:003
37693199270,26cyclictest0-21swapper/320:00:143
37693199270,1cyclictest0-21swapper/321:15:143
376929992724,2cyclictest0-21swapper/223:10:582
376929992724,2cyclictest0-21swapper/222:55:012
376929992724,2cyclictest0-21swapper/220:35:592
37692999270,25cyclictest0-21swapper/223:41:592
37692999270,25cyclictest0-21swapper/219:37:592
37692999270,24cyclictest0-21swapper/222:10:002
4670791260,6ptp4l61690irq/136-eno1-TxRx-000:05:013
4670791260,6ptp4l441ktimers/322:35:013
4670791260,6ptp4l441ktimers/300:40:003
4670791260,5ptp4l441ktimers/321:55:003
4670791260,5ptp4l441ktimers/321:00:013
4670791260,4ptp4l441ktimers/323:15:003
376931992624,1cyclictest0-21swapper/322:44:593
376931992623,2cyclictest453430-21turbostat23:04:593
37692999260,24cyclictest0-21swapper/200:18:592
37692999260,1cyclictest0-21swapper/200:08:002
4670791250,5ptp4l61690irq/136-eno1-TxRx-023:40:003
4670791250,5ptp4l441ktimers/323:20:003
4670791250,5ptp4l441ktimers/321:50:003
4670791250,5ptp4l441ktimers/319:40:013
4670791250,5ptp4l441ktimers/300:34:563
4670791250,5ptp4l441ktimers/300:15:003
4670791250,4ptp4l441ktimers/322:05:023
37693199250,24cyclictest0-21swapper/319:31:583
376929992523,1cyclictest0-21swapper/221:29:592
376929992521,2cyclictest0-21swapper/219:27:002
37692999250,24cyclictest0-21swapper/219:10:162
4670791240,5ptp4l61690irq/136-eno1-TxRx-020:34:563
376931992422,1cyclictest392978-21turbostat20:00:003
376931992421,2cyclictest0-21swapper/323:09:593
376931992320,2cyclictest0-21swapper/320:40:143
376929992320,2cyclictest0-21swapper/223:47:592
376929992320,2cyclictest0-21swapper/220:33:592
376929992320,2cyclictest0-21swapper/200:27:582
4670791220,3ptp4l441ktimers/323:34:563
376929992219,2cyclictest0-21swapper/221:57:582
376928992016,3cyclictest0-21swapper/119:10:011
376929991917,1cyclictest0-21swapper/221:24:592
376929991816,1cyclictest0-21swapper/220:11:002
376929991815,2cyclictest0-21swapper/219:44:552
376929991310,2cyclictest0-21swapper/200:33:592
3769299996,2cyclictest0-21swapper/222:25:062
3769299990,8cyclictest0-21swapper/221:14:032
3769299930,2cyclictest0-21swapper/219:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional