You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 08:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6.osadl.org (updated Wed Nov 19, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27747599120107,11cyclictest371543-40logrotate00:00:000
2774759910799,6cyclictest381192-21kworker/u8:2+events_unbound00:40:000
2774809910292,7cyclictest344238-21latency_hist22:35:011
277475999994,3cyclictest270346-21kworker/u8:3+events_unbound19:15:280
277480999582,9cyclictest382751-21latency_hist00:35:011
277475999382,8cyclictest350859-21latency_hist22:55:020
277475998982,5cyclictest350894-21kworker/u8:3+flush-179:023:05:000
277475998982,5cyclictest332071-21latency_hist22:00:010
277480998773,10cyclictest347431-21latency_hist22:45:011
277475998581,3cyclictest327334-21kworker/u8:2+flush-179:022:15:260
277475998379,3cyclictest261321-21kworker/u8:1+flush-179:019:25:260
277475998370,10cyclictest319338-21latency_hist21:20:010
277480997867,8cyclictest271-21systemd-journal23:55:001
277475997871,5cyclictest332110-21kworker/u8:0+events_unbound22:00:190
277475997569,5cyclictest297813-21kworker/u8:1+events_unbound20:10:280
277475997563,4cyclictest371616-21kworker/u8:1+events_unbound00:05:270
277480997463,8cyclictest349274-21latency_hist22:50:021
277475997466,6cyclictest297813-21kworker/u8:1+flush-179:020:25:010
277475997466,6cyclictest293534-21kworker/u8:0+events_unbound20:00:010
277475997466,6cyclictest293534-21kworker/u8:0+events_unbound20:00:010
277475997267,4cyclictest346103-21kworker/u8:2+events_unbound22:55:250
277475997267,4cyclictest321172-21kworker/u8:3+flush-179:021:45:250
277480997164,6cyclictest297813-21kworker/u8:1+events_unbound20:15:011
277475997155,4cyclictest309511-21kworker/u8:0+events_unbound20:50:230
277475997064,4cyclictest301572-21kworker/u8:0+flush-179:020:35:230
277480996959,8cyclictest317539-21kworker/u8:1+flush-179:021:40:011
27748199680,65cyclictest0-21swapper/222:20:592
277475996863,4cyclictest366364-21kworker/u8:0+events_unbound23:45:250
277480996759,5cyclictest285349-21latency_hist19:35:001
277480996759,5cyclictest285349-21latency_hist19:35:001
277475996758,7cyclictest360398-21kworker/u8:1+flush-179:023:35:290
277475996659,5cyclictest362450-21kworker/u8:3+events_unbound23:40:280
277475996656,7cyclictest296749-21latency_hist20:10:010
277480996554,8cyclictest312705-21latency_hist21:00:011
277475996559,4cyclictest288689-21kworker/u8:1+flush-179:019:50:230
277480996451,11cyclictest332069-21apt-get22:00:001
277475996459,4cyclictest368411-21kworker/u8:2+events_unbound00:04:010
277475996457,5cyclictest308201-21apt-get20:45:140
277480996356,4cyclictest301572-21kworker/u8:0+flush-179:020:25:221
277475996257,4cyclictest316132-21kworker/u8:0+events_unbound21:30:280
277475996257,4cyclictest277346-21kworker/u8:0+events_unbound19:20:180
277475996256,4cyclictest362450-21kworker/u8:3+events_unbound23:50:140
277475996251,9cyclictest381412-21apt-get00:30:130
277480996152,7cyclictest290642-21apt-get19:50:131
277480996151,8cyclictest379825-21apt-get00:25:131
277475996155,4cyclictest374824-21kworker/u8:0+flush-179:000:20:170
277475996154,6cyclictest339174-21kworker/u8:1+flush-179:022:40:000
277475996154,5cyclictest301572-21kworker/u8:0+events_unbound20:30:210
277483996058,1cyclictest0-21swapper/323:35:583
277475996045,12cyclictest340816-21apt-get22:25:010
277480995953,5cyclictest316132-21kworker/u8:0+flush-179:021:15:241
277475995951,5cyclictest342702-21kworker/u8:2+events_unbound22:30:230
277475995852,5cyclictest360450-21kworker/u8:2+flush-179:023:25:000
277480995754,2cyclictest0-21swapper/121:32:591
277480995751,4cyclictest723-21Xorg19:47:591
277481995652,2cyclictest0-21swapper/222:56:002
277480995650,4cyclictest363844-21apt-get23:35:141
277475995651,4cyclictest327334-21kworker/u8:2+events_unbound22:05:260
277475995650,5cyclictest309564-21kworker/u8:3+events_unbound21:10:000
277475995648,5cyclictest371616-21kworker/u8:1+flush-179:000:10:260
277475995548,5cyclictest346103-21kworker/u8:2+events_unbound22:45:010
277475995546,6cyclictest339174-21kworker/u8:1+events_unbound22:25:010
277481995451,2cyclictest0-21swapper/222:33:582
277475995449,4cyclictest277346-21kworker/u8:0+events_unbound19:10:240
277475995447,5cyclictest346103-21kworker/u8:2+events_unbound22:40:230
277475995447,5cyclictest287211-21apt-get19:40:120
277480995345,6cyclictest360398-21kworker/u8:1+flush-179:023:35:001
277480995342,8cyclictest311117-21latency_hist20:55:011
277475995348,3cyclictest316132-21kworker/u8:0+flush-179:021:20:220
277475995347,4cyclictest360398-21kworker/u8:1+flush-179:023:25:160
277475995347,4cyclictest283739-21kworker/u8:3+flush-179:020:15:230
277475995344,7cyclictest374824-21kworker/u8:0+events_unbound00:15:220
277480995245,5cyclictest320929-21cat21:24:541
277475995249,2cyclictest0-21swapper/023:32:590
277475995248,3cyclictest283739-21kworker/u8:3+flush-179:019:49:550
277475995247,4cyclictest301572-21kworker/u8:0+events_unbound20:40:280
277481995148,2cyclictest0-21swapper/222:11:592
277480995145,4cyclictest362450-21kworker/u8:3+flush-179:000:00:011
277480995138,11cyclictest360672-21apt-get23:25:141
277475995145,5cyclictest317539-21kworker/u8:1+flush-179:021:35:220
277475995143,5cyclictest350894-21kworker/u8:3+events_unbound23:10:360
277475995139,7cyclictest330459-21kthreadd21:53:590
4670791500,13ptp4l441ktimers/319:10:013
277480995045,3cyclictest321172-21kworker/u8:3+events_unbound21:25:281
277475995044,4cyclictest309564-21kworker/u8:3+events_unbound21:00:220
277475995041,4cyclictest350894-21kworker/u8:3+events_unbound23:05:260
277481994937,10cyclictest0-21swapper/221:50:002
277480994842,5cyclictest325952-21kworker/u8:0+flush-179:021:45:191
277480994842,5cyclictest271-21systemd-journal21:10:291
277480994842,5cyclictest271-21systemd-journal21:10:291
277480994840,5cyclictest277367-21latency_hist19:10:031
277480994839,7cyclictest321172-21kworker/u8:3+flush-179:021:45:001
277480994741,4cyclictest283739-21kworker/u8:3+flush-179:020:00:001
277480994741,4cyclictest283739-21kworker/u8:3+flush-179:020:00:001
277480994740,5cyclictest371616-21kworker/u8:1+events_unbound00:10:241
277480994739,6cyclictest357223-21latency_hist23:15:001
277480994738,7cyclictest271-21systemd-journal00:25:001
277480994641,4cyclictest283739-21kworker/u8:3+events_unbound20:15:281
277480994640,5cyclictest309564-21kworker/u8:3+flush-179:021:00:231
277480994634,9cyclictest293534-21kworker/u8:0+flush-179:020:00:591
277480994633,10cyclictest335780-21apt-get22:10:011
277480994616,28cyclictest723-21Xorg20:05:591
277475994642,3cyclictest279016-21kworker/u8:2+flush-179:019:35:260
277475994637,6cyclictest337611-21latency_hist22:15:010
277480994539,4cyclictest371616-21kworker/u8:1+flush-179:000:00:221
277480994538,5cyclictest327334-21kworker/u8:2+events_unbound22:10:031
27748099452,38cyclictest329140-21apt-get21:50:161
277480994441,2cyclictest0-21swapper/119:15:591
277480994434,7cyclictest362450-21kworker/u8:3+flush-179:023:45:211
277475994437,4cyclictest321172-21kworker/u8:3+flush-179:021:40:230
277480994338,4cyclictest301572-21kworker/u8:0+flush-179:020:40:271
277475994335,6cyclictest309511-21kworker/u8:0+events_unbound20:55:180
27747599432,39cyclictest301599-21turbostat20:29:560
27747599430,2cyclictest723-21Xorg00:28:000
277480994236,4cyclictest362450-21kworker/u8:3+flush-179:023:45:011
27748099420,40cyclictest723-21Xorg20:38:591
277475994235,6cyclictest316132-21kworker/u8:0+events_unbound21:25:210
277475994235,5cyclictest283739-21kworker/u8:3+events_unbound19:33:290
277475994235,5cyclictest283739-21kworker/u8:3+events_unbound19:33:280
27748399410,39cyclictest307965-21turbostat20:45:003
277480994135,4cyclictest354309-21apt-get23:05:141
277480994135,4cyclictest301572-21kworker/u8:0+flush-179:020:30:191
277480994131,7cyclictest339227-21latency_hist22:20:001
27747599410,39cyclictest315907-21turbostat21:14:550
27747599410,39cyclictest315907-21turbostat21:14:550
277483994038,1cyclictest0-21swapper/320:26:593
277480994037,2cyclictest374824-21kworker/u8:0+events_unbound00:15:191
277480994035,4cyclictest346103-21kworker/u8:2+flush-179:023:00:311
277480994034,5cyclictest321172-21kworker/u8:3+events_unbound22:00:011
277480994033,5cyclictest283739-21kworker/u8:3+flush-179:019:40:171
277475994035,4cyclictest354088-21kworker/u8:0+flush-179:023:15:240
277483993933,5cyclictest282196-21turbostat19:25:003
27748399390,4cyclictest333934-21turbostat22:05:013
27748399390,38cyclictest322536-21turbostat21:30:003
27748399390,38cyclictest295181-21turbostat20:05:003
27748199390,36cyclictest0-21swapper/220:14:552
277480993935,3cyclictest350894-21kworker/u8:3+flush-179:023:20:271
277480993932,5cyclictest261321-21kworker/u8:1+flush-179:019:25:241
27748099390,38cyclictest0-21swapper/122:26:001
277483993833,4cyclictest339208-21turbostat22:20:003
277483993833,4cyclictest301599-21turbostat20:25:013
277483993832,5cyclictest379574-21turbostat00:25:003
277481993834,2cyclictest0-21swapper/222:09:562
277480993833,4cyclictest279016-21kworker/u8:2+flush-179:019:40:001
27748099380,1cyclictest346103-21kworker/u8:2+i91522:54:591
277475993833,4cyclictest283739-21kworker/u8:3+events_unbound20:00:230
277483993735,1cyclictest0-21swapper/320:34:593
27748399370,36cyclictest0-21swapper/320:39:593
27748399370,36cyclictest0-21swapper/319:33:003
27748399370,36cyclictest0-21swapper/319:32:593
277481993734,2cyclictest0-21swapper/221:13:592
277481993734,2cyclictest0-21swapper/221:13:592
277480993732,4cyclictest298371-21kworker/u8:2+flush-179:020:20:181
277480993728,7cyclictest277346-21kworker/u8:0+events_unbound19:20:221
277480993725,9cyclictest340807-21latency_hist22:25:001
277483993634,1cyclictest0-21swapper/321:44:003
277483993631,4cyclictest291975-21turbostat19:55:003
27748399360,35cyclictest0-21swapper/321:38:593
27748399360,35cyclictest0-21swapper/321:22:593
27748399360,35cyclictest0-21swapper/300:34:553
27748399360,1cyclictest0-21swapper/323:21:593
27748399360,1cyclictest0-21swapper/321:16:003
277481993633,2cyclictest0-21swapper/223:05:582
277481993627,8cyclictest0-21swapper/221:07:592
277483993533,1cyclictest0-21swapper/323:41:593
277483993533,1cyclictest0-21swapper/322:47:583
27748399350,34cyclictest0-21swapper/323:29:563
27748199350,1cyclictest0-21swapper/223:30:592
27748199350,1cyclictest0-21swapper/219:24:562
277480993530,3cyclictest352470-21plymouth22:59:591
277483993432,1cyclictest0-21swapper/322:31:593
277481993430,2cyclictest0-21swapper/222:16:002
27748199340,27cyclictest0-21swapper/222:25:022
27748199340,26cyclictest0-21swapper/219:12:592
277480993427,5cyclictest354088-21kworker/u8:0+events_unbound23:15:281
277483993332,1cyclictest0-21swapper/321:02:593
277483993331,1cyclictest0-21swapper/323:18:583
277483993331,1cyclictest0-21swapper/321:45:593
277483993330,2cyclictest0-21swapper/323:57:593
277483993328,4cyclictest357231-21turbostat23:15:013
277483993328,4cyclictest312720-21turbostat20:59:593
27748399330,4cyclictest324122-21turbostat21:35:003
277481993330,2cyclictest0-21swapper/223:55:002
277481993330,2cyclictest0-21swapper/223:10:592
277481993330,2cyclictest0-21swapper/200:03:582
277483993230,1cyclictest0-21swapper/323:47:593
277483993230,1cyclictest0-21swapper/319:59:553
277483993230,1cyclictest0-21swapper/319:59:553
277483993230,1cyclictest0-21swapper/300:11:593
277481993230,1cyclictest0-21swapper/220:26:592
27748099320,30cyclictest0-21swapper/100:05:141
277483993126,4cyclictest340812-21turbostat22:25:013
277481993128,2cyclictest0-21swapper/220:24:552
277481993128,2cyclictest0-21swapper/219:31:002
277481993128,2cyclictest0-21swapper/219:30:592
277481993127,2cyclictest0-21swapper/219:57:592
277481993127,2cyclictest0-21swapper/219:57:592
277480993128,2cyclictest345845-21perf22:39:591
277483993028,1cyclictest0-21swapper/319:39:543
277481993028,1cyclictest0-21swapper/221:31:592
277481993027,2cyclictest0-21swapper/200:31:592
27748199300,2cyclictest0-21swapper/223:39:562
277480993025,4cyclictest309564-21kworker/u8:3+flush-179:021:05:261
277480993024,5cyclictest383567-21latency00:35:221
277483992927,1cyclictest0-21swapper/322:05:013
277483992926,2cyclictest0-21swapper/319:25:153
277483992926,2cyclictest0-21swapper/300:00:143
27748399290,28cyclictest0-21swapper/321:05:163
27748399290,28cyclictest0-21swapper/320:50:163
27748399290,21cyclictest0-21swapper/321:15:023
27748399290,21cyclictest0-21swapper/321:15:013
277481992926,2cyclictest0-21swapper/221:55:142
277481992926,2cyclictest0-21swapper/221:50:162
277481992926,2cyclictest0-21swapper/221:25:592
277481992926,2cyclictest0-21swapper/221:00:142
27748199290,2cyclictest0-21swapper/222:45:162
27748199290,2cyclictest0-21swapper/219:45:012
27748199290,28cyclictest0-21swapper/200:05:142
27748199290,27cyclictest0-21swapper/223:40:152
27748199290,27cyclictest0-21swapper/221:15:012
27748199290,27cyclictest0-21swapper/220:40:182
277480992924,4cyclictest277346-21kworker/u8:0+flush-179:019:10:211
4670791280,6ptp4l441ktimers/319:50:013
4670791280,3ptp4l61690irq/136-eno1-TxRx-020:09:553
277483992826,1cyclictest0-21swapper/322:10:043
277483992826,1cyclictest0-21swapper/320:10:173
27748399280,27cyclictest0-21swapper/300:35:163
27748399280,26cyclictest0-21swapper/323:00:153
27748399280,26cyclictest0-21swapper/322:55:153
277481992826,1cyclictest0-21swapper/222:50:132
277481992825,2cyclictest0-21swapper/223:45:132
277481992825,2cyclictest0-21swapper/221:21:592
277481992825,2cyclictest0-21swapper/220:15:172
27748199280,2cyclictest0-21swapper/219:45:012
27748199280,26cyclictest0-21swapper/220:35:152
4670791270,6ptp4l441ktimers/323:10:013
27748399270,26cyclictest0-21swapper/321:54:593
27748399270,1cyclictest0-21swapper/323:30:153
277481992724,2cyclictest0-21swapper/220:01:592
27748199270,25cyclictest0-21swapper/200:36:592
4670791260,5ptp4l61690irq/136-eno1-TxRx-021:59:553
4670791260,4ptp4l441ktimers/322:40:003
4670791260,4ptp4l441ktimers/300:20:003
27748399260,2cyclictest0-21swapper/300:29:593
27748399260,25cyclictest0-21swapper/323:51:003
27748399260,24cyclictest0-21swapper/319:10:153
277481992623,2cyclictest0-21swapper/221:41:592
277481992623,2cyclictest0-21swapper/221:35:582
277481992623,2cyclictest0-21swapper/219:53:002
277481992623,2cyclictest0-21swapper/219:37:592
277481992623,2cyclictest0-21swapper/219:29:552
277481992623,2cyclictest0-21swapper/219:16:292
277481992623,2cyclictest0-21swapper/200:23:592
27748199260,24cyclictest0-21swapper/223:21:592
27748199260,24cyclictest0-21swapper/200:18:592
202199260,16rtkit-daemon3244400-21munin-node19:05:160
4670791250,4ptp4l441ktimers/320:50:013
277483992519,5cyclictest374802-21turbostat00:10:013
27748399250,24cyclictest0-21swapper/319:18:073
277481992523,1cyclictest0-21swapper/200:14:592
27748199250,2cyclictest0-21swapper/220:58:592
277480992521,2cyclictest723-21Xorg20:47:591
4670791240,3ptp4l61690irq/136-eno1-TxRx-022:29:553
4670791240,3ptp4l441ktimers/320:19:553
277483992422,1cyclictest0-21swapper/322:52:593
277483992422,1cyclictest0-21swapper/322:43:593
277481992420,2cyclictest0-21swapper/222:45:002
4670791230,3ptp4l61690irq/136-eno1-TxRx-019:44:553
277481992219,2cyclictest0-21swapper/222:00:032
277481992119,1cyclictest0-21swapper/220:34:592
277481992118,2cyclictest0-21swapper/223:00:162
277481992018,1cyclictest0-21swapper/223:19:592
277481992017,2cyclictest0-21swapper/223:59:552
277481992016,2cyclictest0-21swapper/223:28:592
277481991916,2cyclictest0-21swapper/220:09:002
277481991512,2cyclictest0-21swapper/200:30:002
27748199129,2cyclictest0-21swapper/220:52:592
27748199107,2cyclictest0-21swapper/220:45:332
2774819996,2cyclictest0-21swapper/222:35:502
2774819930,2cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional