You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 18:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot6.osadl.org (updated Tue Aug 26, 2025 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
414439899124115,6cyclictest34856-21kworker/u8:0+flush-179:011:35:021
414439599121110,7cyclictest39948-21kworker/u8:2+events_unbound12:10:010
414439599119106,9cyclictest12095-21kworker/u8:0+flush-179:011:05:020
414439899117109,5cyclictest16807-21kworker/u8:2+flush-179:010:35:011
414439599117103,11cyclictest28750-21apt-get11:10:140
414439899109102,5cyclictest4171684-21kworker/u8:1+events_unbound08:40:251
414439899108101,5cyclictest4180750-21kworker/u8:2+flush-179:009:30:011
41443989910799,5cyclictest5397-21kworker/u8:0+flush-179:010:05:021
41443989910690,6cyclictest4180750-21kworker/u8:2+events_unbound09:30:011
41443959910495,7cyclictest44951-21apt-get12:00:120
41443989910291,9cyclictest20275-21latency_hist10:45:021
41443989910291,9cyclictest20275-21latency_hist10:45:011
41443989910194,5cyclictest4192673-21kworker/u8:0+events_unbound09:45:241
41443989910189,9cyclictest4171684-21kworker/u8:1+flush-179:008:45:201
41443959910093,5cyclictest13677-21kworker/u8:3+flush-179:010:30:220
4144398999993,4cyclictest4180750-21kworker/u8:2+events_unbound09:10:301
4144395999993,4cyclictest39889-21kworker/u8:0+events_unbound11:45:010
4144395999891,5cyclictest4180750-21kworker/u8:2+flush-179:009:05:270
4144395999891,5cyclictest39948-21kworker/u8:2+flush-179:012:10:210
4144398999792,3cyclictest4149876-21kworker/u8:0+flush-179:007:30:211
4144398999790,5cyclictest4192717-21kworker/u8:2+flush-179:010:15:021
4144398999787,7cyclictest4146114-21kworker/u8:2+events_unbound07:15:231
4144398999786,9cyclictest4147745-21apt-get07:20:151
4144395999789,6cyclictest4181324-21kworker/u8:3+events_unbound09:20:260
4144398999585,7cyclictest4192717-21kworker/u8:2+events_unbound09:55:251
4144398999580,5cyclictest12095-21kworker/u8:0+flush-179:010:55:001
4144398999488,5cyclictest4150969-21kworker/u8:2+events_unbound07:45:011
4144398999487,5cyclictest4147499-21kworker/u8:3+flush-179:008:05:241
4144395999482,9cyclictest4150924-21latency_hist07:30:010
4144398999387,4cyclictest4191080-21kworker/u8:3+events_unbound09:40:201
4144398999386,5cyclictest4176740-21kworker/u8:0+flush-179:008:55:161
4144398999385,5cyclictest12095-21kworker/u8:0+events_unbound11:20:151
4144395999384,6cyclictest4165380-21kworker/u8:0+flush-179:008:20:200
4144398999285,6cyclictest4155834-21kworker/u8:0+flush-179:007:55:001
4144398999285,5cyclictest46342-21kworker/u8:1+flush-179:012:35:011
4144398999281,8cyclictest4176741-21/usr/sbin/munin08:50:241
4144398999273,5cyclictest4192673-21kworker/u8:0+events_unbound09:50:241
4144395999285,5cyclictest4165380-21kworker/u8:0+flush-179:008:15:250
4144395999281,8cyclictest502-21rs:main0
4144398999184,5cyclictest4154106-21cp07:36:221
4144395999185,4cyclictest46342-21kworker/u8:1+events_unbound12:25:130
4144395999184,4cyclictest4180750-21kworker/u8:2+events_unbound09:25:250
4144398999080,6cyclictest12095-21kworker/u8:0+flush-179:010:45:201
4144395999082,5cyclictest4171684-21kworker/u8:1+events_unbound08:40:210
4144398998984,4cyclictest4192717-21kworker/u8:2+events_unbound10:10:241
4144398998980,7cyclictest47879-21latency_hist12:10:011
4144395998982,5cyclictest4180750-21kworker/u8:2+flush-179:009:30:050
4144395998980,7cyclictest4147499-21kworker/u8:3+flush-179:007:40:220
4144395998980,6cyclictest4176740-21kworker/u8:0+flush-179:008:55:010
4144398998882,5cyclictest21844-21kworker/u8:2+flush-179:011:00:001
4144398998882,4cyclictest39948-21kworker/u8:2+flush-179:012:10:251
4144398998881,5cyclictest4176740-21kworker/u8:0+flush-179:009:05:011
4144398998880,6cyclictest4180750-21kworker/u8:2+events_unbound09:05:201
4144395998881,5cyclictest39948-21kworker/u8:2+flush-179:011:55:000
4144395998878,8cyclictest4162427-21apt-get08:05:140
4144398998782,4cyclictest4180750-21kworker/u8:2+flush-179:009:15:231
4144398998782,4cyclictest4180750-21kworker/u8:2+flush-179:009:15:221
4144398998781,5cyclictest4191080-21kworker/u8:3+events_unbound09:35:251
4144398998773,5cyclictest33265-21kworker/u8:1+flush-179:011:40:221
4144395998775,9cyclictest4140665-21kworker/u8:1+flush-179:007:10:210
4144398998679,5cyclictest26649-21latency_hist11:05:011
4144395998676,7cyclictest51323-21latency_hist12:20:010
4144398998579,5cyclictest39889-21kworker/u8:0+events_unbound11:55:201
4144398998577,6cyclictest4167193-21/usr/sbin/munin08:20:241
4144400998453,27cyclictest0-21swapper/211:41:592
4144398998478,4cyclictest4140665-21kworker/u8:1+flush-179:007:10:211
4144398998471,11cyclictest6767-21apt-get10:05:001
4144395998476,6cyclictest4176740-21kworker/u8:0+flush-179:008:55:220
414439599841,80cyclictest1-21systemd12:22:360
4144398998377,4cyclictest4165380-21kworker/u8:0+flush-179:008:15:161
4144398998376,5cyclictest4147499-21kworker/u8:3+events_unbound08:25:241
4144398998376,4cyclictest4147499-21kworker/u8:3+events_unbound08:35:151
4144398998367,13cyclictest56356-21apt-get12:35:121
4144398998367,13cyclictest56356-21apt-get12:35:121
4144398998275,5cyclictest49193-21kworker/u8:0+flush-179:012:15:221
4144398998275,5cyclictest4192717-21kworker/u8:2+events_unbound10:20:141
4144398998273,7cyclictest601706-21systemd-journal10:35:011
4144395998275,5cyclictest12095-21kworker/u8:0+flush-179:010:55:230
4144398998175,4cyclictest49193-21kworker/u8:0+flush-179:012:20:161
4144398998175,4cyclictest4150969-21kworker/u8:2+flush-179:008:00:231
4144395998176,4cyclictest56083-21kworker/u8:0+flush-179:012:35:260
4144395998176,4cyclictest56083-21kworker/u8:0+flush-179:012:35:260
4144395998174,5cyclictest5398-21kworker/u8:3+flush-179:010:00:230
4144395998171,7cyclictest12095-21kworker/u8:0+flush-179:010:35:280
4144398998075,4cyclictest33265-21kworker/u8:1+events_unbound11:25:251
4144395998067,10cyclictest36463-21apt-get11:35:000
4144395997870,6cyclictest4147499-21kworker/u8:3+events_unbound07:50:130
4144398997770,5cyclictest4147499-21kworker/u8:3+flush-179:007:45:161
4144398997770,5cyclictest4147499-21kworker/u8:3+flush-179:007:45:151
4144398997764,4cyclictest4165380-21kworker/u8:0+events_unbound08:30:221
4144398997764,4cyclictest39889-21kworker/u8:0+events_unbound11:45:291
4144395997765,9cyclictest4192696-21latency_hist09:40:010
4144398997663,5cyclictest39889-21kworker/u8:0+events_unbound11:50:151
4144395997670,5cyclictest4150969-21kworker/u8:2+flush-179:008:00:260
4144395997670,4cyclictest4165380-21kworker/u8:0+events_unbound08:30:120
4144395997670,4cyclictest12095-21kworker/u8:0+flush-179:010:40:170
4144395997670,4cyclictest12095-21kworker/u8:0+flush-179:010:40:170
4144395997664,8cyclictest4194297-21latency_hist09:45:000
4144395997662,11cyclictest601706-21systemd-journal10:20:000
4144398997567,6cyclictest4150969-21kworker/u8:2+events_unbound08:10:211
4144398997467,6cyclictest39948-21kworker/u8:2+events_unbound12:00:201
4144395997466,6cyclictest4159227-21apt-get07:55:130
4144402997365,6cyclictest4160598-21turbostat08:00:003
4144398997367,4cyclictest33265-21kworker/u8:1+flush-179:011:30:221
4144395997367,4cyclictest34856-21kworker/u8:0+flush-179:011:35:150
4144398997264,6cyclictest4180750-21kworker/u8:2+flush-179:009:20:211
4144402997169,1cyclictest0-21swapper/310:29:543
4144398997064,4cyclictest21844-21kworker/u8:2+events_unbound11:10:201
4144398997063,4cyclictest46342-21kworker/u8:1+events_unbound12:25:201
4144395997061,6cyclictest4156054-21apt-get07:45:120
4144395997061,6cyclictest4156054-21apt-get07:45:110
4144395997060,7cyclictest4192717-21kworker/u8:2+flush-179:010:10:210
4144395996964,3cyclictest52969-21kworker/u8:2+events_unbound12:35:010
4144395996962,6cyclictest13656-21turbostat10:29:540
4144395996960,7cyclictest601706-21systemd-journal09:50:000
4144395996863,2cyclictest4150969-21kworker/u8:2+events_unbound08:10:230
4144398996761,4cyclictest4147499-21kworker/u8:3+flush-179:007:55:151
4144395996558,5cyclictest21844-21kworker/u8:2+flush-179:010:50:210
4144395996556,4cyclictest39889-21kworker/u8:0+flush-179:011:55:210
4144395996555,7cyclictest4180750-21kworker/u8:2+flush-179:009:10:250
4144395996457,6cyclictest12095-21kworker/u8:0+flush-179:011:20:210
4144395996454,8cyclictest4147471-21latency_hist07:20:010
4144395996357,4cyclictest4150969-21kworker/u8:2+events_unbound07:35:230
4144395996350,10cyclictest4168799-21apt-get08:25:120
4144400996231,2cyclictest0-21swapper/210:13:002
4144395996253,5cyclictest12095-21kworker/u8:0+flush-179:010:45:220
4144395996250,9cyclictest4173326-21latency_hist08:40:000
4144395996156,3cyclictest4192717-21kworker/u8:2+flush-179:010:20:130
4144395995852,4cyclictest4176740-21kworker/u8:0+flush-179:009:00:180
4144398995752,4cyclictest4140665-21kworker/u8:1+flush-179:007:25:161
4144395995754,2cyclictest4181324-21kworker/u8:3+events_unbound09:16:370
4144395995754,2cyclictest4181324-21kworker/u8:3+events_unbound09:16:370
4144398995551,3cyclictest12095-21kworker/u8:0+flush-179:011:15:161
4144398995546,3cyclictest13677-21kworker/u8:3+events_unbound10:25:211
4144395995550,4cyclictest4192717-21kworker/u8:2+flush-179:009:55:360
4144395995548,5cyclictest12095-21kworker/u8:0+events_unbound11:15:150
4144395995446,5cyclictest41499-21latency_hist11:50:010
4144395995344,6cyclictest21844-21kworker/u8:2+events_unbound11:25:130
4144395995247,4cyclictest26110-21kworker/u8:1+flush-179:011:00:270
4144395995143,6cyclictest4192673-21kworker/u8:0+flush-179:009:50:200
4144395995045,4cyclictest4171684-21kworker/u8:1+flush-179:008:45:230
4505291480,13ptp4l61790irq/134-eno1-TxRx-007:09:563
4144400994642,2cyclictest0-21swapper/212:10:592
4144400994542,2cyclictest0-21swapper/212:22:002
4144395994440,3cyclictest4149876-21kworker/u8:0+events_unbound07:30:160
4144395994438,5cyclictest5397-21kworker/u8:0+events_unbound10:06:360
4144402994239,2cyclictest0-21swapper/312:19:593
4144402994239,2cyclictest0-21swapper/307:49:593
4144402994239,2cyclictest0-21swapper/307:49:593
4144398994233,7cyclictest12095-21kworker/u8:0+flush-179:011:05:211
4144402994032,6cyclictest4154201-21turbostat07:39:593
414440299390,4cyclictest46309-21turbostat12:05:003
414440099380,36cyclictest0-21swapper/210:55:592
414440099380,35cyclictest0-21swapper/210:24:592
414440099380,35cyclictest0-21swapper/207:51:592
4144402993735,1cyclictest0-21swapper/311:43:593
4144402993734,2cyclictest0-21swapper/308:15:003
414440299370,36cyclictest0-21swapper/307:34:593
4144400993735,1cyclictest0-21swapper/210:54:592
4144400993734,2cyclictest0-21swapper/209:44:582
4144400993734,2cyclictest0-21swapper/209:09:002
414440099370,35cyclictest0-21swapper/207:26:592
414440099370,29cyclictest0-21swapper/208:45:592
4144402993634,1cyclictest0-21swapper/311:26:593
4144402993634,1cyclictest0-21swapper/308:54:553
4144402993634,1cyclictest0-21swapper/308:08:583
414440299360,35cyclictest0-21swapper/310:10:003
414440299360,35cyclictest0-21swapper/309:57:593
414440299360,35cyclictest0-21swapper/308:04:553
414440299360,35cyclictest0-21swapper/307:19:553
4144400993633,2cyclictest0-21swapper/212:00:002
414440099360,1cyclictest0-21swapper/212:02:592
4144402993533,1cyclictest0-21swapper/309:29:553
414440299350,1cyclictest0-21swapper/307:41:593
4144402993431,2cyclictest0-21swapper/308:39:583
4144400993432,1cyclictest0-21swapper/212:29:592
4144400993431,2cyclictest0-21swapper/211:46:592
4144400993431,2cyclictest0-21swapper/211:30:592
4144400993430,2cyclictest0-21swapper/212:09:592
4144400993330,2cyclictest0-21swapper/208:44:592
4144402993230,1cyclictest0-21swapper/312:24:553
4144402993230,1cyclictest0-21swapper/310:36:003
4144402993230,1cyclictest0-21swapper/307:28:593
4144400993229,2cyclictest0-21swapper/210:09:562
4144400993229,2cyclictest0-21swapper/209:56:592
4144400993229,2cyclictest0-21swapper/209:51:002
4144400993229,2cyclictest0-21swapper/207:42:582
4144400993229,2cyclictest0-21swapper/207:34:002
4505291310,4ptp4l441ktimers/310:35:003
4505291310,11ptp4l61790irq/134-eno1-TxRx-010:25:013
4144402993129,1cyclictest0-21swapper/312:09:593
4144402993129,1cyclictest0-21swapper/309:18:593
4144402993129,1cyclictest0-21swapper/309:18:593
414440099310,29cyclictest0-21swapper/208:30:142
4505291300,5ptp4l441ktimers/307:14:553
4144402993028,1cyclictest0-21swapper/310:04:563
4144402993027,2cyclictest0-21swapper/310:55:143
414440299300,29cyclictest0-21swapper/310:47:593
4144400993027,2cyclictest0-21swapper/211:39:592
4144400993027,2cyclictest0-21swapper/208:04:592
414440099300,2cyclictest0-21swapper/208:05:162
414440099300,29cyclictest0-21swapper/207:45:142
414440099300,29cyclictest0-21swapper/207:45:142
414440099300,28cyclictest0-21swapper/210:15:162
414440099300,28cyclictest0-21swapper/209:10:172
4144402992927,1cyclictest0-21swapper/312:30:183
4144402992927,1cyclictest0-21swapper/311:58:593
414440299290,27cyclictest0-21swapper/311:35:033
414440299290,27cyclictest0-21swapper/311:00:173
414440299290,27cyclictest0-21swapper/308:55:143
4144400992926,2cyclictest0-21swapper/209:00:142
4144400992926,2cyclictest0-21swapper/207:24:552
4144400992925,2cyclictest0-21swapper/209:48:582
414440099290,2cyclictest0-21swapper/209:30:022
414440099290,28cyclictest0-21swapper/211:05:032
414440099290,27cyclictest0-21swapper/211:52:592
414440099290,27cyclictest0-21swapper/211:20:132
414440099290,27cyclictest0-21swapper/210:25:142
414440099290,27cyclictest0-21swapper/209:35:142
414440099290,27cyclictest0-21swapper/209:15:142
414440099290,27cyclictest0-21swapper/209:15:132
414440099290,27cyclictest0-21swapper/207:55:152
4144398992922,5cyclictest4144262-21latency_hist07:10:011
4505291280,7ptp4l61790irq/134-eno1-TxRx-012:40:013
4505291280,7ptp4l61790irq/134-eno1-TxRx-012:40:013
4505291280,7ptp4l61790irq/134-eno1-TxRx-011:15:003
4505291280,7ptp4l441ktimers/309:05:013
4505291280,7ptp4l441ktimers/308:45:003
4505291280,6ptp4l441ktimers/310:45:003
4505291280,6ptp4l441ktimers/310:45:003
4144402992826,1cyclictest0-21swapper/309:30:023
4144402992825,2cyclictest0-21swapper/310:50:593
414440299280,2cyclictest0-21swapper/307:50:133
414440299280,27cyclictest0-21swapper/311:31:593
414440299280,27cyclictest0-21swapper/311:05:023
414440299280,27cyclictest0-21swapper/307:20:163
414440299280,26cyclictest0-21swapper/309:20:143
414440299280,26cyclictest0-21swapper/309:05:183
4144400992825,2cyclictest0-21swapper/211:25:142
4144400992825,2cyclictest0-21swapper/210:35:012
4144400992825,2cyclictest0-21swapper/208:15:162
414440099280,2cyclictest0-21swapper/212:15:142
414440099280,27cyclictest0-21swapper/210:35:012
414440099280,25cyclictest0-21swapper/211:04:552
4505291270,7ptp4l441ktimers/308:25:003
4505291270,6ptp4l61790irq/134-eno1-TxRx-009:49:553
4505291270,6ptp4l61790irq/134-eno1-TxRx-008:50:013
4144402992725,1cyclictest0-21swapper/312:10:013
4144400992725,1cyclictest0-21swapper/210:04:592
4144400992724,2cyclictest0-21swapper/212:37:002
4144400992724,2cyclictest0-21swapper/212:37:002
4144400992724,2cyclictest0-21swapper/209:20:592
4505291260,6ptp4l441ktimers/309:40:003
4505291260,6ptp4l441ktimers/308:34:563
4505291260,5ptp4l441ktimers/312:30:003
4505291260,5ptp4l441ktimers/311:54:563
4505291260,5ptp4l441ktimers/309:45:003
4505291260,5ptp4l441ktimers/309:14:553
4144402992624,1cyclictest0-21swapper/308:17:003
4144400992623,2cyclictest0-21swapper/211:13:002
4144400992623,2cyclictest0-21swapper/210:44:592
4144400992623,2cyclictest0-21swapper/210:44:582
4144400992623,2cyclictest0-21swapper/208:50:152
414440099260,2cyclictest0-21swapper/207:35:142
414440099260,24cyclictest0-21swapper/208:20:142
414440099260,24cyclictest0-21swapper/207:10:092
4505291250,5ptp4l441ktimers/310:19:553
4144402992523,1cyclictest0-21swapper/309:51:593
4144402992523,1cyclictest0-21swapper/308:29:563
4144402992522,2cyclictest0-21swapper/311:20:143
414440299250,24cyclictest0-21swapper/311:15:153
414440299250,24cyclictest0-21swapper/310:14:583
4144400992521,2cyclictest0-21swapper/208:57:592
4144400992421,2cyclictest0-21swapper/208:12:592
4505291230,4ptp4l61790irq/134-eno1-TxRx-011:49:553
4144400992319,2cyclictest0-21swapper/209:25:592
4144400992219,2cyclictest0-21swapper/208:36:592
4144400992018,1cyclictest0-21swapper/210:45:592
4144395992015,3cyclictest0-21swapper/007:10:010
4144400991917,1cyclictest0-21swapper/208:29:592
4144400991916,2cyclictest0-21swapper/207:19:002
4144400991512,2cyclictest0-21swapper/211:16:592
4144400991411,2cyclictest0-21swapper/212:34:552
41444009930,2cyclictest0-21swapper/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional