You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-02-11 - 03:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Mon Feb 10, 2025 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
69528499117102,11cyclictest766070-21apt-get11:15:000
69528699111102,6cyclictest785692-21kworker/u8:1+flush-179:012:30:151
69528699110101,6cyclictest749128-21kworker/u8:1+events_unbound10:20:131
6952869910991,10cyclictest710066-21/usr/sbin/munin07:55:251
6952869910893,5cyclictest713799-21kworker/u8:1+events_unbound08:30:231
6952869910799,6cyclictest762616-21kworker/u8:2+events_unbound11:20:181
69528699107101,4cyclictest729980-21kworker/u8:0+flush-179:009:05:211
6952849910699,5cyclictest719422-21kworker/u8:0+flush-179:008:25:200
6952849910699,5cyclictest700018-21kworker/u8:0+events_unbound07:55:140
6952869910595,7cyclictest700018-21kworker/u8:0+events_unbound07:50:231
6952869910595,7cyclictest700018-21kworker/u8:0+events_unbound07:50:231
6952869910493,7cyclictest744340-21kworker/u8:3+events_unbound09:55:191
6952849910188,10cyclictest722818-21apt-get08:35:120
695286999993,4cyclictest774583-21kworker/u8:0+events_unbound11:55:241
695286999992,5cyclictest689390-21kworker/u8:2+events_unbound07:25:021
695284999991,5cyclictest700018-21kworker/u8:0+events_unbound07:55:010
695284999991,5cyclictest700018-21kworker/u8:0+events_unbound07:55:000
695284999891,5cyclictest767876-21kworker/u8:3+events_unbound11:30:000
695284999889,6cyclictest766017-21kworker/u8:0+events_unbound11:15:140
695286999787,7cyclictest788324-21latency_hist12:40:011
695286999689,6cyclictest774583-21kworker/u8:0+events_unbound11:50:021
695286999681,5cyclictest732359-21kworker/u8:2+events_unbound09:25:011
695286999590,4cyclictest705119-21kworker/u8:3+flush-179:008:00:261
695286999583,9cyclictest3105085-21systemd-journal09:55:001
695286999487,5cyclictest705119-21kworker/u8:3+flush-179:007:45:211
695284999488,3cyclictest732359-21kworker/u8:2+events_unbound09:40:210
695286999386,6cyclictest732359-21kworker/u8:2+events_unbound09:30:011
695284999378,12cyclictest739005-21latency_hist09:30:010
695286999283,7cyclictest725791-21kworker/u8:3+events_unbound08:45:211
695284999285,5cyclictest696862-21kworker/u8:3+flush-179:007:15:230
695286999079,4cyclictest748377-21kworker/u8:0+events_unbound10:00:201
695286998983,4cyclictest762616-21kworker/u8:2+flush-179:011:30:141
695286998982,5cyclictest732359-21kworker/u8:2+flush-179:009:20:231
695286998982,5cyclictest689390-21kworker/u8:2+events_unbound07:20:141
695286998874,10cyclictest703701-21apt-get07:35:111
695284998879,6cyclictest744340-21kworker/u8:3+events_unbound10:00:150
695286998781,4cyclictest778083-21kworker/u8:2+events_unbound12:00:151
695286998781,4cyclictest778083-21kworker/u8:2+events_unbound12:00:151
695286998678,6cyclictest749128-21kworker/u8:1+events_unbound10:15:211
695286998477,5cyclictest713799-21kworker/u8:1+flush-179:008:20:141
695284998476,6cyclictest729980-21kworker/u8:0+flush-179:009:00:010
695284998476,6cyclictest729980-21kworker/u8:0+flush-179:009:00:000
695284998472,9cyclictest724423-21apt-get08:40:130
695284998472,9cyclictest3105085-21systemd-journal09:15:010
695284998471,10cyclictest757835-21apt-get10:40:110
695286998379,3cyclictest749128-21kworker/u8:1+events_unbound10:40:251
695286998373,4cyclictest774583-21kworker/u8:0+events_unbound12:15:221
695284998268,6cyclictest700018-21kworker/u8:0+events_unbound07:25:140
695286998174,5cyclictest748377-21kworker/u8:0+flush-179:010:10:171
695284998174,4cyclictest752502-21kworker/u8:3+flush-179:010:35:210
695286998075,4cyclictest679567-21kworker/u8:1+flush-179:007:10:251
695284998073,4cyclictest739051-21kworker/u8:0+events_unbound09:50:190
695284998069,7cyclictest502-21rs:main0
695284997972,5cyclictest689390-21kworker/u8:2+flush-179:007:20:210
695284997967,9cyclictest754385-21latency_hist10:30:010
695284997966,6cyclictest780100-21kworker/u8:3+events_unbound12:10:200
695286997871,4cyclictest725791-21kworker/u8:3+events_unbound08:50:251
695284997864,7cyclictest770007-21kworker/u8:1+events_unbound11:55:150
695286997771,4cyclictest732359-21kworker/u8:2+events_unbound09:15:221
695284997768,6cyclictest744340-21kworker/u8:3+flush-179:009:55:150
695286997671,4cyclictest778083-21kworker/u8:2+events_unbound12:20:211
695286997666,8cyclictest3105085-21systemd-journal07:20:001
695286997664,4cyclictest732359-21kworker/u8:2+events_unbound09:40:221
695286997567,6cyclictest767876-21kworker/u8:3+events_unbound11:50:001
695286997469,4cyclictest766017-21kworker/u8:0+flush-179:011:15:211
695284997465,5cyclictest762616-21kworker/u8:2+flush-179:011:30:160
695286997365,3cyclictest713799-21kworker/u8:1+events_unbound08:40:231
695284997366,5cyclictest778083-21kworker/u8:2+flush-179:012:00:170
695284997366,5cyclictest778083-21kworker/u8:2+flush-179:012:00:160
695284997366,5cyclictest778083-21kworker/u8:2+events_unbound12:30:010
695284997363,7cyclictest700018-21kworker/u8:0+events_unbound07:30:250
695284997362,8cyclictest713799-21kworker/u8:1+events_unbound08:15:200
695286997267,3cyclictest749128-21kworker/u8:1+events_unbound10:30:121
695286997266,4cyclictest767876-21kworker/u8:3+flush-179:011:40:211
695284997265,5cyclictest759441-21kworker/u8:3+flush-179:010:50:220
695286997166,4cyclictest749128-21kworker/u8:1+flush-179:010:50:001
695286997165,4cyclictest729980-21kworker/u8:0+events_unbound09:00:141
695286997162,6cyclictest780109-21latency_hist12:10:011
695284997164,5cyclictest736898-21kworker/u8:1+flush-179:009:20:260
695284997158,8cyclictest739051-21kworker/u8:0+events_unbound09:35:140
695284997158,10cyclictest732666-21diskstats09:05:160
695286997064,4cyclictest749128-21kworker/u8:1+events_unbound10:55:181
695286997062,6cyclictest3105085-21systemd-journal10:40:001
695284997065,3cyclictest700018-21kworker/u8:0+flush-179:008:10:010
695284997060,7cyclictest774583-21kworker/u8:0+events_unbound12:15:140
695286996959,7cyclictest702115-21apt-get07:30:161
695286996959,3cyclictest749128-21kworker/u8:1+events_unbound11:00:141
695284996962,5cyclictest762616-21kworker/u8:2+events_unbound11:20:220
695284996960,4cyclictest749128-21kworker/u8:1+events_unbound10:30:140
695286996862,5cyclictest767876-21kworker/u8:3+flush-179:011:35:251
695286996857,8cyclictest695191-21latency_hist07:10:021
695284996859,6cyclictest689390-21kworker/u8:2+events_unbound07:35:250
695284996858,8cyclictest689390-21kworker/u8:2+flush-179:007:50:010
695286996761,4cyclictest733694-21kworker/u8:3+events_unbound09:10:181
695286996757,4cyclictest719422-21kworker/u8:0+events_unbound08:35:161
695284996759,6cyclictest719422-21kworker/u8:0+events_unbound08:50:000
695284996756,8cyclictest739520-21apt-get09:30:070
695284996660,4cyclictest732359-21kworker/u8:2+events_unbound09:15:180
695284996660,4cyclictest713799-21kworker/u8:1+flush-179:008:30:230
695284996657,6cyclictest762616-21kworker/u8:2+events_unbound11:35:190
69528799650,1cyclictest0-21swapper/211:57:592
695284996550,8cyclictest703432-21kworker/u8:1+flush-179:007:40:200
695284996457,5cyclictest748377-21kworker/u8:0+flush-179:010:10:140
695284996456,6cyclictest713799-21kworker/u8:1+flush-179:008:20:220
695284996452,9cyclictest3105085-21systemd-journal08:15:000
695284996050,5cyclictest762574-21kworker/u8:0+events_unbound11:00:240
695286995947,10cyclictest3105085-21systemd-journal10:55:001
695284995954,3cyclictest729980-21kworker/u8:0+events_unbound09:00:190
695284995946,10cyclictest751186-21apt-get10:15:130
695286995854,3cyclictest719422-21kworker/u8:0+flush-179:008:25:201
695286995853,4cyclictest705119-21kworker/u8:3+events_unbound08:05:161
695284995846,10cyclictest695488-21apt-get07:10:120
695286995752,4cyclictest689390-21kworker/u8:2+events_unbound07:43:151
695287995653,2cyclictest0-21swapper/208:46:002
695286995651,4cyclictest762616-21kworker/u8:2+events_unbound11:28:141
695284995651,4cyclictest767876-21kworker/u8:3+events_unbound11:40:180
695284995646,8cyclictest786737-21latency_hist12:35:000
695284995543,9cyclictest753083-21diskstats10:20:140
695286995451,2cyclictest778083-21kworker/u8:2+flush-179:012:10:161
695284995445,6cyclictest744304-21latency_hist09:50:000
695286995348,3cyclictest732359-21kworker/u8:2+events_unbound09:45:141
695284995346,6cyclictest774583-21kworker/u8:0+events_unbound12:35:200
695284995343,5cyclictest725791-21kworker/u8:3+events_unbound08:50:260
695284995341,5cyclictest705119-21kworker/u8:3+events_unbound08:00:230
695286995246,4cyclictest774583-21kworker/u8:0+events_unbound12:25:141
695286995146,4cyclictest713799-21kworker/u8:1+flush-179:008:15:201
695286995139,9cyclictest716196-21latency_hist08:15:021
695286995042,6cyclictest739051-21kworker/u8:0+flush-179:009:40:011
695284995044,5cyclictest774583-21kworker/u8:0+events_unbound11:50:140
695284994836,9cyclictest784889-21latency_hist12:25:010
4505291480,13ptp4l61790irq/134-eno1-TxRx-007:10:003
695286994743,3cyclictest764443-21kworker/u8:4+flush-179:011:10:151
695286994541,3cyclictest725791-21kworker/u8:3+flush-179:008:55:201
695286994541,3cyclictest725791-21kworker/u8:3+flush-179:008:55:201
695286994439,4cyclictest744340-21kworker/u8:3+events_unbound10:08:131
69529099430,42cyclictest0-21swapper/308:12:593
69528799430,41cyclictest0-21swapper/212:12:592
69528499430,2cyclictest722-21Xorg10:08:590
69529099420,1cyclictest0-21swapper/309:19:583
695287994138,2cyclictest0-21swapper/211:10:592
695286994135,4cyclictest722-21Xorg10:28:581
695287994038,1cyclictest0-21swapper/207:14:592
695287994037,2cyclictest0-21swapper/209:50:592
695286994036,3cyclictest762616-21kworker/u8:2+events_unbound11:09:151
695290993937,1cyclictest0-21swapper/310:35:593
695284993934,2cyclictest752577-21kworker/0:0-cgroup_destroy10:50:000
69529099380,36cyclictest0-21swapper/307:25:583
695287993835,2cyclictest0-21swapper/211:07:592
695287993835,2cyclictest0-21swapper/208:31:002
69528799380,35cyclictest0-21swapper/210:12:592
69528799380,35cyclictest0-21swapper/207:46:592
695284993833,3cyclictest764489-21perf11:10:000
695290993735,1cyclictest0-21swapper/309:42:003
69529099370,36cyclictest0-21swapper/312:40:003
69529099370,36cyclictest0-21swapper/310:59:553
695287993734,2cyclictest0-21swapper/212:21:592
695287993734,2cyclictest0-21swapper/210:46:592
695287993734,2cyclictest0-21swapper/209:15:592
695287993734,2cyclictest0-21swapper/207:22:592
695290993634,1cyclictest0-21swapper/312:24:553
695290993634,1cyclictest0-21swapper/310:54:593
695290993634,1cyclictest0-21swapper/309:14:003
695290993633,2cyclictest0-21swapper/312:32:583
69529099360,35cyclictest0-21swapper/312:04:593
69529099360,35cyclictest0-21swapper/312:04:583
69529099360,35cyclictest0-21swapper/308:49:543
69529099360,1cyclictest0-21swapper/311:37:593
69529099360,1cyclictest0-21swapper/311:32:593
69529099360,1cyclictest0-21swapper/310:28:583
695287993633,2cyclictest0-21swapper/212:04:552
695287993633,2cyclictest0-21swapper/212:04:542
695287993633,2cyclictest0-21swapper/209:55:592
695287993633,2cyclictest0-21swapper/208:22:582
695290993533,1cyclictest0-21swapper/311:19:553
695290993533,1cyclictest0-21swapper/309:00:593
695290993528,5cyclictest755980-21turbostat10:35:003
695287993533,1cyclictest0-21swapper/212:05:592
695287993533,1cyclictest0-21swapper/211:20:002
695287993533,1cyclictest0-21swapper/210:45:002
695287993533,1cyclictest0-21swapper/209:10:002
695287993533,1cyclictest0-21swapper/207:34:592
695290993431,2cyclictest0-21swapper/311:55:003
695290993427,4cyclictest714620-21turbostat08:10:013
695287993431,2cyclictest0-21swapper/208:56:002
695287993431,2cyclictest0-21swapper/208:55:592
4505291340,5ptp4l441ktimers/308:30:003
695290993331,1cyclictest0-21swapper/311:56:593
695290993331,1cyclictest0-21swapper/310:05:003
695290993331,1cyclictest0-21swapper/309:20:593
695290993330,2cyclictest0-21swapper/308:36:593
695287993330,2cyclictest0-21swapper/211:33:592
695287993330,2cyclictest0-21swapper/211:22:592
695287993329,2cyclictest0-21swapper/207:57:002
695284993322,10cyclictest0-21swapper/011:50:000
695287993229,2cyclictest0-21swapper/209:36:592
695290993129,1cyclictest0-21swapper/308:59:563
695290993129,1cyclictest0-21swapper/308:59:553
695287993128,2cyclictest0-21swapper/211:52:582
695287993128,2cyclictest0-21swapper/211:41:002
695287993128,2cyclictest0-21swapper/207:38:592
69528799310,29cyclictest0-21swapper/208:15:122
69529099300,29cyclictest0-21swapper/308:20:143
695287993028,1cyclictest0-21swapper/207:19:592
695287993027,2cyclictest0-21swapper/210:35:142
695287993027,2cyclictest0-21swapper/210:20:122
695287993027,2cyclictest0-21swapper/210:01:582
695287993027,2cyclictest0-21swapper/209:30:032
695290992926,2cyclictest0-21swapper/312:18:593
695290992926,2cyclictest0-21swapper/309:35:143
695290992926,2cyclictest0-21swapper/307:10:143
695290992922,5cyclictest781698-21turbostat12:15:003
695287992926,2cyclictest0-21swapper/207:43:002
695287992925,2cyclictest0-21swapper/207:53:592
695287992925,2cyclictest0-21swapper/207:53:592
69528799290,27cyclictest0-21swapper/210:30:592
69528799290,27cyclictest0-21swapper/209:12:592
69528799290,27cyclictest0-21swapper/208:05:152
69528799290,27cyclictest0-21swapper/208:00:132
695284992924,4cyclictest778083-21kworker/u8:2+events_unbound12:05:130
695290992826,1cyclictest0-21swapper/311:10:093
695290992826,1cyclictest0-21swapper/308:40:143
69529099280,26cyclictest0-21swapper/309:55:123
69529099280,26cyclictest0-21swapper/309:25:023
695287992826,1cyclictest0-21swapper/209:49:592
695287992825,2cyclictest0-21swapper/208:35:582
69528799280,26cyclictest0-21swapper/211:50:012
69528799280,26cyclictest0-21swapper/211:00:592
69528799280,26cyclictest0-21swapper/210:55:592
4505291280,6ptp4l441ktimers/309:50:003
695290992725,1cyclictest0-21swapper/310:20:003
695290992725,1cyclictest0-21swapper/309:10:003
695290992725,1cyclictest0-21swapper/307:20:143
69529099270,25cyclictest0-21swapper/310:20:113
695287992725,1cyclictest0-21swapper/210:30:002
695287992724,2cyclictest0-21swapper/207:25:132
69528799270,24cyclictest0-21swapper/208:54:552
69528799270,24cyclictest0-21swapper/208:45:002
4505291270,7ptp4l441ktimers/310:15:003
4505291270,6ptp4l441ktimers/309:55:003
4505291270,6ptp4l441ktimers/308:55:003
4505291270,5ptp4l61790irq/134-eno1-TxRx-012:30:013
4505291270,5ptp4l441ktimers/307:35:013
695290992623,2cyclictest749362-21turbostat10:10:003
695287992622,2cyclictest0-21swapper/211:39:552
69528799260,24cyclictest0-21swapper/208:25:132
4505291260,6ptp4l61790irq/134-eno1-TxRx-011:10:003
4505291260,6ptp4l61790irq/134-eno1-TxRx-007:40:013
4505291260,6ptp4l441ktimers/310:50:003
4505291260,6ptp4l441ktimers/307:54:563
4505291260,6ptp4l441ktimers/307:54:553
4505291260,6ptp4l441ktimers/307:50:013
4505291260,5ptp4l61790irq/134-eno1-TxRx-012:10:003
4505291260,5ptp4l441ktimers/311:49:553
4505291260,5ptp4l441ktimers/311:44:563
4505291260,5ptp4l441ktimers/311:25:003
4505291260,4ptp4l441ktimers/309:35:013
695290992523,1cyclictest0-21swapper/307:18:003
695290992522,2cyclictest0-21swapper/307:57:593
69529099250,24cyclictest0-21swapper/308:19:553
695287992523,1cyclictest0-21swapper/210:09:562
695287992522,2cyclictest0-21swapper/212:16:592
695287992522,2cyclictest0-21swapper/209:20:592
4505291250,4ptp4l441ktimers/308:04:553
4505291240,5ptp4l61790irq/134-eno1-TxRx-008:35:003
4505291240,4ptp4l61790irq/134-eno1-TxRx-011:29:553
4505291240,4ptp4l441ktimers/310:44:553
4505291240,4ptp4l441ktimers/307:44:553
695287992321,1cyclictest0-21swapper/209:45:002
695287992320,2cyclictest0-21swapper/209:25:592
4505291230,4ptp4l61790irq/134-eno1-TxRx-011:04:563
695287992219,2cyclictest0-21swapper/212:36:592
695287992219,1cyclictest0-21swapper/210:50:592
695284991713,3cyclictest0-21swapper/007:10:010
695287991411,2cyclictest0-21swapper/211:27:002
695287991310,2cyclictest0-21swapper/212:25:592
695287991210,1cyclictest0-21swapper/210:19:592
69528799107,2cyclictest0-21swapper/208:14:002
6952879996,2cyclictest0-21swapper/212:34:422
6952879950,2cyclictest0-21swapper/209:00:012
6952879930,2cyclictest0-21swapper/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional