You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 04:24
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Fri Nov 14, 2025 00:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
293418799114103,8cyclictest3027986-40logrotate00:00:000
293419099107101,4cyclictest3026261-21kworker/u8:0+events_unbound00:04:011
2934187999987,9cyclictest2982391-21latency_hist21:40:010
2934187999282,7cyclictest3003611-21latency_hist22:45:000
2934190999078,9cyclictest271-21systemd-journal21:45:021
2934190998782,4cyclictest2934101-21kworker/u8:3+events_unbound19:10:251
2934190998679,5cyclictest2992334-21kworker/u8:2+events_unbound22:20:261
2934187998676,7cyclictest2987685-21latency_hist21:55:010
2934187998576,7cyclictest3005436-21/usr/sbin/munin22:50:260
2934190998278,3cyclictest3001726-21kworker/u8:3+events_unbound22:45:291
2934190998176,4cyclictest3032862-21kworker/u8:1+events_unbound00:20:291
2934190998174,5cyclictest2938848-21kworker/u8:2+events_unbound20:05:031
2934190998168,4cyclictest2964945-21kworker/u8:3+events_unbound20:50:231
2934187998175,4cyclictest3005196-21latency_hist22:50:010
2934190997664,9cyclictest3028011-40mandb00:00:001
2934190997560,5cyclictest2938848-21kworker/u8:2+events_unbound19:25:141
2934187997564,8cyclictest2979459-21/usr/sbin/munin21:30:270
2934190997463,8cyclictest2969673-21latency_hist21:00:001
2934187997466,6cyclictest2984257-21latency_hist21:45:010
2934187997464,7cyclictest3034427-21latency_hist00:20:010
2934190997268,3cyclictest2955376-21kworker/u8:3+flush-179:020:30:241
2934190997266,5cyclictest3003599-21kworker/u8:1+events_unbound23:20:241
2934190997261,3cyclictest2934040-21kworker/u8:1+events_unbound19:20:241
2934187997266,4cyclictest3016637-21kworker/u8:3+flush-179:023:35:250
2934190997167,3cyclictest2944696-21kworker/u8:0+flush-179:020:25:251
2934190997166,4cyclictest3005433-21kworker/u8:0+events_unbound22:55:241
2934190997065,4cyclictest3016583-21kworker/u8:0+flush-179:023:45:251
2934190997065,4cyclictest2969725-21kworker/u8:2+events_unbound21:25:241
2934190996964,4cyclictest2994033-21kworker/u8:3+flush-179:022:15:201
2934187996961,6cyclictest3026136-21latency_hist23:55:010
2934187996936,28cyclictest2963316-21latency_hist20:40:010
2934190996762,4cyclictest3008602-21kworker/u8:2+flush-179:023:05:261
2934190996749,15cyclictest2982400-21apt-get21:40:001
2934187996754,11cyclictest2953509-21apt-get20:09:590
293419299660,58cyclictest0-21swapper/220:03:002
2934190996655,8cyclictest2946819-21latency_hist19:50:011
2934190996561,3cyclictest2938848-21kworker/u8:2+events_unbound20:10:321
2934190996555,8cyclictest2980808-21latency_hist21:35:001
2934190996460,3cyclictest3016637-21kworker/u8:3+flush-179:023:50:231
2934190996459,4cyclictest2969725-21kworker/u8:2+flush-179:021:19:551
2934190996454,3cyclictest2997213-21kworker/u8:0+events_unbound22:40:201
2934190996452,4cyclictest3016637-21kworker/u8:3+events_unbound23:30:191
2934187996457,5cyclictest2968062-21kworker/u8:1+flush-179:020:55:010
2934187996454,8cyclictest2969937-21apt-get21:00:140
2934190996360,2cyclictest723-21Xorg19:35:591
2934190996257,4cyclictest3031469-21kworker/u8:0+events_unbound00:15:241
2934187996251,8cyclictest2964901-21latency_hist20:45:000
2934190996157,3cyclictest3016637-21kworker/u8:3+flush-179:023:40:171
2934190996157,3cyclictest3013643-21kworker/u8:2+events_unbound23:15:241
2934190996157,3cyclictest2969725-21kworker/u8:2+events_unbound21:05:241
2934190996157,3cyclictest2958580-21kworker/u8:2+events_unbound20:35:311
2934187996152,7cyclictest3029648-21latency_hist00:05:010
2934190996056,3cyclictest2992334-21kworker/u8:2+events_unbound22:30:241
2934190995955,3cyclictest2955325-21kworker/u8:1+events_unbound20:15:241
2934187995955,2cyclictest723-21Xorg23:25:590
2934187995949,9cyclictest2969725-21kworker/u8:2+flush-179:021:15:230
2934190995853,4cyclictest3005433-21kworker/u8:0+events_unbound23:10:191
2934190995853,4cyclictest2969648-21kworker/u8:0+flush-179:021:00:261
2934187995845,10cyclictest271-21systemd-journal22:40:000
2934190995753,3cyclictest3024586-21kworker/u8:2+events_unbound00:05:291
2934190995753,3cyclictest2989317-21kworker/u8:3+flush-179:022:00:231
2934190995752,4cyclictest2934040-21kworker/u8:1+events_unbound19:30:141
2934190995652,3cyclictest723-21Xorg22:50:591
2934190995652,3cyclictest3031469-21kworker/u8:0+flush-179:000:30:201
2934190995651,4cyclictest3016583-21kworker/u8:0+flush-179:023:39:551
2934190995647,7cyclictest2984771-21apt-get21:45:121
2934187995645,8cyclictest3031469-21kworker/u8:0+flush-179:000:30:250
2934187995553,1cyclictest0-21swapper/019:46:590
2934190995449,4cyclictest3029637-21kworker/u8:3+flush-179:000:10:281
2934190995449,4cyclictest2998809-21kworker/u8:1+flush-179:022:29:551
2934190995449,4cyclictest2986080-21kworker/u8:0+flush-179:021:50:271
2934190995449,4cyclictest2958580-21kworker/u8:2+flush-179:020:45:301
2934190995446,5cyclictest2957205-21apt-get20:20:111
2934190995438,14cyclictest2938848-21kworker/u8:2+flush-179:020:00:011
2934187995445,6cyclictest2945234-21latency_hist19:45:010
293418799540,39cyclictest723-21Xorg23:34:590
2934187995343,7cyclictest2977633-21latency_hist21:25:000
2934187995334,17cyclictest2956980-21systemd-run20:20:010
2934190995247,4cyclictest3003599-21kworker/u8:1+flush-179:023:00:221
2934190995247,4cyclictest3003599-21kworker/u8:1+flush-179:023:00:211
2934190995247,4cyclictest2934040-21kworker/u8:1+events_unbound19:40:141
2934190995247,3cyclictest723-21Xorg21:55:591
2934190995146,4cyclictest2997213-21kworker/u8:0+events_unbound22:35:221
2934190995146,4cyclictest2969725-21kworker/u8:2+events_unbound21:20:201
2934190995143,5cyclictest2974439-21latency_hist21:15:021
2934187995143,5cyclictest3031469-21kworker/u8:0+flush-179:000:15:000
2934196995048,1cyclictest0-21swapper/300:22:593
2934190995046,3cyclictest3031469-21kworker/u8:0+flush-179:000:25:301
2934187995042,6cyclictest2966488-21latency_hist20:50:010
2934190994942,6cyclictest2977891-21kworker/u8:1+events_unbound22:10:221
2934187994942,5cyclictest3039162-21kworker/u8:1+flush-179:000:40:000
2934187994940,7cyclictest3008886-21/usr/sbin/munin23:00:170
2934187994940,7cyclictest3008886-21/usr/sbin/munin23:00:160
2934187994938,9cyclictest2942298-21apt-get19:35:120
2934187994839,6cyclictest2957186-21/usr/sbin/munin20:20:240
293418799480,44cyclictest0-21swapper/022:30:150
2934187994743,2cyclictest2934040-21kworker/u8:1+flush-179:019:30:010
2934187994726,18cyclictest0-21swapper/021:10:140
2934187994633,9cyclictest2992334-21kworker/u8:2+flush-179:022:20:000
2934187994626,18cyclictest3016293-21switchtime23:20:260
293418799462,2cyclictest2987940-21apt-get21:55:120
4670791450,12ptp4l441ktimers/319:09:553
2934187994434,7cyclictest2934040-21kworker/u8:1+flush-179:019:20:270
2934190994338,4cyclictest2934101-21kworker/u8:3+events_unbound19:15:271
293419099430,41cyclictest2963319-21turbostat20:44:551
293418799432,39cyclictest2990892-21turbostat22:09:550
2934187994230,10cyclictest2947361-21diskstats19:50:180
2934192994138,2cyclictest0-21swapper/223:07:592
2934190994137,3cyclictest2950099-21kworker/u8:1+events_unbound20:01:411
2934190994136,4cyclictest3039162-21kworker/u8:1+events_unbound00:35:301
2934187994132,6cyclictest2950099-21kworker/u8:1+events_unbound20:00:230
2934187994131,8cyclictest3013644-21/usr/sbin/munin23:15:250
293419699400,38cyclictest3040796-21turbostat00:40:003
293419699400,38cyclictest2980816-21turbostat21:35:013
293419699400,38cyclictest2963319-21turbostat20:40:003
2934190994035,4cyclictest2945268-21kworker/u8:3+events_unbound19:53:421
2934187994030,8cyclictest3008652-21latency_hist23:00:010
293419699390,4cyclictest2998844-21turbostat22:30:003
293419699390,38cyclictest0-21swapper/321:59:553
293419299390,36cyclictest0-21swapper/223:05:002
293419299390,36cyclictest0-21swapper/223:05:002
2934196993832,5cyclictest2974457-21turbostat21:15:013
293419699380,37cyclictest0-21swapper/320:20:593
2934192993835,2cyclictest0-21swapper/200:01:592
293419299380,36cyclictest0-21swapper/200:31:592
293419299380,36cyclictest0-21swapper/200:28:592
293419299380,35cyclictest0-21swapper/222:50:002
293419299380,35cyclictest0-21swapper/221:45:002
2934187993834,2cyclictest723-21Xorg21:07:580
2934187993830,6cyclictest2953992-21/usr/sbin/munin20:10:230
2934187993827,8cyclictest271-21systemd-journal22:25:000
2934196993734,2cyclictest0-21swapper/323:12:593
293419699370,36cyclictest0-21swapper/320:27:583
2934192993734,2cyclictest0-21swapper/223:44:552
2934192993734,2cyclictest0-21swapper/223:26:592
2934192993734,2cyclictest0-21swapper/222:05:582
2934192993734,2cyclictest0-21swapper/219:56:582
2934192993733,2cyclictest0-21swapper/222:29:592
293419299370,35cyclictest0-21swapper/219:12:592
293419299370,35cyclictest0-21swapper/200:10:592
2934187993727,7cyclictest2944696-21kworker/u8:0+flush-179:019:55:220
4670791360,28ptp4l61690irq/136-eno1-TxRx-022:15:123
2934196993633,2cyclictest0-21swapper/320:45:003
2934196993633,2cyclictest0-21swapper/320:07:583
293419699360,35cyclictest0-21swapper/321:19:563
2934192993634,1cyclictest0-21swapper/220:50:012
293419299360,1cyclictest0-21swapper/221:57:002
293419299360,1cyclictest0-21swapper/221:29:592
293419299360,1cyclictest0-21swapper/221:20:592
293419299360,1cyclictest0-21swapper/219:49:552
2934187993633,2cyclictest723-21Xorg23:10:590
2934187993633,2cyclictest0-21swapper/019:13:580
2934187993630,5cyclictest2992480-21turbostat22:14:560
293418799362,6cyclictest0-21swapper/020:55:590
293418799362,5cyclictest0-21swapper/022:28:590
2934187993614,20cyclictest0-21swapper/000:25:200
2934187993611,22cyclictest0-21swapper/021:49:580
2934196993529,5cyclictest2969666-21turbostat21:00:003
293419699350,1cyclictest0-21swapper/320:53:593
2934192993533,1cyclictest0-21swapper/221:55:002
2934192993533,1cyclictest0-21swapper/220:20:002
2934192993532,2cyclictest0-21swapper/222:54:592
2934192993532,2cyclictest0-21swapper/220:26:592
293419299350,1cyclictest0-21swapper/219:43:592
2934190993525,9cyclictest3016857-21apt-get23:25:151
2934187993518,15cyclictest0-21swapper/023:49:590
293418799351,6cyclictest0-21swapper/019:30:590
293418799351,6cyclictest0-21swapper/000:21:590
2934187993516,16cyclictest0-21swapper/023:05:120
2934187993510,6cyclictest2978142-21expr21:25:130
2934196993432,1cyclictest0-21swapper/320:16:003
2934196993428,5cyclictest3029660-21turbostat00:05:003
2934190993429,4cyclictest2977891-21kworker/u8:1+events_unbound22:05:271
2934187993431,2cyclictest723-21Xorg23:41:590
2934187993431,2cyclictest723-21Xorg20:32:000
2934187993425,6cyclictest2937268-21latency_hist19:20:010
293418799341,6cyclictest0-21swapper/022:04:580
2934196993331,1cyclictest0-21swapper/322:20:583
2934196993331,1cyclictest0-21swapper/319:49:003
2934196993328,4cyclictest3034441-21turbostat00:20:013
2934192993331,1cyclictest0-21swapper/222:05:002
2934192993330,2cyclictest0-21swapper/223:31:592
293418799331,6cyclictest0-21swapper/000:05:160
2934196993230,1cyclictest0-21swapper/323:32:593
2934192993230,1cyclictest0-21swapper/221:49:582
2934192993229,2cyclictest0-21swapper/220:54:552
2934192993229,2cyclictest0-21swapper/220:14:552
2934192993228,2cyclictest0-21swapper/220:36:582
293418799321,5cyclictest2958790-21apt-get20:25:120
4670791310,3ptp4l431rcuc/322:50:003
4670791300,10ptp4l441ktimers/320:35:013
2934196993028,1cyclictest0-21swapper/319:22:593
2934196993027,2cyclictest0-21swapper/323:53:593
2934196993027,2cyclictest0-21swapper/321:03:593
2934196993025,4cyclictest3006795-21turbostat22:55:013
4670791290,5ptp4l441ktimers/322:39:563
2934196992927,1cyclictest0-21swapper/319:35:143
2934196992926,2cyclictest0-21swapper/319:55:593
293419699290,28cyclictest0-21swapper/323:05:143
293419699290,1cyclictest0-21swapper/321:25:123
2934192992926,2cyclictest0-21swapper/223:50:162
2934192992926,2cyclictest0-21swapper/221:10:582
2934192992926,2cyclictest0-21swapper/220:55:132
2934192992926,2cyclictest0-21swapper/219:30:142
2934192992926,2cyclictest0-21swapper/200:00:012
293419299290,27cyclictest0-21swapper/223:16:592
293419299290,27cyclictest0-21swapper/222:55:032
293419299290,27cyclictest0-21swapper/200:35:152
293419299290,21cyclictest0-21swapper/223:35:152
293419299290,1cyclictest0-21swapper/220:40:162
2934196992827,1cyclictest0-21swapper/322:00:143
2934196992826,1cyclictest0-21swapper/323:35:153
2934196992826,1cyclictest0-21swapper/323:15:143
2934196992826,1cyclictest0-21swapper/319:15:153
2934196992826,1cyclictest0-21swapper/300:29:553
2934196992825,2cyclictest0-21swapper/322:55:023
2934196992824,2cyclictest2992482-21turbostat22:10:003
2934196992822,5cyclictest2972865-21turbostat21:10:003
293419699280,27cyclictest0-21swapper/321:40:023
293419699280,27cyclictest0-21swapper/321:36:593
293419699280,27cyclictest0-21swapper/319:28:593
293419699280,26cyclictest0-21swapper/323:55:033
293419699280,26cyclictest0-21swapper/323:45:143
293419699280,26cyclictest0-21swapper/320:10:023
293419699280,26cyclictest0-21swapper/319:10:133
2934192992826,1cyclictest0-21swapper/220:05:022
2934196992725,1cyclictest0-21swapper/322:41:593
2934196992725,1cyclictest0-21swapper/319:31:583
2934196992724,2cyclictest0-21swapper/319:40:593
293419699270,1cyclictest0-21swapper/320:05:003
2934192992724,2cyclictest0-21swapper/221:03:002
293419299270,25cyclictest0-21swapper/222:12:592
293419299270,1cyclictest0-21swapper/220:34:582
4670791260,6ptp4l441ktimers/322:15:013
4670791260,6ptp4l441ktimers/321:49:593
4670791260,5ptp4l61690irq/136-eno1-TxRx-023:45:013
2934196992624,1cyclictest0-21swapper/323:28:593
2934192992623,2cyclictest0-21swapper/221:33:582
2934192992623,2cyclictest0-21swapper/219:29:552
293419299260,2cyclictest0-21swapper/222:40:592
293419299260,2cyclictest0-21swapper/219:15:582
293419299260,24cyclictest0-21swapper/220:21:592
2934196992523,1cyclictest0-21swapper/322:31:003
293419699250,24cyclictest0-21swapper/323:01:593
293419699250,24cyclictest0-21swapper/323:01:593
2934192992522,2cyclictest0-21swapper/221:07:592
2934192992522,2cyclictest0-21swapper/200:15:122
4670791240,6ptp4l61690irq/136-eno1-TxRx-021:55:013
4670791240,6ptp4l441ktimers/300:09:553
4670791240,5ptp4l61690irq/136-eno1-TxRx-019:54:543
4670791240,5ptp4l441ktimers/321:24:593
4670791240,5ptp4l441ktimers/300:14:563
2934192992421,2cyclictest0-21swapper/223:13:592
2934192992421,2cyclictest0-21swapper/200:20:592
4670791230,5ptp4l61690irq/136-eno1-TxRx-023:24:563
4670791230,4ptp4l441ktimers/300:34:553
2934192992320,2cyclictest0-21swapper/222:40:002
2934192992320,2cyclictest0-21swapper/222:33:582
2934190992313,2cyclictest171rcu_preempt19:10:021
4670791220,4ptp4l61690irq/136-eno1-TxRx-020:49:553
2934192992119,1cyclictest0-21swapper/219:36:582
2934192991916,2cyclictest0-21swapper/219:20:592
2934192991814,2cyclictest0-21swapper/221:18:592
2934192991512,2cyclictest0-21swapper/222:18:592
2934192991411,2cyclictest0-21swapper/222:20:132
293418799149,4cyclictest0-21swapper/019:10:020
293419299108,1cyclictest0-21swapper/223:23:592
293419299107,2cyclictest0-21swapper/221:35:242
293419299107,2cyclictest0-21swapper/219:52:432
29341929985,2cyclictest0-21swapper/200:05:172
29341929980,7cyclictest0-21swapper/223:48:142
29341929920,1cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional