You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-03 - 14:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6.osadl.org (updated Thu Jul 03, 2025 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28259789910899,7cyclictest2882200-21gpgv22:10:011
28259789910795,9cyclictest3105085-21systemd-journal23:15:001
28259759910498,5cyclictest2890929-21gpgv22:35:000
28259759910086,11cyclictest2897681-21apt-get22:55:160
2825978999887,9cyclictest2869583-21kworker/u8:0+flush-179:021:35:011
2825978999779,7cyclictest2838589-21latency_hist19:50:011
2825975999783,11cyclictest2859537-21apt-get21:00:000
2825978999584,8cyclictest2829043-21latency_hist19:20:001
2825975999483,8cyclictest2846820-21latency_hist20:20:010
2825975999380,10cyclictest2924808-21apt-get00:20:120
2825978999281,9cyclictest2866428-21apt-get21:20:111
2825975999082,7cyclictest2922978-21kworker/u8:2+flush-179:000:30:010
2825978998977,10cyclictest3105085-21systemd-journal20:00:001
2825978998877,9cyclictest2856595-21/usr/sbin/munin20:50:221
2825978998776,9cyclictest2826115-21apt-get19:10:111
2825975998681,4cyclictest2878846-21kworker/u8:0+events_unbound22:35:040
2825978998478,5cyclictest2843195-21kworker/u8:1+flush-179:020:40:001
2825975998473,8cyclictest2930923-21latency_hist00:40:010
2825975998277,4cyclictest2897662-21kworker/u8:1+events_unbound23:09:530
2825975998277,4cyclictest2869583-21kworker/u8:0+events_unbound21:44:520
2825978998177,3cyclictest2838831-21kworker/u8:0+events_unbound20:01:131
2825975998176,4cyclictest2843195-21kworker/u8:1+events_unbound22:01:340
2825978998076,3cyclictest2897662-21kworker/u8:1+events_unbound23:31:531
2825975998075,4cyclictest2902190-21kworker/u8:0+events_unbound23:22:530
2825975998069,9cyclictest2894511-21apt-get22:45:160
2825975998069,9cyclictest2894511-21apt-get22:45:160
2825975997975,3cyclictest2843195-21kworker/u8:1+events_unbound20:50:190
2825975997974,4cyclictest2869583-21kworker/u8:0+events_unbound21:35:300
2825978997874,3cyclictest2869583-21kworker/u8:0+events_unbound21:41:501
2825978997873,4cyclictest2921594-21kworker/u8:0+events_unbound00:37:171
2825975997870,6cyclictest2877548-21apt-get21:55:120
2825978997773,3cyclictest2922978-21kworker/u8:2+events_unbound00:18:521
2825975997766,8cyclictest2830640-21latency_hist19:25:000
2825978997672,3cyclictest2895875-21kworker/u8:3+events_unbound22:51:471
2825978997671,4cyclictest2878846-21kworker/u8:0+events_unbound22:12:511
2825975997672,3cyclictest2921594-21kworker/u8:0+events_unbound00:11:520
2825975997671,4cyclictest2843195-21kworker/u8:1+events_unbound20:08:510
282597599761,71cyclictest1-21systemd19:59:590
2825975997570,4cyclictest2902190-21kworker/u8:0+events_unbound23:47:520
2825975997570,4cyclictest2865792-21kworker/u8:2+events_unbound21:27:500
2825978997470,3cyclictest2897662-21kworker/u8:1+events_unbound23:59:571
2825978997470,3cyclictest2832262-21kworker/u8:3+events_unbound20:12:511
2825975997470,3cyclictest2869583-21kworker/u8:0+events_unbound21:31:300
2825975997469,4cyclictest2895875-21kworker/u8:3+events_unbound22:51:520
2825975997469,4cyclictest2832262-21kworker/u8:3+events_unbound20:13:500
2825978997367,5cyclictest2897662-21kworker/u8:1+events_unbound23:46:521
2825975997369,3cyclictest2882208-21kworker/u8:2+events_unbound22:21:400
2825975997368,4cyclictest2851565-21kworker/u8:3+events_unbound20:33:520
2825975997366,5cyclictest2911746-21latency_hist23:40:010
2825978997268,3cyclictest2843195-21kworker/u8:1+events_unbound22:04:561
2825978997267,4cyclictest2892637-21kworker/u8:2+events_unbound22:40:271
2825978997267,4cyclictest2832262-21kworker/u8:3+events_unbound19:50:491
2825975997267,4cyclictest2838831-21kworker/u8:0+events_unbound20:02:500
2825975997267,4cyclictest2832262-21kworker/u8:3+events_unbound20:20:510
2825978997167,3cyclictest2878846-21kworker/u8:0+events_unbound22:23:521
2825978997167,3cyclictest2843195-21kworker/u8:1+events_unbound21:12:511
2825978997166,4cyclictest2897662-21kworker/u8:1+events_unbound00:07:521
2825975997167,3cyclictest2882208-21kworker/u8:2+events_unbound22:14:510
2825975997166,4cyclictest2862172-21kworker/u8:0+events_unbound21:06:290
2825975997161,8cyclictest2832227-21latency_hist19:30:010
2825975997159,10cyclictest2918401-21apt-get00:00:140
2825978997066,3cyclictest2701566-21kworker/u8:0+events_unbound19:26:511
2825978997065,4cyclictest2821204-21kworker/u8:1+events_unbound19:22:511
2825978997064,5cyclictest2897662-21kworker/u8:1+flush-179:023:35:241
2825978997060,7cyclictest2892679-21latency_hist22:40:011
2825975997066,3cyclictest2832262-21kworker/u8:3+events_unbound19:36:090
2825975997065,4cyclictest2882208-21kworker/u8:2+events_unbound22:16:510
2825975997064,4cyclictest2862172-21kworker/u8:0+events_unbound21:11:500
2825978996965,3cyclictest2897662-21kworker/u8:1+events_unbound23:04:521
2825978996965,3cyclictest2882208-21kworker/u8:2+events_unbound22:17:521
2825978996965,3cyclictest2849069-21kworker/u8:2+events_unbound20:25:261
2825978996964,4cyclictest2832262-21kworker/u8:3+events_unbound19:42:501
2825975996965,3cyclictest2929284-21kworker/u8:1+events_unbound00:31:520
2825975996965,3cyclictest2902190-21kworker/u8:0+events_unbound23:11:500
2825975996965,3cyclictest2838831-21kworker/u8:0+events_unbound19:53:520
2825975996964,4cyclictest2902190-21kworker/u8:0+events_unbound23:56:530
2825975996964,4cyclictest2902190-21kworker/u8:0+events_unbound23:43:530
2825975996964,4cyclictest2849069-21kworker/u8:2+events_unbound20:28:490
2825978996861,5cyclictest2872527-21latency_hist21:40:001
2825975996864,3cyclictest2922978-21kworker/u8:2+events_unbound00:17:530
2825975996864,3cyclictest2853224-21kworker/u8:2+events_unbound21:03:500
2825975996863,4cyclictest2897662-21kworker/u8:1+events_unbound23:32:520
2825975996863,4cyclictest2843195-21kworker/u8:1+events_unbound21:21:520
2825975996863,4cyclictest2821204-21kworker/u8:1+events_unbound19:31:490
2825975996863,3cyclictest2897662-21kworker/u8:1+events_unbound23:25:180
2825975996863,3cyclictest2701566-21kworker/u8:0+events_unbound19:10:490
2825975996862,5cyclictest2878846-21kworker/u8:0+flush-179:022:25:230
2825978996763,3cyclictest2902190-21kworker/u8:0+events_unbound23:53:521
2825978996760,6cyclictest2843195-21kworker/u8:1+flush-179:021:05:261
2825975996763,3cyclictest2832262-21kworker/u8:3+events_unbound19:47:510
2825978996655,9cyclictest2860062-21apt-get21:00:141
2825975996661,4cyclictest2843195-21kworker/u8:1+events_unbound20:39:520
2825975996661,4cyclictest2701566-21kworker/u8:0+events_unbound19:15:210
2825978996561,3cyclictest2843195-21kworker/u8:1+flush-179:021:45:251
2825978996558,5cyclictest3105085-21systemd-journal00:34:591
2825978996556,7cyclictest502-21rs:main1
2825978996556,6cyclictest3105085-21systemd-journal00:00:511
2825975996555,8cyclictest2853426-21/usr/sbin/munin20:40:240
2825978996454,8cyclictest2835414-21latency_hist19:40:001
2825975996459,4cyclictest2892637-21kworker/u8:2+events_unbound22:40:020
2825978996361,1cyclictest0-21swapper/120:43:581
2825978996153,6cyclictest3105085-21systemd-journal00:15:001
2825975996149,10cyclictest2880743-21apt-get22:05:130
2825975996052,6cyclictest2920010-21/usr/sbin/munin00:05:240
2825978995955,3cyclictest2890779-21kworker/u8:1+events_unbound22:45:031
2825978995955,3cyclictest2890779-21kworker/u8:1+events_unbound22:45:031
2825975995955,3cyclictest2843195-21kworker/u8:1+events_unbound21:51:330
2825978995651,4cyclictest2903786-21kworker/u8:3+events_unbound23:16:521
2825978995651,4cyclictest2701566-21kworker/u8:0+events_unbound19:30:021
2825978995551,3cyclictest2922978-21kworker/u8:2+events_unbound00:24:531
2825975995551,3cyclictest2897662-21kworker/u8:1+events_unbound23:01:000
2825978995450,3cyclictest2843195-21kworker/u8:1+events_unbound21:57:501
2825978995449,4cyclictest2865792-21kworker/u8:2+events_unbound21:25:511
2825978995428,13cyclictest2854781-21turbostat20:49:561
2825978995428,13cyclictest2854781-21turbostat20:49:561
2825975995449,4cyclictest2843195-21kworker/u8:1+events_unbound21:16:280
2825975995447,6cyclictest2902190-21kworker/u8:0+flush-179:023:50:190
2825978995350,2cyclictest2843195-21kworker/u8:1+events_unbound21:50:261
2825978995349,3cyclictest2843195-21kworker/u8:1+events_unbound20:31:201
2825978995348,4cyclictest2922978-21kworker/u8:2+events_unbound00:27:521
2825975995349,3cyclictest2843195-21kworker/u8:1+events_unbound20:45:020
2825975995349,3cyclictest2843195-21kworker/u8:1+events_unbound20:45:010
2825978995247,4cyclictest2897662-21kworker/u8:1+events_unbound23:42:521
2825978995240,10cyclictest2858178-21/usr/sbin/munin20:55:271
2825978995045,4cyclictest2897662-21kworker/u8:1+flush-179:023:05:271
4505291490,13ptp4l441ktimers/319:10:003
2825978994743,3cyclictest2832262-21kworker/u8:3+events_unbound20:20:011
2825978994743,3cyclictest2832262-21kworker/u8:3+events_unbound20:15:021
2825975994742,4cyclictest2903786-21kworker/u8:3+events_unbound23:16:500
282597599471,42cyclictest2837024-21dump-pmu-power19:44:590
2825978994640,4cyclictest1-21systemd21:20:011
2825978994541,3cyclictest2895875-21kworker/u8:3+events_unbound22:55:581
282597899451,7cyclictest2843629-21munin-run20:09:591
282598199430,41cyclictest0-21swapper/222:10:002
282598699420,40cyclictest2875727-21turbostat21:50:003
2825978994234,6cyclictest3105085-21systemd-journal23:28:511
2825986994138,2cyclictest0-21swapper/319:58:583
2825986994034,5cyclictest2900625-21turbostat23:05:003
2825986994034,5cyclictest2890811-21turbostat22:35:003
282598699400,4cyclictest2857949-21turbostat20:55:003
282598699390,38cyclictest2895846-21turbostat22:49:593
282598699390,38cyclictest2895846-21turbostat22:49:593
282598699390,36cyclictest0-21swapper/323:21:593
282598199390,36cyclictest0-21swapper/200:02:002
2825986993833,4cyclictest2885512-21turbostat22:20:003
2825986993833,4cyclictest2842054-21turbostat20:05:003
2825986993832,5cyclictest2864568-21turbostat21:14:593
282598699380,35cyclictest0-21swapper/300:17:003
2825981993835,2cyclictest0-21swapper/200:09:592
2825981993834,2cyclictest0-21swapper/221:55:592
2825981993834,2cyclictest0-21swapper/200:31:002
282598199380,35cyclictest0-21swapper/221:19:592
282598199380,2cyclictest0-21swapper/223:21:592
2825978993828,8cyclictest2825833-21latency_hist19:10:021
2825986993735,1cyclictest0-21swapper/321:24:553
2825986993735,1cyclictest0-21swapper/320:24:593
282598699370,2cyclictest0-21swapper/320:42:593
2825981993734,2cyclictest0-21swapper/221:15:002
2825981993734,2cyclictest0-21swapper/219:59:562
2825981993733,2cyclictest0-21swapper/223:59:562
282598199370,35cyclictest0-21swapper/223:40:592
282598199370,2cyclictest0-21swapper/200:17:002
2825978993732,4cyclictest2897662-21kworker/u8:1+events_unbound23:23:571
282597899370,5cyclictest2887125-21apt-get22:25:031
2825986993634,1cyclictest0-21swapper/321:26:583
2825986993632,2cyclictest0-21swapper/321:56:003
2825986993629,6cyclictest2862992-21turbostat21:10:013
282598699360,1cyclictest0-21swapper/322:35:583
282598699360,1cyclictest0-21swapper/320:14:583
2825981993634,1cyclictest0-21swapper/222:20:012
2825981993634,1cyclictest0-21swapper/220:04:592
2825981993633,2cyclictest0-21swapper/200:23:592
2825981993632,2cyclictest0-21swapper/221:09:552
2825986993533,1cyclictest0-21swapper/300:31:003
2825986993533,1cyclictest0-21swapper/300:09:593
2825986993533,1cyclictest0-21swapper/300:04:553
2825981993532,2cyclictest0-21swapper/223:45:592
2825981993532,2cyclictest0-21swapper/222:54:552
2825986993432,1cyclictest0-21swapper/300:23:593
2825986993431,2cyclictest0-21swapper/323:59:563
2825986993428,5cyclictest2930914-21turbostat00:40:003
2825981993430,2cyclictest0-21swapper/222:14:552
2825975993430,3cyclictest2843195-21kworker/u8:1+events_unbound21:46:170
2825986993329,3cyclictest2913365-21turbostat23:45:003
2825986993328,4cyclictest2888975-21turbostat22:30:003
2825981993329,3cyclictest0-21swapper/221:40:592
2825986993230,1cyclictest0-21swapper/323:45:583
2825986993230,1cyclictest0-21swapper/322:54:553
2825981993230,1cyclictest0-21swapper/200:40:012
2825981993229,2cyclictest0-21swapper/223:05:002
2825981993229,2cyclictest0-21swapper/221:33:592
2825986993127,2cyclictest0-21swapper/321:40:583
2825981993128,2cyclictest0-21swapper/220:33:002
282598199310,29cyclictest0-21swapper/222:20:172
282598199310,29cyclictest0-21swapper/219:25:142
2825986993028,1cyclictest0-21swapper/322:14:563
2825986993028,1cyclictest0-21swapper/321:33:593
2825986993027,2cyclictest0-21swapper/320:32:593
2825981993028,1cyclictest0-21swapper/221:24:582
2825981993026,2cyclictest0-21swapper/222:47:592
2825981993026,2cyclictest0-21swapper/222:47:592
282598199300,2cyclictest0-21swapper/222:35:192
2825986992927,1cyclictest0-21swapper/323:30:133
2825986992926,2cyclictest0-21swapper/320:35:143
2825986992926,2cyclictest0-21swapper/319:25:143
282598699290,27cyclictest0-21swapper/322:03:593
282598699290,27cyclictest0-21swapper/319:35:143
282598699290,26cyclictest0-21swapper/322:20:173
2825981992926,2cyclictest0-21swapper/220:35:132
282598199290,28cyclictest0-21swapper/219:10:122
282598199290,27cyclictest0-21swapper/223:30:132
282598199290,27cyclictest0-21swapper/222:30:022
282598199290,27cyclictest0-21swapper/221:25:152
282598199290,27cyclictest0-21swapper/219:43:592
282598199290,27cyclictest0-21swapper/219:35:142
282598199290,26cyclictest0-21swapper/221:04:552
282598199290,21cyclictest0-21swapper/222:04:002
282598699280,27cyclictest0-21swapper/321:04:553
282598699280,27cyclictest0-21swapper/319:10:133
282598699280,26cyclictest0-21swapper/323:10:593
282598699280,26cyclictest0-21swapper/319:15:593
2825981992825,2cyclictest0-21swapper/223:25:142
282598199280,26cyclictest0-21swapper/220:44:592
282598199280,25cyclictest0-21swapper/219:15:592
4505291270,6ptp4l441ktimers/320:09:563
4505291270,5ptp4l61790irq/134-eno1-TxRx-021:55:003
4505291270,5ptp4l441ktimers/323:20:003
4505291270,5ptp4l441ktimers/321:39:563
4505291270,5ptp4l441ktimers/320:30:003
282598699270,24cyclictest0-21swapper/319:51:003
2825981992724,2cyclictest0-21swapper/219:55:002
2825981992724,1cyclictest0-21swapper/221:54:582
2825981992723,2cyclictest0-21swapper/221:47:002
282598199270,25cyclictest0-21swapper/223:10:592
4505291260,6ptp4l61790irq/134-eno1-TxRx-023:40:003
4505291260,6ptp4l441ktimers/322:59:563
4505291260,6ptp4l441ktimers/319:49:563
4505291260,5ptp4l441ktimers/323:09:563
4505291260,5ptp4l441ktimers/322:09:553
4505291260,5ptp4l441ktimers/320:19:553
4505291260,5ptp4l441ktimers/319:44:553
2825986992624,1cyclictest0-21swapper/323:25:143
282598699260,1cyclictest0-21swapper/322:44:593
2825981992623,2cyclictest0-21swapper/220:58:592
2825981992623,2cyclictest0-21swapper/220:17:592
2825981992623,2cyclictest0-21swapper/220:09:552
2825981992622,2cyclictest0-21swapper/221:38:582
282598199260,2cyclictest0-21swapper/223:15:142
282598199260,24cyclictest0-21swapper/219:47:582
4505291250,5ptp4l61790irq/134-eno1-TxRx-019:35:013
4505291250,5ptp4l441ktimers/319:24:543
4505291250,4ptp4l61790irq/134-eno1-TxRx-023:54:563
4505291250,4ptp4l61790irq/134-eno1-TxRx-021:00:003
4505291250,4ptp4l441ktimers/320:49:563
4505291250,4ptp4l441ktimers/320:49:553
4505291250,4ptp4l441ktimers/300:29:553
2825986992523,1cyclictest0-21swapper/300:10:163
2825986992522,2cyclictest0-21swapper/321:19:543
2825981992522,2cyclictest0-21swapper/223:37:592
2825981992522,2cyclictest0-21swapper/222:25:142
2825981992522,2cyclictest0-21swapper/200:25:582
2825981992320,2cyclictest0-21swapper/219:20:142
2825981992319,2cyclictest0-21swapper/222:44:552
2825975992318,4cyclictest0-21swapper/019:10:010
2825981992220,1cyclictest0-21swapper/220:30:002
2825981992219,2cyclictest0-21swapper/219:32:592
2825981992118,2cyclictest0-21swapper/200:14:592
2825981991916,2cyclictest0-21swapper/223:08:592
2825981991916,2cyclictest0-21swapper/222:57:592
2825981991814,2cyclictest0-21swapper/220:54:552
2825981991613,2cyclictest0-21swapper/220:21:592
2825981991613,2cyclictest0-21swapper/220:13:592
2825981991311,1cyclictest0-21swapper/220:49:562
2825981991311,1cyclictest0-21swapper/220:49:552
28259819963,2cyclictest0-21swapper/223:51:452
28259819930,2cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional