You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 06:29
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Mon Dec 01, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4037039910493,8cyclictest477904-21latency_hist23:00:010
403703999685,4cyclictest398571-21kworker/u8:1+events_unbound19:25:230
403707999589,5cyclictest482284-21kworker/u8:1+flush-179:000:04:021
403707999589,5cyclictest482284-21kworker/u8:1+flush-179:000:04:011
403703999487,5cyclictest398571-21kworker/u8:1+events_unbound19:45:220
403707999185,4cyclictest402136-21kworker/u8:0+flush-179:019:20:001
403707998573,8cyclictest445799-21latency_hist21:20:001
403703998277,4cyclictest428276-21kworker/u8:3+flush-179:020:25:250
403703998268,4cyclictest482284-21kworker/u8:1+events_unbound23:40:010
403707998170,4cyclictest428276-21kworker/u8:3+events_unbound20:40:001
403707998070,7cyclictest487690-21/usr/sbin/munin23:30:231
403703998076,3cyclictest421695-21kworker/u8:2+events_unbound20:10:270
403707997969,7cyclictest499427-21/usr/sbin/munin00:10:191
403703997975,3cyclictest436278-21kworker/u8:1+events_unbound21:10:250
403703997972,6cyclictest413483-21kworker/u8:3+flush-179:020:00:010
403703997966,6cyclictest431498-21kworker/u8:0+events_unbound20:35:200
403703997873,4cyclictest471502-21kworker/u8:0+events_unbound23:00:240
403703997766,8cyclictest464892-21latency_hist22:20:010
403707997670,4cyclictest421695-21kworker/u8:2+events_unbound20:20:011
403703997668,6cyclictest482284-21kworker/u8:1+events_unbound23:50:220
403703997668,6cyclictest271-21systemd-journal19:40:000
403707997569,4cyclictest428276-21kworker/u8:3+flush-179:020:35:001
403707997565,7cyclictest463302-21latency_hist22:15:011
403703997570,4cyclictest497133-21kworker/u8:3+events_unbound00:20:240
403703997569,4cyclictest406902-21kworker/u8:3+events_unbound19:20:230
403703997461,5cyclictest431498-21kworker/u8:0+events_unbound20:55:200
403703997369,3cyclictest436278-21kworker/u8:1+events_unbound21:00:240
403707997265,4cyclictest471502-21kworker/u8:0+flush-179:023:00:251
403703997268,3cyclictest499235-21kworker/u8:2+events_unbound00:25:240
403703997265,5cyclictest413483-21kworker/u8:3+events_unbound20:00:150
403703997242,28cyclictest723-21Xorg21:37:580
403703997167,3cyclictest398571-21kworker/u8:1+flush-179:019:30:250
403703997161,7cyclictest490885-21/usr/sbin/munin23:40:240
403703997064,5cyclictest447372-21kworker/u8:0+events_unbound21:55:260
403707996862,4cyclictest436278-21kworker/u8:1+flush-179:021:20:161
403707996861,4cyclictest455586-21kworker/u8:2+flush-179:022:00:241
403703996862,4cyclictest482284-21kworker/u8:1+flush-179:000:15:200
403703996855,11cyclictest498676-21apt-get00:05:010
403703996855,11cyclictest498676-21apt-get00:05:000
403707996761,4cyclictest406902-21kworker/u8:3+flush-179:019:20:261
403703996762,3cyclictest505596-21kworker/u8:1+events_unbound00:35:220
40370399670,1cyclictest0-21swapper/023:12:580
403707996660,4cyclictest431498-21kworker/u8:0+events_unbound20:50:251
403703996662,3cyclictest471502-21kworker/u8:0+flush-179:023:05:230
403703996661,4cyclictest479530-21kworker/u8:2+events_unbound23:15:210
403703996661,4cyclictest447372-21kworker/u8:0+events_unbound21:40:230
403703996659,6cyclictest398571-21kworker/u8:1+events_unbound19:10:190
403703996659,5cyclictest410299-21kworker/u8:2+events_unbound19:40:240
403703996658,6cyclictest428276-21kworker/u8:3+events_unbound20:40:160
403709996561,3cyclictest0-21swapper/220:49:592
403707996556,6cyclictest499235-21kworker/u8:2+flush-179:000:25:251
403707996554,8cyclictest469934-21latency_hist22:35:001
403707996554,8cyclictest469934-21latency_hist22:35:001
403703996561,3cyclictest439472-21kworker/u8:3+events_unbound21:15:170
403703996561,3cyclictest421695-21kworker/u8:2+flush-179:020:20:220
403703996556,8cyclictest421695-21kworker/u8:2+flush-179:020:20:010
403703996457,5cyclictest468381-21kworker/u8:3+flush-179:022:40:250
403713996361,1cyclictest0-21swapper/320:49:593
403707996357,4cyclictest398571-21kworker/u8:1+events_unbound19:55:021
403703996359,3cyclictest482284-21kworker/u8:1+events_unbound23:55:240
403703996351,10cyclictest430131-21apt-get20:30:140
403707996257,4cyclictest468381-21kworker/u8:3+flush-179:022:50:261
202199620,3rtkit-daemon2020-21rtkit-daemon23:46:050
403707996156,4cyclictest434443-21kworker/u8:2+events_unbound21:05:011
403707996156,3cyclictest482284-21kworker/u8:1+flush-179:023:50:231
403707996152,6cyclictest474714-21latency_hist22:50:011
403703996144,5cyclictest447372-21kworker/u8:0+events_unbound22:25:230
403707996054,4cyclictest723-21Xorg23:37:591
403707995958,1cyclictest0-21swapper/123:59:591
403707995953,4cyclictest428276-21kworker/u8:3+flush-179:020:25:241
403707995952,5cyclictest439472-21kworker/u8:3+flush-179:021:05:241
403703995953,4cyclictest471578-21kworker/u8:1+events_unbound22:50:210
403707995852,3cyclictest452211-21kworker/u8:1+flush-179:021:45:151
403707995851,5cyclictest482284-21kworker/u8:1+flush-179:023:25:251
403703995853,4cyclictest468381-21kworker/u8:3+events_unbound22:45:240
403707995751,4cyclictest479530-21kworker/u8:2+flush-179:023:20:011
403707995749,6cyclictest453765-21latency_hist21:45:001
403707995748,6cyclictest503976-21latency_hist00:25:011
403707995742,13cyclictest398571-21kworker/u8:1+flush-179:019:50:011
403703995752,4cyclictest431498-21kworker/u8:0+events_unbound20:45:150
403703995746,9cyclictest468602-21apt-get22:30:130
403703995746,9cyclictest468602-21apt-get22:30:130
403703995746,8cyclictest505817-21apt-get00:30:130
403707995652,2cyclictest471578-21kworker/u8:1+events_unbound22:55:251
403707995651,3cyclictest482284-21kworker/u8:1+flush-179:023:20:211
403703995651,4cyclictest497133-21kworker/u8:3+events_unbound00:10:210
403707995549,4cyclictest466483-21kworker/u8:2+flush-179:022:40:171
403707995549,4cyclictest387258-21kworker/u8:2+events_unbound19:15:011
403707995543,9cyclictest481082-21latency_hist23:10:011
403703995551,3cyclictest482284-21kworker/u8:1+events_unbound23:30:200
403703995550,4cyclictest466483-21kworker/u8:2+flush-179:022:35:260
403703995545,7cyclictest485865-21latency_hist23:25:010
403707995451,2cyclictest0-21swapper/120:44:001
403707995446,5cyclictest508737-21latency_hist00:40:011
403703995447,6cyclictest464866-21kworker/u8:1+events_unbound22:20:170
403707995348,4cyclictest410299-21kworker/u8:2+flush-179:019:40:261
403707995348,4cyclictest398571-21kworker/u8:1+flush-179:019:39:551
4670791510,8ptp4l441ktimers/319:09:243
403707995144,4cyclictest431498-21kworker/u8:0+events_unbound20:45:261
403707995046,3cyclictest398571-21kworker/u8:1+events_unbound19:25:131
403707995044,4cyclictest436278-21kworker/u8:1+flush-179:021:10:221
403707995037,11cyclictest421695-21kworker/u8:2+flush-179:020:10:201
403703995045,4cyclictest723-21Xorg22:07:590
403707994943,5cyclictest387258-21kworker/u8:2+events_unbound19:10:021
403707994935,4cyclictest398571-21kworker/u8:1+flush-179:019:30:201
403703994944,3cyclictest455586-21kworker/u8:2+events_unbound22:00:150
40371399480,5cyclictest485878-21turbostat23:25:003
403707994844,3cyclictest421695-21kworker/u8:2+flush-179:020:05:221
403707994843,4cyclictest489023-21kworker/u8:0+flush-179:023:45:211
403703994743,3cyclictest436278-21kworker/u8:1+events_unbound21:20:200
403703994743,3cyclictest431498-21kworker/u8:0+flush-179:020:50:270
403707994640,5cyclictest398571-21kworker/u8:1+events_unbound19:55:191
403703994639,5cyclictest447372-21kworker/u8:0+events_unbound21:50:240
403707994540,4cyclictest427745-21kworker/u8:1+flush-179:020:24:551
403707994539,5cyclictest482284-21kworker/u8:1+events_unbound00:15:171
403707994539,5cyclictest468381-21kworker/u8:3+flush-179:022:39:551
403703994539,4cyclictest413483-21kworker/u8:3+events_unbound19:50:210
403703994537,6cyclictest452211-21kworker/u8:1+flush-179:021:45:270
40370999440,41cyclictest0-21swapper/200:04:552
40370999440,41cyclictest0-21swapper/200:04:552
403707994439,3cyclictest482284-21kworker/u8:1+flush-179:023:40:151
403707994438,5cyclictest497133-21kworker/u8:3+events_unbound00:30:201
403703994440,3cyclictest436278-21kworker/u8:1+flush-179:021:05:280
403703994440,3cyclictest402136-21kworker/u8:0+events_unbound19:15:230
403703994437,4cyclictest447400-21kworker/u8:2+events_unbound21:25:030
403707994338,4cyclictest479530-21kworker/u8:2+flush-179:023:10:161
403707994337,5cyclictest447372-21kworker/u8:0+flush-179:022:25:041
403703994338,3cyclictest447372-21kworker/u8:0+flush-179:022:10:010
40371399420,40cyclictest439433-21turbostat21:00:003
40370999420,39cyclictest0-21swapper/219:30:012
403707994235,6cyclictest413483-21kworker/u8:3+events_unbound20:04:591
403703994235,5cyclictest447400-21kworker/u8:2+events_unbound21:30:160
403713994138,1cyclictest0-21swapper/321:02:593
403713994135,5cyclictest471529-21turbostat22:40:003
40371399410,39cyclictest497005-21turbostat00:00:003
403709994137,2cyclictest0-21swapper/221:02:592
403709994132,7cyclictest0-21swapper/221:23:582
403707994134,5cyclictest449220-21kworker/u8:1+flush-179:021:30:221
403703994136,4cyclictest421695-21kworker/u8:2+flush-179:020:05:250
403713994034,4cyclictest479503-21turbostat23:05:003
40371399400,38cyclictest411861-21turbostat19:35:013
40371399400,38cyclictest405227-21turbostat19:14:593
403709994036,2cyclictest0-21swapper/200:11:592
403707994035,4cyclictest447372-21kworker/u8:0+flush-179:021:25:251
403707994035,3cyclictest499226-21systemd-run00:10:001
403707994035,3cyclictest499226-21systemd-run00:10:001
403707994034,4cyclictest450825-21apt-get21:35:121
403713993936,2cyclictest0-21swapper/300:11:593
403713993934,4cyclictest448998-21turbostat21:30:013
403713993934,4cyclictest428294-21turbostat20:24:593
403713993932,6cyclictest0-21swapper/321:23:583
403709993934,2cyclictest0-21swapper/222:20:592
40370999390,36cyclictest0-21swapper/223:34:002
40370999390,36cyclictest0-21swapper/220:59:002
40370999390,36cyclictest0-21swapper/220:36:002
40370999390,35cyclictest0-21swapper/223:21:592
403713993833,4cyclictest507161-21turbostat00:35:013
403713993833,4cyclictest418218-21turbostat19:55:013
40371399380,1cyclictest0-21swapper/319:17:593
403709993835,1cyclictest0-21swapper/221:30:002
403709993834,2cyclictest0-21swapper/223:09:552
403709993834,2cyclictest0-21swapper/220:04:562
40370999380,36cyclictest0-21swapper/221:32:002
40370999380,36cyclictest0-21swapper/220:54:582
40370999380,35cyclictest0-21swapper/200:38:592
403707993833,4cyclictest464866-21kworker/u8:1+events_unbound22:25:011
403703993836,1cyclictest0-21swapper/000:09:560
403703993836,1cyclictest0-21swapper/000:09:560
403703993834,3cyclictest482284-21kworker/u8:1+events_unbound23:25:170
403713993734,2cyclictest0-21swapper/322:21:003
403713993734,2cyclictest0-21swapper/322:14:553
403713993731,4cyclictest477909-21turbostat23:00:003
40371399370,36cyclictest0-21swapper/321:44:593
403709993733,2cyclictest0-21swapper/222:14:552
403709993733,2cyclictest0-21swapper/219:39:552
40370999370,35cyclictest0-21swapper/221:44:592
40370999370,2cyclictest0-21swapper/219:17:582
403707993732,4cyclictest431498-21kworker/u8:0+flush-179:020:55:251
403713993634,1cyclictest0-21swapper/323:09:553
403713993634,1cyclictest0-21swapper/320:04:553
403713993633,2cyclictest0-21swapper/321:10:593
403713993631,4cyclictest473137-21turbostat22:45:003
40371399360,35cyclictest0-21swapper/321:31:593
40371399360,35cyclictest0-21swapper/320:54:593
40371399360,35cyclictest0-21swapper/320:35:593
40371399360,1cyclictest0-21swapper/323:33:593
403709993634,1cyclictest0-21swapper/219:55:002
403709993634,1cyclictest0-21swapper/200:35:012
403709993633,2cyclictest0-21swapper/223:05:002
403709993633,2cyclictest0-21swapper/220:25:002
403709993632,2cyclictest0-21swapper/221:10:582
40370999360,34cyclictest0-21swapper/223:40:592
40370999360,2cyclictest0-21swapper/220:08:592
40370999360,1cyclictest0-21swapper/219:15:002
403707993630,5cyclictest455586-21kworker/u8:2+events_unbound22:05:251
403713993533,1cyclictest0-21swapper/319:28:583
403713993531,3cyclictest0-21swapper/319:44:553
403713993529,5cyclictest413435-21turbostat19:40:003
403713993528,5cyclictest482685-21turbostat23:15:003
40371399350,34cyclictest0-21swapper/300:38:583
403709993531,2cyclictest0-21swapper/222:54:552
403709993531,2cyclictest0-21swapper/220:29:542
403709993530,2cyclictest0-21swapper/220:19:562
403709993530,2cyclictest0-21swapper/219:44:552
403713993429,4cyclictest424841-21turbostat20:15:013
40371399340,33cyclictest0-21swapper/323:40:593
403709993427,6cyclictest0-21swapper/222:37:592
40370999340,27cyclictest0-21swapper/223:10:152
403707993430,3cyclictest447372-21kworker/u8:0+events_unbound22:15:161
403713993331,1cyclictest0-21swapper/320:29:543
403713993330,2cyclictest0-21swapper/322:54:553
403713993330,2cyclictest0-21swapper/320:19:563
403713993327,5cyclictest434674-21turbostat20:45:003
403709993331,1cyclictest0-21swapper/223:00:002
403713993227,4cyclictest474732-21turbostat22:50:003
403713993225,6cyclictest0-21swapper/323:25:143
403713993225,6cyclictest0-21swapper/322:25:593
403709993230,1cyclictest0-21swapper/222:45:002
403709993228,2cyclictest0-21swapper/221:51:002
403709993227,2cyclictest0-21swapper/219:46:592
403713993128,2cyclictest0-21swapper/321:08:593
403713993126,4cyclictest452166-21turbostat21:39:593
403709993129,1cyclictest0-21swapper/220:15:002
403709993127,2cyclictest0-21swapper/221:17:002
403709993124,5cyclictest0-21swapper/222:25:592
40370999310,2cyclictest0-21swapper/223:25:142
403713993028,1cyclictest0-21swapper/321:50:593
403713993028,1cyclictest0-21swapper/321:16:593
403713993027,2cyclictest0-21swapper/319:46:583
403713993024,5cyclictest0-21swapper/322:15:153
40371399300,28cyclictest0-21swapper/323:35:153
40371399300,1cyclictest0-21swapper/319:55:033
403709993027,2cyclictest0-21swapper/221:08:582
403709993027,2cyclictest0-21swapper/220:45:012
403709993024,5cyclictest0-21swapper/222:15:152
40370999300,28cyclictest0-21swapper/200:05:032
40370999300,28cyclictest0-21swapper/200:05:022
40370999300,24cyclictest0-21swapper/219:55:022
403713992926,2cyclictest0-21swapper/300:20:123
403713992925,2cyclictest0-21swapper/321:59:553
40371399290,27cyclictest0-21swapper/300:05:033
40371399290,27cyclictest0-21swapper/300:05:033
403709992926,2cyclictest0-21swapper/222:30:142
403709992926,2cyclictest0-21swapper/222:30:142
403709992926,1cyclictest0-21swapper/222:50:002
403709992925,2cyclictest0-21swapper/221:59:552
40370999290,27cyclictest0-21swapper/223:35:142
40370999290,27cyclictest0-21swapper/200:20:112
403713992825,2cyclictest0-21swapper/322:30:143
403713992825,2cyclictest0-21swapper/322:30:143
40371399280,26cyclictest0-21swapper/320:05:123
40371399280,26cyclictest0-21swapper/300:00:153
40371399280,26cyclictest0-21swapper/300:00:153
40370999280,25cyclictest0-21swapper/200:17:002
403707992823,4cyclictest447372-21kworker/u8:0+events_unbound21:50:221
4670791270,7ptp4l61690irq/136-eno1-TxRx-022:09:553
4670791270,6ptp4l441ktimers/323:20:013
403709992725,1cyclictest0-21swapper/221:39:592
403709992723,2cyclictest0-21swapper/223:17:592
403709992722,2cyclictest0-21swapper/220:31:592
403707992720,5cyclictest723-21Xorg21:56:001
4670791260,6ptp4l61690irq/136-eno1-TxRx-023:55:003
4670791260,6ptp4l61690irq/136-eno1-TxRx-021:49:563
403709992623,2cyclictest0-21swapper/222:09:592
403709992622,2cyclictest0-21swapper/219:22:132
4670791250,6ptp4l61690irq/136-eno1-TxRx-022:04:553
4670791250,6ptp4l441ktimers/319:24:563
403713992522,2cyclictest0-21swapper/300:30:003
40371399250,24cyclictest0-21swapper/300:16:593
403709992522,2cyclictest0-21swapper/223:45:592
403709992521,2cyclictest0-21swapper/223:53:592
403713992422,1cyclictest0-21swapper/323:45:593
403713992421,2cyclictest0-21swapper/320:32:003
403709992422,1cyclictest0-21swapper/200:30:002
403709992421,2cyclictest0-21swapper/221:45:592
403709992320,2cyclictest0-21swapper/219:32:582
403709992219,2cyclictest0-21swapper/223:57:582
403709992016,2cyclictest0-21swapper/222:03:592
403703991713,2cyclictest403853-21latency_hist19:10:010
4037099930,2cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional