You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-21 - 22:15
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Sat Jun 21, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
378454699119109,8cyclictest3798813-21apt-get19:55:031
378454699119106,10cyclictest3798803-21latency_hist19:55:011
378454599110104,4cyclictest3860551-21kworker/u8:0+flush-179:000:04:010
37845469910594,8cyclictest3868522-21latency_hist23:30:011
37845459910594,9cyclictest3883669-21apt-get00:15:160
37845469910392,9cyclictest3802495-21/usr/sbin/munin20:05:251
37845459910392,8cyclictest3790855-21latency_hist19:30:010
37845469910292,8cyclictest3828101-21gpgv21:25:011
37845459910089,8cyclictest3857379-21latency_hist22:55:000
3784546999972,19cyclictest3836159-21kworker/u8:0+flush-179:022:05:231
3784546999885,10cyclictest3865609-21apt-get23:20:141
3784545999783,11cyclictest3787886-21apt-get19:20:150
3784546999685,8cyclictest3852422-21kworker/u8:3+events_unbound22:40:021
3784545999482,10cyclictest3888709-21apt-get00:30:190
3784545999482,10cyclictest3888709-21apt-get00:30:190
3784545999383,8cyclictest3786022-21latency_hist19:15:010
3784546999182,7cyclictest3105085-21systemd-journal19:20:001
3784546999180,9cyclictest3881570-21latency_hist00:10:011
3784546999179,10cyclictest3838021-21apt-get21:55:121
3784545999080,8cyclictest1153969-21cups-browsed00:00:000
3784546998978,8cyclictest3871962-21latency_hist23:40:011
3784546998877,9cyclictest3863997-21apt-get23:15:161
3784546998781,4cyclictest3873588-21kworker/u8:3+flush-179:023:55:151
3784546998777,7cyclictest3823228-21kworker/u8:1+events_unbound21:25:021
3784546998775,10cyclictest3849427-21apt-get22:30:121
3784546998576,7cyclictest3792442-21latency_hist19:35:011
3784546998573,10cyclictest3856050-21apt-get22:50:151
3784545998475,7cyclictest3847839-21apt-get22:25:140
3784545998475,6cyclictest3852346-21latency_hist22:40:000
3784545998372,8cyclictest3870369-21latency_hist23:35:010
3784545998268,11cyclictest3833266-21apt-get21:40:120
3784546998067,10cyclictest3879984-21latency_hist00:05:021
3784546998066,11cyclictest502-21rs:main1
3784546997967,8cyclictest3868972-21cat23:30:021
3784546997664,9cyclictest3858986-21latency_hist23:00:011
3784545997669,4cyclictest3798851-21kworker/u8:3+events_unbound19:55:150
3784545997668,5cyclictest3862395-21kworker/u8:2+flush-179:023:15:210
3784545997563,9cyclictest3795889-21apt-get19:45:120
3784546997463,8cyclictest3862396-21/usr/sbin/munin23:10:311
3784545997465,6cyclictest3888455-21latency_hist00:30:020
3784545997464,8cyclictest3831640-21/usr/sbin/munin21:35:260
3784545997361,10cyclictest3105085-21systemd-journal00:15:000
3784545997350,20cyclictest3872232-21apt-get23:40:120
3784546997264,6cyclictest3842782-21kworker/u8:2+flush-179:023:05:001
3784546997261,8cyclictest3845866-21gpgv22:20:011
3784546997259,10cyclictest3105085-21systemd-journal00:20:011
3784545997265,5cyclictest3852422-21kworker/u8:3+flush-179:023:00:200
3784545997165,5cyclictest3770005-21kworker/u8:2+events_unbound19:45:000
3784545997164,5cyclictest502-21rs:main0
3784545997144,8cyclictest3862161-21latency_hist23:10:010
3784546997064,4cyclictest3770005-21kworker/u8:2+flush-179:019:20:281
3784546997061,7cyclictest3813638-21kworker/u8:2+events_unbound20:45:271
3784546996951,15cyclictest3830073-21apt-get21:30:151
3784546996863,4cyclictest3770005-21kworker/u8:2+flush-179:019:45:221
3784546996859,6cyclictest3873588-21kworker/u8:3+flush-179:023:50:221
3784546996858,7cyclictest3818410-21apt-get20:55:121
3784545996855,11cyclictest3857634-21apt-get22:55:150
3784545996761,4cyclictest3836159-21kworker/u8:0+flush-179:021:50:240
3784545996759,6cyclictest3781615-21kworker/u8:3+flush-179:019:15:250
3784545996757,7cyclictest3854212-21latency_hist22:45:010
3784546996658,6cyclictest3804097-21apt-get20:10:111
3784546996557,6cyclictest3890304-21apt-get00:35:161
3784546996554,9cyclictest3851012-21apt-get22:35:131
3784545996557,6cyclictest3820017-21apt-get21:00:160
3784545996557,6cyclictest3813638-21kworker/u8:2+events_unbound20:45:150
3784545996554,8cyclictest3824777-21latency_hist21:15:010
3784548996461,2cyclictest0-21swapper/219:30:582
3784546996455,7cyclictest3825319-21diskstats21:15:171
3784545996456,7cyclictest3876735-21latency_hist23:55:010
3784545996356,5cyclictest3812336-21kworker/u8:0+flush-179:020:35:250
3784552996260,1cyclictest0-21swapper/319:30:593
378455299620,60cyclictest0-21swapper/323:34:003
378454899620,1cyclictest0-21swapper/223:33:592
3784546996155,4cyclictest3816592-21kworker/u8:3+flush-179:021:10:151
3784545996154,5cyclictest3798851-21kworker/u8:3+flush-179:020:30:240
3784546996041,17cyclictest3852422-21kworker/u8:3+flush-179:022:45:181
3784552995952,5cyclictest3833006-21turbostat21:40:013
3784546995956,2cyclictest3784464-21kworker/u8:1+events_unbound19:35:181
3784546995952,5cyclictest3878386-21kworker/u8:2+flush-179:000:10:031
3784545995953,5cyclictest3807272-21kworker/u8:1+flush-179:020:20:210
3784545995946,10cyclictest3813657-21apt-get20:40:130
3784546995852,4cyclictest3873588-21kworker/u8:3+events_unbound23:45:151
3784546995836,19cyclictest3860829-21apt-get23:05:151
3784545995854,3cyclictest3860551-21kworker/u8:0+flush-179:000:35:270
3784545995852,5cyclictest3816592-21kworker/u8:3+events_unbound21:00:010
3784545995852,4cyclictest3875147-21latency_hist23:50:000
3784545995851,5cyclictest3860551-21kworker/u8:0+events_unbound23:20:190
3784546995744,10cyclictest3810444-21kworker/u8:1+flush-179:021:00:201
3784545995751,5cyclictest3798851-21kworker/u8:3+events_unbound20:15:250
3784545995647,7cyclictest3840962-21latency_hist22:05:010
3784546995548,6cyclictest3833046-21kworker/u8:3+flush-179:022:00:181
3784546995547,6cyclictest3816592-21kworker/u8:3+events_unbound21:05:031
3784545995549,5cyclictest3823228-21kworker/u8:1+events_unbound21:15:160
3784546995248,3cyclictest3829853-21kworker/u8:2+flush-179:021:45:271
3784546995246,4cyclictest3813638-21kworker/u8:2+events_unbound20:40:151
3784545995243,7cyclictest3829853-21kworker/u8:2+flush-179:021:45:200
3784545995144,5cyclictest3816592-21kworker/u8:3+flush-179:020:50:180
3784545995139,10cyclictest3826641-21apt-get21:20:160
3784546995042,6cyclictest3784464-21kworker/u8:1+events_unbound19:10:231
3784545995042,5cyclictest3885263-21kworker/u8:2+events_unbound00:20:100
3784545995041,7cyclictest3860551-21kworker/u8:0+flush-179:023:25:190
3784546994943,5cyclictest3798851-21kworker/u8:3+events_unbound20:40:011
3784546994942,5cyclictest3838024-21kworker/u8:1+flush-179:022:25:211
3784546994939,8cyclictest3842782-21kworker/u8:2+flush-179:022:10:131
3784545994939,8cyclictest3105085-21systemd-journal22:23:420
3784546994842,4cyclictest3798851-21kworker/u8:3+flush-179:020:05:011
3784546994840,6cyclictest3105085-21systemd-journal00:30:001
3784545994845,2cyclictest0-21swapper/000:07:000
3784545994842,5cyclictest3770005-21kworker/u8:2+events_unbound19:50:260
3784545994842,4cyclictest1-21systemd22:05:420
4505291470,12ptp4l61790irq/134-eno1-TxRx-019:09:553
3784546994743,3cyclictest3798851-21kworker/u8:3+flush-179:020:30:161
3784546994741,4cyclictest3105085-21systemd-journal20:27:011
3784545994742,4cyclictest3770005-21kworker/u8:2+flush-179:019:30:240
3784546994641,4cyclictest3833046-21kworker/u8:3+flush-179:021:40:211
3784545994641,4cyclictest3823228-21kworker/u8:1+events_unbound21:30:230
3784546994540,4cyclictest3860551-21kworker/u8:0+flush-179:000:30:241
3784546994540,4cyclictest3860551-21kworker/u8:0+flush-179:000:30:231
3784546994532,11cyclictest3770005-21kworker/u8:2+events_unbound19:40:161
3784545994539,5cyclictest3784464-21kworker/u8:1+flush-179:019:35:150
3784545994539,4cyclictest3798851-21kworker/u8:3+events_unbound20:15:010
3784545994436,6cyclictest3105085-21systemd-journal23:35:420
3784545994432,10cyclictest3828772-21diskstats21:25:160
378455299430,1cyclictest0-21swapper/321:42:593
3784545994334,7cyclictest3838024-21kworker/u8:1+flush-179:022:15:190
378455299420,1cyclictest0-21swapper/322:08:003
3784548994238,2cyclictest0-21swapper/220:33:592
378454899420,39cyclictest0-21swapper/223:39:562
378454899420,2cyclictest0-21swapper/221:42:592
3784546994236,4cyclictest1-21systemd19:30:011
3784545994237,4cyclictest3105085-21systemd-journal23:14:410
3784545994236,4cyclictest3105085-21systemd-journal22:45:420
3784545994234,6cyclictest3842782-21kworker/u8:2+events_unbound22:30:150
378454699410,39cyclictest722-21Xorg20:20:001
378454699410,39cyclictest3833008-21ls21:40:011
378454599410,39cyclictest3836159-21kworker/u8:0+i91521:59:590
3784552994035,4cyclictest3807031-21turbostat20:20:003
3784552994033,5cyclictest3808639-21turbostat20:25:003
3784548994037,2cyclictest0-21swapper/220:28:582
378454899400,1cyclictest0-21swapper/222:08:002
3784552993936,2cyclictest0-21swapper/320:28:583
378455299390,36cyclictest0-21swapper/320:39:593
3784548993936,2cyclictest0-21swapper/220:20:002
3784548993935,2cyclictest0-21swapper/219:58:002
378454899390,36cyclictest0-21swapper/223:19:592
3784545993934,3cyclictest3844169-21systemd-run22:15:010
3784552993836,1cyclictest0-21swapper/320:33:593
3784548993835,2cyclictest0-21swapper/220:21:592
3784548993834,2cyclictest0-21swapper/222:19:552
378454699380,37cyclictest722-21Xorg20:23:591
3784545993833,4cyclictest3802266-21turbostat20:09:550
3784552993735,1cyclictest0-21swapper/319:46:583
378455299370,36cyclictest0-21swapper/322:36:593
378455299370,1cyclictest0-21swapper/319:19:563
3784548993733,2cyclictest0-21swapper/221:57:592
3784548993733,2cyclictest0-21swapper/220:04:552
378454899370,35cyclictest0-21swapper/200:39:582
378454899370,2cyclictest0-21swapper/221:49:552
378454899370,1cyclictest0-21swapper/220:39:592
3784546993730,5cyclictest502-21rs:main1
3784546993729,7cyclictest3836804-21fschecks_time21:50:181
3784552993634,1cyclictest0-21swapper/323:39:593
3784552993633,2cyclictest0-21swapper/322:24:543
3784552993633,2cyclictest0-21swapper/319:57:593
378455299360,35cyclictest0-21swapper/321:28:593
378455299360,1cyclictest0-21swapper/323:52:593
3784548993633,2cyclictest0-21swapper/219:47:592
3784548993632,2cyclictest0-21swapper/220:46:592
378454899360,1cyclictest0-21swapper/222:37:002
3784552993532,2cyclictest0-21swapper/321:57:593
3784548993532,2cyclictest0-21swapper/223:40:592
3784548993532,2cyclictest0-21swapper/200:34:562
3784548993532,2cyclictest0-21swapper/200:34:552
3784546993530,4cyclictest3813638-21kworker/u8:2+events_unbound20:50:271
3784552993432,1cyclictest0-21swapper/322:19:553
3784552993431,2cyclictest0-21swapper/320:46:593
3784552993429,4cyclictest3826373-21turbostat21:24:553
3784552993428,5cyclictest3854211-21turbostat22:45:013
3784548993431,2cyclictest0-21swapper/222:56:592
3784548993430,2cyclictest0-21swapper/222:45:002
3784548993430,2cyclictest0-21swapper/219:41:592
3784546993426,7cyclictest3872649-21gltestperf23:40:181
3784552993331,1cyclictest0-21swapper/323:40:583
3784552993331,1cyclictest0-21swapper/321:46:583
3784552993331,1cyclictest0-21swapper/320:04:553
3784548993330,2cyclictest0-21swapper/200:07:582
3784552993230,1cyclictest0-21swapper/300:34:553
3784552993230,1cyclictest0-21swapper/300:34:553
3784552993229,2cyclictest0-21swapper/323:46:593
3784548993228,2cyclictest0-21swapper/221:30:592
378454899320,29cyclictest0-21swapper/223:20:172
378454899320,29cyclictest0-21swapper/223:15:012
378454899320,29cyclictest0-21swapper/221:14:592
3784552993129,1cyclictest0-21swapper/322:56:593
3784552993128,2cyclictest0-21swapper/319:41:593
378455299310,29cyclictest0-21swapper/323:20:173
3784548993129,1cyclictest0-21swapper/223:29:592
3784548993128,2cyclictest0-21swapper/222:29:552
3784548993128,2cyclictest0-21swapper/220:06:592
378454899310,29cyclictest0-21swapper/200:10:042
3784545993128,2cyclictest3809020-21tr20:25:150
3784545993127,3cyclictest0-21swapper/020:04:590
3784552993028,1cyclictest0-21swapper/300:07:593
3784552993027,2cyclictest0-21swapper/323:29:583
378455299300,2cyclictest0-21swapper/323:10:193
378455299300,27cyclictest0-21swapper/321:10:143
378455299300,27cyclictest0-21swapper/319:25:163
3784548993027,2cyclictest0-21swapper/221:00:182
3784548993027,2cyclictest0-21swapper/220:42:592
3784548993027,2cyclictest0-21swapper/219:16:582
3784548993027,2cyclictest0-21swapper/200:29:002
378454899300,2cyclictest0-21swapper/221:20:182
378454899300,21cyclictest0-21swapper/219:25:162
3784552992927,1cyclictest0-21swapper/321:30:593
3784552992927,1cyclictest0-21swapper/321:00:183
3784552992926,2cyclictest0-21swapper/323:05:183
3784552992926,2cyclictest0-21swapper/320:05:163
378455299290,27cyclictest0-21swapper/322:50:153
378455299290,27cyclictest0-21swapper/321:05:023
378455299290,27cyclictest0-21swapper/300:10:023
3784548992926,2cyclictest0-21swapper/219:39:002
3784548992925,2cyclictest0-21swapper/219:54:552
3784548992924,2cyclictest0-21swapper/221:53:592
378454899290,2cyclictest0-21swapper/222:20:022
378454899290,27cyclictest0-21swapper/223:05:182
378454899290,27cyclictest0-21swapper/223:01:582
378454899290,27cyclictest0-21swapper/222:50:162
378454899290,27cyclictest0-21swapper/221:05:022
378454899290,21cyclictest0-21swapper/222:00:152
4505291280,6ptp4l61790irq/134-eno1-TxRx-020:15:003
4505291280,6ptp4l441ktimers/319:35:013
3784552992826,1cyclictest0-21swapper/322:29:563
3784552992826,1cyclictest0-21swapper/320:55:143
3784552992825,2cyclictest0-21swapper/300:29:003
378455299280,27cyclictest0-21swapper/322:00:153
378455299280,21cyclictest3786037-21turbostat19:15:003
3784548992825,2cyclictest0-21swapper/220:55:142
3784548992825,2cyclictest0-21swapper/220:50:592
378454899280,27cyclictest0-21swapper/221:25:012
4505291270,6ptp4l61790irq/134-eno1-TxRx-020:55:003
3784552992725,1cyclictest0-21swapper/300:20:113
3784552992724,2cyclictest0-21swapper/323:18:003
378455299270,25cyclictest0-21swapper/322:45:163
378455299270,25cyclictest0-21swapper/320:40:153
378454899270,25cyclictest0-21swapper/223:55:592
378454899270,25cyclictest0-21swapper/223:48:592
4505291260,6ptp4l61790irq/134-eno1-TxRx-023:05:013
4505291260,6ptp4l441ktimers/319:55:013
4505291260,5ptp4l441ktimers/321:20:013
378455299260,2cyclictest0-21swapper/300:16:593
3784548992624,1cyclictest0-21swapper/200:20:102
4505291250,5ptp4l441ktimers/322:34:553
4505291250,5ptp4l441ktimers/322:15:013
4505291250,4ptp4l441ktimers/319:25:003
3784552992523,1cyclictest0-21swapper/300:36:593
3784552992523,1cyclictest0-21swapper/300:03:593
3784552992522,2cyclictest0-21swapper/321:54:003
3784548992522,2cyclictest0-21swapper/200:19:592
378454899250,1cyclictest0-21swapper/221:39:592
4505291240,4ptp4l61790irq/134-eno1-TxRx-000:00:003
3784545992414,8cyclictest3784409-21latency_hist19:10:010
3784548992321,1cyclictest0-21swapper/200:04:592
3784548992320,2cyclictest0-21swapper/223:54:582
3784548992320,2cyclictest0-21swapper/222:14:552
3784548992320,2cyclictest0-21swapper/221:15:142
201699220,12rtkit-daemon486-21gpm19:07:391
3784548991714,2cyclictest0-21swapper/222:33:592
3784548991513,1cyclictest0-21swapper/222:48:592
3784548991411,2cyclictest0-21swapper/220:13:002
37845489996,2cyclictest0-21swapper/219:10:132
37845489971,4cyclictest3789285-21turbostat19:25:012
37845489930,2cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional