You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 10:17
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Thu Nov 27, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
180397699112101,9cyclictest271-21systemd-journal00:04:010
18039769910999,8cyclictest271-21systemd-journal23:40:000
18039789910696,8cyclictest271-21systemd-journal00:00:001
18039789910181,17cyclictest1836248-21kworker/u8:3+flush-179:021:00:001
1803978999085,4cyclictest1811835-21kworker/u8:3+flush-179:019:45:241
1803978999085,4cyclictest1811835-21kworker/u8:3+flush-179:019:45:231
1803976999080,6cyclictest1830133-21kworker/u8:1+flush-179:020:40:190
1803978998884,3cyclictest1830133-21kworker/u8:1+events_unbound21:50:231
1803978998883,4cyclictest1828540-21kworker/u8:3+events_unbound20:30:191
1803976998272,7cyclictest1805476-21latency_hist19:15:010
180397699810,4cyclictest723-21Xorg21:31:590
1803978998075,4cyclictest1872002-21kworker/u8:1+events_unbound22:50:271
1803976998065,11cyclictest1843157-21latency_hist21:20:000
1803976997968,8cyclictest1902080-21/usr/sbin/munin00:20:240
1803976997967,8cyclictest1813700-21latency_hist19:40:010
1803978997667,3cyclictest1865686-21kworker/u8:0+events_unbound22:40:191
1803976997666,7cyclictest1815304-21latency_hist19:45:020
1803978997469,4cyclictest1867369-21kworker/u8:3+events_unbound22:35:251
1803976997466,5cyclictest1865686-21kworker/u8:0+flush-179:022:40:000
1803978997256,14cyclictest1806142-21kworker/u8:0+flush-179:020:00:261
1803976997165,4cyclictest1845720-21kworker/u8:2+flush-179:022:05:010
1803978997065,4cyclictest1853900-21kworker/u8:3+flush-179:022:10:211
1803978996964,4cyclictest1836248-21kworker/u8:3+events_unbound21:10:201
1803978996863,4cyclictest1833650-21kworker/u8:2+flush-179:021:05:251
1803978996863,4cyclictest1830133-21kworker/u8:1+flush-179:021:00:211
1803978996863,4cyclictest1810801-21kworker/u8:2+events_unbound19:35:151
1803978996863,4cyclictest1803924-21kworker/u8:2+flush-179:019:19:561
1803978996761,4cyclictest1876085-21kworker/u8:3+events_unbound23:05:131
1803978996759,3cyclictest1900267-21kworker/u8:0+events_unbound00:25:251
1803978996662,3cyclictest1832001-21kworker/u8:0+flush-179:020:45:001
1803978996661,4cyclictest1845720-21kworker/u8:2+flush-179:021:34:551
1803978996660,4cyclictest1895365-21kworker/u8:2+events_unbound00:15:141
180397899660,2cyclictest723-21Xorg21:44:591
1803976996659,4cyclictest1900267-21kworker/u8:0+flush-179:000:35:160
1803976996659,4cyclictest1806142-21kworker/u8:0+flush-179:020:00:180
1803976996658,6cyclictest1893553-21apt-get23:54:590
1803976996658,5cyclictest1785142-21kworker/u8:1+events_unbound19:15:240
1803978996561,3cyclictest1880854-21kworker/u8:0+events_unbound23:50:271
1803978996556,3cyclictest1880854-21kworker/u8:0+events_unbound23:45:181
1803978996552,10cyclictest1862782-21latency_hist22:20:021
1803978996460,3cyclictest1861905-21kworker/u8:2+events_unbound22:30:231
1803978996459,4cyclictest1845720-21kworker/u8:2+events_unbound21:45:251
1803976996457,4cyclictest1830133-21kworker/u8:1+flush-179:021:35:200
1803978996358,4cyclictest1830133-21kworker/u8:1+events_unbound20:50:221
1803978996258,3cyclictest1876085-21kworker/u8:3+flush-179:023:10:251
1803978996258,3cyclictest1785142-21kworker/u8:1+events_unbound19:10:141
1803978996257,4cyclictest1811835-21kworker/u8:3+flush-179:020:10:281
1803978996253,3cyclictest1880854-21kworker/u8:0+events_unbound23:20:181
1803976996251,8cyclictest1880809-21latency_hist23:15:000
1803978996157,3cyclictest1811835-21kworker/u8:3+events_unbound19:40:141
1803976996155,4cyclictest1897314-21apt-get00:05:110
1803976996154,4cyclictest1853900-21kworker/u8:3+flush-179:022:20:220
1803978996056,3cyclictest1830133-21kworker/u8:1+flush-179:021:25:241
1803978996056,3cyclictest1811835-21kworker/u8:3+flush-179:019:59:541
1803978996056,3cyclictest1811835-21kworker/u8:3+flush-179:019:50:161
1803978996055,4cyclictest1864103-21kworker/u8:1+events_unbound22:25:141
1803976996054,3cyclictest1895365-21kworker/u8:2+flush-179:000:00:010
1803978995955,3cyclictest1845720-21kworker/u8:2+flush-179:021:55:211
1803978995954,4cyclictest1887160-21kworker/u8:3+flush-179:023:35:251
1803978995954,4cyclictest1826675-21kworker/u8:0+flush-179:020:25:211
1803978995950,3cyclictest1856660-21kworker/u8:0+events_unbound22:20:131
1803978995853,4cyclictest1806142-21kworker/u8:0+events_unbound19:25:131
1803976995850,6cyclictest1880854-21kworker/u8:0+events_unbound23:20:000
1803976995849,6cyclictest1848175-21kworker/u8:0+flush-179:021:40:210
180397699581,54cyclictest1838428-21munin-run21:05:000
1803978995753,3cyclictest1897042-21kworker/u8:4+flush-179:000:14:561
1803978995752,4cyclictest1880854-21kworker/u8:0+events_unbound23:30:171
1803978995752,4cyclictest1828540-21kworker/u8:3+events_unbound20:36:361
1803978995748,3cyclictest1895365-21kworker/u8:2+events_unbound00:05:181
1803976995751,4cyclictest1852984-21apt-get21:50:030
1803978995634,3cyclictest29-21ksoftirqd/123:05:001
1803976995650,4cyclictest1856660-21kworker/u8:0+flush-179:022:15:200
1803978995551,3cyclictest1900267-21kworker/u8:0+flush-179:000:35:301
1803978995550,4cyclictest1880854-21kworker/u8:0+flush-179:023:44:551
1803976995545,7cyclictest1825104-21latency_hist20:15:010
1803976995451,2cyclictest0-21swapper/023:46:000
1803978995349,3cyclictest1806142-21kworker/u8:0+events_unbound19:20:281
1803978995349,3cyclictest1806142-21kworker/u8:0+events_unbound19:20:281
1803978995348,4cyclictest1830133-21kworker/u8:1+events_unbound21:15:171
1803976995350,2cyclictest1808908-21kworker/u8:1+events_unbound19:25:220
1803978995242,4cyclictest1895365-21kworker/u8:2+events_unbound00:20:231
1803981995047,2cyclictest0-21swapper/223:46:002
1803978995045,4cyclictest1880854-21kworker/u8:0+flush-179:023:17:201
1803976995043,5cyclictest1861905-21kworker/u8:2+flush-179:023:00:010
1803976995041,7cyclictest1856408-21latency_hist22:00:000
1803976995039,9cyclictest1878181-21diskstats23:05:140
1803978994944,4cyclictest1845720-21kworker/u8:2+events_unbound22:00:241
1803978994944,4cyclictest1806142-21kworker/u8:0+events_unbound20:07:211
1803976994946,2cyclictest0-21swapper/021:49:580
1803976994943,5cyclictest1806142-21kworker/u8:0+flush-179:019:20:240
1803976994943,5cyclictest1806142-21kworker/u8:0+flush-179:019:20:240
1803978994844,3cyclictest1830133-21kworker/u8:1+flush-179:021:24:561
1803978994843,4cyclictest1861905-21kworker/u8:2+events_unbound22:57:191
4670791470,12ptp4l61690irq/136-eno1-TxRx-019:10:013
180398299470,44cyclictest1807082-21turbostat19:20:013
1803976994742,4cyclictest1830133-21kworker/u8:1+flush-179:020:45:230
1803978994642,3cyclictest1900267-21kworker/u8:0+events_unbound00:33:311
1803978994642,3cyclictest1830133-21kworker/u8:1+events_unbound20:46:391
1803976994640,4cyclictest1887160-21kworker/u8:3+events_unbound23:35:010
1803981994542,2cyclictest0-21swapper/221:49:582
1803978994541,3cyclictest1811835-21kworker/u8:3+events_unbound20:16:201
1803978994440,3cyclictest1865686-21kworker/u8:0+events_unbound22:45:191
1803981994339,2cyclictest0-21swapper/223:13:592
1803978994339,3cyclictest475-21dbus-daemon00:04:011
1803978994339,3cyclictest1880854-21kworker/u8:0+events_unbound23:25:251
1803978994335,6cyclictest1810514-21apt-get19:30:121
1803976994340,2cyclictest1833650-21kworker/u8:2+events_unbound21:05:140
1803976994337,4cyclictest1880854-21kworker/u8:0+flush-179:023:45:000
1803978994236,4cyclictest1848198-21turbostat21:39:551
1803976994237,3cyclictest1900267-21kworker/u8:0+flush-179:000:25:180
1803976994237,3cyclictest1846617-21awk21:30:010
1803976994234,6cyclictest1828540-21kworker/u8:3+events_unbound20:30:160
1803976994234,5cyclictest1803863-21latency_hist19:10:020
1803976994232,7cyclictest1884005-21latency_hist23:25:010
1803982994134,5cyclictest1818485-21turbostat19:55:003
180398299410,39cyclictest1869421-21turbostat22:40:003
180398299410,39cyclictest1839986-21turbostat21:10:003
1803978994134,6cyclictest1853900-21kworker/u8:3+events_unbound22:10:001
1803976994135,4cyclictest1861905-21kworker/u8:2+flush-179:022:30:190
1803976994134,5cyclictest1811835-21kworker/u8:3+flush-179:019:55:220
1803982994034,5cyclictest1854821-21turbostat21:55:003
1803982994034,4cyclictest1825120-21turbostat20:15:003
180398299400,38cyclictest1897069-21turbostat00:05:003
180398299400,38cyclictest1895403-21turbostat00:00:003
180398299400,38cyclictest1890385-21turbostat23:45:003
180398299400,38cyclictest1858017-21turbostat22:05:013
180398299400,38cyclictest1856415-21turbostat21:59:593
1803978994036,3cyclictest1825089-21kworker/u8:1+events_unbound20:20:031
1803976994030,7cyclictest1810801-21kworker/u8:2+flush-179:019:30:200
1803976994027,7cyclictest1876009-21kworker/u8:1+events_unbound23:00:140
180397699400,2cyclictest0-21swapper/021:14:000
1803982993933,4cyclictest1813708-21turbostat19:40:013
1803982993932,6cyclictest1849799-21turbostat21:40:003
1803982993932,5cyclictest1851395-21turbostat21:45:013
180398299390,37cyclictest1891964-21turbostat23:50:003
180398299390,37cyclictest1887189-21turbostat23:35:003
180398199390,37cyclictest0-21swapper/221:14:002
180398199390,35cyclictest0-21swapper/223:27:592
1803976993933,4cyclictest1876009-21kworker/u8:1+flush-179:023:25:240
1803976993929,6cyclictest0-21swapper/022:45:140
1803982993832,4cyclictest1867822-21turbostat22:35:003
180398299380,3cyclictest1903435-21turbostat00:25:003
180398199380,36cyclictest0-21swapper/221:55:592
1803976993835,2cyclictest723-21Xorg22:28:590
1803976993833,3cyclictest271-21systemd-journal20:10:000
1803976993827,8cyclictest1859598-21kworker/u8:1+events_unbound22:10:160
1803976993826,9cyclictest271-21systemd-journal21:00:000
180397699380,30cyclictest0-21swapper/000:12:590
1803982993734,2cyclictest0-21swapper/323:09:593
1803982993734,2cyclictest0-21swapper/322:29:543
1803982993732,4cyclictest1833616-21turbostat20:50:003
1803981993734,2cyclictest0-21swapper/221:50:592
1803981993734,2cyclictest0-21swapper/220:52:592
1803981993734,2cyclictest0-21swapper/220:41:592
180398199370,35cyclictest0-21swapper/222:02:592
1803976993731,4cyclictest1830133-21kworker/u8:1+flush-179:021:25:000
1803976993731,4cyclictest1830133-21kworker/u8:1+flush-179:020:50:240
1803976993731,4cyclictest1811835-21kworker/u8:3+events_unbound19:45:140
1803976993731,4cyclictest1811835-21kworker/u8:3+events_unbound19:45:130
1803976993728,6cyclictest1826953-21/usr/sbin/munin20:20:150
180397699370,1cyclictest0-21swapper/020:38:580
1803982993631,4cyclictest1821947-21turbostat20:05:013
1803982993630,5cyclictest1832032-21turbostat20:44:593
180398299360,35cyclictest0-21swapper/323:28:593
180398299360,35cyclictest0-21swapper/322:51:593
180398299360,35cyclictest0-21swapper/320:18:003
180398299360,35cyclictest0-21swapper/319:49:003
180398299360,35cyclictest0-21swapper/319:48:593
180398299360,1cyclictest0-21swapper/323:03:593
1803981993634,1cyclictest0-21swapper/221:24:582
1803981993633,2cyclictest0-21swapper/200:34:552
1803981993633,2cyclictest0-21swapper/200:04:552
180398199360,1cyclictest0-21swapper/219:29:552
1803976993631,3cyclictest1906610-21munin-run00:35:000
1803976993629,5cyclictest1853900-21kworker/u8:3+flush-179:022:05:240
1803981993533,1cyclictest0-21swapper/222:34:582
1803981993533,1cyclictest0-21swapper/221:30:002
1803981993532,2cyclictest0-21swapper/223:19:592
1803981993532,2cyclictest0-21swapper/221:34:002
1803976993532,2cyclictest1828558-21cron20:30:010
1803976993530,4cyclictest1811835-21kworker/u8:3+flush-179:019:50:140
1803976993528,3cyclictest1869681-21apt-get22:40:110
1803982993432,1cyclictest0-21swapper/322:46:593
1803982993428,4cyclictest1836789-21turbostat21:00:003
1803981993431,2cyclictest0-21swapper/223:40:592
1803981993431,2cyclictest0-21swapper/222:29:002
1803981993431,2cyclictest0-21swapper/221:08:582
1803982993331,1cyclictest0-21swapper/323:14:553
1803982993331,1cyclictest0-21swapper/300:26:593
1803982993330,2cyclictest0-21swapper/320:05:583
1803982993329,2cyclictest0-21swapper/319:25:593
180398299330,1cyclictest0-21swapper/319:24:593
180398299330,1cyclictest0-21swapper/319:24:593
1803981993330,2cyclictest0-21swapper/221:19:552
1803976993326,5cyclictest1825089-21kworker/u8:1+events_unbound20:15:260
1803982993230,1cyclictest0-21swapper/300:14:543
1803982993228,2cyclictest0-21swapper/321:16:593
1803981993229,2cyclictest0-21swapper/220:57:002
1803981993229,2cyclictest0-21swapper/220:49:582
1803981993229,2cyclictest0-21swapper/220:22:592
1803981993229,2cyclictest0-21swapper/219:33:002
180398199320,30cyclictest0-21swapper/221:35:582
1803982993128,2cyclictest0-21swapper/300:07:593
180398199310,27cyclictest0-21swapper/223:30:142
180398199310,27cyclictest0-21swapper/222:40:112
180398199310,26cyclictest0-21swapper/219:20:152
180398199310,26cyclictest0-21swapper/219:20:142
4670791300,11ptp4l61690irq/136-eno1-TxRx-020:29:563
1803982993028,1cyclictest0-21swapper/319:10:133
1803981993027,2cyclictest0-21swapper/223:35:122
1803981993027,2cyclictest0-21swapper/222:20:022
180398199300,27cyclictest0-21swapper/223:00:132
180398199300,27cyclictest0-21swapper/219:55:152
180398199300,1cyclictest0-21swapper/222:59:592
4670791290,6ptp4l441ktimers/323:00:013
1803982992925,2cyclictest0-21swapper/322:15:123
1803982992924,4cyclictest1841573-21turbostat21:15:003
180398299290,27cyclictest0-21swapper/300:16:003
180398299290,1cyclictest0-21swapper/322:20:043
1803981992926,2cyclictest0-21swapper/222:45:142
1803981992926,2cyclictest0-21swapper/221:00:122
1803981992925,2cyclictest0-21swapper/219:19:002
1803976992923,4cyclictest1873940-21phc2sys22:50:230
4670791280,6ptp4l441ktimers/321:50:023
4670791280,6ptp4l441ktimers/320:55:013
180398299280,2cyclictest0-21swapper/323:35:123
180398299280,26cyclictest0-21swapper/322:10:143
180398299280,26cyclictest0-21swapper/321:00:123
180398299280,1cyclictest0-21swapper/320:34:003
1803981992825,2cyclictest0-21swapper/223:05:122
1803981992825,2cyclictest0-21swapper/219:50:132
180398199280,2cyclictest0-21swapper/223:21:592
180398199280,27cyclictest0-21swapper/219:35:012
4670791270,6ptp4l441ktimers/323:25:003
1803982992724,2cyclictest0-21swapper/322:43:593
1803981992724,2cyclictest0-21swapper/219:10:122
1803981992723,2cyclictest0-21swapper/220:16:002
4670791260,7ptp4l61690irq/136-eno1-TxRx-020:00:003
4670791260,6ptp4l61690irq/136-eno1-TxRx-020:39:593
4670791260,6ptp4l61690irq/136-eno1-TxRx-000:35:013
4670791260,6ptp4l441ktimers/323:55:013
4670791260,5ptp4l61690irq/136-eno1-TxRx-021:30:023
1803982992624,1cyclictest0-21swapper/319:44:563
1803982992623,2cyclictest0-21swapper/322:05:583
1803981992624,1cyclictest0-21swapper/222:10:002
1803981992624,1cyclictest0-21swapper/200:15:002
1803981992623,2cyclictest0-21swapper/222:38:592
1803981992623,2cyclictest0-21swapper/221:40:122
1803981992623,2cyclictest0-21swapper/200:17:582
1803981992622,2cyclictest0-21swapper/220:31:592
180398199260,2cyclictest0-21swapper/222:19:582
180398199260,24cyclictest0-21swapper/219:45:012
1803976992621,4cyclictest1901967-21latency_hist00:20:000
4670791250,6ptp4l61690irq/136-eno1-TxRx-021:34:553
4670791250,5ptp4l61690irq/136-eno1-TxRx-020:24:553
4670791250,4ptp4l441ktimers/321:25:013
1803982992523,1cyclictest0-21swapper/319:31:583
1803982992522,2cyclictest0-21swapper/300:35:153
1803981992522,2cyclictest0-21swapper/223:56:592
1803981992522,2cyclictest0-21swapper/220:06:592
1803981992522,2cyclictest0-21swapper/220:02:002
1803981992522,2cyclictest0-21swapper/200:35:582
1803981992522,2cyclictest0-21swapper/200:29:002
180398199250,2cyclictest0-21swapper/200:24:592
1803981992421,2cyclictest0-21swapper/222:14:552
4670791230,3ptp4l441ktimers/323:19:563
1803981992321,1cyclictest0-21swapper/220:35:592
1803981992219,2cyclictest0-21swapper/220:14:592
1803981992118,2cyclictest0-21swapper/223:55:002
1803981992116,2cyclictest0-21swapper/219:45:132
1803981992116,2cyclictest0-21swapper/219:45:132
180397899216,3cyclictest29-21ksoftirqd/119:10:021
1803981991815,2cyclictest0-21swapper/220:28:592
1803981991715,1cyclictest0-21swapper/200:09:592
1803981991714,2cyclictest0-21swapper/222:53:592
18039819930,2cyclictest0-21swapper/219:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional