You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 20:58
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Wed Nov 19, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16986919910392,8cyclictest1747417-21latency_hist09:40:001
1698691999282,8cyclictest1711867-21/usr/sbin/munin07:50:231
1698691999282,7cyclictest1805470-21latency_hist12:40:011
1698691998880,4cyclictest1698646-21kworker/u8:2+flush-179:007:20:011
1698687998876,9cyclictest271-21systemd-journal11:50:280
1698691998774,9cyclictest1779513-21latency_hist11:20:011
1698687998780,6cyclictest1747441-21kworker/u8:3+flush-179:009:45:310
1698687998273,7cyclictest271-21systemd-journal07:40:000
1698691997973,4cyclictest1744470-21kworker/u8:2+flush-179:010:15:001
1698687997868,7cyclictest1779745-21/usr/sbin/munin11:20:270
1698687997565,8cyclictest1719839-21latency_hist08:15:000
1698687997563,10cyclictest1714814-21apt-get07:59:590
1698691997366,5cyclictest1710080-21kworker/u8:1+events_unbound08:25:241
1698687997363,8cyclictest1774967-21/usr/sbin/munin11:05:290
1698687997363,8cyclictest1774967-21/usr/sbin/munin11:05:280
1698687997261,8cyclictest1737609-21latency_hist09:10:000
1698687997160,8cyclictest1782675-21latency_hist11:30:010
1698687997060,8cyclictest1700196-21latency_hist07:15:010
1698691996962,4cyclictest1747441-21kworker/u8:3+flush-179:009:50:261
169869399680,65cyclictest0-21swapper/207:38:592
1698691996861,5cyclictest1782709-21kworker/u8:1+events_unbound11:40:001
1698687996858,7cyclictest1763584-21latency_hist10:30:010
1698691996760,5cyclictest1774298-21kworker/u8:2+flush-179:011:10:181
1698698996664,1cyclictest0-21swapper/307:38:593
1698687996658,7cyclictest1729693-21kworker/u8:3+flush-179:009:00:010
1698691996560,4cyclictest1802282-21kworker/u8:2+events_unbound12:30:001
1698687996559,5cyclictest1768400-21kworker/u8:3+flush-179:010:50:260
1698687996357,4cyclictest1755653-21kworker/u8:3+flush-179:010:05:000
1698687996354,7cyclictest1787454-21latency_hist11:45:010
1698693996258,2cyclictest0-21swapper/211:00:592
1698691996256,4cyclictest1777932-21kworker/u8:0+flush-179:012:15:001
1698691996252,8cyclictest1795662-21latency_hist12:10:011
1698691996154,5cyclictest1739432-21/usr/sbin/munin09:15:291
1698691996054,4cyclictest1788489-21kworker/u8:1+flush-179:011:45:251
1698691996054,4cyclictest1729693-21kworker/u8:3+flush-179:008:55:291
1698687996054,4cyclictest1736001-21kworker/u8:0+events_unbound09:10:270
1698687996054,4cyclictest1714861-21kworker/u8:3+flush-179:008:05:150
1698687996054,4cyclictest1714861-21kworker/u8:3+flush-179:008:05:150
1698691995952,5cyclictest1737864-21apt-get09:10:131
1698687995952,5cyclictest271-21systemd-journal10:50:010
1698698995854,2cyclictest0-21swapper/311:00:593
1698691995854,3cyclictest1729693-21kworker/u8:3+flush-179:009:24:551
1698687995853,4cyclictest1747441-21kworker/u8:3+flush-179:009:45:000
1698691995752,4cyclictest1774298-21kworker/u8:2+events_unbound11:25:301
1698687995748,6cyclictest1788489-21kworker/u8:1+events_unbound12:10:190
1698691995649,4cyclictest1788489-21kworker/u8:1+flush-179:012:30:191
1698687995650,4cyclictest1718288-21kworker/u8:2+flush-179:008:20:000
1698687995646,7cyclictest1710045-21latency_hist07:45:010
1698687995544,9cyclictest1721963-21apt-get08:20:130
1698691995446,6cyclictest1742623-21/usr/sbin/munin09:25:251
1698691995446,6cyclictest1742623-21/usr/sbin/munin09:25:251
1698691995443,9cyclictest1754303-21apt-get10:00:131
1698687995449,4cyclictest1721661-21kworker/u8:3+flush-179:008:30:250
1698687995447,6cyclictest1799152-21kworker/u8:3+flush-179:012:40:010
1698693995350,2cyclictest0-21swapper/211:18:592
1698691995342,8cyclictest1714802-21latency_hist08:00:011
1698687995348,4cyclictest191rcuc/007:20:010
1698687995348,4cyclictest1768400-21kworker/u8:3+flush-179:011:15:240
1698687995343,8cyclictest271-21systemd-journal09:35:000
1698691995247,4cyclictest1755653-21kworker/u8:3+flush-179:010:10:011
1698691995246,4cyclictest1726036-21kworker/u8:2+events_unbound08:35:251
1698691995243,6cyclictest1799128-21latency_hist12:20:011
1698691995233,13cyclictest1736001-21kworker/u8:0+events_unbound09:05:211
1698687995242,7cyclictest1789299-21latency_hist11:50:010
1698691995148,2cyclictest723-21Xorg10:52:591
1698687995147,3cyclictest1705495-21kworker/u8:0+events_unbound07:45:250
1698687995146,4cyclictest1788489-21kworker/u8:1+flush-179:012:15:230
1698687995044,5cyclictest1771570-21kworker/u8:1+flush-179:011:00:270
4670791490,13ptp4l441ktimers/307:09:553
1698691994942,6cyclictest1758837-21kworker/u8:0+events_unbound10:25:301
1698687994941,6cyclictest1788489-21kworker/u8:1+flush-179:012:00:260
1698691994841,5cyclictest1758837-21kworker/u8:0+flush-179:010:45:261
1698691994840,5cyclictest1729658-21latency_hist08:45:001
1698691994838,8cyclictest1794085-21munin-run12:05:001
1698687994842,5cyclictest1788489-21kworker/u8:1+flush-179:012:25:280
1698691994741,4cyclictest1777932-21kworker/u8:0+events_unbound11:45:011
1698687994743,3cyclictest1729693-21kworker/u8:3+flush-179:009:15:160
1698687994742,4cyclictest1782709-21kworker/u8:1+flush-179:011:30:170
1698687994742,4cyclictest1782709-21kworker/u8:1+flush-179:011:30:170
1698687994742,4cyclictest1758837-21kworker/u8:0+events_unbound10:55:250
1698687994640,5cyclictest1728843-21kworker/u8:1+flush-179:008:40:260
1698687994637,7cyclictest271-21systemd-journal12:34:550
1698691994540,4cyclictest1729693-21kworker/u8:3+flush-179:009:00:241
1698691994540,4cyclictest1718288-21kworker/u8:2+events_unbound08:15:271
1698687994536,7cyclictest1799152-21kworker/u8:3+events_unbound12:25:000
1698687994533,10cyclictest271-21systemd-journal08:35:290
1698691994440,3cyclictest1758837-21kworker/u8:0+events_unbound10:35:281
1698687994438,5cyclictest1729693-21kworker/u8:3+events_unbound09:00:290
169868799442,7cyclictest0-21swapper/009:50:160
169868799440,1cyclictest0-21swapper/009:55:590
1698691994339,3cyclictest723-21Xorg09:55:591
1698691994333,9cyclictest1788489-21kworker/u8:1+flush-179:011:50:251
169869199432,36cyclictest1767034-21apt-get10:40:161
1698687994338,4cyclictest1788489-21kworker/u8:1+flush-179:012:05:240
1698687994338,4cyclictest1758837-21kworker/u8:0+flush-179:010:40:280
1698691994238,3cyclictest1784268-21munin-run11:35:001
1698691994238,3cyclictest1784268-21munin-run11:34:591
1698691994237,3cyclictest1743953-21kworker/u8:1+events_unbound09:30:241
1698691994235,5cyclictest1715624-21diskstats08:00:191
169869199420,40cyclictest723-21Xorg07:38:591
1698687994238,3cyclictest1729693-21kworker/u8:3+flush-179:008:55:000
1698687994237,4cyclictest1744470-21kworker/u8:2+flush-179:010:20:220
1698698994139,1cyclictest0-21swapper/307:13:593
169869399410,1cyclictest0-21swapper/212:26:592
1698691994136,4cyclictest1749528-21kworker/u8:0+flush-179:009:49:561
1698691994136,4cyclictest1695619-21kworker/u8:1+flush-179:007:30:231
1698691994136,3cyclictest1721661-21kworker/u8:3+flush-179:008:25:011
169868799410,39cyclictest723-21Xorg08:04:590
169869899400,38cyclictest1742396-21turbostat09:29:553
169869899400,38cyclictest1742396-21turbostat09:29:553
1698693994037,2cyclictest0-21swapper/209:19:582
1698691994033,4cyclictest1765171-21latency_hist10:35:011
1698691994029,10cyclictest1714861-21kworker/u8:3+flush-179:008:05:211
1698691994029,10cyclictest1714861-21kworker/u8:3+flush-179:008:05:211
169869199400,38cyclictest1726475-21ls08:35:011
1698687994036,3cyclictest1710080-21kworker/u8:1+events_unbound07:50:230
1698687994035,4cyclictest1705495-21kworker/u8:0+events_unbound07:30:280
1698687994033,5cyclictest271-21systemd-journal09:25:010
169868799402,5cyclictest1703904-21apt-get07:25:150
202199390,3rtkit-daemon2020-21rtkit-daemon11:02:551
169869899390,38cyclictest1773144-21turbostat11:00:003
169869899390,37cyclictest1711638-21turbostat07:54:553
169869399390,36cyclictest0-21swapper/210:57:592
1698691993934,4cyclictest1774298-21kworker/u8:2+events_unbound11:05:201
1698691993934,4cyclictest1774298-21kworker/u8:2+events_unbound11:05:201
169869199390,2cyclictest723-21Xorg07:21:591
1698687993936,2cyclictest723-21Xorg11:55:590
1698687993935,2cyclictest723-21Xorg11:12:580
1698687993935,2cyclictest723-21Xorg10:12:590
1698687993934,4cyclictest1777932-21kworker/u8:0+flush-179:011:35:250
202199380,4rtkit-daemon2020-21rtkit-daemon11:21:021
1698698993836,1cyclictest0-21swapper/312:00:593
1698693993834,2cyclictest0-21swapper/210:37:002
1698693993834,2cyclictest0-21swapper/208:33:592
169869399380,36cyclictest0-21swapper/209:25:002
169869399380,35cyclictest0-21swapper/210:49:552
169869399380,35cyclictest0-21swapper/209:47:582
1698691993833,4cyclictest1744470-21kworker/u8:2+flush-179:010:19:551
1698691993833,4cyclictest1724862-21kworker/u8:0+flush-179:008:45:291
1698691993832,4cyclictest1744470-21kworker/u8:2+flush-179:010:20:261
169868799380,37cyclictest0-21swapper/010:33:000
1698698993734,2cyclictest0-21swapper/309:25:003
169869899370,36cyclictest0-21swapper/311:34:593
169869899370,36cyclictest0-21swapper/311:34:593
169869899370,36cyclictest0-21swapper/311:18:593
169869899370,36cyclictest0-21swapper/309:03:593
169869899370,35cyclictest0-21swapper/307:30:593
169869899370,1cyclictest0-21swapper/311:29:593
1698693993734,2cyclictest0-21swapper/208:15:002
169869399370,35cyclictest0-21swapper/208:43:592
169869399370,35cyclictest0-21swapper/208:26:002
1698691993735,1cyclictest0-21swapper/108:54:551
1698691993732,4cyclictest1799152-21kworker/u8:3+events_unbound12:20:271
1698691993731,4cyclictest1705235-21latency_hist07:30:011
1698687993733,2cyclictest723-21Xorg10:16:580
1698687993732,4cyclictest1736001-21kworker/u8:0+flush-179:009:35:230
1698687993730,5cyclictest1756115-21cat10:05:160
202199360,3rtkit-daemon2020-21rtkit-daemon08:26:380
1698698993634,1cyclictest0-21swapper/309:47:583
1698698993634,1cyclictest0-21swapper/308:43:593
1698698993633,2cyclictest0-21swapper/311:09:563
1698698993633,2cyclictest0-21swapper/311:09:553
1698698993633,2cyclictest0-21swapper/309:42:583
169869899360,35cyclictest0-21swapper/310:14:593
169869899360,1cyclictest0-21swapper/312:14:593
1698687993628,6cyclictest1724862-21kworker/u8:0+events_unbound08:45:220
1698698993533,1cyclictest0-21swapper/308:26:003
1698698993530,4cyclictest1719852-21turbostat08:14:593
169869199351,6cyclictest1791140-21apt-get11:55:151
1698698993433,1cyclictest0-21swapper/310:39:553
1698698993432,1cyclictest0-21swapper/309:19:593
1698693993432,1cyclictest0-21swapper/208:25:002
1698693993431,2cyclictest0-21swapper/211:28:002
1698691993430,3cyclictest1709055-21fschecks_count07:40:171
1698691993429,4cyclictest1772654-21proc_pri10:55:251
1698691993424,8cyclictest1747941-21apt-get09:40:151
1698687993429,4cyclictest1702334-21kworker/u8:0+flush-179:007:20:210
1698687993423,9cyclictest1765426-21apt-get10:35:160
1698698993331,1cyclictest0-21swapper/312:25:593
1698698993328,4cyclictest1723298-21turbostat08:25:003
1698693993330,2cyclictest0-21swapper/209:43:592
1698693993329,3cyclictest0-21swapper/209:36:002
1698691993329,3cyclictest1710739-21hwlatdetect07:45:171
1698693993230,1cyclictest351rcuc/211:31:592
1698693993230,1cyclictest351rcuc/211:31:592
1698693993229,2cyclictest0-21swapper/208:59:552
1698693993228,2cyclictest0-21swapper/212:33:592
1698698993129,1cyclictest0-21swapper/308:33:583
1698693993129,1cyclictest351rcuc/211:46:592
1698691993127,3cyclictest1719210-21needreboot08:10:201
1698691993122,6cyclictest1698582-21latency_hist07:10:021
169869899300,29cyclictest0-21swapper/312:05:133
169869899300,29cyclictest0-21swapper/308:06:593
169869899300,29cyclictest0-21swapper/308:06:593
1698693993027,2cyclictest0-21swapper/207:31:592
169869399300,2cyclictest0-21swapper/209:25:142
169869399300,2cyclictest0-21swapper/209:25:132
169869399300,2cyclictest0-21swapper/207:10:132
169869399300,28cyclictest0-21swapper/208:35:132
1698691993023,6cyclictest1694564-21kworker/u8:0+events_unbound07:10:261
1698687993026,2cyclictest1698712-21latency_hist07:10:010
1698698992926,2cyclictest0-21swapper/311:20:173
1698698992926,2cyclictest0-21swapper/308:35:143
169869899290,28cyclictest0-21swapper/311:40:143
169869899290,28cyclictest0-21swapper/308:15:163
169869899290,27cyclictest0-21swapper/311:10:153
1698693992927,1cyclictest0-21swapper/212:24:592
1698693992927,1cyclictest0-21swapper/209:10:152
1698693992925,2cyclictest0-21swapper/210:42:582
169869399290,2cyclictest0-21swapper/210:52:592
169869399290,2cyclictest0-21swapper/208:45:142
169869399290,27cyclictest0-21swapper/212:05:142
169869399290,27cyclictest0-21swapper/211:40:142
169869399290,27cyclictest0-21swapper/211:20:162
169869399290,27cyclictest0-21swapper/211:10:152
169869399290,27cyclictest0-21swapper/210:30:142
169869399290,27cyclictest0-21swapper/210:10:142
169869399290,27cyclictest0-21swapper/208:00:022
169869399290,27cyclictest0-21swapper/207:50:132
169869399290,27cyclictest0-21swapper/207:45:142
169869399290,21cyclictest0-21swapper/210:15:142
1698687992925,3cyclictest1736001-21kworker/u8:0+flush-179:009:25:270
1698687992925,3cyclictest1736001-21kworker/u8:0+flush-179:009:25:270
1698698992826,1cyclictest0-21swapper/309:10:143
1698698992826,1cyclictest0-21swapper/308:45:133
1698698992824,3cyclictest0-21swapper/312:33:593
1698698992824,2cyclictest0-21swapper/309:36:003
169869899280,2cyclictest0-21swapper/310:15:133
169869899280,27cyclictest0-21swapper/310:30:153
169869899280,27cyclictest0-21swapper/307:45:143
169869899280,27cyclictest0-21swapper/307:20:153
169869899280,26cyclictest0-21swapper/308:55:163
169869899280,26cyclictest0-21swapper/308:00:033
1698693992825,2cyclictest0-21swapper/211:05:182
1698693992825,2cyclictest0-21swapper/211:05:172
1698693992825,2cyclictest0-21swapper/207:20:152
1698693992824,2cyclictest0-21swapper/211:58:592
1698693992824,2cyclictest0-21swapper/207:44:552
169869399280,2cyclictest0-21swapper/208:15:152
169869399280,27cyclictest0-21swapper/208:05:142
169869399280,27cyclictest0-21swapper/208:05:142
169869399280,26cyclictest0-21swapper/210:29:582
169869399280,25cyclictest0-21swapper/208:54:552
4670791270,4ptp4l441ktimers/309:35:003
1698693992724,2cyclictest0-21swapper/207:26:592
169869399270,25cyclictest0-21swapper/210:04:002
4670791260,4ptp4l441ktimers/312:00:003
4670791260,4ptp4l441ktimers/311:54:563
4670791260,4ptp4l441ktimers/307:45:013
4670791260,3ptp4l61690irq/136-eno1-TxRx-010:25:003
4670791260,3ptp4l441ktimers/310:55:013
4670791260,3ptp4l441ktimers/309:55:013
4670791260,0ptp4l71-21kworker/3:1+events11:45:023
169869899260,25cyclictest0-21swapper/312:15:593
1698693992623,2cyclictest0-21swapper/212:16:592
169869399260,2cyclictest0-21swapper/209:01:592
169869399260,24cyclictest0-21swapper/212:10:032
169869399260,24cyclictest0-21swapper/207:16:592
46722912522,1phc2sys0-21swapper/307:15:133
4670791250,3ptp4l61690irq/136-eno1-TxRx-009:10:003
4670791250,3ptp4l441ktimers/311:39:553
4670791250,3ptp4l441ktimers/310:45:013
1698698992524,1cyclictest0-21swapper/310:29:593
1698698992523,1cyclictest0-21swapper/312:24:593
1698698992523,1cyclictest0-21swapper/310:00:143
4670791240,3ptp4l61690irq/136-eno1-TxRx-010:49:573
4670791240,3ptp4l61690irq/136-eno1-TxRx-009:59:553
4670791240,3ptp4l61690irq/136-eno1-TxRx-008:55:003
4670791240,3ptp4l61690irq/136-eno1-TxRx-007:59:543
4670791240,3ptp4l441ktimers/312:39:563
4670791240,3ptp4l441ktimers/310:09:553
1698698992422,1cyclictest0-21swapper/307:25:083
1698693992420,2cyclictest0-21swapper/207:58:592
1698693992320,2cyclictest0-21swapper/209:09:552
1698693992017,2cyclictest0-21swapper/210:24:592
1698693991917,1cyclictest0-21swapper/211:39:592
1698693991916,2cyclictest0-21swapper/209:30:042
1698693991815,2cyclictest351rcuc/211:54:552
1698693991814,2cyclictest1754046-21turbostat10:00:002
1698693991713,2cyclictest1794088-21turbostat12:05:002
1698693991613,2cyclictest0-21swapper/210:05:142
1698693991513,1cyclictest0-21swapper/209:54:592
1698693991310,2cyclictest351rcuc/212:36:452
16986939940,2cyclictest0-21swapper/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional