You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 21:13
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6.osadl.org (updated Tue Jul 15, 2025 12:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
318340799125114,8cyclictest3196407-21apt-get07:50:121
318340799112100,9cyclictest3246008-21apt-get10:23:231
318340599110101,7cyclictest502-21rs:main0
31834079910091,6cyclictest3221904-21kworker/u8:2+events_unbound09:10:231
3183407999993,4cyclictest3186544-21kworker/u8:1+flush-179:007:20:251
3183407999789,6cyclictest3262214-21latency_hist11:15:011
3183407999689,5cyclictest3228226-21kworker/u8:1+flush-179:009:30:141
3183405999685,8cyclictest3225053-21latency_hist09:20:010
3183407999590,4cyclictest3247679-21kworker/u8:2+events_unbound10:30:341
3183407999586,6cyclictest3218719-21kworker/u8:3+flush-179:009:15:151
3183405999380,10cyclictest3272550-21apt-get11:45:100
3183407999281,9cyclictest3267751-21kworker/u8:3+events_unbound11:35:001
3183407999184,5cyclictest3233253-21kworker/u8:0+events_unbound10:55:131
3183407999184,5cyclictest3199560-21kworker/u8:2+flush-179:008:55:261
3183407999084,4cyclictest3197181-21kworker/u8:0+events_unbound08:05:241
3183407999080,8cyclictest3288180-21kworker/u8:4+flush-179:012:35:001
3183407999080,7cyclictest3242856-21kworker/u8:1+flush-179:010:25:201
3183405999079,9cyclictest3246057-21latency_hist10:25:010
3183407998982,5cyclictest3267499-21kworker/u8:2+flush-179:011:55:161
3183407998982,5cyclictest3239675-21kworker/u8:2+flush-179:010:10:011
3183407998981,5cyclictest3201236-21kworker/u8:1+events_unbound08:15:231
3183407998975,6cyclictest3189743-21kworker/u8:2+events_unbound07:50:021
3183407998975,6cyclictest3189743-21kworker/u8:2+events_unbound07:50:011
3183405998978,9cyclictest3238039-21latency_hist10:00:010
3183407998880,7cyclictest3206208-21kworker/u8:3+events_unbound08:40:011
3183407998880,5cyclictest3257233-21kworker/u8:3+events_unbound11:05:181
3183407998880,5cyclictest3257233-21kworker/u8:3+events_unbound11:05:171
3183407998879,7cyclictest3285058-21kworker/u8:0+flush-179:012:40:001
3183405998876,9cyclictest3105085-21systemd-journal11:45:000
3183407998678,5cyclictest3228226-21kworker/u8:1+flush-179:009:39:591
3183407998579,4cyclictest3277307-21kworker/u8:0+flush-179:012:05:201
3183407998579,3cyclictest3233253-21kworker/u8:0+flush-179:010:45:161
3183407998572,6cyclictest3228272-21kworker/u8:2+events_unbound09:50:141
3183407998475,6cyclictest3234644-21kworker/u8:3+flush-179:010:10:181
3183407998473,8cyclictest3251074-21/usr/sbin/munin10:40:251
3183407998473,8cyclictest3251074-21/usr/sbin/munin10:40:251
3183407998378,4cyclictest3199560-21kworker/u8:2+flush-179:008:00:011
3183407998377,5cyclictest3213955-21kworker/u8:1+flush-179:008:50:251
3183405998378,4cyclictest3260867-21kworker/u8:0+events_unbound11:10:230
3183405998375,6cyclictest3260867-21kworker/u8:0+events_unbound11:35:240
3183407998276,4cyclictest3267499-21kworker/u8:2+events_unbound11:40:151
3183407998275,4cyclictest3197181-21kworker/u8:0+events_unbound08:15:011
3183407998274,6cyclictest3260867-21kworker/u8:0+flush-179:011:19:591
3183407998273,7cyclictest3193219-21/usr/sbin/munin07:40:231
3183407998270,8cyclictest3281864-21kworker/u8:3+flush-179:012:25:221
3183407998174,5cyclictest3206208-21kworker/u8:3+flush-179:008:45:251
3183407998066,5cyclictest3278694-21kworker/u8:1+flush-179:012:15:271
3183407998066,5cyclictest3278694-21kworker/u8:1+flush-179:012:15:261
3183405998075,4cyclictest3239675-21kworker/u8:2+events_unbound10:09:340
3183405998075,3cyclictest3218719-21kworker/u8:3+events_unbound09:30:190
3183411997950,1cyclictest0-21swapper/308:38:593
3183407997865,7cyclictest3197181-21kworker/u8:0+events_unbound08:25:241
3183405997873,4cyclictest3178683-21kworker/u8:3+events_unbound07:32:360
3183405997772,4cyclictest3267499-21kworker/u8:2+events_unbound11:34:360
3183407997672,3cyclictest3178683-21kworker/u8:3+events_unbound07:35:141
3183405997672,3cyclictest3242856-21kworker/u8:1+events_unbound10:44:390
3183405997672,3cyclictest3242856-21kworker/u8:1+events_unbound10:44:380
3183405997672,3cyclictest3197181-21kworker/u8:0+events_unbound08:04:040
3183405997668,3cyclictest3226638-21kworker/u8:0+events_unbound09:28:350
3183407997570,4cyclictest3206208-21kworker/u8:3+flush-179:008:25:001
3183407997469,3cyclictest3226638-21kworker/u8:0+flush-179:009:25:161
3183407997468,4cyclictest3270667-21kworker/u8:1+events_unbound11:40:001
3183407997468,4cyclictest3213955-21kworker/u8:1+flush-179:009:20:191
3183405997470,3cyclictest3272275-21kworker/u8:0+events_unbound11:51:370
3183405997464,8cyclictest3277308-21/usr/sbin/munin12:00:230
3183405997463,8cyclictest3247643-21latency_hist10:30:010
3183407997364,6cyclictest3189700-21latency_hist07:30:001
3183405997369,3cyclictest3242856-21kworker/u8:1+events_unbound10:36:370
3183405997368,4cyclictest3267499-21kworker/u8:2+events_unbound11:59:350
3183405997364,7cyclictest3204600-21/usr/sbin/munin08:15:230
3183407997263,7cyclictest3219240-21diskstats09:00:181
3183405997268,3cyclictest3197181-21kworker/u8:0+events_unbound08:25:360
3183405997267,4cyclictest3247679-21kworker/u8:2+events_unbound10:30:290
3183405997267,4cyclictest3233253-21kworker/u8:0+events_unbound10:16:360
3183405997267,4cyclictest3218719-21kworker/u8:3+events_unbound09:07:360
3183405997262,8cyclictest3105085-21systemd-journal07:50:020
3183405997262,8cyclictest3105085-21systemd-journal07:50:010
3183407997165,5cyclictest3267499-21kworker/u8:2+events_unbound12:20:251
3183405997167,3cyclictest3265713-21gpgv11:25:000
3183405997167,3cyclictest3255202-21kworker/u8:2+events_unbound10:54:390
3183405997167,3cyclictest3242856-21kworker/u8:1+events_unbound10:48:350
3183405997166,4cyclictest3267499-21kworker/u8:2+events_unbound12:12:360
3183405997061,7cyclictest3198209-21cp07:55:060
3183407996964,3cyclictest3233253-21kworker/u8:0+events_unbound10:15:181
3183407996960,7cyclictest3178683-21kworker/u8:3+events_unbound07:30:201
3183405996965,3cyclictest3267499-21kworker/u8:2+events_unbound12:28:350
3183405996965,3cyclictest3255202-21kworker/u8:2+events_unbound11:09:360
3183405996965,3cyclictest3255202-21kworker/u8:2+events_unbound11:09:350
3183405996965,3cyclictest3233253-21kworker/u8:0+events_unbound10:12:350
3183407996862,4cyclictest3234644-21kworker/u8:3+flush-179:010:00:171
3183407996862,4cyclictest3183359-21kworker/u8:2+events_unbound07:20:001
3183405996864,3cyclictest3189743-21kworker/u8:2+events_unbound07:42:350
318340599670,1cyclictest0-21swapper/008:58:590
3183407996658,6cyclictest3178683-21kworker/u8:3+events_unbound07:15:011
318341099650,1cyclictest0-21swapper/207:40:592
3183407996561,3cyclictest3213955-21kworker/u8:1+events_unbound09:05:251
3183407996558,5cyclictest3234644-21kworker/u8:3+flush-179:009:55:161
3183405996561,3cyclictest3215550-21kworker/u8:0+events_unbound09:01:350
3183407996454,8cyclictest3213921-21latency_hist08:45:011
3183405996460,3cyclictest3199560-21kworker/u8:2+events_unbound08:42:360
3183411996360,1cyclictest0-21swapper/310:55:593
3183407996355,6cyclictest3260867-21kworker/u8:0+events_unbound11:25:161
3183407996354,7cyclictest3278659-21latency_hist12:05:011
3183405996354,6cyclictest3215506-21latency_hist08:50:010
3183405996352,9cyclictest3105085-21systemd-journal08:40:000
3183407996256,5cyclictest3262249-21kworker/u8:1+events_unbound11:20:291
3183407996255,6cyclictest3199560-21kworker/u8:2+flush-179:008:30:171
3183407996154,5cyclictest3272275-21kworker/u8:0+flush-179:011:45:191
3183405996157,3cyclictest3221904-21kworker/u8:2+events_unbound09:20:020
3183411996056,2cyclictest0-21swapper/307:39:003
3183407996054,5cyclictest3247679-21kworker/u8:2+events_unbound10:35:281
3183407996053,5cyclictest3228272-21kworker/u8:2+events_unbound09:45:261
3183410995855,2cyclictest0-21swapper/208:58:592
3183407995852,4cyclictest3228226-21kworker/u8:1+flush-179:009:40:151
3183407995851,5cyclictest3267499-21kworker/u8:2+events_unbound12:10:231
3183407995850,6cyclictest3183304-21latency_hist07:10:011
3183407995850,6cyclictest3183304-21latency_hist07:10:011
3183405995854,3cyclictest3278694-21kworker/u8:1+events_unbound12:18:380
3183405995854,3cyclictest3278694-21kworker/u8:1+events_unbound12:18:370
3183405995854,3cyclictest3197181-21kworker/u8:0+events_unbound08:13:360
3183405995853,4cyclictest3285058-21kworker/u8:0+events_unbound12:37:370
3183410995755,1cyclictest0-21swapper/210:49:592
3183407995751,4cyclictest3257233-21kworker/u8:3+events_unbound11:00:201
3183407995751,4cyclictest3257233-21kworker/u8:3+events_unbound11:00:201
3183405995753,4cyclictest3199560-21kworker/u8:2+events_unbound08:50:020
3183405995748,7cyclictest3234644-21kworker/u8:3+flush-179:009:55:010
3183405995450,3cyclictest3262249-21kworker/u8:1+events_unbound11:16:380
3183405995450,3cyclictest3234644-21kworker/u8:3+events_unbound10:00:010
3183405995450,3cyclictest3186480-21kworker/u8:0+events_unbound07:18:370
3183405995348,4cyclictest3285058-21kworker/u8:0+events_unbound12:33:360
3183405995344,6cyclictest3231417-21latency_hist09:40:010
3183405995146,4cyclictest3255202-21kworker/u8:2+events_unbound11:02:370
3183405995146,4cyclictest3255202-21kworker/u8:2+events_unbound11:02:360
3183405995046,3cyclictest3281864-21kworker/u8:3+events_unbound12:23:350
3183405995041,6cyclictest3197737-21latency_hist07:55:000
3183405994944,4cyclictest3221904-21kworker/u8:2+events_unbound09:11:360
3183405994944,4cyclictest3197181-21kworker/u8:0+events_unbound08:06:350
3183405994937,9cyclictest3105085-21systemd-journal08:30:010
3183405994927,6cyclictest151ktimers/007:25:150
4505291480,13ptp4l61790irq/134-eno1-TxRx-007:09:573
4505291480,13ptp4l61790irq/134-eno1-TxRx-007:09:563
3183407994843,4cyclictest3197181-21kworker/u8:0+events_unbound08:04:361
3183411994641,3cyclictest3283433-21turbostat12:20:013
3183411994641,3cyclictest3283433-21turbostat12:20:003
3183405994638,6cyclictest3105085-21systemd-journal09:40:350
3183407994537,6cyclictest3274134-21apt-get11:50:171
3183411994139,1cyclictest0-21swapper/310:26:593
3183411994138,2cyclictest0-21swapper/311:39:593
3183407994132,4cyclictest3233253-21kworker/u8:0+events_unbound10:50:151
3183411994035,4cyclictest0-21swapper/312:38:003
3183411994034,5cyclictest0-21swapper/309:49:553
3183411994034,4cyclictest3223464-21turbostat09:15:003
318341199400,38cyclictest3278670-21turbostat12:05:003
318341199400,38cyclictest3269106-21turbostat11:35:003
318341199400,38cyclictest3262222-21turbostat11:15:003
318341199400,38cyclictest3217104-21turbostat08:55:003
318341199400,38cyclictest3204381-21turbostat08:15:003
3183405994036,3cyclictest3199560-21kworker/u8:2+events_unbound08:21:000
318341199390,37cyclictest3250837-21turbostat10:40:013
3183410993934,2cyclictest0-21swapper/209:09:562
318341099390,1cyclictest0-21swapper/210:15:012
3183405993935,3cyclictest3183359-21kworker/u8:2+events_unbound07:11:360
3183405993934,4cyclictest3228272-21kworker/u8:2+events_unbound09:49:580
3183411993835,2cyclictest0-21swapper/312:10:593
3183410993836,1cyclictest0-21swapper/212:40:002
3183410993835,2cyclictest0-21swapper/208:16:592
318341099380,36cyclictest0-21swapper/209:28:592
318341099380,36cyclictest0-21swapper/207:35:002
3183405993833,4cyclictest3257155-21kworker/u8:1+events_unbound10:56:340
318341199370,36cyclictest0-21swapper/309:03:593
318341199370,30cyclictest0-21swapper/310:53:593
318341099370,35cyclictest0-21swapper/212:06:592
318341099370,35cyclictest0-21swapper/210:34:592
3183405993732,4cyclictest3267499-21kworker/u8:2+events_unbound12:08:000
3183411993634,1cyclictest0-21swapper/309:23:593
3183411993630,5cyclictest3252427-21turbostat10:45:003
3183411993630,5cyclictest3252427-21turbostat10:45:003
3183411993630,4cyclictest3225058-21turbostat09:20:003
3183411993630,4cyclictest3188124-21turbostat07:24:593
318341199360,35cyclictest0-21swapper/308:49:593
318341199360,1cyclictest0-21swapper/307:14:593
3183410993634,1cyclictest0-21swapper/208:45:002
3183410993633,2cyclictest0-21swapper/211:55:012
3183410993633,2cyclictest0-21swapper/211:30:002
3183410993633,2cyclictest0-21swapper/209:05:012
3183410993633,2cyclictest0-21swapper/207:21:592
318341099360,1cyclictest0-21swapper/211:39:002
3183405993631,4cyclictest3186480-21kworker/u8:0+events_unbound07:21:010
318340599361,5cyclictest3191539-21apt-get07:35:140
318341199350,2cyclictest0-21swapper/311:40:133
3183410993533,1cyclictest0-21swapper/209:54:592
3183410993533,1cyclictest0-21swapper/208:00:002
3183410993532,2cyclictest0-21swapper/211:48:582
3183410993532,2cyclictest0-21swapper/210:09:592
3183410993532,2cyclictest0-21swapper/207:30:002
318341099350,1cyclictest0-21swapper/211:23:592
3183410993432,1cyclictest0-21swapper/207:20:002
3183411993331,1cyclictest0-21swapper/311:54:003
3183411993229,2cyclictest0-21swapper/312:30:583
3183411993125,5cyclictest0-21swapper/308:55:143
3183410993128,2cyclictest0-21swapper/208:12:592
318341099310,27cyclictest0-21swapper/208:38:592
3183411993028,1cyclictest0-21swapper/312:20:593
3183411993025,4cyclictest3231422-21turbostat09:40:003
318341199300,29cyclictest0-21swapper/310:31:593
318341199300,1cyclictest0-21swapper/307:25:153
3183410993028,1cyclictest0-21swapper/208:49:592
3183410993027,2cyclictest0-21swapper/212:25:142
3183410993027,2cyclictest0-21swapper/211:11:592
3183410993027,2cyclictest0-21swapper/210:21:582
3183410993027,2cyclictest0-21swapper/209:35:172
3183410993027,2cyclictest0-21swapper/209:18:582
318341099300,29cyclictest0-21swapper/208:20:152
4505291290,6ptp4l441ktimers/310:05:003
3183411992927,1cyclictest0-21swapper/312:25:143
3183411992924,4cyclictest3265681-21turbostat11:25:003
3183411992924,4cyclictest0-21swapper/307:55:153
318341199290,28cyclictest0-21swapper/311:00:143
318341199290,28cyclictest0-21swapper/311:00:133
318341199290,27cyclictest0-21swapper/309:55:023
318341199290,27cyclictest0-21swapper/307:40:123
3183410992926,2cyclictest0-21swapper/212:20:142
3183410992926,2cyclictest0-21swapper/211:57:002
3183410992926,2cyclictest0-21swapper/211:40:142
3183410992926,2cyclictest0-21swapper/211:30:132
3183410992926,2cyclictest0-21swapper/207:50:142
318341099290,27cyclictest0-21swapper/210:52:582
318341099290,27cyclictest0-21swapper/208:25:172
318341099290,27cyclictest0-21swapper/207:49:552
318341099290,27cyclictest0-21swapper/207:49:552
318341099290,27cyclictest0-21swapper/207:35:132
3183411992826,1cyclictest0-21swapper/310:45:143
3183411992826,1cyclictest0-21swapper/308:04:593
3183411992826,1cyclictest0-21swapper/307:30:143
3183411992822,5cyclictest0-21swapper/312:05:133
318341199280,2cyclictest0-21swapper/310:15:163
318341199280,26cyclictest0-21swapper/311:25:033
318341199280,26cyclictest0-21swapper/307:50:133
3183410992826,1cyclictest0-21swapper/212:30:142
3183410992825,2cyclictest0-21swapper/208:30:142
318341099280,2cyclictest0-21swapper/208:00:142
318341099280,26cyclictest0-21swapper/210:00:152
3183411992725,1cyclictest0-21swapper/310:14:553
3183411992725,1cyclictest0-21swapper/309:25:583
3183411992722,4cyclictest0-21swapper/309:51:593
318341199270,26cyclictest0-21swapper/309:05:143
3183410992724,2cyclictest0-21swapper/212:15:002
3183410992724,2cyclictest0-21swapper/211:03:592
3183410992724,2cyclictest0-21swapper/211:03:582
318341099270,3cyclictest0-21swapper/210:55:592
318341099270,25cyclictest0-21swapper/211:15:592
318341099270,25cyclictest0-21swapper/207:10:592
318341099270,24cyclictest0-21swapper/210:44:562
318341099270,24cyclictest0-21swapper/210:44:552
4505291260,6ptp4l61790irq/134-eno1-TxRx-010:25:003
4505291260,6ptp4l61790irq/134-eno1-TxRx-008:35:013
4505291260,6ptp4l441ktimers/311:59:593
4505291260,6ptp4l441ktimers/308:24:553
4505291260,6ptp4l441ktimers/307:20:023
3183411992623,2cyclictest0-21swapper/308:42:593
3183410992623,2cyclictest0-21swapper/210:35:142
3183410992623,2cyclictest0-21swapper/210:18:002
3183410992623,2cyclictest0-21swapper/209:30:122
3183410992623,2cyclictest0-21swapper/208:08:592
4505291250,6ptp4l441ktimers/311:19:553
4505291250,5ptp4l61790irq/134-eno1-TxRx-011:49:563
4505291250,5ptp4l61790irq/134-eno1-TxRx-011:09:563
4505291250,5ptp4l61790irq/134-eno1-TxRx-011:09:553
4505291250,5ptp4l61790irq/134-eno1-TxRx-009:34:563
4505291250,5ptp4l441ktimers/310:09:563
4505291250,5ptp4l441ktimers/309:44:563
4505291250,5ptp4l441ktimers/308:29:553
3183410992522,2cyclictest0-21swapper/209:14:552
3183410992521,2cyclictest0-21swapper/209:58:592
4505291240,5ptp4l61790irq/134-eno1-TxRx-008:19:563
3183411992422,1cyclictest0-21swapper/308:05:593
3183411992422,1cyclictest0-21swapper/307:46:003
3183411992422,1cyclictest0-21swapper/307:45:593
3183410992422,1cyclictest0-21swapper/209:40:592
201699240,13rtkit-daemon3182291-21cstates07:05:180
201699240,13rtkit-daemon3182291-21cstates07:05:170
3183410992321,1cyclictest0-21swapper/211:07:592
3183410992321,1cyclictest0-21swapper/211:07:592
3183410992321,1cyclictest0-21swapper/209:24:592
3183410992320,2cyclictest0-21swapper/212:15:142
3183410992320,2cyclictest0-21swapper/212:15:142
3183410992220,1cyclictest0-21swapper/210:30:002
3183410992218,2cyclictest0-21swapper/212:03:582
3183410991815,2cyclictest0-21swapper/209:48:592
3183410991411,2cyclictest0-21swapper/208:53:592
31834109940,3cyclictest0-21swapper/207:10:022
31834109940,3cyclictest0-21swapper/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional