You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 00:21
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Fri Nov 21, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31327949910190,8cyclictest3137475-21latency_hist07:25:010
3132798999787,7cyclictest3149205-21latency_hist08:00:011
3132798999591,3cyclictest3132719-21kworker/u8:3+events_unbound07:15:241
3132798999281,8cyclictest271-21systemd-journal09:00:001
3132798999082,3cyclictest3158805-21kworker/u8:3+events_unbound08:45:251
3132794998876,9cyclictest3161946-21latency_hist08:40:020
3132798998777,7cyclictest3187664-21latency_hist10:00:011
3132798998776,8cyclictest3218143-21latency_hist11:35:011
3132794998574,8cyclictest3176263-21latency_hist09:25:000
3132798998479,4cyclictest3207253-21kworker/u8:0+events_unbound11:50:251
3132794998374,7cyclictest3195618-21latency_hist10:25:010
3132798998272,4cyclictest3199083-21kworker/u8:2+events_unbound11:00:331
3132794998274,6cyclictest3227313-21kworker/u8:1+flush-179:012:05:260
3132798998070,4cyclictest3132719-21kworker/u8:3+events_unbound07:30:221
3132794997969,7cyclictest3211775-21latency_hist11:15:010
3132798997873,4cyclictest3168549-21kworker/u8:2+events_unbound09:05:291
3132798997570,4cyclictest3157154-21kworker/u8:0+events_unbound08:25:241
3132798997469,4cyclictest3161970-21kworker/u8:0+events_unbound08:40:241
3132798997465,3cyclictest3154011-21kworker/u8:1+events_unbound08:20:201
3132798997363,7cyclictest3173070-21latency_hist09:15:011
3132798997064,4cyclictest3204881-21kworker/u8:1+events_unbound10:55:161
3132798997061,7cyclictest3168549-21kworker/u8:2+flush-179:009:20:001
3132798997061,3cyclictest3213343-21kworker/u8:1+events_unbound11:20:251
3132798997059,8cyclictest3197459-21latency_hist10:30:001
3132798997059,8cyclictest3197459-21latency_hist10:30:001
3132798996960,4cyclictest3147851-21kworker/u8:2+events_unbound08:05:181
3132798996859,7cyclictest3189252-21latency_hist10:05:011
3132798996857,9cyclictest3227952-21apt-get12:05:141
313280099670,65cyclictest0-21swapper/209:19:592
3132798996763,3cyclictest3132645-21kworker/u8:2+flush-179:007:20:241
3132798996662,3cyclictest3229263-21kworker/u8:0+flush-179:012:19:561
3132798996661,4cyclictest3208597-21kworker/u8:3+flush-179:011:14:551
3132798996661,4cyclictest3144178-21kworker/u8:0+flush-179:007:45:231
3132798996554,8cyclictest3150809-21latency_hist08:05:001
3132798996455,4cyclictest3188397-21kworker/u8:2+events_unbound10:10:131
3132794996458,4cyclictest3171518-21kworker/u8:0+flush-179:009:20:000
3132798996359,3cyclictest3207253-21kworker/u8:0+flush-179:011:05:191
3132798996354,6cyclictest3158805-21kworker/u8:3+flush-179:008:40:011
3132798996258,3cyclictest3144178-21kworker/u8:0+flush-179:007:50:221
3132798996257,4cyclictest3168549-21kworker/u8:2+flush-179:009:25:261
3132798996256,4cyclictest3147851-21kworker/u8:2+flush-179:008:15:281
3132794996254,5cyclictest3158805-21kworker/u8:3+flush-179:009:10:150
3132794996229,23cyclictest3202240-21latency_hist10:45:020
3132798996156,4cyclictest3158805-21kworker/u8:3+flush-179:008:30:271
3132798996156,4cyclictest3135912-21kworker/u8:1+flush-179:007:40:291
3132798996156,4cyclictest3135912-21kworker/u8:1+flush-179:007:40:291
3132798996155,4cyclictest3199083-21kworker/u8:2+flush-179:010:55:001
3132798996056,3cyclictest3188397-21kworker/u8:2+flush-179:010:19:551
3132798996056,3cyclictest3147851-21kworker/u8:2+events_unbound08:10:251
3132798996055,4cyclictest3227313-21kworker/u8:1+flush-179:012:04:551
3132798996055,4cyclictest3168549-21kworker/u8:2+events_unbound09:40:211
3132798996055,4cyclictest3135912-21kworker/u8:1+flush-179:007:35:271
3132794996051,7cyclictest3216554-21latency_hist11:30:010
3132794996049,9cyclictest271-21systemd-journal12:05:000
3132803995957,1cyclictest0-21swapper/312:19:593
3132798995954,4cyclictest3236146-21kworker/u8:2+flush-179:012:34:551
3132803995856,1cyclictest0-21swapper/309:03:003
3132798995854,3cyclictest3203275-21kworker/u8:0+flush-179:010:49:551
3132798995852,5cyclictest3174681-21kworker/u8:1+events_unbound09:30:191
3132798995851,4cyclictest3186058-21latency_hist09:55:011
3132798995753,3cyclictest3132719-21kworker/u8:3+flush-179:007:10:271
3132794995746,8cyclictest3234064-21latency_hist12:25:010
3132798995651,4cyclictest3174681-21kworker/u8:1+events_unbound10:31:211
3132798995647,3cyclictest3229263-21kworker/u8:0+events_unbound12:10:251
3132798995551,3cyclictest3216590-21kworker/u8:3+events_unbound11:35:181
3132798995450,3cyclictest3199083-21kworker/u8:2+events_unbound11:15:181
3132798995450,3cyclictest3199083-21kworker/u8:2+events_unbound11:15:181
3132794995436,16cyclictest3166961-21apt-get08:55:130
3132798995348,4cyclictest3234043-21kworker/u8:1+flush-179:012:24:551
3132798995348,4cyclictest3229263-21kworker/u8:0+events_unbound12:29:561
3132798995348,4cyclictest3168549-21kworker/u8:2+events_unbound09:04:191
3132798995347,5cyclictest3224483-21kworker/u8:2+events_unbound11:59:541
3132794995342,8cyclictest3204881-21kworker/u8:1+flush-179:011:00:220
313279499531,2cyclictest3132791-21cyclictest10:02:590
3132798995248,3cyclictest3214954-21kworker/u8:2+events_unbound11:29:211
3132798995247,4cyclictest3183701-21kworker/u8:0+events_unbound09:49:341
3132798995247,4cyclictest3158805-21kworker/u8:3+events_unbound09:24:211
3132794995248,3cyclictest3213343-21kworker/u8:1+flush-179:011:20:260
3132794995248,2cyclictest3135912-21kworker/u8:1+events_unbound07:50:010
3132794995243,6cyclictest3139104-21kworker/u8:0+flush-179:007:35:250
4670791500,13ptp4l441ktimers/307:10:003
3132794995036,9cyclictest3206996-21latency_hist11:00:000
3132794995033,12cyclictest3140646-21latency_hist07:35:010
3132794994942,5cyclictest3212029-21apt-get11:15:120
3132794994942,5cyclictest3212029-21apt-get11:15:110
3132794994842,4cyclictest3207253-21kworker/u8:0+flush-179:011:05:170
3132794994840,6cyclictest3188397-21kworker/u8:2+events_unbound10:10:240
3132794994838,7cyclictest3171483-21latency_hist09:10:010
3132798994743,3cyclictest3216590-21kworker/u8:3+events_unbound11:40:221
3132794994739,5cyclictest3135873-21latency_hist07:20:010
3132794994738,6cyclictest3158805-21kworker/u8:3+flush-179:008:30:240
3132794994642,3cyclictest3144248-21apt-config07:45:010
3132794994642,3cyclictest3144248-21apt-config07:45:010
3132794994633,10cyclictest271-21systemd-journal07:54:590
313279499460,45cyclictest0-21swapper/010:25:040
313279499460,45cyclictest0-21swapper/010:25:030
3132794994542,2cyclictest723-21Xorg09:46:590
3132794994538,4cyclictest3234043-21kworker/u8:1+events_unbound12:25:240
3132794994535,6cyclictest3174681-21kworker/u8:1+flush-179:009:40:200
3132794994532,11cyclictest3236123-21cp12:30:050
3132794994525,17cyclictest0-21swapper/009:40:010
3132794994521,22cyclictest3189894-21fschecks_time10:05:180
3132798994440,3cyclictest3199083-21kworker/u8:2+events_unbound10:39:221
3132798994440,3cyclictest3174681-21kworker/u8:1+events_unbound10:20:271
3132798994440,3cyclictest3158805-21kworker/u8:3+events_unbound08:54:221
3132798994439,4cyclictest3135912-21kworker/u8:1+events_unbound07:26:211
313279899440,42cyclictest723-21Xorg09:39:581
3132798994339,3cyclictest3174681-21kworker/u8:1+events_unbound10:41:201
3132794994323,18cyclictest3138558-21phc2sys07:25:200
313279499430,42cyclictest0-21swapper/011:54:550
3132794994234,5cyclictest3216590-21kworker/u8:3+flush-179:011:35:190
3132794994225,16cyclictest723-21Xorg09:27:580
3132794994216,24cyclictest3217807-21switchtime11:30:230
3132794994214,9cyclictest3192962-21diskstats10:15:150
313279499420,40cyclictest3186069-21turbostat09:59:550
3132803994138,2cyclictest0-21swapper/307:24:593
3132794994131,7cyclictest271-21systemd-journal08:45:000
3132794994130,9cyclictest3196537-21kworker/u8:3+flush-179:010:30:180
3132794994119,20cyclictest0-21swapper/010:45:180
313280399400,39cyclictest0-21swapper/308:07:003
3132794994034,3cyclictest3224483-21kworker/u8:2+flush-179:011:55:130
3132794994030,7cyclictest3154011-21kworker/u8:1+flush-179:008:20:000
3132794994018,19cyclictest0-21swapper/012:35:150
3132803993933,4cyclictest3168300-21turbostat09:00:013
3132794993932,4cyclictest3222955-21systemd-run11:50:000
3132794993932,4cyclictest3222955-21systemd-run11:50:000
3132794993930,7cyclictest3190869-21kworker/u8:0+flush-179:010:35:240
3132794993913,5cyclictest3166048-21cat08:50:200
313279499390,38cyclictest3230877-21turbostat12:19:560
3132803993832,4cyclictest3150818-21turbostat08:05:013
3132800993836,1cyclictest0-21swapper/208:52:582
313280099380,35cyclictest0-21swapper/210:44:552
3132794993829,7cyclictest3184570-21cat09:50:020
313280399370,35cyclictest0-21swapper/308:10:583
313280399370,1cyclictest0-21swapper/309:17:003
3132800993734,2cyclictest0-21swapper/212:35:592
3132800993734,2cyclictest0-21swapper/208:16:582
3132800993734,2cyclictest0-21swapper/208:12:592
313280099370,35cyclictest0-21swapper/210:20:592
313280099370,35cyclictest0-21swapper/210:17:002
313280099370,35cyclictest0-21swapper/209:38:592
313280099370,35cyclictest0-21swapper/207:47:592
313280099370,2cyclictest0-21swapper/207:45:012
313280099370,2cyclictest0-21swapper/207:45:002
3132798993732,4cyclictest3236146-21kworker/u8:2+events_unbound12:36:211
3132798993732,4cyclictest3207253-21kworker/u8:0+events_unbound11:47:221
3132798993732,4cyclictest3207253-21kworker/u8:0+events_unbound11:47:221
3132794993725,9cyclictest3229538-21apt-get12:10:120
3132794993718,18cyclictest0-21swapper/008:25:240
3132803993634,1cyclictest0-21swapper/307:50:593
3132803993634,1cyclictest0-21swapper/307:38:593
3132803993633,2cyclictest0-21swapper/310:55:593
313280399360,35cyclictest0-21swapper/310:19:593
313280399360,35cyclictest0-21swapper/308:43:593
313280399360,35cyclictest0-21swapper/308:19:003
3132800993634,1cyclictest0-21swapper/209:29:592
3132800993634,1cyclictest0-21swapper/207:30:592
3132800993633,2cyclictest0-21swapper/210:27:592
3132800993633,2cyclictest0-21swapper/210:27:592
3132800993633,2cyclictest0-21swapper/209:34:552
3132800993633,2cyclictest0-21swapper/208:27:592
3132800993633,2cyclictest0-21swapper/208:22:002
313280099360,1cyclictest0-21swapper/212:24:592
313280099360,1cyclictest0-21swapper/211:07:592
313280099360,1cyclictest0-21swapper/210:49:592
313280099360,1cyclictest0-21swapper/209:14:552
3132794993629,5cyclictest3132719-21kworker/u8:3+events_unbound07:10:220
3132794993614,2cyclictest0-21swapper/008:24:180
3132794993613,5cyclictest0-21swapper/008:00:170
3132803993533,1cyclictest0-21swapper/311:43:593
3132803993533,1cyclictest0-21swapper/310:44:593
3132800993533,1cyclictest0-21swapper/212:14:592
3132800993532,2cyclictest0-21swapper/212:01:592
3132800993532,2cyclictest0-21swapper/211:44:002
3132800993532,2cyclictest0-21swapper/209:51:582
3132800993532,2cyclictest0-21swapper/207:55:582
313280099350,1cyclictest0-21swapper/211:35:002
3132803993432,1cyclictest0-21swapper/310:54:593
3132803993431,2cyclictest0-21swapper/310:08:583
3132803993428,4cyclictest3135893-21turbostat07:20:013
3132800993431,2cyclictest0-21swapper/210:54:002
313279899343,29cyclictest3189507-21apt-get10:05:161
313279499342,7cyclictest723-21Xorg08:47:590
3132800993331,1cyclictest0-21swapper/210:05:002
3132800993330,2cyclictest0-21swapper/207:16:592
313279499333,6cyclictest723-21Xorg08:11:580
3132794993326,5cyclictest3147852-21kworker/u8:3+flush-179:007:55:210
313279499332,3cyclictest0-21swapper/009:00:130
3132794993311,5cyclictest3221400-21cat11:45:010
3132803993230,1cyclictest0-21swapper/309:09:583
3132803993229,2cyclictest0-21swapper/310:10:593
3132800993229,2cyclictest0-21swapper/207:12:592
313280399310,29cyclictest0-21swapper/312:05:163
3132800993128,2cyclictest0-21swapper/211:26:592
3132794993126,4cyclictest3205412-21/usr/sbin/munin10:55:010
3132794993125,4cyclictest3152519-21ls08:10:010
3132803993028,1cyclictest0-21swapper/310:27:003
3132803993028,1cyclictest0-21swapper/310:26:593
3132803993028,1cyclictest0-21swapper/308:29:553
3132803993028,1cyclictest0-21swapper/307:40:593
3132803993028,1cyclictest0-21swapper/307:40:593
3132803993028,1cyclictest0-21swapper/307:25:593
313280399300,28cyclictest0-21swapper/311:45:153
313280399300,28cyclictest0-21swapper/311:45:153
313280399300,28cyclictest0-21swapper/309:40:143
313280399300,28cyclictest0-21swapper/307:45:033
3132800993027,2cyclictest0-21swapper/212:25:592
3132800993027,2cyclictest0-21swapper/211:15:122
3132800993027,2cyclictest0-21swapper/211:15:122
313280099300,29cyclictest0-21swapper/212:30:592
313280099300,29cyclictest0-21swapper/210:35:132
4670791290,7ptp4l441ktimers/312:05:003
4670791290,7ptp4l441ktimers/309:35:013
3132803992927,1cyclictest0-21swapper/311:27:593
3132803992927,1cyclictest0-21swapper/311:20:153
3132803992927,1cyclictest0-21swapper/311:08:593
3132803992927,1cyclictest0-21swapper/308:54:543
313280399290,2cyclictest0-21swapper/309:10:123
313280399290,28cyclictest3186069-21turbostat09:55:013
313280399290,27cyclictest0-21swapper/310:30:153
3132800992926,2cyclictest0-21swapper/209:59:002
3132800992926,2cyclictest0-21swapper/209:40:152
313280099290,27cyclictest0-21swapper/212:15:142
313280099290,27cyclictest0-21swapper/207:35:022
4670791280,6ptp4l441ktimers/311:05:013
3132803992826,1cyclictest0-21swapper/312:10:143
3132803992826,1cyclictest0-21swapper/311:10:123
3132803992826,1cyclictest0-21swapper/310:45:143
3132803992825,2cyclictest0-21swapper/308:45:143
313280399280,26cyclictest0-21swapper/312:30:073
313280399280,26cyclictest0-21swapper/310:00:123
3132800992825,2cyclictest0-21swapper/211:03:592
3132800992825,2cyclictest0-21swapper/210:30:162
3132800992825,2cyclictest0-21swapper/208:45:152
3132800992825,2cyclictest0-21swapper/208:30:592
313280099280,26cyclictest0-21swapper/212:05:152
313280099280,26cyclictest0-21swapper/211:45:142
313280099280,26cyclictest0-21swapper/211:45:142
4670791270,7ptp4l61690irq/136-eno1-TxRx-010:40:003
4670791270,6ptp4l61690irq/136-eno1-TxRx-010:25:003
4670791270,6ptp4l441ktimers/309:25:003
4670791270,6ptp4l441ktimers/308:35:003
4670791270,5ptp4l441ktimers/312:25:003
3132803992725,1cyclictest0-21swapper/309:26:583
313280399270,1cyclictest0-21swapper/311:15:133
313280399270,1cyclictest0-21swapper/311:15:123
3132800992725,1cyclictest0-21swapper/210:59:592
3132800992724,2cyclictest0-21swapper/210:05:592
3132800992724,2cyclictest0-21swapper/208:01:592
313280099270,26cyclictest0-21swapper/210:13:592
313280099270,25cyclictest0-21swapper/207:26:582
313280099270,19cyclictest0-21swapper/209:48:592
46722912624,1phc2sys0-21swapper/311:55:113
4670791260,6ptp4l441ktimers/311:35:003
4670791260,6ptp4l441ktimers/307:35:003
4670791260,5ptp4l441ktimers/310:00:013
3132803992624,1cyclictest0-21swapper/312:35:123
3132803992624,1cyclictest0-21swapper/311:50:133
3132803992624,1cyclictest0-21swapper/308:36:003
3132803992623,2cyclictest0-21swapper/312:27:583
3132803992623,2cyclictest0-21swapper/309:39:593
3132803992623,2cyclictest0-21swapper/307:12:593
313280399260,2cyclictest0-21swapper/308:23:593
313280099260,24cyclictest0-21swapper/207:20:472
4670791250,6ptp4l61690irq/136-eno1-TxRx-009:50:003
4670791250,5ptp4l61690irq/136-eno1-TxRx-011:40:013
3132800992523,1cyclictest0-21swapper/209:04:002
3132800992522,2cyclictest0-21swapper/208:37:582
313280099250,24cyclictest0-21swapper/211:35:132
4670791240,5ptp4l441ktimers/307:59:553
3132800992422,1cyclictest0-21swapper/211:56:592
3132794992422,1cyclictest0-21swapper/009:33:000
3132800992320,2cyclictest0-21swapper/211:23:002
3132800992320,2cyclictest0-21swapper/208:08:002
3132800992220,1cyclictest0-21swapper/211:14:552
313279899203,15cyclictest3132664-21latency_hist07:10:011
3132800991816,1cyclictest0-21swapper/209:23:592
3132794991814,3cyclictest0-21swapper/007:10:020
3132800991714,2cyclictest0-21swapper/211:52:592
3132800991412,1cyclictest0-21swapper/207:53:592
3132800991411,2cyclictest0-21swapper/209:07:002
31328009996,2cyclictest0-21swapper/208:41:222
31328009985,2cyclictest0-21swapper/208:56:102
31328009950,4cyclictest0-21swapper/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional