You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 23:57
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot6.osadl.org (updated Wed Dec 03, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
322714199118105,11cyclictest3301292-21latency_hist11:00:011
32271419910095,4cyclictest3306351-21kworker/u8:3+events_unbound11:30:251
32271419910088,4cyclictest3254660-21kworker/u8:3+events_unbound09:15:241
3227141999891,5cyclictest3306351-21kworker/u8:3+events_unbound11:25:231
3227140999786,8cyclictest3240047-21latency_hist07:50:000
3227140999786,8cyclictest3240047-21latency_hist07:50:000
3227141999388,3cyclictest3254660-21kworker/u8:3+events_unbound09:05:231
3227141999287,4cyclictest3287200-21kworker/u8:1+events_unbound10:15:251
3227141999085,4cyclictest3251435-21kworker/u8:1+events_unbound08:30:241
3227140998978,8cyclictest3242129-21/usr/sbin/munin07:55:230
3227140998878,8cyclictest3296762-21/usr/sbin/munin10:45:240
3227141998581,3cyclictest3306351-21kworker/u8:3+events_unbound11:45:231
3227141998480,3cyclictest3254660-21kworker/u8:3+events_unbound08:35:251
3227141998375,3cyclictest3306351-21kworker/u8:3+events_unbound11:40:241
3227141998273,3cyclictest3269530-21kworker/u8:1+events_unbound10:05:261
3227140998275,5cyclictest1-21systemd10:19:590
3227141998171,7cyclictest3330446-21latency_hist12:30:021
3227140998073,5cyclictest3298346-21/usr/sbin/munin10:50:220
3227140997767,7cyclictest3262588-21latency_hist09:00:010
3227141997672,3cyclictest3269530-21kworker/u8:1+events_unbound10:00:251
3227141997672,3cyclictest3269530-21kworker/u8:1+events_unbound09:45:241
3227141997672,3cyclictest3254660-21kworker/u8:3+events_unbound08:45:231
3227141997672,3cyclictest3230251-21kworker/u8:2+events_unbound07:25:431
3227141997671,4cyclictest3230251-21kworker/u8:2+events_unbound07:35:261
3227140997465,6cyclictest3259402-21latency_hist08:50:010
3227141997160,4cyclictest3269530-21kworker/u8:1+events_unbound09:20:201
3227140997161,7cyclictest3231811-21latency_hist07:25:010
3227140997161,7cyclictest3231811-21latency_hist07:25:000
3227141997066,3cyclictest3322474-21kworker/u8:0+events_unbound12:20:261
3227141997061,3cyclictest3230251-21kworker/u8:2+events_unbound07:50:191
3227141996958,8cyclictest3286963-21latency_hist10:15:011
3227141996864,3cyclictest3254660-21kworker/u8:3+events_unbound09:30:251
3227141996863,4cyclictest3287200-21kworker/u8:1+flush-179:010:45:261
3227141996859,6cyclictest3306315-21latency_hist11:15:011
3227140996861,5cyclictest3328328-21kworker/u8:1+flush-179:012:30:000
3227140996861,5cyclictest3233430-21kworker/u8:3+flush-179:008:10:170
3227141996762,4cyclictest3275287-21kworker/u8:0+flush-179:009:40:271
3227141996560,4cyclictest3285365-21kworker/u8:0+events_unbound10:35:241
3227141996560,4cyclictest3270525-21kworker/u8:2+flush-179:009:35:161
3227141996560,4cyclictest3233430-21kworker/u8:3+events_unbound08:10:191
3227140996560,4cyclictest3269530-21kworker/u8:1+flush-179:009:50:240
3227140996556,5cyclictest3306351-21kworker/u8:3+events_unbound11:55:250
3227141996460,3cyclictest3304707-21kworker/u8:2+events_unbound11:15:231
3227141996459,4cyclictest3306351-21kworker/u8:3+flush-179:011:39:561
3227141996457,5cyclictest3270781-21/usr/sbin/munin09:25:231
3227140996456,6cyclictest3331326-21kworker/u8:0+flush-179:012:30:230
3227140996455,7cyclictest3270545-21latency_hist09:25:000
3227141996358,4cyclictest3285365-21kworker/u8:0+flush-179:010:54:541
3227141996358,4cyclictest3233430-21kworker/u8:3+flush-179:008:09:551
3227141996357,4cyclictest3280352-21apt-get09:55:021
3227140996356,4cyclictest3247337-21kworker/u8:0+events_unbound08:25:240
3227141996257,4cyclictest3233430-21kworker/u8:3+flush-179:007:59:551
3227140996255,5cyclictest3285365-21kworker/u8:0+flush-179:010:59:540
3227141996157,3cyclictest3228852-21kworker/u8:0+events_unbound07:15:211
3227141996156,4cyclictest3212799-21kworker/u8:1+flush-179:007:14:551
3227140996154,5cyclictest3272129-21latency_hist09:30:010
3227140996153,6cyclictest3269530-21kworker/u8:1+flush-179:009:35:260
3227141996055,4cyclictest3247337-21kworker/u8:0+flush-179:008:24:551
3227141996054,5cyclictest3285365-21kworker/u8:0+flush-179:011:05:151
3227141995955,3cyclictest3233430-21kworker/u8:3+flush-179:007:40:241
3227141995854,3cyclictest3322533-21kworker/u8:2+events_unbound12:35:271
3227141995854,3cyclictest3228852-21kworker/u8:0+flush-179:007:30:281
3227141995853,4cyclictest3291154-21kworker/u8:2+flush-179:010:29:551
3227141995847,4cyclictest3251435-21kworker/u8:1+events_unbound08:40:131
3227140995849,6cyclictest3278755-21latency_hist09:50:010
3227141995753,4cyclictest3285365-21kworker/u8:0+events_unbound11:02:251
3227141995752,4cyclictest3257801-21kworker/u8:0+flush-179:008:54:551
3227141995751,4cyclictest3322474-21kworker/u8:0+flush-179:012:10:181
3227140995747,8cyclictest3324339-21apt-get12:10:140
3227141995653,3cyclictest3322533-21kworker/u8:2+flush-179:012:09:561
3227141995653,3cyclictest3322533-21kworker/u8:2+flush-179:012:09:551
3227141995652,3cyclictest3276898-21kworker/u8:3+flush-179:009:54:551
3227141995652,3cyclictest3247337-21kworker/u8:0+flush-179:008:19:551
3227141995551,3cyclictest3254660-21kworker/u8:3+events_unbound08:55:291
3227141995551,3cyclictest3243505-21kworker/u8:1+events_unbound08:00:311
3227140995549,5cyclictest3233430-21kworker/u8:3+flush-179:008:05:250
3227140995547,5cyclictest3251435-21kworker/u8:1+events_unbound08:40:230
3227141995451,3cyclictest3322533-21kworker/u8:2+events_unbound12:15:331
3227141995449,4cyclictest3228852-21kworker/u8:0+events_unbound07:22:241
3227141995449,4cyclictest3228852-21kworker/u8:0+events_unbound07:22:241
3227140995449,3cyclictest1-21systemd11:05:010
3227141995349,3cyclictest3287200-21kworker/u8:1+events_unbound10:30:171
3227141995349,3cyclictest3233430-21kworker/u8:3+events_unbound07:45:201
3227141995349,3cyclictest3233430-21kworker/u8:3+events_unbound07:45:191
3227140995348,4cyclictest3243505-21kworker/u8:1+events_unbound08:00:280
3227140995342,9cyclictest3240303-21apt-get07:50:110
3227147995149,1cyclictest0-21swapper/311:04:593
3227141995146,4cyclictest3285365-21kworker/u8:0+flush-179:010:44:561
3227140995140,8cyclictest3236597-21apt-get07:40:010
3227141995046,3cyclictest3306351-21kworker/u8:3+events_unbound11:53:261
3227140995045,4cyclictest3285365-21kworker/u8:0+flush-179:010:35:170
3227141994945,3cyclictest3254660-21kworker/u8:3+flush-179:009:04:561
3227141994944,4cyclictest3251435-21kworker/u8:1+events_unbound08:27:251
3227140994945,3cyclictest3287200-21kworker/u8:1+flush-179:010:40:260
3227140994942,6cyclictest3306351-21kworker/u8:3+flush-179:011:45:250
3227141994843,4cyclictest3306351-21kworker/u8:3+events_unbound11:21:231
3227141994841,5cyclictest3227004-21latency_hist07:10:011
3227141994840,3cyclictest3331326-21kworker/u8:0+events_unbound12:30:221
3227140994845,2cyclictest3261032-21kworker/u8:1+events_unbound09:00:250
4670791470,13ptp4l441ktimers/307:10:013
3227141994642,3cyclictest3285365-21kworker/u8:0+events_unbound10:24:551
3227141994637,3cyclictest3306351-21kworker/u8:3+events_unbound12:00:161
3227140994640,5cyclictest3276898-21kworker/u8:3+events_unbound10:10:240
3227140994636,6cyclictest3306351-21kworker/u8:3+events_unbound11:50:190
3227140994634,9cyclictest3311094-21latency_hist11:30:020
3227140994536,7cyclictest3322533-21kworker/u8:2+flush-179:012:35:240
3227140994536,6cyclictest3322474-21kworker/u8:0+flush-179:012:20:210
3227140994536,6cyclictest3270525-21kworker/u8:2+events_unbound09:40:240
3227140994536,6cyclictest3253277-21/usr/sbin/munin08:30:160
3227147994442,1cyclictest0-21swapper/311:32:583
3227141994440,3cyclictest3254660-21kworker/u8:3+flush-179:009:10:011
3227140994441,2cyclictest0-21swapper/008:52:590
3227140994439,3cyclictest3228852-21kworker/u8:0+events_unbound07:25:190
3227140994435,6cyclictest3304707-21kworker/u8:2+flush-179:011:15:240
3227140994338,4cyclictest3306351-21kworker/u8:3+events_unbound11:40:170
3227140994336,5cyclictest3233430-21kworker/u8:3+events_unbound08:15:230
3227140994334,7cyclictest3309746-21kworker/u8:0+flush-179:011:30:150
3227147994240,1cyclictest0-21swapper/310:41:003
322714099420,40cyclictest3236603-21turbostat07:44:550
322714799410,39cyclictest3299701-21turbostat10:55:003
322714799410,39cyclictest3231816-21turbostat07:25:003
322714799410,39cyclictest3231816-21turbostat07:24:593
3227146994138,2cyclictest0-21swapper/208:42:592
3227140994136,4cyclictest3287200-21kworker/u8:1+flush-179:010:30:230
3227140994135,5cyclictest3285365-21kworker/u8:0+events_unbound11:05:250
3227140994131,8cyclictest3264215-21kworker/u8:2+events_unbound09:10:170
322714799400,39cyclictest3245102-21turbostat08:05:003
322714799400,38cyclictest3290168-21turbostat10:25:003
3227141994035,4cyclictest3306351-21kworker/u8:3+events_unbound11:57:261
322714099401,2cyclictest3230238-21sendmail07:20:000
322714699390,36cyclictest0-21swapper/207:24:552
322714699390,36cyclictest0-21swapper/207:24:552
3227140993932,5cyclictest3251435-21kworker/u8:1+events_unbound08:35:190
3227140993932,5cyclictest3234982-21latency_hist07:35:010
322714099390,38cyclictest3267356-21turbostat09:19:550
3227147993833,4cyclictest3265775-21turbostat09:10:003
3227147993832,4cyclictest3228632-21turbostat07:15:003
322714799380,35cyclictest0-21swapper/311:46:583
3227146993834,2cyclictest0-21swapper/211:17:582
322714699380,36cyclictest0-21swapper/209:26:582
322714699380,36cyclictest0-21swapper/209:10:592
322714699380,36cyclictest0-21swapper/208:15:592
322714699380,36cyclictest0-21swapper/208:03:582
3227140993835,2cyclictest723-21Xorg10:01:590
3227140993834,3cyclictest3306351-21kworker/u8:3+flush-179:011:20:270
3227140993832,5cyclictest3309746-21kworker/u8:0+flush-179:011:35:210
3227147993735,1cyclictest0-21swapper/311:35:593
3227147993734,2cyclictest0-21swapper/307:58:593
322714799370,36cyclictest0-21swapper/307:27:593
3227146993734,2cyclictest0-21swapper/209:21:592
3227146993734,2cyclictest0-21swapper/207:49:552
3227146993734,2cyclictest0-21swapper/207:49:552
322714699370,29cyclictest0-21swapper/207:31:592
3227140993734,2cyclictest723-21Xorg09:06:590
3227140993734,2cyclictest3306354-21systemd-run11:14:590
3227147993634,1cyclictest0-21swapper/312:19:593
3227147993634,1cyclictest0-21swapper/311:29:003
322714799360,29cyclictest0-21swapper/308:44:593
322714799360,1cyclictest0-21swapper/312:11:003
3227146993633,2cyclictest0-21swapper/209:03:592
3227146993632,2cyclictest0-21swapper/211:29:592
322714699360,1cyclictest0-21swapper/212:03:592
322714699360,1cyclictest0-21swapper/210:08:582
3227147993533,1cyclictest0-21swapper/307:18:593
3227146993431,2cyclictest0-21swapper/209:45:592
3227140993431,2cyclictest3306351-21kworker/u8:3+flush-179:012:00:210
3227140993429,3cyclictest3282247-21systemd10:00:000
3227147993331,1cyclictest0-21swapper/311:56:593
3227147993331,1cyclictest0-21swapper/311:23:003
3227147993331,1cyclictest0-21swapper/309:39:553
3227147993331,1cyclictest0-21swapper/309:31:593
3227146993330,2cyclictest0-21swapper/207:42:002
3227146993329,2cyclictest0-21swapper/208:26:592
3227140993329,2cyclictest3269530-21kworker/u8:1+events_unbound10:05:160
3227147993230,1cyclictest0-21swapper/308:48:593
3227147993226,5cyclictest3254636-21turbostat08:35:003
3227146993230,1cyclictest0-21swapper/212:14:582
3227146993230,1cyclictest0-21swapper/208:59:592
3227146993228,2cyclictest0-21swapper/209:37:592
322714699320,2cyclictest0-21swapper/207:26:002
3227140993228,3cyclictest3224050-21kworker/u8:3+flush-179:007:10:250
3227140993225,5cyclictest3322474-21kworker/u8:0+events_unbound12:05:180
3227140993225,5cyclictest3322474-21kworker/u8:0+events_unbound12:05:180
3227147993129,1cyclictest0-21swapper/308:05:593
3227147993127,2cyclictest0-21swapper/308:26:003
3227146993128,2cyclictest0-21swapper/209:44:002
3227146993127,2cyclictest0-21swapper/211:04:552
3227140993120,4cyclictest3287200-21kworker/u8:1+flush-179:010:25:260
322714799300,29cyclictest0-21swapper/311:40:153
3227146993028,1cyclictest0-21swapper/207:39:592
3227146993027,2cyclictest0-21swapper/207:18:592
3227146993026,2cyclictest0-21swapper/212:25:152
3227146993026,2cyclictest0-21swapper/209:18:592
322714699300,29cyclictest0-21swapper/209:05:132
322714699300,28cyclictest0-21swapper/210:41:592
3227140993025,4cyclictest3290170-21dump-pmu-power10:25:000
3227147992927,1cyclictest0-21swapper/311:10:123
3227147992927,1cyclictest0-21swapper/310:10:133
3227147992927,1cyclictest0-21swapper/309:46:593
3227147992926,2cyclictest0-21swapper/311:15:153
3227147992926,2cyclictest0-21swapper/308:10:133
3227146992926,2cyclictest0-21swapper/208:35:132
322714699290,2cyclictest0-21swapper/208:30:142
322714699290,27cyclictest0-21swapper/212:05:182
322714699290,27cyclictest0-21swapper/212:05:172
322714699290,27cyclictest0-21swapper/211:45:032
322714699290,27cyclictest0-21swapper/211:35:122
322714699290,27cyclictest0-21swapper/211:20:152
322714699290,27cyclictest0-21swapper/210:25:122
322714699290,27cyclictest0-21swapper/210:00:142
3227140992922,5cyclictest3326758-21phc2sys12:15:230
3227140992920,7cyclictest271-21systemd-journal09:34:230
4670791280,7ptp4l441ktimers/309:30:013
4670791280,6ptp4l441ktimers/312:25:003
3227147992826,1cyclictest0-21swapper/310:55:123
3227147992825,2cyclictest0-21swapper/312:34:003
3227147992825,2cyclictest0-21swapper/309:55:023
3227147992825,2cyclictest0-21swapper/308:15:133
3227147992825,2cyclictest0-21swapper/307:50:123
322714799280,27cyclictest0-21swapper/311:50:123
322714799280,27cyclictest0-21swapper/310:45:143
322714799280,26cyclictest0-21swapper/310:15:113
3227146992826,1cyclictest0-21swapper/208:09:592
3227146992825,2cyclictest0-21swapper/212:20:142
3227146992825,2cyclictest0-21swapper/211:30:142
3227146992825,2cyclictest0-21swapper/207:55:142
322714699280,2cyclictest0-21swapper/208:14:552
322714699280,26cyclictest0-21swapper/210:16:592
4670791270,7ptp4l441ktimers/308:25:003
322714799270,1cyclictest0-21swapper/307:38:003
3227146992723,2cyclictest0-21swapper/210:20:582
322714699270,1cyclictest0-21swapper/212:30:132
322714699270,1cyclictest0-21swapper/211:13:002
4670791260,6ptp4l61690irq/136-eno1-TxRx-011:10:013
4670791260,6ptp4l441ktimers/309:25:003
4670791260,5ptp4l441ktimers/312:10:003
4670791260,5ptp4l441ktimers/312:10:003
3227147992624,1cyclictest0-21swapper/309:04:553
3227147992623,2cyclictest0-21swapper/307:30:143
322714799260,25cyclictest0-21swapper/312:28:583
322714799260,25cyclictest0-21swapper/310:30:003
3227146992623,2cyclictest0-21swapper/212:15:592
3227146992623,2cyclictest0-21swapper/210:54:002
322714699260,24cyclictest0-21swapper/212:35:152
322714699260,24cyclictest0-21swapper/207:14:462
4670791250,6ptp4l441ktimers/308:59:563
4670791250,5ptp4l441ktimers/309:55:003
3227147992523,1cyclictest0-21swapper/312:00:133
322714799250,24cyclictest0-21swapper/312:39:593
322714799250,24cyclictest0-21swapper/310:06:593
322714799250,24cyclictest0-21swapper/310:04:583
322714799250,24cyclictest0-21swapper/309:40:133
3227146992523,1cyclictest0-21swapper/211:06:592
3227146992522,2cyclictest0-21swapper/210:31:592
3227146992522,2cyclictest0-21swapper/209:59:552
3227140992520,3cyclictest0-21swapper/008:20:160
4670791240,5ptp4l61690irq/136-eno1-TxRx-009:14:553
4670791240,5ptp4l441ktimers/310:40:003
4670791240,5ptp4l441ktimers/308:39:553
4670791240,5ptp4l441ktimers/307:44:553
3227146992421,2cyclictest0-21swapper/211:56:592
3227146992421,2cyclictest0-21swapper/210:38:592
4670791230,4ptp4l61690irq/136-eno1-TxRx-009:19:563
4670791230,4ptp4l441ktimers/310:34:563
4670791230,4ptp4l441ktimers/308:54:563
3227147992321,1cyclictest0-21swapper/307:46:593
3227147992321,1cyclictest0-21swapper/307:46:593
3227146992321,1cyclictest0-21swapper/211:40:142
3227146992320,2cyclictest0-21swapper/208:48:592
3227146992220,1cyclictest0-21swapper/208:55:002
3227146992118,2cyclictest0-21swapper/211:54:552
3227146992118,2cyclictest0-21swapper/209:54:562
3227146992118,2cyclictest0-21swapper/209:31:592
3227146992117,2cyclictest0-21swapper/210:45:592
3227146992017,2cyclictest0-21swapper/207:50:122
3227146991917,1cyclictest0-21swapper/208:23:592
3227146991916,2cyclictest0-21swapper/211:00:012
3227146991715,1cyclictest0-21swapper/210:15:012
322714099162,11cyclictest3227136-21cyclictest07:10:010
32271469930,2cyclictest0-21swapper/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional