You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-18 - 01:04
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot6.osadl.org (updated Fri Oct 17, 2025 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
267981899134124,7cyclictest2767366-21kworker/u8:4+events_unbound11:50:030
267982099119104,12cyclictest2734670-21kworker/u8:0+flush-179:010:15:251
267981899118112,5cyclictest2677622-21kworker/u8:1+events_unbound07:30:240
267982099115104,8cyclictest2747977-21latency_hist10:40:011
267981899112107,4cyclictest2687917-21kworker/u8:0+flush-179:007:55:250
26798209910693,10cyclictest2736812-21/usr/sbin/munin10:05:241
26798189910499,4cyclictest2711913-21kworker/u8:3+events_unbound08:55:230
26798189910486,4cyclictest2677622-21kworker/u8:1+events_unbound08:00:250
26798189910397,4cyclictest2673678-21kworker/u8:3+flush-179:007:15:160
26798189910395,6cyclictest2729712-21kworker/u8:3+events_unbound09:55:020
26798189910395,6cyclictest2729712-21kworker/u8:3+events_unbound09:55:010
26798189910292,4cyclictest2775589-21kworker/u8:1+events_unbound12:10:240
26798189910287,4cyclictest2703897-21kworker/u8:1+events_unbound08:35:200
2679818999893,4cyclictest2738144-21kworker/u8:2+events_unbound11:55:250
2679818999893,4cyclictest2711866-21kworker/u8:1+flush-179:008:50:230
2679818999892,5cyclictest2767366-21kworker/u8:4+events_unbound11:45:000
2679818999691,4cyclictest2731504-21kworker/u8:2+flush-179:009:50:250
2679818999691,4cyclictest2687917-21kworker/u8:0+events_unbound08:15:140
2679818999690,5cyclictest2673678-21kworker/u8:3+events_unbound07:25:220
2679820999585,7cyclictest2738704-21diskstats10:10:151
2679820999585,7cyclictest2731267-21latency_hist09:50:011
2679818999587,6cyclictest2731504-21kworker/u8:2+events_unbound10:00:020
2679818999587,6cyclictest2731504-21kworker/u8:2+events_unbound10:00:020
2679820999484,7cyclictest2728068-21latency_hist09:40:011
2679818999489,4cyclictest2677622-21kworker/u8:1+events_unbound07:20:240
2679820999381,11cyclictest2677622-21kworker/u8:1+flush-179:007:55:001
2679820999286,4cyclictest2721413-21kworker/u8:2+flush-179:009:25:191
2679818999287,4cyclictest2744597-21kworker/u8:0+events_unbound10:45:280
2679818999286,5cyclictest2738144-21kworker/u8:2+events_unbound10:55:000
2679818999286,4cyclictest2734670-21kworker/u8:0+events_unbound10:20:130
2679818999277,12cyclictest601706-21systemd-journal12:30:020
2679818999277,12cyclictest601706-21systemd-journal12:30:010
2679818999175,13cyclictest2723041-21latency_hist09:25:010
2679818999085,4cyclictest2775589-21kworker/u8:1+flush-179:012:35:290
2679818999081,7cyclictest2719846-21latency_hist09:15:010
2679818999081,7cyclictest2719846-21latency_hist09:15:010
2679820998978,8cyclictest2766060-21diskstats11:35:181
2679820998974,11cyclictest601706-21systemd-journal07:20:001
2679820998970,16cyclictest2786502-21latency_hist12:40:011
2679818998979,7cyclictest601706-21systemd-journal10:20:010
2679818998970,14cyclictest2744554-21apt-get10:30:000
2679820998879,7cyclictest2738144-21kworker/u8:2+flush-179:010:34:591
2679818998883,4cyclictest2708661-21kworker/u8:0+flush-179:008:40:250
2679818998883,4cyclictest2687917-21kworker/u8:0+events_unbound08:10:240
2679818998880,6cyclictest2756199-21kworker/u8:1+events_unbound11:20:150
2679818998875,10cyclictest2697795-21apt-get08:05:130
2679820998776,9cyclictest2728346-21apt-get09:40:121
2679818998781,4cyclictest2687917-21kworker/u8:0+flush-179:007:45:150
2679818998677,7cyclictest2738144-21kworker/u8:2+events_unbound12:05:220
2679820998575,8cyclictest2770433-21gpgv11:50:011
2679818998580,4cyclictest2715042-21kworker/u8:0+events_unbound09:05:160
2679820998469,8cyclictest2673678-21kworker/u8:3+flush-179:007:10:171
2679818998478,4cyclictest2756199-21kworker/u8:1+events_unbound11:25:220
2679820998376,5cyclictest2711913-21kworker/u8:3+flush-179:008:59:551
2679820998373,8cyclictest2767385-21apt-get11:40:111
2679820998271,8cyclictest2691158-21latency_hist07:45:001
2679818998278,3cyclictest2738144-21kworker/u8:2+events_unbound11:35:240
2679818998277,4cyclictest2673678-21kworker/u8:3+flush-179:007:11:210
2679818998275,5cyclictest2703897-21kworker/u8:1+events_unbound08:30:130
2679818998274,6cyclictest2677622-21kworker/u8:1+events_unbound07:50:140
2679820998173,5cyclictest2729712-21kworker/u8:3+flush-179:009:55:001
2679818998174,5cyclictest601706-21systemd-journal08:27:240
2679818998172,6cyclictest2739750-21latency_hist10:15:010
2679820998070,7cyclictest2759170-21kworker/u8:3+flush-179:011:15:241
2679818998069,9cyclictest2689831-21apt-get07:40:110
2679820997969,7cyclictest2744597-21kworker/u8:0+flush-179:011:05:241
2679820997967,8cyclictest2767366-21kworker/u8:4+flush-179:011:50:171
2679818997974,4cyclictest2744597-21kworker/u8:0+events_unbound11:00:250
2679818997974,3cyclictest2771866-21kworker/u8:0+flush-179:012:20:200
2679818997970,6cyclictest2744597-21kworker/u8:0+events_unbound11:05:160
2679820997870,5cyclictest2738144-21kworker/u8:2+flush-179:012:05:221
2679820997869,7cyclictest601706-21systemd-journal07:22:221
2679820997866,9cyclictest2784907-21latency_hist12:35:011
2679818997873,4cyclictest2721413-21kworker/u8:2+events_unbound09:25:240
2679818997866,10cyclictest2745055-21apt-get10:30:130
2679818997865,5cyclictest2708661-21kworker/u8:0+events_unbound08:45:120
2679820997768,7cyclictest2687917-21kworker/u8:0+flush-179:007:45:231
2679818997772,4cyclictest2738144-21kworker/u8:2+events_unbound10:55:260
2679818997768,7cyclictest2744597-21kworker/u8:0+events_unbound10:40:240
2679820997669,5cyclictest2756199-21kworker/u8:1+flush-179:011:15:011
2679818997671,3cyclictest2781966-21kworker/u8:2+events_unbound12:30:150
2679818997671,3cyclictest2744597-21kworker/u8:0+events_unbound11:10:270
2679818997666,3cyclictest2728057-21kworker/u8:1+events_unbound09:45:190
2679820997567,6cyclictest2703897-21kworker/u8:1+flush-179:008:35:001
2679820997567,5cyclictest2738144-21kworker/u8:2+flush-179:010:30:011
2679820997564,9cyclictest2694595-21apt-get07:55:121
2679818997569,4cyclictest2775589-21kworker/u8:1+flush-179:012:15:210
2679818997469,4cyclictest2756199-21kworker/u8:1+events_unbound11:30:260
2679818997468,4cyclictest2687917-21kworker/u8:0+flush-179:008:20:150
2679818997367,5cyclictest2738144-21kworker/u8:2+flush-179:010:35:250
2679820997260,8cyclictest601706-21systemd-journal10:45:241
2679820997165,4cyclictest2687917-21kworker/u8:0+flush-179:007:35:161
2679820997064,5cyclictest2673678-21kworker/u8:3+flush-179:007:29:561
2679818997064,4cyclictest2721413-21kworker/u8:2+events_unbound09:35:180
2679820996963,3cyclictest2738144-21kworker/u8:2+flush-179:011:20:221
2679818996961,5cyclictest2711913-21kworker/u8:3+events_unbound09:00:200
2679820996858,7cyclictest2723057-21apt-get09:25:001
267982099680,67cyclictest271rcuc/111:56:591
267982499670,65cyclictest0-21swapper/209:56:592
267982499670,65cyclictest0-21swapper/209:56:592
2679820996761,4cyclictest2771866-21kworker/u8:0+flush-179:012:19:551
2679820996660,4cyclictest2687917-21kworker/u8:0+flush-179:008:25:111
2679820996659,4cyclictest2771866-21kworker/u8:0+events_unbound12:25:021
2679820996655,8cyclictest2753017-21apt-get10:55:151
2679818996661,4cyclictest2759170-21kworker/u8:3+events_unbound11:15:240
2679818996658,5cyclictest2767366-21kworker/u8:4+events_unbound11:45:120
2679826996558,5cyclictest0-21swapper/309:52:593
2679820996559,5cyclictest2703897-21kworker/u8:1+events_unbound08:35:241
2679820996558,5cyclictest2715042-21kworker/u8:0+flush-179:009:15:261
2679818996560,3cyclictest2734670-21kworker/u8:0+events_unbound10:05:180
2679818996559,5cyclictest2738144-21kworker/u8:2+events_unbound12:00:230
2679820996358,4cyclictest2734670-21kworker/u8:0+events_unbound10:00:251
2679820996358,4cyclictest2734670-21kworker/u8:0+events_unbound10:00:251
2679820996353,8cyclictest601706-21systemd-journal10:42:251
2679818996358,4cyclictest2721413-21kworker/u8:2+flush-179:009:40:210
2679820996255,5cyclictest2729712-21kworker/u8:3+flush-179:009:55:151
2679820996255,5cyclictest2729712-21kworker/u8:3+flush-179:009:55:151
2679818996257,4cyclictest2677622-21kworker/u8:1+flush-179:007:38:240
2679820996150,9cyclictest2778529-21latency_hist12:15:001
2679820996148,9cyclictest601706-21systemd-journal09:00:241
2679818996153,6cyclictest2720097-21apt-get09:15:110
2679820996054,4cyclictest2756199-21kworker/u8:1+flush-179:011:25:161
2679820996054,4cyclictest2708661-21kworker/u8:0+flush-179:008:40:171
2679820995953,5cyclictest2715042-21kworker/u8:0+events_unbound09:05:261
2679826995850,6cyclictest2741375-21turbostat10:20:013
2679820995650,5cyclictest2711866-21kworker/u8:1+flush-179:008:54:561
2679820995649,5cyclictest2687917-21kworker/u8:0+flush-179:008:20:001
2679820995552,2cyclictest2775589-21kworker/u8:1+events_unbound12:25:231
2679820995552,2cyclictest2775589-21kworker/u8:1+events_unbound12:25:231
2679820995344,7cyclictest2725146-21apt-get09:30:121
2679818995247,4cyclictest2721413-21kworker/u8:2+flush-179:009:34:560
2679820995043,5cyclictest2738144-21kworker/u8:2+flush-179:010:50:211
2679820995042,6cyclictest601706-21systemd-journal12:00:251
2679820995039,9cyclictest2702578-21apt-get08:20:111
2679820994838,8cyclictest2741618-21apt-get10:20:111
4505291470,12ptp4l61790irq/134-eno1-TxRx-007:09:553
2679826994745,1cyclictest0-21swapper/310:51:583
2679820994741,5cyclictest2738144-21kworker/u8:2+flush-179:011:00:231
2679820994741,5cyclictest2718490-21kworker/u8:1+flush-179:009:10:221
2679820994741,5cyclictest2718490-21kworker/u8:1+flush-179:009:10:221
2679820994739,6cyclictest601706-21systemd-journal11:33:251
2679820994739,6cyclictest2673678-21kworker/u8:3+events_unbound07:30:191
2679820994539,4cyclictest2687917-21kworker/u8:0+flush-179:008:05:171
2679824994340,2cyclictest0-21swapper/209:32:002
2679824994239,2cyclictest0-21swapper/208:55:592
267982499390,36cyclictest0-21swapper/212:06:592
267982499390,36cyclictest0-21swapper/208:33:592
2679826993831,6cyclictest0-21swapper/309:21:593
2679824993835,2cyclictest0-21swapper/209:10:592
2679824993835,2cyclictest0-21swapper/209:10:592
2679824993835,2cyclictest0-21swapper/207:40:592
2679824993834,2cyclictest0-21swapper/211:45:582
2679824993834,2cyclictest0-21swapper/211:43:592
2679824993834,2cyclictest0-21swapper/211:28:002
2679824993834,2cyclictest0-21swapper/211:03:582
2679824993834,2cyclictest0-21swapper/208:09:592
267982499380,36cyclictest0-21swapper/209:17:002
2679826993735,1cyclictest0-21swapper/311:23:003
2679826993734,2cyclictest0-21swapper/310:44:003
267982699370,35cyclictest0-21swapper/310:00:003
267982699370,35cyclictest0-21swapper/309:59:593
2679820993731,5cyclictest2708661-21kworker/u8:0+flush-179:008:45:161
4505291360,3ptp4l441ktimers/309:10:133
4505291360,3ptp4l441ktimers/309:10:133
2679826993629,6cyclictest0-21swapper/308:05:593
267982699360,35cyclictest0-21swapper/308:20:593
267982099362,31cyclictest2699379-21apt-get08:10:141
2679826993533,1cyclictest0-21swapper/311:56:593
2679826993533,1cyclictest0-21swapper/307:58:593
2679826993526,7cyclictest0-21swapper/311:36:003
2679824993532,2cyclictest0-21swapper/209:35:592
267982499350,1cyclictest0-21swapper/210:25:012
2679820993530,3cyclictest2697620-21latency_hist08:05:011
2679826993432,1cyclictest0-21swapper/309:25:593
267982699340,24cyclictest0-21swapper/309:17:003
2679824993431,2cyclictest0-21swapper/212:34:592
2679824993431,2cyclictest0-21swapper/207:29:552
2679826993331,1cyclictest0-21swapper/307:35:003
2679826993330,2cyclictest0-21swapper/308:01:593
267982699330,21cyclictest0-21swapper/307:25:143
2679824993330,2cyclictest0-21swapper/207:17:592
4505291320,6ptp4l441ktimers/312:05:003
2679826993230,1cyclictest0-21swapper/311:49:593
2679826993224,7cyclictest0-21swapper/308:27:593
2679826993224,6cyclictest2765529-21turbostat11:35:003
267982699320,1cyclictest0-21swapper/308:10:133
2679824993230,1cyclictest0-21swapper/209:29:592
2679824993229,2cyclictest0-21swapper/208:26:592
2679826993129,1cyclictest0-21swapper/312:29:593
2679826993129,1cyclictest0-21swapper/312:29:583
2679826993129,1cyclictest0-21swapper/309:34:593
2679826993118,3cyclictest2757558-21turbostat11:10:013
2679824993129,1cyclictest0-21swapper/210:04:592
2679824993129,1cyclictest0-21swapper/210:04:592
2679824993128,2cyclictest0-21swapper/212:14:552
267982499310,29cyclictest0-21swapper/212:20:182
4505291300,7ptp4l441ktimers/311:05:013
2679826993025,4cyclictest2738171-21turbostat10:14:553
267982699300,29cyclictest0-21swapper/308:45:113
267982699300,28cyclictest0-21swapper/311:40:123
267982699300,28cyclictest0-21swapper/310:30:023
2679824993027,2cyclictest0-21swapper/210:50:592
2679824993027,2cyclictest0-21swapper/209:40:142
267982499300,2cyclictest0-21swapper/208:40:142
267982499300,29cyclictest0-21swapper/211:15:142
2679826992927,1cyclictest0-21swapper/308:36:593
2679826992926,2cyclictest0-21swapper/311:50:023
2679826992925,2cyclictest2731284-21turbostat09:50:003
2679826992922,1cyclictest0-21swapper/307:15:143
2679826992921,1cyclictest0-21swapper/312:34:593
267982699290,28cyclictest0-21swapper/310:35:123
267982699290,28cyclictest0-21swapper/308:15:593
267982699290,27cyclictest0-21swapper/308:50:153
2679824992926,2cyclictest0-21swapper/212:15:142
2679824992926,2cyclictest0-21swapper/209:00:172
2679824992926,2cyclictest0-21swapper/208:20:132
2679824992926,2cyclictest0-21swapper/208:00:142
2679824992926,2cyclictest0-21swapper/207:35:152
2679824992925,2cyclictest0-21swapper/209:20:142
267982499290,27cyclictest0-21swapper/212:35:142
267982499290,27cyclictest0-21swapper/210:30:002
267982499290,27cyclictest0-21swapper/210:05:142
267982499290,27cyclictest0-21swapper/209:05:132
267982499290,27cyclictest0-21swapper/208:35:132
267982499290,27cyclictest0-21swapper/208:15:132
2679820992923,4cyclictest2679874-21latency_hist07:10:011
2679826992826,1cyclictest0-21swapper/311:25:123
2679826992826,1cyclictest0-21swapper/307:51:593
267982699280,27cyclictest0-21swapper/307:40:133
267982699280,27cyclictest0-21swapper/307:36:583
267982699280,26cyclictest0-21swapper/310:45:173
267982699280,26cyclictest0-21swapper/307:10:133
2679824992825,2cyclictest0-21swapper/211:58:592
2679824992825,2cyclictest0-21swapper/210:30:012
2679824992825,2cyclictest0-21swapper/208:14:552
4505291270,7ptp4l61790irq/134-eno1-TxRx-010:25:003
4505291270,7ptp4l441ktimers/312:10:003
267982699270,26cyclictest0-21swapper/310:28:003
267982699270,26cyclictest0-21swapper/307:24:583
267982499270,25cyclictest0-21swapper/210:10:122
267982499270,25cyclictest0-21swapper/207:45:142
4505291260,6ptp4l61790irq/134-eno1-TxRx-011:20:003
4505291260,6ptp4l61790irq/134-eno1-TxRx-010:05:013
4505291260,6ptp4l61790irq/134-eno1-TxRx-010:05:013
4505291260,6ptp4l61790irq/134-eno1-TxRx-009:40:013
4505291260,6ptp4l441ktimers/310:59:593
4505291260,6ptp4l441ktimers/309:44:553
4505291260,6ptp4l441ktimers/309:05:013
4505291260,5ptp4l61790irq/134-eno1-TxRx-012:19:553
2679826992624,1cyclictest0-21swapper/311:14:553
2679826992623,2cyclictest0-21swapper/308:33:593
2679824992624,1cyclictest0-21swapper/207:55:002
2679824992623,2cyclictest0-21swapper/211:22:002
2679824992623,2cyclictest0-21swapper/207:32:592
267982499260,24cyclictest0-21swapper/207:20:452
4505291250,5ptp4l61790irq/134-eno1-TxRx-007:49:553
4505291250,5ptp4l441ktimers/310:09:593
4505291250,5ptp4l441ktimers/308:44:563
2679826992523,1cyclictest0-21swapper/308:55:593
267982699250,24cyclictest0-21swapper/309:08:593
2679824992523,1cyclictest0-21swapper/210:36:592
2679824992522,2cyclictest0-21swapper/211:12:002
4505291240,5ptp4l61790irq/134-eno1-TxRx-012:14:553
4505291240,5ptp4l441ktimers/312:39:563
4505291240,4ptp4l61790irq/134-eno1-TxRx-012:24:553
2679824992421,2cyclictest0-21swapper/208:45:112
2679824992421,2cyclictest0-21swapper/207:10:242
2679824992320,2cyclictest0-21swapper/212:29:562
2679824992320,2cyclictest0-21swapper/212:29:552
2679824992320,2cyclictest0-21swapper/211:52:592
2679824992319,2cyclictest0-21swapper/210:56:002
2679824992118,2cyclictest0-21swapper/211:38:592
2679824992118,2cyclictest0-21swapper/211:31:582
2679824992118,2cyclictest0-21swapper/210:47:592
2679824992118,2cyclictest0-21swapper/210:42:592
2679824991916,2cyclictest0-21swapper/209:51:592
2679824991512,2cyclictest0-21swapper/207:58:002
267981899132,8cyclictest2679814-21cyclictest07:10:010
267982499118,2cyclictest0-21swapper/211:08:002
267982499118,2cyclictest0-21swapper/208:54:562
267982499107,2cyclictest0-21swapper/209:48:122
26798249980,7cyclictest0-21swapper/212:04:012
26798249980,7cyclictest0-21swapper/210:18:192
26798249930,3cyclictest0-21swapper/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional