You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-11 - 23:52
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot6 (updated Mon Dec 11, 2023 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34482769913093,34cyclictest3507849-21latency_hist10:15:010
34482789910597,6cyclictest3486651-21latency_hist09:10:011
344827899105100,4cyclictest3528100-21kworker/u8:2+flush-179:012:15:351
34482789910092,6cyclictest3493279-21latency_hist09:30:011
3448276999885,10cyclictest2158010-21systemd-journal11:25:010
3448276999490,3cyclictest3461175-21kworker/u8:2+flush-179:009:10:360
3448276999490,3cyclictest3461175-21kworker/u8:2+flush-179:009:10:360
3448278999284,6cyclictest3517424-21apt-get10:45:011
3448276999284,6cyclictest3429190-21kworker/u8:1+flush-179:007:20:350
3448278998980,7cyclictest3490104-21latency_hist09:20:011
3448276998776,9cyclictest3469143-21apt-get08:15:220
3448278998575,7cyclictest3484808-21latency_hist09:05:021
3448278998277,3cyclictest3454532-21kworker/u8:0+events_unbound07:50:251
3448276998269,10cyclictest3543400-21apt-get12:04:590
3448276998071,7cyclictest2158010-21systemd-journal10:45:340
3448276997867,9cyclictest3464386-21apt-get08:00:240
3448276997766,9cyclictest3538861-21apt-get11:50:200
3448276997672,3cyclictest3470527-21kworker/u8:3+events_unbound08:29:560
3448278997367,4cyclictest3478430-21kworker/u8:3+flush-179:008:50:391
3448276997360,10cyclictest3453221-21apt-get07:25:230
3448276997162,7cyclictest529-21rs:main0
3448276997161,7cyclictest2158010-21systemd-journal08:15:000
3448276997062,6cyclictest3504934-21apt-get10:05:190
3448276996962,5cyclictest3522660-21kworker/u8:2+flush-179:011:05:010
3448276996957,9cyclictest3535692-21apt-get11:40:200
3448276996730,7cyclictest3484862-21pluginstate09:05:010
3448278996554,8cyclictest3515815-21latency_hist10:40:011
3448278996453,9cyclictest3456393-21apt-get07:35:171
3448278996251,9cyclictest3448442-21apt-get07:10:201
3448276996258,3cyclictest3454532-21kworker/u8:0+events_unbound07:40:020
4652991610,42ptp4l441ktimers/307:07:443
3448276996157,3cyclictest3509415-21kworker/u8:1+flush-179:010:20:310
3448278996052,6cyclictest2158010-21systemd-journal10:42:341
3448278996051,7cyclictest3483221-21latency_hist09:00:011
3448276996055,3cyclictest2158010-21systemd-journal12:05:030
3448278995948,9cyclictest3490646-21diskstats09:20:271
3448276995951,6cyclictest3537277-21apt-get11:45:200
3448278995847,9cyclictest2158010-21systemd-journal12:30:341
3448276995852,5cyclictest3461175-21kworker/u8:2+flush-179:008:55:370
3448276995848,7cyclictest3481613-21latency_hist08:55:010
3448278995746,9cyclictest3521352-21apt-get10:55:181
3448278995649,5cyclictest3545218-21kworker/u8:0+flush-179:012:10:271
344827699561,51cyclictest1-21systemd10:42:330
3448278995546,6cyclictest3506262-21latency_hist10:10:011
3448276995547,6cyclictest3488481-21kworker/u8:1+flush-179:009:50:290
3448276995546,7cyclictest3550245-21/usr/sbin/munin12:25:270
3448276995546,7cyclictest3534067-21/usr/sbin/munin11:35:240
3448276995450,3cyclictest3448200-21kworker/u8:2+events_unbound07:30:010
3448276995445,7cyclictest2158010-21systemd-journal10:20:010
3448281995350,2cyclictest0-21swapper/207:13:592
3448276995345,6cyclictest3520838-21latency_hist10:55:010
3448276995343,7cyclictest3448156-21latency_hist07:10:020
3448278995246,5cyclictest3488481-21kworker/u8:1+flush-179:009:40:241
344827899522,48cyclictest3503078-21turbostat10:04:541
3448276995244,6cyclictest3534822-21kworker/u8:3+events_unbound12:15:230
3448282995147,2cyclictest0-21swapper/309:53:583
3448278995142,6cyclictest3470731-21/usr/sbin/munin08:20:341
3448276995147,3cyclictest3512611-21kworker/u8:0+events_unbound10:35:440
3448276995142,7cyclictest3483184-21kworker/u8:3+flush-179:009:30:010
3448276995141,8cyclictest3498314-21latency_hist09:45:010
3448278995046,3cyclictest3501465-21kworker/u8:0+events_unbound09:55:231
3448278995042,6cyclictest529-21rs:main1
3448282994947,1cyclictest0-21swapper/308:54:593
3448278994942,5cyclictest3499897-21latency_hist09:50:011
3448278994941,6cyclictest3540176-21kworker/u8:0+flush-179:011:55:001
3448278994940,7cyclictest2158010-21systemd-journal07:15:341
3448278994936,11cyclictest2158010-21systemd-journal11:31:311
3448276994944,4cyclictest3461175-21kworker/u8:2+flush-179:009:30:310
3448278994842,4cyclictest3479106-21journal-offline08:45:291
3448278994836,10cyclictest3540448-21apt-get11:55:191
3448276994845,2cyclictest3478481-21systemd-run08:45:000
3448278994743,2cyclictest725-21Xorg10:13:591
3448278994742,3cyclictest3459390-21systemd-run07:45:001
3448278994739,7cyclictest3461175-21kworker/u8:2+flush-179:010:20:321
3448278994729,17cyclictest3509415-21kworker/u8:1+flush-179:010:30:231
3448276994740,6cyclictest3461175-21kworker/u8:2+flush-179:010:00:010
3448276994740,5cyclictest3553193-21turbostat12:39:550
3448276994738,6cyclictest3551850-21apt-get12:30:230
3448278994639,5cyclictest3517905-21kworker/u8:0+flush-179:011:00:251
3448278994638,6cyclictest3476866-21latency_hist08:40:011
3448278994540,4cyclictest3528100-21kworker/u8:2+flush-179:012:20:331
344827899452,41cyclictest3535437-21turbostat11:44:541
3448276994541,3cyclictest3461175-21kworker/u8:2+events_unbound09:22:590
3448276994539,4cyclictest725-21Xorg07:52:000
3448278994440,3cyclictest3451413-21kworker/u8:3+flush-179:007:45:271
3448276994440,3cyclictest3449807-21systemd-run07:15:000
3448276994434,8cyclictest3462780-21apt-get07:55:190
3448278994341,1cyclictest0-21swapper/112:39:561
3448278994339,2cyclictest0-21swapper/109:55:021
344827899431,9cyclictest3476830-21cron08:40:011
3448276994337,5cyclictest3522660-21kworker/u8:2+flush-179:011:05:240
3448282994240,1cyclictest0-21swapper/310:08:583
3448282994240,1cyclictest0-21swapper/307:13:583
3448278994238,2cyclictest0-21swapper/108:15:011
3448278994234,6cyclictest3543882-21/usr/sbin/munin12:05:301
344827899423,36cyclictest271rcuc/110:17:001
3448276994238,3cyclictest3480064-21systemd-run08:50:010
3448282994139,1cyclictest0-21swapper/311:42:593
3448282994139,1cyclictest0-21swapper/310:18:003
3448282994134,6cyclictest0-21swapper/311:38:003
3448278994138,2cyclictest3525866-21kworker/u8:1+flush-179:011:20:301
3448278994135,5cyclictest3509415-21kworker/u8:1+flush-179:010:25:311
3448278994134,4cyclictest0-21swapper/111:37:591
3448278994133,6cyclictest3528100-21kworker/u8:2+flush-179:011:25:291
3448278994127,12cyclictest3521036-21apt-config10:55:001
3448276994137,3cyclictest3461175-21kworker/u8:2+events_unbound10:33:590
3448276994137,2cyclictest725-21Xorg08:06:590
3448276994137,2cyclictest0-21swapper/007:35:200
3448282994034,4cyclictest3503078-21turbostat10:00:003
3448278994037,2cyclictest0-21swapper/108:06:591
3448278994035,4cyclictest0-21swapper/109:10:591
3448278994035,4cyclictest0-21swapper/109:10:591
3448278994031,5cyclictest3470700-21sort08:20:011
3448276994037,2cyclictest0-21swapper/011:14:540
3448276994036,3cyclictest3476871-21munin-plugin-st08:40:000
3448276994030,8cyclictest3461175-21kworker/u8:2+events_unbound09:10:010
344827699400,3cyclictest725-21Xorg11:32:580
4652991390,3ptp4l441ktimers/311:05:223
3448282993933,4cyclictest3520855-21turbostat10:55:003
3448282993933,4cyclictest3498323-21turbostat09:45:003
344828299390,38cyclictest0-21swapper/310:30:003
344828299390,37cyclictest3451375-21turbostat07:20:003
344827899391,37cyclictest0-21swapper/107:29:591
344827699392,34cyclictest3450027-21apt-get07:15:200
344827699391,8cyclictest3461098-21cat07:50:000
3448282993836,1cyclictest0-21swapper/312:10:253
3448282993832,4cyclictest3454564-21turbostat07:30:003
3448278993835,2cyclictest0-21swapper/108:00:251
3448276993835,2cyclictest0-21swapper/012:24:550
3448276993835,2cyclictest0-21swapper/008:20:590
3448276993829,7cyclictest3495359-21/usr/sbin/munin09:35:250
344827699381,5cyclictest0-21swapper/010:25:230
3448282993732,4cyclictest3551611-21turbostat12:30:013
3448281993734,2cyclictest0-21swapper/210:24:562
3448278993734,2cyclictest0-21swapper/111:05:581
3448278993733,3cyclictest3543520-21apt-config12:05:011
3448278993727,8cyclictest3551759-21grep12:30:001
3448278993727,8cyclictest3464286-21latency_hist08:00:011
3448276993733,2cyclictest725-21Xorg10:59:590
3448276993731,5cyclictest3498303-21kworker/u8:3+flush-179:009:45:340
3448282993633,2cyclictest0-21swapper/310:45:013
3448282993633,2cyclictest0-21swapper/310:36:593
344828299360,35cyclictest0-21swapper/312:09:003
3448281993634,1cyclictest0-21swapper/209:40:012
3448281993634,1cyclictest0-21swapper/208:14:592
3448281993633,2cyclictest0-21swapper/210:26:592
3448281993633,2cyclictest0-21swapper/210:16:002
3448278993632,3cyclictest3475281-21systemd-run08:35:001
3448278993631,3cyclictest0-21swapper/109:40:001
3448276993633,2cyclictest14-21ksoftirqd/009:15:190
3448276993624,10cyclictest3545485-21apt-get12:10:250
3448282993530,4cyclictest3484820-21sleep09:05:003
3448282993528,5cyclictest3468892-21turbostat08:15:003
3448281993532,2cyclictest0-21swapper/209:49:002
344828199350,34cyclictest0-21swapper/207:30:232
3448278993524,6cyclictest0-21swapper/107:20:211
344827899351,6cyclictest3472594-21expr08:25:231
3448276993530,4cyclictest3529106-21sh11:20:010
3448276993529,4cyclictest543-21gdbus10:00:330
3448282993432,1cyclictest0-21swapper/310:55:583
3448282993432,1cyclictest0-21swapper/310:13:593
3448282993432,1cyclictest0-21swapper/309:08:593
3448282993432,1cyclictest0-21swapper/308:39:553
3448282993432,1cyclictest0-21swapper/307:25:003
344828299340,26cyclictest0-21swapper/308:20:013
344828299340,1cyclictest0-21swapper/312:30:003
3448281993432,1cyclictest0-21swapper/208:35:002
3448281993430,2cyclictest0-21swapper/209:54:552
344828199340,1cyclictest0-21swapper/210:40:212
344827899348,6cyclictest3456158-21idleruntime-cro07:35:011
3448282993331,1cyclictest0-21swapper/310:03:003
3448282993330,2cyclictest0-21swapper/312:40:013
3448281993331,1cyclictest0-21swapper/212:30:242
3448281993330,2cyclictest0-21swapper/209:21:592
344828199330,31cyclictest0-21swapper/212:15:592
3448278993329,3cyclictest3528100-21kworker/u8:2+flush-179:011:15:261
3448278993327,4cyclictest2158010-21systemd-journal11:49:561
3448276993327,4cyclictest3541777-21sendmail12:00:000
344827699331,29cyclictest0-21swapper/011:25:180
3448282993230,1cyclictest0-21swapper/308:45:593
3448282993229,2cyclictest0-21swapper/307:54:593
3448282993227,4cyclictest3529067-21turbostat11:20:013
3448282993225,6cyclictest0-21swapper/312:15:223
344828299320,31cyclictest0-21swapper/311:20:183
344828299320,2cyclictest0-21swapper/309:20:263
344828299320,1cyclictest0-21swapper/309:30:013
3448281993230,1cyclictest0-21swapper/211:54:592
3448281993229,2cyclictest0-21swapper/209:11:592
3448281993229,2cyclictest0-21swapper/209:11:592
3448281993228,2cyclictest0-21swapper/210:47:582
3448281993228,2cyclictest0-21swapper/208:00:592
344828199320,30cyclictest0-21swapper/210:30:192
3448278993223,7cyclictest2158010-21systemd-journal09:30:341
3448282993129,1cyclictest0-21swapper/311:59:563
3448282993125,5cyclictest0-21swapper/311:02:593
3448282993119,11cyclictest3496721-21turbostat09:40:003
344828199310,30cyclictest0-21swapper/210:10:202
344828199310,30cyclictest0-21swapper/207:25:252
344828199310,30cyclictest0-21swapper/207:15:202
344828199310,2cyclictest0-21swapper/210:58:002
3448282993025,4cyclictest0-21swapper/307:55:213
3448282993024,5cyclictest0-21swapper/311:55:023
3448282993024,5cyclictest0-21swapper/309:31:593
344828299300,2cyclictest0-21swapper/308:20:243
344828299300,29cyclictest0-21swapper/310:40:213
3448281993028,1cyclictest0-21swapper/208:24:552
3448281993027,2cyclictest0-21swapper/211:45:232
3448281993027,2cyclictest0-21swapper/210:40:022
3448281993027,2cyclictest0-21swapper/209:15:192
3448281993027,2cyclictest0-21swapper/208:15:232
3448281993027,2cyclictest0-21swapper/208:08:002
344828199300,2cyclictest0-21swapper/211:25:192
344828199300,2cyclictest0-21swapper/209:30:032
344828199300,2cyclictest0-21swapper/207:25:002
344828199300,29cyclictest0-21swapper/212:05:022
344828199300,29cyclictest0-21swapper/211:10:212
344828199300,29cyclictest0-21swapper/209:40:202
344828199300,29cyclictest0-21swapper/208:55:262
344828199300,28cyclictest0-21swapper/212:20:232
344828199300,28cyclictest0-21swapper/211:55:232
344828199300,28cyclictest0-21swapper/209:05:012
344828199300,28cyclictest0-21swapper/207:50:012
4652991290,6ptp4l61790irq/134-eno1-TxRx-007:50:003
3448282992927,1cyclictest0-21swapper/309:20:003
3448282992927,1cyclictest0-21swapper/309:10:023
3448282992927,1cyclictest0-21swapper/309:10:023
3448282992927,1cyclictest0-21swapper/307:40:263
3448282992926,2cyclictest0-21swapper/312:20:213
3448282992926,2cyclictest0-21swapper/312:05:003
344828299290,28cyclictest0-21swapper/309:45:203
344828299290,28cyclictest0-21swapper/308:45:013
344828299290,1cyclictest0-21swapper/311:35:003
3448281992927,1cyclictest0-21swapper/211:00:202
3448281992926,2cyclictest0-21swapper/211:40:212
3448281992926,2cyclictest0-21swapper/209:55:192
3448281992926,2cyclictest0-21swapper/207:55:012
3448281992926,2cyclictest0-21swapper/207:35:182
344828199290,2cyclictest0-21swapper/209:05:012
344828199290,28cyclictest0-21swapper/211:30:202
344828199290,28cyclictest0-21swapper/208:40:022
344828199290,27cyclictest0-21swapper/212:35:202
344828199290,27cyclictest0-21swapper/212:00:242
344828199290,27cyclictest0-21swapper/211:15:222
344828199290,27cyclictest0-21swapper/210:05:212
344828199290,27cyclictest0-21swapper/208:40:012
344828199290,27cyclictest0-21swapper/208:25:222
3448282992826,1cyclictest0-21swapper/311:25:203
3448282992825,2cyclictest0-21swapper/310:30:223
3448282992825,2cyclictest0-21swapper/310:25:013
344828299280,27cyclictest0-21swapper/311:45:013
344828299280,27cyclictest0-21swapper/311:10:213
344828299280,26cyclictest0-21swapper/308:55:263
344828299280,26cyclictest0-21swapper/308:00:263
3448281992826,1cyclictest0-21swapper/212:25:212
3448281992826,1cyclictest0-21swapper/207:40:022
3448281992825,2cyclictest0-21swapper/212:10:252
3448281992825,2cyclictest0-21swapper/211:06:592
3448281992825,2cyclictest0-21swapper/209:25:282
3448281992825,2cyclictest0-21swapper/207:55:012
344828199280,27cyclictest0-21swapper/208:51:592
344828199280,26cyclictest0-21swapper/211:35:182
344828199280,26cyclictest0-21swapper/210:50:592
4652991270,7ptp4l441ktimers/308:30:003
4652991270,6ptp4l61790irq/134-eno1-TxRx-007:40:023
4652991270,6ptp4l441ktimers/308:10:003
344828299270,2cyclictest0-21swapper/307:30:013
344828299270,26cyclictest0-21swapper/308:30:233
344828199260,2cyclictest0-21swapper/211:20:182
344828199250,24cyclictest0-21swapper/210:01:002
3448281992219,2cyclictest0-21swapper/208:45:242
344827899224,16cyclictest3448380-21cat07:10:011
34482819930,2cyclictest0-21swapper/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional