You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 16:41
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6.osadl.org (updated Thu Nov 06, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2693317999684,9cyclictest2734945-21latency_hist09:20:011
2693313998978,8cyclictest2720842-21/usr/sbin/munin08:35:250
2693313998378,4cyclictest2775708-21kworker/u8:3+events_unbound11:40:000
2693313998277,4cyclictest1969335-21kworker/u8:1+events_unbound07:20:010
2693313998174,5cyclictest1969335-21kworker/u8:1+events_unbound08:10:010
2693317998068,9cyclictest2791963-21latency_hist12:15:011
2693313998069,9cyclictest271-21systemd-journal10:00:000
2693317997968,8cyclictest2696711-21/usr/sbin/munin07:20:251
2693313997871,5cyclictest1969335-21kworker/u8:1+events_unbound08:30:000
2693313997871,5cyclictest1969335-21kworker/u8:1+events_unbound08:29:590
2693317997769,6cyclictest2741316-21apt-get09:40:031
2693313997570,4cyclictest2733873-21kworker/u8:1+events_unbound09:30:230
2693313996961,6cyclictest2730199-21kworker/u8:3+flush-179:009:15:000
2693313996957,8cyclictest271-21systemd-journal07:30:010
2693317996858,7cyclictest2759369-21latency_hist10:35:011
2693313996863,3cyclictest2787160-21kworker/u8:1+flush-179:012:00:240
2693313996858,7cyclictest2792459-21kworker/u8:3+flush-179:012:40:000
2693313996759,6cyclictest2796979-21kworker/u8:0+flush-179:012:30:010
2693313996757,8cyclictest271-21systemd-journal10:45:000
2693313996656,8cyclictest2739735-21kworker/u8:3+events_unbound09:50:000
269332599650,63cyclictest0-21swapper/310:42:593
2693317996554,7cyclictest2741296-21latency_hist09:40:011
2693313996557,6cyclictest2754640-21kworker/u8:1+events_unbound10:20:000
2693313996459,3cyclictest2730199-21kworker/u8:3+events_unbound09:05:000
2693313996358,4cyclictest2764289-21kworker/u8:2+events_unbound11:10:360
2693313996356,5cyclictest2767832-21kworker/u8:0+events_unbound11:05:270
2693313996356,5cyclictest2767832-21kworker/u8:0+events_unbound11:05:260
2693313996256,4cyclictest2775708-21kworker/u8:3+flush-179:011:45:000
2693313996255,5cyclictest2693535-21kworker/u8:0+flush-179:007:50:180
2693313996255,5cyclictest1969335-21kworker/u8:1+flush-179:007:45:240
2693317996049,8cyclictest2743390-21/usr/sbin/munin09:45:241
2693313996054,4cyclictest2751448-21kworker/u8:0+events_unbound10:20:240
2693313995949,7cyclictest2793834-21latency_hist12:20:010
2693313995948,8cyclictest2701229-21latency_hist07:35:020
2693322995855,2cyclictest0-21swapper/211:27:592
2693317995855,2cyclictest0-21swapper/112:27:001
2693313995751,4cyclictest2782457-21kworker/u8:0+events_unbound11:55:140
2693313995744,9cyclictest2774205-21latency_hist11:20:010
2693317995646,8cyclictest2716086-21/usr/sbin/munin08:20:231
2693313995649,4cyclictest2693535-21kworker/u8:0+flush-179:007:10:240
2693317995544,9cyclictest2735484-21diskstats09:20:151
2693317995544,9cyclictest2735484-21diskstats09:20:151
2693313995549,4cyclictest2751448-21kworker/u8:0+flush-179:010:40:010
2693317995436,16cyclictest271-21systemd-journal08:58:251
2693317995343,8cyclictest2724055-21apt-get08:45:121
2693313995345,6cyclictest2762816-21apt-get10:45:110
2693317995146,4cyclictest2787160-21kworker/u8:1+flush-179:012:00:001
2693313995140,9cyclictest2784268-21apt-get11:50:120
2693313995044,4cyclictest2759354-21kworker/u8:3+flush-179:010:50:220
2693313995044,4cyclictest2743164-21kworker/u8:2+flush-179:010:05:000
2693313995044,4cyclictest2739735-21kworker/u8:3+flush-179:009:55:010
2693313995041,7cyclictest2766225-21/usr/sbin/munin10:55:240
2693313995039,7cyclictest2754644-21kworker/u8:2+flush-179:010:30:110
2693317994943,4cyclictest2720052-21kworker/u8:0+flush-179:008:30:261
2693317994942,5cyclictest2748426-21kworker/u8:1+flush-179:010:00:161
2693317994940,7cyclictest2767832-21kworker/u8:0+events_unbound11:10:001
2693317994940,7cyclictest2767832-21kworker/u8:0+events_unbound11:10:001
2693313994936,9cyclictest2798591-21latency_hist12:35:010
269332599480,45cyclictest2788811-21turbostat12:05:013
2693317994845,2cyclictest723-21Xorg08:13:001
2693317994841,5cyclictest2704752-21kworker/u8:3+events_unbound08:05:001
2693317994839,7cyclictest2775708-21kworker/u8:3+flush-179:011:30:001
2693313994844,3cyclictest2720052-21kworker/u8:0+events_unbound09:40:150
2693313994842,4cyclictest2730199-21kworker/u8:3+events_unbound09:15:260
2693313994841,5cyclictest2722225-21kworker/u8:2+flush-179:008:50:000
269331399481,44cyclictest2728614-21munin-run09:00:010
4670791470,13ptp4l441ktimers/307:10:013
2693317994738,7cyclictest2767832-21kworker/u8:0+flush-179:011:00:191
2693313994640,4cyclictest2723828-21kworker/u8:4+flush-179:008:50:180
2693317994537,6cyclictest2792459-21kworker/u8:3+flush-179:012:39:591
2693317994536,7cyclictest2777682-21kworker/u8:0+events_unbound11:30:241
2693313994541,3cyclictest723-21Xorg08:10:580
2693313994534,8cyclictest271-21systemd-journal10:14:260
2693313994440,3cyclictest2736566-21munin-run09:25:000
2693313994440,3cyclictest2736566-21munin-run09:25:000
2693313994439,4cyclictest2772846-21kworker/u8:1+flush-179:011:49:550
2693317994339,3cyclictest2693535-21kworker/u8:0+flush-179:007:40:271
2693317994336,5cyclictest2720052-21kworker/u8:0+flush-179:008:55:011
2693317994335,6cyclictest2767832-21kworker/u8:0+flush-179:011:15:001
269331799430,41cyclictest723-21Xorg08:36:591
2693313994338,4cyclictest2738090-21kworker/u8:2+flush-179:009:29:550
2693313994338,4cyclictest2693535-21kworker/u8:0+events_unbound07:35:240
2693317994236,4cyclictest2797259-21apt-get12:30:131
2693317994232,8cyclictest2756432-21/usr/sbin/munin10:25:251
2693313994234,6cyclictest2792459-21kworker/u8:3+flush-179:012:20:220
269332299410,2cyclictest0-21swapper/210:01:002
2693317994136,4cyclictest2759354-21kworker/u8:3+flush-179:010:55:001
2693317994136,4cyclictest2751448-21kworker/u8:0+flush-179:010:35:261
2693317994134,5cyclictest2717440-21turbostat08:29:561
2693317994134,5cyclictest2717440-21turbostat08:29:551
269331799410,39cyclictest723-21Xorg07:36:591
2693313994136,4cyclictest2772846-21kworker/u8:1+flush-179:011:29:550
2693313994135,4cyclictest2754644-21kworker/u8:2+flush-179:010:30:000
2693313994128,9cyclictest2741316-21apt-get09:40:000
2693325994034,5cyclictest2720622-21turbostat08:34:593
2693325994034,5cyclictest2694870-21turbostat07:15:003
269332599400,38cyclictest2736534-21turbostat09:25:003
269332599400,38cyclictest2736534-21turbostat09:24:593
2693317994033,5cyclictest2739735-21kworker/u8:3+flush-179:010:15:001
2693317994029,10cyclictest2720052-21kworker/u8:0+flush-179:009:30:001
269331799400,38cyclictest723-21Xorg12:23:591
269331799400,38cyclictest723-21Xorg11:42:591
2693313994035,4cyclictest2782457-21kworker/u8:0+events_unbound12:05:250
2693313994035,4cyclictest2782457-21kworker/u8:0+events_unbound12:05:250
2693313994031,7cyclictest2769198-21apt-get11:05:000
4670791390,3ptp4l431rcuc/309:15:003
2693325993934,4cyclictest2785613-21turbostat11:55:003
2693325993933,4cyclictest2787206-21turbostat12:00:003
269332599390,4cyclictest2767578-21turbostat11:00:003
269332599390,4cyclictest2759382-21turbostat10:35:003
269332599390,38cyclictest2731759-21turbostat09:10:003
2693317993934,4cyclictest723-21Xorg12:00:591
269331799390,3cyclictest2733334-21cron09:15:011
2693313993935,2cyclictest723-21Xorg10:08:590
2693313993934,4cyclictest2715883-21kworker/u8:2+flush-179:008:30:210
269331399391,37cyclictest723-21Xorg07:23:590
269332599380,3cyclictest2782435-21turbostat11:45:003
2693322993835,2cyclictest0-21swapper/210:16:582
269332299380,36cyclictest0-21swapper/211:08:592
269332299380,36cyclictest0-21swapper/211:08:592
269332299380,36cyclictest0-21swapper/208:21:582
269332299380,36cyclictest0-21swapper/208:17:002
2693317993833,4cyclictest2693535-21kworker/u8:0+flush-179:007:25:141
2693317993820,13cyclictest2751448-21kworker/u8:0+flush-179:010:20:001
269331799380,37cyclictest0-21swapper/107:58:001
2693313993832,4cyclictest2779240-21dump-pmu-power11:35:000
2693313993828,8cyclictest2791972-21apt-get12:15:000
269331399380,2cyclictest723-21Xorg07:55:590
269331399380,2cyclictest191rcuc/008:18:590
2693325993735,1cyclictest0-21swapper/310:29:593
2693325993731,4cyclictest2707887-21turbostat07:55:013
2693325993730,5cyclictest2784020-21turbostat11:50:003
269332599370,35cyclictest0-21swapper/312:10:593
2693322993734,2cyclictest0-21swapper/212:18:592
2693322993734,2cyclictest0-21swapper/211:15:592
2693322993734,2cyclictest0-21swapper/210:53:592
2693322993734,2cyclictest0-21swapper/207:37:592
269332299370,35cyclictest0-21swapper/212:29:002
269332299370,35cyclictest0-21swapper/210:48:592
269332299370,35cyclictest0-21swapper/208:49:592
2693317993733,2cyclictest723-21Xorg12:17:591
2693317993732,4cyclictest2704752-21kworker/u8:3+flush-179:008:15:261
2693317993732,3cyclictest2712688-21systemd-run08:10:001
2693317993728,7cyclictest2743164-21kworker/u8:2+flush-179:009:50:211
2693325993634,1cyclictest0-21swapper/307:35:583
2693325993633,2cyclictest0-21swapper/309:26:593
2693325993631,4cyclictest2779245-21turbostat11:35:003
2693325993631,4cyclictest2728565-21turbostat09:00:003
2693325993631,4cyclictest2715830-21turbostat08:20:003
269332599360,35cyclictest0-21swapper/311:20:593
269332599360,35cyclictest0-21swapper/309:53:003
269332599360,35cyclictest0-21swapper/307:44:003
269332599360,1cyclictest0-21swapper/312:09:563
269332599360,1cyclictest0-21swapper/312:09:553
269332599360,1cyclictest0-21swapper/308:26:003
269332599360,1cyclictest0-21swapper/308:25:593
2693322993633,2cyclictest0-21swapper/211:35:592
2693322993633,2cyclictest0-21swapper/211:23:592
269332299360,1cyclictest0-21swapper/212:35:592
269332299360,1cyclictest0-21swapper/208:39:542
269332299360,1cyclictest0-21swapper/207:24:582
2693325993533,1cyclictest0-21swapper/308:09:553
2693325993532,2cyclictest0-21swapper/308:53:583
2693325993529,5cyclictest2800448-21turbostat12:40:003
269332599350,1cyclictest0-21swapper/309:40:003
2693317993533,1cyclictest0-21swapper/110:24:541
2693317993532,2cyclictest0-21swapper/109:06:591
2693317993530,3cyclictest2705341-21fschecks_time07:45:151
2693313993530,4cyclictest515-21gdbus11:20:250
2693325993429,4cyclictest2771032-21turbostat11:09:593
2693325993429,4cyclictest2771032-21turbostat11:09:593
2693325993429,4cyclictest2754597-21turbostat10:19:593
2693322993431,2cyclictest0-21swapper/208:41:002
269331799343,29cyclictest2782684-21apt-get11:45:121
2693317993430,3cyclictest0-21swapper/107:35:011
2693313993431,2cyclictest723-21Xorg08:21:590
269331399341,31cyclictest2722470-21apt-get08:40:140
2693325993328,4cyclictest2753005-21turbostat10:15:013
2693325993328,4cyclictest2699654-21turbostat07:30:013
2693322993330,2cyclictest0-21swapper/210:56:592
2693322993330,2cyclictest0-21swapper/209:11:592
2693322993329,2cyclictest0-21swapper/207:52:592
2693317993330,2cyclictest2774230-21perf11:20:001
2693317993329,3cyclictest2707878-21munin-plugin-st07:55:011
2693317993329,2cyclictest723-21Xorg07:19:591
2693317993327,5cyclictest2780406-21kworker/u8:2+flush-179:011:50:171
2693317993327,5cyclictest2739735-21kworker/u8:3+flush-179:010:05:291
2693317993323,8cyclictest2789050-21apt-get12:05:111
2693317993323,8cyclictest2789050-21apt-get12:05:111
269331399331,30cyclictest0-21swapper/008:00:160
2693325993230,1cyclictest0-21swapper/309:16:593
2693322993229,2cyclictest0-21swapper/210:37:592
2693322993229,2cyclictest0-21swapper/210:09:552
2693322993229,2cyclictest0-21swapper/208:04:552
2693322993229,2cyclictest0-21swapper/207:18:592
2693317993229,2cyclictest723-21Xorg09:34:581
2693317993226,5cyclictest2694369-21phc2sys07:10:241
2693317993225,5cyclictest723-21Xorg10:55:591
2693317993225,5cyclictest2762570-21dump-pmu-power10:45:001
2693325993129,1cyclictest0-21swapper/307:33:593
2693325993125,5cyclictest2714246-21turbostat08:14:593
2693322993127,2cyclictest0-21swapper/209:25:592
2693317993125,5cyclictest2777682-21kworker/u8:0+flush-179:011:35:201
2693317993122,7cyclictest2723798-21latency_hist08:45:001
2693313993124,5cyclictest2730199-21kworker/u8:3+flush-179:009:05:230
2693325993025,4cyclictest2711077-21turbostat08:05:003
269332599300,28cyclictest0-21swapper/308:21:593
2693322993027,2cyclictest0-21swapper/207:30:132
2693322993027,2cyclictest0-21swapper/207:10:162
2693322993026,2cyclictest0-21swapper/210:22:002
2693322993026,2cyclictest0-21swapper/209:45:592
2693317993027,2cyclictest2774735-21apt-get11:20:151
2693317993027,2cyclictest2730167-21munin-run09:04:591
2693325992926,2cyclictest0-21swapper/312:15:163
2693325992924,4cyclictest0-21swapper/309:00:153
269332599290,28cyclictest0-21swapper/312:22:593
269332599290,27cyclictest0-21swapper/310:45:133
2693322992927,1cyclictest0-21swapper/208:30:002
2693322992927,1cyclictest0-21swapper/208:30:002
2693322992926,2cyclictest0-21swapper/209:30:122
2693322992926,2cyclictest0-21swapper/207:55:112
2693322992925,2cyclictest0-21swapper/209:39:002
269332299290,27cyclictest0-21swapper/212:34:582
269332299290,27cyclictest0-21swapper/212:05:132
269332299290,27cyclictest0-21swapper/212:05:132
269332299290,27cyclictest0-21swapper/209:40:042
269332299290,27cyclictest0-21swapper/207:25:112
2693317992926,2cyclictest723-21Xorg10:48:001
269331799291,27cyclictest0-21swapper/109:55:161
2693313992923,4cyclictest2704202-21phc2sys07:40:220
2693325992826,1cyclictest0-21swapper/311:12:593
2693325992825,2cyclictest0-21swapper/310:54:593
269332599280,27cyclictest0-21swapper/311:00:103
2693322992825,2cyclictest0-21swapper/211:31:592
2693322992825,2cyclictest0-21swapper/207:45:132
2693322992824,2cyclictest0-21swapper/209:09:002
2693325992725,1cyclictest0-21swapper/311:39:553
2693325992725,1cyclictest0-21swapper/307:21:593
2693325992724,2cyclictest0-21swapper/307:55:583
2693322992724,2cyclictest0-21swapper/210:28:592
2693322992724,2cyclictest0-21swapper/208:30:592
2693322992724,2cyclictest0-21swapper/207:44:542
269332299270,25cyclictest0-21swapper/212:02:592
269332299270,1cyclictest0-21swapper/209:50:122
2693325992624,1cyclictest0-21swapper/307:15:263
2693325992621,4cyclictest2756207-21turbostat10:25:003
269332599260,2cyclictest2777659-21turbostat11:30:003
2693322992623,2cyclictest0-21swapper/208:51:582
269332299260,19cyclictest0-21swapper/209:55:592
2693325992523,1cyclictest0-21swapper/310:02:583
2693325992523,1cyclictest0-21swapper/309:58:003
2693325992523,1cyclictest0-21swapper/308:40:143
269332599250,24cyclictest0-21swapper/312:28:593
269332599250,24cyclictest0-21swapper/308:49:583
269332599250,1cyclictest0-21swapper/308:37:593
2693322992522,2cyclictest0-21swapper/211:50:132
2693322992522,2cyclictest0-21swapper/211:49:552
2693322992520,2cyclictest0-21swapper/211:10:132
269332299250,2cyclictest0-21swapper/209:19:552
4670791240,3ptp4l441ktimers/311:20:013
4670791230,3ptp4l441ktimers/312:35:003
4670791230,3ptp4l441ktimers/310:09:563
4670791230,3ptp4l441ktimers/309:49:593
2693325992321,1cyclictest0-21swapper/309:30:133
2693325992320,2cyclictest0-21swapper/307:47:593
2693325992317,5cyclictest2743166-21turbostat09:45:003
2693322992320,2cyclictest0-21swapper/212:12:592
2693322992320,2cyclictest0-21swapper/208:08:592
4670791220,3ptp4l441ktimers/310:39:563
2693322992219,2cyclictest0-21swapper/211:55:592
2693322992219,2cyclictest0-21swapper/211:03:592
2693322992118,2cyclictest0-21swapper/210:11:592
202199210,10rtkit-daemon2692667-21ntp_192.168.11407:05:260
2693322991916,2cyclictest0-21swapper/209:23:002
2693322991916,2cyclictest0-21swapper/209:23:002
2693322991916,2cyclictest0-21swapper/209:02:592
2693317991714,2cyclictest2693230-21latency_hist07:10:021
2693322991613,2cyclictest0-21swapper/211:41:592
2693322991411,2cyclictest0-21swapper/208:13:002
2693322991310,2cyclictest0-21swapper/210:30:582
269332299118,2cyclictest0-21swapper/208:58:122
269332299107,2cyclictest0-21swapper/210:44:192
26933229996,2cyclictest0-21swapper/212:23:422
26933229930,2cyclictest0-21swapper/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional