You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 22:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6.osadl.org (updated Sun Nov 23, 2025 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
373375998983,5cyclictest381276-21kworker/u8:3+flush-179:007:40:011
373373998776,8cyclictest502-21rs:main0
373373998676,7cyclictest384506-21latency_hist07:45:010
373373998476,6cyclictest418686-21latency_hist09:30:010
373375998176,3cyclictest441728-21kworker/u8:0+flush-179:010:55:001
373375998174,5cyclictest359298-21kworker/u8:1+events_unbound07:25:231
373375997969,7cyclictest441484-21latency_hist10:40:001
373373997765,8cyclictest400682-21apt-get08:35:000
373375997670,4cyclictest428278-21kworker/u8:3+flush-179:010:25:261
373375997666,8cyclictest430071-21/usr/sbin/munin10:05:221
373375997666,7cyclictest462168-21latency_hist11:45:011
373375997643,30cyclictest0-21swapper/108:12:001
373375997467,5cyclictest372023-21kworker/u8:2+flush-179:007:10:171
373375997365,6cyclictest449486-21kworker/u8:2+events_unbound11:15:261
373375997364,6cyclictest372023-21kworker/u8:2+flush-179:007:30:211
373375997268,3cyclictest462215-21kworker/u8:2+flush-179:012:00:241
373373997262,7cyclictest454215-21latency_hist11:20:010
373375997164,5cyclictest390338-21kworker/u8:2+events_unbound08:20:001
373375997157,10cyclictest400666-21latency_hist08:35:001
373375997061,7cyclictest414147-21/usr/sbin/munin09:15:251
373375996961,6cyclictest439611-21kworker/u8:2+flush-179:010:45:161
373375996961,6cyclictest387912-21/usr/sbin/munin07:55:261
373375996961,5cyclictest405993-21kworker/u8:3+flush-179:009:05:211
373375996860,6cyclictest405993-21kworker/u8:3+flush-179:009:00:011
373375996859,7cyclictest406200-21/usr/sbin/munin08:50:221
373373996858,7cyclictest415506-21latency_hist09:20:000
373373996857,9cyclictest379648-21apt-get07:26:430
37338099660,63cyclictest0-21swapper/311:56:593
373375996655,9cyclictest470378-21apt-get12:10:141
373375996559,4cyclictest446086-21kworker/u8:1+flush-179:010:55:231
373375996559,4cyclictest405993-21kworker/u8:3+flush-179:009:00:161
373373996563,1cyclictest0-21swapper/009:08:590
373373996458,4cyclictest452452-21kworker/u8:0+flush-179:011:35:260
373373996359,2cyclictest723-21Xorg10:07:590
373373996354,7cyclictest468770-21/usr/sbin/munin12:05:240
373375996251,8cyclictest476998-21apt-get12:30:141
373373996253,7cyclictest405959-21latency_hist08:50:010
373373996253,7cyclictest405959-21latency_hist08:50:010
373375996155,4cyclictest384348-21kworker/u8:1+flush-179:007:45:221
373373996151,7cyclictest413907-21latency_hist09:15:010
373375996055,4cyclictest427305-21kworker/u8:1+events_unbound10:10:241
373375996053,5cyclictest390338-21kworker/u8:2+flush-179:008:20:231
373375996052,6cyclictest449486-21kworker/u8:2+events_unbound11:20:241
373373996056,3cyclictest723-21Xorg11:56:590
373373996053,5cyclictest395649-21latency_hist08:20:010
373373996048,8cyclictest471702-21latency_hist12:15:010
373375995954,4cyclictest462198-21kworker/u8:1+flush-179:012:05:221
373375995953,4cyclictest418721-21kworker/u8:2+flush-179:010:00:151
373375995947,9cyclictest271-21systemd-journal12:00:001
373375995853,4cyclictest386070-21kworker/u8:0+flush-179:007:54:551
373375995850,5cyclictest386070-21kworker/u8:0+flush-179:009:25:201
373373995848,7cyclictest375104-21/usr/sbin/munin07:15:240
373373995848,6cyclictest402585-21kworker/u8:1+flush-179:008:40:160
373373995824,26cyclictest478312-21anacron12:31:440
373375995751,5cyclictest456044-21kworker/u8:3+flush-179:011:25:181
373375995751,4cyclictest412366-21kworker/u8:2+flush-179:009:10:251
373375995749,6cyclictest418721-21kworker/u8:2+flush-179:009:35:201
373375995748,7cyclictest428278-21kworker/u8:3+events_unbound10:40:251
373373995647,7cyclictest468533-21latency_hist12:05:010
373373995645,8cyclictest271-21systemd-journal08:05:010
373373995549,4cyclictest397232-21latency_hist08:25:020
373373995444,8cyclictest475155-21latency_hist12:25:000
373373995444,8cyclictest456063-21apt-get11:25:110
37337399541,43cyclictest433014-21grep10:15:000
373375995349,3cyclictest462198-21kworker/u8:1+flush-179:012:25:201
373375995346,6cyclictest402590-21kworker/u8:2+flush-179:008:45:261
373375995346,6cyclictest402590-21kworker/u8:2+flush-179:008:45:251
373373995346,4cyclictest372023-21kworker/u8:2+flush-179:007:15:000
373380995250,1cyclictest0-21swapper/307:16:003
373375995249,2cyclictest0-21swapper/111:31:591
373375995245,5cyclictest403049-21apt-get08:40:121
373375995245,5cyclictest271-21systemd-journal12:15:121
373373995230,19cyclictest271-21systemd-journal10:57:110
373378995148,2cyclictest0-21swapper/209:09:002
373375995143,7cyclictest462198-21kworker/u8:1+flush-179:012:40:011
373375995143,7cyclictest462198-21kworker/u8:1+flush-179:012:40:001
373375995140,10cyclictest462198-21kworker/u8:1+flush-179:011:45:261
373375995039,8cyclictest399077-21latency_hist08:30:011
373375995039,8cyclictest399077-21latency_hist08:30:011
37337599500,49cyclictest1-21systemd08:10:001
373373995043,6cyclictest475144-21kworker/u8:2+flush-179:012:35:170
373373995043,6cyclictest475144-21kworker/u8:2+flush-179:012:35:160
373373995038,8cyclictest437790-21apt-get10:30:000
37337399500,2cyclictest0-21swapper/009:55:140
37337399491,45cyclictest443092-21awk10:45:000
37337399490,45cyclictest0-21swapper/008:25:040
37337399490,45cyclictest0-21swapper/008:25:030
373373994840,6cyclictest386088-21latency_hist07:50:000
4670791470,13ptp4l441ktimers/307:10:013
373378994744,2cyclictest0-21swapper/210:07:592
373375994744,2cyclictest428278-21kworker/u8:3+events_unbound10:30:141
373375994742,3cyclictest418721-21kworker/u8:2+flush-179:009:45:251
373375994739,6cyclictest271-21systemd-journal09:54:101
373375994642,3cyclictest405993-21kworker/u8:3+flush-179:009:20:211
373375994641,4cyclictest386070-21kworker/u8:0+flush-179:008:39:551
373375994641,4cyclictest359298-21kworker/u8:1+events_unbound07:20:211
373375994640,4cyclictest456044-21kworker/u8:3+flush-179:011:50:211
373375994640,4cyclictest456044-21kworker/u8:3+flush-179:011:50:211
373375994639,5cyclictest387710-21kworker/u8:3+events_unbound08:00:161
373373994642,2cyclictest0-21swapper/011:10:590
373373994634,9cyclictest381545-21apt-get07:35:110
37337399460,1cyclictest0-21swapper/011:50:140
37337399460,1cyclictest0-21swapper/011:50:140
373380994537,7cyclictest0-21swapper/309:08:003
373375994540,4cyclictest452452-21kworker/u8:0+flush-179:011:35:221
373373994541,2cyclictest723-21Xorg10:37:590
373373994540,3cyclictest386070-21kworker/u8:0+flush-179:007:55:260
373373994537,5cyclictest394036-21latency_hist08:15:010
373373994535,8cyclictest457648-21apt-get11:30:120
373373994535,8cyclictest386070-21kworker/u8:0+flush-179:009:40:000
37337399450,30cyclictest428483-21/usr/sbin/munin10:00:180
37337399450,1cyclictest0-21swapper/009:46:580
373378994441,2cyclictest0-21swapper/211:56:592
373375994437,5cyclictest449486-21kworker/u8:2+flush-179:011:05:171
373373994439,2cyclictest0-21swapper/011:40:140
373373994428,15cyclictest723-21Xorg09:00:590
373373994423,19cyclictest407630-21latency_hist08:55:000
37337399440,40cyclictest418946-21apt-get09:30:130
373378994340,2cyclictest0-21swapper/210:31:592
373373994334,7cyclictest455805-21sh11:25:000
373373994334,7cyclictest425290-21/usr/sbin/munin09:50:280
373375994135,5cyclictest418721-21kworker/u8:2+flush-179:010:15:191
373380994034,4cyclictest447852-21turbostat11:00:003
373380994033,5cyclictest410753-21turbostat09:05:013
373380994033,5cyclictest390879-21turbostat08:05:013
37338099400,39cyclictest0-21swapper/309:53:593
373375994035,3cyclictest449486-21kworker/u8:2+flush-179:011:10:221
373373994036,3cyclictest439699-21apt-key10:35:010
37337399400,26cyclictest443597-21expr10:45:130
373380993931,7cyclictest0-21swapper/310:15:593
37338099390,37cyclictest479937-21turbostat12:40:013
37338099390,37cyclictest479937-21turbostat12:40:003
373375993935,3cyclictest462198-21kworker/u8:1+flush-179:012:25:001
373375993933,5cyclictest418721-21kworker/u8:2+flush-179:010:00:001
373373993935,2cyclictest421868-21turbostat09:44:560
37337399392,31cyclictest433257-21apt-get10:15:170
37337399390,37cyclictest446253-21dump-pmu-power10:55:000
37337399390,2cyclictest0-21swapper/008:56:590
373380993832,4cyclictest475167-21turbostat12:24:593
373380993832,4cyclictest465342-21turbostat11:55:003
373380993832,4cyclictest465342-21turbostat11:54:593
373380993811,1cyclictest0-21swapper/309:21:003
373378993835,2cyclictest0-21swapper/208:07:592
373375993834,3cyclictest433029-21kworker/u8:0+events_unbound10:20:271
373375993834,2cyclictest723-21Xorg09:40:591
373373993832,5cyclictest434598-21turbostat10:24:550
373380993734,2cyclictest0-21swapper/310:06:593
373380993732,4cyclictest386103-21turbostat07:49:593
373378993734,2cyclictest0-21swapper/211:41:592
373378993734,2cyclictest0-21swapper/208:58:592
37337899370,35cyclictest0-21swapper/207:24:592
37337899370,29cyclictest0-21swapper/212:20:592
373375993732,4cyclictest446284-21kworker/u8:3+events_unbound11:00:211
37337399370,1cyclictest0-21swapper/011:47:590
373380993634,1cyclictest0-21swapper/312:09:553
373380993634,1cyclictest0-21swapper/310:47:593
373380993634,1cyclictest0-21swapper/310:28:583
373380993634,1cyclictest0-21swapper/307:44:563
373380993633,2cyclictest0-21swapper/308:41:593
373380993633,2cyclictest0-21swapper/307:50:583
373380993630,4cyclictest399087-21turbostat08:30:003
373380993630,4cyclictest399087-21turbostat08:29:593
37338099360,35cyclictest0-21swapper/308:37:593
37338099360,1cyclictest0-21swapper/308:24:563
37338099360,1cyclictest0-21swapper/307:58:003
37337899360,1cyclictest0-21swapper/211:20:592
373373993619,16cyclictest0-21swapper/012:25:210
373380993528,5cyclictest407549-21turbostat08:54:593
373378993532,2cyclictest0-21swapper/209:44:552
37337899350,1cyclictest0-21swapper/210:54:592
373375993530,4cyclictest359298-21kworker/u8:1+flush-179:007:15:271
373373993531,3cyclictest449444-21cron11:05:000
373380993432,1cyclictest0-21swapper/309:59:563
373378993431,2cyclictest0-21swapper/212:05:592
373378993431,2cyclictest0-21swapper/209:50:592
373373993430,2cyclictest417187-21latency_hist09:25:010
37337399341,30cyclictest0-21swapper/007:20:120
373380993331,1cyclictest0-21swapper/309:10:593
373380993329,3cyclictest473557-21turbostat12:20:003
373380993327,4cyclictest478337-21turbostat12:35:013
373380993326,5cyclictest451028-21turbostat11:10:003
373380993326,5cyclictest449445-21turbostat11:05:003
373378993330,2cyclictest0-21swapper/212:38:002
373378993330,2cyclictest0-21swapper/212:38:002
373378993330,2cyclictest0-21swapper/211:04:592
373378993330,2cyclictest0-21swapper/207:45:002
373378993329,2cyclictest0-21swapper/210:10:592
373373993325,6cyclictest373252-21latency_hist07:10:020
373380993230,1cyclictest0-21swapper/312:29:553
373380993230,1cyclictest0-21swapper/311:24:553
37338099320,31cyclictest0-21swapper/308:08:583
373378993230,1cyclictest0-21swapper/210:45:012
373378993229,2cyclictest0-21swapper/208:24:002
37337399322,26cyclictest690-21snmpd07:30:110
373380993129,1cyclictest0-21swapper/307:35:593
373380993128,2cyclictest0-21swapper/310:39:553
373380993125,4cyclictest409154-21turbostat08:59:593
373380993118,11cyclictest381313-21turbostat07:35:003
373378993129,1cyclictest0-21swapper/210:25:002
373378993129,1cyclictest0-21swapper/207:30:012
373378993128,2cyclictest0-21swapper/211:13:592
373373993127,3cyclictest401620-21hwlatdetect08:35:160
373373993114,15cyclictest0-21swapper/007:53:500
37337399310,26cyclictest723-21Xorg12:18:590
373380993027,2cyclictest0-21swapper/310:43:583
373380993027,2cyclictest0-21swapper/309:25:593
37338099300,21cyclictest0-21swapper/310:10:153
37338099300,1cyclictest0-21swapper/310:24:593
373378993027,2cyclictest0-21swapper/212:01:592
373378993027,2cyclictest0-21swapper/208:19:582
373378993026,3cyclictest0-21swapper/211:50:142
373378993026,3cyclictest0-21swapper/211:50:132
373378993026,2cyclictest0-21swapper/210:37:592
373378993025,2cyclictest0-21swapper/208:25:042
373378993025,2cyclictest0-21swapper/208:25:032
373375993024,4cyclictest384543-21systemd-run07:44:591
37337399301,26cyclictest0-21swapper/011:07:590
373380992926,2cyclictest0-21swapper/311:25:593
373380992924,4cyclictest452621-21turbostat11:15:003
37338099290,28cyclictest0-21swapper/309:30:133
37338099290,27cyclictest0-21swapper/310:50:143
37338099290,26cyclictest0-21swapper/311:30:133
373378992927,1cyclictest0-21swapper/208:13:592
373378992926,2cyclictest0-21swapper/211:30:132
373378992926,2cyclictest0-21swapper/209:38:592
37337899290,27cyclictest0-21swapper/212:10:142
37337899290,27cyclictest0-21swapper/211:45:132
37337899290,27cyclictest0-21swapper/208:40:032
37337899290,27cyclictest0-21swapper/207:10:142
37337899290,1cyclictest0-21swapper/209:55:132
373380992821,5cyclictest460585-21turbostat11:40:003
373380992821,5cyclictest415508-21turbostat09:19:593
37338099280,26cyclictest0-21swapper/311:40:143
37338099280,26cyclictest0-21swapper/307:25:123
373378992826,1cyclictest0-21swapper/209:03:582
373378992825,2cyclictest0-21swapper/210:15:152
373378992825,2cyclictest0-21swapper/210:04:552
373378992825,2cyclictest0-21swapper/209:30:582
373378992825,2cyclictest0-21swapper/208:00:132
373378992824,2cyclictest0-21swapper/207:15:592
37337899280,2cyclictest0-21swapper/208:35:032
37337899280,26cyclictest0-21swapper/212:30:162
37337899280,26cyclictest0-21swapper/211:25:122
373375992823,4cyclictest405993-21kworker/u8:3+flush-179:009:30:281
373380992719,6cyclictest468542-21turbostat12:05:003
37338099270,26cyclictest0-21swapper/308:45:153
37338099270,26cyclictest0-21swapper/308:45:153
37338099270,24cyclictest0-21swapper/307:20:223
373378992725,1cyclictest351rcuc/209:46:592
373378992724,2cyclictest0-21swapper/211:36:592
37337899270,26cyclictest0-21swapper/209:28:002
37337899270,25cyclictest0-21swapper/207:38:342
373380992624,1cyclictest0-21swapper/309:35:133
373380992618,6cyclictest0-21swapper/309:45:583
373378992622,2cyclictest0-21swapper/210:57:582
37337899260,24cyclictest0-21swapper/207:46:162
37337899260,24cyclictest0-21swapper/207:31:012
4670791250,4ptp4l441ktimers/308:35:013
4670791250,4ptp4l441ktimers/308:20:013
4670791250,4ptp4l441ktimers/307:14:553
373380992522,2cyclictest0-21swapper/312:13:593
373380992519,5cyclictest0-21swapper/311:48:593
37338099250,2cyclictest0-21swapper/311:15:593
373378992523,1cyclictest0-21swapper/212:27:592
373378992522,2cyclictest0-21swapper/209:18:582
373378992522,2cyclictest0-21swapper/208:45:582
373378992522,2cyclictest0-21swapper/208:45:582
373378992522,2cyclictest0-21swapper/207:51:592
4670791240,4ptp4l441ktimers/310:34:563
4670791240,4ptp4l441ktimers/308:14:553
373378992422,1cyclictest0-21swapper/209:14:002
373378992422,1cyclictest0-21swapper/207:58:222
373378992421,2cyclictest0-21swapper/212:16:592
373378992421,2cyclictest0-21swapper/210:29:552
373378992421,2cyclictest0-21swapper/208:52:002
4670791230,4ptp4l61690irq/136-eno1-TxRx-010:04:553
373380992321,1cyclictest0-21swapper/309:43:003
373378992017,2cyclictest0-21swapper/209:24:592
373378991916,2cyclictest0-21swapper/210:47:592
37337599190,17cyclictest0-21swapper/107:10:021
373378991815,2cyclictest0-21swapper/208:33:582
373378991614,1cyclictest0-21swapper/211:08:592
373378991310,2cyclictest0-21swapper/211:15:592
3733789974,2cyclictest361ktimers/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional