You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 04:37
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Fri Nov 28, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4040559910893,11cyclictest271-21systemd-journal00:00:000
404059999486,6cyclictest451976-21apt-get21:40:001
404055999268,21cyclictest497592-21/usr/sbin/munin00:00:260
404059999181,7cyclictest492294-21latency_hist23:45:021
404055998982,5cyclictest437669-21kworker/u8:3+events_unbound21:00:010
404059998872,14cyclictest468388-21kworker/u8:3+flush-179:023:20:261
404055998572,10cyclictest271-21systemd-journal20:20:000
404055998372,8cyclictest416941-21latency_hist19:50:000
404059998276,4cyclictest421759-21kworker/u8:2+flush-179:020:40:251
404060997945,32cyclictest0-21swapper/200:27:592
404059997968,8cyclictest474777-21latency_hist22:50:011
404059997265,6cyclictest416976-21kworker/u8:1+flush-179:020:00:001
404059997058,9cyclictest431267-21latency_hist20:35:011
404059997054,13cyclictest420547-21kworker/u8:0+flush-179:020:05:211
404059996860,6cyclictest500559-21kworker/u8:0+flush-179:000:10:211
404055996860,5cyclictest418528-21latency_hist19:55:010
404055996857,8cyclictest434462-21latency_hist20:45:010
40405599680,66cyclictest1-21systemd00:40:010
404059996762,4cyclictest468388-21kworker/u8:3+flush-179:022:54:561
404059996660,4cyclictest421759-21kworker/u8:2+flush-179:021:05:251
404055996656,7cyclictest468388-21kworker/u8:3+flush-179:022:35:240
404059996559,4cyclictest487559-21kworker/u8:3+events_unbound00:20:251
404059996559,4cyclictest437669-21kworker/u8:3+events_unbound21:20:151
404055996554,8cyclictest271-21systemd-journal20:35:000
404059996457,6cyclictest437669-21kworker/u8:3+flush-179:021:20:001
404059996358,4cyclictest490742-21kworker/u8:0+flush-179:023:54:551
404059996356,5cyclictest389719-21kworker/u8:2+flush-179:019:15:211
404059996353,7cyclictest460455-21latency_hist22:05:011
404059996353,6cyclictest465896-21kworker/u8:0+flush-179:022:30:161
404060996231,2cyclictest0-21swapper/222:02:592
404059996256,4cyclictest468387-21kworker/u8:1+events_unbound22:55:201
404059996256,4cyclictest407807-21kworker/u8:0+flush-179:019:20:261
404059996248,9cyclictest271-21systemd-journal21:45:011
404055996257,4cyclictest487559-21kworker/u8:3+flush-179:000:05:240
404059996156,4cyclictest487559-21kworker/u8:3+flush-179:000:15:261
404059996155,5cyclictest485848-21kworker/u8:2+events_unbound23:30:221
404055996154,5cyclictest420547-21kworker/u8:0+flush-179:020:25:180
404055996151,7cyclictest473193-21latency_hist22:45:010
404059996049,9cyclictest467076-21apt-get22:25:121
404059996049,8cyclictest497353-40mandb00:00:001
404055996053,4cyclictest484321-21latency_hist23:20:020
404059995953,4cyclictest479537-21kworker/u8:1+flush-179:023:05:231
404055995950,6cyclictest411903-21latency_hist19:35:010
404059995853,4cyclictest496765-21kworker/u8:1+flush-179:000:00:361
404059995851,5cyclictest389719-21kworker/u8:2+events_unbound19:15:011
404059995849,6cyclictest494138-21latency_hist23:50:021
404059995848,7cyclictest450374-21latency_hist21:35:011
404059995752,4cyclictest456978-21kworker/u8:1+flush-179:022:10:221
404059995750,5cyclictest437669-21kworker/u8:3+events_unbound21:10:261
404059995649,4cyclictest473183-21kworker/u8:0+events_unbound23:00:211
404059995649,4cyclictest473183-21kworker/u8:0+events_unbound23:00:201
404059995647,6cyclictest465896-21kworker/u8:0+flush-179:022:20:211
404055995648,5cyclictest476363-21latency_hist22:55:000
404055995647,6cyclictest469997-21latency_hist22:35:000
404059995550,4cyclictest420547-21kworker/u8:0+flush-179:020:10:171
404059995550,4cyclictest420547-21kworker/u8:0+flush-179:020:10:171
404059995548,6cyclictest479537-21kworker/u8:1+flush-179:023:15:151
404059995548,4cyclictest458849-21latency_hist22:00:011
404059995544,8cyclictest415594-21/usr/sbin/munin19:45:221
404055995544,8cyclictest440831-21latency_hist21:05:000
404059995448,4cyclictest420547-21kworker/u8:0+flush-179:020:25:001
404055995449,3cyclictest456978-21kworker/u8:1+events_unbound22:20:140
404055995347,4cyclictest487559-21kworker/u8:3+flush-179:000:15:200
404059995248,3cyclictest500559-21kworker/u8:0+flush-179:000:09:551
404059995247,4cyclictest452109-21kworker/u8:0+flush-179:021:50:011
404059995246,4cyclictest500559-21kworker/u8:0+flush-179:000:25:231
404055995243,7cyclictest442959-21diskstats21:10:160
404059995149,1cyclictest0-21swapper/120:36:591
404059995039,9cyclictest508788-21apt-get00:35:111
404055995041,7cyclictest271-21systemd-journal23:39:590
404055995041,7cyclictest271-21systemd-journal23:00:010
404055995039,9cyclictest271-21systemd-journal19:15:000
404059994943,5cyclictest420547-21kworker/u8:0+events_unbound20:05:011
404059994941,5cyclictest436062-21basename20:50:001
404055994944,4cyclictest420547-21kworker/u8:0+flush-179:020:40:010
404055994944,4cyclictest389719-21kworker/u8:2+flush-179:019:15:170
404055994944,3cyclictest421759-21kworker/u8:2+flush-179:020:45:170
404055994940,7cyclictest456999-21latency_hist21:55:020
404059994843,4cyclictest485848-21kworker/u8:2+flush-179:023:25:231
404059994843,4cyclictest455391-21kworker/u8:2+flush-179:021:50:251
404055994842,5cyclictest452109-21kworker/u8:0+events_unbound21:40:240
404055994840,7cyclictest408765-21kworker/u8:3+flush-179:019:25:210
404055994839,7cyclictest468388-21kworker/u8:3+flush-179:023:10:210
404055994837,9cyclictest478512-21diskstats23:00:160
404055994837,9cyclictest478512-21diskstats23:00:150
404059994741,4cyclictest485848-21kworker/u8:2+flush-179:023:35:151
4670791460,13ptp4l61690irq/136-eno1-TxRx-019:10:003
404059994640,4cyclictest402455-21kworker/u8:1+events_unbound19:35:141
40405999461,43cyclictest418530-21munin-run19:55:001
404055994541,3cyclictest490742-21kworker/u8:0+flush-179:023:40:250
404055994440,3cyclictest407807-21kworker/u8:0+flush-179:019:20:270
404059994337,4cyclictest408765-21kworker/u8:3+flush-179:019:30:191
404059994334,7cyclictest408765-21kworker/u8:3+events_unbound19:40:191
404059994334,7cyclictest408765-21kworker/u8:3+events_unbound19:40:191
404059994330,10cyclictest471603-21basename22:40:001
404059994328,12cyclictest420547-21kworker/u8:0+flush-179:020:25:261
404055994334,6cyclictest502-21rs:main0
404055994333,7cyclictest508533-21latency_hist00:35:010
40405999421,3cyclictest473203-21ls22:44:591
404055994238,3cyclictest455391-21kworker/u8:2+events_unbound21:50:010
404059994136,3cyclictest437669-21kworker/u8:3+flush-179:021:25:231
404059994132,7cyclictest408985-21apt-get19:25:121
404055994135,5cyclictest479537-21kworker/u8:1+flush-179:023:30:140
404055994134,5cyclictest421759-21kworker/u8:2+flush-179:021:05:230
404055994034,4cyclictest452109-21kworker/u8:0+flush-179:021:55:260
404059993931,6cyclictest426504-21latency_hist20:20:021
404055993936,2cyclictest723-21Xorg22:20:000
404060993835,2cyclictest0-21swapper/222:49:592
404060993834,2cyclictest0-21swapper/200:23:592
404059993835,2cyclictest0-21swapper/100:30:591
404059993834,3cyclictest420547-21kworker/u8:0+events_unbound20:50:161
404055993832,5cyclictest494376-21/usr/sbin/munin23:50:200
404055993828,8cyclictest462296-21apt-get22:10:110
404062993734,2cyclictest0-21swapper/320:29:593
404060993734,2cyclictest0-21swapper/220:38:592
404060993734,2cyclictest0-21swapper/220:19:542
404060993733,2cyclictest0-21swapper/220:57:592
40406099370,1cyclictest0-21swapper/221:39:552
404059993731,4cyclictest459104-21kworker/u8:2+flush-179:022:15:181
404059993731,4cyclictest439240-21turbostat21:04:551
404055993732,4cyclictest389719-21kworker/u8:2+events_unbound19:40:130
404055993732,4cyclictest389719-21kworker/u8:2+events_unbound19:40:130
404055993729,6cyclictest421759-21kworker/u8:2+events_unbound20:14:590
404055993729,6cyclictest421759-21kworker/u8:2+events_unbound20:14:590
404055993726,9cyclictest412704-21diskstats19:35:160
404062993631,4cyclictest451992-21turbostat21:44:553
404062993630,5cyclictest479564-21turbostat23:05:003
404062993630,5cyclictest479564-21turbostat23:04:593
40406299360,35cyclictest0-21swapper/322:39:563
40406299360,35cyclictest0-21swapper/322:28:593
40406299360,2cyclictest0-21swapper/300:30:003
404060993633,2cyclictest0-21swapper/221:22:592
404055993632,3cyclictest479537-21kworker/u8:1+flush-179:023:05:260
404055993631,4cyclictest408765-21kworker/u8:3+flush-179:020:00:180
404062993533,1cyclictest0-21swapper/323:54:563
404062993533,1cyclictest0-21swapper/320:14:553
404062993533,1cyclictest0-21swapper/320:14:553
404060993532,2cyclictest0-21swapper/222:44:552
404060993532,2cyclictest0-21swapper/220:44:552
40406099350,1cyclictest0-21swapper/220:04:592
404059993530,4cyclictest459104-21kworker/u8:2+events_unbound22:05:251
404059993526,7cyclictest439247-21latency_hist21:00:011
404055993530,4cyclictest437669-21kworker/u8:3+flush-179:020:55:000
404062993432,1cyclictest0-21swapper/320:48:593
404062993432,1cyclictest0-21swapper/320:31:593
404062993432,1cyclictest0-21swapper/320:08:583
404062993428,5cyclictest468427-21turbostat22:34:553
404060993431,2cyclictest0-21swapper/223:37:592
404060993431,2cyclictest0-21swapper/219:33:592
404055993431,2cyclictest723-21Xorg23:23:580
404055993431,2cyclictest468407-21cron22:30:000
404055993429,4cyclictest447181-21kworker/u8:1+flush-179:021:30:010
404060993330,2cyclictest0-21swapper/220:24:552
40406099330,30cyclictest0-21swapper/221:34:002
404055993328,4cyclictest500559-21kworker/u8:0+flush-179:000:10:230
404055993328,4cyclictest485935-21turbostat23:29:560
404055993327,5cyclictest456978-21kworker/u8:1+events_unbound22:00:180
40405599330,32cyclictest492309-21apt-get23:45:030
404062993230,1cyclictest0-21swapper/323:39:583
404062993230,1cyclictest0-21swapper/321:49:553
404062993229,2cyclictest0-21swapper/321:24:593
404062993229,2cyclictest0-21swapper/319:55:593
404062993229,2cyclictest0-21swapper/300:16:593
40406299320,31cyclictest0-21swapper/322:56:593
404060993229,2cyclictest0-21swapper/223:24:592
40406099320,2cyclictest0-21swapper/219:37:592
404055993227,3cyclictest445382-21timerwakeupswit21:15:240
404055993226,4cyclictest421759-21kworker/u8:2+events_unbound20:20:280
404062993127,3cyclictest490716-21turbostat23:44:553
404062993125,5cyclictest481153-21turbostat23:09:593
404060993129,1cyclictest0-21swapper/200:40:002
404060993128,2cyclictest0-21swapper/223:59:562
40405999310,3cyclictest723-21Xorg23:15:001
404055993127,3cyclictest505297-21unixbench_singl00:20:260
404055993126,4cyclictest447181-21kworker/u8:1+flush-179:021:30:210
4670791300,4ptp4l441ktimers/323:50:003
404062993028,1cyclictest0-21swapper/323:59:553
40406299300,28cyclictest0-21swapper/300:30:173
404060993028,1cyclictest0-21swapper/223:49:582
404060993027,2cyclictest0-21swapper/221:14:552
404060993027,2cyclictest0-21swapper/200:10:132
40406099300,28cyclictest0-21swapper/221:40:012
40406099300,28cyclictest0-21swapper/220:30:142
40406099300,27cyclictest0-21swapper/222:10:142
404055993026,3cyclictest474814-21systemd-run22:50:000
404055993024,5cyclictest448821-21kworker/u8:2+flush-179:021:35:170
404062992927,1cyclictest0-21swapper/321:30:143
40406299290,28cyclictest0-21swapper/323:30:133
40406299290,28cyclictest0-21swapper/322:05:173
40406299290,28cyclictest0-21swapper/319:10:143
40406299290,27cyclictest0-21swapper/322:00:133
40406299290,27cyclictest0-21swapper/320:35:133
40406299290,27cyclictest0-21swapper/319:40:123
40406299290,27cyclictest0-21swapper/319:40:123
40406299290,1cyclictest0-21swapper/321:35:153
40406299290,1cyclictest0-21swapper/319:30:123
404060992926,2cyclictest0-21swapper/223:00:132
404060992926,2cyclictest0-21swapper/223:00:132
404060992926,2cyclictest0-21swapper/221:15:592
404060992926,2cyclictest0-21swapper/220:05:142
40406099290,27cyclictest0-21swapper/223:15:142
40406099290,27cyclictest0-21swapper/221:45:142
40406099290,27cyclictest0-21swapper/221:05:152
40406099290,27cyclictest0-21swapper/219:55:132
40406099290,27cyclictest0-21swapper/219:40:142
40406099290,27cyclictest0-21swapper/219:40:132
40406099290,1cyclictest0-21swapper/223:30:122
404055992928,1cyclictest0-21swapper/020:05:590
404055992924,4cyclictest447181-21kworker/u8:1+flush-179:021:24:560
404055992923,5cyclictest500559-21kworker/u8:0+events_unbound00:25:240
404062992826,1cyclictest0-21swapper/323:13:003
404062992826,1cyclictest0-21swapper/320:00:593
40406299280,27cyclictest0-21swapper/321:05:163
40406299280,26cyclictest0-21swapper/322:45:143
40406299280,26cyclictest0-21swapper/319:35:013
40406299280,1cyclictest0-21swapper/322:10:133
404060992825,2cyclictest0-21swapper/221:25:172
40406099280,26cyclictest0-21swapper/200:30:182
404062992725,1cyclictest0-21swapper/323:15:143
404062992725,1cyclictest0-21swapper/322:40:143
404062992725,1cyclictest0-21swapper/300:35:133
404062992725,1cyclictest0-21swapper/300:10:143
40406299270,26cyclictest0-21swapper/320:17:593
404060992725,1cyclictest0-21swapper/219:19:592
404060992724,2cyclictest0-21swapper/223:50:582
404060992724,2cyclictest0-21swapper/220:53:592
40406099270,25cyclictest0-21swapper/200:00:592
404055992724,2cyclictest723-21Xorg19:55:590
404062992624,1cyclictest0-21swapper/300:09:003
404062992623,2cyclictest0-21swapper/320:52:003
40406299260,25cyclictest0-21swapper/320:21:473
40406299260,24cyclictest0-21swapper/319:45:573
404060992624,1cyclictest0-21swapper/223:14:462
404060992624,1cyclictest0-21swapper/222:39:582
404060992624,1cyclictest0-21swapper/220:15:002
404060992624,1cyclictest0-21swapper/220:14:592
404060992623,2cyclictest0-21swapper/222:50:592
404060992623,2cyclictest0-21swapper/220:28:362
404060992623,2cyclictest0-21swapper/219:28:002
404060992623,2cyclictest0-21swapper/219:10:172
40406099262,1cyclictest0-21swapper/221:56:352
40406099260,24cyclictest0-21swapper/223:07:252
40406099260,24cyclictest0-21swapper/221:04:152
40406099260,24cyclictest0-21swapper/219:50:052
40406099260,24cyclictest0-21swapper/219:45:422
40406099260,24cyclictest0-21swapper/219:20:232
40406099260,1cyclictest0-21swapper/200:19:592
4670791250,4ptp4l61690irq/136-eno1-TxRx-022:20:003
404062992523,1cyclictest0-21swapper/319:15:153
40406299250,24cyclictest0-21swapper/321:27:593
40406299250,24cyclictest0-21swapper/320:42:583
40406299250,24cyclictest0-21swapper/319:51:533
40406299250,1cyclictest0-21swapper/321:10:143
404060992522,2cyclictest0-21swapper/222:24:222
404060992522,1cyclictest0-21swapper/222:35:012
4670791240,4ptp4l61690irq/136-eno1-TxRx-021:00:013
4670791240,4ptp4l441ktimers/321:55:013
4670791240,3ptp4l441ktimers/323:25:013
404062992422,1cyclictest0-21swapper/319:25:373
404062992422,1cyclictest0-21swapper/300:23:593
40406299240,1cyclictest0-21swapper/321:58:583
404060992421,2cyclictest0-21swapper/222:58:592
404060992421,2cyclictest0-21swapper/222:05:592
40406099240,1cyclictest0-21swapper/220:47:502
4670791230,4ptp4l61690irq/136-eno1-TxRx-023:24:553
4670791230,3ptp4l61690irq/136-eno1-TxRx-022:24:553
4670791230,3ptp4l61690irq/136-eno1-TxRx-000:04:553
404062992319,2cyclictest0-21swapper/319:20:203
404060992320,2cyclictest0-21swapper/200:07:582
4670791220,3ptp4l441ktimers/321:04:563
4670791210,2ptp4l441ktimers/322:54:553
4670791210,2ptp4l441ktimers/321:19:553
404060992119,1cyclictest0-21swapper/221:52:202
404060992118,2cyclictest0-21swapper/223:43:002
202199210,12rtkit-daemon485-21gpm19:05:550
404060992017,2cyclictest0-21swapper/222:18:592
404059992014,4cyclictest0-21swapper/119:10:011
404060991816,1cyclictest0-21swapper/223:30:002
404060991613,2cyclictest351rcuc/222:25:302
4040609953,1cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional