You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 03:22
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot6.osadl.org (updated Mon Dec 29, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5287339910382,6cyclictest598704-21kworker/u8:1+events_unbound23:00:020
528736999873,21cyclictest614635-21latency_hist23:35:021
528733998979,7cyclictest590443-21latency_hist22:20:010
528736998881,5cyclictest575832-21/usr/sbin/munin21:35:241
528733998578,5cyclictest608191-21kworker/u8:0+flush-179:023:35:020
528733998576,7cyclictest538232-21kworker/u8:1+events_unbound20:10:000
528733998576,7cyclictest538232-21kworker/u8:1+events_unbound20:09:590
52873399850,84cyclictest1-21systemd20:50:010
528733998479,4cyclictest580617-21kworker/u8:2+flush-179:022:00:000
528733998478,5cyclictest536578-21kworker/u8:2+flush-179:019:35:000
528733998272,7cyclictest531851-21latency_hist19:20:010
528733998272,7cyclictest531851-21latency_hist19:20:010
528736998171,7cyclictest588846-21latency_hist22:15:021
528733998173,6cyclictest608191-21kworker/u8:0+flush-179:000:35:020
528733998172,6cyclictest528606-21kworker/u8:0+flush-179:019:10:210
528733997972,6cyclictest553203-21kworker/u8:3+events_unbound20:40:000
528733997972,5cyclictest598707-21kworker/u8:2+events_unbound22:50:120
528733997963,12cyclictest553062-21latency_hist20:25:010
528733997869,7cyclictest555965-21kworker/u8:0+flush-179:021:15:010
528733997668,6cyclictest608191-21kworker/u8:0+flush-179:023:50:220
528733997667,6cyclictest542160-21kworker/u8:0+flush-179:019:50:230
528733997667,5cyclictest561306-21kworker/u8:3+events_unbound20:55:260
528733997570,4cyclictest580617-21kworker/u8:2+events_unbound22:10:250
528733997567,6cyclictest558333-21kworker/u8:1+events_unbound21:04:590
528733997567,4cyclictest528606-21kworker/u8:0+events_unbound19:25:150
528733997468,5cyclictest626137-21kworker/u8:1+flush-179:000:25:210
528733997468,5cyclictest593664-21kworker/u8:3+events_unbound22:40:240
528733997366,5cyclictest608191-21kworker/u8:0+flush-179:023:55:210
528733997359,5cyclictest555965-21kworker/u8:0+events_unbound20:40:150
528733997265,4cyclictest555965-21kworker/u8:0+events_unbound21:50:150
528733997164,5cyclictest528606-21kworker/u8:0+flush-179:019:20:230
528736997061,7cyclictest596814-21latency_hist22:40:021
528733997063,5cyclictest538232-21kworker/u8:1+events_unbound20:00:190
528733997055,6cyclictest553203-21kworker/u8:3+events_unbound20:25:240
52873399700,68cyclictest1-21systemd22:25:010
528733996963,4cyclictest580617-21kworker/u8:2+events_unbound22:10:020
528733996962,5cyclictest575054-21kworker/u8:2+flush-179:021:35:200
528736996759,5cyclictest630891-21latency_hist00:25:011
528736996758,6cyclictest564444-21latency_hist21:00:001
528736996757,8cyclictest542974-21apt-get19:55:001
528733996760,5cyclictest542160-21kworker/u8:0+events_unbound20:20:010
528736996656,8cyclictest536858-21apt-get19:35:111
528733996663,2cyclictest570275-21kworker/0:2-events_freezable21:30:000
528733996660,4cyclictest598707-21kworker/u8:2+flush-179:023:10:000
528733996560,4cyclictest593664-21kworker/u8:3+flush-179:022:45:310
528733996460,3cyclictest553203-21kworker/u8:3+flush-179:020:30:170
528741996361,1cyclictest0-21swapper/319:37:593
528736996253,6cyclictest546686-21latency_hist20:05:011
528736996252,8cyclictest572963-21diskstats21:25:171
528733996257,4cyclictest608191-21kworker/u8:0+events_unbound23:15:250
528733996255,5cyclictest608191-21kworker/u8:0+events_unbound00:20:170
528733996254,6cyclictest538232-21kworker/u8:1+events_unbound19:45:260
528736996054,5cyclictest606603-21kworker/u8:3+events_unbound00:04:011
528733996055,4cyclictest558333-21kworker/u8:1+flush-179:021:05:300
528733996054,4cyclictest592020-21kworker/u8:0+events_unbound22:35:010
528733996050,6cyclictest536583-21kworker/u8:3+flush-179:019:40:000
528736995956,2cyclictest0-21swapper/123:23:581
528733995953,4cyclictest608191-21kworker/u8:0+flush-179:000:15:140
528733995853,3cyclictest592020-21kworker/u8:0+events_unbound22:40:020
528736995646,7cyclictest634062-21latency_hist00:35:011
528733995651,4cyclictest608191-21kworker/u8:0+events_unbound23:45:260
528733995651,4cyclictest569213-21kworker/u8:1+flush-179:021:15:250
528736995544,9cyclictest608467-21apt-get23:15:111
528733995549,5cyclictest608191-21kworker/u8:0+flush-179:023:40:220
528733995548,5cyclictest606603-21kworker/u8:3+events_unbound23:30:290
528733995546,7cyclictest582205-21kworker/u8:1+events_unbound22:00:210
528736995449,4cyclictest528606-21kworker/u8:0+events_unbound19:10:011
528736995346,5cyclictest266-21systemd-journal21:04:321
528733995342,8cyclictest562857-21latency_hist20:55:020
528736995243,7cyclictest603686-21kworker/u8:0+flush-179:023:05:211
528736995145,4cyclictest608191-21kworker/u8:0+events_unbound00:05:251
528736995144,6cyclictest580617-21kworker/u8:2+events_unbound22:25:001
528733995145,4cyclictest575054-21kworker/u8:2+events_unbound21:40:130
528733995141,7cyclictest266-21systemd-journal00:03:340
528736995043,5cyclictest559921-21/usr/sbin/munin20:45:181
528733995043,5cyclictest569213-21kworker/u8:1+events_unbound21:25:000
528733995043,5cyclictest569213-21kworker/u8:1+events_unbound21:24:590
528736994940,6cyclictest629299-21latency_hist00:20:011
528733994945,3cyclictest603686-21kworker/u8:0+flush-179:023:00:300
528736994843,4cyclictest608191-21kworker/u8:0+flush-179:023:45:251
528736994842,5cyclictest608191-21kworker/u8:0+flush-179:000:25:251
528736994841,6cyclictest555965-21kworker/u8:0+flush-179:020:40:211
528736994841,5cyclictest551722-21apt-get20:20:151
528733994843,4cyclictest608191-21kworker/u8:0+flush-179:000:05:150
528733994843,4cyclictest606603-21kworker/u8:3+events_unbound23:20:200
528733994842,4cyclictest608191-21kworker/u8:0+events_unbound00:10:210
528736994740,5cyclictest582205-21kworker/u8:1+events_unbound22:05:181
528736994740,5cyclictest575054-21kworker/u8:2+flush-179:021:34:591
528736994739,6cyclictest528606-21kworker/u8:0+flush-179:019:30:001
528736994737,8cyclictest580632-21latency_hist21:50:011
528733994743,3cyclictest592020-21kworker/u8:0+flush-179:022:25:120
528733994742,4cyclictest555965-21kworker/u8:0+flush-179:021:45:270
528733994739,6cyclictest538232-21kworker/u8:1+events_unbound19:40:240
4577091470,13ptp4l61490irq/134-eno1-TxRx-019:10:003
52874199450,44cyclictest561278-21turbostat20:49:593
528736994539,4cyclictest266-21systemd-journal22:46:311
528736994534,9cyclictest266-21systemd-journal23:40:001
528736994437,5cyclictest555965-21kworker/u8:0+events_unbound21:20:161
528736994437,5cyclictest555965-21kworker/u8:0+events_unbound21:20:161
528736994436,6cyclictest542160-21kworker/u8:0+flush-179:020:15:241
528736994435,6cyclictest610860-21kworker/u8:1+events_unbound23:25:151
528736994334,7cyclictest575054-21kworker/u8:2+flush-179:021:40:211
528736994333,7cyclictest556489-21latency_hist20:35:011
528736994325,7cyclictest582227-21apt-get21:55:001
528733994337,4cyclictest613047-21latency_hist23:30:000
528733994335,6cyclictest634067-21sh00:35:010
528736994236,3cyclictest619928-21apt-get23:50:141
52873699422,39cyclictest600267-21turbostat22:54:561
528733994237,4cyclictest606603-21kworker/u8:3+events_unbound23:13:310
528736994130,10cyclictest598707-21kworker/u8:2+events_unbound22:55:281
52873699411,11cyclictest603705-21apt-get23:00:171
52874199400,39cyclictest626109-21turbostat00:10:003
528736994032,7cyclictest544849-21latency_hist20:00:011
528733994036,3cyclictest555965-21kworker/u8:0+events_unbound21:30:170
528741993933,5cyclictest580645-21turbostat21:50:013
52874199390,38cyclictest618108-21turbostat23:45:003
528736993931,6cyclictest582205-21kworker/u8:1+flush-179:021:55:221
528736993928,9cyclictest567892-21apt-get21:10:141
528736993928,9cyclictest530466-21kworker/u8:1+flush-179:019:15:211
528736993928,9cyclictest530466-21kworker/u8:1+flush-179:019:15:211
528736993926,10cyclictest569767-21diskstats21:15:161
52874199380,37cyclictest630903-21turbostat00:25:003
52874199380,37cyclictest606633-21turbostat23:09:593
52874199380,37cyclictest0-21swapper/300:30:593
528736993834,3cyclictest585679-21munin-plugin-st22:05:011
528736993824,11cyclictest634076-21apt-get00:35:031
528733993833,4cyclictest542160-21kworker/u8:0+flush-179:019:55:150
528741993735,1cyclictest0-21swapper/322:21:003
528741993735,1cyclictest0-21swapper/320:20:593
528738993734,2cyclictest0-21swapper/223:33:592
528738993734,2cyclictest0-21swapper/200:34:552
528736993734,2cyclictest714-21Xorg00:12:591
528736993732,3cyclictest562859-21munin-run20:55:011
528736993731,4cyclictest592050-21turbostat22:29:551
528736993728,5cyclictest0-21swapper/120:35:581
52873699370,1cyclictest0-21swapper/121:08:001
528733993728,3cyclictest542160-21kworker/u8:0+events_unbound20:10:240
528741993634,1cyclictest0-21swapper/322:32:593
528741993634,1cyclictest0-21swapper/321:53:583
528741993634,1cyclictest0-21swapper/320:39:003
528741993633,2cyclictest0-21swapper/321:29:593
52874199360,35cyclictest0-21swapper/321:41:593
528738993633,2cyclictest0-21swapper/223:13:592
528738993633,2cyclictest0-21swapper/222:59:552
52873899360,1cyclictest0-21swapper/222:27:592
528736993629,5cyclictest530466-21kworker/u8:1+flush-179:019:30:161
528741993533,1cyclictest0-21swapper/322:16:593
528741993530,4cyclictest572436-21turbostat21:25:003
528741993530,4cyclictest572436-21turbostat21:24:593
528741993530,4cyclictest567654-21turbostat21:09:593
52874199350,1cyclictest0-21swapper/320:02:593
52874199350,1cyclictest0-21swapper/319:22:593
528738993532,2cyclictest0-21swapper/222:10:592
528738993532,2cyclictest0-21swapper/222:06:002
528738993531,2cyclictest0-21swapper/223:46:592
52873899350,1cyclictest0-21swapper/220:34:592
528736993531,3cyclictest622904-40sh00:00:001
528736993528,5cyclictest598707-21kworker/u8:2+events_unbound23:10:161
52873699350,8cyclictest540052-21apt-get19:45:141
528741993432,1cyclictest0-21swapper/322:07:593
528741993432,1cyclictest0-21swapper/320:54:553
528741993432,1cyclictest0-21swapper/319:49:563
528738993431,2cyclictest0-21swapper/223:52:582
528738993431,2cyclictest0-21swapper/222:21:592
528738993431,2cyclictest0-21swapper/220:29:592
528738993431,2cyclictest0-21swapper/200:09:552
52873899340,2cyclictest0-21swapper/220:05:142
52873899340,2cyclictest0-21swapper/220:05:142
52873899340,26cyclictest0-21swapper/223:17:592
528736993429,4cyclictest542160-21kworker/u8:0+events_unbound20:05:241
528736993429,4cyclictest542160-21kworker/u8:0+events_unbound20:05:241
528736993428,4cyclictest542160-21kworker/u8:0+flush-179:020:15:001
528741993331,1cyclictest0-21swapper/323:54:593
528738993330,2cyclictest0-21swapper/221:43:582
528738993325,7cyclictest0-21swapper/219:10:142
52873899330,26cyclictest0-21swapper/221:25:162
52873699333,29cyclictest281ktimers/120:25:131
52873699332,3cyclictest589100-21apt-get22:15:151
528741993230,1cyclictest0-21swapper/322:02:003
528741993230,1cyclictest0-21swapper/300:11:593
528738993229,2cyclictest0-21swapper/223:36:582
52873899320,30cyclictest0-21swapper/223:06:002
52873699320,30cyclictest529172-21expr19:10:141
528741993129,1cyclictest0-21swapper/323:16:003
528741993128,2cyclictest0-21swapper/323:29:593
528738993128,2cyclictest0-21swapper/200:20:582
528738993125,5cyclictest0-21swapper/222:15:142
52873899310,29cyclictest0-21swapper/220:25:012
528741993025,4cyclictest0-21swapper/323:57:003
528738993027,2cyclictest0-21swapper/223:00:172
528738993027,2cyclictest0-21swapper/220:37:002
528738993025,4cyclictest0-21swapper/220:55:152
52873899300,28cyclictest0-21swapper/219:55:012
52873899300,1cyclictest0-21swapper/222:39:552
528736993025,4cyclictest539488-21switchtime19:40:261
52873699301,2cyclictest0-21swapper/122:40:031
528741992927,1cyclictest0-21swapper/323:02:003
528741992927,1cyclictest0-21swapper/322:35:133
528741992927,1cyclictest0-21swapper/320:05:143
528741992927,1cyclictest0-21swapper/320:05:143
52874199290,28cyclictest0-21swapper/320:26:583
52874199290,27cyclictest0-21swapper/319:15:143
52874199290,27cyclictest0-21swapper/319:15:133
52874199290,27cyclictest0-21swapper/319:10:133
528738992926,2cyclictest0-21swapper/221:09:552
528738992926,2cyclictest0-21swapper/220:50:112
528738992926,2cyclictest0-21swapper/220:40:132
528738992926,2cyclictest0-21swapper/219:15:132
528738992926,2cyclictest0-21swapper/219:15:132
528738992925,2cyclictest0-21swapper/219:45:142
52873899290,2cyclictest0-21swapper/220:15:162
52873899290,28cyclictest0-21swapper/219:55:012
52873899290,27cyclictest0-21swapper/222:40:022
528741992826,1cyclictest0-21swapper/323:49:553
528741992826,1cyclictest0-21swapper/300:35:023
528741992825,2cyclictest0-21swapper/320:13:593
52874199280,2cyclictest0-21swapper/322:40:023
52874199280,26cyclictest0-21swapper/321:15:143
52874199280,21cyclictest0-21swapper/322:25:143
528738992825,2cyclictest0-21swapper/222:45:142
528738992825,2cyclictest0-21swapper/221:10:142
528738992823,3cyclictest0-21swapper/219:37:292
52873899280,18cyclictest0-21swapper/219:32:062
52873699281,26cyclictest0-21swapper/123:40:141
52873699280,3cyclictest532996-21/usr/sbin/munin19:20:221
4577091280,6ptp4l441ktimers/321:35:013
528741992725,2cyclictest0-21swapper/323:33:593
528741992725,1cyclictest0-21swapper/320:55:153
528741992725,1cyclictest0-21swapper/320:40:123
528741992725,1cyclictest0-21swapper/320:15:163
528738992725,1cyclictest0-21swapper/200:35:022
528738992724,2cyclictest0-21swapper/221:53:002
528738992724,2cyclictest0-21swapper/200:25:592
52873899270,26cyclictest0-21swapper/221:45:162
52873899270,1cyclictest0-21swapper/200:14:592
528736992724,2cyclictest0-21swapper/122:31:001
52874199260,24cyclictest0-21swapper/319:40:043
528738992623,2cyclictest0-21swapper/223:55:592
52873899260,24cyclictest0-21swapper/220:46:552
52873899260,24cyclictest0-21swapper/219:40:362
52873899260,24cyclictest0-21swapper/219:25:262
52873899260,24cyclictest0-21swapper/219:20:172
52873899260,24cyclictest0-21swapper/200:00:132
4577091260,3ptp4l441ktimers/321:05:003
528741992522,2cyclictest0-21swapper/322:45:133
52874199250,2cyclictest441ktimers/319:25:313
52874199250,24cyclictest0-21swapper/319:31:093
528738992522,2cyclictest0-21swapper/220:04:102
52873899250,2cyclictest0-21swapper/221:19:002
52873899250,1cyclictest0-21swapper/222:04:592
4577091250,3ptp4l441ktimers/321:10:013
528741992422,1cyclictest0-21swapper/319:57:593
528741992422,1cyclictest0-21swapper/319:50:133
528738992421,2cyclictest0-21swapper/221:33:002
528738992421,2cyclictest0-21swapper/220:12:592
52873899240,1cyclictest0-21swapper/222:35:002
4577091240,3ptp4l61490irq/134-eno1-TxRx-022:54:553
4577091240,3ptp4l61490irq/134-eno1-TxRx-021:59:543
4577091240,3ptp4l441ktimers/322:59:563
4577091240,3ptp4l441ktimers/321:39:553
4577091240,3ptp4l441ktimers/320:35:003
4577091240,3ptp4l441ktimers/300:29:563
4577091240,3ptp4l441ktimers/300:19:553
4577091240,3ptp4l441ktimers/300:05:003
4577091230,3ptp4l61490irq/134-eno1-TxRx-023:24:553
4577091230,3ptp4l441ktimers/323:40:013
4577091230,2ptp4l61490irq/134-eno1-TxRx-022:15:013
4577091230,2ptp4l441ktimers/323:15:003
528738992219,2cyclictest0-21swapper/223:29:552
528738992219,2cyclictest0-21swapper/223:23:592
528738992119,1cyclictest0-21swapper/200:20:002
528738992118,2cyclictest0-21swapper/221:23:002
528738992118,2cyclictest0-21swapper/221:22:592
528738992018,1cyclictest0-21swapper/221:01:592
528738992017,2cyclictest0-21swapper/223:44:002
528738992011,2cyclictest0-21swapper/222:51:582
528738991715,1cyclictest0-21swapper/221:59:592
528738991715,1cyclictest0-21swapper/221:39:592
52873399111,8cyclictest528745-21sed19:10:010
5287389930,2cyclictest0-21swapper/219:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional