You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-30 - 11:04
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6 (updated Sat Sep 30, 2023 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
40358949910192,6cyclictest4137533-21kworker/u8:0+flush-179:000:30:011
4035894999282,7cyclictest4123193-21latency_hist23:40:011
4035894998780,5cyclictest4015083-21kworker/u8:3+flush-179:019:25:321
4035894998778,6cyclictest4073011-21latency_hist21:05:011
4035891998780,5cyclictest4104334-21/usr/sbin/munin22:40:310
4035891998780,5cyclictest4104334-21/usr/sbin/munin22:40:310
4035891998680,5cyclictest4092675-21kworker/u8:2+events_unbound22:35:010
4035894998474,7cyclictest4090825-21latency_hist22:00:001
4035891998173,5cyclictest4066619-21kworker/u8:0+events_unbound21:05:200
4035894998064,14cyclictest4099100-21kworker/u8:3+flush-179:023:19:591
4035891998067,5cyclictest4137533-21kworker/u8:0+events_unbound00:25:210
4035894997974,3cyclictest4120043-21kworker/u8:2+flush-179:000:35:191
4035891997972,5cyclictest4116868-21kworker/u8:1+events_unbound23:30:190
4035891997964,5cyclictest4052040-21kworker/u8:3+events_unbound20:20:220
4035891997872,4cyclictest4074574-21kworker/u8:1+events_unbound21:35:200
4035891997872,4cyclictest4074574-21kworker/u8:1+events_unbound21:35:200
4035891997871,6cyclictest4052040-21kworker/u8:3+events_unbound21:00:010
4035894997768,6cyclictest4095858-21latency_hist22:15:011
4035891997765,9cyclictest4092943-21apt-get22:05:170
4035891997671,3cyclictest4052040-21kworker/u8:3+events_unbound20:00:190
4035891997668,6cyclictest4074574-21kworker/u8:1+events_unbound22:00:220
4035891997662,5cyclictest4052040-21kworker/u8:3+events_unbound20:40:220
4035894997567,5cyclictest4052040-21kworker/u8:3+flush-179:020:10:311
4035891997565,8cyclictest2158010-21systemd-journal22:00:010
4035891997469,4cyclictest4108261-21kworker/u8:0+flush-179:023:20:330
4035891997457,3cyclictest4123441-21kworker/u8:3+events_unbound23:40:270
4035894997363,8cyclictest4109093-21/usr/sbin/munin22:55:291
4035891997367,5cyclictest4059739-21kworker/u8:0+events_unbound20:25:310
4035891997362,9cyclictest4039330-21apt-get19:20:220
4035891997267,3cyclictest4089237-21kworker/u8:3+events_unbound22:10:330
4035891997265,5cyclictest4050180-21kworker/u8:0+events_unbound19:55:230
4035891997262,7cyclictest4081277-21latency_hist21:30:010
4035894997158,10cyclictest4099070-21apt-get22:25:041
4035891997163,6cyclictest4120043-21kworker/u8:2+events_unbound00:10:210
4035891997161,4cyclictest4099100-21kworker/u8:3+events_unbound23:10:260
4035891997065,4cyclictest4116868-21kworker/u8:1+events_unbound23:35:290
4035891997064,4cyclictest4015083-21kworker/u8:3+events_unbound19:35:200
4035891996964,4cyclictest4074574-21kworker/u8:1+flush-179:021:40:320
4035891996964,4cyclictest4066619-21kworker/u8:0+events_unbound20:55:330
4035891996963,5cyclictest4103499-21kworker/u8:1+events_unbound22:50:260
4035891996963,4cyclictest4074574-21kworker/u8:1+events_unbound22:15:230
403589799680,65cyclictest0-21swapper/321:28:593
4035894996862,4cyclictest4108261-21kworker/u8:0+flush-179:023:00:291
4035894996862,4cyclictest4047262-21kworker/u8:4+flush-179:019:50:331
4035891996864,3cyclictest4123441-21kworker/u8:3+events_unbound23:45:330
4035891996863,4cyclictest4034135-21kworker/u8:1+flush-179:019:49:550
403589199680,66cyclictest4112091-21munin-run23:05:000
4035894996762,3cyclictest4079356-21kworker/u8:0+flush-179:021:25:231
4035894996762,3cyclictest4074574-21kworker/u8:1+flush-179:021:20:311
4035894996560,3cyclictest4120043-21kworker/u8:2+flush-179:023:35:001
4035894996558,6cyclictest4099100-21kworker/u8:3+flush-179:022:30:281
4035894996458,5cyclictest4015083-21kworker/u8:3+flush-179:019:19:561
4035894996457,4cyclictest4071125-21kworker/u8:2+flush-179:021:15:221
4035891996459,4cyclictest4015083-21kworker/u8:3+flush-179:019:25:290
4035891996448,4cyclictest4103499-21kworker/u8:1+events_unbound22:35:320
4035894996358,3cyclictest4102100-21kworker/u8:0+flush-179:022:40:201
4035894996358,3cyclictest4102100-21kworker/u8:0+flush-179:022:40:191
4035894996355,6cyclictest4124791-21latency_hist23:45:011
4035894996346,15cyclictest4134590-21/usr/sbin/munin00:15:281
403589199630,62cyclictest0-21swapper/022:57:590
4035894996256,4cyclictest4066619-21kworker/u8:0+flush-179:021:05:181
4035894996254,7cyclictest4058123-21kworker/u8:2+events_unbound20:20:001
4035894996154,5cyclictest4092675-21kworker/u8:2+flush-179:022:20:311
4035891996157,3cyclictest4050180-21kworker/u8:0+flush-179:020:15:230
4035891996151,3cyclictest4092675-21kworker/u8:2+events_unbound22:25:340
4035891996146,12cyclictest4129582-40mandb00:00:000
4035894996056,3cyclictest4139127-21kworker/u8:1+events_unbound00:30:341
4035894996055,4cyclictest4126300-21kworker/u8:1+flush-179:000:04:551
4035894996055,4cyclictest4015083-21kworker/u8:3+events_unbound19:40:011
4035894996053,5cyclictest4099100-21kworker/u8:3+flush-179:023:30:011
4035891996056,3cyclictest4126300-21kworker/u8:1+flush-179:000:08:310
4035891996054,5cyclictest4052040-21kworker/u8:3+events_unbound20:10:300
4035891996048,3cyclictest4015083-21kworker/u8:3+events_unbound19:15:350
4035894995953,4cyclictest4059739-21kworker/u8:0+flush-179:020:30:261
4035891995954,4cyclictest4052040-21kworker/u8:3+flush-179:020:05:270
4035894995850,5cyclictest4064781-21latency_hist20:40:011
4035891995853,4cyclictest4032562-21kworker/u8:2+events_unbound19:10:230
4035891995838,10cyclictest529-21rs:main0
4035894995752,4cyclictest2158010-21systemd-journal23:48:291
4035894995752,3cyclictest4116868-21kworker/u8:1+flush-179:023:25:011
4035894995648,6cyclictest4082856-21kworker/u8:4+flush-179:021:40:181
4035891995652,3cyclictest4099100-21kworker/u8:3+flush-179:023:15:320
4035891995652,3cyclictest4047262-21kworker/u8:4+events_unbound19:50:340
4035891995647,7cyclictest4061597-21apt-get20:30:200
4035894995549,5cyclictest4088176-21kworker/u8:2+flush-179:021:50:311
4035894995549,4cyclictest4034135-21kworker/u8:1+events_unbound19:50:001
4035894995545,8cyclictest4042502-21apt-get19:30:211
4035891995550,4cyclictest4139127-21kworker/u8:1+events_unbound00:30:210
4035891995550,4cyclictest4061393-21kworker/u8:2+flush-179:020:40:000
4035894995449,3cyclictest4066619-21kworker/u8:0+events_unbound20:50:231
4035894995434,18cyclictest2158010-21systemd-journal22:03:311
4035891995449,4cyclictest4120043-21kworker/u8:2+flush-179:000:20:080
4035894995347,4cyclictest4074574-21kworker/u8:1+flush-179:021:10:211
4035891995348,4cyclictest4074574-21kworker/u8:1+events_unbound21:20:310
4035891995348,4cyclictest4047010-21kworker/u8:0+flush-179:019:44:550
4035891995344,7cyclictest4076183-21latency_hist21:15:010
4035894995246,4cyclictest4096924-21kworker/u8:0+flush-179:022:20:011
4035891995248,3cyclictest4071125-21kworker/u8:2+flush-179:021:18:320
4035891995245,6cyclictest4120043-21kworker/u8:2+flush-179:000:05:010
4035894995145,5cyclictest4108261-21kworker/u8:0+flush-179:023:09:561
4035891995147,3cyclictest4108261-21kworker/u8:0+events_unbound23:50:320
4035891995146,4cyclictest4043813-21kworker/u8:0+events_unbound19:33:320
4035891995144,5cyclictest4069555-21latency_hist20:55:010
4035897995048,1cyclictest0-21swapper/322:57:593
4035897995048,1cyclictest0-21swapper/320:21:593
4035897995043,5cyclictest4104092-21turbostat22:40:003
4035894995045,4cyclictest4050180-21kworker/u8:0+events_unbound19:55:321
4035894995044,4cyclictest4103499-21kworker/u8:1+events_unbound22:35:351
4035894995043,5cyclictest4120043-21kworker/u8:2+flush-179:023:55:321
4652991490,13ptp4l441ktimers/319:09:563
4035894994943,4cyclictest4074574-21kworker/u8:1+events_unbound21:45:211
4035891994945,3cyclictest4074574-21kworker/u8:1+flush-179:021:45:290
4035891994944,4cyclictest4116868-21kworker/u8:1+flush-179:023:25:360
4035894994839,6cyclictest2158010-21systemd-journal22:53:321
4035894994827,18cyclictest0-21swapper/121:35:011
4035891994844,3cyclictest4103499-21kworker/u8:1+flush-179:022:45:310
4035891994844,3cyclictest4088176-21kworker/u8:2+flush-179:021:50:290
4035894994642,3cyclictest4131215-21kworker/u8:3+flush-179:000:10:301
4035894994642,3cyclictest4052040-21kworker/u8:3+flush-179:020:05:201
4035894994539,4cyclictest4120043-21kworker/u8:2+flush-179:000:05:241
4035894994537,7cyclictest4064236-21kworker/u8:1+flush-179:020:40:061
4035891994540,4cyclictest4099100-21kworker/u8:3+flush-179:023:05:350
4035897994438,4cyclictest4054988-21turbostat20:10:003
4035894994440,2cyclictest4128099-21latency_hist23:55:011
4035894994439,4cyclictest4074574-21kworker/u8:1+flush-179:021:35:311
4035894994439,4cyclictest4074574-21kworker/u8:1+flush-179:021:35:311
4035895994340,2cyclictest0-21swapper/223:11:582
4035894994339,3cyclictest4092675-21kworker/u8:2+events_unbound22:05:331
4035894994338,4cyclictest4108261-21kworker/u8:0+flush-179:023:10:301
4035894994337,4cyclictest4120043-21kworker/u8:2+events_unbound00:20:301
403589499431,32cyclictest0-21swapper/119:40:191
4035891994336,6cyclictest4126300-21kworker/u8:1+events_unbound00:15:230
4035894994235,5cyclictest4053392-21latency_hist20:05:011
4035894994229,11cyclictest4067968-21latency_hist20:50:011
403589499420,30cyclictest4070605-21ntp_states20:55:311
4035891994238,3cyclictest4137533-21kworker/u8:0+events_unbound00:35:040
403589799410,39cyclictest4061355-21turbostat20:30:003
4035897994038,1cyclictest0-21swapper/323:28:593
403589599400,38cyclictest0-21swapper/219:34:582
4035894994035,4cyclictest4103499-21kworker/u8:1+flush-179:022:45:361
4652991390,4ptp4l441ktimers/300:30:013
4652991390,3ptp4l441ktimers/300:00:003
4035897993936,2cyclictest0-21swapper/322:34:593
403589799390,38cyclictest0-21swapper/323:20:593
403589499390,38cyclictest0-21swapper/120:20:211
4035891993933,4cyclictest4052040-21kworker/u8:3+events_unbound20:45:090
4035897993836,1cyclictest0-21swapper/322:50:283
4035897993836,1cyclictest0-21swapper/322:00:223
4035897993831,5cyclictest4137563-21turbostat00:25:013
403589799380,36cyclictest0-21swapper/323:48:593
4035895993835,2cyclictest0-21swapper/219:52:582
4035895993835,2cyclictest0-21swapper/219:29:002
403589599380,35cyclictest0-21swapper/222:44:562
403589599380,35cyclictest0-21swapper/222:44:552
4035897993731,4cyclictest4053408-21turbostat20:05:003
403589799370,36cyclictest0-21swapper/322:27:593
4035895993734,2cyclictest0-21swapper/223:38:002
4035895993734,2cyclictest0-21swapper/200:07:592
403589599370,35cyclictest0-21swapper/221:49:592
4035894993733,2cyclictest725-21Xorg19:24:591
4035894993723,12cyclictest4036422-21diskstats19:10:241
403589499370,1cyclictest0-21swapper/120:25:191
4035891993733,3cyclictest4081260-21kworker/u8:3+events_unbound21:31:320
4035897993633,2cyclictest0-21swapper/322:48:583
403589799360,35cyclictest0-21swapper/319:40:593
4035895993628,6cyclictest0-21swapper/220:05:002
4652991350,13ptp4l441ktimers/323:05:013
4035897993533,1cyclictest0-21swapper/322:42:003
4035897993533,1cyclictest0-21swapper/322:42:003
4035897993533,1cyclictest0-21swapper/322:16:593
403589799350,1cyclictest0-21swapper/323:43:593
4035895993532,2cyclictest0-21swapper/222:30:192
4035895993532,2cyclictest0-21swapper/221:04:592
4035895993532,1cyclictest0-21swapper/220:25:002
403589599350,34cyclictest0-21swapper/200:10:202
403589599350,33cyclictest0-21swapper/200:35:022
4652991340,3ptp4l61790irq/134-eno1-TxRx-022:20:203
4035897993432,1cyclictest0-21swapper/300:16:003
4035897993428,4cyclictest4116828-21turbostat23:20:003
4035895993431,2cyclictest0-21swapper/219:11:592
4035895993430,2cyclictest0-21swapper/220:14:592
403589599340,32cyclictest0-21swapper/223:50:242
4035897993329,2cyclictest0-21swapper/321:36:593
4035897993329,2cyclictest0-21swapper/321:36:593
403589799330,32cyclictest0-21swapper/323:50:223
4035895993330,2cyclictest0-21swapper/219:16:592
4035895993325,7cyclictest0-21swapper/222:00:242
4035895993324,7cyclictest0-21swapper/223:30:202
403589599330,32cyclictest0-21swapper/221:50:002
403589599330,31cyclictest0-21swapper/221:15:222
403589599330,31cyclictest0-21swapper/200:02:592
403589599330,1cyclictest0-21swapper/220:35:002
4035897993230,1cyclictest0-21swapper/319:10:213
4035897993230,1cyclictest0-21swapper/300:03:003
4035897993229,2cyclictest0-21swapper/321:09:593
403589799320,31cyclictest0-21swapper/323:33:593
4035895993229,2cyclictest0-21swapper/220:40:592
4035895993226,5cyclictest0-21swapper/222:45:232
4035895993226,5cyclictest0-21swapper/200:25:002
4035895993225,6cyclictest0-21swapper/220:05:282
403589599320,6cyclictest0-21swapper/200:25:202
4035897993128,2cyclictest0-21swapper/319:30:213
4035897993127,2cyclictest0-21swapper/319:38:003
403589799310,30cyclictest0-21swapper/320:10:593
403589799310,30cyclictest0-21swapper/300:15:013
4035895993128,2cyclictest0-21swapper/221:12:592
4035895993127,2cyclictest0-21swapper/222:08:592
4035895993124,6cyclictest0-21swapper/220:35:322
403589599310,30cyclictest0-21swapper/222:35:212
403589599310,29cyclictest0-21swapper/223:05:222
4035897993028,1cyclictest0-21swapper/320:35:593
4035897993028,1cyclictest0-21swapper/319:45:233
4035897993027,2cyclictest0-21swapper/323:35:213
4035897993027,2cyclictest0-21swapper/321:45:193
4035897993027,2cyclictest0-21swapper/321:40:203
4035897993027,2cyclictest0-21swapper/300:35:183
403589799300,29cyclictest0-21swapper/319:25:203
403589799300,28cyclictest0-21swapper/322:10:253
403589799300,28cyclictest0-21swapper/321:15:213
403589799300,28cyclictest0-21swapper/319:20:333
4035895993027,2cyclictest0-21swapper/222:15:222
4035895993027,2cyclictest0-21swapper/221:55:202
4035895993027,2cyclictest0-21swapper/221:25:012
4035895993027,2cyclictest0-21swapper/220:15:222
4035895993022,7cyclictest0-21swapper/223:25:292
403589599300,4cyclictest0-21swapper/221:05:202
403589599300,2cyclictest0-21swapper/221:30:272
403589599300,2cyclictest0-21swapper/220:55:222
403589599300,28cyclictest0-21swapper/222:50:172
403589599300,28cyclictest0-21swapper/221:20:292
403589599300,28cyclictest0-21swapper/220:45:222
403589599300,28cyclictest0-21swapper/220:25:212
403589599300,28cyclictest0-21swapper/219:40:222
4035897992927,1cyclictest0-21swapper/321:50:193
4035897992927,1cyclictest0-21swapper/320:16:593
4035897992927,1cyclictest0-21swapper/300:30:023
4035897992926,2cyclictest0-21swapper/323:05:233
4035897992926,2cyclictest0-21swapper/321:10:593
403589799290,28cyclictest0-21swapper/321:20:013
403589799290,28cyclictest0-21swapper/321:00:313
403589799290,28cyclictest0-21swapper/320:50:023
403589799290,28cyclictest0-21swapper/320:40:223
403589799290,28cyclictest0-21swapper/319:15:223
403589799290,27cyclictest0-21swapper/323:10:203
403589799290,27cyclictest0-21swapper/320:55:213
403589799290,27cyclictest0-21swapper/319:55:223
403589799290,27cyclictest0-21swapper/319:50:213
403589799290,1cyclictest0-21swapper/321:34:593
4035895992926,2cyclictest0-21swapper/223:49:552
4035895992926,2cyclictest0-21swapper/223:00:202
4035895992926,2cyclictest0-21swapper/222:55:212
4035895992926,2cyclictest0-21swapper/222:10:222
4035895992926,2cyclictest0-21swapper/219:45:212
4035895992926,1cyclictest0-21swapper/222:20:212
403589599290,2cyclictest0-21swapper/223:15:312
403589599290,28cyclictest0-21swapper/221:35:182
403589599290,28cyclictest0-21swapper/221:35:182
403589599290,27cyclictest0-21swapper/223:40:172
403589599290,27cyclictest0-21swapper/220:50:232
403589599290,1cyclictest0-21swapper/223:20:222
403589599290,1cyclictest0-21swapper/222:25:022
403589599290,1cyclictest0-21swapper/219:40:012
4035897992826,1cyclictest0-21swapper/322:05:183
403589799280,27cyclictest0-21swapper/320:50:213
403589799280,26cyclictest0-21swapper/321:55:223
4035895992825,2cyclictest0-21swapper/221:40:212
4035895992825,2cyclictest0-21swapper/219:55:222
4035895992825,2cyclictest0-21swapper/219:20:222
403589599280,27cyclictest0-21swapper/200:30:232
403589599280,26cyclictest0-21swapper/223:55:222
403589599280,26cyclictest0-21swapper/200:15:222
4035894992824,3cyclictest4036055-21apt-key19:10:021
4035897992725,1cyclictest0-21swapper/300:05:243
4035897992623,2cyclictest0-21swapper/320:30:233
4035891992521,3cyclictest4035595-21latency_hist19:10:030
40358959950,4cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional