You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 06:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6.osadl.org (updated Sat Jul 27, 2024 00:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4157597999080,7cyclictest4167186-21latency_hist19:40:000
4157597998881,5cyclictest54330-21kworker/u8:0+flush-179:000:25:010
4157600998778,6cyclictest54295-21latency_hist23:50:011
4157600998676,8cyclictest30684-21/usr/sbin/munin22:35:321
4157597998574,6cyclictest9573-21kworker/u8:3+events_unbound22:00:020
4157600998477,4cyclictest4165822-21kworker/u8:3+flush-179:019:40:271
4157597998480,3cyclictest44727-21kworker/u8:1+flush-179:023:25:240
4157597998076,3cyclictest67966-21kworker/u8:1+flush-179:000:30:240
4157597998076,3cyclictest41555-21kworker/u8:0+flush-179:023:10:250
4157597998076,3cyclictest41555-21kworker/u8:0+flush-179:023:10:250
4157597997974,4cyclictest4489-21kworker/u8:2+flush-179:022:20:240
4157597997874,3cyclictest44727-21kworker/u8:1+flush-179:023:35:250
4157597997765,9cyclictest55878-21latency_hist23:55:010
4157600997671,3cyclictest4176952-21kworker/u8:0+events_unbound20:20:241
4157600997670,4cyclictest45004-21kworker/u8:2+flush-179:023:25:001
4157597997665,3cyclictest41555-21kworker/u8:0+events_unbound23:15:240
4157597997569,4cyclictest4489-21kworker/u8:2+events_unbound22:30:120
4157600997461,6cyclictest9573-21kworker/u8:3+events_unbound22:10:001
4157597997469,4cyclictest48937-21kworker/u8:2+flush-179:023:40:210
4157597997368,4cyclictest6338-21kworker/u8:0+events_unbound21:20:230
4157597997368,4cyclictest4154071-21kworker/u8:0+flush-179:019:50:250
4157600997265,5cyclictest4165822-21kworker/u8:3+flush-179:019:35:291
4157597997268,3cyclictest9573-21kworker/u8:3+flush-179:022:05:310
4157597997268,3cyclictest4489-21kworker/u8:2+flush-179:022:25:270
4157597997267,4cyclictest4144461-21kworker/u8:2+flush-179:019:10:210
4157600997159,7cyclictest54330-21kworker/u8:0+flush-179:023:50:181
4157597997167,3cyclictest4173555-21kworker/u8:2+events_unbound20:05:250
4157597997166,4cyclictest4489-21kworker/u8:2+events_unbound22:15:240
4157597997166,4cyclictest4489-21kworker/u8:2+events_unbound22:15:230
4157600997062,6cyclictest27278-21latency_hist22:25:001
4157597997066,3cyclictest4187061-21kworker/u8:2+events_unbound20:50:250
4157597997066,3cyclictest30482-21kworker/u8:1+flush-179:022:40:120
4157597997066,3cyclictest30482-21kworker/u8:1+flush-179:022:40:120
4157600996959,8cyclictest9776-21/usr/sbin/munin21:30:231
4157597996964,4cyclictest44727-21kworker/u8:1+flush-179:023:55:280
4157597996964,4cyclictest44727-21kworker/u8:1+flush-179:023:55:270
4157597996961,6cyclictest4186512-21latency_hist20:40:010
4157600996862,4cyclictest4489-21kworker/u8:2+events_unbound22:10:241
415759799680,66cyclictest716-21Xorg21:04:580
4157600996756,3cyclictest35198-21kworker/u8:0+flush-179:022:50:251
4157597996762,4cyclictest36783-21kworker/u8:3+flush-179:023:05:220
4157597996759,6cyclictest35220-21latency_hist22:50:000
4157600996661,4cyclictest4154071-21kworker/u8:0+flush-179:019:45:121
4157600996660,4cyclictest4173555-21kworker/u8:2+events_unbound20:00:011
4157600996660,4cyclictest4173555-21kworker/u8:2+events_unbound20:00:001
4157597996662,3cyclictest7994-21kworker/u8:1+flush-179:021:30:480
4157597996662,3cyclictest36783-21kworker/u8:3+events_unbound22:55:300
4157600996558,4cyclictest36783-21kworker/u8:3+flush-179:023:05:211
4157597996563,1cyclictest0-21swapper/021:49:580
4157597996561,3cyclictest9573-21kworker/u8:3+events_unbound21:35:240
4157597996553,9cyclictest4161072-21/usr/sbin/munin19:20:250
4157597996460,3cyclictest4191514-21kworker/u8:0+events_unbound21:10:190
4157597996460,3cyclictest4163459-21kworker/u8:1+flush-179:019:30:150
4157597996458,5cyclictest4489-21kworker/u8:2+events_unbound21:55:170
4157597996458,5cyclictest4165822-21kworker/u8:3+events_unbound20:00:250
4157597996359,3cyclictest4187061-21kworker/u8:2+events_unbound20:45:220
4157597996358,4cyclictest35198-21kworker/u8:0+events_unbound23:00:260
4157600996255,5cyclictest4489-21kworker/u8:2+flush-179:022:45:241
4157600996254,5cyclictest4489-21kworker/u8:2+events_unbound21:25:351
4157597996258,3cyclictest4154071-21kworker/u8:0+events_unbound19:25:220
4157597996257,4cyclictest4187061-21kworker/u8:2+flush-179:020:40:250
4157597996157,3cyclictest7994-21kworker/u8:1+events_unbound21:25:450
4157597996156,3cyclictest44727-21kworker/u8:1+events_unbound00:00:270
4157600996056,3cyclictest54330-21kworker/u8:0+flush-179:000:05:251
4157600996056,3cyclictest54330-21kworker/u8:0+flush-179:000:05:241
4157600996054,4cyclictest753-21kworker/u8:1+flush-179:021:15:161
4157600996052,6cyclictest35198-21kworker/u8:0+flush-179:023:00:001
4157597996056,3cyclictest4176952-21kworker/u8:0+flush-179:020:19:560
4157597996056,3cyclictest4176952-21kworker/u8:0+events_unbound20:20:220
4157597996055,4cyclictest753-21kworker/u8:1+events_unbound21:15:280
4157600995953,4cyclictest30482-21kworker/u8:1+flush-179:022:40:161
4157600995953,4cyclictest30482-21kworker/u8:1+flush-179:022:40:161
4157597995955,3cyclictest35198-21kworker/u8:0+flush-179:022:50:280
4157600995854,3cyclictest4154071-21kworker/u8:0+flush-179:019:25:261
4157597995854,3cyclictest44727-21kworker/u8:1+events_unbound00:10:150
4157600995750,6cyclictest44727-21kworker/u8:1+flush-179:023:35:011
4157600995749,6cyclictest54330-21kworker/u8:0+flush-179:000:00:011
4157600995749,6cyclictest54330-21kworker/u8:0+flush-179:000:00:001
4157597995753,3cyclictest4191514-21kworker/u8:0+events_unbound20:58:320
4157597995646,4cyclictest4154071-21kworker/u8:0+events_unbound19:45:240
4157600995549,4cyclictest4157546-21kworker/u8:3+flush-179:019:15:211
4157600995543,10cyclictest4185168-21apt-get20:35:091
4157597995551,3cyclictest70219-21kworker/u8:2+events_unbound00:35:280
4157597995551,3cyclictest70219-21kworker/u8:2+events_unbound00:35:280
4157597995551,3cyclictest54330-21kworker/u8:0+events_unbound00:15:260
4157597995550,4cyclictest4165822-21kworker/u8:3+flush-179:019:40:150
4157600995449,3cyclictest4176952-21kworker/u8:0+flush-179:020:45:001
4157600995445,6cyclictest4489-21kworker/u8:2+flush-179:022:15:191
4157600995445,6cyclictest4489-21kworker/u8:2+flush-179:022:15:181
4157600995445,6cyclictest2664-21apt-get21:10:011
4157597995449,4cyclictest4176952-21kworker/u8:0+events_unbound20:10:300
4157600995347,4cyclictest54330-21kworker/u8:0+flush-179:000:35:011
4157600995347,4cyclictest44727-21kworker/u8:1+events_unbound23:30:011
4157600995342,9cyclictest19522-21cat22:00:011
4157597995349,3cyclictest48937-21kworker/u8:2+events_unbound23:45:220
4157600995246,4cyclictest4154071-21kworker/u8:0+flush-179:019:50:201
4157600995242,7cyclictest4176952-21kworker/u8:0+flush-179:020:30:231
4157597995248,3cyclictest44727-21kworker/u8:1+flush-179:023:34:550
4157597995247,4cyclictest54330-21kworker/u8:0+events_unbound00:28:330
4157597995242,8cyclictest4181999-21apt-get20:25:100
4157600995146,3cyclictest4173555-21kworker/u8:2+events_unbound20:05:131
4157600995145,4cyclictest63792-21kworker/u8:2+events_unbound00:20:131
4157597995147,3cyclictest41555-21kworker/u8:0+events_unbound23:20:320
4157600995044,4cyclictest4189668-21kworker/u8:1+flush-179:020:50:201
4157597995046,3cyclictest4191514-21kworker/u8:0+events_unbound21:09:310
4157597995042,6cyclictest14308-21latency_hist21:45:010
4157600994943,4cyclictest4489-21kworker/u8:2+events_unbound21:55:161
4157600994943,4cyclictest4489-21kworker/u8:2+events_unbound21:50:271
4157600994943,4cyclictest4489-21kworker/u8:2+events_unbound21:50:271
4157600994942,5cyclictest66000-21apt-get00:25:091
4157597994945,3cyclictest4489-21kworker/u8:2+flush-179:021:50:320
4157597994945,3cyclictest4489-21kworker/u8:2+flush-179:021:50:320
4157597994945,3cyclictest4157546-21kworker/u8:3+flush-179:019:15:260
4157600994843,4cyclictest4489-21kworker/u8:2+flush-179:021:24:561
4157600994841,5cyclictest4175388-21latency_hist20:05:001
4157600994839,6cyclictest41555-21kworker/u8:0+flush-179:023:15:211
4157600994836,4cyclictest44727-21kworker/u8:1+events_unbound00:05:011
4157597994844,3cyclictest4489-21kworker/u8:2+events_unbound22:10:220
4157597994844,3cyclictest4181729-21kworker/u8:1+flush-179:020:30:030
4157600994737,8cyclictest41831-21apt-get23:10:101
4157600994737,8cyclictest41831-21apt-get23:10:101
4157597994743,3cyclictest4489-21kworker/u8:2+events_unbound22:38:330
4667591460,10ptp4l61790irq/135-eno1-TxRx-019:05:223
4667591460,10ptp4l61790irq/135-eno1-TxRx-019:05:223
415760799440,41cyclictest0-21swapper/322:51:593
4157600994440,3cyclictest4157546-21kworker/u8:3+flush-179:019:24:561
4157600994438,5cyclictest4176952-21kworker/u8:0+flush-179:020:10:321
415760099442,40cyclictest28875-21turbostat22:34:561
4157600994337,4cyclictest4191514-21kworker/u8:0+events_unbound21:00:291
4157600994332,8cyclictest266-21systemd-journal19:10:051
4157597994339,3cyclictest4165822-21kworker/u8:3+events_unbound19:55:170
4157597994339,3cyclictest4165822-21kworker/u8:3+events_unbound19:55:170
4157607994238,2cyclictest0-21swapper/323:37:593
4157607994238,2cyclictest0-21swapper/320:51:593
4157602994238,2cyclictest0-21swapper/223:37:592
415760299420,40cyclictest0-21swapper/222:51:592
4157600994239,2cyclictest4181729-21kworker/u8:1+events_unbound20:25:211
4157600994237,4cyclictest35198-21kworker/u8:0+flush-179:023:00:261
4157600994236,4cyclictest54330-21kworker/u8:0+flush-179:000:15:211
4157602994138,2cyclictest0-21swapper/220:51:582
4157600994136,4cyclictest54330-21kworker/u8:0+flush-179:000:35:341
4157600994136,4cyclictest54330-21kworker/u8:0+flush-179:000:35:331
4157600994136,4cyclictest4189668-21kworker/u8:1+events_unbound20:50:011
4157600994135,5cyclictest4489-21kworker/u8:2+events_unbound22:25:241
4157600994135,4cyclictest4154071-21kworker/u8:0+events_unbound19:30:231
4157597994137,3cyclictest54330-21kworker/u8:0+events_unbound00:08:330
4157597994137,3cyclictest54330-21kworker/u8:0+events_unbound00:08:330
415760799400,4cyclictest33638-21turbostat22:45:003
415760799400,4cyclictest33638-21turbostat22:45:003
415760799400,38cyclictest65751-21turbostat00:25:003
4157600994037,2cyclictest44727-21kworker/u8:1+events_unbound00:10:181
4157600994033,5cyclictest281ktimers/121:40:131
415760799390,4cyclictest4184927-21turbostat20:35:013
415760799390,1cyclictest0-21swapper/320:27:593
415760299390,36cyclictest0-21swapper/223:54:552
415760299390,2cyclictest0-21swapper/223:11:592
415760299390,2cyclictest0-21swapper/223:11:592
415760099390,38cyclictest271rcuc/120:56:591
4157607993831,5cyclictest776-21turbostat21:04:593
415760799380,37cyclictest0-21swapper/323:54:553
415760799380,36cyclictest0-21swapper/321:52:003
415760799380,36cyclictest0-21swapper/321:51:593
4157602993835,2cyclictest0-21swapper/219:29:002
4157602993835,2cyclictest0-21swapper/200:20:592
415760299380,36cyclictest0-21swapper/220:21:592
415760299380,1cyclictest0-21swapper/223:19:592
4157600993833,4cyclictest4489-21kworker/u8:2+flush-179:021:45:281
4157602993734,2cyclictest0-21swapper/221:12:592
4157602993734,2cyclictest0-21swapper/220:14:542
4157602993734,2cyclictest0-21swapper/219:44:592
4157602993733,2cyclictest0-21swapper/223:43:592
4157602993733,2cyclictest0-21swapper/221:27:592
415760299370,2cyclictest0-21swapper/220:28:002
4157600993731,5cyclictest48937-21kworker/u8:2+flush-179:023:40:111
4157607993633,2cyclictest0-21swapper/323:43:593
4157607993633,2cyclictest0-21swapper/321:13:003
4157607993633,2cyclictest0-21swapper/319:29:003
4157607993632,2cyclictest0-21swapper/321:32:003
415760799360,2cyclictest0-21swapper/323:11:593
415760799360,2cyclictest0-21swapper/323:11:593
415760799360,1cyclictest0-21swapper/323:19:593
4157602993632,2cyclictest0-21swapper/222:44:552
4157602993632,2cyclictest0-21swapper/222:44:552
415760299360,1cyclictest0-21swapper/221:52:002
415760299360,1cyclictest0-21swapper/221:51:592
4667591350,26ptp4l67-21kworker/3:1+events20:10:143
4157607993533,1cyclictest0-21swapper/319:45:003
4157607993532,2cyclictest0-21swapper/321:27:593
4157602993532,2cyclictest0-21swapper/221:31:592
4157602993531,2cyclictest0-21swapper/221:05:002
4157600993532,2cyclictest0-21swapper/120:19:541
4157600993528,5cyclictest716-21Xorg21:13:591
4157607993430,3cyclictest0-21swapper/321:07:583
4157602993431,2cyclictest0-21swapper/221:07:582
4157602993430,2cyclictest0-21swapper/222:34:552
4157602993430,2cyclictest0-21swapper/219:20:582
415760299340,2cyclictest0-21swapper/219:49:552
4157600993430,3cyclictest44727-21kworker/u8:1+flush-179:023:35:281
4157607993330,2cyclictest0-21swapper/319:20:593
415760799330,2cyclictest4160842-21turbostat19:19:593
4157602993330,2cyclictest0-21swapper/221:49:552
4157602993330,2cyclictest0-21swapper/219:51:592
4157602993329,2cyclictest0-21swapper/220:04:562
4667591320,5ptp4l61790irq/135-eno1-TxRx-022:20:003
4667591320,5ptp4l61790irq/135-eno1-TxRx-022:20:003
4157607993230,1cyclictest0-21swapper/322:34:553
4157607993229,2cyclictest0-21swapper/319:51:583
4157602993229,2cyclictest0-21swapper/219:19:002
4157602993228,2cyclictest0-21swapper/222:27:592
4157600993227,4cyclictest9573-21kworker/u8:3+flush-179:021:35:171
4157597993227,3cyclictest0-21swapper/019:10:030
4157597993227,3cyclictest0-21swapper/019:10:030
4157607993129,1cyclictest0-21swapper/321:49:553
4157607993126,4cyclictest7965-21turbostat21:25:003
4157602993128,2cyclictest0-21swapper/221:18:002
4157602993127,2cyclictest0-21swapper/220:32:592
415760299310,29cyclictest0-21swapper/220:55:152
4157607993028,1cyclictest0-21swapper/320:04:563
4157607993027,2cyclictest0-21swapper/323:59:553
4157607993027,2cyclictest0-21swapper/323:59:553
4157607993027,2cyclictest0-21swapper/322:27:593
4157607993027,2cyclictest0-21swapper/321:15:013
415760799300,29cyclictest0-21swapper/320:55:153
415760799300,28cyclictest0-21swapper/322:00:023
415760799300,27cyclictest0-21swapper/323:05:133
415760799300,21cyclictest0-21swapper/320:20:123
415760799300,1cyclictest0-21swapper/319:49:553
4157602993027,2cyclictest0-21swapper/223:59:552
4157602993027,2cyclictest0-21swapper/223:59:552
4157602993027,2cyclictest0-21swapper/221:59:542
4157602993027,2cyclictest0-21swapper/220:36:582
415760299300,28cyclictest0-21swapper/221:40:112
415760299300,28cyclictest0-21swapper/200:05:122
415760299300,28cyclictest0-21swapper/200:05:122
4667591290,6ptp4l441ktimers/323:04:563
4157607992926,2cyclictest0-21swapper/322:45:123
4157607992926,2cyclictest0-21swapper/320:36:593
415760799290,2cyclictest0-21swapper/300:12:593
415760799290,28cyclictest0-21swapper/320:40:143
415760799290,27cyclictest0-21swapper/321:40:123
415760799290,27cyclictest0-21swapper/321:35:113
415760799290,26cyclictest0-21swapper/322:20:113
415760799290,21cyclictest0-21swapper/300:05:123
415760799290,21cyclictest0-21swapper/300:05:123
4157602992926,2cyclictest0-21swapper/223:05:132
4157602992926,2cyclictest0-21swapper/222:45:112
415760299290,2cyclictest0-21swapper/221:35:122
415760299290,28cyclictest0-21swapper/222:20:112
415760299290,27cyclictest0-21swapper/222:00:022
415760299290,27cyclictest0-21swapper/221:23:592
415760299290,27cyclictest0-21swapper/220:40:152
4157600992925,3cyclictest4157746-21latency_hist19:10:031
4157600992925,3cyclictest4157746-21latency_hist19:10:031
4157607992826,1cyclictest0-21swapper/321:59:543
4157607992826,1cyclictest0-21swapper/300:04:593
4157607992825,2cyclictest0-21swapper/322:55:133
4157602992825,2cyclictest0-21swapper/222:55:132
4157602992825,2cyclictest0-21swapper/222:15:102
4157602992825,2cyclictest0-21swapper/222:15:102
4157602992825,2cyclictest0-21swapper/219:36:592
4157602992825,2cyclictest0-21swapper/200:02:592
415760299280,2cyclictest0-21swapper/200:13:002
4667591270,6ptp4l441ktimers/322:10:003
4667591270,6ptp4l441ktimers/320:10:003
4667591270,6ptp4l441ktimers/319:15:003
4667591270,6ptp4l441ktimers/300:20:003
4157607992724,2cyclictest0-21swapper/319:36:583
4157602992725,1cyclictest0-21swapper/222:40:002
4157602992724,2cyclictest0-21swapper/200:15:122
4157602992723,2cyclictest0-21swapper/223:31:592
415760299270,25cyclictest0-21swapper/200:38:592
415760299270,25cyclictest0-21swapper/200:38:582
4667591260,6ptp4l441ktimers/320:50:003
4667591260,5ptp4l441ktimers/322:39:563
4667591260,5ptp4l441ktimers/319:35:013
4157607992623,2cyclictest0-21swapper/323:31:593
415760799260,1cyclictest0-21swapper/320:15:143
4157602992623,2cyclictest0-21swapper/219:30:122
4157602992623,2cyclictest0-21swapper/200:30:112
4157602992623,2cyclictest0-21swapper/200:28:002
4667591250,5ptp4l61790irq/135-eno1-TxRx-022:14:553
4667591250,5ptp4l441ktimers/300:39:563
4667591250,5ptp4l441ktimers/300:39:563
4667591250,5ptp4l441ktimers/300:29:563
4157602992522,2cyclictest0-21swapper/220:08:592
4157602992522,2cyclictest0-21swapper/219:11:592
415760299250,2cyclictest0-21swapper/220:15:132
4667591240,5ptp4l441ktimers/323:24:553
4157607992422,1cyclictest0-21swapper/300:30:103
4157602992421,2cyclictest0-21swapper/223:23:592
4667591230,4ptp4l61790irq/135-eno1-TxRx-019:59:563
4667591230,4ptp4l61790irq/135-eno1-TxRx-019:59:563
4157602992321,1cyclictest0-21swapper/219:59:592
4157602992321,1cyclictest0-21swapper/219:59:592
4157602992320,2cyclictest0-21swapper/223:28:592
4157602992220,1cyclictest0-21swapper/223:04:592
4157607992119,1cyclictest0-21swapper/323:28:593
4157602991614,1cyclictest0-21swapper/222:15:002
4667591130,4ptp4l441ktimers/323:45:203
4157602991310,2cyclictest0-21swapper/223:49:592
41576029990,8cyclictest0-21swapper/222:09:432
41576029973,3cyclictest71-21kworker/2:1+events20:45:102
41576029940,2cyclictest0-21swapper/219:10:032
41576029940,2cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional