You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-10-10 - 19:00
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot6.osadl.org (updated Thu Oct 10, 2024 12:45:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30129299118108,7cyclictest393806-21latency_hist12:00:011
3012929910189,8cyclictest298132-21kworker/u8:0+flush-179:007:20:011
301289999889,7cyclictest316055-21cat07:55:000
301289999785,9cyclictest325679-21latency_hist08:25:010
301292999683,10cyclictest384241-21/usr/sbin/munin11:25:231
301289998979,8cyclictest3105085-21systemd-journal07:45:000
301289998979,8cyclictest3105085-21systemd-journal07:44:590
301289998882,5cyclictest309140-21kworker/u8:4+flush-179:007:40:010
301292998676,7cyclictest317450-21latency_hist08:00:011
301289998370,10cyclictest312670-21/usr/sbin/munin07:45:230
301289998273,7cyclictest388765-21latency_hist11:40:010
301289998271,8cyclictest395420-21latency_hist12:30:010
301289998068,9cyclictest380550-21latency_hist11:15:010
301292997969,7cyclictest302768-21latency_hist07:15:011
301289997873,4cyclictest364117-21kworker/u8:0+flush-179:010:35:010
301292997769,6cyclictest3105085-21systemd-journal11:00:011
301289997771,4cyclictest307583-21kworker/u8:3+events_unbound08:00:250
301289997769,5cyclictest1-21systemd12:25:000
301289997665,8cyclictest335472-21latency_hist08:55:020
30128999750,74cyclictest0-21swapper/009:04:590
301292997462,9cyclictest3105085-21systemd-journal12:25:001
301289997365,5cyclictest343491-21kworker/u8:2+events_unbound10:20:240
301289997064,4cyclictest340487-21kworker/u8:1+flush-179:009:15:300
301292996857,9cyclictest328977-21gpgv08:35:011
301292996857,9cyclictest328977-21gpgv08:35:011
301289996759,6cyclictest303825-21kworker/u8:1+flush-179:007:15:230
301289996755,10cyclictest340506-21apt-get09:10:140
301289996754,9cyclictest382153-21latency_hist11:20:010
30129399660,63cyclictest0-21swapper/209:42:592
301292996452,9cyclictest3105085-21systemd-journal11:35:301
301289996457,6cyclictest343491-21kworker/u8:2+events_unbound10:05:220
301292996351,9cyclictest3105085-21systemd-journal07:20:141
301289996357,4cyclictest382655-21kworker/u8:0+flush-179:011:40:180
301289996352,9cyclictest393772-21kworker/u8:0+flush-179:012:20:010
301289996352,9cyclictest393772-21kworker/u8:0+flush-179:012:20:000
301292996254,6cyclictest368897-21apt-get10:40:011
301292996254,6cyclictest368897-21apt-get10:40:001
301289996255,4cyclictest372376-21kworker/u8:3+flush-179:011:00:210
301289996252,8cyclictest385826-21/usr/sbin/munin11:30:250
301292996052,6cyclictest3105085-21systemd-journal12:33:311
301292996050,8cyclictest322722-21/usr/sbin/munin08:15:241
301289995852,4cyclictest335505-21kworker/u8:3+flush-179:009:25:150
301289995847,7cyclictest3105085-21systemd-journal11:20:010
301289995752,3cyclictest304411-21kworker/u8:2+flush-179:007:30:150
301292995647,7cyclictest364166-21kworker/u8:1+flush-179:010:25:231
301292995647,6cyclictest502-21rs:main1
301292995647,6cyclictest502-21rs:main1
301292995644,10cyclictest390402-21kworker/u8:3+flush-179:011:45:261
301289995649,5cyclictest356413-21kworker/u8:1+events_unbound10:10:250
301289995643,6cyclictest364166-21kworker/u8:1+events_unbound10:35:150
301289995643,6cyclictest364166-21kworker/u8:1+events_unbound10:35:150
301289995643,10cyclictest343491-21kworker/u8:2+flush-179:010:00:010
301292995544,9cyclictest370992-21/usr/sbin/munin10:45:231
301292995543,10cyclictest3105085-21systemd-journal10:00:001
301289995549,4cyclictest335505-21kworker/u8:3+flush-179:009:30:150
301289995546,7cyclictest394085-21latency_hist12:05:010
301289995546,7cyclictest327298-21kworker/u8:0+flush-179:008:35:010
301289995546,7cyclictest327298-21kworker/u8:0+flush-179:008:35:000
301292995447,5cyclictest343684-21/usr/sbin/munin09:20:171
301289995448,5cyclictest315623-21kworker/u8:2+events_unbound08:25:250
301289995445,7cyclictest319693-21kworker/u8:0+flush-179:008:05:210
301289995445,7cyclictest319693-21kworker/u8:0+flush-179:008:05:210
301289995346,5cyclictest343491-21kworker/u8:2+events_unbound09:35:160
301289995346,4cyclictest315623-21kworker/u8:2+flush-179:008:45:200
301289995345,6cyclictest3105085-21systemd-journal09:55:000
301292995242,6cyclictest343491-21kworker/u8:2+flush-179:009:35:251
301289995246,5cyclictest315623-21kworker/u8:2+events_unbound08:35:260
301289995144,5cyclictest348182-21kworker/u8:0+flush-179:009:45:220
301289995144,5cyclictest348182-21kworker/u8:0+flush-179:009:45:210
301292995041,7cyclictest319293-21/usr/sbin/munin08:05:171
301292995041,7cyclictest319293-21/usr/sbin/munin08:05:161
301292995039,9cyclictest361172-21/usr/sbin/munin10:15:211
301289995043,5cyclictest327298-21kworker/u8:0+flush-179:009:00:000
4505291490,13ptp4l61790irq/134-eno1-TxRx-007:10:003
4505291490,13ptp4l61790irq/134-eno1-TxRx-007:10:003
301292994939,8cyclictest372596-21apt-get10:50:121
301292994936,11cyclictest314272-21apt-get07:50:111
30129299491,44cyclictest0-21swapper/108:25:121
301292994840,6cyclictest335505-21kworker/u8:3+flush-179:009:05:221
301292994831,14cyclictest0-21swapper/108:59:551
301292994738,7cyclictest357765-21latency_hist10:05:011
301289994740,6cyclictest390402-21kworker/u8:3+flush-179:011:50:220
301289994740,6cyclictest390402-21kworker/u8:3+flush-179:011:50:220
301292994635,8cyclictest3105085-21systemd-journal09:45:301
301292994635,8cyclictest3105085-21systemd-journal09:45:301
30129299461,42cyclictest389035-21apt-get11:40:141
301289994638,6cyclictest307583-21kworker/u8:3+events_unbound08:10:230
301289994632,10cyclictest3105085-21systemd-journal12:36:300
301292994539,5cyclictest393772-21kworker/u8:0+flush-179:012:20:011
301292994539,5cyclictest393772-21kworker/u8:0+flush-179:012:20:011
301292994538,6cyclictest327298-21kworker/u8:0+flush-179:008:50:011
30129299450,44cyclictest0-21swapper/108:35:131
301289994540,4cyclictest335505-21kworker/u8:3+flush-179:009:20:210
301289994540,4cyclictest298132-21kworker/u8:0+events_unbound07:10:240
301289994539,5cyclictest350050-21kworker/u8:1+flush-179:009:40:210
301289994536,7cyclictest364117-21kworker/u8:0+events_unbound11:05:130
301292994434,8cyclictest346620-21latency_hist09:30:001
301289994439,4cyclictest364117-21kworker/u8:0+events_unbound10:40:190
301289994439,4cyclictest364117-21kworker/u8:0+events_unbound10:40:190
301293994340,2cyclictest0-21swapper/212:19:562
301293994340,2cyclictest0-21swapper/212:19:562
30129299430,41cyclictest722-21Xorg09:03:001
301289994334,7cyclictest327298-21kworker/u8:0+events_unbound09:10:010
30128999431,39cyclictest305965-21cron07:25:000
30129599420,1cyclictest0-21swapper/312:24:593
301292994236,5cyclictest350050-21kworker/u8:1+flush-179:009:40:241
301292994235,5cyclictest722-21Xorg07:27:591
301292994232,7cyclictest382461-21apt-key11:20:011
301292994230,10cyclictest334141-21apt-get08:50:121
301292994228,12cyclictest317723-21apt-get08:00:131
30129299420,41cyclictest0-21swapper/107:40:131
30129299420,41cyclictest0-21swapper/107:40:121
30129299420,40cyclictest722-21Xorg09:33:591
301289994237,3cyclictest391954-21munin-plugin-st11:50:000
30128999420,40cyclictest722-21Xorg10:15:590
301292994133,6cyclictest343491-21kworker/u8:2+flush-179:009:50:231
301289994135,4cyclictest307560-21perf07:30:000
301289994129,9cyclictest3105085-21systemd-journal11:28:300
30128999410,39cyclictest315606-21grep07:55:000
301292994030,8cyclictest502-21rs:main1
301289994034,5cyclictest395696-21awk12:34:590
301289994031,8cyclictest357768-21munin-run10:05:000
30129599390,37cyclictest365728-21turbostat10:30:003
30129399390,36cyclictest0-21swapper/212:36:582
301292993928,9cyclictest362800-21apt-get10:20:101
30128999390,2cyclictest0-21swapper/010:26:590
301295993836,1cyclictest0-21swapper/310:37:593
301295993836,1cyclictest0-21swapper/310:37:593
301295993835,2cyclictest0-21swapper/308:25:593
30129399380,36cyclictest0-21swapper/211:22:592
30129399380,35cyclictest0-21swapper/208:06:002
30129399380,35cyclictest0-21swapper/208:05:592
301292993835,2cyclictest722-21Xorg12:03:591
301292993834,2cyclictest377638-21apt-get11:05:121
301292993825,11cyclictest0-21swapper/107:35:001
301292993821,16cyclictest359972-21fschecks_time10:10:151
30129299381,34cyclictest382530-21sed11:20:011
301295993735,1cyclictest0-21swapper/308:17:593
30129599370,36cyclictest0-21swapper/311:28:593
30129599370,36cyclictest0-21swapper/309:24:553
301293993734,2cyclictest0-21swapper/208:05:002
30129399370,35cyclictest0-21swapper/208:38:002
30129399370,35cyclictest0-21swapper/207:56:592
30129399370,35cyclictest0-21swapper/207:52:002
30129399370,1cyclictest0-21swapper/210:33:592
301292993734,2cyclictest315623-21kworker/u8:2+events_unbound08:15:011
301292993734,2cyclictest0-21swapper/111:34:591
301289993732,4cyclictest315623-21kworker/u8:2+events_unbound08:15:230
301289993731,5cyclictest393543-21turbostat11:59:560
301295993634,1cyclictest0-21swapper/307:34:543
30129599360,35cyclictest0-21swapper/311:09:563
30129599360,1cyclictest0-21swapper/308:58:583
30129599360,1cyclictest0-21swapper/308:31:593
30129599360,1cyclictest0-21swapper/308:31:583
301293993633,2cyclictest0-21swapper/209:49:002
301293993633,2cyclictest0-21swapper/209:48:592
30129399360,1cyclictest0-21swapper/210:35:592
30129399360,1cyclictest0-21swapper/210:35:582
301292993633,2cyclictest0-21swapper/110:10:001
301292993631,4cyclictest394099-21turbostat12:09:551
301292993629,4cyclictest1-21systemd12:14:291
301292993628,6cyclictest364117-21kworker/u8:0+flush-179:011:00:231
301292993621,14cyclictest0-21swapper/107:38:001
301292993618,16cyclictest722-21Xorg12:27:591
301289993632,3cyclictest330750-21kworker/u8:1+flush-179:008:40:150
301289993626,7cyclictest502-21rs:main0
301295993533,1cyclictest0-21swapper/307:54:553
301295993532,2cyclictest0-21swapper/309:38:583
301293993532,2cyclictest0-21swapper/211:29:552
301293993532,2cyclictest0-21swapper/209:23:582
30129399350,1cyclictest0-21swapper/210:50:002
30129399350,1cyclictest0-21swapper/209:59:592
301289993531,3cyclictest364117-21kworker/u8:0+flush-179:010:50:480
301295993429,4cyclictest335479-21turbostat08:55:013
301293993431,2cyclictest0-21swapper/208:14:562
301293993431,2cyclictest0-21swapper/207:15:592
301292993426,7cyclictest301157-21latency_hist07:10:021
301292993426,7cyclictest301157-21latency_hist07:10:011
301289993427,5cyclictest394329-21kworker/u8:1+events_unbound12:15:010
30128999342,30cyclictest371010-21apt-get10:45:120
301295993331,1cyclictest0-21swapper/308:42:583
301295993330,2cyclictest0-21swapper/311:54:593
301295993330,2cyclictest0-21swapper/311:54:593
301293993330,2cyclictest0-21swapper/209:34:552
301293993330,2cyclictest0-21swapper/209:13:592
301293993330,2cyclictest0-21swapper/208:47:592
301293993330,2cyclictest0-21swapper/208:26:592
301293993330,2cyclictest0-21swapper/207:43:592
301293993330,2cyclictest0-21swapper/207:43:592
301292993329,3cyclictest343449-21munin-run09:20:001
301292993324,7cyclictest3105085-21systemd-journal12:40:011
301295993230,1cyclictest0-21swapper/307:49:593
301293993229,2cyclictest0-21swapper/210:16:592
301292993227,4cyclictest313709-21switchtime07:45:231
301292993225,5cyclictest722-21Xorg11:11:591
30129299320,30cyclictest688-21snmpd10:30:131
301295993129,1cyclictest0-21swapper/307:21:593
301295993125,5cyclictest364138-21turbostat10:25:003
301293993128,2cyclictest0-21swapper/212:07:582
30129399310,2cyclictest0-21swapper/208:43:592
301292993113,3cyclictest341274-21ntp_kernel_pll_09:10:231
301289993127,3cyclictest374887-21needreboot10:55:190
301295993027,2cyclictest375524-21turbostat11:00:013
301295993023,6cyclictest394624-21turbostat12:15:013
30129599300,28cyclictest0-21swapper/309:45:143
30129599300,28cyclictest0-21swapper/309:45:133
30129599300,28cyclictest0-21swapper/309:00:113
301293993028,1cyclictest0-21swapper/211:14:592
301293993027,2cyclictest0-21swapper/208:55:132
30129399300,2cyclictest0-21swapper/209:50:142
30129399300,28cyclictest0-21swapper/209:00:122
30129399300,27cyclictest0-21swapper/212:12:592
4505291290,7ptp4l441ktimers/307:15:013
301295992927,1cyclictest0-21swapper/309:55:133
301295992927,1cyclictest0-21swapper/309:15:133
301295992926,2cyclictest388778-21turbostat11:40:003
301295992926,2cyclictest0-21swapper/311:00:023
30129599290,28cyclictest0-21swapper/310:15:113
30129599290,27cyclictest0-21swapper/310:05:123
30129599290,27cyclictest0-21swapper/308:45:123
30129599290,1cyclictest0-21swapper/307:25:133
301293992926,2cyclictest0-21swapper/211:30:132
301293992926,2cyclictest0-21swapper/211:15:132
301293992926,2cyclictest0-21swapper/209:35:132
301293992926,2cyclictest0-21swapper/209:05:592
301293992926,2cyclictest0-21swapper/207:30:122
30129399290,28cyclictest0-21swapper/209:15:142
30129399290,27cyclictest0-21swapper/211:42:592
30129399290,27cyclictest0-21swapper/210:40:032
30129399290,27cyclictest0-21swapper/210:40:032
30129399290,27cyclictest0-21swapper/208:51:592
30129399290,27cyclictest0-21swapper/208:31:592
30129399290,27cyclictest0-21swapper/208:31:582
30129399290,1cyclictest0-21swapper/211:00:012
301292992926,2cyclictest722-21Xorg08:43:591
30129299291,26cyclictest722-21Xorg11:54:591
30129299291,26cyclictest722-21Xorg11:54:591
201699290,17rtkit-daemon299875-21apt-get07:05:110
201699290,17rtkit-daemon299875-21apt-get07:05:110
4505291280,7ptp4l441ktimers/311:50:013
4505291280,7ptp4l441ktimers/307:20:013
301295992826,1cyclictest0-21swapper/311:15:133
301295992826,1cyclictest0-21swapper/308:20:133
301295992825,2cyclictest0-21swapper/310:11:593
30129599280,27cyclictest0-21swapper/311:10:123
30129599280,27cyclictest0-21swapper/309:10:163
30129599280,26cyclictest0-21swapper/308:10:023
301293992825,2cyclictest0-21swapper/212:29:552
301293992825,2cyclictest0-21swapper/211:50:132
301293992825,2cyclictest0-21swapper/211:50:132
301293992825,2cyclictest0-21swapper/210:10:122
30129399280,27cyclictest0-21swapper/211:00:022
30129399280,27cyclictest0-21swapper/207:35:102
30129399280,26cyclictest0-21swapper/210:50:582
30129399280,26cyclictest0-21swapper/210:20:122
30129399280,26cyclictest0-21swapper/207:20:132
4505291270,7ptp4l61790irq/134-eno1-TxRx-009:29:563
4505291270,7ptp4l441ktimers/308:05:003
301295992725,1cyclictest0-21swapper/312:37:593
301295992725,1cyclictest0-21swapper/307:55:033
301295992724,2cyclictest0-21swapper/309:52:583
30129599270,1cyclictest0-21swapper/307:35:123
301293992725,1cyclictest0-21swapper/212:21:592
301293992725,1cyclictest0-21swapper/211:59:592
301293992725,1cyclictest0-21swapper/207:25:132
301293992724,2cyclictest0-21swapper/210:02:002
301293992724,2cyclictest0-21swapper/208:15:132
4505291260,7ptp4l61790irq/134-eno1-TxRx-009:45:013
4505291260,6ptp4l441ktimers/308:09:563
4505291260,6ptp4l441ktimers/308:09:563
4505291260,5ptp4l441ktimers/312:30:013
4505291260,4ptp4l441ktimers/312:05:003
4505291260,4ptp4l441ktimers/310:55:003
301295992623,2cyclictest0-21swapper/309:34:593
30129599260,25cyclictest0-21swapper/308:37:003
301293992623,2cyclictest0-21swapper/211:48:592
4505291250,6ptp4l61790irq/134-eno1-TxRx-012:19:563
4505291250,6ptp4l61790irq/134-eno1-TxRx-012:19:563
4505291250,6ptp4l61790irq/134-eno1-TxRx-010:50:003
4505291250,6ptp4l441ktimers/310:34:563
4505291250,5ptp4l441ktimers/311:44:553
4505291250,5ptp4l441ktimers/310:45:003
4505291250,5ptp4l441ktimers/310:45:003
4505291250,4ptp4l441ktimers/312:00:013
30129599250,2cyclictest0-21swapper/307:40:133
30129599250,2cyclictest0-21swapper/307:40:123
301293992522,2cyclictest0-21swapper/207:47:582
30129399250,2cyclictest0-21swapper/210:07:582
4505291240,4ptp4l61790irq/134-eno1-TxRx-009:10:013
4505291240,4ptp4l441ktimers/312:35:003
4505291240,4ptp4l441ktimers/311:34:563
301295992422,1cyclictest0-21swapper/311:24:553
301295992422,1cyclictest0-21swapper/310:03:593
4505291230,4ptp4l441ktimers/312:09:553
301293992321,1cyclictest0-21swapper/211:35:122
301293992220,1cyclictest0-21swapper/209:28:592
301293992119,1cyclictest0-21swapper/210:30:002
301293992118,2cyclictest0-21swapper/208:20:592
301293992016,2cyclictest0-21swapper/211:09:562
301293991714,2cyclictest0-21swapper/212:04:552
3012939970,6cyclictest0-21swapper/207:13:052
3012939950,3cyclictest395434-21turbostat12:34:552
3012939930,2cyclictest0-21swapper/207:10:012
3012939930,2cyclictest0-21swapper/207:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional