You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-23 - 22:29
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot6.osadl.org (updated Tue Apr 23, 2024 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
98260999126105,19cyclictest1075932-21apt-get12:00:001
98260599119108,8cyclictest1071165-21latency_hist11:45:010
98260999118104,12cyclictest1004845-21kworker/u8:2+flush-179:009:15:001
9826059911098,9cyclictest1082555-21latency_hist12:20:020
98260999106100,4cyclictest1014952-21kworker/u8:1+flush-179:008:50:011
9826099910598,5cyclictest334488-21kworker/u8:0+events_unbound07:15:021
9826099910396,5cyclictest1069541-21kworker/u8:0+flush-179:011:40:241
9826099910294,6cyclictest1082705-21gpgv12:20:001
9826059910291,8cyclictest997166-21apt-get07:55:010
982605999984,11cyclictest1034024-21apt-get09:50:000
982609999891,5cyclictest1046044-21kworker/u8:0+flush-179:010:40:261
982605999877,18cyclictest1079632-21apt-get12:10:130
982605999784,10cyclictest1012006-21apt-get08:40:120
982609999683,10cyclictest1034024-21apt-get09:50:011
982605999690,4cyclictest1029492-21apt-get09:35:110
982605999485,6cyclictest1040893-21kworker/u8:2+flush-179:010:35:260
982609999285,5cyclictest1040893-21kworker/u8:2+events_unbound10:20:011
982609999285,5cyclictest1040893-21kworker/u8:2+events_unbound10:20:011
982609999084,4cyclictest1052090-21kworker/u8:3+flush-179:011:10:221
982609999082,5cyclictest997154-21latency_hist07:55:011
982605999083,5cyclictest1026301-21/usr/sbin/munin09:25:270
982605999063,18cyclictest2158010-21systemd-journal09:50:250
982609998881,6cyclictest1075910-21kworker/u8:1+flush-179:012:40:011
982609998782,4cyclictest1042524-21kworker/u8:3+flush-179:010:25:001
982605998780,5cyclictest1056823-21latency_hist11:00:010
982609998675,8cyclictest2158010-21systemd-journal09:19:591
982609998579,4cyclictest1040893-21kworker/u8:2+flush-179:010:50:261
982609998579,4cyclictest1030890-21kworker/u8:4+flush-179:009:40:201
982609998574,8cyclictest529-21rs:main1
982609998480,3cyclictest995603-21kworker/u8:2+events_unbound08:05:251
982609998477,5cyclictest1037691-21kworker/u8:1+events_unbound10:00:001
982609998477,4cyclictest1000596-21kworker/u8:3+flush-179:008:55:241
982605998477,4cyclictest1042481-21kworker/u8:0+flush-179:010:15:270
982605998477,4cyclictest1042481-21kworker/u8:0+flush-179:010:15:260
982605998475,7cyclictest1084420-21latency_hist12:25:000
982609998377,4cyclictest1056880-21kworker/u8:1+flush-179:011:10:011
982609998276,4cyclictest982550-21kworker/u8:1+flush-179:007:40:051
982609998276,4cyclictest1048848-21kworker/u8:1+flush-179:010:50:001
982609998274,5cyclictest1040893-21kworker/u8:2+events_unbound10:10:141
982609998264,15cyclictest1002225-21kworker/u8:0+flush-179:008:15:011
982605998271,9cyclictest990430-21kworker/u8:3+flush-179:007:45:000
982605998270,9cyclictest1000596-21kworker/u8:3+flush-179:008:25:010
982609998074,4cyclictest1061808-21kworker/u8:2+flush-179:011:20:171
982609998073,5cyclictest1080946-21kworker/u8:3+flush-179:012:25:001
982609997870,5cyclictest1027638-21kworker/u8:0+events_unbound09:35:151
982609997772,4cyclictest1067169-21kworker/u8:0+flush-179:011:30:221
982609997772,4cyclictest1067169-21kworker/u8:0+flush-179:011:30:221
982609997771,4cyclictest1004845-21kworker/u8:2+flush-179:008:35:201
982609997771,3cyclictest1006950-21kworker/u8:0+flush-179:008:25:281
982609997754,18cyclictest1002225-21kworker/u8:0+flush-179:008:15:261
982609997671,4cyclictest979518-21kworker/u8:3+events_unbound07:20:211
982605997671,3cyclictest191rcuc/009:35:010
982605997563,10cyclictest1056880-21kworker/u8:1+flush-179:011:20:000
982609997469,4cyclictest1040893-21kworker/u8:2+flush-179:010:30:251
982609997468,5cyclictest998604-21kworker/u8:0+flush-179:008:00:011
982609997468,5cyclictest998604-21kworker/u8:0+flush-179:008:00:001
982609997465,7cyclictest2158010-21systemd-journal08:44:451
982609997465,6cyclictest1004845-21kworker/u8:2+flush-179:009:10:011
982605997468,4cyclictest1006950-21kworker/u8:0+flush-179:008:30:220
982605997364,7cyclictest1003775-21latency_hist08:15:000
982605997358,12cyclictest992125-21apt-get07:40:010
982609997268,3cyclictest1040893-21kworker/u8:2+flush-179:010:09:551
982609997267,4cyclictest1046044-21kworker/u8:0+flush-179:010:39:551
982609997266,4cyclictest1040893-21kworker/u8:2+flush-179:010:55:271
982605997266,4cyclictest1014952-21kworker/u8:1+events_unbound09:00:150
982609997165,4cyclictest1069550-21kworker/u8:2+flush-179:012:30:241
982609997061,7cyclictest992039-21cp07:37:561
982609996862,4cyclictest1015462-21diskstats08:50:131
982609996862,4cyclictest1000596-21kworker/u8:3+flush-179:008:20:131
982609996859,5cyclictest984317-21/usr/sbin/munin07:15:251
982609996660,5cyclictest1075910-21kworker/u8:1+flush-179:012:05:001
982609996659,5cyclictest1034162-21store09:50:011
982605996655,8cyclictest529-21rs:main0
982609996559,4cyclictest1004845-21kworker/u8:2+flush-179:009:25:231
982609996558,5cyclictest1022903-21kworker/u8:0+flush-179:009:20:171
982605996557,6cyclictest1057073-21/usr/sbin/munin11:00:270
982605996554,9cyclictest2158010-21systemd-journal09:45:010
982609996457,5cyclictest1071402-21/usr/sbin/munin11:45:271
98260599643,59cyclictest1-21systemd12:29:450
982609996358,3cyclictest988007-21kworker/u8:0+flush-179:007:25:211
982609996357,5cyclictest1069550-21kworker/u8:2+flush-179:012:10:311
982609996356,5cyclictest1027694-21kworker/u8:1+flush-179:009:30:201
982609996155,4cyclictest1067955-21kworker/u8:1+flush-179:011:35:221
982609996154,5cyclictest1037691-21kworker/u8:1+flush-179:010:04:551
982609996150,9cyclictest1052090-21kworker/u8:3+flush-179:011:30:011
982609996147,13cyclictest1069550-21kworker/u8:2+flush-179:012:30:001
982605996150,9cyclictest985942-21apt-get07:20:090
982605996052,6cyclictest999278-21apt-get08:00:110
982605996051,7cyclictest1055228-21latency_hist10:55:010
982609995947,9cyclictest2158010-21systemd-journal09:00:461
982605995953,4cyclictest1004845-21kworker/u8:2+flush-179:008:45:160
982609995850,6cyclictest985731-21kworker/u8:2+events_unbound07:30:251
982609995751,4cyclictest982550-21kworker/u8:1+flush-179:008:05:001
982605995749,5cyclictest1072749-21latency_hist11:50:010
982605995650,4cyclictest1052090-21kworker/u8:3+events_unbound10:45:010
982605995646,7cyclictest529-21rs:main0
982605995549,4cyclictest982550-21kworker/u8:1+flush-179:007:20:000
982605995546,7cyclictest1037691-21kworker/u8:1+flush-179:010:00:000
982605995544,8cyclictest995603-21kworker/u8:2+flush-179:008:05:210
982605995447,5cyclictest988007-21kworker/u8:0+flush-179:007:30:170
982605995446,6cyclictest1069550-21kworker/u8:2+flush-179:011:55:010
982609995348,4cyclictest1000596-21kworker/u8:3+flush-179:008:30:271
982605995342,9cyclictest1047529-21apt-get10:30:110
982605995330,5cyclictest984110-21systemd-run07:15:000
98260599531,33cyclictest1060425-21grep11:10:140
982605995235,2cyclictest0-21swapper/011:26:590
982605995148,2cyclictest1069541-21kworker/u8:0+events_unbound12:00:130
982605995142,7cyclictest994242-21apt-get07:45:130
982605995044,4cyclictest1004845-21kworker/u8:2+flush-179:009:00:010
982605995027,5cyclictest0-21swapper/010:05:130
982609994944,4cyclictest1056880-21kworker/u8:1+flush-179:011:15:161
982605994937,8cyclictest1041157-21apt-get10:10:100
4652991490,13ptp4l441ktimers/307:10:013
982609994845,2cyclictest1040893-21kworker/u8:2+flush-179:011:01:451
982605994842,5cyclictest1000596-21kworker/u8:3+flush-179:008:50:210
982605994837,7cyclictest529-21rs:main0
982605994638,6cyclictest1084997-21kworker/u8:0+flush-179:012:30:150
982605994637,7cyclictest529-21rs:main0
982605994637,7cyclictest2158010-21systemd-journal07:25:440
982609994538,6cyclictest1069550-21kworker/u8:2+flush-179:011:50:101
982605994538,6cyclictest1040893-21kworker/u8:2+flush-179:010:25:160
982605994538,5cyclictest1052090-21kworker/u8:3+flush-179:011:25:010
982605994536,7cyclictest1023105-21/usr/sbin/munin09:15:160
982605994441,2cyclictest1037691-21kworker/u8:1+events_unbound10:00:290
982605994439,3cyclictest0-21swapper/009:15:010
982605994438,5cyclictest1069550-21kworker/u8:2+flush-179:012:35:150
982605994437,5cyclictest1067955-21kworker/u8:1+flush-179:011:35:150
98260599430,42cyclictest0-21swapper/008:39:560
982610994239,2cyclictest0-21swapper/211:19:552
982605994236,4cyclictest995603-21kworker/u8:2+flush-179:008:00:010
982605994236,4cyclictest995603-21kworker/u8:2+flush-179:008:00:010
982610994138,2cyclictest0-21swapper/209:35:592
982609994136,3cyclictest995599-21systemd-run07:50:011
982609994134,4cyclictest1040893-21kworker/u8:2+flush-179:010:25:241
982612994038,1cyclictest0-21swapper/311:19:553
982612994034,5cyclictest988871-21turbostat07:29:593
98261299400,38cyclictest987287-21turbostat07:25:003
98261299400,38cyclictest1003784-21turbostat08:15:003
982610994037,2cyclictest0-21swapper/209:50:032
98260599407,32cyclictest1024963-21cat09:20:160
982605994034,5cyclictest1004845-21kworker/u8:2+flush-179:008:15:280
982605994031,7cyclictest1021282-21latency_hist09:10:010
982612993937,1cyclictest0-21swapper/309:35:593
982612993936,2cyclictest0-21swapper/309:15:113
982612993933,5cyclictest1080978-21turbostat12:15:013
98261299390,37cyclictest0-21swapper/310:49:563
982610993936,2cyclictest0-21swapper/209:15:122
98261099390,36cyclictest0-21swapper/211:25:592
98261099390,36cyclictest0-21swapper/207:47:582
98261099390,1cyclictest0-21swapper/209:44:542
982605993931,6cyclictest1056880-21kworker/u8:1+events_unbound11:10:010
982612993831,6cyclictest0-21swapper/309:53:593
98261299380,37cyclictest0-21swapper/310:54:593
98261299380,37cyclictest0-21swapper/307:47:583
98261299380,36cyclictest0-21swapper/309:21:583
98261299380,1cyclictest0-21swapper/312:04:553
98261299380,1cyclictest0-21swapper/311:28:593
982610993834,2cyclictest0-21swapper/210:28:592
982610993834,2cyclictest0-21swapper/209:28:592
98261099380,36cyclictest0-21swapper/212:13:592
98261099380,36cyclictest0-21swapper/210:55:002
98261099380,36cyclictest0-21swapper/209:05:592
98261099380,35cyclictest0-21swapper/209:21:582
98261099380,29cyclictest0-21swapper/210:49:552
982612993735,1cyclictest0-21swapper/307:56:593
982612993735,1cyclictest0-21swapper/307:56:583
982612993730,6cyclictest1039327-21turbostat10:05:013
98261299370,36cyclictest0-21swapper/309:05:593
982610993733,2cyclictest0-21swapper/212:09:552
982610993733,2cyclictest0-21swapper/208:40:592
98261099370,36cyclictest0-21swapper/210:15:112
98261099370,36cyclictest0-21swapper/210:15:112
98261099370,2cyclictest0-21swapper/212:04:552
982605993729,6cyclictest1052090-21kworker/u8:3+flush-179:011:30:150
982605993729,6cyclictest1052090-21kworker/u8:3+flush-179:011:30:150
982612993634,1cyclictest0-21swapper/310:55:593
982612993634,1cyclictest0-21swapper/308:45:593
982612993633,2cyclictest0-21swapper/310:28:593
982612993633,2cyclictest0-21swapper/309:29:003
98261299360,35cyclictest0-21swapper/310:15:123
98261299360,35cyclictest0-21swapper/310:15:113
98261299360,2cyclictest0-21swapper/311:45:013
98261299360,1cyclictest0-21swapper/311:14:593
98261299360,1cyclictest0-21swapper/309:44:543
982610993634,1cyclictest0-21swapper/207:30:002
982610993627,6cyclictest0-21swapper/211:59:002
98261099360,33cyclictest0-21swapper/208:35:162
98261099360,2cyclictest0-21swapper/211:10:142
98261099360,1cyclictest0-21swapper/207:25:002
982612993529,5cyclictest1002201-21turbostat08:10:003
982610993532,2cyclictest0-21swapper/208:47:592
982610993532,2cyclictest0-21swapper/208:21:002
982610993531,2cyclictest0-21swapper/211:49:552
982610993531,2cyclictest0-21swapper/211:32:002
982610993531,2cyclictest0-21swapper/211:32:002
982610993531,2cyclictest0-21swapper/207:40:002
982605993528,6cyclictest1075910-21kworker/u8:1+events_unbound12:05:140
4652991350,3ptp4l61790irq/134-eno1-TxRx-011:50:113
982612993432,1cyclictest0-21swapper/312:09:553
982612993432,1cyclictest0-21swapper/308:40:583
982610993432,1cyclictest0-21swapper/207:44:592
98261099340,2cyclictest0-21swapper/211:45:012
982612993331,1cyclictest0-21swapper/307:44:583
982612993330,2cyclictest0-21swapper/308:21:003
982612993330,2cyclictest0-21swapper/307:40:003
982612993327,5cyclictest0-21swapper/311:59:003
982612993327,5cyclictest0-21swapper/311:35:123
98261299330,26cyclictest0-21swapper/311:05:133
982610993331,1cyclictest0-21swapper/208:10:002
98261099330,31cyclictest0-21swapper/209:00:142
98261099330,2cyclictest0-21swapper/210:35:132
98261099330,2cyclictest0-21swapper/210:04:542
98261099330,27cyclictest0-21swapper/210:05:142
98261099330,26cyclictest0-21swapper/211:35:112
982612993230,1cyclictest0-21swapper/311:49:553
982612993229,2cyclictest0-21swapper/308:38:003
982612993226,4cyclictest1048866-21turbostat10:35:003
98261299320,24cyclictest0-21swapper/309:48:593
982610993229,2cyclictest0-21swapper/208:14:552
982610993229,2cyclictest0-21swapper/207:35:012
982610993226,5cyclictest0-21swapper/208:50:102
98261099320,30cyclictest0-21swapper/210:20:152
98261099320,30cyclictest0-21swapper/209:30:142
98261099320,2cyclictest0-21swapper/212:35:142
98261099320,2cyclictest0-21swapper/212:23:592
98261099320,19cyclictest0-21swapper/211:05:132
982612993129,1cyclictest0-21swapper/311:32:003
982612993129,1cyclictest0-21swapper/311:31:593
982612993124,6cyclictest0-21swapper/310:05:123
98261299310,29cyclictest0-21swapper/309:00:133
98261299310,20cyclictest0-21swapper/310:35:133
98261099310,30cyclictest0-21swapper/212:25:102
98261099310,30cyclictest0-21swapper/209:15:002
98261099310,2cyclictest0-21swapper/209:48:592
98261099310,29cyclictest0-21swapper/208:30:102
98261099310,28cyclictest0-21swapper/212:20:012
982612993027,2cyclictest0-21swapper/307:30:143
982612993024,5cyclictest0-21swapper/312:35:133
98261299300,2cyclictest0-21swapper/309:55:013
98261299300,2cyclictest0-21swapper/308:00:143
98261299300,29cyclictest0-21swapper/310:10:123
98261299300,29cyclictest0-21swapper/308:30:103
98261299300,28cyclictest0-21swapper/312:25:113
98261299300,28cyclictest0-21swapper/309:30:133
982610993027,2cyclictest0-21swapper/211:22:592
98261099300,2cyclictest0-21swapper/211:55:012
98261099300,2cyclictest0-21swapper/210:10:112
98261099300,2cyclictest0-21swapper/208:00:132
98261099300,28cyclictest0-21swapper/209:55:022
98261099300,27cyclictest0-21swapper/210:55:172
98261099300,1cyclictest0-21swapper/211:00:162
982612992927,1cyclictest0-21swapper/312:20:013
982612992927,1cyclictest0-21swapper/311:22:593
982612992926,2cyclictest0-21swapper/310:42:593
982612992926,2cyclictest0-21swapper/308:50:113
98261299290,28cyclictest0-21swapper/309:10:153
98261299290,27cyclictest0-21swapper/310:20:153
98261299290,1cyclictest0-21swapper/312:23:583
982610992926,2cyclictest0-21swapper/210:40:152
98261099290,28cyclictest0-21swapper/208:25:132
98261099290,28cyclictest0-21swapper/207:55:192
98261099290,28cyclictest0-21swapper/207:55:192
982605992923,4cyclictest982465-21latency_hist07:10:020
98261299280,27cyclictest0-21swapper/312:30:123
98261299280,27cyclictest0-21swapper/311:00:163
98261299280,27cyclictest0-21swapper/308:25:123
982610992821,6cyclictest0-21swapper/210:30:122
98261099280,27cyclictest0-21swapper/212:30:122
982612992724,2cyclictest0-21swapper/307:10:143
982610992725,1cyclictest0-21swapper/207:10:152
982610992724,2cyclictest0-21swapper/209:00:002
982610992723,2cyclictest0-21swapper/207:52:592
4652991270,6ptp4l61790irq/134-eno1-TxRx-008:59:563
4652991270,6ptp4l441ktimers/307:20:013
98261299260,24cyclictest0-21swapper/308:15:593
98261099250,24cyclictest0-21swapper/208:15:582
4652991250,4ptp4l441ktimers/307:54:563
982610992319,2cyclictest0-21swapper/207:15:582
982609992212,9cyclictest0-21swapper/107:10:011
9826109930,2cyclictest0-21swapper/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional