You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-26 - 12:13
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot6 (updated Tue Sep 26, 2023 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1372047999581,10cyclictest2158010-21systemd-journal23:35:010
1372047999284,5cyclictest1380223-21latency_hist19:35:010
1372047999184,4cyclictest1385297-21kworker/u8:1+events_unbound20:35:010
1372047998881,5cyclictest1380252-21kworker/u8:4+flush-179:020:05:340
1372047998881,5cyclictest1380252-21kworker/u8:4+flush-179:020:05:340
1372050998783,3cyclictest1418918-21kworker/u8:3+events_unbound22:00:351
1372050998679,5cyclictest1456371-21latency_hist23:35:001
1372047998578,4cyclictest1375161-21kworker/u8:2+flush-179:019:20:200
1372047998478,4cyclictest1385297-21kworker/u8:1+flush-179:020:10:370
1372050998375,6cyclictest1446569-21latency_hist23:05:011
1372047998370,9cyclictest2158010-21systemd-journal20:25:010
1372047998272,7cyclictest1452937-21latency_hist23:25:000
1372050998177,3cyclictest1456425-21kworker/u8:2+events_unbound23:45:341
1372050998176,4cyclictest1418918-21kworker/u8:3+events_unbound23:05:341
1372047998175,4cyclictest1385297-21kworker/u8:1+events_unbound21:55:010
1372050997874,3cyclictest1474461-21kworker/u8:1+events_unbound00:30:321
1372047997872,4cyclictest1407835-21kworker/u8:2+flush-179:021:15:350
1372047997872,4cyclictest1407835-21kworker/u8:2+flush-179:021:15:350
1372047997771,4cyclictest1391023-21kworker/u8:2+events_unbound20:20:310
1372047997767,8cyclictest1443573-21apt-get22:55:180
1372047997670,4cyclictest1435085-21kworker/u8:1+events_unbound22:30:000
1372050997570,4cyclictest1391023-21kworker/u8:2+events_unbound20:45:291
1372050997567,6cyclictest1385262-21latency_hist19:55:001
1372050997566,7cyclictest1463235-21/usr/sbin/munin23:55:361
1372047997569,4cyclictest1418918-21kworker/u8:3+flush-179:023:55:370
1372047997569,4cyclictest1418918-21kworker/u8:3+flush-179:022:35:350
1372047997468,4cyclictest1415790-21kworker/u8:2+events_unbound21:30:430
1372050997268,3cyclictest1391023-21kworker/u8:2+events_unbound20:50:281
1372050997268,3cyclictest1378396-21kworker/u8:2+events_unbound19:55:281
1372050997267,4cyclictest1444847-21kworker/u8:2+events_unbound23:15:341
1372047997261,9cyclictest1476269-21apt-get00:35:180
1372047997261,9cyclictest1445233-21apt-get23:00:180
1372047997257,12cyclictest2158010-21systemd-journal19:15:000
1372050997166,4cyclictest1365709-21kworker/u8:3+events_unbound19:25:231
1372047997165,4cyclictest1385297-21kworker/u8:1+flush-179:021:00:240
1372050997065,4cyclictest1397266-21kworker/u8:3+events_unbound20:55:341
1372050997065,4cyclictest1385297-21kworker/u8:1+events_unbound20:00:241
1372050997057,12cyclictest1378396-21kworker/u8:2+events_unbound19:35:231
1372050996963,5cyclictest1463042-21kworker/u8:2+flush-179:000:10:341
1372047996962,5cyclictest1424728-21kworker/u8:0+events_unbound22:25:010
1372047996961,5cyclictest1418918-21kworker/u8:3+events_unbound21:40:300
1372050996863,4cyclictest1415790-21kworker/u8:2+flush-179:021:35:331
1372050996858,7cyclictest1477602-21latency_hist00:40:011
1372050996851,15cyclictest1456425-21kworker/u8:2+flush-179:023:40:311
1372047996859,4cyclictest1418918-21kworker/u8:3+events_unbound22:00:010
1372050996763,3cyclictest1418918-21kworker/u8:3+events_unbound21:55:291
1372050996763,3cyclictest1378396-21kworker/u8:2+flush-179:019:45:251
1372050996761,4cyclictest1424728-21kworker/u8:0+events_unbound22:35:241
1372047996760,5cyclictest1418918-21kworker/u8:3+flush-179:021:45:210
1372050996662,3cyclictest1378396-21kworker/u8:2+flush-179:019:44:551
1372050996661,4cyclictest1375161-21kworker/u8:2+flush-179:019:20:011
1372050996659,5cyclictest1397266-21kworker/u8:3+flush-179:021:10:001
1372047996658,5cyclictest1448408-21apt-get23:10:180
1372050996561,3cyclictest1415790-21kworker/u8:2+flush-179:021:50:361
1372047996558,5cyclictest1418918-21kworker/u8:3+flush-179:000:00:360
1372050996459,4cyclictest1391023-21kworker/u8:2+flush-179:020:40:341
1372050996459,4cyclictest1385297-21kworker/u8:1+flush-179:022:15:341
1372047996458,4cyclictest1413526-21kworker/u8:3+flush-179:021:20:280
1372047996456,6cyclictest1418918-21kworker/u8:3+flush-179:000:05:330
1372050996358,4cyclictest1456425-21kworker/u8:2+events_unbound23:35:331
1372047996358,3cyclictest1407835-21kworker/u8:2+events_unbound21:05:240
1372050996258,3cyclictest1415790-21kworker/u8:2+events_unbound21:30:451
1372050996257,4cyclictest1418918-21kworker/u8:3+events_unbound23:50:311
1372050996257,4cyclictest1412820-21kworker/u8:0+flush-179:021:20:261
1372050996253,6cyclictest1410976-21latency_hist21:15:011
1372047996257,4cyclictest1444847-21kworker/u8:2+flush-179:023:15:370
1372047996249,5cyclictest1407835-21kworker/u8:2+flush-179:021:10:330
1372050996157,3cyclictest1424728-21kworker/u8:0+events_unbound22:05:331
1372050996156,4cyclictest1418918-21kworker/u8:3+flush-179:022:50:341
1372050996156,4cyclictest1418918-21kworker/u8:3+flush-179:022:24:541
1372050996149,10cyclictest1372267-21apt-get19:10:211
1372047996157,3cyclictest1378396-21kworker/u8:2+flush-179:020:01:310
1372047996156,4cyclictest1463042-21kworker/u8:2+flush-179:000:15:260
1372047996154,5cyclictest1412820-21kworker/u8:0+flush-179:021:30:000
1372047996148,4cyclictest1459815-21kworker/u8:0+flush-179:023:50:220
1372050996056,3cyclictest1385297-21kworker/u8:1+events_unbound20:15:291
1372050996055,4cyclictest1418918-21kworker/u8:3+flush-179:022:55:331
1372050996051,6cyclictest2158010-21systemd-journal20:36:291
1372047996052,6cyclictest1428997-21apt-get22:10:170
1372047996050,7cyclictest1463042-21kworker/u8:2+events_unbound00:30:220
1372047995952,5cyclictest1391023-21kworker/u8:2+flush-179:020:40:000
1372047995952,4cyclictest1456425-21kworker/u8:2+flush-179:023:40:230
1372047995951,6cyclictest1438556-21apt-get22:40:210
1372047995946,11cyclictest1404868-21apt-get20:55:200
1372050995853,4cyclictest1380252-21kworker/u8:4+flush-179:020:14:561
1372047995853,3cyclictest1424728-21kworker/u8:0+flush-179:022:50:210
1372047995852,4cyclictest1365709-21kworker/u8:3+flush-179:019:25:250
1372047995849,6cyclictest529-21rs:main0
1372050995753,3cyclictest1472238-21kworker/u8:3+flush-179:000:29:551
1372050995752,4cyclictest1418918-21kworker/u8:3+events_unbound22:10:221
1372050995749,6cyclictest1449740-21latency_hist23:15:011
1372050995748,6cyclictest1418918-21kworker/u8:3+flush-179:022:45:001
1372047995642,5cyclictest1418918-21kworker/u8:3+flush-179:022:05:270
1372050995550,4cyclictest1413526-21kworker/u8:3+flush-179:021:29:551
1372050995550,4cyclictest1385297-21kworker/u8:1+flush-179:020:27:151
1372047995551,3cyclictest1380252-21kworker/u8:4+flush-179:019:43:310
1372050995437,14cyclictest2158010-21systemd-journal22:27:291
1372047995449,4cyclictest1418918-21kworker/u8:3+flush-179:023:31:300
1372047995449,4cyclictest1380252-21kworker/u8:4+flush-179:019:51:290
1372047995448,4cyclictest1380252-21kworker/u8:4+flush-179:019:55:180
1372055995351,1cyclictest0-21swapper/319:12:003
1372050995342,9cyclictest1419205-21apt-get21:40:211
1372050995247,4cyclictest1438901-21kworker/u8:1+events_unbound22:48:301
1372050995247,4cyclictest1391023-21kworker/u8:2+events_unbound20:32:291
1372050995247,4cyclictest1385297-21kworker/u8:1+events_unbound21:15:241
1372050995247,4cyclictest1385297-21kworker/u8:1+events_unbound21:15:241
1372050995147,3cyclictest1444847-21kworker/u8:2+flush-179:023:25:211
1372050995143,6cyclictest1435085-21kworker/u8:1+flush-179:022:30:311
1372047995146,4cyclictest1385297-21kworker/u8:1+flush-179:022:15:350
1372047995144,5cyclictest1418918-21kworker/u8:3+flush-179:022:45:040
1372051995047,2cyclictest0-21swapper/219:42:592
1372050995046,3cyclictest1385297-21kworker/u8:1+events_unbound21:48:301
1372047995045,4cyclictest1391023-21kworker/u8:2+events_unbound20:40:330
4652991480,13ptp4l61790irq/134-eno1-TxRx-019:10:013
1372050994840,6cyclictest1444847-21kworker/u8:2+flush-179:023:25:011
1372050994812,3cyclictest29-21ksoftirqd/120:05:231
1372050994812,3cyclictest29-21ksoftirqd/120:05:221
1372047994743,3cyclictest1372020-21kworker/u8:1+flush-179:019:35:310
1372047994742,4cyclictest1365709-21kworker/u8:3+events_unbound19:15:280
1372050994641,4cyclictest1378396-21kworker/u8:2+events_unbound19:31:381
1372047994638,6cyclictest1403027-21latency_hist20:50:010
1372047994631,5cyclictest1418918-21kworker/u8:3+flush-179:023:45:290
1372050994540,4cyclictest1463042-21kworker/u8:2+flush-179:000:05:301
1372047994540,4cyclictest1385297-21kworker/u8:1+flush-179:021:38:300
1372055994441,2cyclictest0-21swapper/319:43:003
137205199440,1cyclictest0-21swapper/223:05:002
1372050994440,3cyclictest1463042-21kworker/u8:2+flush-179:000:03:011
1372050994436,3cyclictest1385297-21kworker/u8:1+events_unbound21:00:321
1372047994333,8cyclictest529-21rs:main0
1372055994239,2cyclictest0-21swapper/300:36:583
137205199420,39cyclictest0-21swapper/221:00:012
137205199420,1cyclictest0-21swapper/223:37:592
1372047994236,4cyclictest1469624-21kworker/u8:1+events_unbound00:20:220
137204799420,41cyclictest0-21swapper/000:28:000
137205599410,5cyclictest1417361-21turbostat21:35:003
1372047994137,3cyclictest1450277-21kworker/u8:0+flush-179:023:27:300
1372047994134,4cyclictest1435085-21kworker/u8:1+flush-179:022:30:250
137205599400,38cyclictest1467812-21turbostat00:10:013
137205599400,1cyclictest0-21swapper/323:42:593
1372051994037,2cyclictest0-21swapper/219:18:592
1372050994036,3cyclictest1463042-21kworker/u8:2+flush-179:000:15:271
1372050994036,3cyclictest1385297-21kworker/u8:1+events_unbound20:24:291
137205599390,4cyclictest1422124-21turbostat21:50:003
1372051993936,2cyclictest0-21swapper/200:18:592
1372051993931,7cyclictest0-21swapper/200:30:002
1372050993929,8cyclictest1375161-21kworker/u8:2+flush-179:019:20:241
1372055993836,1cyclictest0-21swapper/300:19:543
137205199380,35cyclictest0-21swapper/222:04:592
137205199380,35cyclictest0-21swapper/220:15:002
1372047993834,3cyclictest1418918-21kworker/u8:3+flush-179:022:00:240
1372055993734,2cyclictest0-21swapper/320:21:593
1372055993732,4cyclictest1418960-21turbostat21:40:003
137205599370,36cyclictest0-21swapper/323:19:003
137205599370,36cyclictest0-21swapper/322:22:593
1372051993734,2cyclictest0-21swapper/221:02:582
1372051993734,2cyclictest0-21swapper/219:20:192
137205099371,34cyclictest1471796-21diskstats00:20:221
1372055993634,1cyclictest0-21swapper/321:09:563
1372055993634,1cyclictest0-21swapper/320:32:003
1372055993634,1cyclictest0-21swapper/319:35:193
137205599360,35cyclictest0-21swapper/323:49:003
137205599360,35cyclictest0-21swapper/322:49:583
137205599360,35cyclictest0-21swapper/320:25:593
137205599360,35cyclictest0-21swapper/319:29:003
137205599360,1cyclictest0-21swapper/321:10:593
1372051993633,2cyclictest0-21swapper/222:54:552
1372051993633,2cyclictest0-21swapper/219:14:562
137205199360,2cyclictest0-21swapper/222:05:212
4652991350,3ptp4l441ktimers/319:30:203
4652991350,28ptp4l61790irq/134-eno1-TxRx-023:20:213
1372055993533,1cyclictest0-21swapper/321:18:003
1372055993533,1cyclictest0-21swapper/321:18:003
1372055993532,2cyclictest0-21swapper/321:55:223
1372051993532,2cyclictest0-21swapper/221:53:592
1372051993532,2cyclictest0-21swapper/220:31:592
137205199350,34cyclictest0-21swapper/222:55:192
137205199350,2cyclictest0-21swapper/200:05:012
1372047993520,7cyclictest1378396-21kworker/u8:2+events_unbound19:45:310
137204799350,34cyclictest1392618-21ntpq20:15:290
1372055993432,1cyclictest0-21swapper/323:51:593
137205199340,32cyclictest0-21swapper/222:45:332
137205199340,2cyclictest0-21swapper/219:50:012
137205199340,2cyclictest0-21swapper/200:35:192
137205199340,2cyclictest0-21swapper/200:15:012
137205199340,2cyclictest0-21swapper/200:05:192
137205199340,28cyclictest0-21swapper/221:45:192
137205199340,27cyclictest0-21swapper/222:30:022
137205199340,1cyclictest0-21swapper/223:55:012
137205199340,1cyclictest0-21swapper/200:25:002
1372055993331,1cyclictest0-21swapper/321:44:563
137205199330,2cyclictest0-21swapper/222:15:242
137205199330,2cyclictest0-21swapper/220:15:192
137205199330,2cyclictest0-21swapper/200:32:002
1372047993325,7cyclictest1447559-21ntp_192.168.11423:05:300
1372055993229,2cyclictest0-21swapper/300:35:003
1372055993227,4cyclictest1438300-21turbostat22:40:003
137205599320,31cyclictest0-21swapper/323:05:203
1372051993230,1cyclictest0-21swapper/220:49:592
1372051993224,7cyclictest0-21swapper/220:05:242
1372051993224,7cyclictest0-21swapper/220:05:232
137205199320,30cyclictest0-21swapper/219:30:222
137205199320,2cyclictest0-21swapper/223:25:012
137205199320,1cyclictest0-21swapper/222:40:232
137205199320,1cyclictest0-21swapper/221:59:592
4652991310,5ptp4l61790irq/134-eno1-TxRx-000:25:013
1372055993129,1cyclictest0-21swapper/322:00:313
1372055993129,1cyclictest0-21swapper/321:22:593
1372055993128,2cyclictest0-21swapper/320:53:593
137205599310,30cyclictest0-21swapper/323:55:243
137205599310,29cyclictest0-21swapper/319:55:163
1372051993124,6cyclictest0-21swapper/221:05:222
1372055993027,2cyclictest0-21swapper/320:15:203
1372055993027,2cyclictest0-21swapper/320:15:003
1372055993025,4cyclictest1385004-21turbostat19:49:593
137205599300,2cyclictest0-21swapper/322:40:323
137205599300,29cyclictest0-21swapper/323:25:193
137205599300,28cyclictest0-21swapper/322:50:293
137205599300,27cyclictest0-21swapper/320:40:223
1372051993027,2cyclictest0-21swapper/223:40:202
1372051993027,2cyclictest0-21swapper/223:25:012
1372051993027,2cyclictest0-21swapper/222:35:012
1372051993027,2cyclictest0-21swapper/221:20:202
1372051993027,2cyclictest0-21swapper/220:50:192
1372051993027,2cyclictest0-21swapper/220:43:592
1372051993027,2cyclictest0-21swapper/220:00:232
137205199300,2cyclictest0-21swapper/223:30:242
137205199300,2cyclictest0-21swapper/220:25:032
137205199300,28cyclictest0-21swapper/223:45:222
137205199300,28cyclictest0-21swapper/223:05:242
137205199300,28cyclictest0-21swapper/219:55:242
137205199300,28cyclictest0-21swapper/219:35:182
1372055992927,1cyclictest0-21swapper/320:35:223
1372055992927,1cyclictest0-21swapper/320:05:203
1372055992927,1cyclictest0-21swapper/320:05:203
1372055992926,2cyclictest0-21swapper/323:10:193
1372055992925,2cyclictest0-21swapper/322:30:223
137205599290,28cyclictest0-21swapper/322:10:183
137205599290,28cyclictest0-21swapper/322:05:023
137205599290,28cyclictest0-21swapper/321:50:223
137205599290,28cyclictest0-21swapper/321:25:223
137205599290,28cyclictest0-21swapper/320:05:023
137205599290,28cyclictest0-21swapper/300:00:223
137205599290,27cyclictest0-21swapper/322:55:203
137205599290,27cyclictest0-21swapper/321:00:213
137205599290,27cyclictest0-21swapper/320:45:223
1372051992926,2cyclictest0-21swapper/222:35:232
1372051992926,2cyclictest0-21swapper/222:20:212
1372051992926,2cyclictest0-21swapper/221:25:252
1372051992926,2cyclictest0-21swapper/221:15:232
1372051992926,2cyclictest0-21swapper/221:15:232
1372051992926,2cyclictest0-21swapper/220:35:242
137205199290,28cyclictest0-21swapper/223:55:242
137205199290,28cyclictest0-21swapper/223:10:182
137205199290,27cyclictest0-21swapper/221:40:242
137205199290,27cyclictest0-21swapper/221:36:592
137205199290,27cyclictest0-21swapper/221:30:332
137205199290,27cyclictest0-21swapper/221:10:222
137205199290,27cyclictest0-21swapper/220:20:202
137205199290,21cyclictest0-21swapper/222:10:212
4652991280,7ptp4l441ktimers/323:35:013
1372055992826,1cyclictest0-21swapper/323:35:233
1372055992826,1cyclictest0-21swapper/322:25:293
1372055992826,1cyclictest0-21swapper/300:10:193
137205599280,2cyclictest0-21swapper/319:20:203
137205599280,27cyclictest0-21swapper/320:55:243
137205599280,26cyclictest0-21swapper/322:15:233
137205599280,26cyclictest0-21swapper/319:20:013
1372051992825,2cyclictest0-21swapper/219:53:002
1372051992825,2cyclictest0-21swapper/219:25:232
137205199280,2cyclictest0-21swapper/223:15:022
4652991270,7ptp4l441ktimers/319:54:553
1372055992725,1cyclictest0-21swapper/323:00:213
1372055992725,1cyclictest0-21swapper/300:25:333
1372050992724,2cyclictest281ktimers/119:10:011
1372047991811,5cyclictest1371965-21latency_hist19:10:020
13720519980,1cyclictest0-21swapper/219:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional