You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-28 - 14:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot6 (updated Wed Feb 28, 2024 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
99884599109105,3cyclictest1091302-21kworker/u8:1+flush-179:000:04:021
99884599109101,6cyclictest1027423-21kworker/u8:0+events_unbound21:00:001
9988459910799,6cyclictest1042270-21kworker/u8:1+events_unbound21:50:391
9988459910196,4cyclictest1078106-21kworker/u8:1+events_unbound23:30:321
9988409910067,25cyclictest1100952-21apt-get00:25:190
998845999891,5cyclictest1039322-21kworker/u8:2+events_unbound21:20:011
998840999879,16cyclictest1094323-21latency_hist00:05:010
998845999690,5cyclictest1052126-21kworker/u8:4+events_unbound21:55:391
998840999586,5cyclictest1065078-21latency_hist22:35:010
998845999486,6cyclictest1024240-21kworker/u8:2+flush-179:020:45:381
998845999388,4cyclictest1054314-21kworker/u8:2+flush-179:022:00:411
998840999383,7cyclictest1013044-21latency_hist19:55:010
998840999383,7cyclictest1013044-21latency_hist19:55:010
998845999283,7cyclictest1074898-21latency_hist23:05:011
998845999175,12cyclictest1029101-21gpgv20:45:011
998840999181,7cyclictest1019411-21latency_hist20:15:010
998845999085,4cyclictest1042270-21kworker/u8:1+events_unbound22:20:011
998845999085,4cyclictest1042270-21kworker/u8:1+events_unbound22:20:001
998845998985,3cyclictest1047023-21kworker/u8:0+flush-179:022:50:421
998840998982,5cyclictest1042270-21kworker/u8:1+flush-179:021:25:290
998845998882,4cyclictest1014887-21kworker/u8:0+flush-179:020:00:281
998845998780,5cyclictest1018896-21kworker/u8:3+events_unbound20:30:011
998845998779,6cyclictest1073096-21kworker/u8:2+events_unbound00:20:341
998845998777,7cyclictest1003494-21kworker/u8:3+events_unbound19:25:281
998840998781,4cyclictest1076463-21kworker/u8:3+flush-179:023:20:340
998845998679,6cyclictest1011693-21kworker/u8:1+events_unbound19:55:001
998845998679,6cyclictest1011693-21kworker/u8:1+events_unbound19:54:591
998845998581,3cyclictest1000644-21kworker/u8:2+flush-179:019:15:371
998845998578,5cyclictest1017802-21kworker/u8:1+flush-179:020:20:321
998845998573,5cyclictest1037476-21kworker/u8:3+events_unbound21:40:361
998845998479,4cyclictest1047023-21kworker/u8:0+flush-179:022:05:401
998845998478,4cyclictest1043891-21kworker/u8:2+flush-179:021:30:371
998845998378,4cyclictest977787-21kworker/u8:1+events_unbound19:10:391
998845998377,4cyclictest1017802-21kworker/u8:1+flush-179:020:30:361
998845998370,5cyclictest1073096-21kworker/u8:2+events_unbound23:35:371
998845998275,4cyclictest1000644-21kworker/u8:2+events_unbound19:20:351
998840998268,11cyclictest1073096-21kworker/u8:2+flush-179:023:25:350
998845998176,4cyclictest1081253-21kworker/u8:4+flush-179:023:50:361
998845998176,4cyclictest1065111-21kworker/u8:1+events_unbound22:45:001
998845998170,5cyclictest1014887-21kworker/u8:0+events_unbound20:10:011
998845998076,3cyclictest1047023-21kworker/u8:0+flush-179:022:55:321
998845998076,3cyclictest1037476-21kworker/u8:3+flush-179:021:45:311
998845997974,4cyclictest1027423-21kworker/u8:0+flush-179:021:20:361
998845997971,5cyclictest1086307-21kworker/u8:0+events_unbound00:10:231
998845997873,4cyclictest1095894-21kworker/u8:3+events_unbound00:15:361
998845997873,4cyclictest1095894-21kworker/u8:3+events_unbound00:15:361
998845997872,4cyclictest1091302-21kworker/u8:1+events_unbound00:05:351
998845997868,7cyclictest1081268-21latency_hist23:25:011
998840997873,3cyclictest1003494-21kworker/u8:3+flush-179:019:45:310
998840997870,5cyclictest1042270-21kworker/u8:1+flush-179:022:10:290
998845997773,3cyclictest1100671-21kworker/u8:1+events_unbound00:35:311
998845997773,3cyclictest1100671-21kworker/u8:1+events_unbound00:35:311
998845997670,4cyclictest1047023-21kworker/u8:0+flush-179:022:20:311
998845997670,4cyclictest1013023-21kworker/u8:2+flush-179:019:55:361
998840997666,7cyclictest1068508-21/usr/sbin/munin22:45:240
4652991750,19ptp4l441ktimers/319:07:323
998845997469,4cyclictest1073096-21kworker/u8:2+events_unbound23:10:351
998845997469,4cyclictest1032396-21kworker/u8:1+flush-179:021:10:381
998845997367,5cyclictest1065111-21kworker/u8:1+flush-179:023:05:311
998845997367,5cyclictest1047023-21kworker/u8:0+events_unbound22:35:371
998845997367,4cyclictest1047023-21kworker/u8:0+events_unbound22:30:281
998840997362,8cyclictest1047023-21kworker/u8:0+flush-179:022:50:370
998845997268,3cyclictest1078106-21kworker/u8:1+events_unbound23:25:291
998845997266,5cyclictest1073096-21kworker/u8:2+events_unbound23:40:401
998845997266,4cyclictest1047023-21kworker/u8:0+events_unbound22:25:331
998845997165,4cyclictest1065111-21kworker/u8:1+flush-179:022:45:261
998845997062,5cyclictest1027423-21kworker/u8:0+events_unbound20:50:231
998845996960,4cyclictest1014887-21kworker/u8:0+events_unbound20:10:271
998840996962,5cyclictest1005127-21kworker/u8:0+flush-179:019:40:000
998840996961,5cyclictest1047023-21kworker/u8:0+flush-179:022:35:330
998840996959,7cyclictest1014887-21kworker/u8:0+flush-179:020:05:300
998840996957,9cyclictest1049165-21apt-get21:45:180
998840996863,3cyclictest1027423-21kworker/u8:0+flush-179:021:20:310
998840996856,10cyclictest1035605-21apt-get21:01:560
998845996660,4cyclictest1076463-21kworker/u8:3+events_unbound23:15:271
998840996658,6cyclictest1058945-21/usr/sbin/munin22:15:360
998840996658,6cyclictest1058945-21/usr/sbin/munin22:15:360
998840996552,10cyclictest2158010-21systemd-journal19:34:560
998845996459,4cyclictest1100671-21kworker/u8:1+flush-179:000:30:301
998845996459,4cyclictest1086307-21kworker/u8:0+events_unbound00:25:371
998845996456,6cyclictest1045712-21apt-get21:35:251
998845996455,7cyclictest1005127-21kworker/u8:0+events_unbound19:35:431
998840996456,6cyclictest1002172-21apt-get19:20:190
998840996455,7cyclictest1066922-21/usr/sbin/munin22:40:370
99884099643,60cyclictest1-21systemd00:00:000
998845996358,4cyclictest1037476-21kworker/u8:3+events_unbound21:25:381
998845996348,11cyclictest2531033-21cups-browsed00:00:011
998840996354,7cyclictest2158010-21systemd-journal19:45:000
998840996354,7cyclictest1027423-21kworker/u8:0+events_unbound20:45:020
998840996352,9cyclictest1050748-21apt-get21:50:270
998840996242,17cyclictest1084697-21apt-get23:35:250
998845996158,2cyclictest1017802-21kworker/u8:1+flush-179:020:40:001
998845996049,9cyclictest2158010-21systemd-journal21:10:001
998840996051,7cyclictest1001897-21latency_hist19:20:010
998840996051,6cyclictest1043891-21kworker/u8:2+events_unbound21:40:280
998850995957,1cyclictest0-21swapper/300:18:593
998850995957,1cyclictest0-21swapper/300:18:593
998845995954,4cyclictest1005127-21kworker/u8:0+flush-179:019:40:361
998840995948,8cyclictest1024240-21kworker/u8:2+flush-179:020:35:350
998840995851,5cyclictest1055318-21gpgv22:05:000
998840995851,5cyclictest1032396-21kworker/u8:1+flush-179:021:10:320
998840995850,6cyclictest1073096-21kworker/u8:2+flush-179:000:20:360
998840995850,5cyclictest1054314-21kworker/u8:2+events_unbound22:05:010
998845995651,4cyclictest1018896-21kworker/u8:3+events_unbound20:15:381
998845995650,4cyclictest1003494-21kworker/u8:3+events_unbound19:45:251
998845995646,8cyclictest1057359-21/usr/sbin/munin22:10:371
998840995651,4cyclictest2158010-21systemd-journal19:15:000
998840995650,4cyclictest1081253-21kworker/u8:4+events_unbound23:45:250
998840995548,5cyclictest1083091-21kworker/u8:0+flush-179:023:30:240
998840995545,8cyclictest1024240-21kworker/u8:2+flush-179:020:50:330
998845995448,4cyclictest1005347-21apt-get19:30:241
998840995344,7cyclictest2158010-21systemd-journal23:14:560
998840995344,6cyclictest1024240-21kworker/u8:2+events_unbound20:40:370
998840995246,5cyclictest1003741-21/usr/sbin/munin19:25:380
998840995242,8cyclictest1035891-21/usr/sbin/munin21:05:400
998840995145,5cyclictest1032396-21kworker/u8:1+events_unbound20:55:380
998840995145,4cyclictest1100671-21kworker/u8:1+flush-179:000:30:300
998840995145,4cyclictest1018896-21kworker/u8:3+events_unbound20:15:360
998840995142,7cyclictest1073096-21kworker/u8:2+flush-179:000:10:300
998840995128,6cyclictest1013298-21apt-get19:55:270
998840994943,4cyclictest1047023-21kworker/u8:0+flush-179:022:25:330
998840994840,6cyclictest1073096-21kworker/u8:2+flush-179:023:05:000
998840994838,8cyclictest1047060-21latency_hist21:40:010
998840994837,9cyclictest1022862-21apt-get20:25:210
998840994744,2cyclictest1076463-21kworker/u8:3+flush-179:023:20:000
998840994742,4cyclictest1100671-21kworker/u8:1+flush-179:000:35:290
998840994742,4cyclictest1100671-21kworker/u8:1+flush-179:000:35:290
998847994641,3cyclictest0-21swapper/219:51:002
998847994641,3cyclictest0-21swapper/219:51:002
998845994642,3cyclictest1024240-21kworker/u8:2+events_unbound21:00:291
998840994640,5cyclictest1091302-21kworker/u8:1+flush-179:000:05:370
998840994534,9cyclictest1073032-21apt-get23:00:010
998850994440,2cyclictest0-21swapper/319:51:003
998850994440,2cyclictest0-21swapper/319:51:003
998840994437,5cyclictest1087096-21ntp_states23:40:350
99884599420,6cyclictest725-21Xorg23:47:591
998840994238,3cyclictest1065111-21kworker/u8:1+flush-179:023:05:280
998840994232,8cyclictest1095894-21kworker/u8:3+flush-179:000:15:340
998840994232,8cyclictest1095894-21kworker/u8:3+flush-179:000:15:340
998840994232,8cyclictest1047023-21kworker/u8:0+events_unbound22:20:320
99885099400,39cyclictest0-21swapper/323:15:263
99884799400,38cyclictest0-21swapper/223:15:262
99884799400,36cyclictest0-21swapper/219:38:592
998840994036,3cyclictest1013023-21kworker/u8:2+flush-179:020:05:000
998840994035,4cyclictest1052126-21kworker/u8:4+flush-179:021:55:400
998850993934,4cyclictest1066673-21turbostat22:44:563
998847993935,3cyclictest0-21swapper/220:55:592
99884799390,36cyclictest0-21swapper/200:03:592
998845993929,7cyclictest998692-21latency_hist19:10:021
99884099390,2cyclictest0-21swapper/021:18:580
998850993835,2cyclictest0-21swapper/323:32:003
998850993833,4cyclictest1030827-21turbostat20:50:003
998847993834,2cyclictest0-21swapper/223:31:592
99884799380,35cyclictest0-21swapper/221:59:592
99884099382,31cyclictest1024450-21apt-get20:30:270
99884099381,35cyclictest1044125-21apt-get21:30:260
998847993734,2cyclictest0-21swapper/222:44:552
998847993733,2cyclictest0-21swapper/221:17:002
998850993634,1cyclictest0-21swapper/320:55:593
998850993633,2cyclictest0-21swapper/321:16:593
99885099360,35cyclictest0-21swapper/300:03:593
99885099360,1cyclictest0-21swapper/319:38:593
998847993633,2cyclictest0-21swapper/221:08:582
998847993633,1cyclictest0-21swapper/220:49:592
998847993632,3cyclictest0-21swapper/220:15:582
998847993632,2cyclictest0-21swapper/222:37:592
998847993632,2cyclictest0-21swapper/220:40:592
998847993632,2cyclictest0-21swapper/219:29:552
998847993631,3cyclictest0-21swapper/222:08:592
998850993533,1cyclictest0-21swapper/321:08:593
998850993532,2cyclictest0-21swapper/322:59:003
998850993532,2cyclictest0-21swapper/320:40:583
998850993532,2cyclictest0-21swapper/319:16:593
998847993533,1cyclictest0-21swapper/219:16:592
998847993532,2cyclictest0-21swapper/222:58:592
99884099350,33cyclictest1089478-21http23:50:230
998850993432,1cyclictest0-21swapper/319:29:553
998850993430,2cyclictest0-21swapper/322:08:593
998850993426,6cyclictest1102286-21turbostat00:30:003
998847993431,2cyclictest0-21swapper/222:28:002
998847993431,2cyclictest0-21swapper/220:06:592
998847993430,2cyclictest0-21swapper/223:54:552
998850993331,1cyclictest0-21swapper/323:54:553
998850993331,1cyclictest0-21swapper/322:37:593
998850993331,1cyclictest0-21swapper/322:28:003
998850993331,1cyclictest0-21swapper/320:15:593
998850993330,1cyclictest0-21swapper/320:06:593
998847993330,2cyclictest0-21swapper/200:15:232
998847993330,2cyclictest0-21swapper/200:15:232
99884099331,30cyclictest695-21snmpd20:20:230
998850993228,2cyclictest0-21swapper/321:45:593
998847993229,2cyclictest0-21swapper/200:25:202
998847993227,4cyclictest0-21swapper/223:55:032
99884799320,29cyclictest0-21swapper/222:10:242
99884799320,29cyclictest0-21swapper/221:50:282
99885099310,2cyclictest0-21swapper/323:35:263
99885099310,2cyclictest0-21swapper/321:15:013
998847993128,2cyclictest0-21swapper/223:39:002
998847993128,2cyclictest0-21swapper/221:46:002
998847993127,2cyclictest0-21swapper/222:45:592
998847993127,2cyclictest0-21swapper/222:31:002
998847993127,2cyclictest0-21swapper/221:12:002
998847993127,2cyclictest0-21swapper/219:22:592
99884799310,29cyclictest0-21swapper/222:05:012
99884799310,29cyclictest0-21swapper/221:40:252
998850993028,1cyclictest0-21swapper/322:30:593
998850993027,2cyclictest0-21swapper/319:30:263
998850993027,2cyclictest0-21swapper/300:10:223
99885099300,2cyclictest0-21swapper/323:20:233
99885099300,29cyclictest0-21swapper/323:45:243
99885099300,29cyclictest0-21swapper/322:05:003
99885099300,28cyclictest0-21swapper/323:55:023
99885099300,28cyclictest0-21swapper/320:35:263
99885099300,28cyclictest0-21swapper/320:00:013
99885099300,28cyclictest0-21swapper/300:35:213
99885099300,28cyclictest0-21swapper/300:35:213
99885099300,22cyclictest0-21swapper/321:40:253
998847993027,2cyclictest0-21swapper/221:35:252
998847993026,2cyclictest0-21swapper/221:33:592
998847993025,2cyclictest0-21swapper/220:00:012
99884799300,2cyclictest0-21swapper/223:00:032
99884799300,29cyclictest0-21swapper/223:20:232
99884799300,29cyclictest0-21swapper/219:40:242
99884799300,29cyclictest0-21swapper/200:40:012
99884799300,29cyclictest0-21swapper/200:40:012
99884799300,28cyclictest0-21swapper/223:45:242
99884799300,28cyclictest0-21swapper/223:10:222
99884799300,28cyclictest0-21swapper/219:50:002
99884799300,28cyclictest0-21swapper/200:10:212
99884799300,27cyclictest0-21swapper/219:30:252
99884799300,26cyclictest0-21swapper/220:35:262
99884799300,21cyclictest0-21swapper/221:25:262
998850992927,1cyclictest0-21swapper/321:35:263
998850992926,2cyclictest0-21swapper/322:45:583
998850992926,2cyclictest0-21swapper/319:22:593
998850992926,2cyclictest0-21swapper/319:10:253
99885099290,28cyclictest0-21swapper/322:15:253
99885099290,28cyclictest0-21swapper/322:15:253
99885099290,28cyclictest0-21swapper/321:55:263
99885099290,28cyclictest0-21swapper/319:40:233
99885099290,27cyclictest1022615-21turbostat20:25:003
99885099290,27cyclictest0-21swapper/323:10:223
99885099290,27cyclictest0-21swapper/320:50:233
99885099290,27cyclictest0-21swapper/320:25:223
99885099290,27cyclictest0-21swapper/319:50:013
99885099290,22cyclictest0-21swapper/321:50:283
99885099290,1cyclictest0-21swapper/323:00:023
99885099290,1cyclictest0-21swapper/322:10:253
99885099290,1cyclictest0-21swapper/321:25:263
998847992926,2cyclictest0-21swapper/222:55:002
998847992926,2cyclictest0-21swapper/220:50:232
99884799290,2cyclictest0-21swapper/222:15:262
99884799290,2cyclictest0-21swapper/222:15:252
99884799290,2cyclictest0-21swapper/220:00:012
99884799290,27cyclictest0-21swapper/219:10:252
99884799290,21cyclictest0-21swapper/220:31:592
998850992826,1cyclictest0-21swapper/322:55:003
99885099280,2cyclictest0-21swapper/320:00:013
99885099280,27cyclictest0-21swapper/323:25:013
99884799280,2cyclictest0-21swapper/220:25:232
99884799280,27cyclictest0-21swapper/223:25:012
99884799280,26cyclictest0-21swapper/221:04:002
99884799280,25cyclictest0-21swapper/200:30:212
998850992725,1cyclictest0-21swapper/321:33:583
998850992725,1cyclictest0-21swapper/321:01:593
99884799270,1cyclictest0-21swapper/200:25:012
4652991270,6ptp4l441ktimers/300:09:553
998850992623,2cyclictest0-21swapper/320:32:003
99885099260,19cyclictest0-21swapper/300:30:213
99885099260,18cyclictest0-21swapper/321:20:203
998847992623,2cyclictest0-21swapper/221:20:192
4652991260,6ptp4l61790irq/134-eno1-TxRx-020:14:563
4652991260,5ptp4l441ktimers/323:09:553
4652991260,5ptp4l441ktimers/322:24:563
998847992521,2cyclictest0-21swapper/200:05:592
998847992421,2cyclictest0-21swapper/223:43:592
998847992419,3cyclictest0-21swapper/220:10:592
998850992321,1cyclictest0-21swapper/323:43:593
998850992321,1cyclictest0-21swapper/300:20:023
998847992319,2cyclictest0-21swapper/222:20:222
998840992218,3cyclictest998924-21latency_hist19:10:020
998847992017,2cyclictest0-21swapper/223:05:592
998847992017,2cyclictest0-21swapper/220:20:022
9988479940,2cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional