You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-04 - 01:17

x86 Intel Atom x6425RE @1900 MHz, Linux 5.18.0-rt11 (Profile)

Latency plot of system in rack #e, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h800 -q
Total number of samples: 100 million
Resolution of latency scale: reduced 
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rackeslot8.osadl.org (updated Mon Jun 06, 2022 00:44:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
154732899811,74cyclictest1-21systemd23:59:560
619589670,0phc2sys0-21swapper/121:20:111
619589670,0phc2sys0-21swapper/121:20:111
23903756556,7pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:021:09:383
1547313996558,5cyclictest1560791-21apt-get19:24:270
23903756460,3pubsub_TSN_loop1718421-21kworker/u8:1+flush-8:022:10:113
1547313996055,3cyclictest1781786-21apt-get22:47:030
1547313996055,3cyclictest1591486-21latency_hist19:59:320
23903755955,3pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:19:583
23903755955,3pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:19:583
1547313995853,4cyclictest262-21systemd-journal00:19:590
154733099570,51cyclictest1803870-21munin-run23:04:520
1547313995650,4cyclictest262-21systemd-journal23:45:090
23905225552,2pubsub_TSN_loop1555056-21kworker/u8:0+flush-8:019:19:513
23901785548,5pubsub_TSN_loop1855490-21kworker/u8:2+flush-8:000:05:212
1547328995546,5cyclictest1705880-21apt-get21:49:520
1547313995550,3cyclictest1705100-21gpgv21:49:440
23905225347,4pubsub_TSN_loop1870469-21kworker/u8:4+flush-8:000:30:243
1547328995345,4cyclictest262-21systemd-journal22:44:480
1547328995345,4cyclictest262-21systemd-journal00:26:380
1547328995345,4cyclictest1837819-21apt-get23:30:040
1547328995344,5cyclictest1635660-21apt-get20:49:470
1547328995244,4cyclictest1817012-21latency_hist23:14:530
1547313995248,3cyclictest262-21systemd-journal20:09:310
1547328995143,4cyclictest1804529-21apt-get23:05:000
1547328995042,4cyclictest1917306-21apt-get00:30:110
23905224946,2pubsub_TSN_loop1697022-21kworker/u8:0+flush-8:021:49:573
23905224945,2pubsub_TSN_loop1885562-21kworker/u8:1+flush-8:000:10:143
23905224945,2pubsub_TSN_loop1803887-21kworker/u8:1+flush-8:023:50:183
1547328994941,4cyclictest1600984-21apt-get20:09:420
23903754845,2pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:14:343
1547313994843,4cyclictest1909867-21latency_hist00:25:000
619290470,1ptp4l0-21swapper/121:28:101
23903754743,3pubsub_TSN_loop1747971-21kworker/u8:4+flush-8:023:21:273
23903754743,3pubsub_TSN_loop1682311-21kworker/u8:1+flush-8:021:39:353
1547328994741,4cyclictest262-21systemd-journal20:34:480
23905224642,3pubsub_TSN_loop1747971-21kworker/u8:4+flush-8:023:35:093
23903754643,2pubsub_TSN_loop1855490-21kworker/u8:2+flush-8:000:37:483
23903754643,2pubsub_TSN_loop1748034-21kworker/u8:5+flush-8:023:00:253
1547328994540,4cyclictest262-21systemd-journal19:44:290
1547313994541,3cyclictest1823708-21latency_hist23:19:530
23904204438,3pubsub_TSN_loop1748034-21kworker/u8:5+events_unbound22:39:353
1547328994439,3cyclictest262-21systemd-journal23:20:550
1547328994439,3cyclictest1883846-21latency_hist00:04:580
1547328994438,4cyclictest1745541-21apt-get22:20:030
619589430,0phc2sys0-21swapper/123:45:141
23905224337,4pubsub_TSN_loop745620-21kworker/u8:2+flush-8:019:04:403
23905224336,5pubsub_TSN_loop1833675-21kworker/u8:0+flush-8:023:40:103
23904204338,3pubsub_TSN_loop1755951-21kworker/u8:0+events_unbound22:28:493
23903754340,2pubsub_TSN_loop1682311-21kworker/u8:1+flush-8:021:45:193
23901784338,3pubsub_TSN_loop1803887-21kworker/u8:1+flush-8:023:41:462
23905224236,5pubsub_TSN_loop1748034-21kworker/u8:5+flush-8:022:46:573
23903754239,2pubsub_TSN_loop1682311-21kworker/u8:1+flush-8:021:39:453
23903754239,2pubsub_TSN_loop1671820-21kworker/u8:4+flush-8:021:58:393
1547328994236,4cyclictest1738845-21apt-get22:14:560
1547328994235,5cyclictest262-21systemd-journal19:24:420
1547313994238,3cyclictest1617489-21latency_hist20:29:350
619290410,1ptp4l0-21swapper/123:23:501
619290410,1ptp4l0-21swapper/122:55:111
23905224136,3pubsub_TSN_loop1733134-21kworker/u8:2+flush-8:022:20:023
23903754138,2pubsub_TSN_loop1747971-21kworker/u8:4+flush-8:023:29:443
1547313994137,3cyclictest262-21systemd-journal23:38:060
1547313994136,3cyclictest1626908-21apt-get20:39:440
1547313994136,3cyclictest1626908-21apt-get20:39:440
23905224035,3pubsub_TSN_loop1803887-21kworker/u8:1+flush-8:023:30:163
23904204035,3pubsub_TSN_loop1542806-21kworker/u8:3+flush-8:019:38:323
23903754037,2pubsub_TSN_loop1870469-21kworker/u8:4+flush-8:023:58:533
1547328994036,2cyclictest262-21systemd-journal21:41:240
154731399401,38cyclictest1643601-21munin-run20:59:380
619589390,1phc2sys0-21swapper/100:38:341
619589390,1phc2sys0-21swapper/100:30:361
23905223932,4pubsub_TSN_loop1671820-21kworker/u8:4+flush-8:021:31:493
23904203935,3pubsub_TSN_loop1855490-21kworker/u8:2+events_unbound00:26:203
23904203934,3pubsub_TSN_loop1784070-21kworker/u8:2+flush-8:023:11:323
1547313993936,2cyclictest1725818-21apt-get22:04:560
619589380,0phc2sys0-21swapper/123:08:391
23905223833,3pubsub_TSN_loop1747971-21kworker/u8:4+flush-8:022:53:403
23904203835,2pubsub_TSN_loop1617353-21kworker/u8:3+events_unbound21:10:363
23904203834,1pubsub_TSN_loop1726528-21kworker/u8:0+flush-8:022:18:243
23903753835,2pubsub_TSN_loop1748034-21kworker/u8:5+flush-8:022:44:063
23903753833,3pubsub_TSN_loop1870469-21kworker/u8:4+flush-8:000:09:563
23901783834,2pubsub_TSN_loop1718421-21kworker/u8:1+flush-8:022:14:362
1547328993833,4cyclictest262-21systemd-journal22:54:490
1547328993833,4cyclictest262-21systemd-journal21:59:220
1547328993833,4cyclictest262-21systemd-journal00:12:330
1547328993832,4cyclictest262-21systemd-journal23:54:080
1547313993834,3cyclictest262-21systemd-journal23:41:010
1547313993834,2cyclictest262-21systemd-journal22:58:340
23904203733,3pubsub_TSN_loop1747971-21kworker/u8:4+events_unbound22:59:113
23904203733,3pubsub_TSN_loop1617353-21kworker/u8:3+events_unbound22:08:083
23904203732,3pubsub_TSN_loop1755951-21kworker/u8:0+events_unbound22:33:423
23903753734,2pubsub_TSN_loop1662715-21kworker/u8:2+flush-8:021:17:513
23903753734,2pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:14:493
23901783732,3pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:39:582
23901783732,3pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:39:582
1547328993732,3cyclictest262-21systemd-journal21:24:410
1547328993732,3cyclictest262-21systemd-journal21:24:410
1547328993730,5cyclictest262-21systemd-journal22:29:480
619589360,1phc2sys0-21swapper/122:40:001
619290360,1ptp4l0-21swapper/121:46:431
619290360,1ptp4l0-21swapper/100:27:311
23905223632,2pubsub_TSN_loop1833675-21kworker/u8:0+flush-8:023:45:323
23905223630,3pubsub_TSN_loop1617353-21kworker/u8:3+flush-8:020:29:493
23904203632,2pubsub_TSN_loop1647922-21kworker/u8:0+events_unbound21:26:013
23901783632,2pubsub_TSN_loop1870469-21kworker/u8:4+flush-8:000:04:312
23901783630,4pubsub_TSN_loop1617353-21kworker/u8:3+flush-8:021:18:352
1547328993631,4cyclictest1246-21rs:main0
1547328993631,3cyclictest262-21systemd-journal22:11:230
1547328993630,4cyclictest262-21systemd-journal22:04:120
619589350,1phc2sys0-21swapper/119:53:411
619589350,0phc2sys0-21swapper/120:34:171
23904203530,3pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:019:58:453
23903753532,2pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:39:533
23903753532,2pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:39:523
23903753532,2pubsub_TSN_loop1542806-21kworker/u8:3+flush-8:019:31:113
23903753531,3pubsub_TSN_loop1718421-21kworker/u8:1+flush-8:022:02:453
23901783530,3pubsub_TSN_loop1617353-21kworker/u8:3+flush-8:020:59:562
23901783529,4pubsub_TSN_loop1647922-21kworker/u8:0+flush-8:021:22:082
23901783529,4pubsub_TSN_loop1647922-21kworker/u8:0+flush-8:021:22:072
23901783528,4pubsub_TSN_loop1855490-21kworker/u8:2+flush-8:023:45:392
1547313993531,3cyclictest262-21systemd-journal23:28:510
619589340,1phc2sys0-21swapper/123:58:471
619589340,1phc2sys0-21swapper/123:30:081
619290340,1ptp4l0-21swapper/122:46:241
619290340,1ptp4l0-21swapper/121:54:561
23905223428,4pubsub_TSN_loop1892093-21kworker/u8:0+flush-8:000:15:253
23903753429,4pubsub_TSN_loop1748034-21kworker/u8:5+flush-8:023:16:393
1547328993429,3cyclictest262-21systemd-journal21:12:570
619290330,1ptp4l0-21swapper/122:20:551
619290330,1ptp4l0-21swapper/121:50:571
23905223328,3pubsub_TSN_loop1555056-21kworker/u8:0+flush-8:019:24:443
23905223326,5pubsub_TSN_loop1555056-21kworker/u8:0+flush-8:019:17:503
23901783329,2pubsub_TSN_loop1919749-21kworker/u8:1+flush-8:000:38:002
23901783328,3pubsub_TSN_loop1697022-21kworker/u8:0+flush-8:021:51:422
619589320,1phc2sys0-21swapper/122:07:461
619290320,1ptp4l0-21swapper/123:36:581
619290320,1ptp4l0-21swapper/121:11:301
1547328993227,4cyclictest262-21systemd-journal20:22:100
1547328993227,4cyclictest262-21systemd-journal20:22:100
1547328993227,3cyclictest1621816-21latency_hist20:34:360
619589310,1phc2sys0-21swapper/100:06:431
23905223126,3pubsub_TSN_loop1542806-21kworker/u8:3+flush-8:019:40:233
23901783126,3pubsub_TSN_loop1647922-21kworker/u8:0+flush-8:021:26:542
23901783126,3pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:021:09:402
1547328993127,3cyclictest262-21systemd-journal22:32:180
1547328993126,4cyclictest262-21systemd-journal21:16:030
619589300,1phc2sys0-21swapper/123:43:391
619589300,1phc2sys0-21swapper/120:52:341
619290300,1ptp4l0-21swapper/123:10:081
23901783025,3pubsub_TSN_loop1870469-21kworker/u8:4+flush-8:000:16:482
1547328993025,4cyclictest1648650-21apt-get21:04:480
1547328993025,4cyclictest1608811-21latency_hist20:19:350
1547328993025,4cyclictest1583596-21apt-get19:49:400
1547328993023,4cyclictest262-21systemd-journal21:25:310
619589290,1phc2sys0-21swapper/119:20:161
619589290,1phc2sys0-21swapper/100:15:291
619589290,0phc2sys0-21swapper/120:23:561
619589290,0phc2sys0-21swapper/120:23:561
23904202925,2pubsub_TSN_loop1870469-21kworker/u8:4+events_unbound00:24:003
23903752926,2pubsub_TSN_loop1617353-21kworker/u8:3+flush-8:020:56:053
23903752925,3pubsub_TSN_loop1590273-21kworker/u8:0+flush-8:020:26:503
1547328992923,4cyclictest262-21systemd-journal21:00:220
1547313992924,4cyclictest262-21systemd-journal21:35:210
619290280,4ptp4l271ktimers/100:11:201
619290280,1ptp4l0-21swapper/123:25:551
619290280,1ptp4l0-21swapper/122:35:001
619290280,1ptp4l0-21swapper/122:12:571
619290280,1ptp4l0-21swapper/121:34:481
23903752825,2pubsub_TSN_loop745620-21kworker/u8:2+flush-8:019:09:303
23903752825,2pubsub_TSN_loop1542806-21kworker/u8:3+flush-8:019:49:023
23901782825,2pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:019:53:282
1547328992824,3cyclictest262-21systemd-journal19:29:460
154731399285,22cyclictest1652150-21diskmemload21:31:220
1547313992825,2cyclictest1652150-21diskmemload00:36:150
1547313992823,3cyclictest262-21systemd-journal19:35:010
619589270,1phc2sys0-21swapper/100:04:081
619290270,1ptp4l0-21swapper/123:04:051
619290270,1ptp4l0-21swapper/122:29:491
619290270,1ptp4l0-21swapper/122:28:571
619290270,1ptp4l0-21swapper/122:15:381
619290270,1ptp4l0-21swapper/121:16:581
619290270,1ptp4l0-21swapper/120:26:121
619290270,1ptp4l0-21swapper/119:45:381
619290270,0ptp4l0-21swapper/119:36:481
23905222722,3pubsub_TSN_loop1855490-21kworker/u8:2+flush-8:000:00:203
23901782724,2pubsub_TSN_loop1617353-21kworker/u8:3+flush-8:021:35:472
23901782723,3pubsub_TSN_loop1671820-21kworker/u8:4+flush-8:021:43:532
23901782721,4pubsub_TSN_loop1726528-21kworker/u8:0+flush-8:022:05:232
619589260,1phc2sys0-21swapper/121:40:121
619290260,1ptp4l0-21swapper/123:54:261
619290260,1ptp4l0-21swapper/122:50:091
619290260,1ptp4l0-21swapper/122:01:441
619290260,1ptp4l0-21swapper/121:30:591
619290260,1ptp4l0-21swapper/119:28:221
23905222622,3pubsub_TSN_loop1590273-21kworker/u8:0+flush-8:020:02:383
23905222621,3pubsub_TSN_loop1617353-21kworker/u8:3+flush-8:020:49:413
23903752623,2pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:48:153
23903752623,2pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:019:54:253
1547313992624,1cyclictest1885357-21gltestperf00:05:150
619290250,1ptp4l0-21swapper/123:17:131
619290250,1ptp4l0-21swapper/119:40:361
619290250,0ptp4l0-21swapper/120:04:521
619290250,0ptp4l0-21swapper/100:22:281
23903752522,2pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:020:07:113
23901782520,3pubsub_TSN_loop1892093-21kworker/u8:0+flush-8:000:12:052
2390022250,1pubsub_TSN_loop0-21swapper/200:25:172
619290240,3ptp4l0-21swapper/119:57:351
619290240,1ptp4l0-21swapper/121:05:281
619290240,1ptp4l0-21swapper/120:17:341
23901782421,2pubsub_TSN_loop1755951-21kworker/u8:0+flush-8:022:36:052
23901782420,3pubsub_TSN_loop1772486-21kworker/u8:1+flush-8:022:43:042
23901782417,5pubsub_TSN_loop745620-21kworker/u8:2+flush-8:019:04:482
2390022240,1pubsub_TSN_loop0-21swapper/221:32:412
1547328992419,4cyclictest262-21systemd-journal20:48:100
1547313992420,3cyclictest1552840-21apt-get19:14:360
23905222319,3pubsub_TSN_loop1748034-21kworker/u8:5+flush-8:023:07:353
23905222318,3pubsub_TSN_loop1647922-21kworker/u8:0+flush-8:021:19:533
23905222318,3pubsub_TSN_loop1647922-21kworker/u8:0+flush-8:021:19:533
23904202320,2pubsub_TSN_loop1617353-21kworker/u8:3+flush-8:021:00:253
2390178231,20pubsub_TSN_loop0-21swapper/222:24:482
2390022230,1pubsub_TSN_loop0-21swapper/221:47:122
2390022230,18pubsub_TSN_loop0-21swapper/223:33:432
619290220,0ptp4l0-21swapper/120:54:531
23901782219,2pubsub_TSN_loop0-21swapper/223:26:582
23901782218,3pubsub_TSN_loop1833675-21kworker/u8:0+flush-8:023:57:322
23901782218,2pubsub_TSN_loop0-21swapper/222:32:162
2390178221,1pubsub_TSN_loop0-21swapper/222:14:592
2390178221,19pubsub_TSN_loop0-21swapper/222:02:392
2390178220,19pubsub_TSN_loop0-21swapper/222:54:512
2390178220,19pubsub_TSN_loop0-21swapper/222:49:562
2390178220,19pubsub_TSN_loop0-21swapper/222:45:452
1547328992216,4cyclictest262-21systemd-journal19:45:030
619290210,1ptp4l0-21swapper/120:34:521
619290210,10ptp4l271ktimers/120:59:471
23901782119,1pubsub_TSN_loop0-21swapper/223:36:002
23901782118,2pubsub_TSN_loop0-21swapper/223:11:592
2390178211,1pubsub_TSN_loop0-21swapper/220:49:482
2390178210,19pubsub_TSN_loop0-21swapper/223:20:012
2390178210,19pubsub_TSN_loop0-21swapper/223:15:592
2390178210,19pubsub_TSN_loop0-21swapper/223:05:192
2390178210,19pubsub_TSN_loop0-21swapper/221:55:032
2390178210,19pubsub_TSN_loop0-21swapper/200:30:152
1547328992119,1cyclictest1592330-21apt-get19:59:430
619290200,2ptp4l271ktimers/119:29:431
23901782018,1pubsub_TSN_loop0-21swapper/223:50:412
2390178201,18pubsub_TSN_loop0-21swapper/223:00:402
2390178200,1pubsub_TSN_loop0-21swapper/222:25:092
2390178200,19pubsub_TSN_loop0-21swapper/200:22:282
1547328992015,3cyclictest262-21systemd-journal19:11:530
619290190,4ptp4l261rcuc/120:04:301
619290190,1ptp4l0-21swapper/120:48:151
619290190,1ptp4l0-21swapper/120:39:521
619290190,1ptp4l0-21swapper/120:39:511
619290190,1ptp4l0-21swapper/120:11:381
619290190,1ptp4l0-21swapper/119:14:441
619290190,1ptp4l0-21swapper/119:09:441
23903751918,1pubsub_TSN_loop0-21swapper/320:34:473
23901781917,1pubsub_TSN_loop0-21swapper/219:45:382
2390178190,1pubsub_TSN_loop0-21swapper/221:09:372
619290180,1ptp4l0-21swapper/119:04:361
23901781614,1pubsub_TSN_loop0-21swapper/219:37:182
23901781611,3pubsub_TSN_loop1567759-21kworker/u8:1+flush-8:019:54:402
23900221611,3pubsub_TSN_loop745620-21kworker/u8:2+events_unbound19:09:442
23901781512,2pubsub_TSN_loop0-21swapper/220:24:452
154731399127,3cyclictest1547365-21date19:09:270
2390178119,1pubsub_TSN_loop0-21swapper/220:54:482
2390178119,1pubsub_TSN_loop0-21swapper/220:09:442
2390022110,6pubsub_TSN_loop0-21swapper/220:01:252
2390022110,1pubsub_TSN_loop0-21swapper/220:19:522
2390022110,1pubsub_TSN_loop0-21swapper/220:19:522
2390022110,1pubsub_TSN_loop0-21swapper/220:09:322
2390022110,1pubsub_TSN_loop0-21swapper/219:33:502
2390178108,1pubsub_TSN_loop0-21swapper/220:31:362
2390178106,3pubsub_TSN_loop1617353-21kworker/u8:3+flush-8:020:37:192
2390178100,0pubsub_TSN_loop0-21swapper/220:49:202
2390178100,0pubsub_TSN_loop0-21swapper/219:14:522
2390022100,1pubsub_TSN_loop0-21swapper/220:17:442
239017890,3pubsub_TSN_loop1555056-21kworker/u8:0+flush-8:019:19:432
239002290,2pubsub_TSN_loop1562620-21modprobe19:24:472
239002280,2pubsub_TSN_loop1575728-21modprobe19:39:472
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional