You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2020-03-29 - 10:14

Intel(R) Atom(TM) Processor E3950 @ 1.60GHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -p90 -i200 -h400 -a0-2 -t3 -q done
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
The worst-case latency is NOT higher under extreme CPU load.
Results of all systems under extreme CPU load are combined on this summary graph.
The worst-case latency is NOT higher under graphics load.
Results of all systems with accelerated graphics are combined on this summary graph.
Characteristics of the highest latencies:
System rackdslot6.osadl.org (updated Sat Feb 29, 2020 12:45:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16422982232178,46sleep20-21swapper/207:06:212
16424722226180,18sleep30-21swapper/307:08:363
16423722223182,33sleep10-21swapper/107:07:161
16423442219180,31sleep00-21swapper/007:06:560
169781021780,4sleep1164271990cyclictest09:05:091
169781021780,4sleep1164271990cyclictest09:05:081
174402521720,4sleep30-21swapper/310:40:093
16492922580,5sleep11649288-21grep07:20:201
16492922580,5sleep11649288-21grep07:20:191
17061072500,1sleep00-21swapper/009:20:200
16502932480,3sleep20-21swapper/207:25:062
17301392450,1sleep335-21ksoftirqd/310:10:133
17301392450,1sleep335-21ksoftirqd/310:10:133
164272090441,25cyclictest1725710-21/usr/sbin/munin10:00:162
164272090441,25cyclictest1714605-21munin-run09:40:012
164272090436,28cyclictest1670373-21cat08:05:172
164272090430,24cyclictest1738566-21sed10:30:012
164272090420,23cyclictest1691665-21cat08:50:192
164272090420,23cyclictest1691665-21cat08:50:182
164272090411,25cyclictest1792235-21/usr/sbin/munin12:20:162
164272090411,23cyclictest0-21swapper/210:32:272
164272090411,21cyclictest1789583-21awk12:15:112
164272090411,21cyclictest0-21swapper/209:11:352
164272090410,26cyclictest1649461-21chrt07:21:372
164272090410,26cyclictest1649461-21chrt07:21:362
164272090410,25cyclictest1783607-21chrt12:02:552
164272090410,25cyclictest1783607-21chrt12:02:542
164272090410,24cyclictest16470002chrt07:15:242
164272090410,23cyclictest1664985-21/usr/sbin/munin07:55:102
164272090410,23cyclictest1664985-21/usr/sbin/munin07:55:092
164271890418,15cyclictest1721759-21munin-run09:55:020
164272090401,29cyclictest1696872-21chrt09:02:242
164272090401,25cyclictest17692922sleep211:32:222
164272090401,25cyclictest16804032sleep208:28:442
164272090401,24cyclictest1693140-21fw_conntrack08:55:072
164272090401,23cyclictest1730432-21/usr/sbin/munin10:10:162
164272090401,23cyclictest1730432-21/usr/sbin/munin10:10:162
164272090401,23cyclictest16780762sleep208:24:142
164272090400,30cyclictest1663706-21chrt07:52:052
164272090400,25cyclictest1661456-21chrt07:48:372
164272090400,23cyclictest1698700-21/usr/sbin/munin09:05:192
164272090400,23cyclictest1698700-21/usr/sbin/munin09:05:192
164272090400,21cyclictest1687684-21sed08:45:012
164272090400,20cyclictest1772940-21seq11:40:112
164272090391,27cyclictest1776697-21munin-run11:50:002
164272090391,24cyclictest1723138-21/usr/sbin/munin09:55:172
164272090391,23cyclictest1764706-21taskset11:24:272
164272090391,23cyclictest1764706-21taskset11:24:272
164272090391,22cyclictest1735894-21taskset10:22:152
164272090391,22cyclictest1688627-21/usr/sbin/munin08:45:112
164272090391,21cyclictest1800059-21taskset12:35:352
164272090391,21cyclictest1800059-21taskset12:35:352
164272090390,29cyclictest1654468-21cat07:34:572
164272090390,25cyclictest1707068-21df09:25:042
164272090390,25cyclictest1685027-21taskset08:37:222
164272090390,24cyclictest1750190-21chrt10:53:232
164272090390,24cyclictest16826672sleep208:32:252
164272090390,24cyclictest16826672sleep208:32:242
164272090390,22cyclictest17217032sleep209:54:362
164272090390,22cyclictest17190582sleep209:46:042
164272090387,17cyclictest0-21swapper/212:30:202
164272090387,17cyclictest0-21swapper/212:30:192
164272090381,28cyclictest1675731-21chrt08:19:292
164272090381,28cyclictest1675731-21chrt08:19:282
164272090381,23cyclictest17860202sleep212:08:252
164272090381,23cyclictest17452462sleep210:40:432
164272090381,23cyclictest1656570-21chrt07:36:362
164272090381,21cyclictest1755360-21sed11:05:022
164272090380,29cyclictest1778977-21chrt11:54:372
164272090380,24cyclictest1788472-21taskset12:14:172
164272090380,21cyclictest1703138-21sed09:15:152
164271890388,14cyclictest1752877-21perf11:00:010
164272090371,23cyclictest17670842chrt11:29:302
164272090371,22cyclictest0-21swapper/210:38:402
164272090371,22cyclictest0-21swapper/208:04:052
164272090371,21cyclictest1770623-21cat11:35:122
164272090370,24cyclictest1762131-21taskset11:16:582
164272090370,23cyclictest1759952-21chrt11:14:262
164272090370,23cyclictest1728726-21chrt10:06:232
164272090370,23cyclictest1706187-21chrt09:20:302
164272090370,23cyclictest1644793-21taskset07:12:232
164272090370,22cyclictest17523742sleep210:56:052
164272090370,22cyclictest16588592sleep207:40:392
164272090361,21cyclictest0-21swapper/212:25:322
164272090360,23cyclictest17336352sleep210:18:342
164272090360,22cyclictest1756069-21sh11:05:082
164272090360,22cyclictest17475922sleep210:45:302
164272090360,22cyclictest17475922sleep210:45:302
164272090360,22cyclictest17110012sleep209:31:332
164272090360,21cyclictest1781490-21munin-run12:00:022
164272090360,21cyclictest1781490-21munin-run12:00:012
164272090350,21cyclictest1673469-21wc08:15:012
164271890355,15cyclictest1750340-21munin-run10:55:000
164272090340,22cyclictest1716949-21seq09:44:182
164271890339,11cyclictest1645056-21perf07:15:010
164271890338,11cyclictest1733771-21munin-run10:20:020
17608182300,5sleep30-21swapper/311:15:103
164271890304,12cyclictest1779030-21munin-run11:55:000
1642718902910,16cyclictest1788551-21munin-run12:15:010
1642718902810,16cyclictest1652159-21perf07:30:010
164271990261,22cyclictest1613-21systemd-logind10:57:481
164271990251,21cyclictest1613-21systemd-logind10:01:491
164271890252,21cyclictest0-21swapper/009:00:020
164271890251,12cyclictest1776655-21munin-run11:50:010
164271890251,12cyclictest1760030-21date11:15:010
164271890250,12cyclictest1783795-21munin-run12:05:000
164271890250,12cyclictest1783795-21munin-run12:05:000
164271890241,21cyclictest1613-21systemd-logind08:40:000
164271990231,20cyclictest1613-21systemd-logind09:40:001
164271990231,19cyclictest1613-21systemd-logind07:20:011
164271890231,19cyclictest1613-21systemd-logind12:10:010
164271890231,19cyclictest1613-21systemd-logind10:50:010
164271890231,19cyclictest1613-21systemd-logind10:50:000
164271890230,21cyclictest1613-21systemd-logind09:43:040
17894002220,3sleep1164271990cyclictest12:15:081
164271890221,18cyclictest393-21systemd-journal09:35:050
164271890221,12cyclictest0-21swapper/011:35:200
164271890220,20cyclictest1613-21systemd-logind07:40:000
164271890220,11cyclictest1740875-21munin-run10:35:010
164271990211,18cyclictest1613-21systemd-logind07:50:011
164271990210,19cyclictest1613-21systemd-logind11:30:001
164271890211,18cyclictest1613-21systemd-logind08:50:010
177398322014,3sleep31540-21ptp4l11:41:343
171670022015,3sleep31540-21ptp4l09:41:543
1642719902014,4cyclictest1699831-21tr09:10:051
1642719902014,4cyclictest1680881-21cpuspeed_turbos08:30:041
1642719902014,4cyclictest1680881-21cpuspeed_turbos08:30:031
1642719902013,5cyclictest21-21ksoftirqd/107:15:021
164271990201,17cyclictest1613-21systemd-logind10:20:001
164271990200,19cyclictest1613-21systemd-logind09:28:041
164271990200,18cyclictest393-21systemd-journal11:20:001
164271990200,17cyclictest393-21systemd-journal08:15:041
164271990200,17cyclictest393-21systemd-journal08:15:031
164271890202,16cyclictest1668733-21wc08:05:020
1642718902013,4cyclictest1514-21sssd_nss12:20:100
164271890201,17cyclictest393-21systemd-journal10:01:480
164271890201,17cyclictest393-21systemd-journal09:55:050
164271890200,17cyclictest393-21systemd-journal08:15:000
1642719901916,2cyclictest1764657-21kworker/1:211:24:311
1642719901916,2cyclictest1764657-21kworker/1:211:24:311
1642719901913,4cyclictest1684769-21awk08:35:211
1642719901913,4cyclictest1665411-21grep07:55:161
1642719901913,4cyclictest1665411-21grep07:55:151
1642719901912,5cyclictest1792902-21who12:20:211
164271990190,17cyclictest393-21systemd-journal12:00:011
164271990190,17cyclictest393-21systemd-journal12:00:001
164271990190,17cyclictest393-21systemd-journal09:57:171
1642718901913,4cyclictest1797040-21awk12:30:160
1642718901913,4cyclictest1797040-21awk12:30:150
164271890191,15cyclictest393-21systemd-journal09:31:030
164271890191,15cyclictest393-21systemd-journal07:35:010
164271890191,15cyclictest1613-21systemd-logind10:10:440
164271890191,15cyclictest1613-21systemd-logind10:10:430
1642719901814,2cyclictest1691443-21head08:50:171
1642719901814,2cyclictest1691443-21head08:50:161
1642719901813,4cyclictest1-21systemd12:10:011
1642719901813,3cyclictest1759098-21head11:10:171
1642719901813,3cyclictest1696372-21sed09:00:181
1642719901812,4cyclictest1774679-21cpuspeed_turbos11:45:051
1642719901812,4cyclictest1742128-21sed10:35:131
1642719901812,4cyclictest1737386-21sed10:25:151
1642719901811,4cyclictest1729215-21cut10:10:021
1642718901815,2cyclictest1656498-21kworker/0:007:56:160
1642718901815,2cyclictest1656498-21kworker/0:007:56:150
1642718901813,3cyclictest1733968-21/usr/sbin/munin10:20:140
1642718901813,3cyclictest1709079-21date09:30:010
1642718901813,3cyclictest1673831-21grep08:15:060
1642718901813,3cyclictest1673831-21grep08:15:050
164271890181,14cyclictest1678504-21grep08:25:040
164271890180,16cyclictest393-21systemd-journal08:10:000
164271990179,5cyclictest1514-21sssd_nss09:50:091
164271990172,8cyclictest1704200-21perf09:20:021
1642719901712,3cyclictest1795247-21grep12:25:201
1642719901712,3cyclictest1768850-21ls11:30:181
1642719901712,3cyclictest1743440-21/usr/sbin/munin10:40:101
1642719901712,3cyclictest1741133-21awk10:35:031
1642719901712,3cyclictest1705331-21tr09:20:121
1642719901712,3cyclictest1671036-21seq08:09:571
164271990171,14cyclictest393-21systemd-journal11:45:011
164271990171,14cyclictest393-21systemd-journal07:36:481
1642719901711,4cyclictest21-21ksoftirqd/108:47:561
1642719901711,4cyclictest1679060-21wc08:25:081
1642719901711,4cyclictest1652439-21ls07:30:041
164271990171,13cyclictest393-21systemd-journal10:52:291
164271990171,13cyclictest393-21systemd-journal08:01:001
164271990171,13cyclictest393-21systemd-journal07:51:041
1642719901711,3cyclictest1709439-21sh09:30:071
1642719901710,4cyclictest1786598-21ls12:10:051
1642719901710,4cyclictest1514-21sssd_nss11:05:091
164271990170,15cyclictest393-21systemd-journal09:42:221
164271990170,14cyclictest393-21systemd-journal07:28:041
164271890177,8cyclictest1765156-21cpuspeed_turbos11:25:060
1642718901714,2cyclictest1669808-1kworker/0:2H11:06:090
1642718901712,3cyclictest1779443-21cat11:55:050
1642718901712,3cyclictest1779443-21cat11:55:040
1642718901712,3cyclictest1677525-21wc08:20:180
1642718901711,4cyclictest1755370-21date11:05:020
1642718901711,4cyclictest1702208-21expr09:15:040
164271890171,14cyclictest1613-21systemd-logind07:46:360
1642718901710,5cyclictest1663991-21wc07:55:000
164271890170,15cyclictest393-21systemd-journal10:30:010
164271890170,14cyclictest393-21systemd-journal11:30:050
164271890170,14cyclictest1695573-21grep09:00:080
1642719901613,2cyclictest1738155-1kworker/1:0H12:00:551
1642719901613,2cyclictest1738155-1kworker/1:0H12:00:541
1642719901611,3cyclictest1733968-21/usr/sbin/munin10:20:061
1642719901611,3cyclictest1718623-21which09:45:181
164271990161,13cyclictest1692697-21tr08:55:041
1642719901611,3cyclictest1672197-21tr08:10:141
164271990161,12cyclictest1657982-21cut07:40:131
1642719901610,4cyclictest1748058-21pluginstate10:50:021
1642719901610,4cyclictest1748058-21pluginstate10:50:011
164271990160,14cyclictest393-21systemd-journal12:30:041
164271990160,14cyclictest393-21systemd-journal12:30:031
164271990160,13cyclictest393-21systemd-journal11:35:041
164271990160,13cyclictest393-21systemd-journal11:02:041
164271890169,4cyclictest1772699-21fw_conntrack11:40:090
164271890161,13cyclictest393-21systemd-journal09:12:070
164271890161,13cyclictest393-21systemd-journal07:25:010
164271890161,13cyclictest393-21systemd-journal07:25:000
1642718901611,3cyclictest1794525-21ls12:25:120
1642718901611,3cyclictest1788751-21sort12:15:030
1642718901611,3cyclictest1745486-21seq10:43:460
1642718901611,3cyclictest1690318-21ls08:50:040
1642718901611,3cyclictest1690318-21ls08:50:030
1642718901611,3cyclictest1682123-21which08:30:170
1642718901611,3cyclictest1682123-21which08:30:160
1642718901610,4cyclictest9-21ksoftirqd/008:45:010
1642718901610,4cyclictest1762655-21cpu11:20:030
1642718901610,4cyclictest1762655-21cpu11:20:030
1642718901610,4cyclictest1516-21runrttasks10:05:210
164271990158,5cyclictest1798721-21sort12:35:081
164271990158,5cyclictest1798721-21sort12:35:081
164271990158,5cyclictest1676704-21hwlatdetect08:20:101
1642719901511,2cyclictest1685400-21/usr/sbin/munin08:40:081
1642719901510,3cyclictest1777453-21cat11:50:091
1642719901510,3cyclictest1730926-21grep10:10:211
1642719901510,3cyclictest1730926-21grep10:10:201
164271890159,3cyclictest1799498-21cut12:35:180
164271890159,3cyclictest1799498-21cut12:35:170
164271890157,6cyclictest1717893-21sed09:45:090
1642718901510,3cyclictest1646249-21missed_timers07:15:140
164271890150,5cyclictest1657235-21cpuspeed_turbos07:40:030
164271890149,3cyclictest1760472-21cat11:15:060
1642718901412,1cyclictest0-21swapper/010:37:100
1642718901410,2cyclictest1697297-21/usr/sbin/munin09:05:080
1642718901410,2cyclictest1697297-21/usr/sbin/munin09:05:070
17416442120,5sleep3341rcuc/310:35:073
1788362290,1sleep30-21swapper/312:12:563
1739735290,6sleep30-21swapper/310:30:143
1769723281,4sleep30-21swapper/311:35:043
1768772281,3sleep30-21swapper/311:30:203
1764713280,1sleep30-21swapper/311:24:343
1764713280,1sleep30-21swapper/311:24:343
1752363280,6sleep30-21swapper/310:55:583
1723872280,1sleep30-21swapper/309:57:103
1698312281,3sleep30-21swapper/309:05:153
1698312281,3sleep30-21swapper/309:05:153
1673180280,5sleep30-21swapper/308:12:043
1779536271,3sleep30-21swapper/311:55:073
1779536271,3sleep30-21swapper/311:55:063
1756684271,2sleep30-21swapper/311:05:173
1727320271,3sleep30-21swapper/310:05:073
1710955270,4sleep30-21swapper/309:30:583
1708051271,3sleep30-21swapper/309:25:173
1694698271,3sleep30-21swapper/309:00:033
1685685271,3sleep30-21swapper/308:40:043
1681039271,3sleep30-21swapper/308:30:063
1681039271,3sleep30-21swapper/308:30:063
1674838270,1sleep30-21swapper/308:15:173
1674838270,1sleep30-21swapper/308:15:163
1670397271,3sleep30-21swapper/308:05:193
1667108271,3sleep30-21swapper/308:00:103
1794935261,2sleep30-21swapper/312:25:193
1789639261,2sleep30-21swapper/312:15:113
1782375261,2sleep30-21swapper/312:00:123
1782375261,2sleep30-21swapper/312:00:113
1778532261,2sleep30-21swapper/311:50:213
1775314261,2sleep30-21swapper/311:45:123
1765566261,2sleep30-21swapper/311:25:103
1759138261,2sleep30-21swapper/311:10:173
1737246261,2sleep30-21swapper/310:25:143
1734463261,2sleep30-21swapper/310:20:083
1721494261,2sleep30-21swapper/309:52:003
1718248261,2sleep30-21swapper/309:45:153
1705926261,2sleep30-21swapper/309:20:193
1702335261,2sleep30-21swapper/309:15:063
1701859261,2sleep30-21swapper/309:15:033
1692270260,4sleep30-21swapper/308:54:163
1692270260,4sleep30-21swapper/308:54:163
1689920261,3sleep31518-21getstats08:49:283
1684113261,2sleep30-21swapper/308:35:153
1679569261,2sleep30-21swapper/308:25:163
1677020260,3sleep30-21swapper/308:20:163
1659941261,2sleep30-21swapper/307:45:093
1658849260,4sleep30-21swapper/307:40:323
1655716261,2sleep30-21swapper/307:35:153
1650974261,2sleep30-21swapper/307:25:153
1645287261,2sleep30-21swapper/307:15:053
1798786250,2sleep30-21swapper/312:35:103
1798786250,2sleep30-21swapper/312:35:103
1795876250,2sleep30-21swapper/312:30:033
1795876250,2sleep30-21swapper/312:30:033
1792045250,2sleep30-21swapper/312:20:133
1784810250,2sleep30-21swapper/312:05:143
1753382250,2sleep30-21swapper/311:00:073
1749343250,2sleep30-21swapper/310:50:173
1746191250,2sleep30-21swapper/310:45:073
1746191250,2sleep30-21swapper/310:45:073
1733407251,2sleep30-21swapper/310:15:423
1726127250,2sleep30-21swapper/310:00:433
1711510250,2sleep30-21swapper/309:35:033
1695307251,2sleep30-21swapper/309:00:073
1665867250,2sleep30-21swapper/307:55:203
1665867250,2sleep30-21swapper/307:55:203
1662570250,2sleep30-21swapper/307:50:113
1652438250,2sleep30-21swapper/307:30:063
1649680250,2sleep30-21swapper/307:24:243
1649680250,2sleep30-21swapper/307:24:243
1643250250,2sleep30-21swapper/307:10:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional