You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-05-12 - 12:41

Intel(R) Atom(TM) Processor E3950 @ 1.60GHz, Linux 4.19.182-rt74 (Profile)

Latency plot of system in rack #d, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -p90 -i200 -h400 -a0-2 -t3 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot7.osadl.org (updated Wed Mar 24, 2021 00:45:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15568622234178,48sleep20-21swapper/219:08:562
15569302223174,41sleep00-21swapper/019:09:460
15565922222184,31sleep30-21swapper/319:05:313
15568512218179,31sleep10-21swapper/119:08:461
159848221850,2sleep30-21swapper/320:00:023
181813421730,4sleep0155720990cyclictest00:25:140
181813421730,4sleep0155720990cyclictest00:25:140
173382521700,3sleep30-21swapper/322:45:043
156229321610,3sleep30-21swapper/319:15:053
156229321610,3sleep30-21swapper/319:15:053
17154022950,4sleep01715391-21irqstats22:20:130
16892442890,2sleep10-21swapper/121:50:051
16892442890,2sleep10-21swapper/121:50:041
17497182710,4sleep233-21ksoftirqd/223:04:552
17497182710,4sleep233-21ksoftirqd/223:04:552
16270642690,4sleep01627066-21gltestperf20:35:010
1557209906458,4cyclictest1778893-21grep23:40:030
1557209906458,4cyclictest1778893-21grep23:40:020
1557234906357,3cyclictest1760634-21grep23:15:122
17648242610,1sleep20-21swapper/223:20:132
1557234906156,3cyclictest1618087-21head20:20:202
155720990611,57cyclictest1796774-21if_enp3s000:00:100
1557234905954,3cyclictest1683684-21sed21:40:202
1557234905953,3cyclictest1571840-21sed19:25:122
1557209905954,3cyclictest1665184-21cpuspeed_turbos21:20:090
1557209905952,5cyclictest1825181-21cpuspeed_turbos00:35:090
155720990592,54cyclictest1587146-21entropy19:45:090
1557234905852,3cyclictest1656638-21grep21:10:062
1557234905852,3cyclictest1656638-21grep21:10:062
1557234905852,3cyclictest1578882-21cut19:35:072
1557209905853,3cyclictest1571405-21if_enp2s019:25:090
1557209905852,4cyclictest1695442-21tail21:55:160
1557209905852,4cyclictest1644447-21fschecks_time20:55:080
1557234905751,3cyclictest1767455-21uniq23:25:072
1557234905751,3cyclictest1673841-21ls21:30:112
1557234905751,3cyclictest1673841-21ls21:30:102
1557234905749,4cyclictest1640550-21cut20:50:092
1557209905750,4cyclictest1597338-21grep19:55:190
17566362560,2sleep30-21swapper/323:10:153
1557234905650,3cyclictest1738640-21unixbench_multi22:50:062
1557209905651,3cyclictest1659736-21unixbench-2d21:15:020
1557209905651,3cyclictest1659736-21unixbench-2d21:15:010
1557209905649,4cyclictest1643438-21grep20:55:020
1557234905549,3cyclictest1627950-21tune2fs20:35:072
1557234905549,3cyclictest1608954-21cut20:10:132
1557209905550,3cyclictest1677357-21cpuspeed_turbos21:35:070
1557209905548,4cyclictest1636188-21grep20:45:060
1557234905446,5cyclictest1697098-21cpuspeed_turbos22:00:042
1557234905348,3cyclictest1688927-21sed21:50:042
1557234905348,3cyclictest1688927-21sed21:50:032
1557234905347,3cyclictest1827959-21grep00:40:032
155723490531,49cyclictest1782113-21unixbench_singl23:40:212
1557209905348,3cyclictest1631231-21cut20:40:020
1557209905347,4cyclictest1768246-21tail23:25:110
1557209905347,3cyclictest1814919-21unixbench_singl00:20:190
17844642520,1sleep00-21swapper/023:45:090
1557234905246,3cyclictest1807336-21grep00:15:022
1557234905246,3cyclictest1807336-21grep00:15:012
1557234905244,5cyclictest1562768-21cpuspeed_turbos19:15:092
1557234905244,5cyclictest1562768-21cpuspeed_turbos19:15:082
1557234905243,5cyclictest1752694-21timerwakeupswit23:05:152
1557209905246,4cyclictest1583859-21ls19:40:130
17323232510,2sleep10-21swapper/122:40:151
1557234905146,3cyclictest1775752-21grep23:35:082
1557234905146,3cyclictest1775752-21grep23:35:072
1557234905146,3cyclictest1655730-21grep21:10:022
1557234905146,3cyclictest1603967-21tune2fs20:05:092
1557234905145,4cyclictest1756030-21grep23:10:112
1557234905143,5cyclictest1821790-21ls00:30:112
155723490513,43cyclictest1668563-21cstates21:25:042
1557234905132,17cyclictest1730716-21ls22:40:082
1557234905045,3cyclictest1796770-21uniq00:00:102
1557234905044,3cyclictest1635199-21latency_hist20:45:012
155723490504,42cyclictest1643955-21cstates20:55:042
1557234905043,4cyclictest1638549-21sed20:45:192
1557209905045,3cyclictest1622139-21grep20:25:190
1557209905045,3cyclictest1611835-21fschecks_time20:15:080
155720990500,47cyclictest1761606-21unixbench_singl23:15:200
155720990500,47cyclictest1705028-21head22:10:020
155720990500,47cyclictest1651402-21cstates21:05:010
1557234904944,3cyclictest1623970-21unixbench_multi20:30:062
155723490494,41cyclictest1680332-21gltestperf21:40:012
1557234904943,3cyclictest1688409-21wc21:50:012
1557234904942,4cyclictest1660875-21cpuspeed_turbos21:15:082
155723490491,45cyclictest1583148-21uniq19:40:092
1557209904944,3cyclictest1561584-21gltestperf19:15:030
1557209904944,3cyclictest1561584-21gltestperf19:15:020
1557234904844,2cyclictest1783116-21/usr/sbin/munin23:45:062
1557234904842,3cyclictest1773091-21sort23:30:132
1557234904842,3cyclictest1734321-21ls22:45:042
1557234904842,3cyclictest1592035-21find19:50:102
1557234904841,4cyclictest1718472-21cpuspeed_turbos22:25:092
1557234904841,4cyclictest1718472-21cpuspeed_turbos22:25:082
1557209904843,3cyclictest1763889-21cut23:20:100
1557209904843,3cyclictest1722946-21sort22:30:110
1557209904842,4cyclictest1735477-21ls22:45:110
1557209904842,3cyclictest1791644-21ls23:55:050
1557209904842,3cyclictest1709842-21grep22:15:050
155720990482,43cyclictest1674981-21cut21:30:170
155720990482,43cyclictest1674981-21cut21:30:160
1557234904741,4cyclictest1701913-21fschecks_time22:05:082
1557234904741,3cyclictest1621025-21gunzip20:25:122
1557234904740,4cyclictest1813188-21grep00:20:092
1557234904740,4cyclictest1574264-21cstates19:30:042
1557234904740,4cyclictest1574264-21cstates19:30:032
1557209904742,3cyclictest1746556-21chrt23:00:050
1557209904742,3cyclictest1746556-21chrt23:00:040
1557209904741,4cyclictest1652708-21proc_pri21:05:070
1557209904741,3cyclictest1669100-21grep21:25:070
1557234904641,3cyclictest1795148-21latency_hist00:00:022
1557234904640,4cyclictest1744425-21cat22:55:132
1557234904639,4cyclictest1559039-21cpu19:10:112
1557234904639,4cyclictest1559039-21cpu19:10:102
1557209904641,3cyclictest1780365-21cpu23:40:100
1557209904641,3cyclictest1567059-21pmu-power19:20:100
1557209904640,4cyclictest1727280-21if_enp1s022:35:100
155720990462,41cyclictest1774588-21ls23:35:020
1557234904538,4cyclictest1692973-21tail21:55:032
1557209904538,4cyclictest1616702-21find20:20:110
1557209904532,10cyclictest9-21ksoftirqd/023:05:150
1557234904439,3cyclictest1665299-21tail21:20:082
1557234904439,3cyclictest1602370-21grep20:05:012
1557234904438,3cyclictest1800722-21sed00:05:102
1557234904438,3cyclictest1800722-21sed00:05:092
1557209904439,3cyclictest1823915-21gltestperf00:35:020
1557209904439,3cyclictest1684322-21if_enp4s021:45:020
1557209904439,3cyclictest1632810-21if_enp1s020:40:100
1557234904335,4cyclictest1648747-21cpuspeed_turbos21:00:072
1557209904338,3cyclictest9-21ksoftirqd/000:10:090
1557209904338,3cyclictest9-21ksoftirqd/000:10:090
1557209904338,3cyclictest1730491-21grep22:40:070
1557209904338,3cyclictest1558336-21forks19:10:080
1557209904338,3cyclictest1558336-21forks19:10:070
1557209904330,10cyclictest9-21ksoftirqd/019:30:110
1557209904330,10cyclictest9-21ksoftirqd/019:30:100
155720990431,39cyclictest1606202-21latency_hist20:10:000
1557234904236,3cyclictest1610370-21/usr/sbin/munin20:15:132
1557209904237,3cyclictest1719695-21uname22:25:140
1557209904237,3cyclictest1719695-21uname22:25:130
1557209904237,3cyclictest1685055-21expr21:45:050
1557209904237,3cyclictest1661741-21grep21:15:120
1557209904237,3cyclictest1591516-21gltestperf19:50:090
155720990420,39cyclictest1609841-21unixbench_multi20:10:190
1557234904137,2cyclictest33-21ksoftirqd/200:30:022
1557234904137,2cyclictest33-21ksoftirqd/200:30:012
1557234904136,3cyclictest1727516-21cat22:35:122
1557234904134,4cyclictest1725277-21cstates22:35:022
1557209904130,9cyclictest9-21ksoftirqd/023:55:010
1557209904129,10cyclictest9-21ksoftirqd/000:05:090
1557209904129,10cyclictest9-21ksoftirqd/000:05:080
1557209904036,2cyclictest9-21ksoftirqd/022:55:010
1557209904035,3cyclictest9-21ksoftirqd/000:20:040
1557209904034,4cyclictest1599375-21cpuspeed_turbos20:00:070
1557209904034,3cyclictest9-21ksoftirqd/022:10:120
1557209904033,4cyclictest1580384-21sed19:35:130
1557209904029,9cyclictest9-21ksoftirqd/022:55:120
1557234903934,3cyclictest1707777-21pmu-power22:10:152
1557209903933,4cyclictest1697189-21grep22:00:040
1557234903833,2cyclictest33-21ksoftirqd/222:20:022
1557234903827,9cyclictest33-21ksoftirqd/219:55:052
1557209903833,3cyclictest1757496-21unixbench_singl23:10:220
1557209903832,3cyclictest9-21ksoftirqd/021:55:030
1557209903832,3cyclictest9-21ksoftirqd/021:55:020
1557234903732,3cyclictest1789175-21df_abs23:50:122
1557234903730,4cyclictest1810211-21head00:15:152
1557234903530,3cyclictest33-21ksoftirqd/219:50:022
1557234903530,2cyclictest33-21ksoftirqd/219:20:042
1557234903329,2cyclictest33-21ksoftirqd/222:20:042
1557227903227,3cyclictest1664454-21chrt21:20:041
1557227902621,3cyclictest15900842sleep119:50:011
16200482240,3sleep1155722790cyclictest20:25:081
155722790231,17cyclictest0-21swapper/120:14:371
155722790230,20cyclictest0-21swapper/100:05:071
155722790230,20cyclictest0-21swapper/100:05:071
179727322116,2sleep341-21ksoftirqd/300:00:153
16554352210,2sleep10-21swapper/121:10:011
155722790213,15cyclictest0-21swapper/123:20:071
155722790210,18cyclictest0-21swapper/121:10:091
155722790210,18cyclictest0-21swapper/121:10:081
17616872202,12sleep31761688-40dnf23:15:493
172509722014,3sleep31485-21ptp4l22:34:583
155722790201,16cyclictest0-21swapper/123:55:211
155722790200,18cyclictest0-21swapper/122:13:041
155722790200,17cyclictest0-21swapper/121:00:101
18102312190,1sleep10-21swapper/100:15:151
15856982190,2sleep10-21swapper/119:45:001
155722790195,11cyclictest1786495-21taskset23:48:241
155722790191,15cyclictest150150irq/129-enp1s0-23:00:111
155722790191,15cyclictest150150irq/129-enp1s0-23:00:101
155722790191,14cyclictest0-21swapper/120:35:081
155722790190,17cyclictest0-21swapper/100:25:081
155722790190,17cyclictest0-21swapper/100:25:081
155722790190,16cyclictest0-21swapper/122:32:361
155722790190,16cyclictest0-21swapper/121:45:061
155722790190,15cyclictest0-21swapper/122:45:191
155722790190,15cyclictest0-21swapper/121:25:151
17834552182,9sleep31783690-21users23:45:063
17179142182,10sleep31718092-21users22:25:063
17179142182,10sleep31718092-21users22:25:063
167585321813,2sleep31485-21ptp4l21:34:323
167585321813,2sleep31485-21ptp4l21:34:313
15777342180,3sleep1155722790cyclictest19:35:011
15777342180,3sleep1155722790cyclictest19:35:011
155722790181,14cyclictest1709647-21chrt22:15:041
155722790181,14cyclictest1562580-21chrt19:15:081
155722790181,14cyclictest1562580-21chrt19:15:071
155722790181,14cyclictest0-21swapper/122:25:061
155722790181,14cyclictest0-21swapper/122:25:051
155722790181,14cyclictest0-21swapper/121:55:041
155722790181,13cyclictest1599815-21taskset20:00:091
155722790181,13cyclictest0-21swapper/119:10:051
155722790181,13cyclictest0-21swapper/119:10:041
155722790180,16cyclictest0-21swapper/123:40:111
155722790180,16cyclictest0-21swapper/122:35:101
155722790180,15cyclictest17982782sleep100:00:201
155722790180,15cyclictest150150irq/129-enp1s0-21:15:091
155722790180,15cyclictest0-21swapper/123:35:081
155722790180,15cyclictest0-21swapper/123:35:081
155722790180,15cyclictest0-21swapper/120:15:141
155722790171,14cyclictest0-21swapper/123:50:241
155722790171,13cyclictest1742373-21taskset22:55:041
155722790171,13cyclictest15909772sleep119:50:061
155722790170,14cyclictest1812426-21chrt00:20:061
155722790170,14cyclictest0-21swapper/123:10:101
155722790170,14cyclictest0-21swapper/121:40:051
155722790170,14cyclictest0-21swapper/121:35:051
155722790170,14cyclictest0-21swapper/121:30:131
155722790170,14cyclictest0-21swapper/121:30:121
155722790170,14cyclictest0-21swapper/120:55:031
155722790170,14cyclictest0-21swapper/120:43:331
155722790170,14cyclictest0-21swapper/120:20:051
155722790170,14cyclictest0-21swapper/119:55:051
155722790170,14cyclictest0-21swapper/100:40:021
155722790170,13cyclictest16237932sleep120:30:061
17470762162,9sleep31747153-21basename23:00:093
17470762162,9sleep31747153-21basename23:00:083
16682812162,8sleep31668645-21df21:25:043
1557227901612,2cyclictest0-21swapper/123:30:171
155722790161,13cyclictest0-21swapper/122:50:141
155722790161,13cyclictest0-21swapper/100:30:161
155722790160,14cyclictest0-21swapper/123:15:041
155722790160,14cyclictest0-21swapper/123:05:111
155722790160,14cyclictest0-21swapper/122:20:031
155722790160,14cyclictest0-21swapper/120:50:051
155722790160,14cyclictest0-21swapper/120:05:051
155722790160,14cyclictest0-21swapper/119:20:021
155722790160,13cyclictest0-21swapper/122:00:041
155722790160,13cyclictest0-21swapper/120:45:061
155722790160,13cyclictest0-21swapper/100:10:151
155722790160,13cyclictest0-21swapper/100:10:141
155722790160,12cyclictest0-21swapper/119:25:051
16874382152,8sleep31687534-21find21:45:193
155722790150,13cyclictest0-21swapper/119:36:211
155722790150,12cyclictest0-21swapper/123:25:041
18190982140,2sleep30-21swapper/300:25:383
18190982140,2sleep30-21swapper/300:25:373
18126462141,8sleep31812819-21fw_conntrack00:20:083
15876832141,7sleep31587884-21fw_forwarded_lo19:45:123
155722790140,12cyclictest0-21swapper/122:05:091
18072302131,7sleep31807261-21/usr/sbin/munin00:15:033
18072302131,7sleep31807261-21/usr/sbin/munin00:15:023
16807472131,7sleep31681093-21/usr/sbin/munin21:40:043
18030512122,5sleep30-21swapper/300:10:013
18030512122,5sleep30-21swapper/300:10:013
17896042127,2sleep3401ktimersoftd/323:50:153
17762462120,1sleep30-21swapper/323:35:093
17762462120,1sleep30-21swapper/323:35:093
17099532121,6sleep31710149-21iostat22:15:073
16482232121,6sleep31648373-21diskstats21:00:053
16250422122,6sleep31625237-21ls20:30:133
16207332122,6sleep31620866-21iostat20:25:113
16152702121,5sleep31615560-21df_inode20:20:053
16102752122,5sleep30-21swapper/320:14:393
18273492110,8sleep30-21swapper/300:36:083
18234622110,1sleep30-21swapper/300:33:483
17709492111,6sleep31771253-21/usr/sbin/munin23:30:063
17311872112,4sleep30-21swapper/322:40:103
17265812111,6sleep31726777-21memory22:35:093
16798982110,8sleep30-21swapper/321:38:523
16443362111,5sleep31644561-21ps20:55:083
16388302111,5sleep31375-26node-red20:48:023
16144022113,5sleep31609994-21kworker/3:220:20:023
16017202110,8sleep30-21swapper/320:00:283
15771252110,8sleep30-21swapper/319:30:243
15771252110,8sleep30-21swapper/319:30:233
18082472102,4sleep30-21swapper/300:15:073
17947592100,7sleep30-21swapper/323:59:183
17640592101,5sleep31764306-21if_enp4s023:20:103
17522392101,5sleep31752580-21/usr/sbin/munin23:05:143
17168432100,8sleep30-21swapper/322:24:223
17064652101,5sleep31704659-21/usr/sbin/munin22:10:103
16982962101,5sleep31696471-21/usr/sbin/munin22:00:103
16524892101,5sleep31651346-21/usr/sbin/munin21:05:063
16320162102,4sleep30-21swapper/320:40:073
15940692102,4sleep31594348-21latency_hist19:55:023
15702802101,4sleep31570478-21users19:25:063
15665052101,5sleep31566712-21fschecks_time19:20:093
15576662102,4sleep30-21swapper/319:10:063
15576662102,4sleep30-21swapper/319:10:053
1780604291,5sleep31780879-21if_err_enp1s023:40:133
1769510291,4sleep30-21swapper/323:25:203
1701165291,4sleep30-21swapper/322:05:043
1690226291,4sleep30-21swapper/321:50:113
1690226291,4sleep30-21swapper/321:50:113
1743746281,3sleep30-21swapper/322:55:113
1664794281,4sleep30-21swapper/321:20:083
1661148281,3sleep31661105-21kworker/3:021:15:093
1659520280,4sleep31659562-21/usr/sbin/munin21:15:023
1659520280,4sleep31659562-21/usr/sbin/munin21:15:013
1627750280,4sleep31626699-21/usr/sbin/munin20:35:053
1581015281,3sleep30-21swapper/319:35:203
1739120271,3sleep30-21swapper/322:50:093
1640704271,3sleep31639489-21/usr/sbin/munin20:50:103
1602961271,3sleep30-21swapper/320:05:053
1694966260,3sleep31695169-21sed21:55:143
1582621250,2sleep30-21swapper/319:40:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional