You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-23 - 19:53

Intel(R) Atom(TM) CPU E3845 @ 1.91GHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rackdslot2.osadl.org (updated Sun Jan 23, 2022 12:46:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1454921780,8sleep3155699cyclictest07:25:093
1300121650,8sleep2155599cyclictest08:00:372
117828812,63sleep30-21swapper/307:09:073
9772872,20sleep20-21swapper/207:06:332
101228111,18sleep00-21swapper/007:06:580
11342764,11sleep10-21swapper/107:08:331
128502710,5sleep112852-21rt-features10:40:341
128102690,8sleep3155699cyclictest09:20:353
149902680,6sleep0155399cyclictest08:05:130
1556996754,9cyclictest41-21ksoftirqd/307:35:153
155399648,13cyclictest780-21date10:25:380
155399619,13cyclictest30714-21/usr/sbin/munin09:45:090
309032600,1sleep10-21swapper/107:45:101
155399596,13cyclictest27123-21grep09:00:110
55598550,1rtkit-daemon0-21swapper/210:51:472
55598520,2rtkit-daemon0-21swapper/209:55:472
55598520,12rtkit-daemon0-21swapper/211:16:332
168992520,3sleep00-21swapper/009:25:340
155399506,20cyclictest1825-21latency_hist11:50:010
1554994716,28cyclictest26416-21kworker/1:211:45:361
55598460,12rtkit-daemon0-21swapper/208:47:422
1554994629,14cyclictest25-21ksoftirqd/109:42:501
155399467,13cyclictest30360-21basename11:05:020
155399465,13cyclictest14763-21awk11:25:070
1555994516,26cyclictest3521-21tr11:50:162
155599451,42cyclictest0-21swapper/209:00:162
155499450,25cyclictest23189-21/usr/sbin/munin09:35:121
155399455,24cyclictest0-21swapper/009:20:150
1555994422,19cyclictest33-21ksoftirqd/212:28:072
1555994422,19cyclictest33-21ksoftirqd/212:28:072
155599430,41cyclictest0-21swapper/211:55:552
1556994229,8cyclictest41-21ksoftirqd/307:40:143
1555994221,18cyclictest33-21ksoftirqd/207:44:092
155599421,3cyclictest0-21swapper/207:15:162
155599421,17cyclictest18929-21grep08:10:112
1555994115,4cyclictest9255-21sed12:35:402
1555994115,4cyclictest24599-21latency10:15:302
1555994113,5cyclictest13202-21cat11:20:372
1555994112,4cyclictest4815-21cat12:30:282
1555994112,4cyclictest20214-21which10:10:252
1555994111,4cyclictest27679-21cut09:40:152
155399414,13cyclictest15523-21bc08:45:220
155599405,10cyclictest29664-21unixbench_multi11:40:392
1555994013,4cyclictest11986-21perl08:40:252
1555994011,4cyclictest19551-21basename11:30:132
155399404,13cyclictest26425-21/usr/sbin/munin10:20:050
1556993925,7cyclictest41-21ksoftirqd/308:55:193
155599399,5cyclictest14046-21date12:05:002
155599392,7cyclictest0-21swapper/210:45:142
1555993913,4cyclictest8883-21timedrift07:55:402
1555993912,3cyclictest27159-21basename08:20:142
1555993911,4cyclictest13799-21sed08:05:022
155499392,33cyclictest16040-21tune2fs08:45:291
1554993917,19cyclictest17792-21cut09:30:001
1554993917,19cyclictest15889-21expr10:05:211
1556993827,6cyclictest41-21ksoftirqd/309:15:143
1555993814,4cyclictest833-21pluginstate11:45:352
1555993814,4cyclictest23005-21processes12:15:092
1555993812,4cyclictest1188-21runrttasks10:08:222
1555993811,4cyclictest26073-21latency_hist11:40:002
1555993811,4cyclictest13779-21wc09:25:002
155499383,29cyclictest27936-21apt-get11:00:171
155499382,28cyclictest7508-21apt-get07:15:201
1554993821,14cyclictest25-21ksoftirqd/109:06:531
55598370,2rtkit-daemon0-21swapper/308:40:123
1556993730,4cyclictest31813-21grep11:45:163
1556993728,5cyclictest41-21ksoftirqd/312:20:143
155599379,5cyclictest19397-21runrttasks07:30:162
155599371,34cyclictest0-21swapper/208:53:062
1555993713,3cyclictest3925-21fschecks_count09:50:252
1555993713,3cyclictest3925-21fschecks_count09:50:242
155499373,26cyclictest27359-21apt-get10:20:131
155499372,28cyclictest7151-21apt-get10:35:151
155499372,27cyclictest19726-21apt-get09:30:201
1554993717,17cyclictest1719-21/usr/sbin/munin09:10:021
1554993716,18cyclictest7580-21cat09:15:191
1554993716,18cyclictest10385-21sed08:00:091
1556993630,4cyclictest41-21ksoftirqd/310:00:013
1556993629,4cyclictest41-21ksoftirqd/309:35:043
1555993612,4cyclictest11389-21cpuspeed07:20:162
1555993612,4cyclictest11389-21cpuspeed07:20:162
1555993611,4cyclictest31393-21tr08:25:152
155499363,27cyclictest0-21swapper/107:40:131
155499362,28cyclictest27828-21apt-get09:00:191
155499362,28cyclictest15595-21apt-get11:25:141
1554993618,15cyclictest15459-21acpi12:05:141
1554993615,18cyclictest6049-21cut11:54:591
155399366,17cyclictest12871-21awk10:40:360
155399365,18cyclictest10165-21systemctl11:20:030
155399363,19cyclictest7634-21cstates12:35:160
126991367,14phc2sys0-21swapper/307:10:013
1556993528,4cyclictest41-21ksoftirqd/308:00:013
1555993512,3cyclictest32040-21cstates09:45:222
155499354,23cyclictest3099-21apt-get09:50:151
155499354,23cyclictest3099-21apt-get09:50:151
155499352,27cyclictest11536-21apt-get12:00:131
155499352,25cyclictest27693-21apt-get11:40:151
1554993514,18cyclictest1997-21irqrtprio12:30:021
1554993513,18cyclictest4422-21ls10:30:311
1554993511,20cyclictest4135-21cut08:30:311
155499350,21cyclictest24973-21ptp4l08:55:341
155399355,17cyclictest13777-21/usr/sbin/munin07:25:120
1556993428,4cyclictest41-21ksoftirqd/308:30:003
1556993428,4cyclictest41-21ksoftirqd/307:20:003
1556993428,4cyclictest13414-21kworker/3:208:05:403
1555993411,4cyclictest33-21ksoftirqd/210:30:522
155499344,24cyclictest11226-21apt-get10:00:141
155499343,26cyclictest23646-21apt-get11:35:131
155499343,25cyclictest23612-21apt-get08:15:201
155499343,23cyclictest11256-21apt-get09:20:141
155499342,27cyclictest3079-21cat07:50:151
1554993413,18cyclictest22044-21df07:35:031
1554993412,19cyclictest497-21cat11:05:321
1554993412,19cyclictest16351-21cut08:05:311
1554993412,18cyclictest18775-21latency10:50:111
1554993411,19cyclictest15820-21tr10:45:201
155399345,8cyclictest19619-21apt-get11:30:130
1553993427,4cyclictest9-21ksoftirqd/008:21:590
1553993427,4cyclictest9-21ksoftirqd/008:17:460
1553993427,4cyclictest9-21ksoftirqd/007:24:340
1553993427,4cyclictest9-21ksoftirqd/007:24:340
1553993419,9cyclictest27200-21apt-get07:40:120
218232330,6sleep221826-21irqrtprio08:15:012
1556993328,3cyclictest41-21ksoftirqd/312:15:393
1556993327,4cyclictest41-21ksoftirqd/311:15:003
1556993327,4cyclictest41-21ksoftirqd/311:15:003
1556993327,4cyclictest41-21ksoftirqd/309:25:373
1556993327,3cyclictest41-21ksoftirqd/308:55:013
1556993327,3cyclictest12950irq/93-ahci[00012:39:593
1556993326,4cyclictest41-21ksoftirqd/308:03:563
155599333,6cyclictest0-21swapper/209:00:002
1555993326,4cyclictest33-21ksoftirqd/210:24:142
155499334,23cyclictest27247-21apt-get08:20:141
155499334,23cyclictest19281-21apt-get10:10:141
155499333,24cyclictest7490-21apt-get11:55:151
155499333,24cyclictest15165-21apt-get07:25:141
155499333,23cyclictest11085-21apt-get08:40:141
1554993326,4cyclictest25-21ksoftirqd/110:26:221
155499332,24cyclictest3656-21apt-get11:10:191
155499332,24cyclictest3656-21apt-get11:10:191
1554993314,16cyclictest23075-21grep10:15:111
155499331,29cyclictest19638-21kworker/1:008:25:071
1554993312,18cyclictest25812-21seq12:17:091
1554993310,20cyclictest20377-21sed12:10:211
155499330,29cyclictest19599-21bc07:30:181
155399334,9cyclictest2599-21iostat07:50:100
155399334,15cyclictest30832-21ls08:25:100
1553993325,5cyclictest9-21ksoftirqd/012:29:140
1553993325,5cyclictest9-21ksoftirqd/012:29:140
155399330,16cyclictest7828-21df_abs07:15:230
1556993227,3cyclictest41-21ksoftirqd/310:30:133
1556993227,3cyclictest41-21ksoftirqd/309:50:103
1556993227,3cyclictest41-21ksoftirqd/309:50:093
1556993226,4cyclictest41-21ksoftirqd/312:00:303
1556993226,4cyclictest41-21ksoftirqd/308:20:403
1556993226,3cyclictest41-21ksoftirqd/308:31:213
1556993225,4cyclictest41-21ksoftirqd/312:06:543
1556993225,4cyclictest41-21ksoftirqd/309:45:153
1556993225,4cyclictest41-21ksoftirqd/308:37:453
1556993224,4cyclictest41-21ksoftirqd/310:05:223
1554993225,4cyclictest25-21ksoftirqd/108:14:321
1554993212,17cyclictest22643-21df_abs10:55:091
1554993211,18cyclictest19638-21if_enp1s011:30:161
1554993211,18cyclictest1004-21grep09:45:381
1554993210,18cyclictest6721-21tr08:35:121
155399324,8cyclictest9788-21sshd11:58:320
155399323,15cyclictest18400-21ls08:10:060
1553993226,4cyclictest9-21ksoftirqd/012:05:340
1553993225,4cyclictest9-21ksoftirqd/009:35:340
1553993225,4cyclictest9-21ksoftirqd/009:31:410
1553993225,4cyclictest9-21ksoftirqd/007:55:420
117552320,7sleep0155399cyclictest08:40:210
1556993126,3cyclictest41-21ksoftirqd/311:30:103
1556993126,3cyclictest41-21ksoftirqd/311:20:373
1556993126,3cyclictest41-21ksoftirqd/310:10:043
1556993126,3cyclictest41-21ksoftirqd/309:10:373
1556993126,3cyclictest41-21ksoftirqd/309:10:003
1556993125,4cyclictest41-21ksoftirqd/311:35:113
1556993125,4cyclictest41-21ksoftirqd/310:00:333
1556993125,3cyclictest41-21ksoftirqd/312:34:533
1556993125,3cyclictest41-21ksoftirqd/309:33:093
1556993124,4cyclictest41-21ksoftirqd/312:11:073
1556993124,4cyclictest41-21ksoftirqd/310:46:043
1556993124,4cyclictest41-21ksoftirqd/310:22:313
1556993124,4cyclictest41-21ksoftirqd/307:51:423
1555993124,4cyclictest33-21ksoftirqd/211:01:222
1555993124,4cyclictest33-21ksoftirqd/210:57:342
1555993124,4cyclictest33-21ksoftirqd/210:28:152
1555993124,4cyclictest33-21ksoftirqd/209:39:302
1555993124,4cyclictest33-21ksoftirqd/207:36:312
1555993124,4cyclictest33-21ksoftirqd/207:13:002
1555993123,5cyclictest33-21ksoftirqd/207:51:502
1555993123,5cyclictest33-21ksoftirqd/207:45:402
1555993111,3cyclictest3295-21/usr/sbin/munin08:30:352
155499319,19cyclictest13279-21sed11:20:371
155499317,21cyclictest3609-21cpuspeed_turbos07:10:201
155499313,23cyclictest7879-21apt-get12:35:191
1554993113,15cyclictest8143-21interrupts07:55:311
1554993112,16cyclictest1-21systemd08:53:591
1554993110,18cyclictest26571-21switchtime12:20:041
1553993124,4cyclictest9-21ksoftirqd/010:57:480
1553993124,4cyclictest9-21ksoftirqd/010:12:030
1553993118,10cyclictest29808-21cron09:44:590
1553993112,14cyclictest0-21swapper/009:50:150
1553993112,14cyclictest0-21swapper/009:50:140
155399310,12cyclictest0-21swapper/010:50:030
98972300,2sleep0111rcuc/011:20:000
1556993025,3cyclictest41-21ksoftirqd/312:28:443
1556993025,3cyclictest41-21ksoftirqd/312:28:443
1556993025,3cyclictest41-21ksoftirqd/310:55:023
1556993025,3cyclictest41-21ksoftirqd/310:15:393
1556993024,4cyclictest41-21ksoftirqd/311:55:103
1556993024,4cyclictest41-21ksoftirqd/311:25:213
1556993024,4cyclictest41-21ksoftirqd/311:09:593
1556993024,4cyclictest41-21ksoftirqd/309:00:333
1556993024,4cyclictest41-21ksoftirqd/307:45:313
1556993023,4cyclictest41-21ksoftirqd/311:00:273
1556993023,4cyclictest41-21ksoftirqd/310:36:273
1555993023,4cyclictest33-21ksoftirqd/209:31:452
1555993023,4cyclictest33-21ksoftirqd/209:07:262
1555993023,4cyclictest33-21ksoftirqd/208:36:542
155499308,19cyclictest7526-21cat09:55:171
1554993013,14cyclictest1323-21sed12:25:381
1554993013,14cyclictest1323-21sed12:25:381
155499301,25cyclictest6586-21tail11:15:081
155399307,8cyclictest28142-21cstates11:00:200
1553993024,4cyclictest9-21ksoftirqd/012:30:380
1553993024,3cyclictest9-21ksoftirqd/007:39:470
1553993023,4cyclictest9-21ksoftirqd/008:01:230
1553993023,4cyclictest9-21ksoftirqd/007:30:200
1553993015,11cyclictest0-21swapper/007:45:150
1556992923,3cyclictest41-21ksoftirqd/310:43:443
1556992923,3cyclictest41-21ksoftirqd/307:31:063
1556992922,5cyclictest41-21ksoftirqd/308:16:163
1556992922,4cyclictest41-21ksoftirqd/310:28:143
1556992914,8cyclictest41-21ksoftirqd/307:20:133
1556992914,8cyclictest41-21ksoftirqd/307:20:133
155599299,5cyclictest4916-21systemd-udevd10:01:582
1555992922,4cyclictest33-21ksoftirqd/211:25:042
155499291,24cyclictest10898-21sed07:20:121
155499291,24cyclictest10898-21sed07:20:111
1553992922,4cyclictest9-21ksoftirqd/010:19:460
1553992922,4cyclictest9-21ksoftirqd/010:06:300
1553992922,4cyclictest9-21ksoftirqd/008:51:140
1553992922,4cyclictest9-21ksoftirqd/008:35:140
1553992912,12cyclictest0-21swapper/010:00:130
1553992912,12cyclictest0-21swapper/009:15:180
1553992912,11cyclictest0-21swapper/010:35:130
1556992823,4cyclictest41-21ksoftirqd/308:10:023
1556992822,4cyclictest41-21ksoftirqd/311:50:333
1556992822,4cyclictest41-21ksoftirqd/308:45:173
1556992821,4cyclictest41-21ksoftirqd/310:53:113
1555992822,4cyclictest33-21ksoftirqd/210:35:102
1555992821,5cyclictest33-21ksoftirqd/211:12:402
1555992821,5cyclictest33-21ksoftirqd/211:12:392
1555992821,4cyclictest33-21ksoftirqd/212:13:232
1555992821,4cyclictest33-21ksoftirqd/212:06:592
1555992821,4cyclictest33-21ksoftirqd/210:43:282
155399289,13cyclictest0-21swapper/012:00:140
1553992822,4cyclictest9-21ksoftirqd/011:12:340
1553992822,4cyclictest9-21ksoftirqd/011:12:340
1553992822,4cyclictest9-21ksoftirqd/010:48:440
1553992821,4cyclictest9-21ksoftirqd/012:21:270
1553992821,4cyclictest9-21ksoftirqd/012:19:540
1553992821,4cyclictest9-21ksoftirqd/009:14:430
1553992821,4cyclictest9-21ksoftirqd/009:05:220
1553992811,13cyclictest0-21swapper/011:40:150
1553992811,12cyclictest0-21swapper/008:30:140
1553992810,15cyclictest2009-21/usr/sbin/munin11:50:060
1556992722,3cyclictest41-21ksoftirqd/309:40:043
1556992721,4cyclictest41-21ksoftirqd/311:43:213
1555992720,4cyclictest9197-21grep09:15:392
155599270,25cyclictest0-21swapper/209:26:262
155399271,11cyclictest0-21swapper/008:55:200
1556992620,4cyclictest41-21ksoftirqd/311:15:263
1555992620,3cyclictest33-21ksoftirqd/212:23:162
1555992618,5cyclictest33-21ksoftirqd/209:12:442
1555992518,4cyclictest33-21ksoftirqd/207:25:242
155399259,13cyclictest3119-21cpuspeed_turbos10:30:140
1553992514,8cyclictest2717-21cut07:10:120
1553992512,9cyclictest23460-21grep11:35:110
1555992417,4cyclictest33-21ksoftirqd/211:06:092
1553992411,9cyclictest6930-21grep09:55:120
1553992410,10cyclictest18578-21unixbench_singl12:10:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional