You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-08-04 - 21:57

Intel(R) Atom(TM) CPU E3845 @ 1.91GHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rackdslot2.osadl.org (updated Wed Aug 04, 2021 12:46:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
278521770,8sleep22966699cyclictest07:55:122
848321620,8sleep32966799cyclictest08:40:293
848321620,8sleep32966799cyclictest08:40:293
1268911340,107ptp4l401ktimersoftd/307:05:393
555981100,21rtkit-daemon0-21swapper/107:07:411
29666998976,7cyclictest33-21ksoftirqd/209:55:132
29666998874,5cyclictest33-21ksoftirqd/210:20:152
29666998569,8cyclictest33-21ksoftirqd/212:20:132
29666998469,8cyclictest33-21ksoftirqd/208:45:122
29666998469,8cyclictest33-21ksoftirqd/208:45:112
29666998369,7cyclictest33-21ksoftirqd/209:20:142
2905028315,18sleep20-21swapper/207:05:482
29666998172,4cyclictest33-21ksoftirqd/210:25:152
29666997662,8cyclictest33-21ksoftirqd/207:10:142
29666997562,7cyclictest33-21ksoftirqd/212:05:162
29666997562,7cyclictest33-21ksoftirqd/209:40:132
29666996957,7cyclictest33-21ksoftirqd/209:25:182
29666996956,7cyclictest33-21ksoftirqd/208:00:172
29666996859,5cyclictest33-21ksoftirqd/208:05:162
29666996756,6cyclictest33-21ksoftirqd/208:50:122
29666996755,7cyclictest33-21ksoftirqd/211:45:162
29666996755,6cyclictest33-21ksoftirqd/211:30:132
29666996755,6cyclictest33-21ksoftirqd/207:45:122
29666996754,7cyclictest33-21ksoftirqd/212:30:192
29666996754,7cyclictest33-21ksoftirqd/212:30:182
125412670,2sleep00-21swapper/010:45:370
29666996653,7cyclictest33-21ksoftirqd/209:15:122
29666996652,8cyclictest33-21ksoftirqd/208:35:192
280052644,45sleep00-21swapper/007:05:210
311212600,2sleep00-21swapper/007:10:120
29666996049,6cyclictest33-21ksoftirqd/212:35:122
29666996048,7cyclictest33-21ksoftirqd/209:50:142
289932600,4sleep128997-21timerwakeupswit09:05:321
29666995946,6cyclictest33-21ksoftirqd/208:15:172
29666995845,6cyclictest33-21ksoftirqd/212:10:132
12972580,5sleep233-21ksoftirqd/211:15:012
126891580,2ptp4l0-21swapper/311:40:073
29666995647,4cyclictest33-21ksoftirqd/207:15:172
29666995647,4cyclictest33-21ksoftirqd/207:15:162
29666995549,3cyclictest33-21ksoftirqd/207:20:122
29666995443,6cyclictest33-21ksoftirqd/207:40:132
29666995245,4cyclictest33-21ksoftirqd/211:50:052
29666995139,6cyclictest33-21ksoftirqd/207:30:132
2966699505,4cyclictest121rcu_preempt09:35:072
126991480,3phc2sys121rcu_preempt12:25:013
29666994736,6cyclictest33-21ksoftirqd/209:00:202
29664994727,4cyclictest9-21ksoftirqd/010:14:020
29666994639,4cyclictest33-21ksoftirqd/211:10:202
29666994540,3cyclictest33-21ksoftirqd/212:25:122
29666994538,4cyclictest33-21ksoftirqd/208:10:142
29665994538,4cyclictest1160-21cron10:35:001
29664994534,6cyclictest9-21ksoftirqd/007:35:140
126891450,1ptp4l401ktimersoftd/309:11:543
55598440,3rtkit-daemon554-21rtkit-daemon10:19:312
29666994438,3cyclictest33-21ksoftirqd/210:00:142
29666994437,4cyclictest33-21ksoftirqd/208:30:142
29666994436,4cyclictest33-21ksoftirqd/210:05:162
29666994434,6cyclictest33-21ksoftirqd/211:35:192
126891440,1ptp4l401ktimersoftd/311:50:023
126891440,1ptp4l401ktimersoftd/310:45:193
126891440,1ptp4l401ktimersoftd/308:39:573
29667994326,14cyclictest41-21ksoftirqd/309:28:463
2966799432,11cyclictest31468-21grep07:50:153
29666994313,5cyclictest33-21ksoftirqd/210:30:542
126891430,1ptp4l401ktimersoftd/309:55:173
126891430,1ptp4l401ktimersoftd/309:15:053
29667994235,4cyclictest41-21ksoftirqd/311:20:133
29666994235,4cyclictest33-21ksoftirqd/212:00:002
29666994235,4cyclictest33-21ksoftirqd/210:55:002
29666994235,4cyclictest33-21ksoftirqd/210:40:012
29666994230,10cyclictest33-21ksoftirqd/209:10:032
29666994229,10cyclictest33-21ksoftirqd/212:15:152
2966799413,7cyclictest0-21swapper/312:30:133
2966799413,7cyclictest0-21swapper/312:30:123
2966799413,7cyclictest0-21swapper/308:30:123
29667994124,14cyclictest41-21ksoftirqd/311:07:233
29666994135,3cyclictest33-21ksoftirqd/212:00:072
29666994128,4cyclictest33-21ksoftirqd/207:50:032
29666994128,10cyclictest33-21ksoftirqd/207:25:082
29666994128,10cyclictest33-21ksoftirqd/207:25:072
29664994134,4cyclictest210-21systemd-journal10:35:000
2966799403,7cyclictest11111-21acpi10:05:133
2966799403,7cyclictest0-21swapper/311:30:153
2966799402,8cyclictest0-21swapper/311:00:133
2966799402,7cyclictest0-21swapper/307:30:223
29666994034,4cyclictest33-21ksoftirqd/210:45:202
29666994033,4cyclictest33-21ksoftirqd/211:25:002
29666994033,4cyclictest33-21ksoftirqd/209:05:152
29666994033,4cyclictest33-21ksoftirqd/208:20:122
29666994031,4cyclictest33-21ksoftirqd/209:30:132
29666994028,10cyclictest33-21ksoftirqd/210:40:502
29666994027,10cyclictest33-21ksoftirqd/208:40:032
29666994027,10cyclictest33-21ksoftirqd/208:40:022
29666994025,4cyclictest33-21ksoftirqd/211:40:072
29664994022,5cyclictest9-21ksoftirqd/011:45:340
2966799392,7cyclictest0-21swapper/309:00:193
2966799392,6cyclictest0-21swapper/310:50:153
2966799391,8cyclictest2690-21acpi07:15:173
2966799391,8cyclictest2690-21acpi07:15:163
29666993932,4cyclictest33-21ksoftirqd/209:45:132
29666993919,11cyclictest33-21ksoftirqd/211:00:202
126891390,3ptp4l2966799cyclictest11:12:063
2966799385,9cyclictest18997-21cut08:55:143
2966799385,10cyclictest29499-21/usr/sbin/munin07:50:013
2966799383,5cyclictest10788-21apt-get08:45:113
2966799383,5cyclictest10788-21apt-get08:45:103
2966799383,5cyclictest0-21swapper/311:35:123
2966799382,8cyclictest0-21swapper/308:20:163
2966799382,6cyclictest0-21swapper/310:55:143
29667993814,21cyclictest26467-21df_abs09:45:073
29667993811,4cyclictest3064-21swap10:35:133
29666993834,3cyclictest33-21ksoftirqd/210:55:032
29666993832,3cyclictest33-21ksoftirqd/208:25:132
29666993831,4cyclictest33-21ksoftirqd/211:25:082
29666993831,4cyclictest33-21ksoftirqd/210:15:002
2966799378,5cyclictest4502-21switchtime11:15:423
2966799373,5cyclictest0-21swapper/311:25:123
2966799372,7cyclictest0-21swapper/310:20:123
2966799372,7cyclictest0-21swapper/309:35:133
29667993713,4cyclictest7393-21cat12:00:183
29667993711,5cyclictest27676-21tr09:05:173
29667993711,4cyclictest30638-21missed_timers10:30:083
29664993731,3cyclictest9-21ksoftirqd/007:24:230
1269913729,4phc2sys41-21ksoftirqd/307:25:033
1269913729,4phc2sys41-21ksoftirqd/307:25:033
2966799368,4cyclictest11932-21fschecks_time08:05:233
2966799362,8cyclictest19456-21apt-get08:15:153
2966799362,7cyclictest27426-21cstates08:25:143
2966799362,6cyclictest10712-21apt-get12:05:153
2966799362,5cyclictest0-21swapper/310:10:143
2966799362,5cyclictest0-21swapper/308:00:123
2966799361,7cyclictest0-21swapper/311:55:153
29667993612,3cyclictest27157-21/usr/sbin/munin10:25:553
29667993611,4cyclictest23431-21fschecks_count07:40:153
29666993630,3cyclictest33-21ksoftirqd/208:55:032
2966799358,4cyclictest17647-21/usr/sbin/munin07:35:013
2966799352,7cyclictest0-21swapper/310:15:193
29667993513,3cyclictest14800-21/usr/sbin/munin12:10:283
29667993512,3cyclictest14367-21df_abs08:10:073
29667993511,4cyclictest13873-21sensors_temp09:30:013
29666993529,3cyclictest33-21ksoftirqd/211:05:022
29665993529,4cyclictest25-21ksoftirqd/110:49:181
29665993529,4cyclictest25-21ksoftirqd/109:30:041
29665993529,4cyclictest25-21ksoftirqd/108:15:261
29665993529,3cyclictest25-21ksoftirqd/109:44:561
29665993528,4cyclictest25-21ksoftirqd/107:51:591
29664993529,3cyclictest9-21ksoftirqd/012:10:020
29664993528,4cyclictest9-21ksoftirqd/009:32:420
29664993528,4cyclictest9-21ksoftirqd/008:10:550
186582350,5sleep1241ktimersoftd/111:35:111
2966799348,4cyclictest3861-21fschecks_time07:55:183
2966799342,6cyclictest14948-21apt-get08:50:133
2966799341,4cyclictest18034-21sed12:15:043
29667993411,3cyclictest7396-21expr07:20:153
29666993422,10cyclictest33-21ksoftirqd/207:35:012
29665993428,4cyclictest25-21ksoftirqd/110:03:341
29665993428,4cyclictest25-21ksoftirqd/107:16:401
29665993428,4cyclictest25-21ksoftirqd/107:16:391
29665993427,4cyclictest25-21ksoftirqd/112:27:001
29665993427,4cyclictest25-21ksoftirqd/112:06:321
29665993427,4cyclictest25-21ksoftirqd/111:26:331
29665993426,5cyclictest25-21ksoftirqd/108:44:091
29665993426,5cyclictest25-21ksoftirqd/108:44:091
29664993428,3cyclictest9-21ksoftirqd/008:18:080
29664993427,4cyclictest9-21ksoftirqd/011:52:330
29664993427,4cyclictest9-21ksoftirqd/011:18:120
29664993427,4cyclictest9-21ksoftirqd/009:15:230
29664993424,4cyclictest9-21ksoftirqd/011:22:210
2966799339,4cyclictest23623-21fschecks_count09:40:153
2966799338,4cyclictest6664-21unixbench_multi10:40:093
2966799338,4cyclictest6651-21ls09:20:103
2966799332,7cyclictest2711-21apt-get12:35:123
29667993327,3cyclictest41-21ksoftirqd/312:27:073
2966799332,6cyclictest27229-21apt-get11:45:163
29665993327,4cyclictest25-21ksoftirqd/112:35:121
29665993327,3cyclictest25-21ksoftirqd/109:24:101
29665993326,4cyclictest25-21ksoftirqd/111:59:381
29665993326,4cyclictest25-21ksoftirqd/110:52:401
29665993326,4cyclictest25-21ksoftirqd/108:31:311
29665993325,5cyclictest25-21ksoftirqd/108:59:101
29665993325,5cyclictest25-21ksoftirqd/107:11:021
2966599330,4cyclictest121rcu_preempt09:10:201
29664993327,4cyclictest9-21ksoftirqd/008:25:380
29664993327,3cyclictest9-21ksoftirqd/010:28:080
29664993327,3cyclictest9-21ksoftirqd/007:57:490
29664993326,4cyclictest9-21ksoftirqd/012:30:550
29664993326,4cyclictest9-21ksoftirqd/012:30:550
29664993326,4cyclictest9-21ksoftirqd/012:24:260
29664993326,4cyclictest9-21ksoftirqd/010:53:210
29664993326,4cyclictest9-21ksoftirqd/009:27:590
29664993326,4cyclictest9-21ksoftirqd/008:36:290
29664993317,4cyclictest9-21ksoftirqd/008:31:140
2966799329,4cyclictest29752-21/usr/sbin/munin09:50:033
2966799329,3cyclictest30510-21missed_timers07:10:083
29667993225,4cyclictest41-21ksoftirqd/310:01:043
29665993226,4cyclictest25-21ksoftirqd/111:11:351
29665993226,4cyclictest25-21ksoftirqd/110:35:261
29665993225,4cyclictest25-21ksoftirqd/111:09:461
29665993225,4cyclictest25-21ksoftirqd/110:56:221
29665993225,4cyclictest25-21ksoftirqd/110:14:251
29665993225,4cyclictest25-21ksoftirqd/109:48:201
29665993225,4cyclictest25-21ksoftirqd/109:16:411
29665993225,4cyclictest25-21ksoftirqd/107:25:111
29665993225,4cyclictest25-21ksoftirqd/107:25:111
29665993225,4cyclictest25-21ksoftirqd/107:23:391
29664993226,4cyclictest9-21ksoftirqd/007:43:130
29664993226,4cyclictest9-21ksoftirqd/007:27:240
29664993226,4cyclictest9-21ksoftirqd/007:27:240
29664993226,3cyclictest9-21ksoftirqd/012:28:040
29664993226,3cyclictest9-21ksoftirqd/008:58:060
29664993225,4cyclictest9-21ksoftirqd/011:36:120
29664993225,4cyclictest9-21ksoftirqd/011:32:060
29664993225,4cyclictest9-21ksoftirqd/008:07:410
29664993225,4cyclictest9-21ksoftirqd/008:02:280
29664993225,4cyclictest9-21ksoftirqd/007:45:380
29664993225,4cyclictest9-21ksoftirqd/007:16:470
29664993225,4cyclictest9-21ksoftirqd/007:16:470
29664993224,5cyclictest9-21ksoftirqd/009:37:310
29664993220,9cyclictest8484-21/usr/sbin/munin08:40:290
29664993220,9cyclictest8484-21/usr/sbin/munin08:40:280
2966599315,24cyclictest21272-21kworker/1:209:26:371
29665993126,3cyclictest25-21ksoftirqd/112:13:171
29665993125,4cyclictest25-21ksoftirqd/107:55:591
29665993124,5cyclictest25-21ksoftirqd/107:44:381
29665993124,4cyclictest25-21ksoftirqd/109:36:191
29665993124,4cyclictest25-21ksoftirqd/108:25:251
29665993124,4cyclictest25-21ksoftirqd/108:07:501
29664993125,4cyclictest9-21ksoftirqd/009:23:300
29664993124,4cyclictest9-21ksoftirqd/011:41:160
29664993124,4cyclictest9-21ksoftirqd/010:58:210
29664993124,4cyclictest9-21ksoftirqd/010:21:200
29664993124,4cyclictest9-21ksoftirqd/008:21:550
29664993116,11cyclictest5307-21cron10:40:010
29664993115,4cyclictest9-21ksoftirqd/009:03:180
29665993024,4cyclictest25-21ksoftirqd/108:49:181
29665993024,4cyclictest25-21ksoftirqd/108:49:181
29665993024,4cyclictest25-21ksoftirqd/107:32:391
29665993023,5cyclictest25-21ksoftirqd/110:41:391
29665993023,4cyclictest25-21ksoftirqd/112:20:121
29665993023,4cyclictest25-21ksoftirqd/111:22:341
29664993024,3cyclictest9-21ksoftirqd/010:17:230
29664993024,3cyclictest9-21ksoftirqd/010:00:360
29664993023,4cyclictest9-21ksoftirqd/012:38:220
29664993023,4cyclictest9-21ksoftirqd/012:06:100
29664993023,4cyclictest9-21ksoftirqd/011:28:330
29664993023,4cyclictest9-21ksoftirqd/009:49:040
29664993023,4cyclictest9-21ksoftirqd/009:12:570
29664993023,4cyclictest9-21ksoftirqd/007:52:120
29665992923,4cyclictest25-21ksoftirqd/112:01:391
29665992923,4cyclictest25-21ksoftirqd/111:34:301
29665992923,3cyclictest25-21ksoftirqd/108:50:411
29665992922,4cyclictest25-21ksoftirqd/112:33:211
29665992922,4cyclictest25-21ksoftirqd/112:33:211
29665992922,4cyclictest25-21ksoftirqd/110:19:311
29665992922,4cyclictest25-21ksoftirqd/108:12:271
29665992922,4cyclictest25-21ksoftirqd/107:37:441
2966599291,3cyclictest121rcu_preempt11:45:351
29664992924,3cyclictest9-21ksoftirqd/012:00:120
29664992923,4cyclictest9-21ksoftirqd/007:31:360
29664992923,3cyclictest9-21ksoftirqd/012:18:450
29664992922,4cyclictest9-21ksoftirqd/011:11:060
29664992922,4cyclictest9-21ksoftirqd/010:06:340
29664992922,4cyclictest9-21ksoftirqd/009:54:330
29664992921,5cyclictest9-21ksoftirqd/011:08:530
29665992822,4cyclictest25-21ksoftirqd/110:23:091
29665992822,4cyclictest25-21ksoftirqd/108:23:251
29665992822,3cyclictest25-21ksoftirqd/110:25:051
29665992821,4cyclictest25-21ksoftirqd/111:52:291
29665992821,4cyclictest25-21ksoftirqd/108:37:131
29664992823,3cyclictest9-21ksoftirqd/009:06:000
29664992822,4cyclictest9-21ksoftirqd/009:55:080
29665992721,3cyclictest25-21ksoftirqd/111:00:491
29665992720,4cyclictest25-21ksoftirqd/112:15:221
29665992720,4cyclictest25-21ksoftirqd/109:01:511
29664992720,4cyclictest9-21ksoftirqd/010:43:080
29664992720,4cyclictest9-21ksoftirqd/008:48:260
29664992720,4cyclictest9-21ksoftirqd/008:48:260
29664992619,5cyclictest9-21ksoftirqd/009:41:520
2966599259,14cyclictest21272-21kworker/1:209:55:261
29665992519,3cyclictest25-21ksoftirqd/111:41:351
29665992518,4cyclictest25-21ksoftirqd/107:46:261
2966599250,23cyclictest0-21swapper/111:17:051
29664992519,3cyclictest9-21ksoftirqd/011:02:530
29664992518,4cyclictest9-21ksoftirqd/011:56:190
29664992518,4cyclictest9-21ksoftirqd/008:53:140
29665992417,4cyclictest25-21ksoftirqd/109:54:181
29665992216,4cyclictest25-21ksoftirqd/108:03:511
29665992215,4cyclictest25-21ksoftirqd/110:06:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional