You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-07-29 - 17:59

Intel(R) Atom(TM) CPU D2700 @ 2.13GHz, Linux 4.4.39-rt50 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6.osadl.org (updated Thu Jul 29, 2021 12:43:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
91680,0rcu_sched3-21ksoftirqd/009:23:150
91640,0rcu_sched37-21ksoftirqd/307:18:093
81630,0rcu_preempt37-21ksoftirqd/309:53:423
81620,0rcu_preempt21-21ksoftirqd/109:08:141
81610,0rcu_preempt3-21ksoftirqd/007:48:140
91600,0rcu_sched37-21ksoftirqd/312:03:163
91600,0rcu_sched21-21ksoftirqd/110:18:151
91590,0rcu_sched37-21ksoftirqd/311:48:153
91590,0rcu_sched37-21ksoftirqd/307:38:143
91590,0rcu_sched21-21ksoftirqd/109:18:151
91590,0rcu_sched21-21ksoftirqd/108:18:141
81590,0rcu_preempt29-21ksoftirqd/208:58:262
81590,0rcu_preempt21-21ksoftirqd/110:43:161
81580,0rcu_preempt21-21ksoftirqd/110:43:151
81570,0rcu_preempt3-21ksoftirqd/012:13:150
91560,0rcu_sched37-21ksoftirqd/312:13:153
81560,0rcu_preempt37-21ksoftirqd/310:38:363
81560,0rcu_preempt37-21ksoftirqd/307:48:143
11870995620,0cyclictest22219-21fschecks_time10:18:311
81550,0rcu_preempt37-21ksoftirqd/309:13:153
81550,0rcu_preempt29-21ksoftirqd/210:55:262
81550,0rcu_preempt29-21ksoftirqd/207:13:372
11870995514,0cyclictest14033-21needreboot12:33:431
81540,0rcu_preempt740-21systemd-cgroups09:28:150
81540,0rcu_preempt37-21ksoftirqd/310:13:303
81540,0rcu_preempt37-21ksoftirqd/308:18:143
81540,0rcu_preempt37-21ksoftirqd/308:08:323
81540,0rcu_preempt29-21ksoftirqd/212:18:402
81540,0rcu_preempt29-21ksoftirqd/211:43:372
81540,0rcu_preempt29-21ksoftirqd/207:38:492
81540,0rcu_preempt21-21ksoftirqd/111:23:391
81540,0rcu_preempt21-21ksoftirqd/111:23:391
11870995422,0cyclictest2572-21crond12:08:151
11870995421,0cyclictest13718-21crond07:23:141
11870995418,0cyclictest5751-21needreboot12:13:391
91530,0rcu_sched21-21ksoftirqd/110:03:151
81530,0rcu_preempt37-21ksoftirqd/312:43:323
81530,0rcu_preempt37-21ksoftirqd/311:13:143
81530,0rcu_preempt37-21ksoftirqd/309:23:363
81530,0rcu_preempt37-21ksoftirqd/308:43:453
81530,0rcu_preempt29-21ksoftirqd/210:28:412
81530,0rcu_preempt29-21ksoftirqd/210:13:472
81530,0rcu_preempt29-21ksoftirqd/208:43:442
81530,0rcu_preempt29-21ksoftirqd/208:33:442
81530,0rcu_preempt29-21ksoftirqd/208:18:312
81530,0rcu_preempt29-21ksoftirqd/207:58:342
81530,0rcu_preempt29-21ksoftirqd/207:48:422
81530,0rcu_preempt21-21ksoftirqd/109:48:141
11870995318,0cyclictest12835-21gltestperf11:13:311
81520,0rcu_preempt37-21ksoftirqd/308:58:293
81520,0rcu_preempt29-21ksoftirqd/211:18:282
81520,0rcu_preempt29-21ksoftirqd/209:13:312
81520,0rcu_preempt29-21ksoftirqd/209:08:302
81520,0rcu_preempt29-21ksoftirqd/209:03:532
81520,0rcu_preempt29-21ksoftirqd/208:58:152
81520,0rcu_preempt29-21ksoftirqd/208:58:152
81520,0rcu_preempt21-21ksoftirqd/109:53:141
11870995217,0cyclictest3290-21grep12:08:331
11870995214,0cyclictest22012-21nscd11:08:151
91510,0rcu_sched21-21ksoftirqd/111:48:151
81510,0rcu_preempt37-21ksoftirqd/310:48:503
81510,0rcu_preempt37-21ksoftirqd/310:28:293
81510,0rcu_preempt37-21ksoftirqd/310:18:363
81510,0rcu_preempt37-21ksoftirqd/309:33:383
81510,0rcu_preempt37-21ksoftirqd/307:38:343
81510,0rcu_preempt29-21ksoftirqd/212:23:312
81510,0rcu_preempt29-21ksoftirqd/211:38:342
81510,0rcu_preempt29-21ksoftirqd/208:48:302
81510,0rcu_preempt29-21ksoftirqd/207:28:342
81510,0rcu_preempt21-21ksoftirqd/110:03:321
81510,0rcu_preempt21-21ksoftirqd/108:58:391
81510,0rcu_preempt21-21ksoftirqd/107:33:141
1599510,0migration/010197-21fschecks_time07:13:270
11870995123,0cyclictest21-21ksoftirqd/109:56:051
11870995118,0cyclictest26501-21grep07:53:271
91500,0rcu_sched21-21ksoftirqd/108:08:141
91500,0rcu_sched21-21ksoftirqd/108:08:141
81500,0rcu_preempt37-21ksoftirqd/310:23:453
81500,0rcu_preempt37-21ksoftirqd/310:03:453
81500,0rcu_preempt37-21ksoftirqd/307:32:483
81500,0rcu_preempt29-21ksoftirqd/211:33:372
81500,0rcu_preempt29-21ksoftirqd/211:13:262
81500,0rcu_preempt29-21ksoftirqd/210:58:262
81500,0rcu_preempt29-21ksoftirqd/210:08:322
81500,0rcu_preempt29-21ksoftirqd/209:38:492
81500,0rcu_preempt29-21ksoftirqd/208:23:352
81500,0rcu_preempt29-21ksoftirqd/208:08:442
81500,0rcu_preempt29-21ksoftirqd/207:28:142
81500,0rcu_preempt21-21ksoftirqd/111:08:421
81500,0rcu_preempt21-21ksoftirqd/110:13:151
81500,0rcu_preempt21-21ksoftirqd/109:38:521
81500,0rcu_preempt21-21ksoftirqd/108:53:301
81500,0rcu_preempt21-21ksoftirqd/108:53:301
81500,0rcu_preempt21-21ksoftirqd/108:28:281
81500,0rcu_preempt21-21ksoftirqd/108:23:461
81500,0rcu_preempt21-21ksoftirqd/107:38:401
1899500,0migration/19978-21cstates07:13:251
11870995024,0cyclictest21-21ksoftirqd/111:51:031
11870995022,0cyclictest21-21ksoftirqd/112:22:221
91490,0rcu_sched21-21ksoftirqd/109:38:141
91490,0rcu_sched21-21ksoftirqd/108:38:151
81490,0rcu_preempt732-21proc_pri10:43:402
81490,0rcu_preempt7111-21crond09:43:150
81490,0rcu_preempt37-21ksoftirqd/312:38:163
81490,0rcu_preempt37-21ksoftirqd/312:18:323
81490,0rcu_preempt37-21ksoftirqd/311:38:253
81490,0rcu_preempt37-21ksoftirqd/311:28:323
81490,0rcu_preempt37-21ksoftirqd/311:13:313
81490,0rcu_preempt37-21ksoftirqd/308:53:303
81490,0rcu_preempt37-21ksoftirqd/308:53:303
81490,0rcu_preempt37-21ksoftirqd/308:03:273
81490,0rcu_preempt37-21ksoftirqd/308:03:273
81490,0rcu_preempt37-21ksoftirqd/307:52:083
81490,0rcu_preempt37-21ksoftirqd/307:23:483
81490,0rcu_preempt29-21ksoftirqd/212:43:412
81490,0rcu_preempt29-21ksoftirqd/212:38:432
81490,0rcu_preempt29-21ksoftirqd/212:33:462
81490,0rcu_preempt29-21ksoftirqd/212:33:162
81490,0rcu_preempt29-21ksoftirqd/212:08:162
81490,0rcu_preempt29-21ksoftirqd/211:48:302
81490,0rcu_preempt29-21ksoftirqd/211:23:262
81490,0rcu_preempt29-21ksoftirqd/211:23:262
81490,0rcu_preempt29-21ksoftirqd/210:48:162
81490,0rcu_preempt29-21ksoftirqd/210:38:322
81490,0rcu_preempt29-21ksoftirqd/210:33:522
81490,0rcu_preempt29-21ksoftirqd/210:03:302
81490,0rcu_preempt29-21ksoftirqd/210:01:512
81490,0rcu_preempt29-21ksoftirqd/209:33:422
81490,0rcu_preempt29-21ksoftirqd/209:23:282
81490,0rcu_preempt29-21ksoftirqd/208:28:432
81490,0rcu_preempt29-21ksoftirqd/207:18:312
81490,0rcu_preempt21-21ksoftirqd/110:58:471
11870994925,0cyclictest21-21ksoftirqd/109:19:181
11870994924,0cyclictest21-21ksoftirqd/112:28:331
11870994924,0cyclictest21-21ksoftirqd/107:51:341
11870994923,0cyclictest21-21ksoftirqd/112:27:341
11870994922,0cyclictest21-21ksoftirqd/107:25:011
11870994921,0cyclictest5825-21grep08:18:471
11870994917,0cyclictest14787-21munin-node08:43:351
81480,0rcu_preempt37-21ksoftirqd/312:38:383
81480,0rcu_preempt37-21ksoftirqd/311:53:313
81480,0rcu_preempt37-21ksoftirqd/311:23:353
81480,0rcu_preempt37-21ksoftirqd/311:23:353
81480,0rcu_preempt37-21ksoftirqd/310:53:373
81480,0rcu_preempt37-21ksoftirqd/309:38:273
81480,0rcu_preempt37-21ksoftirqd/308:38:323
81480,0rcu_preempt37-21ksoftirqd/308:37:323
81480,0rcu_preempt37-21ksoftirqd/308:18:303
81480,0rcu_preempt37-21ksoftirqd/307:18:313
81480,0rcu_preempt29-21ksoftirqd/212:13:292
81480,0rcu_preempt29-21ksoftirqd/211:58:292
81480,0rcu_preempt29-21ksoftirqd/210:23:152
81480,0rcu_preempt29-21ksoftirqd/209:53:282
81480,0rcu_preempt29-21ksoftirqd/209:48:152
81480,0rcu_preempt29-21ksoftirqd/209:28:442
81480,0rcu_preempt29-21ksoftirqd/208:03:352
81480,0rcu_preempt29-21ksoftirqd/208:03:352
81480,0rcu_preempt29-21ksoftirqd/207:43:332
81480,0rcu_preempt21-21ksoftirqd/112:03:071
81480,0rcu_preempt21-21ksoftirqd/111:53:461
81480,0rcu_preempt21-21ksoftirqd/111:38:291
81480,0rcu_preempt21-21ksoftirqd/110:53:371
81480,0rcu_preempt21-21ksoftirqd/110:48:531
81480,0rcu_preempt21-21ksoftirqd/109:28:261
11870994826,0cyclictest21-21ksoftirqd/111:35:521
11870994823,0cyclictest21-21ksoftirqd/111:20:571
11870994823,0cyclictest21-21ksoftirqd/110:29:531
81470,0rcu_preempt37-21ksoftirqd/312:23:283
81470,0rcu_preempt37-21ksoftirqd/310:33:323
81470,0rcu_preempt37-21ksoftirqd/309:48:143
81470,0rcu_preempt37-21ksoftirqd/308:23:343
81470,0rcu_preempt37-21ksoftirqd/307:53:313
81470,0rcu_preempt29-21ksoftirqd/211:53:352
81470,0rcu_preempt29-21ksoftirqd/211:28:262
81470,0rcu_preempt29-21ksoftirqd/211:08:382
81470,0rcu_preempt29-21ksoftirqd/210:18:282
81470,0rcu_preempt29-21ksoftirqd/208:38:272
81470,0rcu_preempt29-21ksoftirqd/207:53:272
81470,0rcu_preempt21-21ksoftirqd/112:43:441
81470,0rcu_preempt21-21ksoftirqd/110:33:521
81470,0rcu_preempt21-21ksoftirqd/110:23:331
81470,0rcu_preempt21-21ksoftirqd/108:48:421
81470,0rcu_preempt21-21ksoftirqd/108:43:101
11870994717,0cyclictest32575-21munin-node08:08:351
81460,0rcu_preempt37-21ksoftirqd/312:13:293
81460,0rcu_preempt37-21ksoftirqd/312:08:153
81460,0rcu_preempt37-21ksoftirqd/311:18:353
81460,0rcu_preempt37-21ksoftirqd/311:03:423
81460,0rcu_preempt37-21ksoftirqd/309:48:473
81460,0rcu_preempt37-21ksoftirqd/309:28:383
81460,0rcu_preempt37-21ksoftirqd/309:18:153
81460,0rcu_preempt37-21ksoftirqd/309:08:283
81460,0rcu_preempt3-21ksoftirqd/007:18:140
81460,0rcu_preempt21-21ksoftirqd/109:08:421
11870994621,0cyclictest21-21ksoftirqd/107:37:231
91450,0rcu_sched21-21ksoftirqd/107:48:141
81450,0rcu_preempt37-21ksoftirqd/312:33:163
81450,0rcu_preempt37-21ksoftirqd/311:33:163
81450,0rcu_preempt37-21ksoftirqd/311:03:153
81450,0rcu_preempt37-21ksoftirqd/310:08:323
81450,0rcu_preempt37-21ksoftirqd/309:58:313
81450,0rcu_preempt37-21ksoftirqd/308:48:263
81450,0rcu_preempt29-21ksoftirqd/211:03:252
81450,0rcu_preempt29-21ksoftirqd/209:52:352
81450,0rcu_preempt29-21ksoftirqd/209:18:262
81450,0rcu_preempt29-21ksoftirqd/208:13:452
81450,0rcu_preempt29-21ksoftirqd/207:37:432
81450,0rcu_preempt21-21ksoftirqd/112:38:341
81450,0rcu_preempt21-21ksoftirqd/111:28:311
81450,0rcu_preempt10022-21munin-run11:08:150
91440,0rcu_sched0-21swapper/007:33:140
81440,0rcu_preempt37-21ksoftirqd/311:48:163
81440,0rcu_preempt37-21ksoftirqd/310:43:463
81440,0rcu_preempt37-21ksoftirqd/309:03:373
81440,0rcu_preempt37-21ksoftirqd/307:58:143
81440,0rcu_preempt29-21ksoftirqd/212:03:422
81440,0rcu_preempt21-21ksoftirqd/109:23:261
81440,0rcu_preempt21-21ksoftirqd/107:58:261
11869994428,0cyclictest3-21ksoftirqd/011:38:430
81430,0rcu_preempt37-21ksoftirqd/308:28:293
11869994238,0cyclictest3-21ksoftirqd/008:23:150
11869994237,0cyclictest3-21ksoftirqd/009:38:150
11869994237,0cyclictest3-21ksoftirqd/007:48:310
11869994137,0cyclictest41ktimersoftd/011:58:520
11869994137,0cyclictest3-21ksoftirqd/010:43:400
11869994137,0cyclictest3-21ksoftirqd/007:38:330
11869994136,0cyclictest3-21ksoftirqd/008:53:260
11869994136,0cyclictest3-21ksoftirqd/008:53:260
11869994136,0cyclictest3-21ksoftirqd/008:28:350
11869994127,0cyclictest3-21ksoftirqd/009:03:480
11869994126,0cyclictest3-21ksoftirqd/011:23:360
11869994126,0cyclictest3-21ksoftirqd/011:23:360
11869994036,0cyclictest41ktimersoftd/012:18:270
11869994036,0cyclictest3-21ksoftirqd/010:38:160
11869994036,0cyclictest3-21ksoftirqd/010:28:340
11869994035,0cyclictest3-21ksoftirqd/012:13:410
11869994035,0cyclictest3-21ksoftirqd/010:53:460
11869994035,0cyclictest3-21ksoftirqd/009:13:320
11869994025,0cyclictest3-21ksoftirqd/011:43:160
11869993935,0cyclictest41ktimersoftd/009:08:350
11869993935,0cyclictest3-21ksoftirqd/011:18:270
11869993935,0cyclictest3-21ksoftirqd/010:27:390
11869993935,0cyclictest3-21ksoftirqd/009:48:450
11869993934,0cyclictest3-21ksoftirqd/012:28:160
11869993934,0cyclictest3-21ksoftirqd/010:58:470
11869993934,0cyclictest3-21ksoftirqd/008:48:400
11869993934,0cyclictest3-21ksoftirqd/008:03:140
11869993934,0cyclictest3-21ksoftirqd/008:03:140
11869993925,0cyclictest3-21ksoftirqd/008:33:510
11869993834,0cyclictest3-21ksoftirqd/012:43:290
11869993834,0cyclictest3-21ksoftirqd/011:53:460
11869993834,0cyclictest3-21ksoftirqd/011:48:320
11869993834,0cyclictest3-21ksoftirqd/011:08:440
11869993834,0cyclictest3-21ksoftirqd/010:48:490
11869993834,0cyclictest3-21ksoftirqd/010:08:430
11869993834,0cyclictest3-21ksoftirqd/009:28:440
11869993834,0cyclictest3-21ksoftirqd/008:58:440
11869993834,0cyclictest3-21ksoftirqd/008:43:280
11869993834,0cyclictest3-21ksoftirqd/007:33:290
11869993833,0cyclictest3-21ksoftirqd/007:53:460
11869993831,0cyclictest3-21ksoftirqd/012:33:360
11869993825,0cyclictest3-21ksoftirqd/010:33:410
1599370,0migration/017053-21unixbench_singl10:03:470
11869993733,0cyclictest41ktimersoftd/011:28:320
11869993733,0cyclictest3-21ksoftirqd/011:13:430
11869993733,0cyclictest3-21ksoftirqd/009:54:530
11869993733,0cyclictest3-21ksoftirqd/008:38:300
11869993733,0cyclictest3-21ksoftirqd/008:13:330
11869993733,0cyclictest3-21ksoftirqd/007:58:250
11869993732,0cyclictest3-21ksoftirqd/012:03:160
11869993732,0cyclictest3-21ksoftirqd/011:33:360
11869993732,0cyclictest3-21ksoftirqd/010:18:310
11869993726,0cyclictest3-21ksoftirqd/008:08:410
11869993723,0cyclictest3-21ksoftirqd/012:38:160
83352362,0sleep00-21swapper/008:28:140
11869993632,0cyclictest3-21ksoftirqd/012:28:430
11869993632,0cyclictest3-21ksoftirqd/009:58:280
11869993632,0cyclictest3-21ksoftirqd/007:23:370
11869993631,0cyclictest3-21ksoftirqd/010:15:110
11869993531,0cyclictest3-21ksoftirqd/009:33:420
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional