You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-08-04 - 21:24

Intel(R) Atom(TM) CPU D2700 @ 2.13GHz, Linux 4.4.39-rt50 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot6.osadl.org (updated Wed Aug 04, 2021 12:43:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
81730,0rcu_preempt6576-21ls07:14:203
81640,0rcu_preempt37-21ksoftirqd/311:19:193
91630,0rcu_sched37-21ksoftirqd/311:39:013
91630,0rcu_sched29-21ksoftirqd/211:39:012
91630,0rcu_sched29-21ksoftirqd/209:54:002
91620,0rcu_sched37-21ksoftirqd/307:49:003
91620,0rcu_sched29-21ksoftirqd/211:19:022
81620,0rcu_preempt37-21ksoftirqd/307:34:343
91610,0rcu_sched37-21ksoftirqd/309:34:013
81600,0rcu_preempt29-21ksoftirqd/212:19:382
91590,0rcu_sched37-21ksoftirqd/310:24:013
91590,0rcu_sched29-21ksoftirqd/207:59:012
81590,0rcu_preempt37-21ksoftirqd/307:19:313
81590,0rcu_preempt29-21ksoftirqd/209:24:012
81590,0rcu_preempt29-21ksoftirqd/209:14:262
91580,0rcu_sched37-21ksoftirqd/310:44:013
81580,0rcu_preempt3-21ksoftirqd/007:26:220
81580,0rcu_preempt3-21ksoftirqd/007:26:220
81570,0rcu_preempt3-21ksoftirqd/010:34:300
81570,0rcu_preempt29-21ksoftirqd/208:19:292
91560,0rcu_sched37-21ksoftirqd/310:14:013
81560,0rcu_preempt37-21ksoftirqd/311:15:593
81560,0rcu_preempt3-21ksoftirqd/008:57:440
81560,0rcu_preempt29-21ksoftirqd/212:24:412
81560,0rcu_preempt29-21ksoftirqd/210:24:172
81560,0rcu_preempt29-21ksoftirqd/208:58:032
81550,0rcu_preempt29-21ksoftirqd/212:29:372
81550,0rcu_preempt29-21ksoftirqd/209:10:572
81550,0rcu_preempt29-21ksoftirqd/209:10:572
81550,0rcu_preempt29-21ksoftirqd/207:36:242
81550,0rcu_preempt29-21ksoftirqd/207:29:142
7824995526,0cyclictest5890-21gltestperf09:49:181
91540,0rcu_sched29-21ksoftirqd/212:14:022
81540,0rcu_preempt37-21ksoftirqd/311:49:323
81540,0rcu_preempt3-21ksoftirqd/007:59:370
81540,0rcu_preempt29-21ksoftirqd/208:29:352
7824995420,0cyclictest27791-21grep09:24:181
91530,0rcu_sched37-21ksoftirqd/308:09:003
91530,0rcu_sched29-21ksoftirqd/208:44:012
81530,0rcu_preempt37-21ksoftirqd/312:04:363
81530,0rcu_preempt37-21ksoftirqd/310:44:413
81530,0rcu_preempt37-21ksoftirqd/310:39:013
81530,0rcu_preempt37-21ksoftirqd/310:24:383
81530,0rcu_preempt37-21ksoftirqd/308:24:243
81530,0rcu_preempt37-21ksoftirqd/307:54:313
81530,0rcu_preempt37-21ksoftirqd/307:24:323
81530,0rcu_preempt37-21ksoftirqd/307:24:323
81530,0rcu_preempt29-21ksoftirqd/210:19:212
81530,0rcu_preempt29-21ksoftirqd/208:34:112
81530,0rcu_preempt29-21ksoftirqd/207:49:182
782499532,0cyclictest19828-21missed_timers09:04:241
91520,0rcu_sched29-21ksoftirqd/210:29:022
81520,0rcu_preempt37-21ksoftirqd/311:39:263
81520,0rcu_preempt37-21ksoftirqd/308:29:163
81520,0rcu_preempt37-21ksoftirqd/308:24:003
81520,0rcu_preempt3-21ksoftirqd/011:21:060
81520,0rcu_preempt3-21ksoftirqd/009:37:080
81520,0rcu_preempt3-21ksoftirqd/009:29:000
81520,0rcu_preempt3-21ksoftirqd/007:19:000
81520,0rcu_preempt29-21ksoftirqd/211:59:322
81520,0rcu_preempt29-21ksoftirqd/207:39:382
782499523,0cyclictest27458-21taskset12:03:301
7824995221,0cyclictest7646-21crond12:34:011
81510,0rcu_preempt37-21ksoftirqd/309:14:013
81510,0rcu_preempt37-21ksoftirqd/309:14:013
81510,0rcu_preempt37-21ksoftirqd/308:47:303
81510,0rcu_preempt3-21ksoftirqd/011:46:230
81510,0rcu_preempt3-21ksoftirqd/010:44:510
81510,0rcu_preempt3-21ksoftirqd/010:29:150
81510,0rcu_preempt3-21ksoftirqd/009:59:230
81510,0rcu_preempt3-21ksoftirqd/009:59:230
81510,0rcu_preempt3-21ksoftirqd/008:06:130
81510,0rcu_preempt3-21ksoftirqd/007:37:400
81510,0rcu_preempt29-21ksoftirqd/212:44:272
81510,0rcu_preempt29-21ksoftirqd/211:19:362
81510,0rcu_preempt29-21ksoftirqd/209:39:012
81510,0rcu_preempt29-21ksoftirqd/208:49:172
81510,0rcu_preempt21-21ksoftirqd/109:34:261
782499514,0cyclictest3355-21gltestperf11:04:191
782499513,0cyclictest29057-21gltestperf08:09:191
7824995123,0cyclictest21-21ksoftirqd/110:35:501
782499512,0cyclictest9523-21munin-node09:59:221
782499512,0cyclictest9523-21munin-node09:59:221
81500,0rcu_preempt37-21ksoftirqd/311:44:183
81500,0rcu_preempt37-21ksoftirqd/311:01:463
81500,0rcu_preempt37-21ksoftirqd/310:14:343
81500,0rcu_preempt37-21ksoftirqd/309:58:593
81500,0rcu_preempt3-21ksoftirqd/012:36:570
81500,0rcu_preempt3-21ksoftirqd/010:49:380
81500,0rcu_preempt3-21ksoftirqd/010:49:380
81500,0rcu_preempt3-21ksoftirqd/009:18:460
81500,0rcu_preempt3-21ksoftirqd/009:05:360
81500,0rcu_preempt3-21ksoftirqd/008:49:510
81500,0rcu_preempt3-21ksoftirqd/007:49:360
81500,0rcu_preempt3-21ksoftirqd/007:14:290
81500,0rcu_preempt29-21ksoftirqd/211:09:332
81500,0rcu_preempt29-21ksoftirqd/210:44:582
81500,0rcu_preempt29-21ksoftirqd/209:55:532
81500,0rcu_preempt29-21ksoftirqd/208:59:192
81500,0rcu_preempt21-21ksoftirqd/108:23:251
782499502,0cyclictest4265-21gltestperf12:24:241
7818995024,0cyclictest9292-21systemd-cgroups09:59:010
7818995021,0cyclictest5584-21sadc12:29:010
7818995012,0cyclictest5922-21seq09:49:180
1899500,0migration/17398-21runrttasks07:16:351
81490,0rcu_preempt37-21ksoftirqd/312:24:223
81490,0rcu_preempt37-21ksoftirqd/311:54:023
81490,0rcu_preempt37-21ksoftirqd/307:49:303
81490,0rcu_preempt37-21ksoftirqd/307:29:013
81490,0rcu_preempt3-21ksoftirqd/012:43:240
81490,0rcu_preempt3-21ksoftirqd/011:15:020
81490,0rcu_preempt3-21ksoftirqd/010:25:250
81490,0rcu_preempt3-21ksoftirqd/010:23:150
81490,0rcu_preempt3-21ksoftirqd/010:08:200
81490,0rcu_preempt3-21ksoftirqd/008:33:500
81490,0rcu_preempt3-21ksoftirqd/008:20:000
81490,0rcu_preempt29-21ksoftirqd/212:44:022
81490,0rcu_preempt29-21ksoftirqd/212:34:342
81490,0rcu_preempt29-21ksoftirqd/211:54:392
81490,0rcu_preempt29-21ksoftirqd/210:54:242
81490,0rcu_preempt29-21ksoftirqd/210:49:502
81490,0rcu_preempt29-21ksoftirqd/210:49:502
81490,0rcu_preempt29-21ksoftirqd/210:04:542
81490,0rcu_preempt29-21ksoftirqd/209:06:232
81490,0rcu_preempt29-21ksoftirqd/207:19:002
81490,0rcu_preempt21-21ksoftirqd/108:34:211
782499493,0cyclictest3026-21sed12:19:441
782499493,0cyclictest27858-21grep08:04:311
782499493,0cyclictest26307-21runrttasks10:39:191
782499493,0cyclictest25549-21grep09:19:181
782499493,0cyclictest22205-21fschecks_count10:29:171
782499493,0cyclictest21290-21unixbench_multi10:24:541
782499492,0cyclictest18823-21proc_pri10:19:371
7824994915,0cyclictest12520-21gltestperf12:44:211
7818994937,0cyclictest25394-21tr09:19:150
81480,0rcu_preempt37-21ksoftirqd/312:19:343
81480,0rcu_preempt37-21ksoftirqd/310:53:233
81480,0rcu_preempt37-21ksoftirqd/310:53:233
81480,0rcu_preempt37-21ksoftirqd/309:44:193
81480,0rcu_preempt37-21ksoftirqd/309:34:303
81480,0rcu_preempt37-21ksoftirqd/308:54:023
81480,0rcu_preempt37-21ksoftirqd/308:34:153
81480,0rcu_preempt3-21ksoftirqd/012:33:060
81480,0rcu_preempt3-21ksoftirqd/011:25:390
81480,0rcu_preempt3-21ksoftirqd/011:03:480
81480,0rcu_preempt3-21ksoftirqd/009:47:450
81480,0rcu_preempt3-21ksoftirqd/008:48:430
81480,0rcu_preempt29-21ksoftirqd/211:53:562
81480,0rcu_preempt29-21ksoftirqd/211:44:352
81480,0rcu_preempt29-21ksoftirqd/211:39:202
81480,0rcu_preempt29-21ksoftirqd/210:59:482
81480,0rcu_preempt29-21ksoftirqd/210:09:222
81480,0rcu_preempt29-21ksoftirqd/209:24:172
81480,0rcu_preempt29-21ksoftirqd/208:44:332
81480,0rcu_preempt29-21ksoftirqd/208:09:182
81480,0rcu_preempt29-21ksoftirqd/208:04:312
81480,0rcu_preempt29-21ksoftirqd/207:44:302
81480,0rcu_preempt29-21ksoftirqd/207:24:202
81480,0rcu_preempt29-21ksoftirqd/207:24:202
81480,0rcu_preempt21-21ksoftirqd/112:05:371
782499484,0cyclictest2477-21cat09:39:341
782499484,0cyclictest24479-21needreboot11:54:311
782499483,0cyclictest6194-21runrttasks08:31:491
782499483,0cyclictest4695-21unixbench_multi09:44:321
782499483,0cyclictest2625-21ls08:24:151
782499483,0cyclictest21967-21runrttasks07:52:361
782499483,0cyclictest21171-21cpuspeed_turbos09:09:151
782499483,0cyclictest21171-21cpuspeed_turbos09:09:151
782499483,0cyclictest16161-21fschecks_time10:14:181
782499483,0cyclictest16107-21taskset07:39:071
782499483,0cyclictest12285-21runrttasks08:45:191
782499483,0cyclictest0-21swapper/112:14:231
782499483,0cyclictest0-21swapper/107:44:361
782499482,0cyclictest8306-21gltestperf07:19:161
782499482,0cyclictest8148-21sendmail_mailqu11:14:361
782499482,0cyclictest654-21munin-node10:59:281
782499482,0cyclictest27402-21grep10:44:201
782499482,0cyclictest11693-21cpuspeed_turbos10:04:121
782499482,0cyclictest10365-21timerandwakeup11:19:381
782499482,0cyclictest10349-21fschecks_time07:24:191
782499482,0cyclictest10349-21fschecks_time07:24:191
782499482,0cyclictest10003-21cpu12:39:161
782499481,0cyclictest4972-21munin-node11:09:251
782499481,0cyclictest23179-21munin-node09:14:241
782499481,0cyclictest15326-21munin-node11:34:421
81470,0rcu_preempt37-21ksoftirqd/311:29:193
81470,0rcu_preempt37-21ksoftirqd/310:04:233
81470,0rcu_preempt37-21ksoftirqd/309:59:153
81470,0rcu_preempt37-21ksoftirqd/309:59:153
81470,0rcu_preempt37-21ksoftirqd/309:49:323
81470,0rcu_preempt37-21ksoftirqd/309:39:203
81470,0rcu_preempt37-21ksoftirqd/308:49:303
81470,0rcu_preempt37-21ksoftirqd/307:59:313
81470,0rcu_preempt3-21ksoftirqd/012:23:580
81470,0rcu_preempt3-21ksoftirqd/012:11:360
81470,0rcu_preempt3-21ksoftirqd/008:38:500
81470,0rcu_preempt3-21ksoftirqd/007:47:230
81470,0rcu_preempt29-21ksoftirqd/212:04:022
81470,0rcu_preempt29-21ksoftirqd/211:04:182
81470,0rcu_preempt29-21ksoftirqd/210:14:282
81470,0rcu_preempt29-21ksoftirqd/209:59:162
81470,0rcu_preempt29-21ksoftirqd/209:59:162
81470,0rcu_preempt29-21ksoftirqd/207:59:162
81470,0rcu_preempt29-21ksoftirqd/207:14:142
81470,0rcu_preempt21-21ksoftirqd/109:29:321
782499473,0cyclictest8209-21fschecks_count12:34:161
782499473,0cyclictest30363-21sed10:49:561
782499473,0cyclictest30363-21sed10:49:561
782499473,0cyclictest24659-21grep07:59:161
782499473,0cyclictest14957-21cpuspeed_turbos08:54:111
782499473,0cyclictest13756-21cpuspeed_turbos10:09:151
782499473,0cyclictest11306-21cpuspeed_turbos11:24:161
782499473,0cyclictest0-21swapper/107:54:301
782499472,0cyclictest7872-21tune2fs09:54:171
782499472,0cyclictest30762-21cpuspeed_turbos08:14:151
782499472,0cyclictest30762-21cpuspeed_turbos08:14:151
782499472,0cyclictest30514-21kernelversion12:09:301
782499472,0cyclictest22359-21missed_timers11:49:271
782499472,0cyclictest17466-21gltestperf08:59:151
782499472,0cyclictest14154-21cpuspeed_turbos07:34:161
782499472,0cyclictest14057-21missed_timers11:29:221
782499472,0cyclictest13861-21runrttasks08:49:271
782499472,0cyclictest12309-21fschecks_count07:29:151
782499471,0cyclictest8678-21munin-node08:39:171
81460,0rcu_preempt37-21ksoftirqd/312:42:533
81460,0rcu_preempt37-21ksoftirqd/312:34:173
81460,0rcu_preempt37-21ksoftirqd/312:29:183
81460,0rcu_preempt37-21ksoftirqd/312:09:223
81460,0rcu_preempt37-21ksoftirqd/311:59:333
81460,0rcu_preempt37-21ksoftirqd/311:24:023
81460,0rcu_preempt37-21ksoftirqd/311:14:023
81460,0rcu_preempt37-21ksoftirqd/311:04:153
81460,0rcu_preempt37-21ksoftirqd/310:54:373
81460,0rcu_preempt37-21ksoftirqd/309:24:173
81460,0rcu_preempt37-21ksoftirqd/309:19:333
81460,0rcu_preempt37-21ksoftirqd/308:14:293
81460,0rcu_preempt37-21ksoftirqd/308:14:293
81460,0rcu_preempt37-21ksoftirqd/307:39:373
81460,0rcu_preempt3-21ksoftirqd/012:08:060
81460,0rcu_preempt3-21ksoftirqd/011:38:110
81460,0rcu_preempt3-21ksoftirqd/008:59:080
81460,0rcu_preempt3-21ksoftirqd/008:09:300
81460,0rcu_preempt3-21ksoftirqd/007:39:430
81460,0rcu_preempt29-21ksoftirqd/212:14:302
81460,0rcu_preempt29-21ksoftirqd/211:24:162
81460,0rcu_preempt29-21ksoftirqd/210:34:022
81460,0rcu_preempt29-21ksoftirqd/209:49:012
81460,0rcu_preempt29-21ksoftirqd/209:29:172
81460,0rcu_preempt29-21ksoftirqd/208:24:152
782499464,0cyclictest19587-21grep11:44:191
782499463,0cyclictest31169-21cpuspeed_turbos10:54:181
782499462,0cyclictest17395-21taskset11:39:131
81450,0rcu_preempt37-21ksoftirqd/312:14:033
81450,0rcu_preempt37-21ksoftirqd/310:29:323
81450,0rcu_preempt37-21ksoftirqd/309:04:193
81450,0rcu_preempt37-21ksoftirqd/308:39:213
81450,0rcu_preempt37-21ksoftirqd/308:14:003
81450,0rcu_preempt29-21ksoftirqd/211:29:162
81450,0rcu_preempt29-21ksoftirqd/210:39:162
81450,0rcu_preempt29-21ksoftirqd/209:39:362
81450,0rcu_preempt29-21ksoftirqd/208:15:072
81450,0rcu_preempt29-21ksoftirqd/208:15:072
7818994521,0cyclictest0-21swapper/008:29:010
7818994519,0cyclictest24165-21munin-plugin-st07:59:010
81440,0rcu_preempt37-21ksoftirqd/312:44:213
81440,0rcu_preempt37-21ksoftirqd/309:14:343
81440,0rcu_preempt37-21ksoftirqd/308:59:013
81440,0rcu_preempt3-21ksoftirqd/012:44:160
81440,0rcu_preempt3-21ksoftirqd/011:55:400
81440,0rcu_preempt3-21ksoftirqd/011:11:010
81440,0rcu_preempt3-21ksoftirqd/010:56:290
7818994430,0cyclictest1284-21runrttasks12:03:300
7818994418,0cyclictest0-21swapper/008:42:120
81430,0rcu_preempt3-21ksoftirqd/011:49:340
81430,0rcu_preempt3-21ksoftirqd/009:29:170
81430,0rcu_preempt3-21ksoftirqd/009:13:120
81430,0rcu_preempt3-21ksoftirqd/009:13:120
81430,0rcu_preempt3-21ksoftirqd/007:29:160
7818994326,0cyclictest910-21grep12:14:460
7818994325,0cyclictest3016-21grep11:04:140
7818994316,0cyclictest17904-21cut11:39:200
7818994316,0cyclictest14354-21cut10:09:220
81420,0rcu_preempt3-21ksoftirqd/010:14:240
7818994225,0cyclictest2293-21taskset09:39:300
7818994220,0cyclictest26053-21latency_hist10:39:010
7818994218,0cyclictest30968-21fschecks_time08:14:180
7818994218,0cyclictest30968-21fschecks_time08:14:180
7818994126,0cyclictest14175-21munin-node11:29:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional