You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-23 - 19:13

Intel(R) Atom(TM) CPU D2700 @ 2.13GHz, Linux 4.4.39-rt50 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot6.osadl.org (updated Sun Jan 23, 2022 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
91660,0rcu_sched3-21ksoftirqd/008:15:390
91620,0rcu_sched21-21ksoftirqd/108:45:381
24715996124,0cyclictest3-21ksoftirqd/007:41:520
24715996124,0cyclictest3-21ksoftirqd/007:41:520
91590,0rcu_sched3-21ksoftirqd/009:35:380
91590,0rcu_sched29-21ksoftirqd/211:25:382
91590,0rcu_sched29-21ksoftirqd/211:25:382
91590,0rcu_sched21-21ksoftirqd/110:35:381
81590,0rcu_preempt37-21ksoftirqd/307:46:103
81590,0rcu_preempt3-21ksoftirqd/011:51:140
91570,0rcu_sched21-21ksoftirqd/111:55:391
81570,0rcu_preempt37-21ksoftirqd/311:20:383
91560,0rcu_sched37-21ksoftirqd/309:30:383
81560,0rcu_preempt37-21ksoftirqd/311:15:383
81560,0rcu_preempt3-21ksoftirqd/010:26:080
81560,0rcu_preempt3-21ksoftirqd/010:26:080
81560,0rcu_preempt21-21ksoftirqd/112:25:571
91550,0rcu_sched29-21ksoftirqd/211:05:402
81550,0rcu_preempt37-21ksoftirqd/311:45:383
81550,0rcu_preempt3-21ksoftirqd/009:15:390
81550,0rcu_preempt3-21ksoftirqd/007:11:110
81550,0rcu_preempt21-21ksoftirqd/111:05:571
91540,0rcu_sched37-21ksoftirqd/310:40:383
91540,0rcu_sched32361-21munin-run12:35:382
91540,0rcu_sched3-21ksoftirqd/011:15:380
91540,0rcu_sched3-21ksoftirqd/010:25:380
91540,0rcu_sched3-21ksoftirqd/007:25:380
91540,0rcu_sched21047-21systemd-cgroups10:50:381
81540,0rcu_preempt21-21ksoftirqd/112:21:091
81540,0rcu_preempt21-21ksoftirqd/108:16:001
81540,0rcu_preempt21-21ksoftirqd/107:25:551
81540,0rcu_preempt21-21ksoftirqd/107:10:391
24715995420,0cyclictest3-21ksoftirqd/009:06:530
91530,0rcu_sched3-21ksoftirqd/011:05:390
91530,0rcu_sched21-21ksoftirqd/108:15:391
81530,0rcu_preempt37-21ksoftirqd/311:00:393
81530,0rcu_preempt3-21ksoftirqd/010:36:090
81530,0rcu_preempt3-21ksoftirqd/010:11:140
81530,0rcu_preempt3-21ksoftirqd/009:41:000
81530,0rcu_preempt21-21ksoftirqd/110:01:001
81530,0rcu_preempt21-21ksoftirqd/107:46:461
91520,0rcu_sched3-21ksoftirqd/010:20:380
81520,0rcu_preempt37-21ksoftirqd/309:50:393
81520,0rcu_preempt37-21ksoftirqd/307:16:093
81520,0rcu_preempt3-21ksoftirqd/008:41:050
81520,0rcu_preempt27503-21munin-run08:30:381
81520,0rcu_preempt21-21ksoftirqd/111:46:161
81520,0rcu_preempt21-21ksoftirqd/110:56:491
24715995218,0cyclictest28296-21hwlatdetect08:31:020
91510,0rcu_sched3-21ksoftirqd/010:45:390
81510,0rcu_preempt37-21ksoftirqd/312:01:173
81510,0rcu_preempt37-21ksoftirqd/310:50:393
81510,0rcu_preempt3-21ksoftirqd/009:20:580
81510,0rcu_preempt3-21ksoftirqd/008:56:000
81510,0rcu_preempt21-21ksoftirqd/112:05:381
81510,0rcu_preempt21-21ksoftirqd/111:16:141
81510,0rcu_preempt21-21ksoftirqd/111:01:081
81510,0rcu_preempt21-21ksoftirqd/110:50:551
81510,0rcu_preempt21-21ksoftirqd/109:25:381
81510,0rcu_preempt21-21ksoftirqd/109:20:381
81510,0rcu_preempt21-21ksoftirqd/108:46:151
81510,0rcu_preempt21-21ksoftirqd/108:01:001
81510,0rcu_preempt21-21ksoftirqd/107:15:581
91500,0rcu_sched3-21ksoftirqd/011:25:380
91500,0rcu_sched3-21ksoftirqd/011:25:380
81500,0rcu_preempt37-21ksoftirqd/312:35:393
81500,0rcu_preempt37-21ksoftirqd/311:31:043
81500,0rcu_preempt37-21ksoftirqd/311:06:143
81500,0rcu_preempt37-21ksoftirqd/310:01:033
81500,0rcu_preempt37-21ksoftirqd/309:50:583
81500,0rcu_preempt37-21ksoftirqd/309:05:393
81500,0rcu_preempt37-21ksoftirqd/307:11:083
81500,0rcu_preempt3-21ksoftirqd/012:01:000
81500,0rcu_preempt3-21ksoftirqd/011:56:160
81500,0rcu_preempt3-21ksoftirqd/008:16:060
81500,0rcu_preempt21-21ksoftirqd/112:31:141
81500,0rcu_preempt21-21ksoftirqd/112:05:391
81500,0rcu_preempt21-21ksoftirqd/111:40:591
81500,0rcu_preempt21-21ksoftirqd/110:21:011
81500,0rcu_preempt21-21ksoftirqd/109:26:101
81500,0rcu_preempt21-21ksoftirqd/109:11:111
81500,0rcu_preempt21-21ksoftirqd/107:35:561
1599500,0migration/023076-21fschecks_time07:01:000
91490,0rcu_sched37-21ksoftirqd/310:00:393
91490,0rcu_sched3-21ksoftirqd/010:50:380
81490,0rcu_preempt37-21ksoftirqd/311:35:583
81490,0rcu_preempt37-21ksoftirqd/310:20:393
81490,0rcu_preempt37-21ksoftirqd/309:21:153
81490,0rcu_preempt37-21ksoftirqd/307:36:093
81490,0rcu_preempt3-21ksoftirqd/010:35:380
81490,0rcu_preempt3-21ksoftirqd/009:16:120
81490,0rcu_preempt21-21ksoftirqd/112:15:391
81490,0rcu_preempt21-21ksoftirqd/112:13:411
81490,0rcu_preempt21-21ksoftirqd/111:55:591
81490,0rcu_preempt21-21ksoftirqd/110:12:381
81490,0rcu_preempt21-21ksoftirqd/109:55:531
81490,0rcu_preempt21-21ksoftirqd/108:30:581
81490,0rcu_preempt21-21ksoftirqd/108:24:081
81490,0rcu_preempt21-21ksoftirqd/108:10:391
81490,0rcu_preempt21-21ksoftirqd/107:42:231
81490,0rcu_preempt21-21ksoftirqd/107:42:231
81490,0rcu_preempt21-21ksoftirqd/107:20:581
24715994922,0cyclictest3-21ksoftirqd/012:35:030
91480,0rcu_sched21-21ksoftirqd/110:20:381
81480,0rcu_preempt37-21ksoftirqd/312:26:123
81480,0rcu_preempt37-21ksoftirqd/309:36:153
81480,0rcu_preempt37-21ksoftirqd/309:11:073
81480,0rcu_preempt37-21ksoftirqd/307:45:393
81480,0rcu_preempt37-21ksoftirqd/307:45:393
81480,0rcu_preempt37-21ksoftirqd/307:26:163
81480,0rcu_preempt3-21ksoftirqd/012:30:380
81480,0rcu_preempt3-21ksoftirqd/011:08:070
81480,0rcu_preempt3-21ksoftirqd/009:51:090
81480,0rcu_preempt3-21ksoftirqd/009:36:160
81480,0rcu_preempt3-21ksoftirqd/008:50:590
81480,0rcu_preempt3-21ksoftirqd/008:30:380
81480,0rcu_preempt3-21ksoftirqd/007:51:160
81480,0rcu_preempt21-21ksoftirqd/111:21:081
81480,0rcu_preempt21-21ksoftirqd/111:21:081
81480,0rcu_preempt21-21ksoftirqd/109:45:581
81480,0rcu_preempt21-21ksoftirqd/108:50:591
2699480,0migration/223724-21rt-features07:01:092
24724994832,0cyclictest29-21ksoftirqd/209:20:392
2471599483,0cyclictest0-21swapper/012:21:090
24715994824,0cyclictest3-21ksoftirqd/010:56:370
1899480,0migration/124260-21runrttasks07:02:471
91470,0rcu_sched3-21ksoftirqd/011:30:380
81470,0rcu_preempt37-21ksoftirqd/312:10:593
81470,0rcu_preempt37-21ksoftirqd/310:30:573
81470,0rcu_preempt37-21ksoftirqd/309:41:073
81470,0rcu_preempt37-21ksoftirqd/308:45:583
81470,0rcu_preempt37-21ksoftirqd/307:55:593
81470,0rcu_preempt3-21ksoftirqd/012:06:140
81470,0rcu_preempt3-21ksoftirqd/010:06:110
81470,0rcu_preempt3-21ksoftirqd/009:46:070
81470,0rcu_preempt3-21ksoftirqd/008:00:550
81470,0rcu_preempt3-21ksoftirqd/007:56:040
81470,0rcu_preempt3-21ksoftirqd/007:30:560
81470,0rcu_preempt21-21ksoftirqd/111:10:591
24724994742,0cyclictest29341-21netstat12:26:032
24724994742,0cyclictest29-21ksoftirqd/210:06:102
24715994723,0cyclictest3-21ksoftirqd/007:09:460
24715994721,0cyclictest3-21ksoftirqd/008:46:430
24715994720,0cyclictest3-21ksoftirqd/010:53:180
91460,0rcu_sched3-21ksoftirqd/007:20:390
81460,0rcu_preempt37-21ksoftirqd/312:21:103
81460,0rcu_preempt37-21ksoftirqd/312:16:133
81460,0rcu_preempt37-21ksoftirqd/311:56:013
81460,0rcu_preempt37-21ksoftirqd/310:46:023
81460,0rcu_preempt37-21ksoftirqd/310:41:093
81460,0rcu_preempt37-21ksoftirqd/310:05:593
81460,0rcu_preempt37-21ksoftirqd/309:30:593
81460,0rcu_preempt37-21ksoftirqd/308:55:393
81460,0rcu_preempt37-21ksoftirqd/308:26:053
81460,0rcu_preempt37-21ksoftirqd/308:11:503
81460,0rcu_preempt37-21ksoftirqd/308:06:163
81460,0rcu_preempt37-21ksoftirqd/307:20:563
81460,0rcu_preempt3-21ksoftirqd/012:16:050
81460,0rcu_preempt3-21ksoftirqd/011:36:030
81460,0rcu_preempt3-21ksoftirqd/010:01:130
81460,0rcu_preempt3-21ksoftirqd/009:26:110
81460,0rcu_preempt3-21ksoftirqd/008:06:080
81460,0rcu_preempt3-21ksoftirqd/007:50:380
81460,0rcu_preempt21-21ksoftirqd/111:36:131
81460,0rcu_preempt21-21ksoftirqd/111:31:091
81460,0rcu_preempt21-21ksoftirqd/110:40:551
81460,0rcu_preempt21-21ksoftirqd/110:35:561
81460,0rcu_preempt21-21ksoftirqd/109:52:151
81460,0rcu_preempt21-21ksoftirqd/109:41:101
81460,0rcu_preempt21-21ksoftirqd/108:55:561
81460,0rcu_preempt21-21ksoftirqd/107:55:581
81460,0rcu_preempt21-21ksoftirqd/107:51:061
81460,0rcu_preempt21-21ksoftirqd/107:31:221
81460,0rcu_preempt21-21ksoftirqd/107:11:151
24724994640,0cyclictest29-21ksoftirqd/211:41:042
2471599469,0cyclictest0-21swapper/009:55:580
24715994623,0cyclictest3-21ksoftirqd/011:45:350
24715994620,0cyclictest3-21ksoftirqd/007:28:340
81450,0rcu_preempt37-21ksoftirqd/311:50:543
81450,0rcu_preempt37-21ksoftirqd/311:46:163
81450,0rcu_preempt37-21ksoftirqd/311:25:573
81450,0rcu_preempt37-21ksoftirqd/309:16:183
81450,0rcu_preempt37-21ksoftirqd/309:05:403
81450,0rcu_preempt37-21ksoftirqd/309:00:343
81450,0rcu_preempt37-21ksoftirqd/308:40:393
81450,0rcu_preempt37-21ksoftirqd/307:06:023
81450,0rcu_preempt3-21ksoftirqd/008:35:590
81450,0rcu_preempt3-21ksoftirqd/008:35:590
81450,0rcu_preempt3-21ksoftirqd/008:20:400
81450,0rcu_preempt21-21ksoftirqd/110:25:591
81450,0rcu_preempt21-21ksoftirqd/110:25:591
81450,0rcu_preempt21-21ksoftirqd/110:06:151
81450,0rcu_preempt21-21ksoftirqd/108:37:431
81450,0rcu_preempt21-21ksoftirqd/108:37:431
24724994539,0cyclictest29-21ksoftirqd/212:10:592
81440,0rcu_preempt37-21ksoftirqd/312:06:143
81440,0rcu_preempt37-21ksoftirqd/311:21:023
81440,0rcu_preempt37-21ksoftirqd/311:21:023
81440,0rcu_preempt37-21ksoftirqd/311:01:083
81440,0rcu_preempt37-21ksoftirqd/310:25:543
81440,0rcu_preempt37-21ksoftirqd/310:25:543
81440,0rcu_preempt37-21ksoftirqd/310:15:563
81440,0rcu_preempt37-21ksoftirqd/308:36:083
81440,0rcu_preempt37-21ksoftirqd/308:36:083
81440,0rcu_preempt37-21ksoftirqd/308:16:173
81440,0rcu_preempt37-21ksoftirqd/308:05:393
81440,0rcu_preempt37-21ksoftirqd/307:51:153
81440,0rcu_preempt37-21ksoftirqd/307:01:003
81440,0rcu_preempt3-21ksoftirqd/012:10:590
81440,0rcu_preempt3-21ksoftirqd/011:45:540
81440,0rcu_preempt3-21ksoftirqd/011:30:580
81440,0rcu_preempt21-21ksoftirqd/111:26:051
81440,0rcu_preempt21-21ksoftirqd/109:36:161
81440,0rcu_preempt21-21ksoftirqd/109:05:571
24724994439,0cyclictest29-21ksoftirqd/209:05:582
24724994439,0cyclictest281ktimersoftd/209:41:102
24724994438,0cyclictest29-21ksoftirqd/210:51:022
24724994438,0cyclictest281ktimersoftd/211:25:582
24724994438,0cyclictest281ktimersoftd/208:21:112
81430,0rcu_preempt37-21ksoftirqd/308:21:153
81430,0rcu_preempt37-21ksoftirqd/307:30:523
81430,0rcu_preempt3-21ksoftirqd/009:01:000
81430,0rcu_preempt3-21ksoftirqd/007:35:500
81430,0rcu_preempt21-21ksoftirqd/109:30:591
81430,0rcu_preempt21-21ksoftirqd/109:00:391
24724994339,0cyclictest281ktimersoftd/211:11:172
24724994338,0cyclictest281ktimersoftd/210:20:582
24724994338,0cyclictest281ktimersoftd/209:26:002
24724994337,0cyclictest281ktimersoftd/212:16:152
2472499431,0cyclictest281ktimersoftd/207:30:382
81420,0rcu_preempt37-21ksoftirqd/308:31:023
81420,0rcu_preempt3-21ksoftirqd/011:17:160
24724994237,0cyclictest29-21ksoftirqd/212:00:572
24724994237,0cyclictest29-21ksoftirqd/209:45:592
24724994236,0cyclictest281ktimersoftd/210:10:582
81410,0rcu_preempt37-21ksoftirqd/310:11:183
24724994136,0cyclictest29-21ksoftirqd/211:16:152
24724994136,0cyclictest29-21ksoftirqd/210:26:082
24724994136,0cyclictest29-21ksoftirqd/210:26:082
24724994136,0cyclictest29-21ksoftirqd/208:51:122
24724994136,0cyclictest29-21ksoftirqd/207:21:142
24724994135,0cyclictest29-21ksoftirqd/211:52:072
24724994135,0cyclictest29-21ksoftirqd/208:56:152
24724994135,0cyclictest29-21ksoftirqd/208:25:592
24724994135,0cyclictest281ktimersoftd/208:41:022
24724994036,0cyclictest29-21ksoftirqd/209:15:552
24724994036,0cyclictest29-21ksoftirqd/208:31:192
24724994035,0cyclictest29-21ksoftirqd/211:46:032
24724994035,0cyclictest29-21ksoftirqd/211:30:532
24724994035,0cyclictest29-21ksoftirqd/209:11:142
24724994035,0cyclictest29-21ksoftirqd/209:00:552
24724994035,0cyclictest29-21ksoftirqd/208:16:002
24724994035,0cyclictest29-21ksoftirqd/208:06:152
24724994034,0cyclictest29-21ksoftirqd/212:10:382
24724994034,0cyclictest29-21ksoftirqd/210:01:102
24724994034,0cyclictest29-21ksoftirqd/208:35:552
24724994034,0cyclictest29-21ksoftirqd/208:35:552
24724994034,0cyclictest29-21ksoftirqd/207:40:392
24724994034,0cyclictest281ktimersoftd/208:01:092
24724994034,0cyclictest281ktimersoftd/207:51:042
24724994034,0cyclictest281ktimersoftd/207:46:002
24724994027,0cyclictest29-21ksoftirqd/210:30:582
24724993935,0cyclictest29-21ksoftirqd/211:55:392
24724993935,0cyclictest29-21ksoftirqd/209:35:572
24724993935,0cyclictest29-21ksoftirqd/207:30:552
24724993935,0cyclictest281ktimersoftd/207:05:542
24724993934,0cyclictest29-21ksoftirqd/209:50:542
24724993934,0cyclictest29-21ksoftirqd/207:16:002
24724993933,0cyclictest29-21ksoftirqd/210:45:582
24724993933,0cyclictest29-21ksoftirqd/207:56:022
24724993933,0cyclictest29-21ksoftirqd/207:40:542
24724993933,0cyclictest29-21ksoftirqd/207:40:542
24724993933,0cyclictest29-21ksoftirqd/207:11:112
24724993834,0cyclictest29-21ksoftirqd/210:56:112
24724993834,0cyclictest29-21ksoftirqd/208:11:042
24724993833,0cyclictest29-21ksoftirqd/211:36:002
24724993833,0cyclictest29-21ksoftirqd/209:56:102
24724993832,0cyclictest29-21ksoftirqd/208:45:562
24724993826,0cyclictest29-21ksoftirqd/210:16:142
24724993824,0cyclictest29-21ksoftirqd/210:41:142
24724993732,0cyclictest29-21ksoftirqd/212:20:582
24724993732,0cyclictest29-21ksoftirqd/209:30:552
24724993732,0cyclictest281ktimersoftd/210:35:552
24724993629,0cyclictest29-21ksoftirqd/211:01:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional