You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-27 - 05:08

x86 Intel Atom D2700 @2130 MHz, Linux 4.4.39-rt50 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6.osadl.org (updated Mon Jun 27, 2022 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
81910,0rcu_preempt25112-21munin-run22:42:550
81670,0rcu_preempt18060-21latency_hist21:07:550
81620,0rcu_preempt29-21ksoftirqd/219:08:222
91610,0rcu_sched3-21ksoftirqd/019:57:550
91610,0rcu_sched29-21ksoftirqd/221:07:552
91580,0rcu_sched3-21ksoftirqd/019:22:540
81580,0rcu_preempt29-21ksoftirqd/200:23:272
81580,0rcu_preempt21-21ksoftirqd/119:28:231
81580,0rcu_preempt21-21ksoftirqd/119:28:231
91570,0rcu_sched21-21ksoftirqd/123:32:541
81570,0rcu_preempt37-21ksoftirqd/322:58:093
81570,0rcu_preempt3-21ksoftirqd/019:03:200
91560,0rcu_sched3-21ksoftirqd/020:17:540
81560,0rcu_preempt37-21ksoftirqd/323:38:183
81560,0rcu_preempt29-21ksoftirqd/223:43:212
81560,0rcu_preempt21-21ksoftirqd/123:18:241
81560,0rcu_preempt21-21ksoftirqd/123:18:241
81560,0rcu_preempt21-21ksoftirqd/120:03:121
91550,0rcu_sched21-21ksoftirqd/119:27:541
81550,0rcu_preempt37-21ksoftirqd/322:23:093
81550,0rcu_preempt3-21ksoftirqd/023:18:160
81550,0rcu_preempt3-21ksoftirqd/023:18:160
81550,0rcu_preempt21-21ksoftirqd/123:48:181
81550,0rcu_preempt21-21ksoftirqd/123:03:201
81550,0rcu_preempt21-21ksoftirqd/121:43:091
703995525,0cyclictest10247-21cut20:48:083
81540,0rcu_preempt21-21ksoftirqd/122:58:121
81540,0rcu_preempt21-21ksoftirqd/122:42:551
81540,0rcu_preempt21-21ksoftirqd/119:43:191
81540,0rcu_preempt21-21ksoftirqd/100:18:251
81540,0rcu_preempt21-21ksoftirqd/100:13:231
81540,0rcu_preempt21-21ksoftirqd/100:13:231
91530,0rcu_sched29-21ksoftirqd/219:07:552
81530,0rcu_preempt37-21ksoftirqd/321:48:063
81530,0rcu_preempt37-21ksoftirqd/319:28:123
81530,0rcu_preempt37-21ksoftirqd/319:28:123
81530,0rcu_preempt37-21ksoftirqd/319:03:083
81530,0rcu_preempt29-21ksoftirqd/223:13:062
81530,0rcu_preempt21-21ksoftirqd/123:13:141
81530,0rcu_preempt21-21ksoftirqd/122:23:261
81530,0rcu_preempt21-21ksoftirqd/122:18:231
81530,0rcu_preempt21-21ksoftirqd/100:28:081
81530,0rcu_preempt21-21ksoftirqd/100:08:051
81520,0rcu_preempt29-21ksoftirqd/223:42:552
81520,0rcu_preempt29-21ksoftirqd/219:13:112
81520,0rcu_preempt21-21ksoftirqd/122:48:251
81520,0rcu_preempt21-21ksoftirqd/121:28:221
81520,0rcu_preempt21-21ksoftirqd/121:13:091
81520,0rcu_preempt21-21ksoftirqd/120:33:201
703995226,0cyclictest37-21ksoftirqd/323:47:213
81510,0rcu_preempt37-21ksoftirqd/323:37:553
81510,0rcu_preempt37-21ksoftirqd/322:18:183
81510,0rcu_preempt37-21ksoftirqd/321:28:273
81510,0rcu_preempt37-21ksoftirqd/321:23:123
81510,0rcu_preempt37-21ksoftirqd/321:23:123
81510,0rcu_preempt37-21ksoftirqd/320:13:123
81510,0rcu_preempt37-21ksoftirqd/319:23:063
81510,0rcu_preempt29-21ksoftirqd/223:12:552
81510,0rcu_preempt29-21ksoftirqd/221:08:172
81510,0rcu_preempt21-21ksoftirqd/123:53:081
81510,0rcu_preempt21-21ksoftirqd/123:12:541
81510,0rcu_preempt21-21ksoftirqd/122:53:121
81510,0rcu_preempt21-21ksoftirqd/122:08:091
81510,0rcu_preempt21-21ksoftirqd/121:23:311
81510,0rcu_preempt21-21ksoftirqd/121:23:311
81510,0rcu_preempt21-21ksoftirqd/119:38:121
91500,0rcu_sched3-21ksoftirqd/000:27:540
81500,0rcu_preempt37-21ksoftirqd/319:38:183
81500,0rcu_preempt29-21ksoftirqd/222:18:172
81500,0rcu_preempt29-21ksoftirqd/221:33:082
81500,0rcu_preempt29-21ksoftirqd/200:18:122
81500,0rcu_preempt21-21ksoftirqd/122:37:551
81500,0rcu_preempt21-21ksoftirqd/121:48:161
81500,0rcu_preempt21-21ksoftirqd/120:58:101
81500,0rcu_preempt21-21ksoftirqd/120:13:081
81500,0rcu_preempt21-21ksoftirqd/119:48:091
81500,0rcu_preempt21-21ksoftirqd/119:33:091
81500,0rcu_preempt21-21ksoftirqd/119:07:551
81500,0rcu_preempt21-21ksoftirqd/100:23:141
693995037,0cyclictest32538-21cpuspeed_turbos00:18:050
81490,0rcu_preempt37-21ksoftirqd/300:07:543
81490,0rcu_preempt29-21ksoftirqd/221:52:552
81490,0rcu_preempt29-21ksoftirqd/221:48:172
81490,0rcu_preempt29-21ksoftirqd/220:38:222
81490,0rcu_preempt29-21ksoftirqd/219:58:042
81490,0rcu_preempt21-21ksoftirqd/123:58:051
81490,0rcu_preempt21-21ksoftirqd/121:08:101
81490,0rcu_preempt21-21ksoftirqd/120:53:061
81490,0rcu_preempt21-21ksoftirqd/119:53:091
81490,0rcu_preempt21-21ksoftirqd/100:03:201
693994934,0cyclictest0-21swapper/019:07:560
91480,0rcu_sched37-21ksoftirqd/300:37:553
81480,0rcu_preempt37-21ksoftirqd/321:38:193
81480,0rcu_preempt37-21ksoftirqd/321:33:053
81480,0rcu_preempt37-21ksoftirqd/321:13:053
81480,0rcu_preempt37-21ksoftirqd/320:08:093
81480,0rcu_preempt29-21ksoftirqd/223:28:092
81480,0rcu_preempt29-21ksoftirqd/200:09:512
81480,0rcu_preempt21-21ksoftirqd/123:38:091
81480,0rcu_preempt21-21ksoftirqd/122:47:551
81480,0rcu_preempt21-21ksoftirqd/122:28:061
81480,0rcu_preempt21-21ksoftirqd/122:02:551
81480,0rcu_preempt21-21ksoftirqd/121:18:081
81480,0rcu_preempt21-21ksoftirqd/120:47:551
81480,0rcu_preempt21-21ksoftirqd/120:28:101
81480,0rcu_preempt21-21ksoftirqd/120:28:101
81480,0rcu_preempt21-21ksoftirqd/120:08:081
81480,0rcu_preempt21-21ksoftirqd/119:58:051
81480,0rcu_preempt21-21ksoftirqd/119:13:061
81480,0rcu_preempt21-21ksoftirqd/100:33:241
91470,0rcu_sched29-21ksoftirqd/223:17:552
91470,0rcu_sched29-21ksoftirqd/223:17:552
81470,0rcu_preempt37-21ksoftirqd/323:48:253
81470,0rcu_preempt29-21ksoftirqd/223:27:542
81470,0rcu_preempt29-21ksoftirqd/222:58:092
81470,0rcu_preempt29-21ksoftirqd/222:37:502
81470,0rcu_preempt29-21ksoftirqd/222:28:062
81470,0rcu_preempt29-21ksoftirqd/221:38:262
81470,0rcu_preempt29-21ksoftirqd/220:12:552
81470,0rcu_preempt29-21ksoftirqd/219:38:072
81470,0rcu_preempt21-21ksoftirqd/123:43:081
81470,0rcu_preempt21-21ksoftirqd/123:23:141
81470,0rcu_preempt21-21ksoftirqd/122:03:211
81470,0rcu_preempt21-21ksoftirqd/121:38:061
81470,0rcu_preempt21-21ksoftirqd/120:43:171
81470,0rcu_preempt21-21ksoftirqd/120:38:351
81470,0rcu_preempt21-21ksoftirqd/120:18:111
81470,0rcu_preempt21-21ksoftirqd/119:08:171
81460,0rcu_preempt37-21ksoftirqd/322:48:253
81460,0rcu_preempt37-21ksoftirqd/319:07:563
81460,0rcu_preempt37-21ksoftirqd/300:27:553
81460,0rcu_preempt29-21ksoftirqd/222:37:552
81460,0rcu_preempt29-21ksoftirqd/222:03:132
81460,0rcu_preempt29-21ksoftirqd/221:58:102
81460,0rcu_preempt29-21ksoftirqd/221:22:542
81460,0rcu_preempt29-21ksoftirqd/220:28:082
81460,0rcu_preempt29-21ksoftirqd/220:28:082
81460,0rcu_preempt29-21ksoftirqd/220:18:152
81460,0rcu_preempt29-21ksoftirqd/200:33:052
81460,0rcu_preempt29-21ksoftirqd/200:13:072
81460,0rcu_preempt29-21ksoftirqd/200:13:072
81460,0rcu_preempt21-21ksoftirqd/121:33:251
81460,0rcu_preempt21-21ksoftirqd/121:03:191
81460,0rcu_preempt21-21ksoftirqd/119:18:091
693994633,0cyclictest0-21swapper/023:41:590
81450,0rcu_preempt37-21ksoftirqd/323:28:183
81450,0rcu_preempt37-21ksoftirqd/323:18:053
81450,0rcu_preempt37-21ksoftirqd/323:18:053
81450,0rcu_preempt37-21ksoftirqd/323:12:553
81450,0rcu_preempt37-21ksoftirqd/322:47:213
81450,0rcu_preempt37-21ksoftirqd/322:13:323
81450,0rcu_preempt37-21ksoftirqd/321:43:083
81450,0rcu_preempt37-21ksoftirqd/320:18:143
81450,0rcu_preempt37-21ksoftirqd/319:54:213
81450,0rcu_preempt37-21ksoftirqd/319:19:473
81450,0rcu_preempt37-21ksoftirqd/319:17:153
81450,0rcu_preempt29-21ksoftirqd/223:48:052
81450,0rcu_preempt29-21ksoftirqd/221:22:552
81450,0rcu_preempt29-21ksoftirqd/221:22:552
81450,0rcu_preempt29-21ksoftirqd/220:48:132
81450,0rcu_preempt29-21ksoftirqd/220:23:262
81450,0rcu_preempt29-21ksoftirqd/219:48:182
81450,0rcu_preempt29-21ksoftirqd/219:43:092
81450,0rcu_preempt29-21ksoftirqd/219:32:552
81450,0rcu_preempt21-21ksoftirqd/123:33:171
81450,0rcu_preempt21-21ksoftirqd/121:53:091
81450,0rcu_preempt21-21ksoftirqd/120:23:161
702994539,0cyclictest443-21cpuspeed_turbos21:43:062
69399451,0cyclictest103550irq/27-eth0-rx-22:34:160
81440,0rcu_preempt37-21ksoftirqd/322:53:173
81440,0rcu_preempt37-21ksoftirqd/322:07:553
81440,0rcu_preempt37-21ksoftirqd/321:53:223
81440,0rcu_preempt37-21ksoftirqd/320:37:553
81440,0rcu_preempt37-21ksoftirqd/320:27:553
81440,0rcu_preempt37-21ksoftirqd/320:27:553
81440,0rcu_preempt37-21ksoftirqd/320:27:553
81440,0rcu_preempt37-21ksoftirqd/320:07:553
81440,0rcu_preempt37-21ksoftirqd/319:33:233
81440,0rcu_preempt37-21ksoftirqd/300:28:293
81440,0rcu_preempt37-21ksoftirqd/300:18:193
81440,0rcu_preempt37-21ksoftirqd/300:08:123
81440,0rcu_preempt29-21ksoftirqd/223:33:082
81440,0rcu_preempt29-21ksoftirqd/222:52:562
81440,0rcu_preempt29-21ksoftirqd/222:47:552
81440,0rcu_preempt29-21ksoftirqd/222:13:192
81440,0rcu_preempt29-21ksoftirqd/222:08:082
81440,0rcu_preempt29-21ksoftirqd/221:13:222
81440,0rcu_preempt29-21ksoftirqd/220:42:552
81440,0rcu_preempt29-21ksoftirqd/220:08:052
81440,0rcu_preempt29-21ksoftirqd/219:28:122
81440,0rcu_preempt29-21ksoftirqd/219:28:122
91430,0rcu_sched37-21ksoftirqd/320:52:563
81430,0rcu_preempt37-21ksoftirqd/323:23:153
81430,0rcu_preempt37-21ksoftirqd/323:12:543
81430,0rcu_preempt37-21ksoftirqd/323:03:073
81430,0rcu_preempt37-21ksoftirqd/322:33:093
81430,0rcu_preempt37-21ksoftirqd/321:02:563
81430,0rcu_preempt29-21ksoftirqd/222:48:102
81430,0rcu_preempt29-21ksoftirqd/219:18:152
81430,0rcu_preempt29-21ksoftirqd/200:28:182
81430,0rcu_preempt29-21ksoftirqd/200:03:142
693994311,0cyclictest0-21swapper/023:13:170
81420,0rcu_preempt37-21ksoftirqd/322:28:103
81420,0rcu_preempt37-21ksoftirqd/320:43:213
81420,0rcu_preempt37-21ksoftirqd/319:58:193
81420,0rcu_preempt29-21ksoftirqd/223:53:102
81420,0rcu_preempt29-21ksoftirqd/221:28:202
81420,0rcu_preempt29-21ksoftirqd/220:53:082
81420,0rcu_preempt29-21ksoftirqd/220:03:052
81420,0rcu_preempt29-21ksoftirqd/219:57:552
81410,0rcu_preempt37-21ksoftirqd/322:37:553
81410,0rcu_preempt37-21ksoftirqd/322:07:553
81410,0rcu_preempt37-21ksoftirqd/321:58:113
81410,0rcu_preempt37-21ksoftirqd/321:18:093
81410,0rcu_preempt37-21ksoftirqd/320:58:113
81410,0rcu_preempt37-21ksoftirqd/300:14:043
81410,0rcu_preempt37-21ksoftirqd/300:14:043
81410,0rcu_preempt29-21ksoftirqd/223:03:092
81410,0rcu_preempt29-21ksoftirqd/220:58:252
69399417,0cyclictest0-21swapper/020:08:050
693994133,0cyclictest23213-21memory21:18:140
693994133,0cyclictest0-21swapper/019:43:130
693994117,0cyclictest28726-21grep00:08:080
693994114,0cyclictest2132-21gdm-simple-gree21:50:530
693994113,0cyclictest10574-21sort20:48:100
81400,0rcu_preempt37-21ksoftirqd/323:58:313
81400,0rcu_preempt37-21ksoftirqd/321:08:243
81400,0rcu_preempt37-21ksoftirqd/320:41:223
81400,0rcu_preempt37-21ksoftirqd/319:48:193
81400,0rcu_preempt29-21ksoftirqd/222:25:102
81400,0rcu_preempt29-21ksoftirqd/220:33:052
81400,0rcu_preempt29-21ksoftirqd/219:23:062
693994033,0cyclictest0-21swapper/022:58:190
693994026,0cyclictest7205-21strings00:33:160
693994019,0cyclictest1242-21runrttasks20:36:270
693994015,0cyclictest0-21swapper/021:10:260
693994014,0cyclictest26343-21munin-node22:43:190
693994014,0cyclictest0-21swapper/022:28:050
693994012,0cyclictest0-21swapper/000:03:260
81390,0rcu_preempt3-21ksoftirqd/021:38:200
81390,0rcu_preempt13532-21latency_hist23:32:540
693993916,0cyclictest27555-21cpuspeed_turbos22:48:050
693993916,0cyclictest19689-21runrttasks23:47:250
693993913,0cyclictest30842-21munin-node20:18:200
81380,0rcu_preempt37-21ksoftirqd/323:53:173
81380,0rcu_preempt37-21ksoftirqd/319:43:063
81380,0rcu_preempt29-21ksoftirqd/223:58:072
69399389,0cyclictest804-21irqbalance20:02:100
693993825,0cyclictest3569-21cpuspeed_turbos23:08:060
693993819,0cyclictest1242-21runrttasks22:57:270
693993819,0cyclictest1242-21runrttasks21:29:130
693993818,0cyclictest9795-21cstates23:23:110
693993815,0cyclictest14231-21munin-node19:38:160
693993815,0cyclictest12438-21sort19:33:210
693993812,0cyclictest6437-21munin-node20:38:100
693993812,0cyclictest4379-21munin-node19:13:250
693993812,0cyclictest0-21swapper/000:13:040
693993812,0cyclictest0-21swapper/000:13:040
693993811,0cyclictest18498-21sed22:23:260
693993731,0cyclictest3-21ksoftirqd/022:17:130
693993725,0cyclictest14760-21proc_pri23:33:190
693993719,0cyclictest1242-21runrttasks20:56:120
693993718,0cyclictest21393-21unixbench_singl23:48:250
693993713,0cyclictest723-21fschecks_time21:43:090
693993713,0cyclictest14193-21cpu20:58:050
693993713,0cyclictest12200-21grep22:08:240
693993712,0cyclictest6926-21munin-node21:58:120
693993712,0cyclictest24812-21munin-node23:58:120
693993711,0cyclictest8923-21seq19:27:510
693993711,0cyclictest21002-21munin-node21:13:120
693993711,0cyclictest16102-21munin-node22:18:190
693993710,0cyclictest9005-21sendmail_mailqu20:43:210
81360,0rcu_preempt21-21ksoftirqd/122:13:351
693993622,0cyclictest0-21swapper/000:28:050
693993611,0cyclictest437-21munin-node20:23:180
693993610,0cyclictest4862-21munin-node21:53:070
693993610,0cyclictest2327-21munin-node20:28:100
693993610,0cyclictest2327-21munin-node20:28:100
69399359,0cyclictest10124-21awk19:28:190
69399359,0cyclictest10124-21awk19:28:190
693993515,0cyclictest23424-21unixbench_multi23:53:240
693993515,0cyclictest10566-21taskset22:07:110
693993514,0cyclictest2277-21ntp_kernel_pll_23:03:180
693993513,0cyclictest19011-21taskset19:49:340
693993510,0cyclictest25453-21munin-node21:23:240
693993510,0cyclictest25453-21munin-node21:23:240
693993510,0cyclictest25118-21munin-node20:03:250
693993419,0cyclictest29034-21fschecks_time21:33:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional