You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-17 - 02:23

Intel(R) Atom(TM) CPU D2700 @ 2.13GHz, Linux 4.4.39-rt50 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot6.osadl.org (updated Mon Jan 17, 2022 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
81640,0rcu_preempt29-21ksoftirqd/223:00:512
91620,0rcu_sched21-21ksoftirqd/123:30:501
91620,0rcu_sched21-21ksoftirqd/123:30:501
81620,0rcu_preempt29-21ksoftirqd/219:26:042
81600,0rcu_preempt21-21ksoftirqd/100:20:501
30450995827,0cyclictest29558-21grep21:36:242
81570,0rcu_preempt3-21ksoftirqd/023:42:440
81560,0rcu_preempt3-21ksoftirqd/022:16:210
81560,0rcu_preempt3-21ksoftirqd/000:15:510
81560,0rcu_preempt29-21ksoftirqd/200:31:032
81560,0rcu_preempt21-21ksoftirqd/121:55:511
81560,0rcu_preempt21-21ksoftirqd/119:31:231
81550,0rcu_preempt29-21ksoftirqd/222:36:082
81550,0rcu_preempt29-21ksoftirqd/200:06:182
91540,0rcu_sched37-21ksoftirqd/300:35:503
81540,0rcu_preempt3-21ksoftirqd/021:25:510
81540,0rcu_preempt29-21ksoftirqd/200:21:072
81540,0rcu_preempt29-21ksoftirqd/200:21:072
81540,0rcu_preempt16217-21gltestperf21:06:092
81530,0rcu_preempt37-21ksoftirqd/323:04:113
81530,0rcu_preempt37-21ksoftirqd/320:35:513
81530,0rcu_preempt3-21ksoftirqd/023:48:220
81530,0rcu_preempt3-21ksoftirqd/020:31:430
81530,0rcu_preempt3-21ksoftirqd/000:01:070
81530,0rcu_preempt29-21ksoftirqd/220:51:102
81530,0rcu_preempt29-21ksoftirqd/220:00:512
81530,0rcu_preempt21-21ksoftirqd/121:50:511
2699530,0migration/230244-21idleruntime-cro19:05:512
91520,0rcu_sched37-21ksoftirqd/322:05:513
91520,0rcu_sched29-21ksoftirqd/223:55:512
81520,0rcu_preempt37-21ksoftirqd/322:00:513
81520,0rcu_preempt3-21ksoftirqd/019:45:590
81520,0rcu_preempt29-21ksoftirqd/223:16:202
81520,0rcu_preempt29-21ksoftirqd/222:16:172
81520,0rcu_preempt29-21ksoftirqd/220:26:252
81520,0rcu_preempt29-21ksoftirqd/219:46:242
81520,0rcu_preempt21-21ksoftirqd/119:15:521
81520,0rcu_preempt21-21ksoftirqd/100:13:521
81520,0rcu_preempt1820-21systemd-cgroups21:50:510
81510,0rcu_preempt3-21ksoftirqd/020:56:180
81510,0rcu_preempt3-21ksoftirqd/000:21:060
81510,0rcu_preempt3-21ksoftirqd/000:21:060
81510,0rcu_preempt29-21ksoftirqd/223:41:122
81510,0rcu_preempt29-21ksoftirqd/223:06:142
81510,0rcu_preempt29-21ksoftirqd/222:01:152
81510,0rcu_preempt29-21ksoftirqd/221:46:192
81510,0rcu_preempt29-21ksoftirqd/220:16:232
81510,0rcu_preempt21-21ksoftirqd/100:26:261
30450995127,0cyclictest29-21ksoftirqd/200:05:152
1599510,0migration/028959-21idleruntime19:01:060
91500,0rcu_sched21-21ksoftirqd/120:15:511
81500,0rcu_preempt3-21ksoftirqd/022:51:080
81500,0rcu_preempt3-21ksoftirqd/021:11:070
81500,0rcu_preempt29-21ksoftirqd/223:46:072
81500,0rcu_preempt29-21ksoftirqd/222:41:082
81500,0rcu_preempt29-21ksoftirqd/221:56:092
81500,0rcu_preempt29-21ksoftirqd/221:10:522
81500,0rcu_preempt29-21ksoftirqd/221:00:512
81500,0rcu_preempt29-21ksoftirqd/220:21:222
81500,0rcu_preempt29-21ksoftirqd/219:41:202
81500,0rcu_preempt29-21ksoftirqd/219:31:072
81500,0rcu_preempt29-21ksoftirqd/219:11:062
81500,0rcu_preempt29-21ksoftirqd/200:26:212
81500,0rcu_preempt29-21ksoftirqd/200:11:092
81500,0rcu_preempt21-21ksoftirqd/123:03:391
81500,0rcu_preempt21-21ksoftirqd/120:55:561
81500,0rcu_preempt21-21ksoftirqd/120:15:521
81500,0rcu_preempt21-21ksoftirqd/119:51:081
81490,0rcu_preempt37-21ksoftirqd/322:36:163
81490,0rcu_preempt3-21ksoftirqd/020:01:120
81490,0rcu_preempt3-21ksoftirqd/019:51:060
81490,0rcu_preempt29-21ksoftirqd/223:21:172
81490,0rcu_preempt29-21ksoftirqd/221:31:052
81490,0rcu_preempt29-21ksoftirqd/221:31:052
81490,0rcu_preempt29-21ksoftirqd/220:31:122
81490,0rcu_preempt29-21ksoftirqd/220:01:132
81490,0rcu_preempt29-21ksoftirqd/219:21:062
81490,0rcu_preempt29-21ksoftirqd/200:16:202
9750480,0irq/25-0000:00:28146-21latency_hist19:00:523
81480,0rcu_preempt37-21ksoftirqd/323:00:513
81480,0rcu_preempt3-21ksoftirqd/023:26:110
81480,0rcu_preempt3-21ksoftirqd/023:26:110
81480,0rcu_preempt3-21ksoftirqd/022:01:090
81480,0rcu_preempt3-21ksoftirqd/021:05:520
81480,0rcu_preempt29-21ksoftirqd/223:56:152
81480,0rcu_preempt29-21ksoftirqd/223:30:502
81480,0rcu_preempt29-21ksoftirqd/223:30:502
81480,0rcu_preempt29-21ksoftirqd/222:46:102
81480,0rcu_preempt29-21ksoftirqd/222:31:022
81480,0rcu_preempt29-21ksoftirqd/222:26:062
81480,0rcu_preempt29-21ksoftirqd/222:26:062
81480,0rcu_preempt29-21ksoftirqd/221:52:462
81480,0rcu_preempt29-21ksoftirqd/221:41:142
81480,0rcu_preempt29-21ksoftirqd/221:21:242
81480,0rcu_preempt29-21ksoftirqd/221:16:032
81480,0rcu_preempt29-21ksoftirqd/220:41:062
81480,0rcu_preempt29-21ksoftirqd/220:40:512
81480,0rcu_preempt29-21ksoftirqd/220:40:512
81480,0rcu_preempt29-21ksoftirqd/220:14:502
81480,0rcu_preempt29-21ksoftirqd/220:06:042
81480,0rcu_preempt29-21ksoftirqd/219:51:022
81480,0rcu_preempt29-21ksoftirqd/219:06:212
81480,0rcu_preempt21-21ksoftirqd/119:05:511
81470,0rcu_preempt29-21ksoftirqd/223:31:232
81470,0rcu_preempt29-21ksoftirqd/223:11:072
81470,0rcu_preempt29-21ksoftirqd/222:51:282
81470,0rcu_preempt29-21ksoftirqd/222:21:222
81470,0rcu_preempt29-21ksoftirqd/222:11:182
81470,0rcu_preempt29-21ksoftirqd/221:25:512
81470,0rcu_preempt29-21ksoftirqd/221:01:152
81470,0rcu_preempt29-21ksoftirqd/220:50:512
81470,0rcu_preempt29-21ksoftirqd/219:16:082
81470,0rcu_preempt21-21ksoftirqd/121:11:081
81470,0rcu_preempt21-21ksoftirqd/120:36:081
81470,0rcu_preempt21-21ksoftirqd/120:36:081
91460,0rcu_sched3-21ksoftirqd/020:55:510
81460,0rcu_preempt29-21ksoftirqd/223:01:142
81460,0rcu_preempt29-21ksoftirqd/222:06:112
81460,0rcu_preempt21-21ksoftirqd/123:31:071
81460,0rcu_preempt21-21ksoftirqd/123:06:171
81460,0rcu_preempt21-21ksoftirqd/121:41:061
81460,0rcu_preempt21-21ksoftirqd/121:31:061
81460,0rcu_preempt21-21ksoftirqd/121:31:061
81460,0rcu_preempt21-21ksoftirqd/121:01:121
81460,0rcu_preempt21-21ksoftirqd/120:25:511
81460,0rcu_preempt21-21ksoftirqd/119:21:141
91450,0rcu_sched37-21ksoftirqd/321:35:513
81450,0rcu_preempt3-21ksoftirqd/023:11:060
81450,0rcu_preempt29-21ksoftirqd/219:35:512
81450,0rcu_preempt29-21ksoftirqd/219:35:512
81450,0rcu_preempt21-21ksoftirqd/123:16:111
81450,0rcu_preempt21-21ksoftirqd/122:56:221
81450,0rcu_preempt21-21ksoftirqd/122:50:511
81450,0rcu_preempt21-21ksoftirqd/122:30:461
81450,0rcu_preempt21-21ksoftirqd/122:30:461
81450,0rcu_preempt21-21ksoftirqd/120:01:421
81440,0rcu_preempt29-21ksoftirqd/223:36:052
81440,0rcu_preempt21-21ksoftirqd/123:44:151
81440,0rcu_preempt21-21ksoftirqd/123:36:231
81440,0rcu_preempt21-21ksoftirqd/100:06:221
30454994412,0cyclictest27516-21sort20:16:073
30437994435,0cyclictest3-21ksoftirqd/000:31:220
81430,0rcu_preempt3-21ksoftirqd/022:56:110
81430,0rcu_preempt3-21ksoftirqd/022:20:510
81430,0rcu_preempt3-21ksoftirqd/021:36:080
81430,0rcu_preempt3-21ksoftirqd/020:11:150
81430,0rcu_preempt21-21ksoftirqd/123:11:101
81430,0rcu_preempt21-21ksoftirqd/122:41:181
81430,0rcu_preempt21-21ksoftirqd/122:22:281
81430,0rcu_preempt21-21ksoftirqd/120:21:181
81430,0rcu_preempt21-21ksoftirqd/119:37:301
81430,0rcu_preempt21-21ksoftirqd/119:37:301
81430,0rcu_preempt21-21ksoftirqd/119:06:161
30437994338,0cyclictest7403-21cpu20:46:060
81420,0rcu_preempt3-21ksoftirqd/023:58:020
81420,0rcu_preempt3-21ksoftirqd/022:50:490
81420,0rcu_preempt3-21ksoftirqd/022:11:020
81420,0rcu_preempt3-21ksoftirqd/019:25:500
81420,0rcu_preempt21-21ksoftirqd/123:48:501
81420,0rcu_preempt21-21ksoftirqd/122:32:301
81420,0rcu_preempt21-21ksoftirqd/122:06:071
81420,0rcu_preempt21-21ksoftirqd/100:02:091
30454994219,0cyclictest18536-21proc_pri23:46:163
91410,0rcu_sched3-21ksoftirqd/020:20:460
81410,0rcu_preempt3-21ksoftirqd/022:40:510
81410,0rcu_preempt3-21ksoftirqd/022:26:060
81410,0rcu_preempt3-21ksoftirqd/022:26:060
81410,0rcu_preempt3-21ksoftirqd/021:56:150
81410,0rcu_preempt3-21ksoftirqd/021:31:160
81410,0rcu_preempt3-21ksoftirqd/021:31:160
81410,0rcu_preempt3-21ksoftirqd/000:16:070
81410,0rcu_preempt21-21ksoftirqd/122:46:091
81410,0rcu_preempt21-21ksoftirqd/122:40:421
81410,0rcu_preempt21-21ksoftirqd/122:16:071
81410,0rcu_preempt21-21ksoftirqd/121:16:061
81410,0rcu_preempt21-21ksoftirqd/120:31:061
81410,0rcu_preempt21-21ksoftirqd/100:21:201
81410,0rcu_preempt21-21ksoftirqd/100:21:201
9750400,0irq/25-0000:00:0-21swapper/319:05:523
81400,0rcu_preempt3-21ksoftirqd/023:40:510
81400,0rcu_preempt3-21ksoftirqd/023:16:230
81400,0rcu_preempt3-21ksoftirqd/022:31:020
81400,0rcu_preempt3-21ksoftirqd/020:21:230
81400,0rcu_preempt3-21ksoftirqd/019:59:090
81400,0rcu_preempt3-21ksoftirqd/019:26:080
81400,0rcu_preempt21-21ksoftirqd/121:06:131
81400,0rcu_preempt21-21ksoftirqd/120:55:511
81400,0rcu_preempt21-21ksoftirqd/120:41:021
81400,0rcu_preempt21-21ksoftirqd/119:11:071
81400,0rcu_preempt21-21ksoftirqd/100:31:071
30454994028,0cyclictest2859-21grep19:16:083
30454994022,0cyclictest27507-21cut22:51:133
30454994019,0cyclictest30132-21tr00:16:033
81390,0rcu_preempt3-21ksoftirqd/023:21:060
81390,0rcu_preempt3-21ksoftirqd/023:06:250
81390,0rcu_preempt3-21ksoftirqd/023:01:170
81390,0rcu_preempt3-21ksoftirqd/022:39:290
81390,0rcu_preempt3-21ksoftirqd/021:21:210
81390,0rcu_preempt3-21ksoftirqd/021:16:100
81390,0rcu_preempt3-21ksoftirqd/020:36:180
81390,0rcu_preempt3-21ksoftirqd/020:36:180
81390,0rcu_preempt3-21ksoftirqd/000:06:180
81390,0rcu_preempt21-21ksoftirqd/122:14:041
81390,0rcu_preempt21-21ksoftirqd/121:26:231
81390,0rcu_preempt21-21ksoftirqd/119:46:251
81390,0rcu_preempt21-21ksoftirqd/119:26:171
30454993925,0cyclictest13216-21gltestperf19:41:053
30454993922,0cyclictest3360-21fschecks_time23:11:063
30454993921,0cyclictest9768-21sed23:26:123
30454993921,0cyclictest9768-21sed23:26:123
30454993920,0cyclictest15903-21switchtime19:46:213
30454993914,0cyclictest11603-21munin-node20:56:073
81380,0rcu_preempt37-21ksoftirqd/322:40:513
81380,0rcu_preempt3-21ksoftirqd/023:50:510
81380,0rcu_preempt3-21ksoftirqd/021:51:050
81380,0rcu_preempt3-21ksoftirqd/020:41:210
81380,0rcu_preempt3-21ksoftirqd/019:16:200
81380,0rcu_preempt3-21ksoftirqd/019:11:080
81380,0rcu_preempt3-21ksoftirqd/019:06:160
81380,0rcu_preempt21-21ksoftirqd/123:56:141
81380,0rcu_preempt21-21ksoftirqd/121:24:391
30454993823,0cyclictest9981-21unixbench_singl19:31:223
30454993823,0cyclictest19370-21kernelversion22:31:093
30454993822,0cyclictest31097-21head21:41:143
30454993821,0cyclictest24919-21switchtime00:01:213
30454993820,0cyclictest5708-21fschecks_time20:41:053
30454993819,0cyclictest18600-21missed_timers21:11:133
30454993817,0cyclictest7496-21tr20:46:093
30454993817,0cyclictest31524-21tr20:26:063
30454993817,0cyclictest11762-21grep22:11:213
30454993813,0cyclictest0-21swapper/320:36:133
30454993813,0cyclictest0-21swapper/320:36:133
30454993813,0cyclictest0-21swapper/319:51:043
81370,0rcu_preempt3-21ksoftirqd/022:05:520
81370,0rcu_preempt3-21ksoftirqd/020:26:120
81370,0rcu_preempt21-21ksoftirqd/123:21:031
81370,0rcu_preempt21-21ksoftirqd/122:00:521
81370,0rcu_preempt21-21ksoftirqd/121:56:151
81370,0rcu_preempt21-21ksoftirqd/121:36:101
81370,0rcu_preempt21-21ksoftirqd/120:46:061
30454993729,0cyclictest30490-21switchtime20:21:193
30454993725,0cyclictest827-21dbus-daemon21:55:513
30454993724,0cyclictest13856-21tail23:36:083
30454993723,0cyclictest7003-21gltestperf19:26:083
30454993723,0cyclictest440-21tune2fs19:11:103
30454993723,0cyclictest37-21ksoftirqd/323:35:473
30454993722,0cyclictest29317-21unixbench_multi00:11:233
30454993721,0cyclictest7686-21sed23:21:073
30454993721,0cyclictest25602-21grep20:11:073
30454993721,0cyclictest16747-21cpuspeed_turbos22:26:063
30454993721,0cyclictest16747-21cpuspeed_turbos22:26:063
30454993720,0cyclictest9617-21cpuspeed_turbos20:51:033
30454993720,0cyclictest16901-21unixbench_singl23:41:263
30454993720,0cyclictest12551-21cpuspeed_turbos22:16:043
30454993719,0cyclictest13996-21fschecks_time21:01:123
30454993718,0cyclictest22976-21timerwakeupswit23:56:243
30454993718,0cyclictest22225-21uniq21:21:093
81360,0rcu_preempt3-21ksoftirqd/019:31:160
81360,0rcu_preempt3-21ksoftirqd/000:27:260
81360,0rcu_preempt21-21ksoftirqd/123:51:211
81360,0rcu_preempt21-21ksoftirqd/120:09:251
30454993624,0cyclictest2291-21tune2fs00:26:113
30454993623,0cyclictest10722-21cpuspeed_turbos19:36:033
30454993623,0cyclictest10722-21cpuspeed_turbos19:36:033
30454993622,0cyclictest32759-21grep21:46:073
30454993622,0cyclictest32351-21fschecks_count00:21:053
30454993622,0cyclictest32351-21fschecks_count00:21:053
30454993622,0cyclictest21332-21fschecks_count20:01:093
30454993622,0cyclictest20486-21needreboot23:51:133
30454993621,0cyclictest9199-21tail22:06:143
30454993621,0cyclictest26008-21cpuspeed_turbos00:06:033
30454993621,0cyclictest23562-21gltestperf20:06:083
30454993620,0cyclictest6246-21crond19:25:503
30454993620,0cyclictest5603-21gltestperf23:16:083
30454993620,0cyclictest24254-21fschecks_count21:26:093
30454993618,0cyclictest25438-21grep22:46:083
30454993618,0cyclictest14650-21tr22:21:043
30454993618,0cyclictest1342-21gltestperf23:06:093
30454993617,0cyclictest17164-21unixbench_singl21:06:243
30454993616,0cyclictest18919-21munin-node19:56:153
30454993614,0cyclictest0-21swapper/321:31:083
30454993614,0cyclictest0-21swapper/321:31:083
81350,0rcu_preempt3-21ksoftirqd/021:41:080
81350,0rcu_preempt3-21ksoftirqd/019:36:100
81350,0rcu_preempt3-21ksoftirqd/019:36:100
81350,0rcu_preempt21-21ksoftirqd/119:56:561
81350,0rcu_preempt21-21ksoftirqd/119:42:311
30454993520,0cyclictest19977-21cpuspeed_turbos21:16:033
81340,0rcu_preempt3-21ksoftirqd/023:31:060
81340,0rcu_preempt3-21ksoftirqd/021:01:190
81340,0rcu_preempt3-21ksoftirqd/019:41:120
30437993426,0cyclictest3-21ksoftirqd/020:06:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional