You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-22 - 02:52

Intel(R) Atom(TM) CPU D2700 @ 2.13GHz, Linux 4.4.39-rt50 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6.osadl.org (updated Fri Oct 22, 2021 00:43:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
91650,0rcu_sched21-21ksoftirqd/122:58:311
91610,0rcu_sched20394-21systemd-cgroups19:33:260
91600,0rcu_sched21-21ksoftirqd/119:58:321
81570,0rcu_preempt3-21ksoftirqd/023:48:450
81570,0rcu_preempt14197-21crond19:18:310
81560,0rcu_preempt3-21ksoftirqd/023:03:260
81560,0rcu_preempt29-21ksoftirqd/200:03:502
81550,0rcu_preempt29-21ksoftirqd/223:58:312
81550,0rcu_preempt29-21ksoftirqd/221:08:272
91540,0rcu_sched5190-21kworker/1:000:13:311
81540,0rcu_preempt29-21ksoftirqd/223:49:032
81530,0rcu_preempt3-21ksoftirqd/021:23:320
81530,0rcu_preempt3-21ksoftirqd/021:14:080
81530,0rcu_preempt3-21ksoftirqd/020:13:480
81530,0rcu_preempt29-21ksoftirqd/221:49:182
81530,0rcu_preempt29-21ksoftirqd/221:14:072
81530,0rcu_preempt29-21ksoftirqd/221:08:522
81530,0rcu_preempt29-21ksoftirqd/220:19:052
81520,0rcu_preempt3-21ksoftirqd/020:43:450
81520,0rcu_preempt3-21ksoftirqd/020:23:310
81520,0rcu_preempt29-21ksoftirqd/223:24:042
81520,0rcu_preempt29-21ksoftirqd/223:13:562
81520,0rcu_preempt29-21ksoftirqd/223:13:562
81520,0rcu_preempt29-21ksoftirqd/222:59:022
81520,0rcu_preempt29-21ksoftirqd/222:33:532
81520,0rcu_preempt29-21ksoftirqd/222:04:232
81520,0rcu_preempt29-21ksoftirqd/221:58:502
81520,0rcu_preempt29-21ksoftirqd/221:54:072
81520,0rcu_preempt29-21ksoftirqd/220:54:072
81520,0rcu_preempt29-21ksoftirqd/220:48:312
81520,0rcu_preempt29-21ksoftirqd/220:28:432
81520,0rcu_preempt29-21ksoftirqd/219:54:002
81520,0rcu_preempt19168-21systemd-cgroups20:48:311
81520,0rcu_preempt15096-21unixbench_singl00:29:052
81510,0rcu_preempt29-21ksoftirqd/222:29:062
81510,0rcu_preempt29-21ksoftirqd/221:38:562
81500,0rcu_preempt37-21ksoftirqd/319:03:593
81500,0rcu_preempt3-21ksoftirqd/021:39:030
81500,0rcu_preempt3-21ksoftirqd/000:38:320
81500,0rcu_preempt29-21ksoftirqd/223:09:082
81500,0rcu_preempt29-21ksoftirqd/223:03:422
81500,0rcu_preempt29-21ksoftirqd/222:18:502
81500,0rcu_preempt29-21ksoftirqd/221:33:512
81500,0rcu_preempt29-21ksoftirqd/220:42:212
81500,0rcu_preempt29-21ksoftirqd/220:23:562
81500,0rcu_preempt29-21ksoftirqd/220:04:032
81490,0rcu_preempt3-21ksoftirqd/023:58:580
81490,0rcu_preempt3-21ksoftirqd/021:03:450
81490,0rcu_preempt3-21ksoftirqd/020:08:560
81490,0rcu_preempt3-21ksoftirqd/019:48:490
81490,0rcu_preempt29-21ksoftirqd/223:29:192
81490,0rcu_preempt29-21ksoftirqd/223:18:482
81490,0rcu_preempt29-21ksoftirqd/220:53:032
81490,0rcu_preempt29-21ksoftirqd/219:43:462
81490,0rcu_preempt29-21ksoftirqd/219:36:202
81490,0rcu_preempt29-21ksoftirqd/219:28:512
81490,0rcu_preempt29-21ksoftirqd/219:13:532
81490,0rcu_preempt29-21ksoftirqd/219:03:322
81490,0rcu_preempt29-21ksoftirqd/200:24:032
81490,0rcu_preempt29-21ksoftirqd/200:10:432
81480,0rcu_preempt37-21ksoftirqd/322:13:543
81480,0rcu_preempt37-21ksoftirqd/322:13:543
81480,0rcu_preempt37-21ksoftirqd/320:04:073
81480,0rcu_preempt37-21ksoftirqd/300:28:533
81480,0rcu_preempt3-21ksoftirqd/023:38:480
81480,0rcu_preempt3-21ksoftirqd/023:14:130
81480,0rcu_preempt3-21ksoftirqd/023:14:130
81480,0rcu_preempt3-21ksoftirqd/022:54:040
81480,0rcu_preempt3-21ksoftirqd/022:28:560
81480,0rcu_preempt3-21ksoftirqd/020:53:500
81480,0rcu_preempt3-21ksoftirqd/019:18:480
81480,0rcu_preempt29-21ksoftirqd/222:53:542
81480,0rcu_preempt29-21ksoftirqd/222:14:052
81480,0rcu_preempt29-21ksoftirqd/222:14:052
81480,0rcu_preempt29-21ksoftirqd/221:24:022
81480,0rcu_preempt29-21ksoftirqd/220:58:562
81480,0rcu_preempt29-21ksoftirqd/220:11:092
81480,0rcu_preempt29-21ksoftirqd/219:23:442
81480,0rcu_preempt29-21ksoftirqd/219:23:442
81480,0rcu_preempt29-21ksoftirqd/219:18:502
81480,0rcu_preempt29-21ksoftirqd/219:09:042
81470,0rcu_preempt37-21ksoftirqd/323:03:313
81470,0rcu_preempt37-21ksoftirqd/321:33:473
81470,0rcu_preempt37-21ksoftirqd/321:19:193
81470,0rcu_preempt37-21ksoftirqd/321:19:193
81470,0rcu_preempt3-21ksoftirqd/023:53:510
81470,0rcu_preempt3-21ksoftirqd/023:33:490
81470,0rcu_preempt3-21ksoftirqd/019:58:510
81470,0rcu_preempt3-21ksoftirqd/000:28:560
81470,0rcu_preempt29-21ksoftirqd/223:34:032
81470,0rcu_preempt29-21ksoftirqd/222:08:502
81470,0rcu_preempt29-21ksoftirqd/221:18:322
81470,0rcu_preempt29-21ksoftirqd/221:18:322
81470,0rcu_preempt29-21ksoftirqd/200:13:582
81460,0rcu_preempt37-21ksoftirqd/322:23:533
81460,0rcu_preempt37-21ksoftirqd/320:18:523
81460,0rcu_preempt37-21ksoftirqd/320:09:013
81460,0rcu_preempt3-21ksoftirqd/022:18:310
81460,0rcu_preempt3-21ksoftirqd/022:18:310
81460,0rcu_preempt3-21ksoftirqd/022:04:050
81460,0rcu_preempt3-21ksoftirqd/021:58:540
81460,0rcu_preempt3-21ksoftirqd/021:43:460
81460,0rcu_preempt3-21ksoftirqd/020:58:460
81460,0rcu_preempt3-21ksoftirqd/019:33:530
81460,0rcu_preempt3-21ksoftirqd/019:23:520
81460,0rcu_preempt3-21ksoftirqd/019:23:520
81460,0rcu_preempt3-21ksoftirqd/019:09:480
81460,0rcu_preempt29-21ksoftirqd/223:58:452
81460,0rcu_preempt29-21ksoftirqd/223:44:152
81460,0rcu_preempt29-21ksoftirqd/222:43:482
81460,0rcu_preempt29-21ksoftirqd/222:39:392
81460,0rcu_preempt29-21ksoftirqd/222:23:552
81460,0rcu_preempt29-21ksoftirqd/221:44:002
81460,0rcu_preempt29-21ksoftirqd/221:28:562
81460,0rcu_preempt29-21ksoftirqd/220:34:002
81460,0rcu_preempt29-21ksoftirqd/220:15:192
81460,0rcu_preempt29-21ksoftirqd/200:34:032
81460,0rcu_preempt21-21ksoftirqd/119:03:511
81450,0rcu_preempt37-21ksoftirqd/323:14:093
81450,0rcu_preempt37-21ksoftirqd/323:14:093
81450,0rcu_preempt37-21ksoftirqd/321:23:583
81450,0rcu_preempt37-21ksoftirqd/321:13:573
81450,0rcu_preempt37-21ksoftirqd/320:58:593
81450,0rcu_preempt37-21ksoftirqd/319:33:533
81450,0rcu_preempt3-21ksoftirqd/023:44:120
81450,0rcu_preempt3-21ksoftirqd/023:18:310
81450,0rcu_preempt3-21ksoftirqd/022:09:030
81450,0rcu_preempt3-21ksoftirqd/021:34:040
81450,0rcu_preempt3-21ksoftirqd/021:09:060
81450,0rcu_preempt3-21ksoftirqd/019:03:460
81450,0rcu_preempt3-21ksoftirqd/000:13:530
81450,0rcu_preempt29-21ksoftirqd/222:48:542
81450,0rcu_preempt29-21ksoftirqd/219:58:552
81450,0rcu_preempt29-21ksoftirqd/200:19:012
81440,0rcu_preempt37-21ksoftirqd/323:48:323
81440,0rcu_preempt37-21ksoftirqd/323:34:043
81440,0rcu_preempt37-21ksoftirqd/323:29:103
81440,0rcu_preempt37-21ksoftirqd/323:09:123
81440,0rcu_preempt37-21ksoftirqd/322:54:073
81440,0rcu_preempt37-21ksoftirqd/322:28:453
81440,0rcu_preempt37-21ksoftirqd/322:04:193
81440,0rcu_preempt37-21ksoftirqd/321:58:573
81440,0rcu_preempt37-21ksoftirqd/321:49:203
81440,0rcu_preempt37-21ksoftirqd/320:53:553
81440,0rcu_preempt37-21ksoftirqd/319:43:463
81440,0rcu_preempt37-21ksoftirqd/319:08:593
81440,0rcu_preempt3-21ksoftirqd/023:30:560
81440,0rcu_preempt3-21ksoftirqd/022:43:530
81440,0rcu_preempt3-21ksoftirqd/022:28:310
81440,0rcu_preempt3-21ksoftirqd/021:28:470
81440,0rcu_preempt3-21ksoftirqd/021:19:090
81440,0rcu_preempt3-21ksoftirqd/021:19:090
81440,0rcu_preempt3-21ksoftirqd/000:23:560
81440,0rcu_preempt29-21ksoftirqd/219:48:552
81440,0rcu_preempt29-21ksoftirqd/219:38:562
10283994439,0cyclictest21-21ksoftirqd/123:53:321
81430,0rcu_preempt37-21ksoftirqd/323:58:453
81430,0rcu_preempt37-21ksoftirqd/323:23:513
81430,0rcu_preempt37-21ksoftirqd/323:18:473
81430,0rcu_preempt37-21ksoftirqd/322:48:323
81430,0rcu_preempt37-21ksoftirqd/322:43:543
81430,0rcu_preempt37-21ksoftirqd/322:33:543
81430,0rcu_preempt37-21ksoftirqd/320:48:473
81430,0rcu_preempt37-21ksoftirqd/320:15:593
81430,0rcu_preempt37-21ksoftirqd/300:18:573
81430,0rcu_preempt3-21ksoftirqd/023:23:490
81430,0rcu_preempt3-21ksoftirqd/021:54:090
81430,0rcu_preempt3-21ksoftirqd/020:38:320
81430,0rcu_preempt3-21ksoftirqd/020:24:000
81430,0rcu_preempt3-21ksoftirqd/019:38:490
81430,0rcu_preempt3-21ksoftirqd/000:18:560
81430,0rcu_preempt29-21ksoftirqd/223:38:522
10283994332,0cyclictest21-21ksoftirqd/122:49:031
10283994328,0cyclictest21-21ksoftirqd/122:43:561
10283994327,0cyclictest21-21ksoftirqd/121:43:511
81420,0rcu_preempt37-21ksoftirqd/323:43:323
81420,0rcu_preempt37-21ksoftirqd/323:03:483
81420,0rcu_preempt37-21ksoftirqd/322:38:563
81420,0rcu_preempt37-21ksoftirqd/321:38:483
81420,0rcu_preempt37-21ksoftirqd/321:28:533
81420,0rcu_preempt37-21ksoftirqd/321:08:513
81420,0rcu_preempt37-21ksoftirqd/320:38:453
81420,0rcu_preempt37-21ksoftirqd/320:23:483
81420,0rcu_preempt37-21ksoftirqd/319:53:533
81420,0rcu_preempt37-21ksoftirqd/319:49:003
81420,0rcu_preempt3-21ksoftirqd/023:03:540
81420,0rcu_preempt3-21ksoftirqd/022:18:560
81420,0rcu_preempt3-21ksoftirqd/021:49:040
81420,0rcu_preempt3-21ksoftirqd/000:13:260
81420,0rcu_preempt3-21ksoftirqd/000:04:070
10283994238,0cyclictest21-21ksoftirqd/121:58:541
10283994237,0cyclictest201ktimersoftd/120:23:311
10283994236,0cyclictest21-21ksoftirqd/123:14:121
10283994236,0cyclictest21-21ksoftirqd/123:14:121
81410,0rcu_preempt37-21ksoftirqd/322:08:483
81410,0rcu_preempt37-21ksoftirqd/321:03:503
81410,0rcu_preempt37-21ksoftirqd/320:43:483
81410,0rcu_preempt37-21ksoftirqd/319:14:053
81410,0rcu_preempt37-21ksoftirqd/300:34:133
81410,0rcu_preempt37-21ksoftirqd/300:23:533
81410,0rcu_preempt37-21ksoftirqd/300:08:473
81410,0rcu_preempt3-21ksoftirqd/022:50:590
81410,0rcu_preempt3-21ksoftirqd/020:43:320
10283994136,0cyclictest21-21ksoftirqd/122:38:451
10283994136,0cyclictest21-21ksoftirqd/122:19:151
10283994136,0cyclictest21-21ksoftirqd/121:33:321
10283994136,0cyclictest21-21ksoftirqd/119:38:491
10283994136,0cyclictest21-21ksoftirqd/100:02:291
10283994135,0cyclictest21-21ksoftirqd/123:53:521
10283994135,0cyclictest21-21ksoftirqd/122:23:541
10283994133,0cyclictest21-21ksoftirqd/119:48:481
10283994127,0cyclictest21-21ksoftirqd/120:38:591
81400,0rcu_preempt37-21ksoftirqd/323:55:423
81400,0rcu_preempt37-21ksoftirqd/322:18:493
81400,0rcu_preempt37-21ksoftirqd/320:28:533
81400,0rcu_preempt37-21ksoftirqd/319:58:443
81400,0rcu_preempt37-21ksoftirqd/319:38:503
81400,0rcu_preempt37-21ksoftirqd/319:23:353
81400,0rcu_preempt37-21ksoftirqd/319:23:353
81400,0rcu_preempt37-21ksoftirqd/319:18:493
81400,0rcu_preempt37-21ksoftirqd/300:03:483
81400,0rcu_preempt3-21ksoftirqd/023:09:060
81400,0rcu_preempt3-21ksoftirqd/022:39:410
81400,0rcu_preempt3-21ksoftirqd/020:28:490
81400,0rcu_preempt3-21ksoftirqd/019:54:010
81400,0rcu_preempt3-21ksoftirqd/019:44:000
10283994035,0cyclictest21-21ksoftirqd/123:39:041
10283994035,0cyclictest21-21ksoftirqd/122:04:031
10283994035,0cyclictest21-21ksoftirqd/120:48:591
10283994035,0cyclictest21-21ksoftirqd/120:23:551
10283994035,0cyclictest21-21ksoftirqd/119:13:561
10283994034,0cyclictest21-21ksoftirqd/121:54:141
10283994034,0cyclictest21-21ksoftirqd/120:34:021
10283994034,0cyclictest201ktimersoftd/123:29:191
10283994026,0cyclictest21-21ksoftirqd/121:18:571
10283994026,0cyclictest21-21ksoftirqd/121:18:571
10283994026,0cyclictest21-21ksoftirqd/119:23:571
10283994026,0cyclictest21-21ksoftirqd/119:23:571
10283994025,0cyclictest21-21ksoftirqd/119:58:581
10283994024,0cyclictest21-21ksoftirqd/122:29:011
81390,0rcu_preempt37-21ksoftirqd/321:43:493
81390,0rcu_preempt37-21ksoftirqd/319:28:513
81390,0rcu_preempt37-21ksoftirqd/300:13:323
81390,0rcu_preempt3-21ksoftirqd/022:33:450
81390,0rcu_preempt3-21ksoftirqd/020:48:450
81390,0rcu_preempt21-21ksoftirqd/119:08:311
10283993934,0cyclictest21-21ksoftirqd/123:24:011
10283993934,0cyclictest21-21ksoftirqd/122:08:451
10283993934,0cyclictest21-21ksoftirqd/120:03:471
10283993934,0cyclictest21-21ksoftirqd/119:43:491
10283993934,0cyclictest21-21ksoftirqd/100:19:051
10283993933,0cyclictest21-21ksoftirqd/123:03:451
10283993933,0cyclictest21-21ksoftirqd/121:29:031
10283993933,0cyclictest21-21ksoftirqd/121:24:081
10283993933,0cyclictest21-21ksoftirqd/120:58:561
10283993933,0cyclictest21-21ksoftirqd/120:13:541
10283993932,0cyclictest21-21ksoftirqd/119:28:471
10283993930,0cyclictest21-21ksoftirqd/100:14:041
10283993926,0cyclictest21-21ksoftirqd/122:13:471
10283993926,0cyclictest21-21ksoftirqd/122:13:471
81380,0rcu_preempt37-21ksoftirqd/323:38:473
81380,0rcu_preempt37-21ksoftirqd/320:34:103
81380,0rcu_preempt3-21ksoftirqd/020:04:080
10283993834,0cyclictest21-21ksoftirqd/123:18:541
10283993834,0cyclictest21-21ksoftirqd/123:09:131
10283993833,0cyclictest21-21ksoftirqd/123:43:591
10283993833,0cyclictest21-21ksoftirqd/122:59:021
10283993833,0cyclictest21-21ksoftirqd/121:49:191
10283993833,0cyclictest21-21ksoftirqd/121:08:511
10283993833,0cyclictest21-21ksoftirqd/100:03:501
10283993832,0cyclictest21-21ksoftirqd/120:53:481
81370,0rcu_preempt37-21ksoftirqd/321:54:083
10283993732,0cyclictest21-21ksoftirqd/121:03:521
10283993732,0cyclictest21-21ksoftirqd/120:08:531
10283993732,0cyclictest21-21ksoftirqd/100:23:561
10283993731,0cyclictest21-21ksoftirqd/123:33:441
10283993731,0cyclictest21-21ksoftirqd/119:18:511
10283993723,0cyclictest21-21ksoftirqd/120:32:151
1799360,0watchdog/1827-21dbus-daemon21:13:321
10283993631,0cyclictest21-21ksoftirqd/122:33:461
10283993631,0cyclictest21-21ksoftirqd/121:39:021
10283993631,0cyclictest21-21ksoftirqd/119:33:441
10283993631,0cyclictest21-21ksoftirqd/100:28:591
10283993531,0cyclictest21-21ksoftirqd/100:34:051
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional