You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-05-12 - 13:57

Intel(R) Atom(TM) CPU D2700 @ 2.13GHz, Linux 4.4.39-rt50 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6.osadl.org (updated Wed May 12, 2021 12:43:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
81700,0rcu_preempt29-21ksoftirqd/208:59:372
81650,0rcu_preempt9847-21turbostat10:29:372
81610,0rcu_preempt18500-21crond08:14:372
81580,0rcu_preempt3-21ksoftirqd/010:25:070
81580,0rcu_preempt3-21ksoftirqd/009:04:580
81560,0rcu_preempt37-21ksoftirqd/307:04:373
91540,0rcu_sched3-21ksoftirqd/010:04:370
91540,0rcu_sched3-21ksoftirqd/009:34:370
91540,0rcu_sched3-21ksoftirqd/008:59:370
81540,0rcu_preempt3-21ksoftirqd/011:24:370
22035995422,0cyclictest37-21ksoftirqd/307:29:153
81530,0rcu_preempt6927-21crond09:04:372
81530,0rcu_preempt3-21ksoftirqd/012:15:070
81530,0rcu_preempt3-21ksoftirqd/012:15:070
81530,0rcu_preempt3-21ksoftirqd/008:19:510
81530,0rcu_preempt21-21ksoftirqd/110:45:031
81530,0rcu_preempt21-21ksoftirqd/110:18:321
81530,0rcu_preempt21-21ksoftirqd/108:45:021
91520,0rcu_sched914-21dbus-daemon10:34:370
91520,0rcu_sched3-21ksoftirqd/012:09:370
91520,0rcu_sched3-21ksoftirqd/009:24:370
81520,0rcu_preempt37-21ksoftirqd/309:34:373
81520,0rcu_preempt3-21ksoftirqd/012:19:590
81520,0rcu_preempt3-21ksoftirqd/010:19:370
81520,0rcu_preempt24243-21gltestperf09:44:513
81520,0rcu_preempt24243-21gltestperf09:44:513
81520,0rcu_preempt21-21ksoftirqd/111:18:081
81520,0rcu_preempt21-21ksoftirqd/110:19:511
81520,0rcu_preempt21-21ksoftirqd/109:24:381
81520,0rcu_preempt21-21ksoftirqd/109:05:041
81520,0rcu_preempt21-21ksoftirqd/108:15:011
81520,0rcu_preempt21-21ksoftirqd/107:20:351
81520,0rcu_preempt11458-21cpuspeed_turbos09:14:473
3499520,0migration/321531-21sshd07:01:153
22035995220,0cyclictest20840-21cpu08:19:483
1899520,0migration/120536-21idleruntime06:59:521
91510,0rcu_sched29-21ksoftirqd/207:04:372
91510,0rcu_sched0-21swapper/011:04:320
81510,0rcu_preempt3-21ksoftirqd/009:25:050
81510,0rcu_preempt3-21ksoftirqd/008:30:060
81510,0rcu_preempt3-21ksoftirqd/007:14:320
81510,0rcu_preempt3-21ksoftirqd/007:14:320
81510,0rcu_preempt21-21ksoftirqd/112:33:341
81510,0rcu_preempt21-21ksoftirqd/111:44:491
81510,0rcu_preempt21-21ksoftirqd/110:28:291
81510,0rcu_preempt21-21ksoftirqd/107:48:451
22035995123,0cyclictest37-21ksoftirqd/307:23:483
91500,0rcu_sched21-21ksoftirqd/107:49:391
81500,0rcu_preempt37-21ksoftirqd/310:54:383
81500,0rcu_preempt3-21ksoftirqd/012:04:370
81500,0rcu_preempt3-21ksoftirqd/011:49:370
81500,0rcu_preempt3-21ksoftirqd/011:44:210
81500,0rcu_preempt3-21ksoftirqd/011:24:520
81500,0rcu_preempt3-21ksoftirqd/009:09:500
81500,0rcu_preempt3-21ksoftirqd/008:12:550
81500,0rcu_preempt3-21ksoftirqd/007:24:500
81500,0rcu_preempt21-21ksoftirqd/111:54:531
81500,0rcu_preempt21-21ksoftirqd/110:54:141
81500,0rcu_preempt21-21ksoftirqd/110:40:271
81500,0rcu_preempt21-21ksoftirqd/109:59:541
81500,0rcu_preempt21-21ksoftirqd/109:29:481
81500,0rcu_preempt21-21ksoftirqd/109:00:041
81500,0rcu_preempt21-21ksoftirqd/108:43:231
81500,0rcu_preempt21-21ksoftirqd/108:20:391
81500,0rcu_preempt21-21ksoftirqd/107:54:471
81500,0rcu_preempt21-21ksoftirqd/107:35:051
22035995015,0cyclictest2796-21processes10:10:063
22035995014,0cyclictest17177-21gltestperf08:09:523
91490,0rcu_sched3-21ksoftirqd/012:34:320
91490,0rcu_sched3-21ksoftirqd/011:14:370
81490,0rcu_preempt37-21ksoftirqd/312:19:463
81490,0rcu_preempt37-21ksoftirqd/311:55:003
81490,0rcu_preempt37-21ksoftirqd/308:55:063
81490,0rcu_preempt37-21ksoftirqd/308:39:483
81490,0rcu_preempt37-21ksoftirqd/308:29:483
81490,0rcu_preempt3-21ksoftirqd/011:09:370
81490,0rcu_preempt3-21ksoftirqd/010:39:320
81490,0rcu_preempt3-21ksoftirqd/010:39:320
81490,0rcu_preempt3-21ksoftirqd/009:34:510
81490,0rcu_preempt21-21ksoftirqd/112:24:221
81490,0rcu_preempt21-21ksoftirqd/107:15:071
81490,0rcu_preempt21-21ksoftirqd/107:15:071
22035994913,0cyclictest2050-21timerandwakeup08:50:043
22035994913,0cyclictest2050-21timerandwakeup08:50:043
22035994912,0cyclictest17631-21cpuspeed_turbos09:29:473
91480,0rcu_sched3-21ksoftirqd/011:54:370
81480,0rcu_preempt3-21ksoftirqd/012:24:550
81480,0rcu_preempt3-21ksoftirqd/011:55:000
81480,0rcu_preempt3-21ksoftirqd/009:54:510
81480,0rcu_preempt3-21ksoftirqd/007:44:530
81480,0rcu_preempt21-21ksoftirqd/111:35:081
81480,0rcu_preempt21-21ksoftirqd/110:54:481
81480,0rcu_preempt21-21ksoftirqd/110:29:551
81480,0rcu_preempt21-21ksoftirqd/108:04:271
2203599482,0cyclictest10779-21fschecks_time07:54:513
22035994817,0cyclictest29108-21fschecks_time11:14:513
22035994816,0cyclictest7227-21cpu09:04:473
22035994816,0cyclictest26671-21gltestperf07:15:003
22035994816,0cyclictest26671-21gltestperf07:15:003
22035994815,0cyclictest22692-21processes09:40:043
91470,0rcu_sched25174-21crond12:24:372
81470,0rcu_preempt37-21ksoftirqd/311:25:123
81470,0rcu_preempt37-21ksoftirqd/311:14:363
81470,0rcu_preempt37-21ksoftirqd/308:35:043
81470,0rcu_preempt3-21ksoftirqd/011:59:510
81470,0rcu_preempt3-21ksoftirqd/011:29:560
81470,0rcu_preempt3-21ksoftirqd/010:49:520
81470,0rcu_preempt3-21ksoftirqd/010:49:370
81470,0rcu_preempt3-21ksoftirqd/010:04:520
81470,0rcu_preempt3-21ksoftirqd/009:49:380
81470,0rcu_preempt3-21ksoftirqd/008:49:500
81470,0rcu_preempt3-21ksoftirqd/008:49:500
81470,0rcu_preempt3-21ksoftirqd/008:04:550
81470,0rcu_preempt3-21ksoftirqd/007:34:490
81470,0rcu_preempt3-21ksoftirqd/007:29:380
81470,0rcu_preempt3-21ksoftirqd/006:59:480
81470,0rcu_preempt21-21ksoftirqd/112:09:491
81470,0rcu_preempt21-21ksoftirqd/110:10:061
81470,0rcu_preempt21-21ksoftirqd/109:55:371
81470,0rcu_preempt21-21ksoftirqd/109:44:591
81470,0rcu_preempt21-21ksoftirqd/109:44:591
81470,0rcu_preempt21-21ksoftirqd/109:44:221
81470,0rcu_preempt21-21ksoftirqd/108:37:321
81470,0rcu_preempt21-21ksoftirqd/107:43:071
22035994718,0cyclictest10630-21unixbench_multi09:10:123
22035994717,0cyclictest22906-21cpu08:24:483
22035994714,0cyclictest26712-21ntpdc09:50:023
22035994714,0cyclictest15825-21timerandwakeup08:05:043
22035994713,0cyclictest32484-21gltestperf10:04:513
81460,0rcu_preempt37-21ksoftirqd/312:10:103
81460,0rcu_preempt37-21ksoftirqd/311:59:553
81460,0rcu_preempt37-21ksoftirqd/311:29:473
81460,0rcu_preempt37-21ksoftirqd/310:34:503
81460,0rcu_preempt37-21ksoftirqd/310:34:503
81460,0rcu_preempt37-21ksoftirqd/309:59:383
81460,0rcu_preempt37-21ksoftirqd/309:54:593
81460,0rcu_preempt37-21ksoftirqd/309:25:053
81460,0rcu_preempt37-21ksoftirqd/309:20:003
81460,0rcu_preempt3-21ksoftirqd/011:10:030
81460,0rcu_preempt3-21ksoftirqd/010:39:530
81460,0rcu_preempt3-21ksoftirqd/009:44:490
81460,0rcu_preempt3-21ksoftirqd/009:44:490
81460,0rcu_preempt3-21ksoftirqd/009:44:370
81460,0rcu_preempt3-21ksoftirqd/009:15:000
81460,0rcu_preempt3-21ksoftirqd/008:59:380
81460,0rcu_preempt3-21ksoftirqd/008:35:000
81460,0rcu_preempt3-21ksoftirqd/008:14:590
81460,0rcu_preempt3-21ksoftirqd/007:19:480
81460,0rcu_preempt3-21ksoftirqd/007:04:370
81460,0rcu_preempt21-21ksoftirqd/112:25:021
81460,0rcu_preempt21-21ksoftirqd/112:04:571
81460,0rcu_preempt21-21ksoftirqd/111:49:471
81460,0rcu_preempt21-21ksoftirqd/111:39:571
81460,0rcu_preempt21-21ksoftirqd/111:30:001
81460,0rcu_preempt21-21ksoftirqd/111:24:511
81460,0rcu_preempt21-21ksoftirqd/111:10:001
81460,0rcu_preempt21-21ksoftirqd/111:00:001
81460,0rcu_preempt21-21ksoftirqd/108:55:041
81460,0rcu_preempt21-21ksoftirqd/108:49:521
81460,0rcu_preempt21-21ksoftirqd/108:49:521
81460,0rcu_preempt21-21ksoftirqd/108:24:381
81460,0rcu_preempt21-21ksoftirqd/108:09:511
2203599469,0cyclictest7464-21switchtime07:45:043
22035994623,0cyclictest37-21ksoftirqd/311:51:333
22035994617,0cyclictest25679-21sshd07:11:153
22035994617,0cyclictest25679-21sshd07:11:153
22035994616,0cyclictest15629-21unixbench_singl10:40:063
22035994615,0cyclictest29306-21crond12:34:373
81450,0rcu_preempt9008-21crond09:14:322
81450,0rcu_preempt37-21ksoftirqd/311:45:153
81450,0rcu_preempt37-21ksoftirqd/310:44:383
81450,0rcu_preempt37-21ksoftirqd/307:59:503
81450,0rcu_preempt3-21ksoftirqd/011:34:460
81450,0rcu_preempt3-21ksoftirqd/010:56:430
81450,0rcu_preempt3-21ksoftirqd/008:44:480
81450,0rcu_preempt3-21ksoftirqd/008:24:530
81450,0rcu_preempt3-21ksoftirqd/008:04:370
81450,0rcu_preempt3-21ksoftirqd/007:55:010
81450,0rcu_preempt3-21ksoftirqd/007:49:500
81450,0rcu_preempt3-21ksoftirqd/007:41:000
81450,0rcu_preempt21-21ksoftirqd/112:17:461
81450,0rcu_preempt21-21ksoftirqd/112:17:461
81450,0rcu_preempt21-21ksoftirqd/111:19:371
81450,0rcu_preempt21-21ksoftirqd/111:04:501
81450,0rcu_preempt21-21ksoftirqd/110:34:491
81450,0rcu_preempt21-21ksoftirqd/110:34:491
81450,0rcu_preempt21-21ksoftirqd/110:05:031
81450,0rcu_preempt21-21ksoftirqd/109:49:481
81450,0rcu_preempt21-21ksoftirqd/109:35:021
81450,0rcu_preempt21-21ksoftirqd/109:24:571
81450,0rcu_preempt21-21ksoftirqd/109:14:501
81450,0rcu_preempt21-21ksoftirqd/109:10:061
2203599458,0cyclictest0-21swapper/307:39:503
2203599452,0cyclictest8581-21ls10:24:533
2203599452,0cyclictest1317-21runrttasks07:50:043
22035994519,0cyclictest21787-21sed12:14:543
22035994519,0cyclictest21787-21sed12:14:543
22035994518,0cyclictest32155-21proc_pri08:45:023
22035994518,0cyclictest17241-21cut12:04:483
22035994517,0cyclictest10218-21cpuspeed_turbos10:29:483
22035994516,0cyclictest8604-21crond11:44:373
22035994516,0cyclictest22756-21fschecks_count10:59:503
22035994511,0cyclictest1337-21lldpd07:34:103
2203599451,0cyclictest26802-21unixbench_multi12:25:143
22031994534,0cyclictest29-21ksoftirqd/211:49:502
91440,0rcu_sched3659-21systemd-cgroups10:14:370
91440,0rcu_sched23517-21systemd-cgroups09:44:372
81440,0rcu_preempt37-21ksoftirqd/311:05:033
81440,0rcu_preempt37-21ksoftirqd/309:00:053
81440,0rcu_preempt37-21ksoftirqd/308:14:523
81440,0rcu_preempt3-21ksoftirqd/010:19:510
81440,0rcu_preempt3-21ksoftirqd/008:39:480
81440,0rcu_preempt29-21ksoftirqd/207:04:372
81440,0rcu_preempt21-21ksoftirqd/112:00:541
81440,0rcu_preempt21-21ksoftirqd/107:34:381
81440,0rcu_preempt21-21ksoftirqd/107:10:071
81440,0rcu_preempt21-21ksoftirqd/107:10:071
22035994417,0cyclictest5228-21gltestperf11:34:533
22035994415,0cyclictest2544-21fschecks_time07:34:503
22035994414,0cyclictest7306-21grep10:20:063
22031994432,0cyclictest29-21ksoftirqd/210:34:522
22031994432,0cyclictest29-21ksoftirqd/210:34:522
22025994439,0cyclictest0-21swapper/107:04:381
91430,0rcu_sched26780-21crond08:34:372
81430,0rcu_preempt37-21ksoftirqd/311:19:503
81430,0rcu_preempt37-21ksoftirqd/310:49:483
81430,0rcu_preempt21-21ksoftirqd/108:34:381
81430,0rcu_preempt21-21ksoftirqd/107:24:381
22031994337,0cyclictest29-21ksoftirqd/209:30:002
91420,0rcu_sched29-21ksoftirqd/210:44:372
81420,0rcu_preempt37-21ksoftirqd/310:14:533
81420,0rcu_preempt3-21ksoftirqd/007:14:550
81420,0rcu_preempt3-21ksoftirqd/007:14:550
81420,0rcu_preempt21-21ksoftirqd/108:04:551
22031994237,0cyclictest29-21ksoftirqd/212:14:532
22031994237,0cyclictest29-21ksoftirqd/212:14:532
22031994136,0cyclictest29-21ksoftirqd/208:40:032
22031994136,0cyclictest29-21ksoftirqd/208:19:522
22031994135,0cyclictest29-21ksoftirqd/207:50:062
22031994122,0cyclictest29-21ksoftirqd/209:49:552
22031994035,0cyclictest29-21ksoftirqd/212:24:542
22031994035,0cyclictest29-21ksoftirqd/211:09:562
22031994035,0cyclictest29-21ksoftirqd/210:29:512
22031994035,0cyclictest29-21ksoftirqd/208:44:582
22031994033,0cyclictest29-21ksoftirqd/210:24:372
22031994027,0cyclictest29-21ksoftirqd/211:55:002
22031993934,0cyclictest29-21ksoftirqd/211:59:512
22031993934,0cyclictest29-21ksoftirqd/211:14:512
22031993934,0cyclictest29-21ksoftirqd/209:20:072
22031993934,0cyclictest29-21ksoftirqd/207:59:502
22031993933,0cyclictest29-21ksoftirqd/212:04:512
22031993933,0cyclictest29-21ksoftirqd/210:54:512
22031993933,0cyclictest29-21ksoftirqd/208:49:502
22031993933,0cyclictest29-21ksoftirqd/208:49:502
22031993933,0cyclictest29-21ksoftirqd/208:34:522
22031993926,0cyclictest29-21ksoftirqd/211:40:152
22031993926,0cyclictest29-21ksoftirqd/209:54:512
22031993925,0cyclictest29-21ksoftirqd/207:09:482
22031993925,0cyclictest29-21ksoftirqd/207:09:482
22031993923,0cyclictest29-21ksoftirqd/207:34:592
22031993834,0cyclictest29-21ksoftirqd/211:30:032
22031993833,0cyclictest29-21ksoftirqd/211:45:022
22031993833,0cyclictest29-21ksoftirqd/210:49:552
22031993833,0cyclictest29-21ksoftirqd/209:45:022
22031993833,0cyclictest29-21ksoftirqd/209:45:022
22031993832,0cyclictest29-21ksoftirqd/210:09:532
22031993832,0cyclictest29-21ksoftirqd/209:25:012
22031993832,0cyclictest29-21ksoftirqd/207:15:092
22031993832,0cyclictest29-21ksoftirqd/207:15:092
22031993825,0cyclictest29-21ksoftirqd/208:25:472
22031993823,0cyclictest29-21ksoftirqd/212:32:312
22031993733,0cyclictest29-21ksoftirqd/211:25:022
22031993731,0cyclictest29-21ksoftirqd/211:35:012
22031993731,0cyclictest29-21ksoftirqd/210:44:482
22031993731,0cyclictest29-21ksoftirqd/209:04:552
22031993731,0cyclictest29-21ksoftirqd/207:29:522
22031993731,0cyclictest29-21ksoftirqd/207:25:062
22031993730,0cyclictest29-21ksoftirqd/212:14:372
22031993722,0cyclictest29-21ksoftirqd/210:59:532
22031993631,0cyclictest29-21ksoftirqd/210:14:492
22031993631,0cyclictest29-21ksoftirqd/209:59:482
22031993631,0cyclictest29-21ksoftirqd/209:35:052
22031993631,0cyclictest29-21ksoftirqd/208:14:572
22031993631,0cyclictest29-21ksoftirqd/207:55:032
22031993631,0cyclictest29-21ksoftirqd/207:20:012
22031993631,0cyclictest271rcuc/210:04:492
22031993630,0cyclictest29-21ksoftirqd/211:19:492
22031993630,0cyclictest29-21ksoftirqd/208:05:062
22031993630,0cyclictest29-21ksoftirqd/207:44:382
22031993630,0cyclictest29-21ksoftirqd/207:39:522
22031993629,0cyclictest29-21ksoftirqd/211:05:042
22031993531,0cyclictest29-21ksoftirqd/209:14:382
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional