You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-17 - 05:38

Intel(R) Atom(TM) CPU D2700 @ 2.13GHz, Linux 4.4.39-rt50 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot6.osadl.org (updated Sun Oct 17, 2021 00:43:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
81540,0rcu_preempt21-21ksoftirqd/123:39:101
81540,0rcu_preempt21-21ksoftirqd/123:39:101
24314995436,0cyclictest37-21ksoftirqd/300:28:403
81530,0rcu_preempt29-21ksoftirqd/223:59:092
81530,0rcu_preempt29-21ksoftirqd/221:48:592
81530,0rcu_preempt29-21ksoftirqd/221:48:592
81530,0rcu_preempt29-21ksoftirqd/221:24:052
81530,0rcu_preempt29-21ksoftirqd/219:08:412
81530,0rcu_preempt21-21ksoftirqd/119:48:581
81530,0rcu_preempt21-21ksoftirqd/119:19:031
1899530,0migration/123562-21timerwakeupswit19:04:101
81520,0rcu_preempt29-21ksoftirqd/221:09:012
81520,0rcu_preempt29-21ksoftirqd/220:03:552
81520,0rcu_preempt29-21ksoftirqd/200:29:122
81520,0rcu_preempt21-21ksoftirqd/123:04:271
81520,0rcu_preempt21-21ksoftirqd/121:49:121
81520,0rcu_preempt21-21ksoftirqd/121:49:121
81510,0rcu_preempt29-21ksoftirqd/222:24:112
81510,0rcu_preempt29-21ksoftirqd/222:19:112
81510,0rcu_preempt21-21ksoftirqd/123:09:151
81510,0rcu_preempt21-21ksoftirqd/121:13:591
81510,0rcu_preempt21-21ksoftirqd/119:59:081
2699510,0migration/223113-21needreboot19:04:042
81500,0rcu_preempt29-21ksoftirqd/223:44:032
81500,0rcu_preempt29-21ksoftirqd/223:09:212
81500,0rcu_preempt29-21ksoftirqd/222:56:372
81500,0rcu_preempt29-21ksoftirqd/221:04:012
81500,0rcu_preempt29-21ksoftirqd/200:23:402
81500,0rcu_preempt29-21ksoftirqd/200:18:552
81500,0rcu_preempt29-21ksoftirqd/200:09:222
81500,0rcu_preempt21-21ksoftirqd/123:49:131
81500,0rcu_preempt21-21ksoftirqd/122:58:591
81500,0rcu_preempt21-21ksoftirqd/122:49:071
81500,0rcu_preempt21-21ksoftirqd/121:24:111
81500,0rcu_preempt21-21ksoftirqd/120:24:111
81500,0rcu_preempt21-21ksoftirqd/120:03:551
81500,0rcu_preempt21-21ksoftirqd/100:28:561
81500,0rcu_preempt21-21ksoftirqd/100:18:561
81500,0rcu_preempt21-21ksoftirqd/100:04:041
81490,0rcu_preempt29-21ksoftirqd/223:23:572
81490,0rcu_preempt29-21ksoftirqd/222:34:002
81490,0rcu_preempt29-21ksoftirqd/222:18:412
81490,0rcu_preempt29-21ksoftirqd/221:59:142
81490,0rcu_preempt29-21ksoftirqd/221:29:102
81490,0rcu_preempt29-21ksoftirqd/220:59:402
81490,0rcu_preempt29-21ksoftirqd/220:24:112
81490,0rcu_preempt29-21ksoftirqd/219:58:562
81490,0rcu_preempt29-21ksoftirqd/219:28:572
81490,0rcu_preempt21-21ksoftirqd/123:43:411
81490,0rcu_preempt21-21ksoftirqd/123:13:571
81490,0rcu_preempt21-21ksoftirqd/122:13:581
81490,0rcu_preempt21-21ksoftirqd/122:04:061
81490,0rcu_preempt21-21ksoftirqd/121:33:271
81490,0rcu_preempt21-21ksoftirqd/121:18:581
81490,0rcu_preempt21-21ksoftirqd/121:04:131
81490,0rcu_preempt21-21ksoftirqd/119:14:121
81490,0rcu_preempt21-21ksoftirqd/100:24:111
105750490,0irq/26-eth0-rx-0-21swapper/019:03:580
81480,0rcu_preempt29-21ksoftirqd/223:54:102
81480,0rcu_preempt29-21ksoftirqd/223:48:412
81480,0rcu_preempt29-21ksoftirqd/222:43:402
81480,0rcu_preempt29-21ksoftirqd/222:28:572
81480,0rcu_preempt29-21ksoftirqd/222:08:582
81480,0rcu_preempt29-21ksoftirqd/222:04:002
81480,0rcu_preempt29-21ksoftirqd/220:29:112
81480,0rcu_preempt29-21ksoftirqd/219:38:412
81480,0rcu_preempt29-21ksoftirqd/219:14:172
81480,0rcu_preempt21-21ksoftirqd/122:41:481
81480,0rcu_preempt21-21ksoftirqd/122:09:091
81480,0rcu_preempt21-21ksoftirqd/121:33:531
81480,0rcu_preempt21-21ksoftirqd/120:45:081
81480,0rcu_preempt21-21ksoftirqd/120:19:461
81480,0rcu_preempt21-21ksoftirqd/119:57:151
81480,0rcu_preempt21-21ksoftirqd/119:57:151
81480,0rcu_preempt21-21ksoftirqd/119:08:591
81480,0rcu_preempt21-21ksoftirqd/100:08:411
81470,0rcu_preempt37-21ksoftirqd/319:04:013
81470,0rcu_preempt29-21ksoftirqd/223:34:102
81470,0rcu_preempt29-21ksoftirqd/223:19:112
81470,0rcu_preempt29-21ksoftirqd/223:03:412
81470,0rcu_preempt29-21ksoftirqd/222:49:052
81470,0rcu_preempt29-21ksoftirqd/221:39:112
81470,0rcu_preempt29-21ksoftirqd/221:33:542
81470,0rcu_preempt29-21ksoftirqd/220:39:092
81470,0rcu_preempt29-21ksoftirqd/220:33:562
81470,0rcu_preempt29-21ksoftirqd/220:18:552
81470,0rcu_preempt29-21ksoftirqd/219:54:072
81470,0rcu_preempt29-21ksoftirqd/219:54:072
81470,0rcu_preempt29-21ksoftirqd/219:24:092
81470,0rcu_preempt29-21ksoftirqd/200:14:052
81470,0rcu_preempt21-21ksoftirqd/122:53:581
81470,0rcu_preempt21-21ksoftirqd/122:38:361
81470,0rcu_preempt21-21ksoftirqd/120:53:081
81470,0rcu_preempt21-21ksoftirqd/119:38:561
81470,0rcu_preempt21-21ksoftirqd/100:13:591
3399470,0watchdog/324287-21cyclictest19:08:413
81460,0rcu_preempt29-21ksoftirqd/223:29:072
81460,0rcu_preempt29-21ksoftirqd/221:14:092
81460,0rcu_preempt29-21ksoftirqd/220:48:552
81460,0rcu_preempt29-21ksoftirqd/219:48:412
81460,0rcu_preempt21-21ksoftirqd/123:33:401
81460,0rcu_preempt21-21ksoftirqd/123:19:041
81460,0rcu_preempt21-21ksoftirqd/122:44:111
81460,0rcu_preempt21-21ksoftirqd/122:44:111
81460,0rcu_preempt21-21ksoftirqd/122:18:581
81460,0rcu_preempt21-21ksoftirqd/121:58:411
81460,0rcu_preempt21-21ksoftirqd/121:39:101
81460,0rcu_preempt21-21ksoftirqd/120:59:041
81460,0rcu_preempt21-21ksoftirqd/120:34:121
1599460,0migration/013121-21fschecks_time19:58:580
81450,0rcu_preempt29-21ksoftirqd/222:44:122
81450,0rcu_preempt29-21ksoftirqd/222:44:122
81450,0rcu_preempt29-21ksoftirqd/221:53:592
81450,0rcu_preempt29-21ksoftirqd/221:18:562
81450,0rcu_preempt29-21ksoftirqd/220:54:002
81450,0rcu_preempt29-21ksoftirqd/220:43:592
81450,0rcu_preempt29-21ksoftirqd/220:13:572
81450,0rcu_preempt29-21ksoftirqd/220:09:112
81450,0rcu_preempt29-21ksoftirqd/219:51:162
81450,0rcu_preempt29-21ksoftirqd/219:19:122
81450,0rcu_preempt29-21ksoftirqd/200:33:542
81450,0rcu_preempt29-21ksoftirqd/200:07:422
81450,0rcu_preempt21-21ksoftirqd/123:59:501
81450,0rcu_preempt21-21ksoftirqd/123:57:371
81450,0rcu_preempt21-21ksoftirqd/123:34:411
81450,0rcu_preempt21-21ksoftirqd/122:24:051
81450,0rcu_preempt21-21ksoftirqd/121:09:111
81450,0rcu_preempt21-21ksoftirqd/120:13:531
81450,0rcu_preempt21-21ksoftirqd/119:23:411
24298994533,0cyclictest3-21ksoftirqd/022:03:550
81440,0rcu_preempt37-21ksoftirqd/323:18:413
81440,0rcu_preempt3-21ksoftirqd/019:28:140
81440,0rcu_preempt29-21ksoftirqd/223:42:252
81440,0rcu_preempt29-21ksoftirqd/223:42:252
81440,0rcu_preempt29-21ksoftirqd/223:13:552
81440,0rcu_preempt29-21ksoftirqd/219:39:042
81440,0rcu_preempt21-21ksoftirqd/122:28:591
81440,0rcu_preempt21-21ksoftirqd/121:53:541
81440,0rcu_preempt21-21ksoftirqd/121:43:591
81440,0rcu_preempt21-21ksoftirqd/119:44:071
81440,0rcu_preempt21-21ksoftirqd/119:34:121
81440,0rcu_preempt21-21ksoftirqd/100:38:401
81430,0rcu_preempt37-21ksoftirqd/322:54:073
81430,0rcu_preempt37-21ksoftirqd/320:53:553
81430,0rcu_preempt3-21ksoftirqd/020:18:410
81430,0rcu_preempt29-21ksoftirqd/221:44:022
81430,0rcu_preempt21-21ksoftirqd/120:54:151
81430,0rcu_preempt21-21ksoftirqd/120:38:591
81430,0rcu_preempt21-21ksoftirqd/120:08:411
81430,0rcu_preempt21-21ksoftirqd/119:31:411
24314994338,0cyclictest6414-21swap20:59:093
24314994334,0cyclictest37-21ksoftirqd/319:18:413
81420,0rcu_preempt37-21ksoftirqd/323:34:073
81420,0rcu_preempt37-21ksoftirqd/321:09:173
81420,0rcu_preempt37-21ksoftirqd/320:19:073
81420,0rcu_preempt37-21ksoftirqd/319:38:163
81420,0rcu_preempt37-21ksoftirqd/300:38:403
81420,0rcu_preempt37-21ksoftirqd/300:18:413
81420,0rcu_preempt29-21ksoftirqd/222:59:122
81420,0rcu_preempt21-21ksoftirqd/120:28:571
1599420,0migration/028055-21latency_hist21:53:410
81410,0rcu_preempt37-21ksoftirqd/323:55:283
81410,0rcu_preempt37-21ksoftirqd/322:44:173
81410,0rcu_preempt37-21ksoftirqd/322:44:173
81410,0rcu_preempt37-21ksoftirqd/321:43:403
81410,0rcu_preempt37-21ksoftirqd/320:16:133
81410,0rcu_preempt37-21ksoftirqd/319:23:553
81410,0rcu_preempt3-21ksoftirqd/022:14:120
81410,0rcu_preempt3-21ksoftirqd/021:28:420
81410,0rcu_preempt21-21ksoftirqd/123:23:581
24314994136,0cyclictest27321-21gltestperf23:09:053
24314994135,0cyclictest10444-21tail23:44:163
24298994135,0cyclictest13944-21fschecks_time21:18:580
24298994128,0cyclictest3-21ksoftirqd/021:39:050
24298994127,0cyclictest3-21ksoftirqd/022:38:400
1599410,0migration/025532-21fschecks_time20:28:580
81400,0rcu_preempt37-21ksoftirqd/323:14:063
81400,0rcu_preempt37-21ksoftirqd/322:26:133
81400,0rcu_preempt37-21ksoftirqd/322:23:413
81400,0rcu_preempt37-21ksoftirqd/322:07:323
81400,0rcu_preempt37-21ksoftirqd/321:28:573
81400,0rcu_preempt37-21ksoftirqd/321:24:063
81400,0rcu_preempt37-21ksoftirqd/320:39:133
81400,0rcu_preempt37-21ksoftirqd/320:06:283
81400,0rcu_preempt37-21ksoftirqd/319:41:593
81400,0rcu_preempt37-21ksoftirqd/319:31:473
81400,0rcu_preempt3-21ksoftirqd/023:29:000
81400,0rcu_preempt3-21ksoftirqd/023:24:000
81400,0rcu_preempt3-21ksoftirqd/023:19:150
81400,0rcu_preempt3-21ksoftirqd/023:13:540
81400,0rcu_preempt3-21ksoftirqd/023:09:110
81400,0rcu_preempt3-21ksoftirqd/021:34:420
81400,0rcu_preempt3-21ksoftirqd/020:49:070
81400,0rcu_preempt3-21ksoftirqd/019:39:040
81400,0rcu_preempt3-21ksoftirqd/019:38:030
81400,0rcu_preempt3-21ksoftirqd/019:29:140
81400,0rcu_preempt3-21ksoftirqd/019:19:110
81400,0rcu_preempt3-21ksoftirqd/000:33:550
3499400,0migration/314119-21gltestperf21:18:593
24314994034,0cyclictest37-21ksoftirqd/320:28:583
24298994034,0cyclictest12400-21needreboot21:14:080
81390,0rcu_preempt37-21ksoftirqd/321:53:573
81390,0rcu_preempt37-21ksoftirqd/320:49:103
81390,0rcu_preempt37-21ksoftirqd/320:43:543
81390,0rcu_preempt37-21ksoftirqd/320:23:573
81390,0rcu_preempt37-21ksoftirqd/320:01:393
81390,0rcu_preempt37-21ksoftirqd/319:53:413
81390,0rcu_preempt37-21ksoftirqd/319:44:133
81390,0rcu_preempt37-21ksoftirqd/300:11:413
81390,0rcu_preempt3-21ksoftirqd/022:53:570
81390,0rcu_preempt3-21ksoftirqd/022:39:160
81390,0rcu_preempt3-21ksoftirqd/022:26:560
81390,0rcu_preempt3-21ksoftirqd/022:19:190
81390,0rcu_preempt3-21ksoftirqd/022:09:400
81390,0rcu_preempt3-21ksoftirqd/021:48:590
81390,0rcu_preempt3-21ksoftirqd/021:48:590
81390,0rcu_preempt3-21ksoftirqd/021:09:020
81390,0rcu_preempt3-21ksoftirqd/000:29:100
24314993934,0cyclictest37-21ksoftirqd/323:48:413
24314993934,0cyclictest37-21ksoftirqd/321:59:113
24314993933,0cyclictest12626-21fschecks_count22:34:013
24314993928,0cyclictest37-21ksoftirqd/322:09:193
24298993924,0cyclictest3-21ksoftirqd/019:54:160
24298993924,0cyclictest3-21ksoftirqd/019:54:160
24298993923,0cyclictest3-21ksoftirqd/022:43:470
24298993923,0cyclictest3-21ksoftirqd/022:43:470
24298993922,0cyclictest3-21ksoftirqd/021:24:140
1599390,0migration/025591-21missed_timers23:04:170
81380,0rcu_preempt37-21ksoftirqd/322:49:083
81380,0rcu_preempt37-21ksoftirqd/322:38:583
81380,0rcu_preempt37-21ksoftirqd/321:43:413
81380,0rcu_preempt37-21ksoftirqd/321:34:523
81380,0rcu_preempt37-21ksoftirqd/300:19:013
81380,0rcu_preempt37-21ksoftirqd/300:06:303
81380,0rcu_preempt3-21ksoftirqd/023:49:000
81380,0rcu_preempt3-21ksoftirqd/022:33:400
81380,0rcu_preempt3-21ksoftirqd/021:43:410
81380,0rcu_preempt3-21ksoftirqd/020:43:550
81380,0rcu_preempt3-21ksoftirqd/020:19:150
81380,0rcu_preempt3-21ksoftirqd/019:49:070
81380,0rcu_preempt3-21ksoftirqd/019:43:410
81380,0rcu_preempt3-21ksoftirqd/019:14:120
81380,0rcu_preempt3-21ksoftirqd/000:24:010
81380,0rcu_preempt3-21ksoftirqd/000:10:140
81380,0rcu_preempt3-21ksoftirqd/000:04:160
24314993833,0cyclictest37-21ksoftirqd/322:58:593
24314993833,0cyclictest37-21ksoftirqd/320:09:053
24314993833,0cyclictest37-21ksoftirqd/319:54:083
24314993833,0cyclictest37-21ksoftirqd/319:54:083
24314993832,0cyclictest37-21ksoftirqd/323:39:043
24314993832,0cyclictest37-21ksoftirqd/323:39:043
24314993825,0cyclictest37-21ksoftirqd/323:29:083
105750380,0irq/26-eth0-rx-18958-21runrttasks22:48:570
81370,0rcu_preempt37-21ksoftirqd/323:03:583
81370,0rcu_preempt37-21ksoftirqd/321:49:583
81370,0rcu_preempt37-21ksoftirqd/321:49:583
81370,0rcu_preempt37-21ksoftirqd/321:14:103
81370,0rcu_preempt3-21ksoftirqd/023:54:070
81370,0rcu_preempt3-21ksoftirqd/022:58:540
81370,0rcu_preempt3-21ksoftirqd/021:59:420
81370,0rcu_preempt3-21ksoftirqd/020:58:530
81370,0rcu_preempt3-21ksoftirqd/020:54:070
81370,0rcu_preempt3-21ksoftirqd/000:19:110
81370,0rcu_preempt3-21ksoftirqd/000:13:410
24314993732,0cyclictest37-21ksoftirqd/323:59:003
24314993732,0cyclictest37-21ksoftirqd/320:38:403
24314993732,0cyclictest37-21ksoftirqd/319:19:033
24314993732,0cyclictest37-21ksoftirqd/300:29:053
24298993732,0cyclictest3-21ksoftirqd/019:08:410
1699370,0watchdog/01252-21runrttasks20:27:050
1599370,0migration/028076-21missed_timers20:34:050
81360,0rcu_preempt37-21ksoftirqd/322:28:533
81360,0rcu_preempt37-21ksoftirqd/322:14:073
81360,0rcu_preempt3-21ksoftirqd/023:58:560
81360,0rcu_preempt3-21ksoftirqd/023:46:270
81360,0rcu_preempt3-21ksoftirqd/020:08:580
81360,0rcu_preempt3-21ksoftirqd/020:04:130
24314993626,0cyclictest37-21ksoftirqd/321:04:083
24298993631,0cyclictest7938-21wget21:04:000
81350,0rcu_preempt3-21ksoftirqd/023:35:040
81350,0rcu_preempt3-21ksoftirqd/020:38:570
24314993530,0cyclictest37-21ksoftirqd/323:23:413
81340,0rcu_preempt3-21ksoftirqd/023:39:030
81340,0rcu_preempt3-21ksoftirqd/023:39:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional