You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-03 - 07:28

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot6.osadl.org (updated Sun Jul 03, 2022 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2911400,0irq_work/2311rcuc/223:25:072
2911400,0irq_work/2311rcuc/221:15:082
2911310,0irq_work/2311rcuc/219:50:092
2911250,0irq_work/2311rcuc/220:55:172
2911240,0irq_work/2311rcuc/221:35:522
2911220,0irq_work/2311rcuc/200:15:232
2911210,0irq_work/2311rcuc/220:34:542
2911200,0irq_work/2311rcuc/219:55:162
2911200,0irq_work/2311rcuc/219:55:162
2911140,0irq_work/2311rcuc/219:20:042
2911130,0irq_work/2311rcuc/219:13:072
2911110,0irq_work/2311rcuc/223:30:432
2911100,0irq_work/2311rcuc/223:10:092
2911090,0irq_work/2311rcuc/222:37:322
2911090,0irq_work/2311rcuc/220:35:522
2911070,0irq_work/2311rcuc/219:16:222
2911060,0irq_work/2311rcuc/223:44:532
2911060,0irq_work/2311rcuc/220:04:532
2911060,0irq_work/2311rcuc/220:04:532
2911050,0irq_work/2311rcuc/200:09:532
2911040,0irq_work/2311rcuc/222:20:142
2911040,0irq_work/2311rcuc/221:55:072
2911040,0irq_work/2311rcuc/219:46:292
2911030,0irq_work/2311rcuc/222:50:172
2911030,0irq_work/2311rcuc/222:34:262
2911030,0irq_work/2311rcuc/222:25:202
2911030,0irq_work/2311rcuc/221:05:142
2911030,0irq_work/2311rcuc/221:05:142
2911020,0irq_work/2311rcuc/223:52:542
2911010,0irq_work/2311rcuc/221:29:362
2911010,0irq_work/2311rcuc/220:50:222
2911010,0irq_work/2311rcuc/220:15:182
2911010,0irq_work/2311rcuc/219:40:212
2911000,0irq_work/2311rcuc/220:05:522
291990,0irq_work/2311rcuc/222:06:282
291990,0irq_work/2311rcuc/221:24:542
291960,0irq_work/2311rcuc/223:09:302
291960,0irq_work/2311rcuc/223:00:092
291960,0irq_work/2311rcuc/221:43:452
291960,0irq_work/2311rcuc/219:39:492
291950,0irq_work/2311rcuc/222:59:512
291950,0irq_work/2311rcuc/221:47:482
291950,0irq_work/2311rcuc/220:40:072
291950,0irq_work/2311rcuc/219:25:192
291950,0irq_work/2311rcuc/200:29:532
291940,0irq_work/2311rcuc/222:15:252
291940,0irq_work/2311rcuc/221:00:172
291940,0irq_work/2311rcuc/220:49:282
291940,0irq_work/2311rcuc/220:13:032
291930,0irq_work/2311rcuc/221:30:182
291930,0irq_work/2311rcuc/220:23:072
291920,0irq_work/2311rcuc/200:33:242
291920,0irq_work/2311rcuc/200:14:532
291910,0irq_work/2311rcuc/223:59:542
291910,0irq_work/2311rcuc/223:20:072
291910,0irq_work/2311rcuc/223:19:092
291910,0irq_work/2311rcuc/222:45:142
291910,0irq_work/2311rcuc/222:00:182
291900,0irq_work/2311rcuc/222:10:072
291900,0irq_work/2311rcuc/222:10:072
291900,0irq_work/2311rcuc/219:30:202
291890,0irq_work/2311rcuc/223:36:552
291890,0irq_work/2311rcuc/221:50:582
291880,0irq_work/2311rcuc/200:35:092
291870,0irq_work/2311rcuc/222:41:202
291860,0irq_work/2311rcuc/223:45:222
291860,0irq_work/2311rcuc/200:21:542
291860,0irq_work/2311rcuc/200:21:542
291860,0irq_work/2311rcuc/200:02:472
291840,0irq_work/2311rcuc/220:27:102
361820,0irq_work/3381rcuc/320:32:103
291780,0irq_work/2311rcuc/221:10:212
103350770,0irq/28-eth0-tx-7650irq/26-ahci[00021:47:100
221720,0irq_work/1241rcuc/120:29:541
181720,0irq_work/0161rcuc/019:20:080
221700,0irq_work/1241rcuc/122:29:481
221700,0irq_work/1241rcuc/120:54:531
361660,0irq_work/3381rcuc/320:20:213
221660,0irq_work/1241rcuc/121:15:181
221650,0irq_work/1241rcuc/123:15:181
221650,0irq_work/1241rcuc/100:34:481
181650,0irq_work/0161rcuc/019:41:150
221630,0irq_work/1241rcuc/123:30:171
141630,0rcu_preempt22415-21timerandwakeup23:20:210
381620,0rcuc/312134-21grep00:15:093
221620,0irq_work/1241rcuc/123:50:181
181620,0irq_work/0161rcuc/023:46:220
221610,0irq_work/1241rcuc/123:00:181
221610,0irq_work/1241rcuc/122:15:081
221600,0irq_work/1241rcuc/100:09:531
103350600,0irq/28-eth0-tx-15319-21unixbench_multi19:05:200
361590,0irq_work/3381rcuc/323:30:233
181590,0irq_work/0161rcuc/020:27:580
1601399594,0cyclictest1799migration/019:30:180
181580,0irq_work/0161rcuc/022:24:270
181580,0irq_work/0161rcuc/019:58:460
181580,0irq_work/0161rcuc/019:58:460
141580,0rcu_preempt0-21swapper/119:06:011
221570,0irq_work/1241rcuc/121:40:181
361560,0irq_work/3381rcuc/300:24:543
361550,0irq_work/3381rcuc/300:39:093
291540,0irq_work/2311rcuc/219:05:202
221540,0irq_work/1241rcuc/123:39:531
181540,0irq_work/0161rcuc/023:25:250
181540,0irq_work/0161rcuc/022:45:090
181540,0irq_work/0161rcuc/021:28:500
221530,0irq_work/1241rcuc/120:35:181
181530,0irq_work/0161rcuc/020:00:140
181530,0irq_work/0161rcuc/020:00:140
181530,0irq_work/0161rcuc/019:27:340
221520,0irq_work/1241rcuc/120:34:491
181520,0irq_work/0161rcuc/021:21:420
181520,0irq_work/0161rcuc/020:32:250
181520,0irq_work/0161rcuc/019:53:390
16016995236,0cyclictest10948-21grep22:55:051
141520,0rcu_preempt5391-21wc20:00:183
141520,0rcu_preempt5391-21wc20:00:183
221510,0irq_work/1241rcuc/100:39:531
181510,0irq_work/0161rcuc/020:10:500
361500,0irq_work/3381rcuc/321:33:093
361500,0irq_work/3381rcuc/321:00:233
221500,0irq_work/1241rcuc/123:49:481
221500,0irq_work/1241rcuc/121:50:171
221500,0irq_work/1241rcuc/121:39:541
381490,0rcuc/3141rcu_preempt19:35:143
361490,0irq_work/3381rcuc/322:50:113
361490,0irq_work/3381rcuc/322:24:593
361490,0irq_work/3381rcuc/319:59:273
361490,0irq_work/3381rcuc/319:59:273
181490,0irq_work/0161rcuc/020:24:320
16013994912,0cyclictest210099rtkit-daemon19:38:400
141490,0rcu_preempt25-21ksoftirqd/120:20:221
221480,0irq_work/1241rcuc/120:44:491
221480,0irq_work/1241rcuc/119:35:141
181480,0irq_work/0161rcuc/023:19:120
181480,0irq_work/0161rcuc/022:33:050
181480,0irq_work/0161rcuc/022:29:270
181480,0irq_work/0161rcuc/020:53:430
181480,0irq_work/0161rcuc/019:45:240
16016994831,0cyclictest16491-21grep19:10:081
141480,0rcu_preempt39-21ksoftirqd/320:29:543
141480,0rcu_preempt20411-21gltestperf00:35:090
103250480,0irq/27-eth0-rx-14794-21ls19:05:123
221470,0irq_work/1241rcuc/100:24:531
221470,0irq_work/1241rcuc/100:24:531
181470,0irq_work/0161rcuc/023:30:480
181470,0irq_work/0161rcuc/020:45:340
181470,0irq_work/0161rcuc/000:05:030
141470,0rcu_preempt25-21ksoftirqd/122:20:201
141470,0rcu_preempt13-21ksoftirqd/022:35:040
361460,0irq_work/3381rcuc/321:50:063
181460,0irq_work/0161rcuc/023:56:040
181460,0irq_work/0161rcuc/023:00:180
181460,0irq_work/0161rcuc/022:12:390
181460,0irq_work/0161rcuc/022:12:390
181460,0irq_work/0161rcuc/022:06:190
181460,0irq_work/0161rcuc/020:57:460
181460,0irq_work/0161rcuc/000:23:510
181460,0irq_work/0161rcuc/000:23:510
141460,0rcu_preempt13235-21seq00:17:020
103250460,0irq/27-eth0-rx-10974-21netstat21:35:163
221450,0irq_work/1241rcuc/121:05:141
221450,0irq_work/1241rcuc/121:05:141
221450,0irq_work/1241rcuc/119:15:181
181450,0irq_work/0161rcuc/022:15:090
181450,0irq_work/0161rcuc/020:44:410
1601899457,0cyclictest3799migration/323:50:203
16016994526,0cyclictest619-21grep22:30:071
16016994526,0cyclictest4719-21cut20:00:081
16016994526,0cyclictest4719-21cut20:00:081
141450,0rcu_preempt28403-21timerwakeupswit20:55:233
141450,0rcu_preempt18880-21missed_timers19:15:133
141450,0rcu_preempt13-21ksoftirqd/000:00:220
361440,0irq_work/3381rcuc/320:45:083
361440,0irq_work/3381rcuc/319:34:543
221440,0irq_work/1241rcuc/100:25:201
181440,0irq_work/0161rcuc/022:58:540
141440,0rcu_preempt39-21ksoftirqd/322:44:543
141440,0rcu_preempt25-21ksoftirqd/120:44:541
141440,0rcu_preempt25-21ksoftirqd/119:40:261
141440,0rcu_preempt13-21ksoftirqd/000:25:240
361430,0irq_work/3381rcuc/322:45:113
361430,0irq_work/3381rcuc/321:15:043
181430,0irq_work/0161rcuc/021:31:320
16018994324,0cyclictest22684-21grep19:25:083
1601899431,0cyclictest210099rtkit-daemon19:13:063
1601699437,0cyclictest2399migration/123:57:251
1601699436,0cyclictest2399migration/120:12:561
1601699434,0cyclictest2399migration/121:10:131
141430,0rcu_preempt39-21ksoftirqd/320:05:123
141430,0rcu_preempt30697-21date21:00:530
141430,0rcu_preempt29108-21runrttasks23:38:470
221420,0irq_work/1241rcuc/119:45:211
181420,0irq_work/0161rcuc/023:08:080
16018994223,0cyclictest13006-21grep23:00:053
1601899421,0cyclictest210099rtkit-daemon00:31:223
1601699427,0cyclictest2399migration/120:58:011
1601699427,0cyclictest2399migration/100:02:251
1601699426,0cyclictest2399migration/121:22:121
1601699425,0cyclictest2399migration/119:25:081
1601399422,0cyclictest1799migration/019:15:230
141420,0rcu_preempt39-21ksoftirqd/323:55:133
141420,0rcu_preempt29101-21kernelversion19:40:133
141420,0rcu_preempt25-21ksoftirqd/120:15:041
361410,0irq_work/3381rcuc/322:10:093
361410,0irq_work/3381rcuc/322:10:093
221410,0irq_work/1241rcuc/121:29:531
181410,0irq_work/0141rcu_preempt22:44:540
1601899416,0cyclictest3799migration/320:34:543
1601899411,0cyclictest210099rtkit-daemon21:09:503
1601899411,0cyclictest210099rtkit-daemon21:09:503
16016994126,0cyclictest20895-21sed22:00:091
16016994123,0cyclictest17453-21cut23:10:081
141410,0rcu_preempt39-21ksoftirqd/322:05:093
141410,0rcu_preempt24114-21taskset22:07:251
103350410,0irq/28-eth0-tx-7650irq/26-ahci[00000:10:090
361400,0irq_work/3381rcuc/321:40:243
221400,0irq_work/1241rcuc/121:45:161
1601899406,0cyclictest3799migration/320:40:153
1601899403,0cyclictest3799migration/323:35:193
1601699404,0cyclictest2399migration/100:10:081
1601399406,0cyclictest1799migration/021:15:080
1601399403,0cyclictest1799migration/021:10:060
161400,0rcuc/029501-21cpuspeed_turbos23:40:040
161400,0rcuc/00-21swapper/019:09:540
141400,0rcu_preempt39-21ksoftirqd/300:20:143
141400,0rcu_preempt39-21ksoftirqd/300:20:143
141400,0rcu_preempt25-21ksoftirqd/119:20:041
141400,0rcu_preempt13-21ksoftirqd/021:05:230
141400,0rcu_preempt13-21ksoftirqd/021:05:230
141400,0rcu_preempt0-21swapper/020:19:540
221390,0irq_work/1241rcuc/123:05:191
181390,0irq_work/0161rcuc/022:54:270
181390,0irq_work/0141rcu_preempt21:50:230
16018993927,0cyclictest812-21rsyslogd23:19:533
16018993925,0cyclictest6448-21grep21:25:083
16018993925,0cyclictest30818-21grep23:40:253
16018993924,0cyclictest20162-21crond21:59:533
1601899391,0cyclictest210099rtkit-daemon21:14:183
1601699396,0cyclictest2399migration/120:09:011
16016993923,0cyclictest7104-21cut22:45:071
141390,0rcu_preempt9286-21gltestperf22:50:081
141390,0rcu_preempt39-21ksoftirqd/300:04:543
141390,0rcu_preempt25-21ksoftirqd/122:40:051
141390,0rcu_preempt25-21ksoftirqd/122:10:221
141390,0rcu_preempt25-21ksoftirqd/122:10:221
141390,0rcu_preempt14584-21fschecks_time21:45:073
181380,0irq_work/0141rcu_preempt23:50:040
181380,0irq_work/0141rcu_preempt23:14:530
1601899384,0cyclictest3799migration/323:07:013
16018993824,0cyclictest11079-21grep20:15:083
1601699386,0cyclictest2399migration/123:20:081
1601699385,0cyclictest2399migration/123:25:071
1601699385,0cyclictest210099rtkit-daemon00:16:131
141380,0rcu_preempt39-21ksoftirqd/320:50:053
141380,0rcu_preempt39-21ksoftirqd/319:45:253
141380,0rcu_preempt29464-21missed_timers22:20:133
141380,0rcu_preempt2699-21grep22:35:043
141380,0rcu_preempt26727-21expr22:15:053
141380,0rcu_preempt10179-21taskset20:12:313
141380,0rcu_preempt0-21swapper/021:56:250
181370,0irq_work/0141rcu_preempt20:04:540
1601699375,0cyclictest2399migration/122:37:041
1601699372,0cyclictest30942-21runrttasks21:03:581
16013993724,0cyclictest10162-21grep21:35:050
141370,0rcu_preempt39-21ksoftirqd/322:00:223
141370,0rcu_preempt39-21ksoftirqd/300:10:203
141370,0rcu_preempt25-21ksoftirqd/123:40:191
141370,0rcu_preempt25-21ksoftirqd/119:55:031
141370,0rcu_preempt25-21ksoftirqd/119:55:031
141370,0rcu_preempt21349-21cstates23:20:043
141370,0rcu_preempt12374-21forks21:40:060
141370,0rcu_preempt0-21swapper/020:39:540
141370,0rcu_preempt0-21swapper/000:30:180
1601899363,0cyclictest3799migration/321:24:313
1601399361,0cyclictest20397-21munin-node22:00:220
16018993523,0cyclictest17561-21sed23:10:093
141350,0rcu_preempt12756-21latency_hist22:59:543
361340,0irq_work/3141rcu_preempt23:45:083
361340,0irq_work/3141rcu_preempt22:29:543
141340,0rcu_preempt0-21swapper/121:54:551
141340,0rcu_preempt0-21swapper/119:51:051
361330,0irq_work/3141rcu_preempt19:50:203
16018993323,0cyclictest23330-21cpuspeed_turbos23:25:043
141330,0rcu_preempt8709-21grep21:30:121
221320,0irq_work/1241rcuc/119:30:181
1601899323,0cyclictest7976-21gltestperf00:05:083
361300,0irq_work/3141rcu_preempt19:20:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional