You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-01 - 11:43

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6.osadl.org (updated Wed May 31, 2023 12:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2911210,0irq_work/2311rcuc/210:23:002
2911180,0irq_work/2311rcuc/207:59:362
2911160,0irq_work/2311rcuc/210:50:392
3611150,0irq_work/3381rcuc/307:59:123
2911150,0irq_work/2311rcuc/209:13:512
3611140,0irq_work/3381rcuc/309:34:073
2911140,0irq_work/2311rcuc/208:41:322
2911130,0irq_work/2311rcuc/211:17:372
2911130,0irq_work/2311rcuc/210:58:432
2911130,0irq_work/2311rcuc/207:10:492
2911120,0irq_work/2311rcuc/208:36:562
2911120,0irq_work/2311rcuc/208:32:502
2911100,0irq_work/2311rcuc/211:37:052
2911070,0irq_work/2311rcuc/210:16:322
2911070,0irq_work/2311rcuc/209:07:302
2911060,0irq_work/2311rcuc/210:24:112
2911050,0irq_work/2311rcuc/209:42:522
2911040,0irq_work/2311rcuc/209:31:342
2911040,0irq_work/2311rcuc/208:47:502
3611030,0irq_work/3381rcuc/308:34:083
2911030,0irq_work/2311rcuc/212:03:422
2911030,0irq_work/2311rcuc/206:59:012
3611020,0irq_work/3381rcuc/310:49:003
2911020,0irq_work/2311rcuc/210:47:542
2911020,0irq_work/2311rcuc/210:42:122
2911020,0irq_work/2311rcuc/209:56:252
2911020,0irq_work/2311rcuc/209:56:252
2911020,0irq_work/2311rcuc/207:35:502
2911010,0irq_work/2311rcuc/212:24:282
2911010,0irq_work/2311rcuc/207:51:542
2911000,0irq_work/2311rcuc/212:07:182
2911000,0irq_work/2311rcuc/207:29:202
361990,0irq_work/3381rcuc/311:19:173
291990,0irq_work/2311rcuc/209:11:062
291990,0irq_work/2311rcuc/208:54:142
291990,0irq_work/2311rcuc/207:21:532
291980,0irq_work/2311rcuc/211:02:402
291980,0irq_work/2311rcuc/207:40:212
361970,0irq_work/3381rcuc/311:12:203
361970,0irq_work/3381rcuc/311:12:203
361970,0irq_work/3381rcuc/308:24:193
291970,0irq_work/2311rcuc/209:28:242
291970,0irq_work/2311rcuc/207:53:542
291970,0irq_work/2311rcuc/207:44:182
291970,0irq_work/2311rcuc/207:44:182
291950,0irq_work/2311rcuc/212:14:132
291950,0irq_work/2311rcuc/212:14:132
291950,0irq_work/2311rcuc/210:13:292
291950,0irq_work/2311rcuc/210:07:352
291950,0irq_work/2311rcuc/209:44:582
291950,0irq_work/2311rcuc/209:19:572
291950,0irq_work/2311rcuc/209:03:272
291950,0irq_work/2311rcuc/208:23:072
291950,0irq_work/2311rcuc/208:12:252
291950,0irq_work/2311rcuc/208:07:432
291950,0irq_work/2311rcuc/207:24:112
291940,0irq_work/2311rcuc/211:28:172
291940,0irq_work/2311rcuc/211:19:592
291940,0irq_work/2311rcuc/211:06:192
291940,0irq_work/2311rcuc/211:06:192
291930,0irq_work/2311rcuc/211:55:242
291930,0irq_work/2311rcuc/211:47:242
291930,0irq_work/2311rcuc/208:24:252
291920,0irq_work/2311rcuc/212:12:162
291920,0irq_work/2311rcuc/210:37:122
291920,0irq_work/2311rcuc/209:52:552
291920,0irq_work/2311rcuc/208:49:022
291920,0irq_work/2311rcuc/208:49:022
291910,0irq_work/2311rcuc/210:32:542
291910,0irq_work/2311rcuc/210:01:592
291910,0irq_work/2311rcuc/210:01:592
291900,0irq_work/2311rcuc/212:18:522
291900,0irq_work/2311rcuc/211:49:572
291900,0irq_work/2311rcuc/211:40:172
291900,0irq_work/2311rcuc/211:31:412
291900,0irq_work/2311rcuc/207:13:592
291890,0irq_work/2311rcuc/209:34:222
291890,0irq_work/2311rcuc/208:17:552
291890,0irq_work/2311rcuc/207:07:432
361880,0irq_work/3381rcuc/312:14:193
361880,0irq_work/3381rcuc/312:14:193
291870,0irq_work/2311rcuc/211:12:162
291870,0irq_work/2311rcuc/211:12:162
361820,0irq_work/3381rcuc/309:09:183
361790,0irq_work/3381rcuc/310:11:443
361770,0irq_work/3381rcuc/311:49:033
361770,0irq_work/3381rcuc/309:23:383
2668399772,0cyclictest210099rtkit-daemon10:33:431
361760,0irq_work/3381rcuc/311:57:513
361750,0irq_work/3381rcuc/310:39:173
361740,0irq_work/3381rcuc/312:18:433
361740,0irq_work/3381rcuc/311:58:443
361740,0irq_work/3381rcuc/311:04:373
361740,0irq_work/3381rcuc/311:04:373
361730,0irq_work/3381rcuc/307:34:263
361720,0irq_work/3381rcuc/312:24:423
361720,0irq_work/3381rcuc/310:27:513
361720,0irq_work/3381rcuc/309:17:063
361710,0irq_work/3381rcuc/308:30:463
361710,0irq_work/3381rcuc/307:39:023
361710,0irq_work/3381rcuc/307:14:263
361700,0irq_work/3381rcuc/307:49:213
361700,0irq_work/3381rcuc/306:59:213
361690,0irq_work/3381rcuc/307:58:383
361690,0irq_work/3381rcuc/307:47:273
361690,0irq_work/3381rcuc/307:47:273
221690,0irq_work/1241rcuc/112:19:041
361680,0irq_work/3381rcuc/307:19:363
361670,0irq_work/3381rcuc/310:54:103
221670,0irq_work/1241rcuc/111:18:431
221670,0irq_work/1241rcuc/110:54:151
361660,0irq_work/3381rcuc/308:55:383
361660,0irq_work/3381rcuc/308:39:003
361650,0irq_work/3381rcuc/311:18:423
181650,0irq_work/0161rcuc/011:19:210
103350650,0irq/28-eth0-tx-7650irq/26-ahci[00011:18:110
361640,0irq_work/3381rcuc/311:43:173
361640,0irq_work/3381rcuc/309:59:113
361640,0irq_work/3381rcuc/309:59:113
361640,0irq_work/3381rcuc/309:41:363
361640,0irq_work/3381rcuc/307:04:213
361630,0irq_work/3381rcuc/311:34:233
361630,0irq_work/3381rcuc/309:04:153
361610,0irq_work/3381rcuc/310:44:093
361610,0irq_work/3381rcuc/310:06:533
361610,0irq_work/3381rcuc/308:19:073
221610,0irq_work/1241rcuc/111:59:221
361600,0irq_work/3381rcuc/310:21:233
361600,0irq_work/3381rcuc/308:44:133
361590,0irq_work/3381rcuc/311:43:593
361590,0irq_work/3381rcuc/309:53:373
221590,0irq_work/1241rcuc/111:43:351
361580,0irq_work/3381rcuc/309:24:243
2668299582,0cyclictest210099rtkit-daemon10:19:010
361570,0irq_work/3381rcuc/307:29:173
141570,0rcu_preempt25-21ksoftirqd/109:23:431
381560,0rcuc/325315-21cut06:54:023
361560,0irq_work/3381rcuc/312:05:573
361560,0irq_work/3381rcuc/307:28:383
181560,0irq_work/0161rcuc/008:29:130
361550,0irq_work/3381rcuc/308:59:233
221550,0irq_work/1241rcuc/111:28:481
141550,0rcu_preempt13-21ksoftirqd/009:38:590
361540,0irq_work/3381rcuc/312:09:123
361540,0irq_work/3381rcuc/309:55:373
361540,0irq_work/3381rcuc/309:55:373
361530,0irq_work/3381rcuc/310:15:173
361530,0irq_work/3381rcuc/309:29:193
361530,0irq_work/3381rcuc/308:49:283
361530,0irq_work/3381rcuc/308:49:283
361530,0irq_work/3381rcuc/308:13:073
2668299532,0cyclictest210099rtkit-daemon10:27:510
241530,0rcuc/126467-21ls06:58:431
221520,0irq_work/1241rcuc/110:24:111
221520,0irq_work/1241rcuc/110:18:431
221520,0irq_work/1241rcuc/108:48:341
221520,0irq_work/1241rcuc/108:38:071
361510,0irq_work/3381rcuc/310:59:313
221510,0irq_work/1241rcuc/107:39:201
221510,0irq_work/1241rcuc/107:19:031
361500,0irq_work/3381rcuc/311:27:203
361500,0irq_work/3381rcuc/310:29:153
361500,0irq_work/3381rcuc/308:05:453
221500,0irq_work/1241rcuc/112:15:551
221500,0irq_work/1241rcuc/112:15:551
221500,0irq_work/1241rcuc/110:39:201
221500,0irq_work/1241rcuc/110:36:001
361490,0irq_work/3381rcuc/310:34:123
361490,0irq_work/3381rcuc/309:44:163
361490,0irq_work/3381rcuc/307:09:243
2668599493,0cyclictest210099rtkit-daemon08:17:513
221490,0irq_work/1241rcuc/107:02:521
181490,0irq_work/0141rcu_preempt07:34:200
141490,0rcu_preempt13-21ksoftirqd/008:08:430
2668299482,0cyclictest210099rtkit-daemon10:32:360
221480,0irq_work/1241rcuc/112:24:221
221480,0irq_work/1241rcuc/109:34:411
221480,0irq_work/1241rcuc/107:57:411
221480,0irq_work/1241rcuc/107:11:181
141480,0rcu_preempt12241-21cat11:44:151
2668299477,0cyclictest1799migration/008:10:570
26682994728,0cyclictest28930-21grep08:19:260
221470,0irq_work/1241rcuc/112:12:331
221470,0irq_work/1241rcuc/110:44:101
141470,0rcu_preempt13-21ksoftirqd/007:44:180
141470,0rcu_preempt13-21ksoftirqd/007:44:180
2668299466,0cyclictest1799migration/010:46:040
26682994628,0cyclictest23479-21grep09:24:240
221460,0irq_work/1241rcuc/111:54:311
221460,0irq_work/1241rcuc/109:57:321
221460,0irq_work/1241rcuc/109:57:321
221460,0irq_work/1241rcuc/107:32:001
161460,0rcuc/026421-21taskset06:58:230
141460,0rcu_preempt13-21ksoftirqd/007:29:070
2668299457,0cyclictest1799migration/009:23:320
2668299454,0cyclictest1799migration/011:53:070
26682994528,0cyclictest23825-21grep12:14:030
26682994528,0cyclictest23825-21grep12:14:030
2668299451,0cyclictest210099rtkit-daemon10:55:080
141450,0rcu_preempt3082-21seq08:37:140
141450,0rcu_preempt25-21ksoftirqd/111:19:061
141450,0rcu_preempt13-21ksoftirqd/012:04:130
141450,0rcu_preempt13-21ksoftirqd/009:34:170
141450,0rcu_preempt0-21swapper/206:54:052
361440,0irq_work/3381rcuc/311:29:143
221440,0irq_work/1241rcuc/111:53:431
221440,0irq_work/1241rcuc/111:04:151
221440,0irq_work/1241rcuc/111:04:151
221440,0irq_work/1241rcuc/110:11:371
221440,0irq_work/1241rcuc/109:59:061
221440,0irq_work/1241rcuc/109:59:061
221440,0irq_work/1241rcuc/109:52:451
141440,0rcu_preempt25-21ksoftirqd/107:24:041
141440,0rcu_preempt13-21ksoftirqd/007:24:070
2668399436,0cyclictest2399migration/110:49:321
2668299437,0cyclictest1799migration/007:09:530
221430,0irq_work/1241rcuc/109:33:081
221430,0irq_work/1241rcuc/109:04:221
221430,0irq_work/1241rcuc/108:49:121
221430,0irq_work/1241rcuc/108:49:121
221430,0irq_work/1241rcuc/107:04:121
181430,0irq_work/0161rcuc/009:14:140
141430,0rcu_preempt25-21ksoftirqd/107:59:171
141430,0rcu_preempt25-21ksoftirqd/107:49:151
141430,0rcu_preempt1234-21runrttasks09:59:280
141430,0rcu_preempt1234-21runrttasks09:59:280
221420,0irq_work/1241rcuc/110:05:591
221420,0irq_work/1241rcuc/108:29:041
141420,0rcu_preempt25-21ksoftirqd/108:54:011
141420,0rcu_preempt25-21ksoftirqd/108:14:151
141420,0rcu_preempt13-21ksoftirqd/012:24:060
141420,0rcu_preempt13-21ksoftirqd/008:44:030
141420,0rcu_preempt13-21ksoftirqd/007:58:430
141420,0rcu_preempt13-21ksoftirqd/006:59:240
2668299416,0cyclictest1799migration/012:19:200
2668299415,0cyclictest1799migration/007:39:090
26682994120,0cyclictest14317-21sed09:04:080
2668299412,0cyclictest8668-21crond11:38:430
141410,0rcu_preempt13-21ksoftirqd/010:39:040
141410,0rcu_preempt13-21ksoftirqd/009:49:190
141410,0rcu_preempt13-21ksoftirqd/008:39:150
141410,0rcu_preempt13-21ksoftirqd/008:14:050
141410,0rcu_preempt13-21ksoftirqd/007:18:440
2668299404,0cyclictest1799migration/011:32:410
2668299403,0cyclictest1799migration/009:29:060
26682994026,0cyclictest26885-21grep11:04:020
26682994026,0cyclictest26885-21grep11:04:020
2668299402,0cyclictest1799migration/009:46:410
221400,0irq_work/1241rcuc/108:09:281
181400,0irq_work/0161rcuc/009:54:170
181400,0irq_work/0161rcuc/009:54:170
141400,0rcu_preempt13-21ksoftirqd/011:09:060
141400,0rcu_preempt13-21ksoftirqd/011:09:060
141400,0rcu_preempt13-21ksoftirqd/009:03:190
2668299393,0cyclictest1799migration/010:11:050
221390,0irq_work/1241rcuc/109:43:511
221390,0irq_work/1241rcuc/107:37:511
181390,0irq_work/0161rcuc/012:01:200
141390,0rcu_preempt9829-21taskset08:53:590
141390,0rcu_preempt20640-21cpuspeed_turbos10:48:580
141390,0rcu_preempt16255-21switchtime11:54:350
141390,0rcu_preempt13-21ksoftirqd/012:09:110
141390,0rcu_preempt13-21ksoftirqd/011:43:590
2668299383,0cyclictest1799migration/008:49:220
2668299383,0cyclictest1799migration/008:49:220
2668299383,0cyclictest1799migration/007:59:580
221380,0irq_work/1241rcuc/109:26:551
221380,0irq_work/1241rcuc/108:23:431
141380,0rcu_preempt7145-21fschecks_count11:34:061
141380,0rcu_preempt2875-21cstates11:24:041
141380,0rcu_preempt18096-21cstates09:14:011
141380,0rcu_preempt1667-21taskset07:14:211
2668399373,0cyclictest2399migration/108:38:581
26682993725,0cyclictest1922-21unixbench-2d07:14:240
221370,0irq_work/1241rcuc/108:06:221
181370,0irq_work/0161rcuc/010:08:100
141370,0rcu_preempt2845-21expr11:24:040
141370,0rcu_preempt25-21ksoftirqd/109:41:081
141370,0rcu_preempt13-21ksoftirqd/007:49:020
2668299363,0cyclictest30593-21crond07:08:430
2668299362,0cyclictest0-21swapper/008:24:140
221360,0irq_work/1141rcu_preempt10:58:431
181360,0irq_work/0161rcuc/010:34:510
2668399354,0cyclictest2399migration/112:08:201
2668399354,0cyclictest2399migration/110:18:561
2668399352,0cyclictest2399migration/107:43:581
2668399352,0cyclictest2399migration/107:43:581
2668299351,0cyclictest210099rtkit-daemon10:15:470
221350,0irq_work/1241rcuc/111:09:211
221350,0irq_work/1241rcuc/111:09:211
2668399344,0cyclictest2399migration/109:09:541
2668399344,0cyclictest2399migration/109:03:101
141340,0rcu_preempt13-21ksoftirqd/011:40:430
141340,0rcu_preempt13-21ksoftirqd/010:59:210
141340,0rcu_preempt13-21ksoftirqd/009:09:180
2668399323,0cyclictest2399migration/108:24:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional