You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-27 - 15:46

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot6.osadl.org (updated Fri Jan 27, 2023 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1811020,0irq_work/0161rcuc/009:03:180
291890,0irq_work/2311rcuc/207:34:442
221810,0irq_work/1241rcuc/108:43:181
221720,0irq_work/1241rcuc/112:12:471
2413999719,0cyclictest2399migration/110:51:441
2413999703,0cyclictest2399migration/112:16:281
2413999703,0cyclictest2399migration/110:18:341
2413899695,0cyclictest1799migration/011:13:210
2413899694,0cyclictest1799migration/011:53:120
2413999689,0cyclictest2399migration/111:07:341
2413999677,0cyclictest2399migration/111:34:441
2413899676,0cyclictest1799migration/009:52:490
361640,0irq_work/3381rcuc/312:00:483
181640,0irq_work/0161rcuc/007:08:240
181640,0irq_work/0161rcuc/007:08:240
361630,0irq_work/3381rcuc/307:38:173
2413999635,0cyclictest2399migration/111:40:521
361620,0irq_work/3381rcuc/311:57:333
2413999625,0cyclictest2399migration/110:34:041
2413999625,0cyclictest2399migration/110:34:041
2413899623,0cyclictest1799migration/011:46:480
2413899623,0cyclictest1799migration/011:46:480
291610,0irq_work/2311rcuc/207:53:042
24139996123,0cyclictest11418-21timerandwakeup10:23:231
103350610,0irq/28-eth0-tx-22764-21runrttasks06:58:050
2414099603,0cyclictest3099migration/209:24:072
2413999594,0cyclictest2399migration/112:22:471
24139995931,0cyclictest1135-21grep09:58:261
24139995920,0cyclictest22934-21cut10:53:101
2413999584,0cyclictest2399migration/110:59:441
24138995827,0cyclictest29457-21latency_hist12:27:470
141580,0rcu_preempt32-21ksoftirqd/210:57:472
2414099577,0cyclictest3099migration/209:00:312
2413999576,0cyclictest2399migration/112:28:201
24139995722,0cyclictest4573-21crond11:27:471
24139995719,0cyclictest27170-21grep08:28:051
24138995726,0cyclictest21727-21munin-node09:33:170
24138995725,0cyclictest2842-21crond08:47:470
24138995724,0cyclictest1314-21latency11:18:110
24138995721,0cyclictest5755-21seq10:12:220
2413899572,0cyclictest28804-21munin-node11:08:120
24138995719,0cyclictest12356-21cpuspeed_turbos10:28:050
141570,0rcu_preempt26741-21grep07:08:112
141570,0rcu_preempt26741-21grep07:08:112
2414099564,0cyclictest3099migration/212:07:222
24139995623,0cyclictest12816-21grep09:08:301
2413899563,0cyclictest1799migration/011:28:050
24138995622,0cyclictest17749-21cut11:58:100
2413899560,0cyclictest1799migration/009:23:200
2414099553,0cyclictest3099migration/211:31:222
2413999553,0cyclictest2399migration/109:16:291
2413999553,0cyclictest2399migration/107:48:061
24139995522,0cyclictest32536-21tr07:23:011
24138995523,0cyclictest9057-21grep11:37:580
24138995519,0cyclictest28046-21grep09:48:030
181550,0irq_work/0161rcuc/009:21:010
361540,0irq_work/3381rcuc/311:38:183
2414099543,0cyclictest3099migration/210:40:372
2414099542,0cyclictest3099migration/212:09:072
2414099542,0cyclictest3099migration/207:03:342
2414099542,0cyclictest28424-21cpuspeed_turbos07:13:042
2414099542,0cyclictest28424-21cpuspeed_turbos07:13:042
2413999543,0cyclictest2399migration/111:18:161
2413999540,0cyclictest2399migration/107:43:151
24138995420,0cyclictest27352-21crond12:22:470
2413899542,0cyclictest1799migration/007:22:490
141540,0rcu_preempt13-21ksoftirqd/008:02:480
361530,0irq_work/3381rcuc/307:28:183
2414099534,0cyclictest3099migration/211:37:422
2414099533,0cyclictest3099migration/209:44:442
24140995323,0cyclictest32220-21grep09:58:052
2414099531,0cyclictest7283-21sort10:13:302
2413999535,0cyclictest2399migration/107:53:091
2413999535,0cyclictest2399migration/107:18:121
2413999533,0cyclictest2399migration/107:28:181
24139995320,0cyclictest8029-21sort08:58:141
24138995322,0cyclictest30434-21missed_timers12:28:090
24138995321,0cyclictest19540-21fschecks_count12:03:010
2413899530,0cyclictest1799migration/010:03:460
221530,0irq_work/1241rcuc/111:13:081
291520,0irq_work/2311rcuc/210:25:092
2414099523,0cyclictest3099migration/212:22:352
2414099522,0cyclictest210099rtkit-daemon09:52:492
24139995221,0cyclictest24302-21sed09:38:071
2413999520,0cyclictest2399migration/110:03:061
24138995223,0cyclictest4823-21gltestperf07:33:060
24138995222,0cyclictest6655-21awk10:13:170
24138995222,0cyclictest14908-21crond11:52:470
24138995221,0cyclictest6545-21grep08:53:250
291510,0irq_work/2311rcuc/211:48:152
24141995125,0cyclictest30875-21needreboot09:53:143
2414099515,0cyclictest3099migration/209:51:282
2414099514,0cyclictest3099migration/211:18:542
24140995122,0cyclictest26351-21grep08:23:212
24140995122,0cyclictest26351-21grep08:23:212
24140995121,0cyclictest2230-21cat08:43:202
24140995120,0cyclictest6626-21vmstat08:53:262
24140995118,0cyclictest10896-21cut07:48:052
24139995121,0cyclictest6539-21unixbench_multi08:53:251
24139995121,0cyclictest0-21swapper/107:03:041
24139995120,0cyclictest0-21swapper/109:28:111
24139995120,0cyclictest0-21swapper/109:28:111
24139995118,0cyclictest27223-21unixbench-2d09:43:231
2413999511,0cyclictest2399migration/108:41:331
2413999510,0cyclictest2399migration/108:09:031
24138995129,0cyclictest21366-21munin-node12:08:250
24138995125,0cyclictest0-21swapper/007:43:190
24138995123,0cyclictest0-21swapper/008:32:470
24138995121,0cyclictest1234-21runrttasks10:27:280
24138995121,0cyclictest11633-21expr07:48:180
181510,0irq_work/0161rcuc/008:48:160
141510,0rcu_preempt28680-21seq12:23:162
141510,0rcu_preempt0-21swapper/307:02:273
141510,0rcu_preempt0-21swapper/207:02:192
103350510,0irq/28-eth0-tx-7650irq/26-ahci[00010:52:550
2414099503,0cyclictest3099migration/210:46:412
24140995021,0cyclictest1107-21nscd08:13:192
24140995020,0cyclictest14766-21expr07:58:052
2413999504,0cyclictest2399migration/111:08:091
2413999503,0cyclictest2399migration/109:48:121
24139995022,0cyclictest29546-21fschecks_time08:33:111
24139995021,0cyclictest6846-21grep07:38:071
24139995021,0cyclictest26371-21cut07:08:061
24139995021,0cyclictest26371-21cut07:08:061
24139995019,0cyclictest17722-21unixbench-2d10:38:261
2413999500,0cyclictest2399migration/110:43:291
24138995021,0cyclictest29129-21cat07:13:150
24138995021,0cyclictest29129-21cat07:13:150
2413899502,0cyclictest1799migration/009:13:510
24138995014,0cyclictest0-21swapper/008:13:350
221500,0irq_work/1241rcuc/108:48:081
141500,0rcu_preempt9682-21cut09:03:051
141500,0rcu_preempt32-21ksoftirqd/207:28:272
2414199497,0cyclictest3799migration/310:22:303
24141994930,0cyclictest0-21swapper/310:16:033
2414099495,0cyclictest3099migration/212:14:102
2414099494,0cyclictest3099migration/209:22:422
2414099493,0cyclictest3099migration/210:12:342
24140994917,0cyclictest18370-21seq08:05:232
24139994919,0cyclictest5559-21latency11:28:101
24139994919,0cyclictest23099-21cstates08:18:041
24139994919,0cyclictest23099-21cstates08:18:041
2413999490,0cyclictest2399migration/107:16:581
2413999490,0cyclictest2399migration/107:16:581
24138994925,0cyclictest6876-21munin-node11:33:180
2413899492,0cyclictest1799migration/008:59:130
181490,0irq_work/0161rcuc/010:18:290
361480,0irq_work/3381rcuc/312:26:493
361480,0irq_work/3381rcuc/309:13:003
2414199486,0cyclictest3799migration/311:22:293
2414099487,0cyclictest0-21swapper/207:38:212
2414099482,0cyclictest3099migration/207:45:502
24140994816,0cyclictest4119-21processes08:48:162
24139994820,0cyclictest18861-21timerwakeupswit09:23:201
24139994819,0cyclictest4431-21cpuspeed_turbos07:33:021
2413899487,0cyclictest1799migration/008:05:280
24138994820,0cyclictest24526-21sendmail_mailqu12:13:180
2413899482,0cyclictest0-21swapper/008:33:140
141480,0rcu_preempt32-21ksoftirqd/209:28:112
141480,0rcu_preempt32-21ksoftirqd/209:28:112
141480,0rcu_preempt13-21ksoftirqd/008:08:290
141480,0rcu_preempt13-21ksoftirqd/007:38:540
2414199474,0cyclictest3799migration/308:23:053
2414199474,0cyclictest3799migration/308:23:053
24141994716,0cyclictest6405-21sed08:53:223
24140994721,0cyclictest29466-21fschecks_count08:33:112
2414099472,0cyclictest210099rtkit-daemon11:37:482
24140994715,0cyclictest27498-21cut08:28:092
2413999477,0cyclictest2399migration/108:02:131
2413899477,0cyclictest1799migration/011:24:350
2413899473,0cyclictest1799migration/008:22:470
2413899473,0cyclictest1799migration/008:22:470
24138994721,0cyclictest20893-21timerandwakeup09:28:260
24138994721,0cyclictest20893-21timerandwakeup09:28:260
241470,0rcuc/122677-21munin-node06:58:031
361460,0irq_work/3381rcuc/312:28:493
361460,0irq_work/3381rcuc/312:16:313
361460,0irq_work/3381rcuc/309:37:073
361460,0irq_work/3381rcuc/309:31:453
361460,0irq_work/3381rcuc/309:31:453
361460,0irq_work/3381rcuc/309:00:143
361460,0irq_work/3381rcuc/307:27:113
361460,0irq_work/3381rcuc/307:08:433
361460,0irq_work/3381rcuc/307:08:433
24141994628,0cyclictest29315-21gltestperf11:08:033
2414099464,0cyclictest3099migration/212:31:472
2414099464,0cyclictest3099migration/210:52:472
24140994619,0cyclictest1234-21runrttasks09:37:292
24140994618,0cyclictest29702-21netstat11:08:132
2413999467,0cyclictest2399migration/108:07:081
24139994620,0cyclictest6377-21fschecks_time10:13:141
24139994620,0cyclictest22032-21cat08:13:341
24139994619,0cyclictest16631-21swap09:18:181
24138994621,0cyclictest26739-21munin-node11:03:130
141460,0rcu_preempt13-21ksoftirqd/010:47:530
141460,0rcu_preempt13-21ksoftirqd/010:43:250
141460,0rcu_preempt13-21ksoftirqd/010:38:230
141460,0rcu_preempt13-21ksoftirqd/009:58:450
141460,0rcu_preempt0-21swapper/007:30:160
361450,0irq_work/3381rcuc/309:25:093
361450,0irq_work/3381rcuc/307:43:193
291450,0irq_work/2311rcuc/211:43:052
291450,0irq_work/2311rcuc/211:43:052
2414099454,0cyclictest3099migration/210:18:132
2414099453,0cyclictest3099migration/210:58:122
24140994521,0cyclictest31588-21fschecks_time08:38:032
2414099452,0cyclictest3099migration/209:04:092
2414099452,0cyclictest0-21swapper/210:03:062
2414099452,0cyclictest0-21swapper/207:23:552
24140994519,0cyclictest26888-21expr11:03:032
2414099451,0cyclictest0-21swapper/207:18:082
24139994521,0cyclictest1234-21runrttasks08:23:481
24139994521,0cyclictest1234-21runrttasks08:23:481
2413899457,0cyclictest1799migration/008:23:040
2413899457,0cyclictest1799migration/008:23:040
141450,0rcu_preempt25-21ksoftirqd/112:03:221
141450,0rcu_preempt13-21ksoftirqd/007:03:190
361440,0irq_work/3381rcuc/312:12:373
361440,0irq_work/3381rcuc/310:39:193
291440,0irq_work/2311rcuc/209:12:482
2414199441,0cyclictest210099rtkit-daemon12:18:023
2414099444,0cyclictest3099migration/211:54:592
2414099442,0cyclictest0-21swapper/208:18:202
2414099442,0cyclictest0-21swapper/208:18:202
24140994417,0cyclictest1234-21runrttasks09:13:012
24140994414,0cyclictest24408-21if_err_eth009:38:082
24139994421,0cyclictest1234-21runrttasks11:44:031
24139994421,0cyclictest1234-21runrttasks11:44:031
2413999440,0cyclictest2399migration/111:58:101
2413899447,0cyclictest1799migration/009:45:330
141440,0rcu_preempt39-21ksoftirqd/311:33:213
141440,0rcu_preempt13-21ksoftirqd/010:59:570
141440,0rcu_preempt13-21ksoftirqd/009:40:410
141440,0rcu_preempt13-21ksoftirqd/007:53:270
141440,0rcu_preempt13-21ksoftirqd/007:18:140
381430,0rcuc/30-21swapper/307:02:483
361430,0irq_work/3381rcuc/309:06:263
291430,0irq_work/2311rcuc/212:02:472
2414199432,0cyclictest210099rtkit-daemon08:13:343
24140994323,0cyclictest4172-21unixbench_singl11:23:202
24140994320,0cyclictest18965-21expr08:08:042
2414099432,0cyclictest14398-21cpuspeed_turbos10:33:022
2414099432,0cyclictest14398-21cpuspeed_turbos10:33:022
2413899430,0cyclictest1799migration/008:40:200
141430,0rcu_preempt0-21swapper/308:30:443
361420,0irq_work/3381rcuc/310:48:033
361420,0irq_work/3381rcuc/309:52:283
2414199426,0cyclictest3799migration/307:33:593
2413999425,0cyclictest2399migration/109:34:191
141420,0rcu_preempt4572-21seq10:08:071
141420,0rcu_preempt39-21ksoftirqd/308:33:113
141420,0rcu_preempt14334-21sed11:48:201
141420,0rcu_preempt12227-21seq10:27:561
2414199416,0cyclictest3799migration/310:31:483
2414199413,0cyclictest3799migration/311:22:563
24141994125,0cyclictest14473-21grep11:48:233
24141994124,0cyclictest3249-21grep10:03:223
2414199411,0cyclictest210099rtkit-daemon07:13:313
2414199411,0cyclictest210099rtkit-daemon07:13:313
2414099413,0cyclictest31156-21grep11:13:022
141410,0rcu_preempt39-21ksoftirqd/310:53:313
141410,0rcu_preempt39-21ksoftirqd/310:33:083
141410,0rcu_preempt39-21ksoftirqd/310:33:083
141410,0rcu_preempt39-21ksoftirqd/307:58:133
361400,0irq_work/3381rcuc/308:40:333
361400,0irq_work/3381rcuc/307:53:153
2414199404,0cyclictest3799migration/308:21:053
2414199404,0cyclictest3799migration/308:21:053
24140994016,0cyclictest12258-21cpu10:28:042
141400,0rcu_preempt39-21ksoftirqd/311:43:183
141400,0rcu_preempt39-21ksoftirqd/311:43:183
141400,0rcu_preempt39-21ksoftirqd/311:03:133
141400,0rcu_preempt39-21ksoftirqd/309:58:253
141400,0rcu_preempt39-21ksoftirqd/309:08:293
2414199394,0cyclictest3799migration/311:12:593
2413999390,0cyclictest2399migration/112:26:591
141390,0rcu_preempt1934-21open_inodes08:43:153
2414199383,0cyclictest3799migration/311:30:483
2414199383,0cyclictest3799migration/309:38:023
24141993822,0cyclictest3223-21tr08:48:023
24141993820,0cyclictest16955-21grep09:18:233
2413999382,0cyclictest2399migration/111:53:021
141380,0rcu_preempt0-21swapper/009:10:300
361370,0irq_work/3381rcuc/312:06:163
361370,0irq_work/3381rcuc/310:23:233
2414199374,0cyclictest3799migration/309:47:393
2414199373,0cyclictest3799migration/310:08:243
2414199373,0cyclictest210099rtkit-daemon07:21:303
141370,0rcu_preempt39-21ksoftirqd/310:46:373
141370,0rcu_preempt39-21ksoftirqd/308:03:213
2413999365,0cyclictest2399migration/109:53:221
24138993621,0cyclictest0-21swapper/010:34:150
24138993621,0cyclictest0-21swapper/010:34:150
141360,0rcu_preempt26082-21unixbench_multi10:58:303
141360,0rcu_preempt19112-21fschecks_count08:08:063
141360,0rcu_preempt12349-21basename07:52:483
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional