You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-02-04 - 23:30

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6.osadl.org (updated Sat Feb 04, 2023 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2911180,0irq_work/2311rcuc/207:37:502
2911120,0irq_work/2311rcuc/209:07:472
2911110,0irq_work/2311rcuc/210:57:262
2911090,0irq_work/2311rcuc/207:48:002
2911080,0irq_work/2311rcuc/212:28:302
2911070,0irq_work/2311rcuc/207:11:012
2911060,0irq_work/2311rcuc/211:51:012
2911060,0irq_work/2311rcuc/211:51:012
2911050,0irq_work/2311rcuc/212:12:022
2911040,0irq_work/2311rcuc/211:58:082
2911020,0irq_work/2311rcuc/210:58:482
2911020,0irq_work/2311rcuc/209:02:402
2911020,0irq_work/2311rcuc/207:53:022
2911010,0irq_work/2311rcuc/210:38:002
2911000,0irq_work/2311rcuc/207:58:312
361990,0irq_work/3381rcuc/307:50:003
291990,0irq_work/2311rcuc/211:25:062
291990,0irq_work/2311rcuc/209:20:322
291980,0irq_work/2311rcuc/211:29:002
291980,0irq_work/2311rcuc/208:36:482
291980,0irq_work/2311rcuc/208:36:482
291980,0irq_work/2311rcuc/208:27:482
291970,0irq_work/2311rcuc/211:03:012
291970,0irq_work/2311rcuc/210:44:262
291970,0irq_work/2311rcuc/210:44:262
291970,0irq_work/2311rcuc/209:57:002
291970,0irq_work/2311rcuc/209:38:032
291970,0irq_work/2311rcuc/209:38:032
291970,0irq_work/2311rcuc/207:33:192
291960,0irq_work/2311rcuc/210:09:282
291960,0irq_work/2311rcuc/207:04:512
291940,0irq_work/2311rcuc/212:14:352
291940,0irq_work/2311rcuc/209:17:272
291940,0irq_work/2311rcuc/207:17:222
291930,0irq_work/2311rcuc/210:36:112
291930,0irq_work/2311rcuc/210:28:482
291930,0irq_work/2311rcuc/208:48:342
291920,0irq_work/2311rcuc/211:22:322
291910,0irq_work/2311rcuc/212:22:262
291910,0irq_work/2311rcuc/209:45:092
291910,0irq_work/2311rcuc/209:33:122
291910,0irq_work/2311rcuc/207:31:482
291910,0irq_work/2311rcuc/207:31:482
291910,0irq_work/2311rcuc/207:22:322
291900,0irq_work/2311rcuc/209:58:482
291900,0irq_work/2311rcuc/209:23:022
291900,0irq_work/2311rcuc/208:07:332
291890,0irq_work/2311rcuc/210:02:422
291890,0irq_work/2311rcuc/208:46:582
361880,0irq_work/3381rcuc/308:54:103
291880,0irq_work/2311rcuc/211:53:482
291880,0irq_work/2311rcuc/208:22:482
291870,0irq_work/2311rcuc/210:48:002
291870,0irq_work/2311rcuc/208:52:342
291870,0irq_work/2311rcuc/208:42:132
291870,0irq_work/2311rcuc/207:44:082
291860,0irq_work/2311rcuc/211:35:132
291860,0irq_work/2311rcuc/208:15:572
291850,0irq_work/2311rcuc/212:25:512
291850,0irq_work/2311rcuc/212:06:172
291850,0irq_work/2311rcuc/210:22:372
291850,0irq_work/2311rcuc/209:29:232
103350850,0irq/28-eth0-tx-7650irq/26-ahci[00011:52:000
103350850,0irq/28-eth0-tx-7650irq/26-ahci[00011:52:000
291830,0irq_work/2311rcuc/211:45:252
291830,0irq_work/2311rcuc/210:18:012
291830,0irq_work/2311rcuc/208:20:302
291830,0irq_work/2311rcuc/207:18:072
291820,0irq_work/2311rcuc/209:48:332
291810,0irq_work/2311rcuc/211:38:072
291810,0irq_work/2311rcuc/210:13:102
291810,0irq_work/2311rcuc/209:00:162
291810,0irq_work/2311rcuc/208:04:232
291790,0irq_work/2311rcuc/211:15:512
291790,0irq_work/2311rcuc/211:12:272
1503099753,0cyclictest3799migration/309:34:373
361740,0irq_work/3381rcuc/307:27:263
1503099747,0cyclictest3799migration/311:59:213
15030996825,0cyclictest12780-21unixbench-2d10:48:043
221660,0irq_work/1241rcuc/108:57:311
221630,0irq_work/1241rcuc/111:12:311
1503099632,0cyclictest3799migration/312:20:313
15030996311,0cyclictest14943-21df09:37:533
15030996311,0cyclictest14943-21df09:37:533
361610,0irq_work/3381rcuc/308:10:243
1503099617,0cyclictest3799migration/308:50:483
1503099615,0cyclictest3799migration/310:57:473
1503099614,0cyclictest3799migration/311:13:573
1501399614,0cyclictest1799migration/010:52:510
1501399614,0cyclictest1799migration/008:15:180
141610,0rcu_preempt13-21ksoftirqd/010:44:190
141610,0rcu_preempt13-21ksoftirqd/010:44:190
361600,0irq_work/3381rcuc/312:08:013
291600,0irq_work/2311rcuc/206:57:442
1503099607,0cyclictest3799migration/310:40:293
1503099602,0cyclictest3799migration/311:52:473
1501399604,0cyclictest1799migration/011:09:200
1501399595,0cyclictest1799migration/010:42:080
15013995919,0cyclictest6385-21sed07:57:540
1503099581,0cyclictest210099rtkit-daemon11:34:143
141580,0rcu_preempt13-21ksoftirqd/007:42:580
1503099573,0cyclictest3799migration/309:00:023
15030995727,0cyclictest7424-21crond08:02:323
1503099572,0cyclictest3799migration/307:04:273
1503099571,0cyclictest210099rtkit-daemon08:03:453
1501399574,0cyclictest1799migration/012:31:080
1501399561,0cyclictest2444-21runrttasks07:47:580
141560,0rcu_preempt39-21ksoftirqd/308:12:523
221550,0irq_work/1241rcuc/110:32:571
181550,0irq_work/0161rcuc/008:42:510
1503099552,0cyclictest3799migration/308:18:053
1503099550,0cyclictest3799migration/310:27:513
1501799553,0cyclictest210099rtkit-daemon12:21:091
141550,0rcu_preempt25-21ksoftirqd/107:38:011
361540,0irq_work/3381rcuc/309:02:423
361540,0irq_work/3381rcuc/307:28:043
361540,0irq_work/3381rcuc/307:28:043
1501399541,0cyclictest8158-21munin-node11:58:110
141540,0rcu_preempt25-21ksoftirqd/107:17:321
141540,0rcu_preempt13-21ksoftirqd/009:28:090
15030995310,0cyclictest14999-21cyclictest10:33:033
1503099530,0cyclictest3799migration/307:38:043
1501799536,0cyclictest2399migration/110:22:471
141530,0rcu_preempt39-21ksoftirqd/308:33:033
141530,0rcu_preempt39-21ksoftirqd/308:33:033
141530,0rcu_preempt13-21ksoftirqd/007:42:320
361520,0irq_work/3381rcuc/308:37:423
221520,0irq_work/1241rcuc/111:22:571
221520,0irq_work/1241rcuc/110:57:591
221520,0irq_work/1241rcuc/107:07:321
1503099520,0cyclictest3799migration/307:36:433
1501799521,0cyclictest210099rtkit-daemon12:00:501
1501399523,0cyclictest1799migration/009:07:430
361510,0irq_work/3381rcuc/312:30:573
361510,0irq_work/3381rcuc/310:02:323
361510,0irq_work/3381rcuc/309:31:103
361510,0irq_work/3381rcuc/309:18:213
361510,0irq_work/3381rcuc/307:56:553
221510,0irq_work/1241rcuc/110:02:571
1503099510,0cyclictest3799migration/308:23:513
1501399514,0cyclictest1799migration/009:13:120
1501399513,0cyclictest1799migration/011:28:290
1501399512,0cyclictest1799migration/007:10:090
1501399511,0cyclictest27508-21munin-node07:32:460
141510,0rcu_preempt30628-21unixbench_multi10:13:051
141510,0rcu_preempt25-21ksoftirqd/109:18:251
141510,0rcu_preempt25-21ksoftirqd/108:02:481
141510,0rcu_preempt13-21ksoftirqd/011:12:470
141510,0rcu_preempt13-21ksoftirqd/010:23:010
141510,0rcu_preempt13-21ksoftirqd/009:22:520
141510,0rcu_preempt13-21ksoftirqd/008:27:310
141510,0rcu_preempt0-21swapper/107:01:261
361500,0irq_work/3381rcuc/310:25:093
221500,0irq_work/1241rcuc/109:42:581
1501399502,0cyclictest23015-21munin-node09:58:060
1501399502,0cyclictest0-21swapper/012:12:470
15013995019,0cyclictest30638-21grep10:13:050
1501399501,0cyclictest32662-21grep09:02:430
141500,0rcu_preempt9055-21seq09:22:491
141500,0rcu_preempt39-21ksoftirqd/311:37:593
141500,0rcu_preempt13-21ksoftirqd/010:32:460
361490,0irq_work/3381rcuc/312:13:053
361490,0irq_work/3381rcuc/310:20:373
361490,0irq_work/3381rcuc/310:16:553
361490,0irq_work/3381rcuc/307:42:463
361490,0irq_work/3381rcuc/307:14:473
15013994924,0cyclictest9728-21munin-node08:07:590
1501399492,0cyclictest0-21swapper/009:43:030
1501399491,0cyclictest1799migration/010:47:440
141490,0rcu_preempt25-21ksoftirqd/109:12:471
141490,0rcu_preempt25-21ksoftirqd/107:22:441
103350490,0irq/28-eth0-tx-13010-21munin-node06:57:460
361480,0irq_work/3381rcuc/307:20:553
221480,0irq_work/1241rcuc/111:37:571
221480,0irq_work/1241rcuc/108:18:061
221480,0irq_work/1241rcuc/107:47:581
221480,0irq_work/1241rcuc/107:07:591
1501799483,0cyclictest210099rtkit-daemon12:07:371
1501399482,0cyclictest0-21swapper/011:07:320
15013994816,0cyclictest31930-21if_eth010:17:490
15013994816,0cyclictest24107-21forks11:17:490
15013994816,0cyclictest19504-21interrupts09:47:500
141480,0rcu_preempt39-21ksoftirqd/309:52:503
141480,0rcu_preempt25-21ksoftirqd/111:02:531
141480,0rcu_preempt25-21ksoftirqd/108:32:471
141480,0rcu_preempt25-21ksoftirqd/108:32:471
141480,0rcu_preempt25-21ksoftirqd/107:42:461
141480,0rcu_preempt13-21ksoftirqd/011:22:470
141480,0rcu_preempt13-21ksoftirqd/009:52:460
141480,0rcu_preempt13-21ksoftirqd/007:17:550
103250480,0irq/27-eth0-rx-14859-21munin-run07:02:323
361470,0irq_work/3381rcuc/311:07:123
361470,0irq_work/3381rcuc/308:28:003
15030994727,0cyclictest0-21swapper/310:45:543
15030994727,0cyclictest0-21swapper/310:45:543
15013994723,0cyclictest28500-21tr08:52:430
15013994720,0cyclictest31833-21grep08:58:030
15013994719,0cyclictest16786-21cut12:17:480
15013994718,0cyclictest12894-21if_err_eth012:07:510
15013994715,0cyclictest20945-21head08:32:560
15013994715,0cyclictest20945-21head08:32:560
141470,0rcu_preempt25-21ksoftirqd/107:57:481
141470,0rcu_preempt13-21ksoftirqd/012:25:490
361460,0irq_work/3381rcuc/312:02:473
221460,0irq_work/1241rcuc/110:42:591
221460,0irq_work/1241rcuc/110:42:591
181460,0irq_work/0161rcuc/010:07:320
1503099461,0cyclictest210099rtkit-daemon11:30:493
15013994627,0cyclictest3163-21munin-node10:27:540
15013994620,0cyclictest1234-21runrttasks11:35:040
15013994617,0cyclictest22629-21sed08:37:470
15013994617,0cyclictest15146-21cpu07:02:430
15013994615,0cyclictest3055-21swap11:43:000
141460,0rcu_preempt25052-21cat08:42:543
141460,0rcu_preempt25-21ksoftirqd/111:18:021
141460,0rcu_preempt25-21ksoftirqd/108:27:311
141460,0rcu_preempt13-21ksoftirqd/011:52:460
361450,0irq_work/3381rcuc/310:08:473
1503099457,0cyclictest3799migration/311:22:463
15013994527,0cyclictest3503-21munin-node07:52:590
15013994521,0cyclictest14726-21munin-node09:38:120
15013994521,0cyclictest14726-21munin-node09:38:120
1501399452,0cyclictest8115-21grep08:02:480
1501399452,0cyclictest14980-21python08:18:080
141450,0rcu_preempt25-21ksoftirqd/110:27:501
141450,0rcu_preempt16178-21unixbench_singl09:38:131
141450,0rcu_preempt16178-21unixbench_singl09:38:131
141450,0rcu_preempt13-21ksoftirqd/010:07:470
141450,0rcu_preempt13-21ksoftirqd/009:32:540
381440,0rcuc/319734-21needreboot09:47:553
15013994420,0cyclictest27757-21vmstat08:48:040
15013994414,0cyclictest23367-21munin-node07:22:560
15013994414,0cyclictest15611-21cpu10:57:430
1501399440,0cyclictest1799migration/011:41:100
141440,0rcu_preempt25-21ksoftirqd/112:27:501
141440,0rcu_preempt25-21ksoftirqd/111:53:011
141440,0rcu_preempt25-21ksoftirqd/111:33:031
141440,0rcu_preempt25-21ksoftirqd/109:52:451
361430,0irq_work/3381rcuc/312:25:543
361430,0irq_work/3381rcuc/309:12:583
221430,0irq_work/1241rcuc/111:27:591
1503099436,0cyclictest3799migration/311:45:483
15030994323,0cyclictest17583-21missed_timers09:42:533
1503099430,0cyclictest3799migration/310:52:483
15017994327,0cyclictest2817-21grep09:07:481
15013994320,0cyclictest1234-21runrttasks07:28:260
15013994320,0cyclictest1234-21runrttasks07:28:260
15013994315,0cyclictest1234-21runrttasks12:02:340
1501399431,0cyclictest18137-21grep08:27:440
141430,0rcu_preempt39-21ksoftirqd/307:07:543
141430,0rcu_preempt25051-21missed_timers08:42:541
141430,0rcu_preempt25-21ksoftirqd/111:52:311
141430,0rcu_preempt25-21ksoftirqd/111:52:311
141430,0rcu_preempt25-21ksoftirqd/107:17:561
141430,0rcu_preempt13887-21tune2fs10:52:471
1501799421,0cyclictest210099rtkit-daemon12:12:521
1501399422,0cyclictest8899-21nscd09:18:120
15013994217,0cyclictest19505-21fschecks_count07:12:520
141420,0rcu_preempt8090-21sed10:37:531
141420,0rcu_preempt39-21ksoftirqd/311:17:573
141420,0rcu_preempt39-21ksoftirqd/311:08:013
141420,0rcu_preempt39-21ksoftirqd/309:22:523
141420,0rcu_preempt25-21ksoftirqd/112:03:021
141420,0rcu_preempt25-21ksoftirqd/107:33:021
141420,0rcu_preempt25-21ksoftirqd/107:27:471
141420,0rcu_preempt25-21ksoftirqd/107:27:471
241410,0rcuc/10-21swapper/111:13:031
221410,0irq_work/1241rcuc/110:47:581
141410,0rcu_preempt3714-21taskset11:46:511
141410,0rcu_preempt28335-21switchtime10:08:001
141410,0rcu_preempt25-21ksoftirqd/110:17:431
141410,0rcu_preempt25-21ksoftirqd/109:28:161
141410,0rcu_preempt23383-21sed08:38:011
141410,0rcu_preempt20020-21switchtime09:48:011
1503099401,0cyclictest210099rtkit-daemon11:51:553
1503099401,0cyclictest210099rtkit-daemon11:51:553
1501799401,0cyclictest210099rtkit-daemon12:24:201
141400,0rcu_preempt773-21taskset09:02:511
141400,0rcu_preempt25-21ksoftirqd/109:33:111
141400,0rcu_preempt25-21ksoftirqd/108:12:561
141400,0rcu_preempt25-21ksoftirqd/107:52:541
361390,0irq_work/3381rcuc/309:12:173
15030993924,0cyclictest25191-21tr10:02:433
141390,0rcu_preempt25-21ksoftirqd/108:27:481
141390,0rcu_preempt20715-21kworker/1:109:57:471
141390,0rcu_preempt10882-21switchtime08:08:031
141380,0rcu_preempt27346-21zcat08:47:571
141340,0rcu_preempt0-21swapper/108:57:321
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional