You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-05 - 15:27

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot6.osadl.org (updated Mon Jun 05, 2023 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
181980,0irq_work/0161rcuc/011:19:060
361880,0irq_work/3381rcuc/308:59:023
361760,0irq_work/3381rcuc/307:38:563
361740,0irq_work/3381rcuc/307:26:113
291720,0irq_work/2311rcuc/211:48:552
735899716,0cyclictest1799migration/010:33:590
361710,0irq_work/3381rcuc/309:57:193
361690,0irq_work/3381rcuc/311:58:233
291680,0irq_work/2311rcuc/209:52:582
291680,0irq_work/2311rcuc/208:38:532
735899675,0cyclictest1799migration/007:28:530
361620,0irq_work/3381rcuc/306:59:253
361620,0irq_work/3381rcuc/306:59:253
735899611,0cyclictest210099rtkit-daemon10:23:130
735899611,0cyclictest210099rtkit-daemon10:23:130
291610,0irq_work/2311rcuc/209:14:142
291610,0irq_work/2311rcuc/209:14:142
221610,0irq_work/1241rcuc/108:04:131
221610,0irq_work/1241rcuc/108:04:131
181610,0irq_work/0161rcuc/006:53:530
103350610,0irq/28-eth0-tx-7650irq/26-ahci[00008:15:250
291600,0irq_work/2311rcuc/208:53:332
221600,0irq_work/1241rcuc/111:44:051
141600,0rcu_preempt28935-21grep10:24:081
735899595,0cyclictest1799migration/007:48:420
735899592,0cyclictest1234-21runrttasks11:55:030
361590,0irq_work/3381rcuc/311:43:593
361590,0irq_work/3381rcuc/310:58:533
291590,0irq_work/2311rcuc/208:23:552
181590,0irq_work/0161rcuc/011:17:290
737099581,0cyclictest210099rtkit-daemon12:24:102
291580,0irq_work/2311rcuc/211:11:592
141580,0rcu_preempt25712-21grep08:59:091
141580,0rcu_preempt12528-21wc07:09:091
7358995634,0cyclictest0-21swapper/006:58:330
7358995634,0cyclictest0-21swapper/006:58:330
735899562,0cyclictest14350-21munin-node08:33:560
361560,0irq_work/3381rcuc/310:04:133
141560,0rcu_preempt32-21ksoftirqd/211:58:552
735899553,0cyclictest1799migration/011:09:470
735899552,0cyclictest23225-21swap11:29:080
735899552,0cyclictest23225-21swap11:29:080
735899551,0cyclictest26094-21cpuspeed_turbos07:43:510
737499541,0cyclictest210099rtkit-daemon09:10:133
735899543,0cyclictest1799migration/008:56:050
361540,0irq_work/3381rcuc/311:24:013
291540,0irq_work/2311rcuc/209:47:562
291540,0irq_work/2311rcuc/208:17:372
737099534,0cyclictest210099rtkit-daemon12:08:572
735899535,0cyclictest1799migration/009:26:090
735899534,0cyclictest210099rtkit-daemon09:18:410
735899533,0cyclictest1799migration/010:06:070
735899533,0cyclictest1799migration/008:00:590
361530,0irq_work/3381rcuc/310:53:333
291530,0irq_work/2311rcuc/207:16:042
221530,0irq_work/1241rcuc/112:08:541
141530,0rcu_preempt3092-21needreboot11:59:000
737499521,0cyclictest210099rtkit-daemon11:08:193
737099521,0cyclictest210099rtkit-daemon08:58:592
735899524,0cyclictest1799migration/010:49:270
7358995220,0cyclictest7925-21forks10:54:050
7358995220,0cyclictest1234-21runrttasks10:44:480
735899522,0cyclictest1799migration/011:34:560
735899522,0cyclictest1799migration/011:34:560
361520,0irq_work/3381rcuc/311:09:073
291520,0irq_work/2311rcuc/207:53:552
181520,0irq_work/0161rcuc/008:46:360
141520,0rcu_preempt39-21ksoftirqd/310:33:483
737099511,0cyclictest210099rtkit-daemon08:58:092
736399511,0cyclictest210099rtkit-daemon07:04:561
735899512,0cyclictest4806-21munin-node09:29:090
735899512,0cyclictest23052-21runrttasks10:12:070
735899512,0cyclictest1799migration/009:33:570
735899512,0cyclictest1799migration/007:15:090
735899512,0cyclictest0-21swapper/012:09:030
735899512,0cyclictest0-21swapper/011:08:280
361510,0irq_work/3381rcuc/310:21:413
361510,0irq_work/3381rcuc/310:21:413
291510,0irq_work/2311rcuc/207:32:212
181510,0irq_work/0161rcuc/009:59:100
737499508,0cyclictest3799migration/311:40:563
737499501,0cyclictest210099rtkit-daemon12:03:073
737099501,0cyclictest210099rtkit-daemon11:44:192
735899503,0cyclictest1799migration/011:24:470
735899503,0cyclictest1799migration/008:42:120
7358995024,0cyclictest31639-21wc11:49:090
361500,0irq_work/3381rcuc/308:53:543
361500,0irq_work/3381rcuc/308:33:463
291500,0irq_work/2311rcuc/210:08:382
221500,0irq_work/1241rcuc/111:04:041
141500,0rcu_preempt19413-21grep11:19:122
737099491,0cyclictest210099rtkit-daemon12:07:092
737099491,0cyclictest210099rtkit-daemon09:04:192
737099491,0cyclictest210099rtkit-daemon07:36:592
736399491,0cyclictest210099rtkit-daemon09:24:541
735899494,0cyclictest10505-21munin-node12:18:550
735899493,0cyclictest17085-21unixbench_singl07:19:100
735899493,0cyclictest1799migration/008:19:310
7358994921,0cyclictest12709-21fschecks_time08:28:530
735899492,0cyclictest9981-21head07:03:570
735899492,0cyclictest0-21swapper/008:09:040
735899492,0cyclictest0-21swapper/008:09:040
141490,0rcu_preempt23997-21tr07:38:480
737499487,0cyclictest3799migration/309:37:333
737499481,0cyclictest210099rtkit-daemon12:26:163
737099481,0cyclictest210099rtkit-daemon12:17:192
735899487,0cyclictest832-13audispd07:38:330
735899484,0cyclictest4757-21gltestperf12:03:560
735899484,0cyclictest23517-21cpuspeed_turbos10:13:530
735899482,0cyclictest28830-21munin-node09:09:020
735899482,0cyclictest14137-21runrttasks12:25:190
7358994818,0cyclictest18703-21proc_pri07:24:100
7358994818,0cyclictest1320-21munin-node10:38:500
7358994818,0cyclictest1234-21runrttasks09:47:580
7358994813,0cyclictest12939-21grep07:09:150
361480,0irq_work/3381rcuc/308:04:003
361480,0irq_work/3381rcuc/308:04:003
291480,0irq_work/2311rcuc/211:53:572
291480,0irq_work/2311rcuc/209:22:512
221480,0irq_work/1241rcuc/108:13:321
221480,0irq_work/1241rcuc/108:13:321
181480,0irq_work/0161rcuc/010:29:230
141480,0rcu_preempt28127-21tr07:48:512
141480,0rcu_preempt25-21ksoftirqd/110:54:061
141480,0rcu_preempt0-21swapper/306:58:033
737099471,0cyclictest210099rtkit-daemon12:19:592
737099471,0cyclictest210099rtkit-daemon09:43:032
737099471,0cyclictest210099rtkit-daemon09:35:592
735899474,0cyclictest27023-21fschecks_count09:03:540
7358994721,0cyclictest2804-21cat08:04:070
7358994721,0cyclictest2804-21cat08:04:070
735899472,0cyclictest29537-21timerandwakeup11:44:090
735899472,0cyclictest0-21swapper/009:03:330
7358994719,0cyclictest8607-21cstates12:13:530
7358994719,0cyclictest1234-21runrttasks10:59:250
7358994716,0cyclictest30932-21cpu09:13:540
7358994716,0cyclictest30932-21cpu09:13:540
141470,0rcu_preempt16528-21cpuspeed_turbos08:38:503
737499468,0cyclictest3799migration/309:05:483
737499466,0cyclictest3799migration/308:14:523
737499464,0cyclictest3799migration/311:30:093
737499464,0cyclictest3799migration/311:30:093
737499461,0cyclictest210099rtkit-daemon07:37:403
736399467,0cyclictest2399migration/111:16:111
735899462,0cyclictest21975-21head08:49:100
7358994618,0cyclictest27594-21wc10:23:480
735899461,0cyclictest10224-21munin-node08:24:040
141460,0rcu_preempt39-21ksoftirqd/309:14:013
141460,0rcu_preempt39-21ksoftirqd/309:14:013
141460,0rcu_preempt32-21ksoftirqd/208:23:332
141460,0rcu_preempt0-21swapper/106:54:041
737499456,0cyclictest3799migration/311:22:203
737099451,0cyclictest210099rtkit-daemon09:09:392
736399455,0cyclictest2399migration/110:59:451
736399454,0cyclictest2399migration/108:32:551
735899453,0cyclictest0-21swapper/007:56:520
7358994524,0cyclictest14235-21sed09:49:060
735899452,0cyclictest16874-21runrttasks09:57:200
361450,0irq_work/3381rcuc/310:54:023
141450,0rcu_preempt32-21ksoftirqd/207:23:492
737499445,0cyclictest3799migration/312:05:483
736399447,0cyclictest2399migration/109:32:291
736399445,0cyclictest2399migration/109:09:391
7358994416,0cyclictest1234-21runrttasks11:43:120
361440,0irq_work/3381rcuc/309:49:043
361440,0irq_work/3381rcuc/308:28:343
311440,0rcuc/26184-21needreboot06:53:592
291440,0irq_work/2311rcuc/211:08:332
141440,0rcu_preempt32-21ksoftirqd/210:03:572
141440,0rcu_preempt32-21ksoftirqd/208:33:532
141440,0rcu_preempt17303-21grep11:14:142
141440,0rcu_preempt15252-21grep09:53:512
737499436,0cyclictest3799migration/310:12:393
7370994328,0cyclictest31003-21sed10:29:262
736399437,0cyclictest2399migration/110:31:311
736399436,0cyclictest2399migration/108:48:171
736399434,0cyclictest2399migration/110:51:431
736399434,0cyclictest2399migration/109:48:541
735899431,0cyclictest10643-21runrttasks09:42:000
361430,0irq_work/3381rcuc/308:23:553
361430,0irq_work/3381rcuc/307:54:123
291430,0irq_work/2311rcuc/209:59:152
291430,0irq_work/2311rcuc/207:09:092
141430,0rcu_preempt32-21ksoftirqd/210:58:542
141430,0rcu_preempt2835-21cpuspeed_turbos09:23:483
141430,0rcu_preempt26059-21grep07:43:512
141430,0rcu_preempt16119-21grep07:18:542
737499422,0cyclictest3799migration/309:28:513
737099424,0cyclictest3099migration/207:58:442
736399424,0cyclictest2399migration/110:18:331
736399424,0cyclictest2399migration/110:18:331
736399423,0cyclictest2399migration/110:04:321
736399423,0cyclictest2399migration/108:27:361
736399423,0cyclictest2399migration/107:54:121
361420,0irq_work/3141rcu_preempt10:42:533
311420,0rcuc/23351-21grep08:04:192
311420,0rcuc/23351-21grep08:04:192
291420,0irq_work/2311rcuc/211:28:552
291420,0irq_work/2311rcuc/211:28:552
141420,0rcu_preempt3626-21grep10:43:502
141420,0rcu_preempt27645-21grep10:23:482
737499412,0cyclictest210099rtkit-daemon07:13:123
737099412,0cyclictest210099rtkit-daemon11:42:042
736399417,0cyclictest2399migration/107:31:361
736399414,0cyclictest2399migration/110:37:311
736399414,0cyclictest2399migration/108:39:481
736399412,0cyclictest2399migration/110:14:491
736399411,0cyclictest2399migration/110:39:361
361410,0irq_work/3381rcuc/310:24:043
361410,0irq_work/3381rcuc/307:14:133
291410,0irq_work/2311rcuc/209:28:562
141410,0rcu_preempt39-21ksoftirqd/311:18:213
736399404,0cyclictest2399migration/109:22:551
736399402,0cyclictest2399migration/112:05:071
736399402,0cyclictest210099rtkit-daemon12:20:291
736399401,0cyclictest210099rtkit-daemon11:51:261
361400,0irq_work/3381rcuc/312:13:553
361400,0irq_work/3381rcuc/308:48:543
291400,0irq_work/2311rcuc/207:42:292
141400,0rcu_preempt39-21ksoftirqd/311:49:043
141400,0rcu_preempt31811-21grep10:33:482
737499394,0cyclictest3799migration/310:45:003
736399394,0cyclictest2399migration/111:25:381
736399393,0cyclictest2399migration/107:25:281
291390,0irq_work/2311rcuc/207:02:152
291390,0irq_work/2311rcuc/207:02:152
141390,0rcu_preempt5689-21grep10:48:492
141390,0rcu_preempt39-21ksoftirqd/309:19:113
141390,0rcu_preempt32-21ksoftirqd/210:14:192
141390,0rcu_preempt25-21ksoftirqd/109:59:011
737099385,0cyclictest3099migration/208:44:332
736399383,0cyclictest2399migration/110:12:511
736399383,0cyclictest2399migration/108:51:551
736399382,0cyclictest2399migration/112:14:121
736399381,0cyclictest24715-21missed_timers07:38:581
736399380,0cyclictest24442-21grep11:33:531
736399380,0cyclictest24442-21grep11:33:531
291380,0irq_work/2311rcuc/210:39:142
291380,0irq_work/2311rcuc/208:29:512
141380,0rcu_preempt39-21ksoftirqd/308:44:103
141380,0rcu_preempt39-21ksoftirqd/307:43:563
141380,0rcu_preempt25-21ksoftirqd/109:39:011
737099374,0cyclictest3099migration/211:23:592
737099372,0cyclictest3099migration/211:33:382
737099372,0cyclictest3099migration/211:33:382
737099371,0cyclictest210099rtkit-daemon10:56:132
736399373,0cyclictest2399migration/107:46:201
7363993725,0cyclictest2695-21grep11:58:531
736399372,0cyclictest2399migration/111:53:541
141370,0rcu_preempt39-21ksoftirqd/308:12:263
141370,0rcu_preempt39-21ksoftirqd/308:12:263
7374993623,0cyclictest20632-21head07:28:593
7374993622,0cyclictest24647-21tail11:33:553
7374993622,0cyclictest24647-21tail11:33:553
737099361,0cyclictest3099migration/210:21:172
737099361,0cyclictest3099migration/210:21:172
736399363,0cyclictest2399migration/109:54:481
736399361,0cyclictest2399migration/108:16:441
241360,0rcuc/17146-21latency_hist06:58:331
241360,0rcuc/17146-21latency_hist06:58:331
141360,0rcu_preempt39-21ksoftirqd/309:39:423
141360,0rcu_preempt2840-21tr09:23:482
141360,0rcu_preempt10306-21seq07:04:023
141360,0rcu_preempt0-21swapper/109:13:551
141360,0rcu_preempt0-21swapper/109:13:551
737499356,0cyclictest3799migration/310:33:323
7374993527,0cyclictest15890-21fschecks_count07:18:533
736399354,0cyclictest2399migration/108:53:511
7363993525,0cyclictest4015-21grep10:43:551
7363993523,0cyclictest7366-21sed09:33:541
7363993522,0cyclictest32633-21grep07:58:561
736399352,0cyclictest2399migration/107:51:171
736399352,0cyclictest2399migration/107:22:091
7363993519,0cyclictest8208-21cpu08:18:501
736399351,0cyclictest2399migration/107:37:401
361350,0irq_work/3381rcuc/309:45:413
221350,0irq_work/1241rcuc/111:18:551
141350,0rcu_preempt25-21ksoftirqd/109:07:491
737099341,0cyclictest1211-21snmpd08:12:362
737099341,0cyclictest1211-21snmpd08:12:362
7363993421,0cyclictest14526-21ntp_states07:14:081
7363993420,0cyclictest22206-21grep11:28:511
7363993420,0cyclictest22206-21grep11:28:511
7363993419,0cyclictest14405-21cpu08:33:491
141340,0rcu_preempt7518-21sed12:09:073
141340,0rcu_preempt39-21ksoftirqd/308:19:113
141340,0rcu_preempt39-21ksoftirqd/308:02:073
141340,0rcu_preempt25-21ksoftirqd/111:43:281
141340,0rcu_preempt11698-21awk12:19:083
7363993318,0cyclictest12468-21awk09:44:121
361330,0irq_work/3381rcuc/310:02:403
736399322,0cyclictest2399migration/112:24:591
7363993216,0cyclictest15683-21crond11:13:331
141320,0rcu_preempt29134-21seq07:49:073
737099312,0cyclictest3099migration/207:03:592
361310,0irq_work/3141rcu_preempt10:13:573
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional