You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-02 - 10:32

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot6.osadl.org (updated Sat Jul 02, 2022 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2911370,0irq_work/2311rcuc/219:40:332
2911340,0irq_work/2311rcuc/221:54:552
2911320,0irq_work/2311rcuc/222:49:502
2911300,0irq_work/2311rcuc/220:59:562
2911300,0irq_work/2311rcuc/219:39:552
2911290,0irq_work/2311rcuc/223:59:552
2911290,0irq_work/2311rcuc/221:14:572
2911260,0irq_work/2311rcuc/221:55:192
2911230,0irq_work/2311rcuc/220:00:062
2911180,0irq_work/2311rcuc/220:44:562
2911160,0irq_work/2311rcuc/223:11:132
2911160,0irq_work/2311rcuc/221:20:062
2911150,0irq_work/2311rcuc/222:20:232
2911150,0irq_work/2311rcuc/222:09:562
2911150,0irq_work/2311rcuc/219:14:552
2911150,0irq_work/2311rcuc/200:20:212
2911120,0irq_work/2311rcuc/222:55:122
2911120,0irq_work/2311rcuc/222:55:122
2911120,0irq_work/2311rcuc/221:09:562
2911110,0irq_work/2311rcuc/221:39:552
2911110,0irq_work/2311rcuc/220:39:312
2911110,0irq_work/2311rcuc/220:39:312
2911100,0irq_work/2311rcuc/219:20:202
2911100,0irq_work/2311rcuc/200:19:552
2911070,0irq_work/2311rcuc/223:20:132
2911070,0irq_work/2311rcuc/219:30:202
2911070,0irq_work/2311rcuc/200:05:182
2911060,0irq_work/2311rcuc/219:50:422
2911060,0irq_work/2311rcuc/200:10:302
2911040,0irq_work/2311rcuc/222:15:142
2911040,0irq_work/2311rcuc/221:40:222
2911030,0irq_work/2311rcuc/220:14:102
2911030,0irq_work/2311rcuc/200:25:202
2911020,0irq_work/2311rcuc/221:00:262
2911020,0irq_work/2311rcuc/220:29:552
2911020,0irq_work/2311rcuc/219:49:562
2911000,0irq_work/2311rcuc/223:39:512
2911000,0irq_work/2311rcuc/223:09:012
2911000,0irq_work/2311rcuc/219:26:082
2911000,0irq_work/2311rcuc/219:26:082
2211000,0irq_work/1241rcuc/123:59:551
291990,0irq_work/2311rcuc/222:31:292
291990,0irq_work/2311rcuc/220:30:212
291980,0irq_work/2311rcuc/223:40:112
291980,0irq_work/2311rcuc/222:40:122
291980,0irq_work/2311rcuc/221:05:552
291980,0irq_work/2311rcuc/200:00:212
291980,0irq_work/2311rcuc/200:00:212
291970,0irq_work/2311rcuc/223:00:212
291970,0irq_work/2311rcuc/220:15:222
291970,0irq_work/2311rcuc/220:08:032
291960,0irq_work/2311rcuc/223:15:202
291950,0irq_work/2311rcuc/223:29:552
291940,0irq_work/2311rcuc/222:54:152
291940,0irq_work/2311rcuc/222:54:152
291940,0irq_work/2311rcuc/220:20:162
291900,0irq_work/2311rcuc/219:18:472
291900,0irq_work/2311rcuc/200:30:182
291880,0irq_work/2311rcuc/220:49:552
291870,0irq_work/2311rcuc/222:00:172
361860,0irq_work/3381rcuc/319:05:083
291860,0irq_work/2311rcuc/222:39:512
291850,0irq_work/2311rcuc/222:25:082
361840,0irq_work/3381rcuc/320:49:143
291840,0irq_work/2311rcuc/223:45:022
291830,0irq_work/2311rcuc/222:10:232
291830,0irq_work/2311rcuc/221:45:552
291830,0irq_work/2311rcuc/221:45:552
291830,0irq_work/2311rcuc/221:25:122
291810,0irq_work/2311rcuc/223:52:502
291800,0irq_work/2311rcuc/220:50:172
361770,0irq_work/3381rcuc/322:05:193
361770,0irq_work/3381rcuc/321:59:553
291770,0irq_work/2311rcuc/219:55:182
291770,0irq_work/2311rcuc/200:35:062
291730,0irq_work/2311rcuc/223:30:252
291730,0irq_work/2311rcuc/221:34:332
361690,0irq_work/3381rcuc/319:40:053
361680,0irq_work/3381rcuc/320:26:253
361670,0irq_work/3381rcuc/319:35:113
361660,0irq_work/3381rcuc/322:00:223
361660,0irq_work/3381rcuc/320:20:223
361650,0irq_work/3381rcuc/322:25:173
361640,0irq_work/3381rcuc/322:34:573
361640,0irq_work/3381rcuc/321:35:273
721899632,0cyclictest210099rtkit-daemon00:10:323
361600,0irq_work/3381rcuc/323:50:083
7218995611,0cyclictest210099rtkit-daemon20:56:263
141560,0rcu_preempt25-21ksoftirqd/121:40:211
721899559,0cyclictest210099rtkit-daemon21:54:253
721599556,0cyclictest2399migration/121:31:301
361540,0irq_work/3381rcuc/300:15:213
141540,0rcu_preempt0-21swapper/219:05:382
721899531,0cyclictest210099rtkit-daemon00:28:063
721899521,0cyclictest210099rtkit-daemon00:20:323
721899521,0cyclictest210099rtkit-daemon00:08:323
721899521,0cyclictest210099rtkit-daemon00:01:353
721899521,0cyclictest210099rtkit-daemon00:01:353
221520,0irq_work/1241rcuc/120:05:121
721899514,0cyclictest210099rtkit-daemon19:54:253
141510,0rcu_preempt15942-21grep19:30:100
103350510,0irq/28-eth0-tx-5150-21munin-node19:05:110
721899504,0cyclictest210099rtkit-daemon20:00:383
721599509,0cyclictest2399migration/100:38:551
361500,0irq_work/3381rcuc/321:05:233
221500,0irq_work/1241rcuc/122:59:551
221500,0irq_work/1241rcuc/122:59:551
721899492,0cyclictest210099rtkit-daemon23:16:103
361490,0irq_work/3381rcuc/320:05:213
141490,0rcu_preempt13-21ksoftirqd/023:30:250
721599489,0cyclictest2399migration/119:48:061
7215994810,0cyclictest2399migration/120:19:031
241480,0rcuc/15382-21munin-node19:05:081
221480,0irq_work/1241rcuc/123:10:261
181480,0irq_work/0161rcuc/022:09:190
181480,0irq_work/0161rcuc/019:45:120
7218994730,0cyclictest8022-21expr21:45:243
7218994730,0cyclictest8022-21expr21:45:243
7218994730,0cyclictest7739-21sed19:10:113
721899471,0cyclictest210099rtkit-daemon21:40:153
721899471,0cyclictest210099rtkit-daemon19:33:523
721599478,0cyclictest2399migration/123:46:531
721599477,0cyclictest2399migration/122:45:091
721599476,0cyclictest2399migration/121:56:011
721599476,0cyclictest2399migration/121:54:171
361470,0irq_work/3381rcuc/320:32:253
361470,0irq_work/3381rcuc/300:34:503
721899461,0cyclictest210099rtkit-daemon23:59:323
721599469,0cyclictest2399migration/122:10:551
721599469,0cyclictest2399migration/120:53:541
721599469,0cyclictest2399migration/119:50:041
721599468,0cyclictest2399migration/122:42:571
721599466,0cyclictest2399migration/123:52:101
721599462,0cyclictest2399migration/119:55:041
381460,0rcuc/3141rcu_preempt23:00:073
7650450,0irq/26-ahci[000103350irq/28-eth0-tx-19:10:260
721599459,0cyclictest2399migration/123:15:361
721599457,0cyclictest2399migration/122:32:531
721599457,0cyclictest2399migration/120:25:091
721599454,0cyclictest2399migration/100:29:451
721599454,0cyclictest2399migration/100:14:121
721599451,0cyclictest210099rtkit-daemon19:12:251
141450,0rcu_preempt31521-21missed_timers21:25:143
141450,0rcu_preempt13-21ksoftirqd/000:14:560
103350450,0irq/28-eth0-tx-7650irq/26-ahci[00021:25:270
721599448,0cyclictest2399migration/100:07:111
721599447,0cyclictest2399migration/123:03:381
721599446,0cyclictest2399migration/121:35:001
721599445,0cyclictest2399migration/119:15:261
721599444,0cyclictest2399migration/122:54:031
721599444,0cyclictest2399migration/122:54:031
721599443,0cyclictest2399migration/119:33:421
221440,0irq_work/1241rcuc/100:30:191
181440,0irq_work/0161rcuc/023:25:190
141440,0rcu_preempt18406-21cpuspeed_turbos20:55:061
141440,0rcu_preempt13-21ksoftirqd/022:54:550
141440,0rcu_preempt13-21ksoftirqd/022:54:550
141440,0rcu_preempt13-21ksoftirqd/021:40:100
141440,0rcu_preempt13-21ksoftirqd/000:25:090
141440,0rcu_preempt13-21ksoftirqd/000:14:560
721899431,0cyclictest210099rtkit-daemon21:00:393
721599437,0cyclictest2399migration/121:06:291
721599434,0cyclictest2399migration/122:20:501
721599434,0cyclictest2399migration/119:41:381
721599433,0cyclictest2399migration/122:17:551
361430,0irq_work/3381rcuc/320:50:173
141430,0rcu_preempt13-21ksoftirqd/023:05:100
141430,0rcu_preempt13-21ksoftirqd/022:49:550
141430,0rcu_preempt13-21ksoftirqd/020:14:560
141430,0rcu_preempt0-21swapper/123:29:551
721599424,0cyclictest2399migration/120:40:561
721599424,0cyclictest2399migration/120:04:371
721599423,0cyclictest2399migration/122:27:061
721599423,0cyclictest2399migration/120:47:181
721599423,0cyclictest2399migration/120:21:401
721599423,0cyclictest2399migration/100:22:241
361420,0irq_work/3381rcuc/322:10:593
361420,0irq_work/3381rcuc/300:39:563
181420,0irq_work/0161rcuc/021:50:120
141420,0rcu_preempt25-21ksoftirqd/121:15:251
141420,0rcu_preempt13-21ksoftirqd/020:19:570
141420,0rcu_preempt10201-21cpuspeed_turbos20:35:061
141420,0rcu_preempt10201-21cpuspeed_turbos20:35:061
141420,0rcu_preempt0-21swapper/019:25:190
141420,0rcu_preempt0-21swapper/019:25:190
721899412,0cyclictest210099rtkit-daemon22:56:163
721899412,0cyclictest210099rtkit-daemon22:56:163
721899412,0cyclictest1639-21gltestperf22:50:123
721899412,0cyclictest1639-21gltestperf22:50:123
721599413,0cyclictest2399migration/123:22:221
721599413,0cyclictest2399migration/121:46:161
721599413,0cyclictest2399migration/121:46:161
361410,0irq_work/3381rcuc/323:49:263
241410,0rcuc/10-21swapper/121:25:141
141410,0rcu_preempt28635-21seq20:00:150
141410,0rcu_preempt25-21ksoftirqd/121:10:221
141410,0rcu_preempt19009-21latency20:55:140
721599404,0cyclictest2399migration/120:30:171
721599403,0cyclictest2399migration/123:39:221
721599403,0cyclictest2399migration/120:14:191
721599403,0cyclictest2399migration/119:22:211
361400,0irq_work/3381rcuc/320:39:553
361400,0irq_work/3381rcuc/320:39:553
181400,0irq_work/0161rcuc/021:00:050
181400,0irq_work/0161rcuc/019:50:190
141400,0rcu_preempt6335-21cut20:25:090
141400,0rcu_preempt21630-21gltestperf22:20:140
141400,0rcu_preempt13-21ksoftirqd/020:05:240
141400,0rcu_preempt0-21swapper/121:21:361
7218993926,0cyclictest13817-21cut19:25:083
7218993926,0cyclictest13817-21cut19:25:083
721899392,0cyclictest210099rtkit-daemon22:33:353
721599393,0cyclictest2399migration/121:00:021
7206993925,0cyclictest13752-21grep23:20:060
7206993923,0cyclictest29924-21sed22:40:090
7206993922,0cyclictest6321-21grep00:20:050
361390,0irq_work/3381rcuc/319:49:063
161390,0rcuc/00-21swapper/020:15:090
141390,0rcu_preempt29558-21cpuspeed_turbos22:40:053
141390,0rcu_preempt21970-21missed_timers22:20:193
141390,0rcu_preempt13339-21timerandwakeup20:40:230
141390,0rcu_preempt13-21ksoftirqd/023:35:120
141390,0rcu_preempt13-21ksoftirqd/022:25:130
141390,0rcu_preempt0-21swapper/323:40:133
103350390,0irq/28-eth0-tx-27340-21munin-node22:35:060
721899383,0cyclictest3799migration/323:22:353
7218993827,0cyclictest17922-21cpuspeed_turbos23:30:063
7206993824,0cyclictest827-21grep21:30:110
720699382,0cyclictest1799migration/020:30:110
720699380,0cyclictest1799migration/023:15:090
361380,0irq_work/3381rcuc/319:54:573
221380,0irq_work/1241rcuc/122:05:131
141380,0rcu_preempt0-21swapper/023:40:060
7218993721,0cyclictest9991-21grep23:10:103
721599374,0cyclictest2399migration/123:08:531
720699373,0cyclictest23976-21munin-node23:45:130
720699373,0cyclictest1799migration/021:14:230
720699372,0cyclictest1799migration/023:56:330
720699371,0cyclictest1799migration/021:20:060
181370,0irq_work/0161rcuc/023:10:080
141370,0rcu_preempt39-21ksoftirqd/322:45:063
141370,0rcu_preempt17160-21expr22:10:070
141370,0rcu_preempt13401-21grep22:00:101
141370,0rcu_preempt13-21ksoftirqd/019:20:050
141370,0rcu_preempt0-21swapper/020:50:060
103350370,0irq/28-eth0-tx-12379-21munin-node00:35:190
7218993622,0cyclictest9586-21cut19:15:113
7218993620,0cyclictest19185-21grep22:15:063
721599366,0cyclictest2399migration/123:40:091
7215993624,0cyclictest5398-21timerwakeupswit00:15:251
7215993622,0cyclictest27554-21df22:35:071
7206993630,0cyclictest30182-21munin-node00:00:170
7206993630,0cyclictest30182-21munin-node00:00:170
7206993624,0cyclictest18040-21grep19:35:100
7206993620,0cyclictest26947-21cat19:55:210
361360,0irq_work/3381rcuc/321:34:503
221360,0irq_work/1241rcuc/119:25:191
221360,0irq_work/1241rcuc/119:25:191
181360,0irq_work/0161rcuc/021:56:020
181360,0irq_work/0161rcuc/021:45:130
181360,0irq_work/0161rcuc/021:45:130
141360,0rcu_preempt13-21ksoftirqd/021:15:100
141360,0rcu_preempt13-21ksoftirqd/019:15:120
141360,0rcu_preempt10176-21cpuspeed_turbos20:35:060
141360,0rcu_preempt10176-21cpuspeed_turbos20:35:060
141360,0rcu_preempt0-21swapper/022:30:130
721899351,0cyclictest3799migration/321:10:103
7206993529,0cyclictest7009-21sshd23:02:030
7206993529,0cyclictest26042-21munin-node23:50:170
1799350,0migration/011316-21munin-node00:30:190
141350,0rcu_preempt39-21ksoftirqd/320:15:223
141350,0rcu_preempt0-21swapper/021:38:510
141350,0rcu_preempt0-21swapper/020:45:270
721899342,0cyclictest3799migration/323:05:053
721899342,0cyclictest210099rtkit-daemon20:10:153
7215993420,0cyclictest18547-21sed23:30:131
720699343,0cyclictest1799migration/022:57:320
720699343,0cyclictest1799migration/022:57:320
720699342,0cyclictest19088-21munin-node22:15:230
7206993419,0cyclictest13103-21sh22:00:070
361340,0irq_work/3381rcuc/319:21:543
181340,0irq_work/0161rcuc/000:05:180
141340,0rcu_preempt25-21ksoftirqd/119:39:561
141340,0rcu_preempt25-21ksoftirqd/100:04:221
141340,0rcu_preempt25-21ksoftirqd/100:04:221
7206993327,0cyclictest23248-21missed_timers21:05:150
141330,0rcu_preempt39-21ksoftirqd/320:40:233
141330,0rcu_preempt13-21ksoftirqd/019:40:180
7218993211,0cyclictest103250irq/27-eth0-rx-21:19:563
361310,0irq_work/3381rcuc/323:35:083
361310,0irq_work/3381rcuc/321:24:483
7218993019,0cyclictest16231-21grep23:25:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional