You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-05 - 14:31

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot6.osadl.org (updated Tue Jul 05, 2022 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2211440,0irq_work/1241rcuc/109:14:481
1811420,0irq_work/0161rcuc/011:49:470
1811290,0irq_work/0161rcuc/011:29:480
1811190,0irq_work/0161rcuc/011:54:480
1811190,0irq_work/0161rcuc/011:54:480
2211150,0irq_work/1241rcuc/111:29:491
2211150,0irq_work/1241rcuc/110:09:491
2211150,0irq_work/1241rcuc/108:39:491
1811120,0irq_work/0161rcuc/012:34:490
1811120,0irq_work/0161rcuc/007:35:470
1811090,0irq_work/0161rcuc/008:09:490
3611080,0irq_work/3381rcuc/311:20:043
2911060,0irq_work/2311rcuc/210:40:152
2211050,0irq_work/1241rcuc/108:19:491
2211040,0irq_work/1241rcuc/112:29:481
2211020,0irq_work/1241rcuc/109:24:481
1811010,0irq_work/0161rcuc/011:44:470
1811000,0irq_work/0161rcuc/009:05:470
221980,0irq_work/1241rcuc/107:59:481
221980,0irq_work/1241rcuc/107:45:131
181980,0irq_work/0161rcuc/009:24:430
221960,0irq_work/1241rcuc/107:10:121
221950,0irq_work/1241rcuc/111:55:141
221950,0irq_work/1241rcuc/111:55:141
221950,0irq_work/1241rcuc/108:39:431
181950,0irq_work/0161rcuc/010:40:140
221930,0irq_work/1241rcuc/110:29:481
181930,0irq_work/0161rcuc/009:39:480
181930,0irq_work/0161rcuc/009:39:480
181920,0irq_work/0161rcuc/007:19:480
221910,0irq_work/1241rcuc/111:05:241
221900,0irq_work/1241rcuc/110:50:131
221900,0irq_work/1241rcuc/110:50:131
221900,0irq_work/1241rcuc/109:34:491
221900,0irq_work/1241rcuc/109:34:491
221900,0irq_work/1241rcuc/109:25:151
181900,0irq_work/0161rcuc/008:20:130
181900,0irq_work/0161rcuc/007:55:140
221890,0irq_work/1241rcuc/112:10:161
181890,0irq_work/0161rcuc/008:30:160
181890,0irq_work/0161rcuc/008:30:160
181890,0irq_work/0161rcuc/007:20:130
221880,0irq_work/1241rcuc/111:29:431
181880,0irq_work/0161rcuc/009:49:430
181880,0irq_work/0161rcuc/009:19:430
181880,0irq_work/0161rcuc/009:00:110
221870,0irq_work/1241rcuc/112:04:481
221870,0irq_work/1241rcuc/107:35:131
181870,0irq_work/0161rcuc/007:10:130
361860,0irq_work/3381rcuc/311:15:043
181850,0irq_work/0161rcuc/012:24:430
221840,0irq_work/1241rcuc/110:59:491
221840,0irq_work/1241rcuc/110:05:131
181840,0irq_work/0161rcuc/009:29:430
221830,0irq_work/1241rcuc/110:19:491
221830,0irq_work/1241rcuc/109:00:121
221830,0irq_work/1241rcuc/107:25:121
181830,0irq_work/0161rcuc/012:25:030
181830,0irq_work/0161rcuc/011:54:420
181830,0irq_work/0161rcuc/010:00:210
221820,0irq_work/1241rcuc/112:15:121
221820,0irq_work/1241rcuc/111:35:131
181820,0irq_work/0161rcuc/012:04:430
181820,0irq_work/0161rcuc/008:14:500
221810,0irq_work/1241rcuc/108:59:431
221810,0irq_work/1241rcuc/108:30:141
221810,0irq_work/1241rcuc/108:30:141
221810,0irq_work/1241rcuc/108:10:121
181810,0irq_work/0161rcuc/010:25:050
181810,0irq_work/0161rcuc/009:10:160
181810,0irq_work/0161rcuc/007:09:480
181800,0irq_work/0161rcuc/010:09:490
181800,0irq_work/0161rcuc/009:44:430
181800,0irq_work/0161rcuc/009:44:430
221790,0irq_work/1241rcuc/109:34:431
361780,0irq_work/3381rcuc/311:09:483
221780,0irq_work/1241rcuc/112:05:151
221780,0irq_work/1241rcuc/107:44:431
221770,0irq_work/1241rcuc/111:10:111
221770,0irq_work/1241rcuc/109:54:491
221770,0irq_work/1241rcuc/109:40:001
221770,0irq_work/1241rcuc/109:40:001
181770,0irq_work/0161rcuc/010:19:490
181770,0irq_work/0161rcuc/007:40:100
221760,0irq_work/1241rcuc/112:35:151
181760,0irq_work/0161rcuc/012:30:070
221750,0irq_work/1241rcuc/111:49:421
221750,0irq_work/1241rcuc/110:45:111
221750,0irq_work/1241rcuc/110:45:111
181750,0irq_work/0161rcuc/011:34:490
181750,0irq_work/0161rcuc/007:45:020
221740,0irq_work/1241rcuc/107:59:591
221740,0irq_work/1241rcuc/107:34:431
181740,0irq_work/0161rcuc/012:10:040
181740,0irq_work/0161rcuc/011:30:230
181740,0irq_work/0161rcuc/009:30:140
181740,0irq_work/0161rcuc/008:09:500
291730,0irq_work/2311rcuc/209:15:152
221730,0irq_work/1241rcuc/110:15:171
221730,0irq_work/1241rcuc/108:29:431
181730,0irq_work/0161rcuc/012:05:030
181730,0irq_work/0161rcuc/008:55:120
221720,0irq_work/1241rcuc/111:24:481
181720,0irq_work/0161rcuc/009:50:210
181720,0irq_work/0161rcuc/008:49:480
221710,0irq_work/1241rcuc/112:20:121
221710,0irq_work/1241rcuc/108:19:441
221710,0irq_work/1241rcuc/107:19:431
181710,0irq_work/0161rcuc/011:05:250
181710,0irq_work/0161rcuc/010:50:020
181710,0irq_work/0161rcuc/010:50:020
181710,0irq_work/0161rcuc/010:34:480
181710,0irq_work/0161rcuc/008:34:490
181710,0irq_work/0161rcuc/008:29:430
221700,0irq_work/1241rcuc/111:50:161
221700,0irq_work/1241rcuc/111:19:131
221700,0irq_work/1241rcuc/109:08:291
361690,0irq_work/3381rcuc/310:05:073
221690,0irq_work/1241rcuc/110:30:141
181690,0irq_work/0161rcuc/011:15:050
221680,0irq_work/1241rcuc/107:49:491
181680,0irq_work/0161rcuc/011:00:000
361670,0irq_work/3381rcuc/307:55:033
291670,0irq_work/2311rcuc/207:15:012
221670,0irq_work/1241rcuc/108:50:131
181670,0irq_work/0161rcuc/012:19:430
181670,0irq_work/0161rcuc/010:34:480
181670,0irq_work/0161rcuc/008:50:130
181660,0irq_work/0161rcuc/011:14:480
181660,0irq_work/0161rcuc/009:59:430
181660,0irq_work/0161rcuc/008:04:430
221650,0irq_work/1241rcuc/112:34:291
221650,0irq_work/1241rcuc/110:55:151
221650,0irq_work/1241rcuc/110:39:491
181650,0irq_work/0161rcuc/007:50:140
1685399652,0cyclictest210099rtkit-daemon10:50:202
1685399652,0cyclictest210099rtkit-daemon10:50:202
221640,0irq_work/1241rcuc/110:00:041
221640,0irq_work/1241rcuc/108:45:141
221630,0irq_work/1241rcuc/109:50:061
221630,0irq_work/1241rcuc/109:49:431
221630,0irq_work/1241rcuc/109:19:431
181630,0irq_work/0161rcuc/008:40:020
181630,0irq_work/0161rcuc/007:25:130
221620,0irq_work/1241rcuc/110:39:291
181620,0irq_work/0161rcuc/010:55:010
221610,0irq_work/1241rcuc/111:40:011
221610,0irq_work/1241rcuc/108:05:031
181610,0irq_work/0161rcuc/010:44:590
181610,0irq_work/0161rcuc/010:44:590
181610,0irq_work/0161rcuc/010:20:020
181600,0irq_work/0161rcuc/007:30:020
141600,0rcu_preempt39-21ksoftirqd/311:40:133
141600,0rcu_preempt39-21ksoftirqd/311:35:143
361590,0irq_work/3381rcuc/312:30:213
181590,0irq_work/0161rcuc/011:20:040
181590,0irq_work/0161rcuc/010:10:160
361580,0irq_work/3381rcuc/309:45:043
221580,0irq_work/1241rcuc/107:20:231
221580,0irq_work/1241rcuc/107:04:591
141580,0rcu_preempt39-21ksoftirqd/308:10:023
361570,0irq_work/3381rcuc/308:59:593
141570,0rcu_preempt14246-21kworker/3:007:10:163
361560,0irq_work/3381rcuc/309:35:153
361560,0irq_work/3381rcuc/309:35:153
1685399553,0cyclictest210099rtkit-daemon07:19:592
1685399551,0cyclictest210099rtkit-daemon12:00:182
141550,0rcu_preempt22901-21fschecks_time10:00:052
291540,0irq_work/2311rcuc/208:55:132
141530,0rcu_preempt39-21ksoftirqd/310:59:483
141530,0rcu_preempt39-21ksoftirqd/308:25:163
141530,0rcu_preempt39-21ksoftirqd/308:22:443
291520,0irq_work/2311rcuc/207:55:202
1685499522,0cyclictest210099rtkit-daemon12:01:383
141520,0rcu_preempt0-21swapper/207:06:002
291510,0irq_work/2311rcuc/209:58:222
1685399512,0cyclictest210099rtkit-daemon12:22:002
361500,0irq_work/3381rcuc/310:39:513
361500,0irq_work/3381rcuc/309:20:213
1685499501,0cyclictest210099rtkit-daemon12:09:003
1685499501,0cyclictest210099rtkit-daemon07:47:053
1685399501,0cyclictest210099rtkit-daemon07:41:182
141500,0rcu_preempt472-21sed11:40:192
141500,0rcu_preempt39-21ksoftirqd/311:45:253
1685499491,0cyclictest210099rtkit-daemon10:45:343
1685499491,0cyclictest210099rtkit-daemon10:45:343
1685499491,0cyclictest210099rtkit-daemon07:35:583
1685399491,0cyclictest210099rtkit-daemon12:28:482
1685399491,0cyclictest210099rtkit-daemon10:47:522
1685399491,0cyclictest210099rtkit-daemon10:47:522
141490,0rcu_preempt39-21ksoftirqd/309:09:493
141490,0rcu_preempt32-21ksoftirqd/209:05:202
141490,0rcu_preempt32-21ksoftirqd/208:50:072
141490,0rcu_preempt19791-21kworker/3:112:35:203
141490,0rcu_preempt13208-21ps09:35:152
141490,0rcu_preempt13208-21ps09:35:152
141490,0rcu_preempt11144-21ps09:30:132
291480,0irq_work/2311rcuc/209:23:092
1685499481,0cyclictest210099rtkit-daemon10:51:373
1685499481,0cyclictest210099rtkit-daemon10:51:373
1685399482,0cyclictest210099rtkit-daemon11:29:192
1685399482,0cyclictest210099rtkit-daemon07:31:242
1685399481,0cyclictest210099rtkit-daemon07:51:582
141480,0rcu_preempt30291-21processes07:40:123
103250480,0irq/27-eth0-rx-0-21swapper/307:04:593
361470,0irq_work/3381rcuc/311:28:103
1685499471,0cyclictest210099rtkit-daemon12:21:383
141470,0rcu_preempt39-21ksoftirqd/310:35:073
141470,0rcu_preempt32-21ksoftirqd/210:15:232
141470,0rcu_preempt32-21ksoftirqd/210:05:152
141470,0rcu_preempt16241-21cat09:44:492
361460,0irq_work/3381rcuc/309:19:493
1685499468,0cyclictest3799migration/309:55:533
1685499468,0cyclictest3799migration/308:37:073
1685399461,0cyclictest210099rtkit-daemon11:09:372
141460,0rcu_preempt21267-21cat12:30:232
141460,0rcu_preempt10109-21cat08:10:062
361450,0irq_work/3381rcuc/312:19:483
1685499458,0cyclictest3799migration/311:55:153
1685499458,0cyclictest3799migration/311:55:153
1685499457,0cyclictest3799migration/311:31:133
1685499457,0cyclictest3799migration/307:34:233
141450,0rcu_preempt32-21ksoftirqd/211:35:092
141450,0rcu_preempt32-21ksoftirqd/210:10:172
291440,0irq_work/2311rcuc/212:18:412
1685499447,0cyclictest3799migration/308:53:413
1685499447,0cyclictest3799migration/307:16:363
1685499444,0cyclictest3799migration/308:32:363
1685499444,0cyclictest3799migration/308:32:363
1685399441,0cyclictest210099rtkit-daemon11:00:262
1685399441,0cyclictest210099rtkit-daemon07:48:122
141440,0rcu_preempt39-21ksoftirqd/308:55:033
141440,0rcu_preempt39-21ksoftirqd/308:45:133
141440,0rcu_preempt32-21ksoftirqd/211:30:492
141440,0rcu_preempt32-21ksoftirqd/211:20:222
141440,0rcu_preempt32-21ksoftirqd/210:20:072
141440,0rcu_preempt32-21ksoftirqd/209:25:192
361430,0irq_work/3381rcuc/309:34:483
291430,0irq_work/2311rcuc/208:15:122
1685499436,0cyclictest3799migration/310:19:033
1685499436,0cyclictest3799migration/309:10:483
1685499435,0cyclictest3799migration/310:25:523
1685499435,0cyclictest3799migration/308:08:043
16853994327,0cyclictest8686-21grep08:05:172
141430,0rcu_preempt39-21ksoftirqd/309:25:083
141430,0rcu_preempt24059-21proc_pri07:25:123
141420,0rcu_preempt9780-21grep12:04:592
141420,0rcu_preempt39-21ksoftirqd/308:15:163
141420,0rcu_preempt32-21ksoftirqd/212:10:232
141420,0rcu_preempt32-21ksoftirqd/208:20:022
1685499411,0cyclictest210099rtkit-daemon07:53:533
16853994129,0cyclictest1228-21munin-node11:09:572
141410,0rcu_preempt39-21ksoftirqd/312:10:013
141410,0rcu_preempt39-21ksoftirqd/310:19:503
141410,0rcu_preempt39-21ksoftirqd/309:40:113
141410,0rcu_preempt39-21ksoftirqd/309:40:113
141410,0rcu_preempt39-21ksoftirqd/308:00:133
361400,0irq_work/3381rcuc/307:20:113
311400,0rcuc/216837-21cyclictest07:09:492
291400,0irq_work/2311rcuc/207:28:172
1685399401,0cyclictest3099migration/208:44:232
141400,0rcu_preempt32-21ksoftirqd/211:45:132
141400,0rcu_preempt19093-21tr09:50:132
311390,0rcuc/25397-21ls08:00:002
311390,0rcuc/20-21swapper/210:29:492
1685499392,0cyclictest3799migration/311:10:063
16853993927,0cyclictest15689-21grep08:24:592
1685399382,0cyclictest3099migration/212:37:172
141380,0rcu_preempt39-21ksoftirqd/311:00:033
141380,0rcu_preempt32-21ksoftirqd/210:34:482
141380,0rcu_preempt0-21swapper/308:44:483
1685499372,0cyclictest3799migration/309:50:043
1685399371,0cyclictest25774-21runrttasks08:49:492
141370,0rcu_preempt39-21ksoftirqd/312:25:083
141370,0rcu_preempt31007-21ps09:00:122
141370,0rcu_preempt18637-21ps08:30:162
141370,0rcu_preempt18637-21ps08:30:162
3099360,0migration/221405-21runrttasks08:38:222
1685499363,0cyclictest3799migration/311:49:583
16853993623,0cyclictest15710-21grep09:40:212
16853993623,0cyclictest15710-21grep09:40:212
1685399361,0cyclictest3099migration/211:55:042
1685399361,0cyclictest3099migration/211:55:042
141360,0rcu_preempt4897-21grep10:34:592
141360,0rcu_preempt381rcuc/310:00:203
361350,0irq_work/3381rcuc/310:10:133
1685399352,0cyclictest3099migration/211:50:212
16853993423,0cyclictest21782-21grep11:15:042
141340,0rcu_preempt39-21ksoftirqd/310:30:093
291330,0irq_work/2141rcu_preempt10:55:002
141330,0rcu_preempt32-21ksoftirqd/209:12:242
141320,0rcu_preempt32-21ksoftirqd/207:38:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional