You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-29 - 21:27

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot6.osadl.org (updated Wed Mar 29, 2023 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3611460,0irq_work/3381rcuc/309:31:183
2911440,0irq_work/2311rcuc/209:56:192
3611370,0irq_work/3381rcuc/308:30:473
3611210,0irq_work/3381rcuc/312:05:473
3611190,0irq_work/3381rcuc/309:46:063
3611160,0irq_work/3381rcuc/310:07:163
3611150,0irq_work/3381rcuc/311:46:023
2211120,0irq_work/1241rcuc/111:40:421
3611110,0irq_work/3381rcuc/312:11:063
3611110,0irq_work/3381rcuc/309:11:203
2211110,0irq_work/1241rcuc/110:50:471
3611100,0irq_work/3381rcuc/311:26:133
3611090,0irq_work/3381rcuc/309:00:483
2211090,0irq_work/1241rcuc/110:10:461
3611080,0irq_work/3381rcuc/311:31:073
2211080,0irq_work/1241rcuc/110:31:151
3611060,0irq_work/3381rcuc/309:54:343
3611030,0irq_work/3381rcuc/307:50:473
3611020,0irq_work/3381rcuc/308:21:213
361990,0irq_work/3381rcuc/311:01:033
361990,0irq_work/3381rcuc/309:16:023
361990,0irq_work/3381rcuc/307:56:023
361990,0irq_work/3381rcuc/307:56:023
221990,0irq_work/1241rcuc/107:25:421
221980,0irq_work/1241rcuc/107:46:131
361970,0irq_work/3381rcuc/308:31:103
361970,0irq_work/3381rcuc/308:01:033
361970,0irq_work/3381rcuc/308:01:033
221970,0irq_work/1241rcuc/109:26:131
361960,0irq_work/3381rcuc/310:03:453
361960,0irq_work/3381rcuc/307:51:163
221960,0irq_work/1241rcuc/108:15:521
221960,0irq_work/1241rcuc/107:36:141
361950,0irq_work/3381rcuc/310:51:123
221950,0irq_work/1241rcuc/107:26:181
361940,0irq_work/3381rcuc/310:20:473
361940,0irq_work/3381rcuc/308:40:473
361920,0irq_work/3381rcuc/311:16:033
361920,0irq_work/3381rcuc/307:15:473
221920,0irq_work/1241rcuc/112:06:161
221920,0irq_work/1241rcuc/110:25:471
361900,0irq_work/3381rcuc/307:01:153
221890,0irq_work/1241rcuc/111:50:421
221890,0irq_work/1241rcuc/109:21:161
221890,0irq_work/1241rcuc/109:11:141
103350890,0irq/28-eth0-tx-7650irq/26-ahci[00010:41:140
361880,0irq_work/3381rcuc/312:10:473
221880,0irq_work/1241rcuc/108:50:481
361870,0irq_work/3381rcuc/312:30:243
361870,0irq_work/3381rcuc/312:30:243
361870,0irq_work/3381rcuc/312:00:473
361870,0irq_work/3381rcuc/310:26:463
361870,0irq_work/3381rcuc/308:10:473
361870,0irq_work/3381rcuc/307:45:473
221860,0irq_work/1241rcuc/111:51:151
221860,0irq_work/1241rcuc/110:56:151
221860,0irq_work/1241rcuc/109:20:471
361850,0irq_work/3381rcuc/308:56:463
361850,0irq_work/3381rcuc/308:56:463
221850,0irq_work/1241rcuc/110:16:121
221840,0irq_work/1241rcuc/109:50:421
361830,0irq_work/3381rcuc/310:40:473
361830,0irq_work/3381rcuc/309:30:473
361830,0irq_work/3381rcuc/308:50:033
361830,0irq_work/3381rcuc/308:16:033
361820,0irq_work/3381rcuc/310:56:033
291820,0irq_work/2311rcuc/211:56:582
361810,0irq_work/3381rcuc/308:51:073
361810,0irq_work/3381rcuc/308:45:423
361810,0irq_work/3381rcuc/308:12:453
221810,0irq_work/1241rcuc/111:16:141
221810,0irq_work/1241rcuc/108:30:471
361800,0irq_work/3381rcuc/309:41:513
361800,0irq_work/3381rcuc/307:36:023
221800,0irq_work/1241rcuc/109:55:421
221800,0irq_work/1241rcuc/107:20:421
361790,0irq_work/3381rcuc/307:10:253
221790,0irq_work/1241rcuc/112:00:481
221790,0irq_work/1241rcuc/111:21:131
221790,0irq_work/1241rcuc/111:21:131
221790,0irq_work/1241rcuc/108:20:481
361780,0irq_work/3381rcuc/312:25:403
361780,0irq_work/3381rcuc/311:23:453
361780,0irq_work/3381rcuc/311:23:453
221780,0irq_work/1241rcuc/109:31:141
361770,0irq_work/3381rcuc/310:41:203
361770,0irq_work/3381rcuc/309:21:023
361770,0irq_work/3381rcuc/307:26:103
361760,0irq_work/3381rcuc/310:23:263
361760,0irq_work/3381rcuc/310:11:133
361760,0irq_work/3381rcuc/310:11:133
361760,0irq_work/3381rcuc/307:25:403
221760,0irq_work/1241rcuc/112:25:471
221760,0irq_work/1241rcuc/112:25:471
221760,0irq_work/1241rcuc/108:35:481
361750,0irq_work/3381rcuc/311:38:103
361750,0irq_work/3381rcuc/311:11:093
361750,0irq_work/3381rcuc/309:55:553
361750,0irq_work/3381rcuc/309:40:473
361750,0irq_work/3381rcuc/307:31:493
221750,0irq_work/1241rcuc/107:55:481
221750,0irq_work/1241rcuc/107:55:481
161750,0rcuc/02850-21cstates07:16:040
221740,0irq_work/1241rcuc/110:00:471
221740,0irq_work/1241rcuc/107:31:161
291730,0irq_work/2311rcuc/211:30:412
221730,0irq_work/1241rcuc/112:11:171
221730,0irq_work/1241rcuc/110:36:031
221730,0irq_work/1241rcuc/109:41:021
361720,0irq_work/3381rcuc/312:16:403
361710,0irq_work/3381rcuc/309:09:403
361710,0irq_work/3381rcuc/309:09:403
291710,0irq_work/2311rcuc/209:46:202
221710,0irq_work/1241rcuc/110:10:521
221710,0irq_work/1241rcuc/110:10:521
361700,0irq_work/3381rcuc/311:10:463
361700,0irq_work/3381rcuc/310:33:403
221700,0irq_work/1241rcuc/108:40:481
361690,0irq_work/3381rcuc/310:47:033
2872899692,0cyclictest210099rtkit-daemon12:01:180
221690,0irq_work/1241rcuc/111:15:411
221690,0irq_work/1241rcuc/109:06:031
221690,0irq_work/1241rcuc/109:06:031
221690,0irq_work/1241rcuc/108:05:471
221690,0irq_work/1241rcuc/108:05:471
221690,0irq_work/1241rcuc/107:10:451
361680,0irq_work/3381rcuc/311:50:563
361680,0irq_work/3381rcuc/307:16:343
291680,0irq_work/2311rcuc/210:51:032
221680,0irq_work/1241rcuc/112:00:471
221670,0irq_work/1241rcuc/111:41:011
221660,0irq_work/1241rcuc/110:41:021
141660,0rcu_preempt13-21ksoftirqd/010:30:470
291650,0irq_work/2311rcuc/210:15:582
221650,0irq_work/1241rcuc/111:30:481
221650,0irq_work/1241rcuc/111:06:161
221650,0irq_work/1241rcuc/111:01:101
291640,0irq_work/2311rcuc/210:56:592
291640,0irq_work/2311rcuc/208:31:192
291640,0irq_work/2311rcuc/206:56:022
221640,0irq_work/1241rcuc/112:16:141
291630,0irq_work/2311rcuc/211:08:452
361620,0irq_work/3381rcuc/311:42:593
291610,0irq_work/2311rcuc/210:26:022
2873099612,0cyclictest210099rtkit-daemon09:07:352
2873099612,0cyclictest210099rtkit-daemon09:07:352
221610,0irq_work/1241rcuc/109:00:471
221610,0irq_work/1241rcuc/109:00:471
291600,0irq_work/2311rcuc/207:14:042
221600,0irq_work/1241rcuc/112:21:151
221600,0irq_work/1241rcuc/109:56:171
2872899592,0cyclictest210099rtkit-daemon07:32:140
221590,0irq_work/1241rcuc/110:26:181
221590,0irq_work/1241rcuc/107:51:031
221590,0irq_work/1241rcuc/107:01:021
291580,0irq_work/2311rcuc/209:26:462
221580,0irq_work/1241rcuc/108:31:131
221580,0irq_work/1241rcuc/107:11:231
291570,0irq_work/2311rcuc/211:41:012
221570,0irq_work/1241rcuc/108:46:021
2872899561,0cyclictest210099rtkit-daemon08:26:090
221560,0irq_work/1241rcuc/109:35:481
221560,0irq_work/1241rcuc/109:05:161
221560,0irq_work/1241rcuc/108:07:501
241540,0rcuc/11234-21runrttasks06:59:111
221540,0irq_work/1241rcuc/110:51:021
291530,0irq_work/2311rcuc/209:01:402
2872899532,0cyclictest210099rtkit-daemon07:59:440
2872899532,0cyclictest210099rtkit-daemon07:59:440
221530,0irq_work/1241rcuc/111:25:471
221530,0irq_work/1241rcuc/107:41:081
311520,0rcuc/21211-21snmpd11:19:332
291520,0irq_work/2311rcuc/212:22:452
291520,0irq_work/2311rcuc/208:56:272
291520,0irq_work/2311rcuc/208:56:272
221520,0irq_work/1241rcuc/108:11:231
291500,0irq_work/2311rcuc/212:06:082
2872899501,0cyclictest210099rtkit-daemon07:44:570
2872899491,0cyclictest210099rtkit-daemon09:25:470
2872899491,0cyclictest210099rtkit-daemon08:20:000
181490,0irq_work/0161rcuc/012:06:160
141490,0rcu_preempt13-21ksoftirqd/011:40:470
291480,0irq_work/2311rcuc/207:01:122
2872899481,0cyclictest210099rtkit-daemon11:03:160
2872899481,0cyclictest210099rtkit-daemon10:18:340
2872899481,0cyclictest210099rtkit-daemon08:40:100
141480,0rcu_preempt32-21ksoftirqd/211:36:192
291470,0irq_work/2311rcuc/208:21:032
28728994736,0cyclictest213332sleep008:01:000
28728994736,0cyclictest213332sleep008:01:000
2872899472,0cyclictest832-13audispd12:20:460
2872899471,0cyclictest210099rtkit-daemon08:11:020
2872899471,0cyclictest210099rtkit-daemon07:46:200
291460,0irq_work/2311rcuc/208:41:072
291460,0irq_work/2311rcuc/207:55:202
28728994614,0cyclictest23626-21fschecks_count08:06:050
2872899461,0cyclictest210099rtkit-daemon11:51:250
2872899461,0cyclictest210099rtkit-daemon11:44:010
141460,0rcu_preempt32-21ksoftirqd/207:56:022
141460,0rcu_preempt32-21ksoftirqd/207:56:022
291450,0irq_work/2311rcuc/210:46:032
2872899455,0cyclictest1799migration/009:51:040
2872899451,0cyclictest210099rtkit-daemon11:19:330
2872899451,0cyclictest210099rtkit-daemon10:46:330
2872899451,0cyclictest210099rtkit-daemon10:24:570
181450,0irq_work/0161rcuc/010:56:150
161450,0rcuc/026904-21munin-node06:56:010
291440,0irq_work/2311rcuc/212:26:082
291440,0irq_work/2311rcuc/212:26:082
291440,0irq_work/2311rcuc/208:46:132
2872899441,0cyclictest210099rtkit-daemon11:26:190
2872899441,0cyclictest210099rtkit-daemon09:39:470
141440,0rcu_preempt13-21ksoftirqd/009:00:470
141440,0rcu_preempt13-21ksoftirqd/009:00:470
381430,0rcuc/328006-21sed06:56:143
291430,0irq_work/2311rcuc/208:51:162
291430,0irq_work/2311rcuc/208:36:142
2872899430,0cyclictest1799migration/011:06:090
141430,0rcu_preempt32-21ksoftirqd/210:31:202
141430,0rcu_preempt13-21ksoftirqd/011:56:170
141430,0rcu_preempt13-21ksoftirqd/009:31:210
141430,0rcu_preempt10787-21perl10:11:022
141430,0rcu_preempt10787-21perl10:11:022
2873099422,0cyclictest210099rtkit-daemon10:04:262
2872899421,0cyclictest210099rtkit-daemon11:23:120
2872899421,0cyclictest210099rtkit-daemon11:23:120
181420,0irq_work/0161rcuc/011:46:140
141420,0rcu_preempt32-21ksoftirqd/210:36:202
141420,0rcu_preempt32-21ksoftirqd/207:16:202
141420,0rcu_preempt311rcuc/211:05:472
141420,0rcu_preempt17453-21grep07:51:050
291410,0irq_work/2311rcuc/212:11:162
291410,0irq_work/2311rcuc/211:11:132
291410,0irq_work/2311rcuc/207:46:052
291410,0irq_work/2141rcu_preempt07:31:062
28728994128,0cyclictest20867-21grep10:36:000
141410,0rcu_preempt20504-21taskset09:16:042
141410,0rcu_preempt13-21ksoftirqd/010:11:020
141410,0rcu_preempt13-21ksoftirqd/010:11:020
141410,0rcu_preempt13-21ksoftirqd/009:11:080
141410,0rcu_preempt13-21ksoftirqd/009:06:170
141410,0rcu_preempt13-21ksoftirqd/009:06:170
141410,0rcu_preempt13-21ksoftirqd/007:00:580
291400,0irq_work/2311rcuc/209:31:002
291400,0irq_work/2311rcuc/208:02:552
291400,0irq_work/2311rcuc/208:02:552
2873099405,0cyclictest3099migration/210:06:162
2873099403,0cyclictest3099migration/209:11:042
2872899402,0cyclictest1799migration/007:36:110
181400,0irq_work/0161rcuc/011:35:190
161400,0rcuc/08855-21gltestperf10:06:050
141400,0rcu_preempt32-21ksoftirqd/209:25:472
141400,0rcu_preempt13-21ksoftirqd/012:26:210
141400,0rcu_preempt13-21ksoftirqd/012:26:210
141400,0rcu_preempt13-21ksoftirqd/010:01:240
141400,0rcu_preempt13-21ksoftirqd/007:21:200
291390,0irq_work/2311rcuc/207:35:492
141390,0rcu_preempt13-21ksoftirqd/009:05:470
7650380,0irq/26-ahci[00031348-21munin-run08:25:470
291380,0irq_work/2311rcuc/208:06:062
2873099384,0cyclictest3099migration/207:06:072
2872899383,0cyclictest1799migration/009:16:030
2872899382,0cyclictest1234-21runrttasks09:41:150
141380,0rcu_preempt32-21ksoftirqd/211:21:182
141380,0rcu_preempt32-21ksoftirqd/211:21:182
141380,0rcu_preempt32-21ksoftirqd/210:41:022
141380,0rcu_preempt20593-21switchtime11:51:182
141380,0rcu_preempt0-21swapper/212:20:112
291370,0irq_work/2311rcuc/207:20:482
28728993722,0cyclictest1960-21grep07:11:230
2872899372,0cyclictest770-21grep12:21:220
2872899372,0cyclictest25505-21grep09:26:200
141370,0rcu_preempt32-21ksoftirqd/212:04:362
141370,0rcu_preempt31500-21timerandwakeup09:41:172
141370,0rcu_preempt13-21ksoftirqd/012:11:120
141370,0rcu_preempt13-21ksoftirqd/009:46:190
28728993621,0cyclictest18804-21grep10:31:010
181360,0irq_work/0161rcuc/007:06:130
141360,0rcu_preempt28732-21timerwakeupswit08:16:192
291350,0irq_work/2311rcuc/208:28:452
291350,0irq_work/2311rcuc/208:11:252
291350,0irq_work/2311rcuc/207:41:032
2873099351,0cyclictest11568-21fschecks_time11:31:072
141350,0rcu_preempt0-21swapper/008:52:360
7650340,0irq/26-ahci[0000-21swapper/010:51:170
2873099342,0cyclictest3099migration/210:23:512
2872899343,0cyclictest1799migration/008:31:070
28728993416,0cyclictest7666-21grep08:45:590
181340,0irq_work/0161rcuc/007:26:460
181340,0irq_work/0141rcu_preempt08:41:160
141340,0rcu_preempt13-21ksoftirqd/009:56:140
141340,0rcu_preempt0-21swapper/209:36:152
28730993322,0cyclictest18239-21tr11:46:122
291320,0irq_work/2311rcuc/207:29:552
28730993221,0cyclictest2593-21grep09:51:042
141320,0rcu_preempt4142-21timerandwakeup11:11:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional