You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-07 - 21:25

x86 Intel Atom E3845 @1910 MHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rackdslot2.osadl.org (updated Wed Jun 07, 2023 12:46:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
345021460,4sleep01521699cyclictest12:30:290
3003321210,8sleep01521699cyclictest08:50:220
1339911100,4ptp4l401ktimersoftd/307:05:383
15217997870,4cyclictest25-21ksoftirqd/107:15:181
15217997459,12cyclictest25-21ksoftirqd/107:20:181
57202730,4sleep31313-21lldpd10:15:463
15217997366,4cyclictest25-21ksoftirqd/109:00:181
15217997263,5cyclictest25-21ksoftirqd/110:50:221
15217997263,5cyclictest25-21ksoftirqd/107:30:151
15217997263,5cyclictest25-21ksoftirqd/107:30:141
15217997157,11cyclictest25-21ksoftirqd/111:30:191
302312680,3sleep20-21swapper/207:40:352
133991680,2ptp4l401ktimersoftd/307:20:283
15217996759,4cyclictest25-21ksoftirqd/111:10:211
15217996655,5cyclictest25-21ksoftirqd/110:35:191
115522660,2sleep30-21swapper/310:30:213
15217996556,5cyclictest25-21ksoftirqd/111:15:231
15217996456,4cyclictest25-21ksoftirqd/109:35:211
133991640,1ptp4l401ktimersoftd/308:08:033
15217996355,4cyclictest25-21ksoftirqd/110:10:201
15217996353,5cyclictest25-21ksoftirqd/110:25:211
15217996353,5cyclictest25-21ksoftirqd/110:25:201
133991630,2ptp4l391rcuc/307:51:203
133991630,1ptp4l401ktimersoftd/312:33:193
15217996149,8cyclictest25-21ksoftirqd/108:55:181
133991610,1ptp4l401ktimersoftd/309:32:573
133991610,1ptp4l401ktimersoftd/309:32:573
133991590,1ptp4l401ktimersoftd/308:17:463
133991580,1ptp4l401ktimersoftd/307:12:163
133991570,2ptp4l401ktimersoftd/310:13:303
133991570,1ptp4l401ktimersoftd/308:29:263
15218995548,4cyclictest33-21ksoftirqd/209:40:212
15217995542,6cyclictest25-21ksoftirqd/108:05:131
133991550,1ptp4l401ktimersoftd/311:59:153
55498520,2rtkit-daemon0-21swapper/109:20:171
15218995244,4cyclictest33-21ksoftirqd/209:55:192
1280025132,8sleep00-21swapper/007:05:120
15218994943,4cyclictest33-21ksoftirqd/208:25:192
252442480,8sleep31521999cyclictest12:09:593
15217994842,3cyclictest25-21ksoftirqd/112:25:191
15217994741,4cyclictest25-21ksoftirqd/111:40:201
133991470,2ptp4l11558-21kworker/3:208:10:223
55498460,8rtkit-daemon553-21rtkit-daemon07:05:422
1521899453,3cyclictest121rcu_preempt10:39:522
15217994537,4cyclictest25-21ksoftirqd/112:10:031
15217994537,4cyclictest25-21ksoftirqd/108:40:041
133991450,2ptp4l10768-21apt-get09:20:013
133991450,2ptp4l10768-21apt-get09:20:003
55498440,10rtkit-daemon1274-21snmpd10:22:591
55498440,10rtkit-daemon1274-21snmpd10:22:591
15217994438,4cyclictest25-21ksoftirqd/107:50:351
15217994431,3cyclictest25-21ksoftirqd/107:55:201
15217994431,10cyclictest25-21ksoftirqd/112:00:241
1521699445,7cyclictest121rcu_preempt07:13:300
15218994337,4cyclictest33-21ksoftirqd/211:35:042
15217994337,4cyclictest25-21ksoftirqd/109:25:311
15217994330,10cyclictest25-21ksoftirqd/112:35:351
251422420,8sleep31521999cyclictest07:30:213
251422420,8sleep31521999cyclictest07:30:203
15218994234,5cyclictest33-21ksoftirqd/210:40:202
1521799428,4cyclictest25-21ksoftirqd/108:30:331
15217994236,4cyclictest25-21ksoftirqd/110:40:311
15217994232,6cyclictest25-21ksoftirqd/111:25:211
1338914233,4phc2sys41-21ksoftirqd/309:55:013
55498410,26rtkit-daemon1274-21snmpd09:42:160
15217994135,4cyclictest25-21ksoftirqd/107:13:281
15217994134,4cyclictest25-21ksoftirqd/111:40:001
15217994134,4cyclictest25-21ksoftirqd/109:55:251
15217994129,10cyclictest25-21ksoftirqd/109:45:401
1477724113,17sleep1204-21systemd-journal07:08:471
15219994020,5cyclictest41-21ksoftirqd/310:39:523
15219994020,5cyclictest41-21ksoftirqd/307:26:403
15217994035,3cyclictest25-21ksoftirqd/110:30:361
15217994035,3cyclictest25-21ksoftirqd/109:10:321
15217994034,4cyclictest25-21ksoftirqd/112:05:221
15217994034,4cyclictest25-21ksoftirqd/110:45:451
15217994034,4cyclictest25-21ksoftirqd/108:00:381
15217994034,3cyclictest25-21ksoftirqd/111:00:381
15217994033,4cyclictest25-21ksoftirqd/111:50:331
15217994033,4cyclictest25-21ksoftirqd/107:45:211
15217994033,4cyclictest25-21ksoftirqd/107:45:011
15217994032,4cyclictest25-21ksoftirqd/110:00:251
15217994028,10cyclictest25-21ksoftirqd/108:35:301
133991400,1ptp4l401ktimersoftd/311:49:173
55498390,4rtkit-daemon16311-21rs:main0
15219993926,10cyclictest41-21ksoftirqd/310:28:353
15219993926,10cyclictest41-21ksoftirqd/310:28:353
1521899398,29cyclictest33-21ksoftirqd/207:57:322
15218993933,4cyclictest33-21ksoftirqd/207:15:002
15218993932,3cyclictest33-21ksoftirqd/212:15:242
15217993934,3cyclictest25-21ksoftirqd/110:20:001
15217993933,4cyclictest25-21ksoftirqd/109:35:001
15217993933,4cyclictest25-21ksoftirqd/109:34:591
15217993932,4cyclictest25-21ksoftirqd/108:10:371
1521699390,3cyclictest121rcu_preempt10:20:510
1521699390,3cyclictest121rcu_preempt10:20:500
133991390,2ptp4l11558-21kworker/3:207:58:253
133991390,1ptp4l401ktimersoftd/310:53:173
133991390,1ptp4l401ktimersoftd/310:00:263
133991390,1ptp4l401ktimersoftd/307:42:063
15218993832,4cyclictest33-21ksoftirqd/207:30:252
15218993832,4cyclictest33-21ksoftirqd/207:30:252
15218993831,4cyclictest33-21ksoftirqd/208:45:002
15217993833,3cyclictest25-21ksoftirqd/111:10:001
15217993833,3cyclictest25-21ksoftirqd/109:05:301
15217993833,3cyclictest25-21ksoftirqd/108:15:291
15217993832,4cyclictest25-21ksoftirqd/111:45:241
15217993832,4cyclictest25-21ksoftirqd/110:05:191
15217993832,4cyclictest25-21ksoftirqd/108:50:161
15217993832,3cyclictest25-21ksoftirqd/108:45:271
1521699383,3cyclictest121rcu_preempt10:55:370
133991380,2ptp4l391rcuc/308:03:173
133991380,1ptp4l401ktimersoftd/312:27:013
133991380,1ptp4l401ktimersoftd/312:20:403
133991380,1ptp4l401ktimersoftd/310:05:123
133991380,1ptp4l401ktimersoftd/309:05:153
133991380,1ptp4l401ktimersoftd/308:35:063
133991380,1ptp4l391rcuc/308:33:303
55498370,6rtkit-daemon1198-21latency_hist07:50:000
15218993730,4cyclictest33-21ksoftirqd/212:33:482
1521799374,4cyclictest121rcu_preempt11:20:261
15217993732,3cyclictest25-21ksoftirqd/109:40:381
15217993732,3cyclictest25-21ksoftirqd/108:20:351
15217993731,4cyclictest25-21ksoftirqd/112:30:351
15217993731,4cyclictest25-21ksoftirqd/112:20:261
15217993731,4cyclictest25-21ksoftirqd/111:55:311
15217993731,4cyclictest25-21ksoftirqd/108:25:311
15217993731,4cyclictest25-21ksoftirqd/107:25:171
15217993731,3cyclictest25-21ksoftirqd/107:35:201
15217993726,3cyclictest25-21ksoftirqd/109:50:051
15216993730,4cyclictest9-21ksoftirqd/009:35:430
133991370,1ptp4l401ktimersoftd/312:12:283
133991370,1ptp4l401ktimersoftd/311:21:463
133991370,1ptp4l401ktimersoftd/311:11:123
133991370,1ptp4l401ktimersoftd/310:41:073
133991370,1ptp4l401ktimersoftd/309:12:503
133991370,1ptp4l401ktimersoftd/309:04:173
133991370,1ptp4l401ktimersoftd/308:42:303
15219993629,4cyclictest41-21ksoftirqd/308:45:263
15218993630,4cyclictest33-21ksoftirqd/208:45:262
15218993627,6cyclictest33-21ksoftirqd/212:00:292
15217993631,3cyclictest25-21ksoftirqd/111:00:011
15216993630,3cyclictest9-21ksoftirqd/008:00:000
133991360,1ptp4l401ktimersoftd/312:16:043
133991360,1ptp4l401ktimersoftd/311:52:123
133991360,1ptp4l401ktimersoftd/311:34:103
133991360,1ptp4l401ktimersoftd/309:45:023
133991360,1ptp4l401ktimersoftd/309:44:213
133991360,1ptp4l401ktimersoftd/309:27:053
133991360,1ptp4l391rcuc/311:03:123
133991360,12ptp4l401ktimersoftd/307:46:103
1338913628,4phc2sys41-21ksoftirqd/307:35:363
15219993519,4cyclictest41-21ksoftirqd/308:50:223
15218993529,4cyclictest33-21ksoftirqd/210:25:222
15218993529,4cyclictest33-21ksoftirqd/210:25:222
15218993529,3cyclictest33-21ksoftirqd/207:55:012
15218993528,4cyclictest33-21ksoftirqd/211:20:202
15218993528,4cyclictest33-21ksoftirqd/209:50:012
15218993528,4cyclictest33-21ksoftirqd/209:20:002
15218993528,4cyclictest33-21ksoftirqd/209:20:002
15218993528,4cyclictest33-21ksoftirqd/208:38:392
15218993528,4cyclictest33-21ksoftirqd/208:15:192
15217993530,3cyclictest25-21ksoftirqd/109:15:371
15217993530,3cyclictest25-21ksoftirqd/109:15:361
15217993528,4cyclictest25-21ksoftirqd/112:15:261
15216993529,4cyclictest9-21ksoftirqd/008:10:320
1521699350,4cyclictest121rcu_preempt07:24:040
133991350,1ptp4l401ktimersoftd/312:37:043
133991350,1ptp4l401ktimersoftd/312:03:493
133991350,1ptp4l401ktimersoftd/311:40:553
133991350,1ptp4l401ktimersoftd/311:16:053
133991350,1ptp4l401ktimersoftd/310:20:173
133991350,1ptp4l401ktimersoftd/310:20:173
133991350,1ptp4l401ktimersoftd/309:20:563
133991350,1ptp4l401ktimersoftd/308:22:093
133991350,12ptp4l401ktimersoftd/311:08:213
133991350,10ptp4l401ktimersoftd/310:58:523
54922340,3sleep3121rcu_preempt11:25:313
15218993428,4cyclictest33-21ksoftirqd/210:55:212
15218993428,4cyclictest33-21ksoftirqd/209:00:252
15218993428,4cyclictest33-21ksoftirqd/208:05:282
15218993428,3cyclictest33-21ksoftirqd/212:05:402
15218993427,4cyclictest33-21ksoftirqd/210:45:372
15218993427,4cyclictest33-21ksoftirqd/210:20:412
15218993427,4cyclictest33-21ksoftirqd/210:20:402
15218993427,4cyclictest33-21ksoftirqd/208:50:242
15216993427,4cyclictest9-21ksoftirqd/011:41:580
133991340,1ptp4l401ktimersoftd/310:48:423
133991340,1ptp4l401ktimersoftd/308:59:313
55498330,3rtkit-daemon1274-21snmpd11:09:400
15218993327,4cyclictest33-21ksoftirqd/211:25:392
15218993327,4cyclictest33-21ksoftirqd/210:10:012
15218993327,4cyclictest33-21ksoftirqd/207:20:212
15218993326,4cyclictest33-21ksoftirqd/211:50:002
15218993326,4cyclictest33-21ksoftirqd/209:10:362
15218993325,6cyclictest33-21ksoftirqd/211:10:282
1521899331,4cyclictest1313-21lldpd07:45:112
15216993328,3cyclictest9-21ksoftirqd/012:10:300
15216993326,4cyclictest9-21ksoftirqd/011:33:420
1521699332,3cyclictest121rcu_preempt10:45:240
133991330,1ptp4l401ktimersoftd/309:58:043
55498320,2rtkit-daemon0-21swapper/008:38:440
15219993227,3cyclictest41-21ksoftirqd/311:35:403
15218993227,3cyclictest33-21ksoftirqd/209:05:272
15218993226,4cyclictest33-21ksoftirqd/211:59:122
15218993226,3cyclictest33-21ksoftirqd/210:54:562
15218993225,4cyclictest33-21ksoftirqd/209:22:442
15218993224,4cyclictest33-21ksoftirqd/208:24:142
1521699323,3cyclictest121rcu_preempt12:05:440
15216993227,3cyclictest9-21ksoftirqd/008:00:260
15216993226,4cyclictest9-21ksoftirqd/009:25:290
15216993225,4cyclictest9-21ksoftirqd/011:55:030
15216993225,4cyclictest9-21ksoftirqd/011:48:480
15216993225,4cyclictest9-21ksoftirqd/009:55:250
15216993225,4cyclictest9-21ksoftirqd/008:07:570
15216993225,4cyclictest9-21ksoftirqd/007:52:390
15216993225,4cyclictest9-21ksoftirqd/007:30:340
15216993225,4cyclictest9-21ksoftirqd/007:30:340
133991320,1ptp4l401ktimersoftd/307:16:343
518923124,3sleep09-21ksoftirqd/010:15:400
213062310,4sleep220117-21/usr/sbin/munin08:30:312
15218993124,4cyclictest33-21ksoftirqd/211:00:272
15218993124,4cyclictest33-21ksoftirqd/207:28:422
1521899311,27cyclictest1313-21lldpd12:25:222
15216993126,3cyclictest9-21ksoftirqd/010:00:420
15216993125,4cyclictest9-21ksoftirqd/011:21:010
15216993125,4cyclictest9-21ksoftirqd/009:20:290
15216993124,4cyclictest9-21ksoftirqd/011:35:390
15216993124,4cyclictest9-21ksoftirqd/011:00:240
15216993124,4cyclictest9-21ksoftirqd/007:40:410
1521699311,6cyclictest1274-21snmpd08:40:180
2667123021,4sleep09-21ksoftirqd/009:51:150
1521899303,3cyclictest121rcu_preempt09:26:482
15218993024,4cyclictest33-21ksoftirqd/210:00:342
15218993023,4cyclictest33-21ksoftirqd/212:15:002
15218993023,4cyclictest33-21ksoftirqd/208:56:142
15218993020,8cyclictest33-21ksoftirqd/209:51:022
15218993019,4cyclictest33-21ksoftirqd/211:40:022
15218993012,15cyclictest1274-21snmpd08:02:382
1521699305,3cyclictest121rcu_preempt08:25:000
15216993025,3cyclictest9-21ksoftirqd/007:15:290
15216993024,4cyclictest9-21ksoftirqd/012:03:260
15216993024,3cyclictest9-21ksoftirqd/009:08:410
15216993023,4cyclictest9-21ksoftirqd/009:45:110
55498290,2rtkit-daemon0-21swapper/010:08:540
15218992924,3cyclictest33-21ksoftirqd/207:15:282
15218992923,4cyclictest33-21ksoftirqd/207:35:462
15218992922,4cyclictest33-21ksoftirqd/211:31:312
15218992922,4cyclictest33-21ksoftirqd/208:10:182
15218992920,5cyclictest1274-21snmpd10:32:552
1521899290,3cyclictest0-21swapper/212:37:522
15218992822,4cyclictest33-21ksoftirqd/209:30:312
15218992822,4cyclictest33-21ksoftirqd/209:30:312
15218992822,3cyclictest17229-21missed_timers11:50:362
15216992822,4cyclictest9-21ksoftirqd/012:27:480
15216992822,4cyclictest9-21ksoftirqd/009:00:130
15216992821,4cyclictest9-21ksoftirqd/009:31:340
15216992821,4cyclictest9-21ksoftirqd/009:31:340
1521699281,3cyclictest121rcu_preempt08:55:280
15216992811,9cyclictest204-21systemd-journal07:35:220
133991280,1ptp4l401ktimersoftd/309:35:453
15218992717,6cyclictest204-21systemd-journal11:05:222
15218992715,8cyclictest15131-21apache212:20:202
1521699279,9cyclictest121rcu_preempt08:30:360
15216992722,3cyclictest9-21ksoftirqd/008:19:460
15216992721,4cyclictest9-21ksoftirqd/010:44:520
15216992720,4cyclictest9-21ksoftirqd/009:18:490
15216992720,4cyclictest9-21ksoftirqd/009:18:490
15216992720,4cyclictest9-21ksoftirqd/007:29:590
15218992621,3cyclictest19672-21timerwakeupswit09:35:432
15218992620,4cyclictest33-21ksoftirqd/210:10:182
15218992619,4cyclictest33-21ksoftirqd/210:16:022
15216992620,3cyclictest9-21ksoftirqd/010:50:230
15216992619,4cyclictest28561-21irqrtprio12:15:310
15218992519,3cyclictest33-21ksoftirqd/211:15:262
1521699259,9cyclictest204-21systemd-journal08:45:170
15216992520,3cyclictest9-21ksoftirqd/012:37:350
15216992518,4cyclictest9-21ksoftirqd/009:10:270
15216992518,4cyclictest30761-21grep12:20:260
1521699251,3cyclictest121rcu_preempt11:50:250
15216992418,4cyclictest9-21ksoftirqd/010:10:050
15216992417,4cyclictest9-21ksoftirqd/010:38:080
15216992316,4cyclictest1274-21snmpd10:25:350
15216992316,4cyclictest1274-21snmpd10:25:350
1521699230,3cyclictest121rcu_preempt11:29:240
15216992217,3cyclictest9-21ksoftirqd/010:30:360
15216992214,4cyclictest16311-21rs:main0
1521699214,8cyclictest121rcu_preempt08:26:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional