You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-11 - 03:52

x86 Intel Atom E3845 @1910 MHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot2.osadl.org (updated Sun Jun 11, 2023 00:45:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1339911920,6ptp4l401ktimersoftd/319:09:193
10226998771,11cyclictest25-21ksoftirqd/121:50:201
133991820,2ptp4l16455-21iostat_ios00:00:253
210072770,5sleep11274-21snmpd20:40:321
10227997616,13cyclictest13054-21taskset23:53:552
10227997615,15cyclictest27189-21if_enp1s019:45:252
10227997517,13cyclictest12186-21sed21:30:332
10227997516,12cyclictest15062-21taskset21:39:582
10227997416,12cyclictest27696-21tune2fs23:15:272
10227997416,12cyclictest27696-21tune2fs23:15:272
10227997416,12cyclictest25361-21/usr/sbin/munin22:00:302
10227997416,12cyclictest1952-21perl21:10:292
10227997416,10cyclictest4695-21hddtemp_smartct23:35:242
10227997415,12cyclictest19635-21cut20:40:002
10227997412,20cyclictest16248-21iostat22:50:292
10227997316,13cyclictest21427-21irqstats00:10:272
10227997316,12cyclictest204-21systemd-journal22:15:252
10227997316,12cyclictest16537-21sed00:00:252
10227997315,12cyclictest31076-21taskset22:11:462
10227997315,12cyclictest18124-21grep00:05:212
10226997314,12cyclictest7818-21sensors_temp23:40:381
10227997216,14cyclictest18125-21df_inode21:45:252
10227997216,12cyclictest7003-21meminfo21:20:342
10227997216,11cyclictest19205-21sensors22:55:302
10227997215,9cyclictest14200-21memory22:45:352
10227997215,14cyclictest835-21chrt21:07:232
10227997215,13cyclictest28848-21/usr/sbin/munin19:50:322
10227997215,12cyclictest16543-21sed21:40:322
10227997215,12cyclictest1313-21lldpd21:01:592
10227997214,9cyclictest31231-21python00:30:332
10227997213,13cyclictest21090-21ls19:30:352
10227997213,13cyclictest20915-21phc2sys20:40:302
10227997213,12cyclictest7227-21sed20:10:312
10227997212,13cyclictest974-21sed23:25:392
10227997115,12cyclictest26747-21ls00:20:392
10227997115,12cyclictest21152-21phc2sys21:50:352
10227997115,12cyclictest17137-21sshd20:34:302
10227997112,13cyclictest14392-21wc23:55:362
10227997112,12cyclictest21959-21apt-get19:35:172
10227997111,14cyclictest1313-21lldpd22:05:252
10227997111,13cyclictest18276-21missed_timers19:25:332
10227997110,15cyclictest3855-21grep21:15:242
10227997014,12cyclictest22368-21apt-get21:55:182
10227997013,14cyclictest8519-21latency_hist23:45:002
10227997013,12cyclictest2286-21if_enp4s022:20:282
10227997013,12cyclictest2286-21if_enp4s022:20:272
10227997012,14cyclictest1313-21lldpd23:45:192
10227997012,13cyclictest4664-21iostat_ios22:25:312
10227997011,13cyclictest29597-21latency_hist00:30:012
10227997011,12cyclictest4569-21ls20:05:292
10227996916,12cyclictest21708-21ls23:00:412
10227996914,15cyclictest9808-21awk21:25:532
10227996912,14cyclictest15886-21ls19:20:302
10227996911,14cyclictest32524-21cstates00:35:202
10227996911,14cyclictest10424-21cpuspeed_turbos19:10:232
10227996911,13cyclictest11213-21iostat_ios20:20:282
10227996911,12cyclictest31770-21hwlatdetect19:55:232
10226996913,12cyclictest1-21systemd22:04:161
55498680,43rtkit-daemon33-21ksoftirqd/219:08:082
10227996815,13cyclictest22803-21iostat20:45:302
10227996814,11cyclictest1274-21snmpd23:31:262
10227996813,13cyclictest11880-21meminfo22:40:322
10227996813,12cyclictest2123-21sed20:00:272
10227996812,13cyclictest26774-21apt-get20:55:192
10227996812,13cyclictest25506-21ptp4l-jitter19:40:422
10227996811,13cyclictest24256-21awk00:15:382
10227996811,13cyclictest24256-21awk00:15:372
10227996811,11cyclictest23003-21fschecks_time23:05:232
10227996810,13cyclictest1313-21lldpd20:25:192
10227996712,13cyclictest25903-21sed20:50:352
10227996710,13cyclictest1313-21lldpd22:30:252
10226996712,11cyclictest1313-21lldpd19:37:071
10226996711,19cyclictest1274-21snmpd21:36:551
10226996711,18cyclictest1313-21lldpd20:51:181
10226996711,17cyclictest204-21systemd-journal22:58:311
10226996710,18cyclictest1313-21lldpd00:07:191
55498650,14rtkit-daemon553-21rtkit-daemon20:28:570
10228996554,6cyclictest41-21ksoftirqd/322:50:213
10227996511,14cyclictest1648-21gnome-shell22:39:582
10226996512,11cyclictest1313-21lldpd23:04:061
10227996412,15cyclictest8632-21tune2fs20:15:252
10227996411,12cyclictest25692-21irqstats23:10:292
10226996412,11cyclictest16309-21in:imuxsock23:52:401
133991630,1ptp4l401ktimersoftd/320:19:183
10227996314,10cyclictest13168-21hddtemp_smartct19:15:282
1022699633,11cyclictest1313-21lldpd19:50:011
10226996312,11cyclictest1313-21lldpd00:28:291
10226996212,11cyclictest1313-21lldpd00:36:221
10228996152,5cyclictest41-21ksoftirqd/319:55:153
10227996113,9cyclictest29545-21apt-get23:20:192
10227996113,9cyclictest29545-21apt-get23:20:182
1022699614,18cyclictest1313-21lldpd23:49:301
10226996112,11cyclictest204-21systemd-journal22:12:331
10226996110,18cyclictest1313-21lldpd23:39:161
10228996048,6cyclictest41-21ksoftirqd/321:45:223
10226996013,11cyclictest204-21systemd-journal21:20:541
10226996012,12cyclictest13899-21fschecks_count23:55:281
10226996010,12cyclictest16311-21rs:main1
10226996010,12cyclictest1274-21snmpd21:18:281
133991590,1ptp4l401ktimersoftd/322:36:113
133991590,1ptp4l401ktimersoftd/321:28:153
133991590,1ptp4l401ktimersoftd/300:21:523
1022699595,17cyclictest1313-21lldpd20:06:301
10226995912,17cyclictest1313-21lldpd23:13:121
10226995912,11cyclictest1313-21lldpd20:14:001
10228995846,6cyclictest41-21ksoftirqd/322:45:213
10228995845,6cyclictest41-21ksoftirqd/319:15:223
1022699585,13cyclictest1313-21lldpd19:20:201
10226995812,11cyclictest1313-21lldpd20:26:211
10226995811,11cyclictest1313-21lldpd22:52:111
10226995711,11cyclictest1313-21lldpd20:17:301
10226995710,12cyclictest1313-21lldpd20:48:161
133991560,1ptp4l401ktimersoftd/319:54:213
10228995646,5cyclictest41-21ksoftirqd/300:35:193
1022699564,18cyclictest1313-21lldpd19:13:561
10228995545,5cyclictest41-21ksoftirqd/322:55:163
133991540,1ptp4l401ktimersoftd/323:55:463
10228995446,4cyclictest41-21ksoftirqd/323:40:183
10228995443,6cyclictest41-21ksoftirqd/320:25:173
67152520,4sleep11274-21snmpd22:30:231
10228995243,5cyclictest41-21ksoftirqd/321:15:223
1022699527,17cyclictest16309-21in:imuxsock22:15:301
1022699524,11cyclictest1313-21lldpd22:20:481
1022699524,11cyclictest1313-21lldpd22:20:481
10228995141,4cyclictest41-21ksoftirqd/322:05:213
1022899510,3cyclictest121rcu_preempt23:35:193
1022699513,13cyclictest1313-21lldpd00:17:471
1022699513,13cyclictest1313-21lldpd00:17:461
1022699510,12cyclictest1313-21lldpd00:30:261
10228995024,12cyclictest41-21ksoftirqd/323:00:283
10228994942,4cyclictest41-21ksoftirqd/320:10:013
1022899493,4cyclictest41-21ksoftirqd/320:35:273
1022699499,10cyclictest1313-21lldpd19:51:501
10228994837,6cyclictest41-21ksoftirqd/300:10:173
10228994813,7cyclictest41-21ksoftirqd/322:20:383
10228994813,7cyclictest41-21ksoftirqd/322:20:383
1022699480,20cyclictest1313-21lldpd21:40:351
1022599483,3cyclictest121rcu_preempt00:06:180
1022899474,3cyclictest121rcu_preempt20:40:183
10228994735,9cyclictest41-21ksoftirqd/323:05:223
1022699472,11cyclictest1313-21lldpd21:32:541
10228994633,10cyclictest41-21ksoftirqd/321:50:403
983124529,7sleep00-21swapper/019:08:350
10228994539,3cyclictest41-21ksoftirqd/300:20:013
10228994539,3cyclictest41-21ksoftirqd/300:20:003
10228994538,4cyclictest41-21ksoftirqd/320:20:253
10228994532,3cyclictest41-21ksoftirqd/300:25:293
10228994530,5cyclictest41-21ksoftirqd/300:05:383
10228994524,7cyclictest41-21ksoftirqd/323:15:423
10228994524,7cyclictest41-21ksoftirqd/323:15:423
10226994512,20cyclictest1313-21lldpd23:28:361
10226994510,11cyclictest1313-21lldpd00:20:361
10228994439,3cyclictest41-21ksoftirqd/320:55:203
10228994438,3cyclictest41-21ksoftirqd/322:30:313
10228994437,4cyclictest41-21ksoftirqd/320:45:333
10228994437,4cyclictest41-21ksoftirqd/319:35:223
10228994437,4cyclictest41-21ksoftirqd/319:25:213
10228994431,10cyclictest41-21ksoftirqd/323:25:173
10228994426,7cyclictest41-21ksoftirqd/320:00:213
10228994425,8cyclictest41-21ksoftirqd/300:30:203
10228994415,5cyclictest41-21ksoftirqd/323:20:333
10228994415,5cyclictest41-21ksoftirqd/323:20:333
10226994412,18cyclictest204-21systemd-journal22:37:271
10226994412,17cyclictest204-21systemd-journal21:02:211
55498430,3rtkit-daemon553-21rtkit-daemon22:49:170
55498430,3rtkit-daemon553-21rtkit-daemon22:43:160
1022899436,7cyclictest41-21ksoftirqd/319:10:383
10228994337,4cyclictest41-21ksoftirqd/320:50:193
10228994337,3cyclictest41-21ksoftirqd/321:35:183
10228994336,4cyclictest41-21ksoftirqd/323:50:283
10228994336,4cyclictest41-21ksoftirqd/322:10:333
10228994336,4cyclictest41-21ksoftirqd/321:10:383
10228994336,4cyclictest41-21ksoftirqd/321:05:213
10228994335,5cyclictest41-21ksoftirqd/322:00:213
10228994330,11cyclictest41-21ksoftirqd/321:45:013
10228994327,5cyclictest41-21ksoftirqd/322:15:413
10228994327,5cyclictest41-21ksoftirqd/321:20:283
10228994310,5cyclictest41-21ksoftirqd/321:01:303
10226994312,18cyclictest204-21systemd-journal19:43:101
10226994311,18cyclictest204-21systemd-journal20:00:151
10226994311,18cyclictest1274-21snmpd00:13:041
10225994336,4cyclictest1274-21snmpd19:55:000
1022599431,7cyclictest121rcu_preempt23:55:360
98152424,25sleep10-21swapper/119:08:241
10228994237,3cyclictest41-21ksoftirqd/323:35:013
10228994236,3cyclictest41-21ksoftirqd/319:40:243
10228994236,3cyclictest41-21ksoftirqd/319:20:353
10228994235,4cyclictest41-21ksoftirqd/323:45:233
10228994235,4cyclictest41-21ksoftirqd/321:55:293
10228994235,4cyclictest41-21ksoftirqd/320:10:333
10228994235,4cyclictest41-21ksoftirqd/319:45:213
10228994234,5cyclictest41-21ksoftirqd/323:10:243
10228994234,4cyclictest41-21ksoftirqd/322:25:223
10228994232,5cyclictest41-21ksoftirqd/319:30:183
10226994211,18cyclictest204-21systemd-journal23:22:591
10226994211,18cyclictest204-21systemd-journal23:22:591
10226994211,16cyclictest204-21systemd-journal22:25:461
10226994210,19cyclictest204-21systemd-journal19:55:251
10226994210,19cyclictest1313-21lldpd23:09:281
10226994210,19cyclictest1313-21lldpd22:49:481
10228994134,4cyclictest41-21ksoftirqd/322:40:213
10226994111,17cyclictest204-21systemd-journal21:13:501
10228994035,3cyclictest41-21ksoftirqd/320:30:303
10228994034,3cyclictest41-21ksoftirqd/321:30:343
10226994011,17cyclictest1313-21lldpd21:05:451
10226994011,17cyclictest1313-21lldpd19:30:471
10226993913,11cyclictest11362-21awk22:40:241
10226993910,17cyclictest1313-21lldpd00:01:181
10226993813,11cyclictest204-21systemd-journal22:06:531
1022699380,3cyclictest121rcu_preempt21:46:191
10225993831,4cyclictest9-21ksoftirqd/023:45:230
1022599380,3cyclictest121rcu_preempt00:11:350
10225993731,4cyclictest9-21ksoftirqd/022:10:210
10225993731,4cyclictest9-21ksoftirqd/020:15:220
10225993730,4cyclictest9-21ksoftirqd/022:35:000
10225993730,4cyclictest9-21ksoftirqd/020:46:090
10225993730,4cyclictest9-21ksoftirqd/019:35:010
10225993730,4cyclictest9-21ksoftirqd/019:20:000
1022599371,10cyclictest1313-21lldpd20:05:000
10226993614,7cyclictest204-21systemd-journal19:15:351
10225993630,4cyclictest9-21ksoftirqd/022:55:160
10225993630,3cyclictest9-21ksoftirqd/019:25:250
10225993622,4cyclictest1313-21lldpd21:40:320
1022599360,5cyclictest121rcu_preempt00:26:300
10226993515,16cyclictest1313-21lldpd23:20:011
10226993515,16cyclictest1313-21lldpd23:20:001
10225993530,3cyclictest9-21ksoftirqd/020:55:270
10225993530,3cyclictest9-21ksoftirqd/000:16:330
10225993530,3cyclictest9-21ksoftirqd/000:16:320
10225993529,4cyclictest9-21ksoftirqd/022:00:220
10225993529,4cyclictest9-21ksoftirqd/019:35:420
10225993528,4cyclictest9-21ksoftirqd/022:15:320
10225993528,4cyclictest9-21ksoftirqd/022:07:130
10225993528,4cyclictest9-21ksoftirqd/022:00:000
10225993528,4cyclictest9-21ksoftirqd/021:12:310
10225993528,4cyclictest9-21ksoftirqd/019:47:190
10225993528,3cyclictest1274-21snmpd19:10:210
10225993519,4cyclictest9-21ksoftirqd/021:09:590
1022599350,3cyclictest121rcu_preempt23:10:060
1022599344,3cyclictest121rcu_preempt00:02:250
10225993428,3cyclictest9-21ksoftirqd/022:50:230
10225993427,4cyclictest9-21ksoftirqd/021:53:270
10225993427,4cyclictest9-21ksoftirqd/021:03:560
10225993427,4cyclictest9-21ksoftirqd/020:30:390
10225993417,5cyclictest1274-21snmpd21:25:010
10226993320,9cyclictest1274-21snmpd21:55:181
10226993315,15cyclictest572-21gmain21:26:431
10226993315,14cyclictest1274-21snmpd20:20:311
10226993314,15cyclictest28988-21latency_hist21:00:011
10226993314,15cyclictest2339-21fschecks_time23:30:231
10226993313,16cyclictest17915-21if_enp4s019:25:271
1022599333,3cyclictest121rcu_preempt23:25:270
1022599331,3cyclictest121rcu_preempt20:38:500
10226993214,14cyclictest1313-21lldpd20:33:541
10225993227,3cyclictest9-21ksoftirqd/023:40:370
10225993226,3cyclictest9-21ksoftirqd/023:20:310
10225993226,3cyclictest9-21ksoftirqd/023:20:310
10225993226,3cyclictest9-21ksoftirqd/022:36:190
10225993225,4cyclictest9-21ksoftirqd/023:30:060
10225993225,4cyclictest9-21ksoftirqd/023:18:210
10225993225,4cyclictest9-21ksoftirqd/023:18:210
10225993225,4cyclictest9-21ksoftirqd/023:09:500
10225993225,4cyclictest9-21ksoftirqd/021:38:420
10225993225,4cyclictest9-21ksoftirqd/021:30:240
10225993225,4cyclictest9-21ksoftirqd/019:20:390
1022599320,3cyclictest121rcu_preempt23:50:180
1022599314,7cyclictest121rcu_preempt21:16:130
10225993125,3cyclictest9-21ksoftirqd/023:37:080
10225993125,3cyclictest9-21ksoftirqd/021:45:230
10225993124,4cyclictest9-21ksoftirqd/020:06:450
1022599312,3cyclictest121rcu_preempt23:02:250
1022599311,3cyclictest121rcu_preempt00:35:210
10225993023,4cyclictest9-21ksoftirqd/022:29:250
10225993023,4cyclictest9-21ksoftirqd/019:55:190
10225993022,5cyclictest1313-21lldpd20:10:180
1022599293,6cyclictest121rcu_preempt19:44:180
1022599292,7cyclictest121rcu_preempt00:21:580
10225992916,10cyclictest9-21ksoftirqd/020:45:000
1022599291,3cyclictest121rcu_preempt21:26:190
10225992822,4cyclictest9-21ksoftirqd/020:50:190
10225992821,4cyclictest1274-21snmpd20:20:230
10225992720,4cyclictest9-21ksoftirqd/000:30:200
10225992619,4cyclictest9-21ksoftirqd/022:22:320
10225992619,4cyclictest9-21ksoftirqd/022:22:310
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional