You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-06 - 19:08

x86 Intel Atom E3845 @1910 MHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot2.osadl.org (updated Tue Jun 06, 2023 12:45:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
133991870,17ptp4l401ktimersoftd/307:09:043
148282810,4sleep1231rcuc/108:30:451
148282810,4sleep1231rcuc/108:30:451
51292720,4sleep38238-21cyclictest10:30:313
79522714,12sleep10-21swapper/107:09:581
8242997059,7cyclictest41-21ksoftirqd/311:10:193
8240997025,9cyclictest25-21ksoftirqd/109:05:371
824099688,4cyclictest25-21ksoftirqd/108:50:201
8240996730,16cyclictest25-21ksoftirqd/111:40:331
8240996633,5cyclictest25-21ksoftirqd/111:45:281
8240996633,4cyclictest25-21ksoftirqd/108:00:011
8240996632,5cyclictest25-21ksoftirqd/110:55:201
8240996632,5cyclictest25-21ksoftirqd/110:00:231
8240996629,5cyclictest25-21ksoftirqd/110:50:461
8240996623,5cyclictest25-21ksoftirqd/108:20:351
8240996623,5cyclictest25-21ksoftirqd/108:20:351
8240996534,4cyclictest25-21ksoftirqd/108:25:401
8240996532,4cyclictest25-21ksoftirqd/109:50:331
8240996532,4cyclictest25-21ksoftirqd/108:00:281
8240996531,4cyclictest25-21ksoftirqd/111:10:011
8240996529,5cyclictest25-21ksoftirqd/107:25:261
8240996528,5cyclictest25-21ksoftirqd/110:45:361
133991650,2ptp4l401ktimersoftd/308:38:453
133991650,1ptp4l401ktimersoftd/312:13:243
8240996432,4cyclictest25-21ksoftirqd/112:25:371
8240996432,4cyclictest25-21ksoftirqd/112:20:401
8240996432,4cyclictest25-21ksoftirqd/111:35:261
8240996431,5cyclictest25-21ksoftirqd/107:10:401
8240996431,4cyclictest25-21ksoftirqd/109:45:201
8240996430,5cyclictest25-21ksoftirqd/110:25:221
8240996429,7cyclictest25-21ksoftirqd/108:55:501
8240996429,5cyclictest25-21ksoftirqd/112:05:001
8240996427,5cyclictest25-21ksoftirqd/112:35:301
8240996427,5cyclictest25-21ksoftirqd/112:35:301
8240996427,5cyclictest25-21ksoftirqd/112:15:331
8240996417,5cyclictest25-21ksoftirqd/112:35:011
135822640,2sleep013585-21tr09:40:230
8240996330,5cyclictest25-21ksoftirqd/108:40:011
8240996330,4cyclictest25-21ksoftirqd/108:15:241
8240996328,5cyclictest25-21ksoftirqd/107:30:311
8240996324,8cyclictest25-21ksoftirqd/110:30:231
8240996324,8cyclictest25-21ksoftirqd/109:25:251
8240996322,5cyclictest25-21ksoftirqd/111:30:191
8240996322,5cyclictest25-21ksoftirqd/111:30:191
823999637,9cyclictest13532-21acpi10:50:190
823999637,10cyclictest30840-21awk07:55:320
823999636,9cyclictest28838-21sort10:10:370
133991630,1ptp4l401ktimersoftd/311:15:273
8242996249,7cyclictest41-21ksoftirqd/309:30:193
8242996248,7cyclictest41-21ksoftirqd/307:30:213
8240996230,4cyclictest25-21ksoftirqd/108:05:451
8240996229,5cyclictest25-21ksoftirqd/110:35:351
8240996229,5cyclictest25-21ksoftirqd/110:15:361
8240996229,5cyclictest25-21ksoftirqd/109:35:311
8240996225,5cyclictest25-21ksoftirqd/109:40:471
823999626,9cyclictest28210-21grep07:50:280
823999626,8cyclictest1767-21latency_hist10:25:000
8239996228,8cyclictest9-21ksoftirqd/008:45:250
8240996129,5cyclictest25-21ksoftirqd/111:15:451
8240996129,5cyclictest25-21ksoftirqd/110:20:351
8240996129,4cyclictest25-21ksoftirqd/109:55:471
8240996128,5cyclictest25-21ksoftirqd/110:40:411
8240996128,4cyclictest25-21ksoftirqd/111:55:351
8240996127,4cyclictest25-21ksoftirqd/112:06:501
8240996125,5cyclictest25-21ksoftirqd/111:30:001
8240996125,5cyclictest25-21ksoftirqd/110:10:191
8240996124,8cyclictest25-21ksoftirqd/108:45:201
8240996123,8cyclictest25-21ksoftirqd/111:12:531
8240996122,9cyclictest25-21ksoftirqd/107:15:421
8240996122,8cyclictest25-21ksoftirqd/107:40:261
823999617,8cyclictest28089-21df_abs12:30:220
823999616,9cyclictest1313-21lldpd09:40:000
823999616,8cyclictest2607-21cat10:25:270
823999616,7cyclictest11287-21sed11:55:010
823999615,9cyclictest1274-21snmpd08:20:300
823999615,9cyclictest1274-21snmpd08:20:300
823999615,8cyclictest23201-21sed11:10:230
823999615,8cyclictest20011-21sed08:45:000
133991610,1ptp4l401ktimersoftd/311:37:253
8240996029,4cyclictest25-21ksoftirqd/108:10:401
8240996028,4cyclictest25-21ksoftirqd/110:05:251
8240996025,5cyclictest25-21ksoftirqd/107:35:221
823999607,8cyclictest16311-21rs:main0
823999606,9cyclictest26792-21sed07:50:000
823999606,8cyclictest1313-21lldpd11:40:220
823999605,9cyclictest22462-21latency_hist10:00:000
823999605,8cyclictest22672-21latency_hist11:10:010
823999605,8cyclictest18065-21cat11:00:010
823999605,8cyclictest15339-21sed08:35:010
823999605,8cyclictest15339-21sed08:35:000
8240995928,4cyclictest25-21ksoftirqd/112:10:221
8240995928,4cyclictest25-21ksoftirqd/111:50:191
8240995928,4cyclictest25-21ksoftirqd/109:20:231
8240995927,8cyclictest25-21ksoftirqd/107:50:191
8240995927,5cyclictest25-21ksoftirqd/111:00:421
8240995927,4cyclictest25-21ksoftirqd/111:20:381
8240995926,5cyclictest25-21ksoftirqd/109:30:381
8240995925,5cyclictest25-21ksoftirqd/109:10:261
8240995924,5cyclictest25-21ksoftirqd/109:00:421
8240995922,5cyclictest25-21ksoftirqd/109:18:111
8240995919,8cyclictest25-21ksoftirqd/107:20:411
823999597,8cyclictest23328-21strings07:40:310
823999596,9cyclictest32079-21latency_hist11:30:000
823999596,9cyclictest1313-21lldpd11:45:370
823999596,9cyclictest1313-21lldpd07:15:010
823999596,8cyclictest29164-21latency_hist09:05:010
823999596,8cyclictest1313-21lldpd12:30:000
823999596,8cyclictest1313-21lldpd10:05:310
823999595,9cyclictest29395-21/usr/sbin/munin09:05:320
823999595,8cyclictest22150-21grep07:40:010
823999595,8cyclictest12314-21sed08:25:420
823999585,8cyclictest6973-21df10:35:240
823999585,8cyclictest1313-21lldpd12:10:220
823999585,7cyclictest5666-21unixbench_singl09:20:410
823999585,7cyclictest2684-21cat09:15:300
823999585,7cyclictest101ktimersoftd/012:20:260
823999584,9cyclictest7514-21phc2sys-jitter09:25:430
823999584,8cyclictest19052-21sed09:50:310
823999575,9cyclictest1430-21/usr/sbin/munin08:05:470
823999575,7cyclictest8710-21dpkg09:30:190
823999574,9cyclictest31925-21df_abs08:00:230
823999574,8cyclictest1274-21snmpd11:00:230
823999573,9cyclictest101ktimersoftd/009:45:350
823999573,8cyclictest19874-21latency_hist07:35:000
8242995642,7cyclictest41-21ksoftirqd/311:30:193
8242995642,7cyclictest41-21ksoftirqd/311:30:193
8240995639,6cyclictest25-21ksoftirqd/108:40:261
8240995624,5cyclictest25-21ksoftirqd/107:45:161
823999564,8cyclictest17522-21sed12:05:370
823999564,8cyclictest1274-21snmpd12:35:240
823999564,8cyclictest1274-21snmpd12:35:230
823999563,9cyclictest1274-21snmpd11:35:310
823999557,8cyclictest16311-21rs:main0
823999557,8cyclictest16311-21rs:main0
823999554,9cyclictest204-21systemd-journal07:20:000
823999554,7cyclictest101ktimersoftd/010:30:260
823999554,6cyclictest266172sleep008:57:310
823999553,9cyclictest12763-21/usr/sbin/munin11:55:390
823999553,8cyclictest1313-21lldpd08:35:270
823999553,8cyclictest1274-21snmpd10:15:270
823999553,7cyclictest9234-21sed10:40:240
823999552,8cyclictest1274-21snmpd11:15:300
8242995435,7cyclictest41-21ksoftirqd/311:20:233
823999546,7cyclictest1274-21snmpd10:02:430
823999544,8cyclictest204-21systemd-journal12:17:550
823999544,8cyclictest12916-21/usr/sbin/munin07:20:350
823999543,8cyclictest1555-21ls09:15:000
823999542,9cyclictest12228-21needreboot10:45:320
133991540,5ptp4l16311-21rs:main3
823999535,8cyclictest1274-21snmpd08:15:230
823999532,8cyclictest15107-21mailstats12:00:350
133991530,1ptp4l401ktimersoftd/308:41:423
8242995240,6cyclictest41-21ksoftirqd/312:20:193
8242995238,7cyclictest41-21ksoftirqd/310:50:213
823999522,8cyclictest1274-21snmpd08:52:050
8239995224,10cyclictest9-21ksoftirqd/008:11:150
766525230,11sleep233-21ksoftirqd/207:06:222
55498520,15rtkit-daemon16309-21in:imuxsock11:22:552
133991520,2ptp4l0-21swapper/309:36:193
823999511,14cyclictest1274-21snmpd11:20:120
133991510,3ptp4l20807-21kworker/3:109:44:423
133991500,1ptp4l401ktimersoftd/307:49:143
8242994936,6cyclictest41-21ksoftirqd/307:20:203
824199481,44cyclictest1313-21lldpd07:49:022
133991480,2ptp4l20807-21kworker/3:112:39:423
133991480,2ptp4l20807-21kworker/3:112:39:423
133991480,2ptp4l20807-21kworker/3:109:00:223
8242994531,6cyclictest41-21ksoftirqd/308:05:223
824199453,11cyclictest1313-21lldpd11:50:172
55498450,6rtkit-daemon1313-21lldpd10:03:123
8242994424,11cyclictest1313-21lldpd07:15:193
8241994410,31cyclictest25529-21irqrtprio08:55:422
8242994231,6cyclictest41-21ksoftirqd/310:15:193
824199422,7cyclictest63-21kswapd011:45:182
824199422,10cyclictest204-21systemd-journal11:55:232
133991420,3ptp4l20807-21kworker/3:107:55:243
133991420,2ptp4l20807-21kworker/3:109:57:133
133991420,1ptp4l401ktimersoftd/312:25:193
133991420,1ptp4l401ktimersoftd/312:16:373
8242994125,8cyclictest41-21ksoftirqd/308:50:203
824299411,3cyclictest121rcu_preempt10:13:193
8241994113,4cyclictest13671-21fschecks_count08:30:302
8241994113,4cyclictest13671-21fschecks_count08:30:292
824199411,10cyclictest1313-21lldpd07:55:132
133991410,1ptp4l401ktimersoftd/312:32:343
133991410,1ptp4l401ktimersoftd/311:51:543
133991410,1ptp4l401ktimersoftd/309:14:063
824199401,9cyclictest1313-21lldpd07:25:212
8241994014,4cyclictest16311-21rs:main2
76732409,19sleep00-21swapper/007:06:270
208662400,3sleep3121rcu_preempt11:05:253
133991400,4ptp4l824299cyclictest11:58:213
133991400,2ptp4l20807-21kworker/3:111:03:303
133991400,1ptp4l401ktimersoftd/309:54:393
133991400,1ptp4l401ktimersoftd/308:23:153
133991400,1ptp4l401ktimersoftd/308:23:153
133991400,1ptp4l401ktimersoftd/308:15:433
8241993913,6cyclictest204-21systemd-journal12:30:212
8241993913,4cyclictest27241-21df07:50:162
8241993913,4cyclictest10989-21acpi09:35:212
824199390,5cyclictest23117-21apt-get12:20:182
133991390,1ptp4l401ktimersoftd/310:25:353
133991390,1ptp4l401ktimersoftd/310:24:353
133991390,1ptp4l401ktimersoftd/309:15:253
8242993825,10cyclictest41-21ksoftirqd/308:30:413
8242993825,10cyclictest41-21ksoftirqd/308:30:403
824199386,11cyclictest572-21gmain09:40:302
824199383,8cyclictest12993-21apt-get07:20:202
8241993813,3cyclictest17677-21sshd09:49:022
8241993812,4cyclictest4286-21cstates09:20:242
8241993810,5cyclictest204-21systemd-journal08:40:442
8241993810,5cyclictest1313-21lldpd07:15:432
133991380,2ptp4l20807-21kworker/3:111:44:263
133991380,1ptp4l401ktimersoftd/309:26:233
133991380,1ptp4l401ktimersoftd/309:24:533
133991380,1ptp4l401ktimersoftd/309:09:143
133991380,1ptp4l401ktimersoftd/307:40:143
8242993731,3cyclictest41-21ksoftirqd/308:10:313
8242993730,4cyclictest41-21ksoftirqd/310:40:213
8242993730,4cyclictest41-21ksoftirqd/310:36:173
8242993723,8cyclictest41-21ksoftirqd/309:45:193
8241993713,4cyclictest20515-21which07:35:262
8241993712,3cyclictest8456-21/usr/sbin/munin08:20:542
8241993712,3cyclictest8456-21/usr/sbin/munin08:20:542
8241993711,4cyclictest25808-21sed12:25:222
133991370,1ptp4l401ktimersoftd/311:27:023
8242993630,3cyclictest41-21ksoftirqd/312:05:393
8242993629,4cyclictest41-21ksoftirqd/308:00:193
824299360,4cyclictest121rcu_preempt07:50:033
824199369,5cyclictest1313-21lldpd12:12:422
824199369,5cyclictest1313-21lldpd07:10:182
824199369,5cyclictest1274-21snmpd11:05:282
824199368,6cyclictest5777-21timerwakeupswit10:30:382
824199362,7cyclictest0-21swapper/210:35:212
8241993612,3cyclictest9647-21sed09:30:312
8241993611,4cyclictest27400-21latency_hist11:20:002
8241993611,3cyclictest23160-21expr11:10:232
824199360,13cyclictest543-21/usr/sbin/munin09:10:382
133991360,1ptp4l401ktimersoftd/308:49:153
133991360,1ptp4l401ktimersoftd/307:36:203
8242993528,4cyclictest41-21ksoftirqd/308:25:023
824199357,10cyclictest12856-21cron08:30:002
824199353,12cyclictest18818-21sed09:50:282
824199351,31cyclictest0-21swapper/209:29:322
824199351,18cyclictest1313-21lldpd10:50:032
8241993510,9cyclictest16311-21rs:main2
8241993510,5cyclictest1313-21lldpd08:19:042
8241993510,4cyclictest3215-21timerwakeupswit09:15:372
133991350,1ptp4l401ktimersoftd/308:57:503
133991350,1ptp4l401ktimersoftd/307:11:103
8242993427,4cyclictest41-21ksoftirqd/312:02:593
824199349,6cyclictest1274-21snmpd08:51:542
824199346,4cyclictest1313-21lldpd10:18:222
824199342,5cyclictest0-21swapper/211:25:182
824199341,13cyclictest0-21swapper/208:10:172
8241993410,5cyclictest1313-21lldpd08:46:222
133991340,1ptp4l401ktimersoftd/310:57:133
8242993327,4cyclictest41-21ksoftirqd/307:25:313
8242993327,3cyclictest41-21ksoftirqd/311:50:013
824199339,6cyclictest1313-21lldpd08:09:342
824199339,5cyclictest1313-21lldpd12:08:152
8241993310,5cyclictest1313-21lldpd12:03:572
8241993310,5cyclictest1313-21lldpd11:37:002
8241993310,5cyclictest1313-21lldpd11:04:202
8241993310,5cyclictest1313-21lldpd09:04:132
824199330,3cyclictest121rcu_preempt10:42:242
8242993226,3cyclictest41-21ksoftirqd/310:45:203
824199329,5cyclictest1313-21lldpd11:40:302
824199329,5cyclictest1313-21lldpd11:32:592
824199329,5cyclictest1313-21lldpd11:32:582
824199329,5cyclictest1313-21lldpd09:09:162
824199329,4cyclictest1313-21lldpd10:25:442
824199321,28cyclictest0-21swapper/207:42:032
8241993211,5cyclictest1313-21lldpd10:09:452
8241993210,5cyclictest1313-21lldpd10:00:452
8241993210,4cyclictest1274-21snmpd12:37:212
8241993210,4cyclictest1274-21snmpd12:37:212
824199319,5cyclictest1313-21lldpd08:37:502
8241993111,4cyclictest1274-21snmpd12:17:192
824199310,5cyclictest1313-21lldpd07:34:022
824199309,5cyclictest1313-21lldpd10:56:462
824199304,6cyclictest1274-21snmpd09:55:292
824199301,27cyclictest0-21swapper/208:01:522
8241992913,13cyclictest1313-21lldpd10:47:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional