You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-08-05 - 07:41

Intel(R) Atom(TM) CPU E3845 @ 1.91GHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rackdslot4.osadl.org (updated Thu Aug 05, 2021 00:46:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
963122070,4sleep10-21swapper/119:05:141
1208911800,6ptp4l401ktimersoftd/319:05:223
1937521660,7sleep21207499cyclictest23:55:172
12073999783,8cyclictest25-21ksoftirqd/119:50:141
12073998470,8cyclictest25-21ksoftirqd/121:05:121
106672780,4sleep233-21ksoftirqd/219:05:232
63872710,8sleep21207499cyclictest00:20:052
1207299700,27cyclictest17226-21cstates21:55:050
1207299690,27cyclictest11193-21latency21:45:250
66372670,9sleep31207599cyclictest23:40:123
12073996558,4cyclictest25-21ksoftirqd/123:15:121
12073996557,4cyclictest25-21ksoftirqd/120:10:111
1207499640,14cyclictest14015-21grep21:50:122
120891630,1ptp4l401ktimersoftd/319:10:353
149802600,2sleep30-21swapper/323:10:173
133672600,3sleep113362-21if_enp1s021:10:071
12073995945,8cyclictest25-21ksoftirqd/120:35:101
129192580,2sleep20-21swapper/223:10:002
1207499570,20cyclictest9762-21/usr/sbin/munin21:05:262
95962550,2sleep30-21swapper/322:25:083
120891550,1ptp4l401ktimersoftd/319:41:573
1207599555,14cyclictest26143-21sort20:45:133
1207599555,14cyclictest26143-21sort20:45:133
1207499550,16cyclictest17261-21needreboot20:35:082
1207499550,13cyclictest19531-21awk19:15:372
120891540,1ptp4l391rcuc/300:33:343
1207499540,17cyclictest17963-21apt-get21:15:122
120891530,2ptp4l401ktimersoftd/323:08:483
1207499530,18cyclictest27146-21apt-get23:25:162
12073995340,7cyclictest25-21ksoftirqd/120:55:121
12073995340,7cyclictest25-21ksoftirqd/120:05:101
321942520,2sleep20-21swapper/223:30:242
1207499520,9cyclictest14372-21apt-get22:30:142
1207499500,9cyclictest5596-21apt-get22:20:132
1207499500,9cyclictest2171-21apt-get22:55:102
1207499500,9cyclictest10138-21ls00:25:032
1207499500,8cyclictest10504-21apt-get23:45:102
1207499500,16cyclictest18624-21apt-get22:35:152
1207499500,10cyclictest730-21ls21:35:022
12073995039,7cyclictest25-21ksoftirqd/123:05:121
12072995020,19cyclictest9-21ksoftirqd/021:30:580
120891490,1ptp4l401ktimersoftd/320:35:133
1207499490,9cyclictest14288-21apt-get21:10:162
1207499490,17cyclictest2602-21apt-get23:35:132
120891480,1ptp4l401ktimersoftd/322:49:383
12075994827,4cyclictest41-21ksoftirqd/319:24:383
1207499482,20cyclictest24474-21/usr/sbin/munin21:25:062
1207499480,9cyclictest25909-21df20:45:132
1207499480,9cyclictest25909-21df20:45:122
1207499480,8cyclictest18216-21ls00:35:032
1207499470,16cyclictest29966-21iostat19:30:142
1207499470,16cyclictest29966-21iostat19:30:142
1207499470,15cyclictest21469-21df_inode21:20:072
12072994730,14cyclictest9-21ksoftirqd/022:01:440
215492460,4sleep121551-21cpuspeed_turbos19:20:151
1207499460,17cyclictest24508-21sensors_temp00:00:352
12073994638,5cyclictest25-21ksoftirqd/119:15:121
120891450,1ptp4l401ktimersoftd/323:30:013
120891450,1ptp4l401ktimersoftd/322:00:363
120891450,1ptp4l401ktimersoftd/321:20:163
1207499450,17cyclictest20883-21df_abs20:00:062
120891440,1ptp4l401ktimersoftd/320:33:533
120891440,1ptp4l401ktimersoftd/300:05:513
120891440,1ptp4l401ktimersoftd/300:00:243
12073994439,3cyclictest25-21ksoftirqd/123:40:131
12073994437,4cyclictest25-21ksoftirqd/121:30:141
12073994431,8cyclictest25-21ksoftirqd/119:40:121
1151824414,19sleep00-21swapper/019:06:560
120891430,1ptp4l401ktimersoftd/323:57:443
12073994338,3cyclictest25-21ksoftirqd/123:25:011
12073994337,3cyclictest25-21ksoftirqd/120:49:591
12073994337,3cyclictest25-21ksoftirqd/120:49:581
12073994336,4cyclictest25-21ksoftirqd/121:35:111
1207399431,4cyclictest25-21ksoftirqd/100:35:131
287442420,4sleep3401ktimersoftd/321:30:013
120891420,1ptp4l401ktimersoftd/321:43:333
12073994236,4cyclictest25-21ksoftirqd/100:30:001
12073994235,4cyclictest25-21ksoftirqd/100:00:011
12073994234,5cyclictest25-21ksoftirqd/122:30:051
120891410,1ptp4l401ktimersoftd/321:00:063
1207499411,18cyclictest21929-21apt-get22:00:102
12073994136,3cyclictest25-21ksoftirqd/123:25:021
12073994135,4cyclictest25-21ksoftirqd/123:35:011
12073994135,4cyclictest25-21ksoftirqd/119:35:111
12073994135,3cyclictest25-21ksoftirqd/121:00:121
12073994135,3cyclictest25-21ksoftirqd/100:25:011
12073994134,4cyclictest25-21ksoftirqd/122:30:011
12073994134,4cyclictest25-21ksoftirqd/122:10:001
12073994134,4cyclictest25-21ksoftirqd/122:10:001
12072994123,11cyclictest9-21ksoftirqd/023:20:090
120891400,2ptp4l0-21swapper/321:51:193
12075994033,4cyclictest41-21ksoftirqd/320:10:133
1207499401,19cyclictest17307-21df_abs21:55:052
1207499400,10cyclictest31556-21df00:10:142
12073994035,3cyclictest25-21ksoftirqd/119:25:261
12073994034,4cyclictest25-21ksoftirqd/121:45:121
120891390,2ptp4l0-21swapper/319:35:203
120891390,1ptp4l401ktimersoftd/323:15:153
120891390,1ptp4l401ktimersoftd/321:15:193
12075993932,4cyclictest41-21ksoftirqd/300:12:283
1207499391,10cyclictest1694-21gnome-shell23:43:582
12073993934,3cyclictest25-21ksoftirqd/123:50:151
12073993933,4cyclictest25-21ksoftirqd/123:45:131
12073993932,4cyclictest25-21ksoftirqd/122:55:001
12073993932,4cyclictest25-21ksoftirqd/121:20:151
12073993932,4cyclictest25-21ksoftirqd/121:20:001
12073993932,4cyclictest25-21ksoftirqd/120:50:061
12073993932,4cyclictest25-21ksoftirqd/120:50:061
12073993932,4cyclictest25-21ksoftirqd/120:35:001
12073993932,4cyclictest25-21ksoftirqd/120:25:151
12073993930,7cyclictest25-21ksoftirqd/100:07:291
1207299392,29cyclictest18432-21apt-get23:15:120
1207299391,31cyclictest17630-21apt-get19:15:130
39162380,5sleep0111rcuc/019:40:000
1207499381,35cyclictest0-21swapper/200:30:072
12074993811,20cyclictest42250irq/98-enp4s0-T22:25:132
1207499380,9cyclictest14707-21apt-get23:10:152
12073993833,3cyclictest25-21ksoftirqd/122:45:131
12073993833,3cyclictest25-21ksoftirqd/122:15:151
12073993832,4cyclictest25-21ksoftirqd/120:40:161
12073993832,4cyclictest25-21ksoftirqd/119:10:111
12073993832,3cyclictest25-21ksoftirqd/122:25:001
12073993831,4cyclictest25-21ksoftirqd/121:55:111
12073993831,4cyclictest25-21ksoftirqd/119:30:151
12073993831,4cyclictest25-21ksoftirqd/119:30:151
12073993831,4cyclictest25-21ksoftirqd/100:25:121
12073993831,4cyclictest25-21ksoftirqd/100:10:141
1207299382,27cyclictest13710-21apt-get19:50:150
12073993732,3cyclictest25-21ksoftirqd/123:30:021
12073993732,3cyclictest25-21ksoftirqd/122:40:011
12073993732,3cyclictest25-21ksoftirqd/121:50:021
12073993732,3cyclictest15887-21kworker/1:120:00:421
12073993731,4cyclictest25-21ksoftirqd/123:00:151
12073993731,4cyclictest25-21ksoftirqd/122:00:151
12073993731,3cyclictest25-21ksoftirqd/119:50:001
12073993730,4cyclictest25-21ksoftirqd/121:40:031
12073993730,4cyclictest25-21ksoftirqd/121:25:141
12073993730,4cyclictest25-21ksoftirqd/100:20:001
1207299373,27cyclictest10352-21apt-get23:05:110
1207299372,28cyclictest13302-21apt-get19:10:100
1207299371,33cyclictest4305-21grep22:55:360
36102360,2sleep00-21swapper/023:35:200
120891360,12ptp4l401ktimersoftd/320:20:233
12075993629,4cyclictest41-21ksoftirqd/321:45:073
12075993628,5cyclictest41-21ksoftirqd/322:19:543
12074993629,4cyclictest33-21ksoftirqd/221:35:002
1207299362,28cyclictest5528-21apt-get19:40:120
1207299362,27cyclictest5494-21apt-get20:20:100
1207299362,27cyclictest17533-21apt-get19:55:160
12072993616,17cyclictest21910-21/usr/sbin/munin21:20:190
12072993615,18cyclictest7714-21wc22:20:280
12072993614,19cyclictest26674-21cat20:05:170
1207299361,30cyclictest0-21swapper/000:30:150
120891350,1ptp4l401ktimersoftd/322:54:123
120891350,12ptp4l401ktimersoftd/320:00:293
12075993529,3cyclictest41-21ksoftirqd/320:43:083
12075993528,4cyclictest41-21ksoftirqd/322:58:383
12073993530,3cyclictest25-21ksoftirqd/122:35:091
12073993529,4cyclictest25-21ksoftirqd/120:20:131
12073993529,4cyclictest25-21ksoftirqd/120:15:001
12073993529,4cyclictest25-21ksoftirqd/119:55:161
1207299353,4cyclictest121rcu_preempt23:45:360
1207299352,26cyclictest5763-21apt-get21:00:110
1207299352,26cyclictest11247-21apt-get00:25:130
12072993519,12cyclictest9-21ksoftirqd/020:34:580
12072993515,17cyclictest2259-21ptp4l21:35:140
12072993513,19cyclictest6287-21sed23:40:100
12072993513,19cyclictest24350-21/usr/sbin/munin20:45:120
12072993513,19cyclictest24350-21/usr/sbin/munin20:45:110
1207299351,30cyclictest1914-21grep22:15:120
12072993512,19cyclictest17747-21df_abs22:35:050
12072993512,19cyclictest14713-21cat23:50:120
120891340,1ptp4l401ktimersoftd/323:37:283
120891340,1ptp4l401ktimersoftd/322:38:063
120891340,1ptp4l401ktimersoftd/322:13:413
120891340,1ptp4l401ktimersoftd/300:36:003
12075993427,4cyclictest41-21ksoftirqd/321:31:403
12075993427,4cyclictest41-21ksoftirqd/320:18:133
12075993427,4cyclictest41-21ksoftirqd/300:26:183
12074993427,4cyclictest33-21ksoftirqd/222:48:152
12073993428,4cyclictest25-21ksoftirqd/123:10:051
12073993428,3cyclictest25-21ksoftirqd/123:00:001
12073993428,3cyclictest25-21ksoftirqd/122:10:031
12072993427,4cyclictest9-21ksoftirqd/022:11:560
12072993426,5cyclictest9-21ksoftirqd/023:33:150
1207299342,25cyclictest17684-21apt-get20:35:110
12072993414,17cyclictest29533-21if_enp1s020:50:060
12072993414,17cyclictest29533-21if_enp1s020:50:060
12072993414,17cyclictest21851-21/usr/sbin/munin20:40:310
12072993413,18cyclictest5335-21taskset23:00:020
12072993413,18cyclictest2639-21/usr/sbin/munin00:15:270
12072993413,17cyclictest13061-21/usr/sbin/munin23:10:060
12072993412,18cyclictest25277-21expr21:25:040
12075993327,3cyclictest41-21ksoftirqd/322:23:013
12075993326,4cyclictest41-21ksoftirqd/323:46:313
12075993326,4cyclictest41-21ksoftirqd/322:44:293
12075993326,4cyclictest41-21ksoftirqd/320:08:313
12075993325,5cyclictest41-21ksoftirqd/319:33:583
12075993325,5cyclictest41-21ksoftirqd/319:33:573
12074993327,4cyclictest33-21ksoftirqd/222:52:132
12074993327,3cyclictest33-21ksoftirqd/220:40:042
12074993326,4cyclictest33-21ksoftirqd/222:18:502
12074993326,4cyclictest33-21ksoftirqd/220:07:162
12074993326,4cyclictest33-21ksoftirqd/219:28:552
1207499330,20cyclictest6209-21cstates21:00:132
12072993327,4cyclictest9-21ksoftirqd/023:29:220
12072993326,4cyclictest9-21ksoftirqd/022:09:200
12072993326,4cyclictest9-21ksoftirqd/022:09:190
1207299332,26cyclictest9839-21apt-get21:05:120
1207299332,24cyclictest1385-21apt-get20:15:120
12072993314,16cyclictest9498-21latency_hist00:25:000
12072993314,16cyclictest20607-21timedrift19:20:030
12072993313,17cyclictest24481-21mailstats00:00:350
1207299331,25cyclictest1854-21apt-get20:55:130
12072993310,19cyclictest20418-21cut00:35:250
120891320,1ptp4l401ktimersoftd/321:14:023
12075993226,3cyclictest41-21ksoftirqd/319:57:313
12075993225,4cyclictest41-21ksoftirqd/320:56:233
12075993225,4cyclictest41-21ksoftirqd/319:25:503
12074993226,4cyclictest33-21ksoftirqd/219:39:382
12074993226,4cyclictest33-21ksoftirqd/219:14:382
12074993226,3cyclictest33-21ksoftirqd/219:58:092
12072993225,4cyclictest9-21ksoftirqd/022:29:430
12072993225,4cyclictest9-21ksoftirqd/019:45:550
12072993214,15cyclictest17843-21/usr/sbin/munin21:15:170
12072993213,16cyclictest21824-21/usr/sbin/munin20:00:290
1207299321,27cyclictest30787-21df22:50:120
1207299321,26cyclictest5830-21apt-get21:40:110
12072993210,19cyclictest28266-21mailstats22:45:300
12072993210,19cyclictest16127-21sed22:30:310
12072993210,19cyclictest12808-21grep21:10:020
1207299320,28cyclictest13126-21/usr/sbin/munin21:50:040
12075993124,4cyclictest41-21ksoftirqd/323:53:453
12075993124,4cyclictest41-21ksoftirqd/323:21:473
12075993124,4cyclictest41-21ksoftirqd/322:05:493
12075993124,4cyclictest41-21ksoftirqd/322:05:493
12075993124,4cyclictest41-21ksoftirqd/321:36:503
12075993124,4cyclictest41-21ksoftirqd/319:48:433
12075993124,4cyclictest41-21ksoftirqd/300:17:303
12074993125,4cyclictest33-21ksoftirqd/220:16:382
12074993125,3cyclictest28594-21cron22:10:012
12074993125,3cyclictest28594-21cron22:10:012
12074993124,4cyclictest33-21ksoftirqd/223:16:592
12074993124,4cyclictest33-21ksoftirqd/220:57:192
12074993124,4cyclictest33-21ksoftirqd/200:18:262
12074993123,5cyclictest33-21ksoftirqd/220:24:432
1207299312,23cyclictest9797-21apt-get20:25:130
12072993113,15cyclictest29290-21missed_timers19:30:100
12072993113,15cyclictest29290-21missed_timers19:30:090
1207299311,26cyclictest29766-21munin-node20:10:090
1207299311,26cyclictest29728-21ps00:10:000
1207299311,26cyclictest25449-21ps00:05:010
1207299311,25cyclictest21750-21/usr/sbin/munin22:40:050
12072993110,18cyclictest17977-21df_abs23:55:050
120891300,1ptp4l401ktimersoftd/323:27:493
12075993025,3cyclictest41-21ksoftirqd/320:50:033
12075993025,3cyclictest41-21ksoftirqd/320:50:023
12075993023,4cyclictest41-21ksoftirqd/300:21:003
12074993023,4cyclictest33-21ksoftirqd/221:43:202
12074993023,4cyclictest33-21ksoftirqd/219:48:182
12074993023,4cyclictest33-21ksoftirqd/219:40:472
12073993025,3cyclictest25-21ksoftirqd/100:00:081
1207299301,25cyclictest24455-21/usr/sbin/munin19:25:020
12075992923,3cyclictest41-21ksoftirqd/319:16:433
1207599290,27cyclictest0-21swapper/320:27:273
12074992923,4cyclictest33-21ksoftirqd/222:40:212
12074992922,4cyclictest33-21ksoftirqd/223:23:002
12074992922,4cyclictest33-21ksoftirqd/219:21:152
1207599286,14cyclictest6321-21awk23:00:123
12075992822,3cyclictest41-21ksoftirqd/321:07:413
12075992821,4cyclictest41-21ksoftirqd/322:30:473
12075992821,4cyclictest41-21ksoftirqd/321:57:263
12074992822,3cyclictest33-21ksoftirqd/219:54:022
12074992821,4cyclictest33-21ksoftirqd/220:51:412
12074992821,4cyclictest33-21ksoftirqd/220:51:412
12074992821,4cyclictest33-21ksoftirqd/220:29:292
12074992821,4cyclictest33-21ksoftirqd/200:08:532
12074992721,3cyclictest33-21ksoftirqd/220:32:052
12074992720,4cyclictest33-21ksoftirqd/220:13:192
120891260,1ptp4l401ktimersoftd/319:53:423
12074992619,4cyclictest33-21ksoftirqd/223:50:432
12074992615,8cyclictest5584-21cat23:00:042
12074992520,3cyclictest33-21ksoftirqd/222:10:352
12074992211,8cyclictest10673-21sed21:45:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional