You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-29 - 02:56

Intel(R) Atom(TM) CPU E3845 @ 1.91GHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot4.osadl.org (updated Wed Sep 29, 2021 00:46:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2371021750,6sleep02633799cyclictest21:45:080
1208911690,6ptp4l401ktimersoftd/319:07:183
3187021620,6sleep02633799cyclictest23:55:050
3092521440,6sleep32634099cyclictest00:30:333
2642421360,4sleep326431-21irqrtprio19:50:003
120891970,2ptp4l31127-21cpuspeed_turbos22:35:033
258452858,20sleep10-21swapper/119:09:281
255842784,18sleep20-21swapper/219:06:072
270272700,3sleep12633899cyclictest23:49:591
200022680,6sleep120005-21idleruntime20:20:141
26337996312,26cyclictest19762-21/usr/sbin/munin19:40:280
26338996255,4cyclictest25-21ksoftirqd/100:15:141
241152620,2sleep00-21swapper/023:45:090
208162620,2sleep20-21swapper/221:40:172
120891610,1ptp4l401ktimersoftd/300:36:443
120891600,1ptp4l401ktimersoftd/322:10:293
83432560,4sleep38340-21iostat22:45:143
26340995643,7cyclictest41-21ksoftirqd/323:10:123
74482530,2sleep10-21swapper/120:45:111
152892530,4sleep015285-21unixbench_multi19:35:090
2565725030,10sleep00-21swapper/019:07:020
120891500,2ptp4l4479-21cpu23:20:173
26340994841,4cyclictest41-21ksoftirqd/319:20:053
26339994837,6cyclictest33-21ksoftirqd/222:35:142
26337994827,18cyclictest9-21ksoftirqd/022:19:250
26340994725,5cyclictest41-21ksoftirqd/323:00:143
26339994736,6cyclictest33-21ksoftirqd/220:15:142
26338994636,6cyclictest25-21ksoftirqd/122:30:151
2633799443,7cyclictest16311-21apt-get22:55:140
2633799443,7cyclictest16311-21apt-get22:55:140
2633799442,37cyclictest0-21swapper/023:00:130
120891440,1ptp4l401ktimersoftd/320:15:123
2633799433,10cyclictest28920-21apt-get00:30:140
2633799431,5cyclictest28435-21apt-get22:30:150
26337994315,24cyclictest26257-21basename21:10:000
120891430,1ptp4l401ktimersoftd/323:30:143
26340994235,4cyclictest41-21ksoftirqd/322:20:013
26337994220,19cyclictest9-21ksoftirqd/021:23:160
26337994213,26cyclictest4994-21sed00:00:180
26337994213,25cyclictest28300-21sort21:10:190
26337994212,26cyclictest6523-21sed19:25:030
120891420,1ptp4l401ktimersoftd/320:30:173
56598410,3rtkit-daemon564-21rtkit-daemon19:27:541
288842410,8sleep12633899cyclictest19:10:251
26339994135,3cyclictest33-21ksoftirqd/221:20:122
26339994125,4cyclictest33-21ksoftirqd/220:50:022
26337994113,25cyclictest19146-21cat21:40:040
26337994113,24cyclictest11385-21sed22:50:060
26337994112,25cyclictest11244-21cat23:30:030
120891410,1ptp4l401ktimersoftd/300:25:023
114191410,7getstats2634099cyclictest00:09:003
26340994033,4cyclictest41-21ksoftirqd/323:20:013
26339994034,4cyclictest33-21ksoftirqd/222:30:152
26339994033,4cyclictest33-21ksoftirqd/220:24:592
2633799402,5cyclictest28292-21apt-get21:50:150
26337994014,23cyclictest32610-21sed21:15:160
26337994013,24cyclictest20612-21fschecks_count00:20:090
26337994012,25cyclictest8634-21cat22:05:160
26337994012,25cyclictest10009-21dump-pmu-power20:50:000
26337994011,26cyclictest15053-21ls21:35:050
26337994011,25cyclictest6434-21basename22:05:010
26337994011,22cyclictest6649-21irqrtprio23:25:000
26337994010,27cyclictest1634-21latency_hist19:20:000
26337994010,26cyclictest31186-21expr22:35:020
120891400,1ptp4l401ktimersoftd/323:35:113
120891400,12ptp4l0-21swapper/322:06:593
26340993926,10cyclictest41-21ksoftirqd/320:40:143
26339993933,4cyclictest33-21ksoftirqd/222:10:152
26339993933,3cyclictest33-21ksoftirqd/219:40:002
26338993932,4cyclictest25-21ksoftirqd/121:55:001
2633799392,34cyclictest21314-21uname21:00:300
26337993911,25cyclictest12226-21cat20:50:160
26337993910,25cyclictest23822-21sed00:25:030
26337993910,25cyclictest10557-21irqrtprio21:30:010
26340993831,4cyclictest41-21ksoftirqd/323:55:143
26340993831,4cyclictest41-21ksoftirqd/321:00:143
26340993831,4cyclictest41-21ksoftirqd/320:50:153
26339993832,4cyclictest33-21ksoftirqd/221:20:002
2633999381,34cyclictest1453-21kworker/2:021:35:162
26338993831,4cyclictest25-21ksoftirqd/119:35:001
2633799386,8cyclictest0-21swapper/019:30:030
2633799382,5cyclictest302-21apt-get00:35:110
2633799382,5cyclictest16886-21apt-get00:15:140
2633799382,5cyclictest12670-21apt-get00:10:130
2633799382,32cyclictest10524-21wc20:10:020
2633799382,32cyclictest0-21swapper/023:25:120
2633799380,8cyclictest27869-21smartctl23:50:050
120891380,11ptp4l0-21swapper/300:12:513
120882380,5sleep233-21ksoftirqd/223:30:102
26340993731,4cyclictest41-21ksoftirqd/321:30:023
26340993731,3cyclictest41-21ksoftirqd/300:20:003
26340993730,4cyclictest41-21ksoftirqd/322:55:033
26340993730,4cyclictest41-21ksoftirqd/322:55:023
26340993730,4cyclictest41-21ksoftirqd/300:00:173
26339993731,4cyclictest33-21ksoftirqd/200:00:072
26339993731,3cyclictest33-21ksoftirqd/222:50:012
26339993730,4cyclictest33-21ksoftirqd/220:55:162
26339993730,4cyclictest33-21ksoftirqd/200:40:002
26339993729,5cyclictest33-21ksoftirqd/200:35:012
26339993729,4cyclictest33-21ksoftirqd/222:55:022
26339993729,4cyclictest33-21ksoftirqd/222:55:022
26338993731,4cyclictest25-21ksoftirqd/123:55:141
2633799379,25cyclictest31620-21fschecks_count19:55:090
2633799379,24cyclictest26547-21irqrtprio20:30:010
2633799372,5cyclictest0-21swapper/021:55:160
2633799371,4cyclictest26895-21/usr/sbin/munin23:10:130
26337993714,20cyclictest26883-21processes19:50:040
26337993712,22cyclictest20273-21/usr/sbin/munin22:20:200
2633799370,5cyclictest18429-21users23:35:330
230642370,8sleep32634099cyclictest23:45:003
26340993631,3cyclictest41-21ksoftirqd/321:45:013
26340993630,3cyclictest41-21ksoftirqd/323:29:533
26340993630,3cyclictest41-21ksoftirqd/321:45:093
26340993629,4cyclictest41-21ksoftirqd/321:55:123
26340993629,4cyclictest41-21ksoftirqd/319:45:023
26340993622,4cyclictest41-21ksoftirqd/322:20:233
26339993631,3cyclictest33-21ksoftirqd/222:00:002
26339993630,4cyclictest33-21ksoftirqd/222:41:182
26339993630,4cyclictest33-21ksoftirqd/222:05:152
26339993629,4cyclictest33-21ksoftirqd/222:25:142
26339993629,4cyclictest33-21ksoftirqd/219:20:022
26339993628,5cyclictest33-21ksoftirqd/219:15:142
26339993628,4cyclictest33-21ksoftirqd/220:40:172
26339993627,6cyclictest33-21ksoftirqd/220:00:142
26339993627,4cyclictest33-21ksoftirqd/223:35:132
26338993631,3cyclictest25-21ksoftirqd/122:10:251
26338993630,4cyclictest25-21ksoftirqd/121:35:131
2633799368,4cyclictest7947-21missed_timers00:05:100
2633799367,4cyclictest6942-21cat21:25:050
26337993630,3cyclictest9-21ksoftirqd/022:46:550
2633799361,6cyclictest4146-21apt-get22:40:130
120891360,1ptp4l401ktimersoftd/322:25:163
120891360,1ptp4l401ktimersoftd/319:15:123
26340993529,4cyclictest41-21ksoftirqd/323:50:333
26340993529,4cyclictest41-21ksoftirqd/322:50:113
26340993529,4cyclictest41-21ksoftirqd/321:10:113
26340993529,4cyclictest41-21ksoftirqd/320:55:093
26340993528,4cyclictest41-21ksoftirqd/321:35:163
26340993528,4cyclictest41-21ksoftirqd/320:25:043
26340993528,4cyclictest41-21ksoftirqd/320:25:043
26340993528,4cyclictest41-21ksoftirqd/319:57:283
26339993529,4cyclictest33-21ksoftirqd/200:15:112
26339993529,3cyclictest33-21ksoftirqd/222:18:172
26339993529,3cyclictest33-21ksoftirqd/219:10:142
26339993528,4cyclictest33-21ksoftirqd/221:48:512
26339993527,5cyclictest33-21ksoftirqd/221:14:082
26339993527,4cyclictest33-21ksoftirqd/223:20:012
26339993525,5cyclictest33-21ksoftirqd/220:05:142
26339993518,6cyclictest33-21ksoftirqd/221:50:252
26338993530,3cyclictest25-21ksoftirqd/123:35:121
26338993530,3cyclictest25-21ksoftirqd/122:40:151
26338993530,3cyclictest25-21ksoftirqd/122:00:141
26338993529,4cyclictest25-21ksoftirqd/120:35:021
26338993528,4cyclictest25-21ksoftirqd/122:45:561
26338993527,5cyclictest25-21ksoftirqd/120:34:241
2633899350,4cyclictest121rcu_preempt00:05:321
2633799358,24cyclictest24247-21acpi23:05:120
2633799357,5cyclictest31624-21df_abs23:15:050
2633799355,4cyclictest6094-21/usr/sbin/munin20:45:000
2633799355,4cyclictest12266-21cpu22:10:150
26337993528,4cyclictest9-21ksoftirqd/020:22:340
26337993528,4cyclictest9-21ksoftirqd/019:23:440
2633799350,11cyclictest6126-21sed20:05:000
120891350,1ptp4l401ktimersoftd/320:48:153
26340993429,3cyclictest41-21ksoftirqd/322:40:143
26340993429,3cyclictest41-21ksoftirqd/322:30:093
26340993428,4cyclictest41-21ksoftirqd/321:21:183
26340993428,4cyclictest41-21ksoftirqd/319:25:253
26340993427,4cyclictest41-21ksoftirqd/321:50:113
26340993427,4cyclictest41-21ksoftirqd/321:16:383
26340993427,4cyclictest41-21ksoftirqd/320:40:003
26340993427,4cyclictest41-21ksoftirqd/319:12:553
26339993429,3cyclictest33-21ksoftirqd/222:20:122
26339993428,4cyclictest33-21ksoftirqd/221:30:022
26339993428,4cyclictest33-21ksoftirqd/220:30:312
26339993427,4cyclictest33-21ksoftirqd/223:57:382
26339993427,4cyclictest33-21ksoftirqd/223:45:122
26339993427,4cyclictest33-21ksoftirqd/222:04:252
26339993427,4cyclictest33-21ksoftirqd/219:47:262
2633899344,3cyclictest121rcu_preempt21:29:511
26338993427,4cyclictest25-21ksoftirqd/122:17:581
26338993427,4cyclictest25-21ksoftirqd/121:21:461
26338993427,4cyclictest25-21ksoftirqd/120:28:051
26338993427,4cyclictest25-21ksoftirqd/120:28:041
26338993427,4cyclictest25-21ksoftirqd/119:40:391
26338993427,4cyclictest25-21ksoftirqd/100:30:141
26338993426,5cyclictest25-21ksoftirqd/121:10:571
2633799346,3cyclictest24275-21cat19:45:240
26337993427,4cyclictest9-21ksoftirqd/019:13:120
26337993426,5cyclictest9-21ksoftirqd/020:26:000
26337993426,5cyclictest9-21ksoftirqd/020:25:590
2633799341,5cyclictest16780-21sed20:55:230
2633799341,4cyclictest10192-21ptp4l-jitter20:10:000
120891340,1ptp4l401ktimersoftd/320:21:403
120891340,1ptp4l401ktimersoftd/320:10:353
120891340,1ptp4l401ktimersoftd/319:44:583
26340993327,4cyclictest41-21ksoftirqd/300:21:303
26340993326,4cyclictest41-21ksoftirqd/319:35:033
26340993326,4cyclictest41-21ksoftirqd/319:30:053
26339993327,4cyclictest33-21ksoftirqd/220:35:342
26339993327,4cyclictest33-21ksoftirqd/220:10:142
26339993327,4cyclictest33-21ksoftirqd/200:05:012
26339993326,4cyclictest33-21ksoftirqd/223:50:012
26339993326,4cyclictest33-21ksoftirqd/223:06:042
26338993327,4cyclictest25-21ksoftirqd/121:20:001
26338993326,4cyclictest25-21ksoftirqd/123:27:281
26338993326,4cyclictest25-21ksoftirqd/123:06:401
26338993326,4cyclictest25-21ksoftirqd/122:22:441
26338993326,4cyclictest25-21ksoftirqd/100:22:321
2633799337,3cyclictest32372-21tr20:35:150
2633799337,3cyclictest19638-21expr23:40:050
120891330,1ptp4l401ktimersoftd/320:05:243
26340993226,4cyclictest41-21ksoftirqd/322:00:263
26340993224,4cyclictest41-21ksoftirqd/323:05:153
26339993226,4cyclictest33-21ksoftirqd/223:20:062
26339993226,4cyclictest33-21ksoftirqd/200:10:122
26339993226,3cyclictest33-21ksoftirqd/220:50:152
26339993226,3cyclictest33-21ksoftirqd/219:25:142
26339993225,4cyclictest33-21ksoftirqd/221:26:562
26339993225,4cyclictest33-21ksoftirqd/219:54:402
26339993225,4cyclictest33-21ksoftirqd/200:27:112
26339993225,3cyclictest33-21ksoftirqd/219:40:022
26339993224,5cyclictest33-21ksoftirqd/222:50:552
26339993216,7cyclictest33-21ksoftirqd/219:55:092
26339993215,4cyclictest33-21ksoftirqd/220:25:052
26339993215,4cyclictest33-21ksoftirqd/220:25:052
26338993227,3cyclictest25-21ksoftirqd/120:15:331
26338993227,3cyclictest25-21ksoftirqd/100:30:011
26338993226,4cyclictest25-21ksoftirqd/123:54:381
26338993226,4cyclictest25-21ksoftirqd/121:55:121
26338993226,4cyclictest25-21ksoftirqd/120:50:111
26338993226,4cyclictest25-21ksoftirqd/120:10:391
26338993225,4cyclictest25-21ksoftirqd/123:15:001
26338993225,4cyclictest25-21ksoftirqd/100:10:481
26338993224,5cyclictest25-21ksoftirqd/123:15:391
26338993224,5cyclictest25-21ksoftirqd/120:58:281
26338993219,7cyclictest241ktimersoftd/123:40:101
2633799329,20cyclictest15537-21entropy20:15:110
2633799324,2cyclictest0-21swapper/022:25:120
26339993125,4cyclictest33-21ksoftirqd/223:40:162
26339993125,3cyclictest33-21ksoftirqd/223:10:152
26339993124,4cyclictest33-21ksoftirqd/223:26:462
26338993126,3cyclictest25-21ksoftirqd/122:59:041
26338993126,3cyclictest25-21ksoftirqd/122:59:041
26338993125,4cyclictest25-21ksoftirqd/119:55:101
26338993125,4cyclictest25-21ksoftirqd/119:45:421
26338993125,4cyclictest25-21ksoftirqd/100:01:451
26338993124,4cyclictest25-21ksoftirqd/123:30:321
26338993124,4cyclictest25-21ksoftirqd/122:37:131
26338993124,4cyclictest25-21ksoftirqd/121:00:271
26338993124,4cyclictest25-21ksoftirqd/119:22:421
120891310,1ptp4l401ktimersoftd/321:29:013
26340993024,4cyclictest41-21ksoftirqd/320:00:053
26340993023,4cyclictest41-21ksoftirqd/323:49:463
26339993025,3cyclictest33-21ksoftirqd/221:08:302
26339993024,3cyclictest33-21ksoftirqd/200:24:582
26338993025,3cyclictest25-21ksoftirqd/120:00:131
26338993024,4cyclictest34992irq/88-enp1s0-T20:05:161
26338993024,4cyclictest25-21ksoftirqd/123:01:301
26338993024,4cyclictest25-21ksoftirqd/120:43:441
26338993023,4cyclictest25-21ksoftirqd/121:06:551
26339992922,4cyclictest33-21ksoftirqd/219:30:142
26338992923,4cyclictest25-21ksoftirqd/100:35:251
26338992923,3cyclictest25-21ksoftirqd/119:18:401
26338992922,4cyclictest25-21ksoftirqd/122:09:511
26338992922,4cyclictest25-21ksoftirqd/121:42:411
26340992822,4cyclictest41-21ksoftirqd/321:05:103
26338992822,4cyclictest25-21ksoftirqd/122:50:281
221002285,19sleep222251-21sed23:00:322
26339992721,4cyclictest33-21ksoftirqd/221:00:322
26338992720,4cyclictest25-21ksoftirqd/119:53:001
26338992719,5cyclictest25-21ksoftirqd/121:32:571
26338992620,3cyclictest25-21ksoftirqd/122:27:211
2633899254,3cyclictest121rcu_preempt21:45:291
26338992519,3cyclictest25-21ksoftirqd/119:36:401
26338992518,5cyclictest25-21ksoftirqd/123:20:341
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional