You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-04-12 - 09:04

Intel(R) Atom(TM) CPU E3845 @ 1.91GHz, Linux 4.19.37-rt20 (Profile)

Latency plot of system in rack #b, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot6.osadl.org (updated Mon Apr 12, 2021 00:43:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1366921070,2sleep20-21swapper/223:07:102
873911000,4ptp4l401ktimersoftd/320:06:403
296062720,5sleep341-21ksoftirqd/320:12:133
237892700,3sleep341-21ksoftirqd/323:17:153
87391650,1ptp4l401ktimersoftd/322:34:413
253392634,10sleep10-21swapper/120:05:321
275812580,2sleep30-21swapper/301:00:433
275812580,2sleep30-21swapper/301:00:433
25700995810,18cyclictest15071-21head23:07:261
245992580,3sleep12570099cyclictest23:17:251
73812570,2sleep30-21swapper/300:39:473
25700995711,13cyclictest28097-21munin-node20:12:011
251612574,42sleep00-21swapper/020:03:130
25700995610,14cyclictest25312-21rm00:57:261
25700995610,14cyclictest25312-21rm00:57:261
25700995510,12cyclictest9639-21if_enp1s0.100:42:161
258732540,2sleep30-21swapper/322:47:033
25700995410,17cyclictest11248-21/usr/sbin/munin22:32:131
56432530,2sleep25645-21ssh22:26:062
2570099539,17cyclictest19067-21diskmemload22:54:181
25700995210,12cyclictest28494-21sh23:22:101
2570099509,17cyclictest6308-21latency23:32:161
25700995017,11cyclictest20932-21apt-get01:27:111
129732500,2sleep20-21swapper/223:39:022
2570099488,11cyclictest8314-21rm01:12:511
87391470,2ptp4l30806-21kworker/3:022:13:103
25701994735,6cyclictest33-21ksoftirqd/222:17:042
87391460,1ptp4l401ktimersoftd/321:17:113
2570099468,15cyclictest25098-21ssh22:46:061
2570099458,16cyclictest28851-21ssh00:27:431
2570099457,17cyclictest9990-21ssh22:30:151
2570099455,17cyclictest0-21swapper/122:23:111
87391440,1ptp4l30806-21kworker/3:022:31:513
2570099447,11cyclictest404-21rm23:59:321
87391430,2ptp4l401ktimersoftd/320:06:593
233602433,25sleep223280-21/usr/sbin/munin20:02:022
25700994214,13cyclictest20892-21ssh22:07:201
87391400,1ptp4l401ktimersoftd/300:22:473
87391400,0ptp4l401ktimersoftd/300:03:003
25700994024,9cyclictest16986-21apt-get00:17:071
25700994013,16cyclictest31300-21sh22:17:541
25700994010,17cyclictest19067-21diskmemload00:10:501
87391390,1ptp4l401ktimersoftd/322:17:313
87391390,1ptp4l401ktimersoftd/301:17:343
87391390,1ptp4l401ktimersoftd/300:47:103
87391390,1ptp4l401ktimersoftd/300:20:293
25700993913,8cyclictest27261-21snmp_o5powerctl01:32:251
25700993911,13cyclictest8667-21cat21:42:221
87391380,1ptp4l401ktimersoftd/323:54:023
87391380,1ptp4l401ktimersoftd/323:45:293
87391380,1ptp4l401ktimersoftd/323:26:583
87391380,1ptp4l401ktimersoftd/322:23:163
87391380,1ptp4l401ktimersoftd/300:11:553
25700993823,9cyclictest28693-21apt-get01:02:101
2569999384,21cyclictest28702-21sh01:02:100
87391370,1ptp4l401ktimersoftd/323:58:443
87391370,1ptp4l401ktimersoftd/323:31:293
87391370,1ptp4l401ktimersoftd/323:13:323
87391370,1ptp4l401ktimersoftd/322:52:473
87391370,1ptp4l401ktimersoftd/322:09:433
87391370,1ptp4l401ktimersoftd/320:21:133
87391370,1ptp4l401ktimersoftd/301:34:193
87391370,1ptp4l401ktimersoftd/301:11:033
87391370,1ptp4l401ktimersoftd/301:02:343
87391370,0ptp4l401ktimersoftd/300:15:483
2570299372,12cyclictest21094-21ssh22:42:043
2570099379,18cyclictest21782-21/usr/sbin/munin00:22:181
25700993713,20cyclictest19771-21cat23:47:041
25700993710,16cyclictest19067-21diskmemload00:40:211
25700993710,14cyclictest18742-21ssh01:23:531
87391360,1ptp4l401ktimersoftd/323:07:073
87391360,0ptp4l401ktimersoftd/300:33:413
25702993612,3cyclictest7441-21phc2sys-jitter01:12:213
2570099369,16cyclictest19067-21diskmemload01:20:451
2570099369,12cyclictest13292-21phc2sys00:12:171
2570099366,11cyclictest2862-21diskmemload23:28:251
25700993621,9cyclictest18115-21apt-get23:12:051
87391350,1ptp4l401ktimersoftd/301:22:463
87391350,0ptp4l401ktimersoftd/301:31:533
25702993513,3cyclictest15036-21head21:57:133
25702993512,4cyclictest19164-21expr00:52:133
25701993526,4cyclictest33-21ksoftirqd/220:37:012
2570099359,8cyclictest19067-21diskmemload22:13:031
2570099356,18cyclictest0-21swapper/123:47:211
2570099356,18cyclictest0-21swapper/120:37:051
2569999356,16cyclictest13214-21ssh23:07:050
87391340,1ptp4l401ktimersoftd/322:39:403
87391340,12ptp4l401ktimersoftd/320:52:193
2570299343,9cyclictest5292-21ssh23:32:063
25702993412,3cyclictest6844-21sed20:32:133
25702993412,3cyclictest30774-21sh00:31:033
25701993424,5cyclictest33-21ksoftirqd/221:12:052
2570099347,13cyclictest0-21swapper/121:02:161
2570099347,11cyclictest0-21swapper/120:07:021
25700993422,6cyclictest10895-21apt-get20:42:021
25700993410,10cyclictest19067-21diskmemload22:58:581
2570299333,7cyclictest0-21swapper/321:47:033
25701993322,5cyclictest33-21ksoftirqd/221:27:042
25701993322,5cyclictest33-21ksoftirqd/221:27:042
2570099339,15cyclictest30604-21if_ens321:22:101
2570099336,14cyclictest1271-21ssh00:32:391
2570099335,9cyclictest0-21swapper/123:40:091
25700993322,5cyclictest31387-21apt-get20:17:051
25700993321,6cyclictest8360-21apt-get23:02:081
20242330,4sleep233-21ksoftirqd/201:07:162
2570299323,7cyclictest25221-21apt-get21:12:063
25702993214,3cyclictest9928-21switchtime20:37:203
25702993210,3cyclictest8629-21sensors_temp21:42:183
25701993222,5cyclictest33-21ksoftirqd/223:52:062
25701993222,5cyclictest33-21ksoftirqd/201:32:092
2570099327,11cyclictest0-21swapper/120:47:101
2570099326,12cyclictest0-21swapper/123:52:231
2570099326,11cyclictest0-21swapper/121:27:171
2570099326,11cyclictest0-21swapper/121:27:171
2570099326,10cyclictest0-21swapper/121:12:261
2570099325,11cyclictest0-21swapper/100:47:171
25700993221,6cyclictest18883-21apt-get00:52:121
259582310,2sleep00-21swapper/022:47:040
25701993126,3cyclictest33-21ksoftirqd/200:02:032
2570099317,16cyclictest19067-21diskmemload01:08:271
25700993114,14cyclictest16240-21ssh22:37:051
25702993012,3cyclictest849-21runrttasks20:31:343
25701993025,3cyclictest33-21ksoftirqd/223:02:082
25701993023,4cyclictest33-21ksoftirqd/223:27:102
25701993013,15cyclictest33-21ksoftirqd/222:48:102
25701992916,6cyclictest33-21ksoftirqd/222:27:032
2570199290,3cyclictest121rcu_preempt22:40:372
2570099294,11cyclictest0-21swapper/100:02:141
2570099293,11cyclictest0-21swapper/121:47:211
25700992915,11cyclictest0-21swapper/121:32:061
25700992915,11cyclictest0-21swapper/121:32:061
98942281,23sleep09901-21snmp_o5powerctl23:02:270
321752280,2sleep00-21swapper/023:58:200
25701992823,3cyclictest5887-21timerwakeupswit00:37:282
25701992823,3cyclictest33-21ksoftirqd/220:07:102
25701992817,5cyclictest33-21ksoftirqd/221:17:022
25700992817,7cyclictest24516-21/usr/sbin/munin21:07:241
25700992816,8cyclictest1629-21cut20:22:061
25700992815,9cyclictest0-21swapper/122:02:051
2569999282,16cyclictest19376-21ssh22:07:040
2570199273,3cyclictest121rcu_preempt00:17:042
25701992722,3cyclictest33-21ksoftirqd/223:47:032
25701992722,3cyclictest33-21ksoftirqd/221:32:162
25701992722,3cyclictest33-21ksoftirqd/221:32:162
25701992722,3cyclictest33-21ksoftirqd/221:11:592
25701992721,4cyclictest33-21ksoftirqd/201:03:382
25701992720,4cyclictest33-21ksoftirqd/200:52:112
25701992720,4cyclictest19474-21cut23:12:182
25701992718,7cyclictest33-21ksoftirqd/223:32:112
25701992717,8cyclictest33-21ksoftirqd/221:02:102
25701992716,9cyclictest33-21ksoftirqd/223:23:592
25700992716,8cyclictest6576-21grep21:37:231
25700992716,8cyclictest6415-21cut20:32:081
25700992716,8cyclictest15530-21/usr/sbin/munin21:57:161
25700992715,9cyclictest27909-21cut21:17:061
25700992715,9cyclictest17346-21sed20:52:251
25700992714,9cyclictest4140-21grep20:27:101
25700992714,9cyclictest26603-21if_ens322:47:121
25700992713,11cyclictest19108-21missed_timers20:57:141
2569999274,13cyclictest1572-21ssh22:22:040
87391260,1ptp4l401ktimersoftd/323:38:123
87391260,1ptp4l401ktimersoftd/321:22:033
87391260,10ptp4l401ktimersoftd/321:02:383
87391260,0ptp4l401ktimersoftd/321:10:303
87391260,0ptp4l401ktimersoftd/320:58:173
2570199264,3cyclictest121rcu_preempt23:17:282
25701992621,3cyclictest33-21ksoftirqd/221:57:262
25701992621,3cyclictest33-21ksoftirqd/221:37:022
25701992621,3cyclictest33-21ksoftirqd/220:16:592
25701992621,3cyclictest33-21ksoftirqd/201:22:102
25701992621,3cyclictest33-21ksoftirqd/200:17:162
25701992620,3cyclictest33-21ksoftirqd/222:12:092
25701992620,3cyclictest33-21ksoftirqd/221:52:122
25701992620,3cyclictest33-21ksoftirqd/220:31:592
25701992620,3cyclictest33-21ksoftirqd/201:02:052
25701992620,3cyclictest33-21ksoftirqd/201:02:052
25700992614,9cyclictest12154-21chrt21:52:041
87391250,1ptp4l401ktimersoftd/323:47:093
87391250,1ptp4l401ktimersoftd/322:57:053
87391250,1ptp4l401ktimersoftd/321:27:043
87391250,1ptp4l401ktimersoftd/321:27:043
87391250,1ptp4l401ktimersoftd/320:42:233
87391250,1ptp4l401ktimersoftd/320:26:023
25701992520,3cyclictest33-21ksoftirqd/223:47:172
25701992520,3cyclictest33-21ksoftirqd/221:42:242
25701992520,3cyclictest33-21ksoftirqd/220:47:062
25701992520,3cyclictest33-21ksoftirqd/200:42:172
25701992519,3cyclictest33-21ksoftirqd/222:32:062
25701992519,3cyclictest33-21ksoftirqd/222:07:052
25701992519,3cyclictest33-21ksoftirqd/220:57:122
25701992519,3cyclictest33-21ksoftirqd/220:42:112
25701992517,5cyclictest12274-21needreboot01:17:192
25701992513,9cyclictest33-21ksoftirqd/200:27:282
2570199250,6cyclictest121rcu_preempt00:47:212
25699992518,4cyclictest10319-21sed21:47:130
25699992517,5cyclictest28535-21sh01:02:050
25699992517,5cyclictest28535-21sh01:02:050
87391240,1ptp4l401ktimersoftd/323:04:433
87391240,1ptp4l401ktimersoftd/322:07:013
87391240,1ptp4l401ktimersoftd/321:37:033
87391240,1ptp4l401ktimersoftd/320:47:053
25701992419,3cyclictest33-21ksoftirqd/222:57:152
25701992419,3cyclictest33-21ksoftirqd/220:22:042
25701992419,3cyclictest33-21ksoftirqd/201:12:102
25701992418,3cyclictest33-21ksoftirqd/221:47:132
25701992418,3cyclictest33-21ksoftirqd/220:17:162
25701992418,3cyclictest33-21ksoftirqd/201:27:202
25701992413,3cyclictest33-21ksoftirqd/222:02:142
2569999243,12cyclictest20949-21ssh01:27:120
25699992417,4cyclictest9-21ksoftirqd/000:12:130
25699992416,6cyclictest9-21ksoftirqd/022:34:430
87391230,1ptp4l401ktimersoftd/321:32:023
87391230,1ptp4l401ktimersoftd/321:32:023
87391230,1ptp4l401ktimersoftd/300:42:113
87391230,0ptp4l401ktimersoftd/321:52:233
2570199233,11cyclictest21105-21ssh22:42:052
25701992318,3cyclictest33-21ksoftirqd/222:52:142
25701992318,3cyclictest33-21ksoftirqd/200:22:262
25701992317,3cyclictest33-21ksoftirqd/220:36:592
25701992317,3cyclictest33-21ksoftirqd/200:07:062
25701992316,4cyclictest32064-21cut00:32:112
2569999232,12cyclictest30832-21ssh22:52:050
25699992317,3cyclictest9-21ksoftirqd/000:52:290
25699992317,3cyclictest9-21ksoftirqd/000:47:090
25699992315,4cyclictest9895-21latency_hist23:37:030
25699992314,5cyclictest24856-21grep23:17:270
25699992311,5cyclictest9-21ksoftirqd/023:12:060
73402220,2sleep00-21swapper/023:01:030
25701992216,3cyclictest33-21ksoftirqd/220:52:542
2569999223,16cyclictest3223-21ssh00:02:190
2569999223,11cyclictest63-21kswapd001:22:080
2569999223,11cyclictest20046-21ssh23:47:100
2569999222,17cyclictest860-21gltestperf23:27:100
2569999222,12cyclictest1602-21kworker/u8:200:17:070
25699992217,3cyclictest9-21ksoftirqd/000:41:190
25699992216,4cyclictest9-21ksoftirqd/021:36:090
25699992216,4cyclictest9-21ksoftirqd/021:36:090
25699992214,5cyclictest16972-21ps22:37:140
25699992213,6cyclictest48550irq/100-ens3-Tx00:42:080
25701992116,3cyclictest33-21ksoftirqd/200:02:062
2569999219,9cyclictest15768-21cut23:42:150
25699992116,3cyclictest9-21ksoftirqd/022:27:130
25699992115,3cyclictest9-21ksoftirqd/021:09:110
25699992115,3cyclictest10913-21grep23:37:150
25699992114,4cyclictest7199-21/usr/sbin/munin21:42:160
25699992114,4cyclictest1620-21forks20:22:060
25699992113,5cyclictest16103-21gltestperf20:52:090
25699992113,4cyclictest6672-21gltestperf01:12:150
25701992015,3cyclictest33-21ksoftirqd/221:27:002
2569999209,8cyclictest26221-21gltestperf01:32:130
25699992015,3cyclictest9-21ksoftirqd/021:28:470
25699992015,3cyclictest9-21ksoftirqd/021:28:470
25699992015,3cyclictest9-21ksoftirqd/021:20:030
25699992015,3cyclictest9-21ksoftirqd/000:11:500
25699992015,3cyclictest27242-21grep00:27:120
25699992014,4cyclictest26147-21sh23:52:180
25699992014,3cyclictest9-21ksoftirqd/021:02:450
25699992014,3cyclictest32327-21sh23:26:440
25699992014,3cyclictest11625-21hddtemp_smartct01:17:130
25699992013,4cyclictest9-21ksoftirqd/020:49:200
25699992013,4cyclictest30181-21memory22:17:130
25699992013,4cyclictest22953-21unixbench_singl22:42:280
25699992013,4cyclictest12305-21forks21:52:060
2569999197,4cyclictest629-21rm00:32:210
2569999193,13cyclictest10045-21grep20:37:210
25699991913,4cyclictest23322-21mailstats00:22:200
25699991913,3cyclictest30086-21cat21:22:040
25699991913,3cyclictest2879-21sed01:07:270
25699991913,3cyclictest26338-21open_files21:12:230
25699991913,3cyclictest17984-21/usr/sbin/munin20:57:120
25699991913,3cyclictest16971-21seq22:02:070
25699991913,3cyclictest10816-21/usr/sbin/munin20:42:120
25699991912,4cyclictest7838-21awk20:32:250
25699991912,4cyclictest4318-21if_ens320:27:110
25699991912,4cyclictest15486-21sed21:57:150
25699991912,4cyclictest1064-21cron20:21:580
25699991812,3cyclictest9-21ksoftirqd/021:37:380
25699991812,3cyclictest29019-21cstates20:12:080
25699991811,4cyclictest25706-21acpi20:07:000
25699991811,4cyclictest24584-21fschecks_time22:12:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional