You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-19 - 20:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Asem

Intel(R) Celeron(R) CPU J1900 @ 1.99GHz, Linux 4.19.13-rt10-intel-pk-preempt-rt-asem-1.2.1+ (Profile)

Latency plot of shadow in rack #2, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack2slot6s.osadl.org (updated Mon Mar 09, 2020 00:43:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
159792346338,4sleep20-21swapper/219:33:542
162472319312,4sleep00-21swapper/019:37:220
162912218210,5sleep30-21swapper/319:37:573
162942211204,4sleep10-21swapper/119:37:591
172442660,5sleep21664299cyclictest23:13:362
7272630,2sleep10-21swapper/123:48:311
203822580,4sleep120384-21latency_hist23:23:141
309362480,2sleep30-21swapper/322:33:223
309362480,2sleep30-21swapper/322:33:223
16645992412,2cyclictest20037-21switchtime22:08:283
251382231,18sleep325135-21apt-get23:33:173
1664299231,19cyclictest0-21swapper/221:23:202
16640992316,6cyclictest29697-21kworker/1:000:32:141
16640992316,6cyclictest29697-21kworker/1:000:32:141
1664299220,19cyclictest0-21swapper/221:28:142
16637992216,3cyclictest31135-21grep23:43:320
243512210,16sleep00-21swapper/000:43:190
1664299217,12cyclictest0-21swapper/221:13:272
16642992115,3cyclictest2392-21grep01:03:352
16640992115,3cyclictest5236-21grep20:23:271
1664099211,17cyclictest24369-21grep23:28:311
16637992117,2cyclictest16645-21cyclictest19:38:090
195862200,2sleep019779-21cstates20:58:140
16645992015,3cyclictest2309-21grep01:03:343
1664299203,9cyclictest0-21swapper/220:33:112
1664299203,9cyclictest0-21swapper/220:33:112
1664299203,10cyclictest0-21swapper/221:38:132
16642992014,3cyclictest22899-21tr19:53:122
16640992016,2cyclictest29697-21kworker/1:023:58:231
16640992015,3cyclictest14626-21grep00:18:381
16640992014,3cyclictest7483-21latency22:53:231
1664099200,2cyclictest0-21swapper/123:03:351
16637992015,3cyclictest32576-21grep21:23:330
16645991912,5cyclictest5958-1kworker/3:2H20:08:393
1664299197,10cyclictest0-21swapper/222:04:372
1664299193,9cyclictest0-21swapper/222:38:142
1664299192,10cyclictest0-21swapper/200:08:162
1664299191,15cyclictest0-21swapper/220:03:122
1664299190,3cyclictest0-21swapper/221:21:052
16640991914,3cyclictest27883-21aten2_rbpower_c22:28:151
16640991914,3cyclictest26474-21grep19:58:281
16640991914,3cyclictest26426-21aten2_rbpower_p21:13:181
16640991914,3cyclictest1575-21if_err_enp3s021:28:211
16640991913,4cyclictest16006-21tr23:13:221
16640991913,4cyclictest12392-21latency_hist20:43:101
16640991913,3cyclictest30901-21cat23:43:301
16640991913,3cyclictest28962-21grep23:38:331
9792180,2sleep00-21swapper/020:15:470
84742180,3sleep00-21swapper/022:55:560
197932180,2sleep00-21swapper/023:18:450
1664599183,9cyclictest0-21swapper/300:33:163
1664599182,13cyclictest9502-21cut22:58:203
1664599182,13cyclictest444-21grep22:38:203
1664599182,13cyclictest20014-21cron19:48:083
1664599182,10cyclictest0-21swapper/321:03:123
16645991813,3cyclictest22608-21/usr/sbin/munin19:53:173
16645991812,4cyclictest7327-21if_enp1s022:53:213
16645991812,4cyclictest5914-21latency_hist00:03:153
16645991812,3cyclictest7504-21threads00:03:323
16645991812,3cyclictest415-21sh20:13:253
16645991812,3cyclictest3946-21cat20:23:133
16645991812,3cyclictest22905-21interrupts00:38:253
16645991812,3cyclictest14211-21/usr/sbin/munin21:58:233
1664599181,14cyclictest0-21swapper/320:28:113
1664299187,10cyclictest0-21swapper/220:23:442
1664299186,10cyclictest6834-21if_enp1s000:03:242
1664299186,10cyclictest0-21swapper/219:58:192
1664299183,9cyclictest0-21swapper/220:53:112
1664299183,9cyclictest0-21swapper/200:13:162
16642991813,3cyclictest3904-21apt-get23:58:162
16642991813,3cyclictest13012-21apt-get00:18:222
16642991812,3cyclictest3671-21cron23:58:142
1664299181,15cyclictest0-21swapper/222:28:252
1664299181,14cyclictest85732sleep222:57:142
1664299181,14cyclictest31598-21apt-get20:13:122
16642991811,4cyclictest14741-21latency_hist20:48:102
16642991811,4cyclictest11753-21cut23:03:202
1664299180,17cyclictest0-21swapper/219:50:292
1664099189,7cyclictest29562-21cpuspeed_turbos20:08:121
1664099188,7cyclictest26230-21gltestperf22:23:231
1664099183,9cyclictest13385-21apt-get23:08:161
16640991813,3cyclictest21883-21threads23:23:401
16640991813,3cyclictest20947-21/usr/sbin/munin22:13:171
16640991812,4cyclictest24027-21sed19:53:251
16640991812,3cyclictest8865-21acpi22:58:141
16640991811,4cyclictest6894-21if_err_enp1s000:03:241
1663799183,8cyclictest13385-21apt-get23:08:160
16637991813,3cyclictest29560-21tr20:08:120
16637991813,3cyclictest20037-21switchtime22:08:280
16637991812,3cyclictest4896-21sed23:58:260
16637991812,3cyclictest32589-21grep22:38:170
16637991812,3cyclictest28193-21head00:48:340
16637991812,3cyclictest26054-21pzem_power19:58:250
16637991812,3cyclictest22669-21/usr/sbin/munin23:28:190
16637991812,3cyclictest14987-21/usr/sbin/munin20:48:360
16637991812,3cyclictest11213-21grep00:13:220
129062180,2sleep00-21swapper/023:05:200
101132180,3sleep10-21swapper/120:37:441
101132180,3sleep10-21swapper/120:37:441
314022170,2sleep20-21swapper/223:45:362
306822170,3sleep10-21swapper/100:56:161
229172170,2sleep00-21swapper/022:17:260
1664599173,9cyclictest0-21swapper/322:43:143
1664599173,9cyclictest0-21swapper/320:38:113
16645991712,3cyclictest8441-21aten2_rbpower_e00:08:183
16645991712,3cyclictest31832-21/usr/sbin/munin23:48:303
16645991712,3cyclictest31548-21gltestperf21:23:223
16645991712,3cyclictest27238-21hddtemp_smartct00:48:233
16645991712,3cyclictest27185-21awk22:23:343
16645991712,3cyclictest27079-21/usr/sbin/munin20:03:253
16645991712,3cyclictest26273-21dpkg21:13:113
16645991712,3cyclictest24318-21/usr/sbin/munin00:43:253
16645991712,3cyclictest23933-21pzem23:28:283
16645991712,3cyclictest17233-21/usr/sbin/munin20:53:193
16645991712,3cyclictest15726-21/usr/sbin/munin23:13:243
16645991712,3cyclictest13649-21ntp_kernel_pll_20:43:213
16645991712,3cyclictest12175-21ntp_states23:03:263
16645991711,4cyclictest23181-21/usr/sbin/munin22:18:163
1664599171,14cyclictest21449-21grep22:13:183
16645991711,4cyclictest20427-21aten2_rbpower_c19:48:143
16645991711,3cyclictest5808-21grep21:38:193
16645991711,3cyclictest29307-21awk21:18:183
16645991711,3cyclictest18344-21grep23:18:213
16645991710,4cyclictest5294-21cat22:48:233
16645991710,4cyclictest20177-21if_enp1s020:58:183
1664299177,7cyclictest0-21swapper/220:48:112
1664299175,10cyclictest0-21swapper/221:08:112
1664299173,8cyclictest0-21swapper/222:23:072
1664299172,9cyclictest0-21swapper/223:48:172
1664299171,3cyclictest25062-21find23:33:152
16642991712,3cyclictest5865-21/usr/sbin/munin20:28:202
16642991712,3cyclictest4383-21/usr/sbin/munin22:48:172
16642991712,3cyclictest22669-21/usr/sbin/munin23:28:242
16642991712,3cyclictest20947-21/usr/sbin/munin22:13:222
1664299171,14cyclictest0-21swapper/223:18:162
1664299171,13cyclictest21731-21apt-get21:03:112
16642991711,3cyclictest19177-21ntp_states19:43:112
16642991711,3cyclictest11977-21/usr/sbin/munin21:53:152
16642991710,4cyclictest17912-21df_inode00:28:212
16642991710,4cyclictest17912-21df_inode00:28:212
1664099178,7cyclictest11215-21gltestperf00:13:221
1664099178,6cyclictest23711-21sed00:38:341
1664099174,8cyclictest28594-21apt-get21:18:111
1664099173,8cyclictest19499-21apt-get20:58:111
1664099173,8cyclictest1664-21apt-get23:53:171
1664099173,12cyclictest27079-21/usr/sbin/munin20:03:211
1664099173,10cyclictest0-21swapper/120:43:121
16640991714,2cyclictest15478-21kworker/1:221:34:001
16640991712,3cyclictest26576-21/usr/sbin/munin00:48:191
16640991712,3cyclictest19090-21cpu19:43:101
16640991712,3cyclictest1099-21pzem_voltage22:38:281
16640991711,4cyclictest23842-21sed21:08:101
1664099171,14cyclictest17599-21grep19:38:221
16640991711,4cyclictest14301-21cut21:58:141
16640991711,3cyclictest9526-21latency_hist21:48:121
16640991711,3cyclictest5847-21grep21:38:191
16640991711,3cyclictest31578-21gltestperf21:23:221
16640991711,3cyclictest23772-21gltestperf22:18:201
1663799178,6cyclictest27604-21cpuspeed_turbos23:38:180
1663799173,9cyclictest27879-21apt-get22:28:150
1663799173,8cyclictest1400-21apt-get20:18:110
16637991712,3cyclictest9394-21mailstats20:33:250
16637991712,3cyclictest9394-21mailstats20:33:250
16637991712,3cyclictest29134-21gltestperf21:18:170
16637991712,3cyclictest26228-21sed22:23:230
16637991712,3cyclictest16028-21cstates23:13:230
16637991712,3cyclictest15489-21sendmail_mailtr21:58:280
16637991712,3cyclictest11431-21ntp_states20:38:210
16637991711,3cyclictest28807-21/usr/sbin/munin00:53:170
16637991711,3cyclictest2799-21latency_hist21:33:120
16637991711,3cyclictest14687-21sed20:48:100
16637991711,3cyclictest14552-21unixbench_multi00:18:380
16637991710,5cyclictest537-21runrttasks00:32:480
16637991710,5cyclictest537-21runrttasks00:32:480
16637991710,4cyclictest8553-21cpu00:08:180
16637991710,4cyclictest509-21dbus-daemon21:43:270
16637991710,4cyclictest22597-21fschecks_time00:38:220
1664599163,10cyclictest41-21ksoftirqd/320:33:123
1664599163,10cyclictest41-21ksoftirqd/320:33:123
16645991611,3cyclictest9801-21aten2_rbpower_e21:48:153
16645991611,3cyclictest9679-21cat21:48:123
16645991611,3cyclictest755-21/usr/sbin/munin21:28:293
16645991611,3cyclictest537-21runrttasks23:23:203
16645991611,3cyclictest31031-21ls23:43:313
16645991611,3cyclictest27343-21/usr/sbin/munin23:38:223
16645991611,3cyclictest2593-21sed20:18:233
16645991611,3cyclictest23961-21/usr/sbin/munin21:08:153
16645991611,3cyclictest16832-21aten2_rbpower_v19:38:163
16645991611,3cyclictest16451-21/usr/sbin/munin22:03:303
16645991611,3cyclictest15300-21cat20:48:213
16645991611,3cyclictest10614-21/usr/sbin/munin00:13:243
16645991610,3cyclictest3669-21ls21:33:183
16645991610,3cyclictest29218-21cat00:53:203
16645991610,3cyclictest26879-21latency_hist20:03:083
16645991610,3cyclictest2681-21ntp_kernel_err23:53:263
1664299167,8cyclictest0-21swapper/223:00:122
1664299167,6cyclictest0-21swapper/223:08:252
1664299166,7cyclictest1970-21gltestperf20:18:152
16642991611,3cyclictest3831-21irqstats21:33:202
16642991611,3cyclictest29649-21seq20:08:122
16642991611,3cyclictest28923-21aten2_rbpower_v00:53:192
16642991611,3cyclictest23307-21pzem00:38:312
16642991611,3cyclictest18758-21aten2_rbpower_c22:08:142
16642991611,3cyclictest16573-21sed00:23:312
16642991611,3cyclictest12522-21cat20:43:102
16642991610,3cyclictest31408-21expr00:58:202
16642991610,3cyclictest30591-21fschecks_time22:33:192
16642991610,3cyclictest30591-21fschecks_time22:33:192
16642991610,3cyclictest10138-21cat21:48:162
1664299160,3cyclictest0-21swapper/222:27:172
1664299160,3cyclictest0-21swapper/221:58:162
1664299160,15cyclictest0-21swapper/220:58:392
1664099164,7cyclictest12005-21apt-get21:53:141
1664099163,8cyclictest31070-21apt-get00:58:181
16640991611,3cyclictest4167-21latency_hist22:48:131
16640991611,3cyclictest32289-21if_enp1s020:13:171
16640991611,3cyclictest31932-21seq22:35:521
16640991611,3cyclictest31932-21seq22:35:521
16640991611,3cyclictest24723-21cat00:43:201
16640991611,3cyclictest19441-21cat22:08:191
16640991611,3cyclictest16532-21ls20:48:381
16640991611,3cyclictest16248-21cut00:23:261
16640991611,3cyclictest11327-21sed21:48:301
16640991611,3cyclictest10029-21users00:08:331
16640991610,3cyclictest5976-21aten2_rbpower_p20:28:131
16640991610,3cyclictest25389-21grep23:33:221
16640991610,3cyclictest24625-21smartctl21:08:191
16640991610,3cyclictest2417-21grep01:03:351
16640991610,3cyclictest21075-21ls19:48:201
16640991610,3cyclictest20196-21sort00:33:211
1664099160,13cyclictest0-21swapper/120:20:341
1663799163,8cyclictest25135-21apt-get23:33:200
1663799163,8cyclictest20508-21apt-get23:23:240
1663799163,7cyclictest6141-21apt-get00:03:160
16637991611,3cyclictest9532-21ls22:58:210
16637991611,3cyclictest537-21runrttasks23:54:180
16637991611,3cyclictest537-21runrttasks21:54:470
16637991611,3cyclictest537-21runrttasks19:43:570
16637991611,3cyclictest537-21runrttasks00:36:330
16637991611,3cyclictest3730-21aten2_rbpower_v20:23:120
16637991611,3cyclictest31303-21pzem_voltage22:33:270
16637991611,3cyclictest31303-21pzem_voltage22:33:270
16637991611,3cyclictest18215-21sed20:53:210
16637991611,3cyclictest16441-21pzem_voltage00:23:300
1663799161,12cyclictest4643-21grep22:48:170
16637991610,4cyclictest31860-21apt-get23:48:160
16637991610,3cyclictest9804-21aten2_rbpower_e21:48:150
16637991610,3cyclictest889-21aten2_rbpower_c01:03:190
16637991610,3cyclictest6178-21cstates20:28:140
16637991610,3cyclictest537-21runrttasks21:07:060
16637991610,3cyclictest27773-21idleruntime20:03:170
16637991610,3cyclictest24691-21if_enp2s021:08:200
16637991610,3cyclictest16763-21cstates22:03:160
1664599159,4cyclictest18840-21sort00:28:323
1664599159,4cyclictest18840-21sort00:28:323
1664599159,3cyclictest15265-21basename00:23:183
1664599158,4cyclictest13749-21expr23:08:183
1664599155,8cyclictest31286-21cpuspeed_turbos00:58:193
16645991510,3cyclictest27849-21/usr/sbin/munin22:28:193
16645991510,3cyclictest13120-21aten2_rbpower_v00:18:243
16645991510,3cyclictest13017-21ntp_kernel_err21:53:233
1664299151,11cyclictest0-21swapper/200:33:212
16642991510,3cyclictest7781-21cstates21:43:152
16642991510,3cyclictest28092-21cat23:38:232
16642991510,3cyclictest26884-21cstates00:48:202
16642991510,3cyclictest24318-21/usr/sbin/munin00:43:312
16642991510,3cyclictest2259-21aten2_rbpower_v22:43:162
16642991510,3cyclictest20527-21aten2_rbpower_c23:23:242
1664299150,3cyclictest0-21swapper/219:38:112
1664099151,11cyclictest17030-21grep22:03:191
16640991510,3cyclictest4508-21aten2_rbpower_v22:48:161
1663799159,3cyclictest22409-21latency_hist19:53:080
16637991510,3cyclictest81ktimersoftd/021:33:120
16637991510,3cyclictest6173-21cut21:38:220
16637991510,3cyclictest537-21runrttasks22:19:140
16637991510,3cyclictest31487-21forks00:58:220
16637991510,3cyclictest26808-21fschecks_count21:13:210
16637991510,3cyclictest2431-21cstates22:43:170
16637991510,3cyclictest23472-21latency19:53:180
1664099149,3cyclictest17528-21cstates20:53:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional