You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-19 - 18:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Pentium(R) CPU N4200 @ 1.10GHz, Linux 5.9.1-rt18 (Profile)

Latency plot of shadow in rack #c, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackcslot4s.osadl.org (updated Sun Sep 19, 2021 12:44:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
712699262,5cyclictest3559-21apt-get10:10:182
7126992618,4cyclictest17812-21apt-get07:40:162
712599261,22cyclictest13721-21snmp_o5powerctl07:25:391
712599261,22cyclictest13721-21snmp_o5powerctl07:25:391
712699252,5cyclictest13541-21apt-get10:40:132
712699252,19cyclictest10208-21apt-get10:30:172
7124992514,7cyclictest10684-21apt-get08:55:120
712499251,19cyclictest5210-21apt-get10:15:160
7127992414,8cyclictest10259-21cat12:05:183
712699242,18cyclictest15127-21apt-get12:20:152
712699241,6cyclictest15874-21file07:35:002
7126992416,4cyclictest18505-21apt-get10:55:142
7127992315,4cyclictest8329-21latency_hist10:25:003
7127992311,8cyclictest5712-21apt-get08:40:143
7126992317,3cyclictest21484-40apt-check07:50:072
712699231,4cyclictest14010-21apt-get09:05:152
7126992314,5cyclictest3091-21apt-get11:45:152
712699231,3cyclictest11864-21apt-get10:35:172
7126992312,7cyclictest9735-21apt-get12:05:142
712699230,20cyclictest30397-21http09:55:012
7125992314,6cyclictest20545-21apt-get11:00:151
712799229,9cyclictest26477-21apt-get08:05:173
7127992213,6cyclictest4982-21cron10:14:593
7126992212,6cyclictest17319-21apt-get09:15:162
712699221,18cyclictest18296-21lsb_release10:55:002
712699221,17cyclictest20880-21kworker/u8:108:10:202
712699220,5cyclictest2156-21perf08:30:002
7124992210,8cyclictest32150-21apt-get11:35:160
712799219,7cyclictest32637-21apt-get10:00:143
712799218,8cyclictest15191-21apt-get10:45:153
7127992112,6cyclictest30404-21http09:55:013
7127992112,6cyclictest11869-21pluginstate07:20:263
7127992110,7cyclictest8554-21apt-get10:25:153
712699212,13cyclictest9029-21apt-get08:50:142
7126992114,5cyclictest26990-21cat08:05:212
7126992113,6cyclictest17272-21cut09:14:592
7126992111,8cyclictest26271-21lsb_release08:05:002
712699211,16cyclictest20533-21apt-get11:00:152
712699210,4cyclictest17538-21cron07:40:002
712599218,9cyclictest15666-21apt-get09:10:131
712499219,8cyclictest22203-21apt-get11:05:160
712499219,8cyclictest13449-21apt-get12:15:170
7124992112,7cyclictest9154-21lsb_release07:15:010
712799203,10cyclictest3091-21apt-get11:45:133
712799202,13cyclictest16818-21apt-esm-hook12:25:163
712799202,12cyclictest695-21apt-esm-hook08:25:153
712799202,12cyclictest13541-21apt-get10:40:133
712799202,11cyclictest7575-21snmpd10:30:163
7127992014,3cyclictest21962-21cron11:05:003
7127992013,5cyclictest17499-21expr09:15:173
712799201,2cyclictest311ksoftirqd/307:25:443
712799201,2cyclictest311ksoftirqd/307:25:433
7127992012,6cyclictest5340-21cat11:50:203
7127992012,6cyclictest16093-21grep12:20:263
712799201,13cyclictest22312-21apt-get09:30:133
7127992010,6cyclictest856-21gdbus07:40:013
712699202,12cyclictest23976-21apt-esm-hook09:35:162
7126992015,3cyclictest17899-21timerandwakeup12:25:282
712699201,3cyclictest1184-21munin-run11:40:002
7126992013,5cyclictest21064-21which12:35:242
7126992013,4cyclictest1685-21munin-run10:05:002
712699201,2cyclictest0-21swapper/212:15:032
7126992012,6cyclictest31417-21sed08:20:002
7126992012,6cyclictest29933-21cat11:25:252
7126992012,5cyclictest12691-21apt-get07:25:212
7126992012,5cyclictest12691-21apt-get07:25:202
712699201,17cyclictest7157-21perf08:45:002
7125992016,2cyclictest4994-21sendmail_mailqu08:35:261
7125992013,5cyclictest8154-21expr07:10:161
7125992013,5cyclictest5852-21memory10:15:241
7125992013,5cyclictest496-21cut08:25:001
7125992013,5cyclictest25834-21tune2fs11:15:161
7125992013,5cyclictest22946-21load09:30:311
7125992013,5cyclictest15352-21expr10:45:151
7125992010,5cyclictest0-21swapper/112:15:031
712499203,12cyclictest8587-21apt-esm-hook10:25:150
712499202,13cyclictest19026-21apt-esm-hook09:20:150
7124992015,3cyclictest28621-21idleruntime-cro11:25:000
7124992015,3cyclictest13777-21sh09:05:000
7124992014,4cyclictest23241-21wc07:55:170
7124992014,4cyclictest15799-21latency_hist07:35:000
7124992013,5cyclictest6092-21grep08:40:170
7124992013,5cyclictest29297-21cat08:10:390
7124992013,5cyclictest11371-21sed10:30:350
712799197,8cyclictest7705-21apt-get07:10:123
712799195,9cyclictest7373-21apt-get08:45:153
712799192,11cyclictest32150-21apt-get11:35:153
7127991915,2cyclictest18474-21sh12:30:183
7127991915,2cyclictest14968-21sendmail_mailqu09:05:263
7127991912,5cyclictest6533-21grep10:15:343
7127991912,5cyclictest2480-21wc10:05:193
7127991912,5cyclictest18005-21expr07:40:183
7127991912,4cyclictest72750irq/130-enp2s008:10:193
7127991911,6cyclictest29055-21sort09:50:153
712799191,14cyclictest22929-21munin-run07:55:003
712799191,13cyclictest27191-21apt-get11:20:163
712799191,12cyclictest15697-21apt-esm-hook09:10:133
712799191,12cyclictest14399-21apt-esm-hook07:30:223
712699193,15cyclictest9118-21cron07:15:002
712699192,15cyclictest27069-21latency_hist09:45:002
7126991914,3cyclictest11394-21sh12:10:152
7126991912,5cyclictest2302-21awk11:40:212
7126991912,4cyclictest27290-21apt-get09:45:142
712699191,13cyclictest8554-21apt-get10:25:162
712699191,12cyclictest25508-21apt-get11:15:142
712699191,12cyclictest23147-21apt-get07:55:172
712699190,3cyclictest763-21cron10:20:002
712699190,3cyclictest25279-21perf11:15:002
712699190,17cyclictest30957-21apt-get09:55:172
712599199,8cyclictest32673-21cat11:35:211
712599199,8cyclictest29365-21cat11:25:171
7125991915,2cyclictest7542-21missed_timers10:20:241
7125991915,2cyclictest4086-21sendmail_mailtr11:45:241
7125991915,2cyclictest17225-21threads07:35:381
7125991914,3cyclictest6176-21latency_hist11:55:001
7125991914,3cyclictest14958-21rt-features09:05:261
7125991914,3cyclictest10277-21kernelversion12:05:181
7125991913,4cyclictest1840-21cat10:05:001
7125991912,5cyclictest9360-21dpkg07:15:141
7125991912,5cyclictest6738-21sensors_temp08:40:231
7125991912,5cyclictest2898-21cat11:45:001
7125991912,5cyclictest27477-21ls09:45:151
7125991912,5cyclictest25643-21open_inodes08:00:261
7125991912,5cyclictest20668-21cat07:45:341
7125991912,5cyclictest20382-21sshd09:21:461
7125991912,5cyclictest18144-21users12:25:301
7125991912,5cyclictest1683-21sendmail_mailqu08:25:261
712599191,12cyclictest11404-21apt-esm-hook12:10:151
712499192,12cyclictest9389-21apt-esm-hook07:15:150
712499192,12cyclictest9041-21apt-esm-hook08:50:150
712499192,11cyclictest9747-21apt-esm-hook12:05:140
7124991915,2cyclictest27523-21sh09:45:150
7124991913,4cyclictest3069-21sed08:30:260
7124991913,4cyclictest1887-21sed08:25:310
7124991912,5cyclictest7752-21awk11:55:300
7124991912,5cyclictest26795-21cut08:05:210
7124991912,5cyclictest18482-21sed10:55:010
712499191,15cyclictest16112-21cut09:10:160
712499191,13cyclictest11895-21apt-esm-hook10:35:170
712499190,15cyclictest32637-21apt-get10:00:150
712799182,12cyclictest1432-21apt-get11:40:133
7127991814,2cyclictest9997-21sendmail_mailst08:50:243
7127991813,3cyclictest30428-21missed_timers08:15:253
7127991813,2cyclictest20632-21kernelversion12:35:203
7127991811,5cyclictest6562-21expr11:55:173
7127991811,5cyclictest2744-21perl08:30:243
7127991811,5cyclictest25447-21sed11:15:013
7127991811,5cyclictest19555-21awk10:55:273
7127991811,5cyclictest17459-21cat10:50:193
712799181,12cyclictest4068-21apt-esm-hook08:35:143
712799181,12cyclictest30495-21apt-get11:30:153
712799181,12cyclictest30495-21apt-get11:30:153
712799181,12cyclictest28841-21apt-get11:25:153
712799181,12cyclictest25508-21apt-get11:15:143
712799181,12cyclictest24823-21apt-get08:00:183
712799181,12cyclictest19471-21apt-get07:45:193
712799181,12cyclictest12836-21apt-get12:15:103
712799181,12cyclictest11864-21apt-get10:35:183
712799181,12cyclictest11391-21apt-get12:10:163
7127991810,5cyclictest21824-21cat09:25:243
7127991810,3cyclictest31-21ksoftirqd/307:15:153
712799180,2cyclictest311ksoftirqd/309:40:113
7126991815,2cyclictest16681-21cat10:50:012
7126991814,2cyclictest19470-21apt07:45:162
7126991813,3cyclictest7370-21/usr/sbin/munin08:45:202
7126991813,3cyclictest27741-21kernelversion11:20:182
7126991813,3cyclictest22309-21/usr/sbin/munin09:30:402
7126991811,5cyclictest8418-21cat10:25:012
7126991811,5cyclictest8173-21awk12:00:192
7126991811,5cyclictest2521-21expr08:30:192
7126991811,5cyclictest21820-21cat09:25:252
712699181,14cyclictest0-21swapper/209:00:172
712599188,8cyclictest27701-21cat11:20:181
712599182,10cyclictest22104-21lsb_release09:30:001
7125991814,2cyclictest8016-21missed_timers08:45:201
7125991814,2cyclictest19266-21nfs4_client10:55:201
7125991814,2cyclictest14210-21missed_timers10:40:181
7125991814,2cyclictest10207-21apt10:30:141
7125991814,2cyclictest10081-21switchtime08:50:271
7125991812,4cyclictest30950-40dpkg09:55:101
7125991812,4cyclictest18528-21awk12:30:181
7125991812,4cyclictest11817-21ntp_kernel_pll_07:20:251
7125991812,3cyclictest0-21swapper/110:50:131
7125991811,5cyclictest9650-21cut12:05:011
7125991811,5cyclictest4027-21idleruntime10:10:251
7125991811,5cyclictest3479-21wc10:10:001
7125991811,5cyclictest25861-21df09:40:151
7125991811,5cyclictest23601-21sshd11:08:451
7125991811,5cyclictest11118-21expr08:55:151
712599181,13cyclictest12899-21snmp_o5powerctl10:35:281
7125991810,6cyclictest12587-21df_inode09:00:221
712499182,9cyclictest22089-21munin-run09:30:000
712499182,12cyclictest30944-40apt-check09:55:120
712499182,11cyclictest30507-21apt-esm-hook11:30:150
712499182,11cyclictest30507-21apt-esm-hook11:30:150
712499182,11cyclictest28841-21apt-get11:25:120
712499182,11cyclictest13553-21apt-esm-hook10:40:120
7124991813,3cyclictest11499-21sed07:20:210
7124991812,4cyclictest30667-21cat09:55:010
7124991812,4cyclictest22563-21sed07:50:330
7124991811,5cyclictest2317-21awk11:40:210
712499181,12cyclictest3559-21apt-get10:10:180
712499181,12cyclictest23964-21apt-get09:35:170
712499181,12cyclictest20525-40apt-check11:00:150
712499181,12cyclictest19502-21apt-esm-hook07:45:190
712499181,12cyclictest16787-21apt-get12:25:150
712499181,12cyclictest15203-21apt-esm-hook10:45:140
712499181,11cyclictest18449-21apt-esm-hook12:30:180
712499181,11cyclictest14022-21apt-esm-hook09:05:160
7124991810,4cyclictest13124-21sed07:25:320
7124991810,4cyclictest13124-21sed07:25:320
712799179,6cyclictest23659-21cat07:55:203
7127991711,4cyclictest28876-21sed09:50:003
7127991711,4cyclictest23686-21sed11:10:013
712799171,12cyclictest12389-21apt-esm-hook09:00:183
712799171,11cyclictest31477-21apt-esm-hook08:20:153
712799171,11cyclictest19006-21apt-esm-hook09:20:143
712699172,14cyclictest12472-21awk07:25:012
7126991713,3cyclictest5479-21munin-run08:40:002
7126991711,4cyclictest450-21munin-run08:25:002
7126991711,4cyclictest30922-21expr11:30:182
7126991711,4cyclictest30922-21expr11:30:182
712699171,14cyclictest0-21swapper/207:15:152
712699171,12cyclictest7442-21snmp_o5powerctl11:55:282
712699171,12cyclictest22236-21apt-esm-hook11:05:162
712699171,12cyclictest1936-21apt-esm-hook10:05:152
712699171,11cyclictest10696-21apt-esm-hook08:55:112
7126991710,5cyclictest19617-21cat09:20:182
7126991710,5cyclictest18655-21expr12:30:192
7125991713,2cyclictest28359-21sh08:10:261
7125991713,2cyclictest22855-21uptime07:50:341
7125991713,2cyclictest17854-21interrupts09:15:211
7125991711,4cyclictest30136-21cat09:50:281
712599171,12cyclictest31223-21sendmail08:20:001
712599171,12cyclictest23159-21apt-esm-hook07:55:161
712599171,12cyclictest17812-21apt-get07:40:161
7125991710,5cyclictest9940-21sshd10:26:471
7125991710,5cyclictest27817-21awk08:05:351
7125991710,5cyclictest21225-21cat12:35:281
712499179,6cyclictest4707-21sed08:35:230
712499172,11cyclictest29815-21apt-esm-hook08:15:160
7124991714,2cyclictest12903-21apt-config12:14:560
7124991712,3cyclictest0-21swapper/012:00:190
7124991711,4cyclictest24873-21tr11:10:250
712499171,12cyclictest24855-21apt-esm-hook08:00:170
712499171,12cyclictest20091-21apt-get12:35:140
712499171,12cyclictest18505-21apt-get10:55:140
712499171,11cyclictest7373-21apt-get08:45:140
712499171,11cyclictest31465-21apt-get08:20:140
712499171,11cyclictest17331-21apt-esm-hook09:15:170
712499171,11cyclictest16115-21apt-get07:35:200
712799163,7cyclictest8083-21apt-get12:00:193
7127991612,2cyclictest10681-21/usr/sbin/munin08:55:153
712799161,11cyclictest23995-21apt-esm-hook09:35:163
712699169,5cyclictest5069-21cut11:50:172
712599169,5cyclictest24218-21fw_packets11:10:161
712599167,7cyclictest2907-21cat08:30:251
7125991611,3cyclictest31295-21sh11:30:221
7125991611,3cyclictest31295-21sh11:30:221
712599161,11cyclictest15397-21snmp_o5powerctl07:30:341
7125991610,4cyclictest7029-21cat11:55:211
7125991610,4cyclictest24103-21cat09:35:181
712499162,10cyclictest25652-21apt-esm-hook09:40:140
7124991613,2cyclictest1904-21apt-get10:05:140
712499161,11cyclictest6902-21apt-esm-hook10:20:170
712499161,11cyclictest25539-21apt-esm-hook11:15:140
712499161,11cyclictest22344-21apt-esm-hook09:30:150
712499161,11cyclictest15139-21apt-esm-hook12:20:150
712499161,10cyclictest3091-21apt-get11:45:160
7124991610,4cyclictest18829-21basename07:40:280
712499160,11cyclictest4744-21apt-get11:50:150
712599157,6cyclictest15455-21tune2fs12:20:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional