You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-06-16 - 21:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Pentium(R) CPU N4200 @ 1.10GHz, Linux 5.9.1-rt18 (Profile)

Latency plot of shadow in rack #c, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackcslot4s.osadl.org (updated Thu Jun 10, 2021 12:44:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1936099348,16cyclictest13646-21ls11:35:230
1936099334,13cyclictest19354-21munin-run11:55:010
1936099333,10cyclictest28368-21unixbench_multi09:05:290
19362993213,17cyclictest31157-21apt-get12:30:122
1936099328,10cyclictest20392-21open_files11:55:230
1936099321,13cyclictest0-21swapper/011:05:060
1936099316,10cyclictest8289-21fschecks_time09:45:200
1936099315,9cyclictest17721-21munin-run11:50:010
1936099314,9cyclictest26215-21awk09:00:190
1936099314,9cyclictest13783-21sed08:25:190
1936099304,8cyclictest30010-21cat12:25:170
1936099303,9cyclictest3510-21latency07:55:180
1936099303,9cyclictest25741-21basename07:25:240
1936099303,12cyclictest23211-21tune2fs12:05:170
1936299292,7cyclictest26498-21apt-get10:40:172
1936099296,8cyclictest20972-21snmp_o5powerctl07:10:230
1936099294,9cyclictest28824-21sed10:45:180
1936099294,9cyclictest16867-21sed10:10:210
1936099294,9cyclictest10256-21sed09:50:260
1936099294,8cyclictest11010-21awk11:25:290
1936099294,8cyclictest11010-21awk11:25:290
1936099294,10cyclictest0-21swapper/010:30:010
1936099293,9cyclictest4802-21grep08:00:150
1936099293,9cyclictest31535-21sed09:15:240
1936099293,9cyclictest21533-21grep12:00:180
1936099293,10cyclictest5581-21sed11:10:270
1936099290,17cyclictest0-21swapper/008:15:080
1936299282,22cyclictest15163-21apt-get08:30:162
1936099284,10cyclictest8701-21snmp_o5powerctl11:20:280
1936099283,8cyclictest12228-21grep09:55:210
1936299273,21cyclictest7845-21apt-get08:10:172
19362992718,5cyclictest3004-21apt-get09:30:152
1936099275,9cyclictest25538-21mailstats12:10:210
1936099275,9cyclictest13548-21memory10:00:170
1936099275,8cyclictest28499-21memory12:20:280
1936099274,9cyclictest5089-21expr09:35:210
1936099273,9cyclictest22553-21sed08:50:150
1936099273,8cyclictest19529-21apt-get10:20:130
1936099270,16cyclictest0-21swapper/007:45:150
19364992614,7cyclictest13162-21apt-get08:25:143
19364992611,3cyclictest31-21ksoftirqd/311:55:193
19362992624,1cyclictest0-21swapper/208:59:052
1936099265,7cyclictest19058-21diskstats08:40:170
1936099264,8cyclictest30425-21memory10:50:220
1936099264,7cyclictest23792-21irqstats07:20:210
1936099263,9cyclictest0-21swapper/011:15:190
1936099263,8cyclictest31315-21cstates12:30:130
19364992511,12cyclictest14387-21apt11:40:003
1936299251,7cyclictest23265-21apt-get07:20:162
19362992517,5cyclictest14557-21apt-get10:05:132
1936299251,20cyclictest24472-21apt-get10:35:162
1936299250,22cyclictest22609-21munin-run10:30:002
1936099255,7cyclictest25259-21ntp_states10:35:280
1936099254,6cyclictest32224-21ntp_states10:55:230
1936099253,8cyclictest27414-21cat07:30:160
1936099253,8cyclictest14776-21cstates11:40:160
1936099253,8cyclictest11745-21diskstats08:20:170
1936099253,7cyclictest30775-21gltestperf07:40:200
1936099253,10cyclictest2453-21timerwakeupswit07:50:310
1936099252,9cyclictest22206-21sed07:15:250
1936099251,15cyclictest0-21swapper/008:45:220
1936099251,15cyclictest0-21swapper/008:30:270
1936099250,16cyclictest0-21swapper/009:25:280
19364992420,2cyclictest29383-21snmpd09:35:303
1936299242,4cyclictest17931-21apt-get11:50:142
19362992417,4cyclictest18830-21apt-get08:40:162
19361992412,8cyclictest24548-21apt-get12:10:131
1936099241,8cyclictest0-21swapper/008:05:290
1936099240,15cyclictest0-21swapper/009:10:250
1936099240,14cyclictest0-21swapper/009:30:170
19364992313,7cyclictest16207-21apt-get10:10:163
1936299232,4cyclictest413-21munin-run11:00:002
19362992317,4cyclictest16285-21apt-get11:45:132
19362992315,5cyclictest1232-21apt-get07:50:162
1936299231,19cyclictest30947-21lsb_release12:30:002
1936299231,18cyclictest0-21swapper/210:50:132
1936299222,17cyclictest6005-21apt-get11:15:142
19362992217,3cyclictest0-21swapper/209:35:152
19362992214,6cyclictest28448-21update-cracklib07:35:012
19362992214,5cyclictest9848-21apt-get08:15:172
1936299221,3cyclictest0-21swapper/208:25:152
19362992211,9cyclictest31867-21awk09:20:002
1936199222,15cyclictest31983-21apt-get07:45:151
19361992214,6cyclictest14510-21users08:25:301
19360992213,5cyclictest11323-21apt-get11:30:140
1936099220,8cyclictest0-21swapper/010:05:320
1936099220,14cyclictest0-21swapper/012:15:240
1936499218,2cyclictest6116-21tr11:15:143
19364992112,7cyclictest2645-21cron07:55:003
1936299219,3cyclictest25-21ksoftirqd/208:20:132
1936299214,3cyclictest111rcu_preempt09:25:212
19362992119,1cyclictest0-21swapper/211:29:352
19362992119,1cyclictest0-21swapper/211:29:342
19362992116,3cyclictest0-21swapper/209:00:142
1936299211,5cyclictest23040-21basename07:20:002
1936299211,4cyclictest19274-21latency_hist10:20:002
19362992113,4cyclictest11250-21apt-get09:55:122
1936299211,18cyclictest2161-21https11:05:002
1936299211,17cyclictest7628-21munin-run08:09:592
1936299211,17cyclictest17633-21perf10:15:002
1936299210,4cyclictest29575-21dpkg10:50:002
1936299210,1cyclictest0-21swapper/210:32:012
19361992113,5cyclictest4439-21perf09:35:001
1936099219,9cyclictest29696-21snmp_o5powerctl07:35:330
1936499209,7cyclictest31-21ksoftirqd/309:20:133
19364992012,5cyclictest72750irq/130-enp2s010:30:133
19364992012,5cyclictest29290-21uname12:24:593
19364992011,7cyclictest30008-21df_inode10:50:143
1936299204,2cyclictest111rcu_preempt07:45:232
1936299202,16cyclictest21926-21awk08:50:002
19362992013,5cyclictest25631-21expr07:25:242
19362992013,5cyclictest21169-21sed10:25:002
1936299201,2cyclictest111rcu_preempt08:35:222
1936299201,1cyclictest0-21swapper/209:52:352
1936299200,4cyclictest0-21swapper/207:55:132
1936199202,5cyclictest19-21ksoftirqd/109:06:301
19361992015,3cyclictest19285-21sendmail10:20:001
19361992013,5cyclictest30382-21apt09:15:111
19361992013,5cyclictest29694-21snmp_o5powerctl07:35:321
19361992013,5cyclictest1690-21sed07:50:211
19361992013,5cyclictest10672-21sed09:50:331
19361992012,6cyclictest32019-21awk12:30:201
19361992012,6cyclictest21172-21unixbench-2d07:10:261
19361992012,5cyclictest30985-21cat07:40:231
1936099209,7cyclictest22829-21apt-get10:30:140
1936099208,9cyclictest24374-21/usr/sbin/munin08:55:230
1936099202,5cyclictest0-21swapper/010:40:310
1936499198,7cyclictest31-21ksoftirqd/310:00:113
1936499198,2cyclictest31-21ksoftirqd/311:50:283
1936499192,3cyclictest111rcu_preempt08:20:123
1936499192,12cyclictest337-21apt-get12:35:163
1936499191,9cyclictest17312-21snmp_o5powerctl11:45:303
19364991913,4cyclictest12428-21threads11:30:273
19364991912,5cyclictest8648-21sed09:45:253
19364991912,5cyclictest32667-21sed07:45:223
19364991912,5cyclictest25074-21awk08:55:323
19364991912,5cyclictest20548-21sensors_temp10:20:303
19364991912,5cyclictest12892-21cat10:00:013
19364991911,6cyclictest19931-21tune2fs07:10:043
19364991911,6cyclictest10066-21idleruntime09:50:233
19364991911,5cyclictest27259-21perl07:30:153
1936499191,13cyclictest29383-21snmpd11:40:143
1936499191,13cyclictest23265-21apt-get07:20:163
1936499191,13cyclictest17848-21apt-get10:15:153
19364991910,7cyclictest26461-21grep09:00:233
1936299195,2cyclictest111rcu_preempt07:40:262
1936299193,14cyclictest633-17auditd11:34:592
1936299192,3cyclictest111rcu_preempt12:10:152
1936299192,3cyclictest111rcu_preempt11:10:172
1936299192,12cyclictest32111-21apt-esm-hook09:20:132
19362991913,4cyclictest22902-21apt-get12:05:122
19362991912,5cyclictest7921-21wc09:45:002
19362991912,5cyclictest28951-21cut07:35:232
19362991912,5cyclictest25-21ksoftirqd/211:55:162
19362991912,5cyclictest13343-21grep11:35:212
1936299190,17cyclictest25-21ksoftirqd/208:55:002
1936299190,17cyclictest0-21swapper/209:10:152
1936299190,16cyclictest2716-21apt-get11:05:162
1936199192,14cyclictest8280-21grep08:10:201
1936199192,12cyclictest6207-21apt-esm-hook08:05:161
1936199192,12cyclictest22902-21apt-get12:05:121
19361991915,2cyclictest9694-21apt-esm-hook11:25:161
19361991915,2cyclictest9694-21apt-esm-hook11:25:161
19361991914,3cyclictest21277-21sh12:00:161
19361991912,5cyclictest9179-21date11:25:001
19361991912,5cyclictest29492-21nfs4_client09:10:201
19361991912,5cyclictest26099-21sed09:00:191
19361991912,5cyclictest1527-21expr09:25:151
19361991911,6cyclictest8366-21cut09:45:201
19361991911,6cyclictest4870-21df09:35:171
19361991911,6cyclictest12448-21grep09:55:231
19361991911,5cyclictest24710-21http07:25:001
19361991911,5cyclictest2159-21basename11:05:001
1936199191,13cyclictest6316-21apt-get09:40:151
1936099198,8cyclictest32731-21load09:20:270
1936099192,5cyclictest0-21swapper/011:00:220
1936099191,7cyclictest9-21ksoftirqd/008:10:170
19360991914,3cyclictest17876-21sh10:15:160
1936499188,2cyclictest31-21ksoftirqd/312:25:133
1936499182,3cyclictest111rcu_preempt09:30:153
1936499181,5cyclictest0-21swapper/310:38:463
19364991814,2cyclictest9776-21cpuspeed_turbos11:25:173
19364991814,2cyclictest9776-21cpuspeed_turbos11:25:173
19364991812,4cyclictest32395-21which10:55:253
19364991812,4cyclictest27677-21sed09:05:233
19364991812,4cyclictest21790-21ls07:15:153
19364991811,5cyclictest8895-21cat08:10:283
19364991811,5cyclictest7815-21cat11:20:173
19364991811,5cyclictest6572-21expr08:05:213
19364991811,5cyclictest30278-21date07:40:003
19364991811,5cyclictest28600-21grep10:45:163
19364991811,5cyclictest21266-21ntpq08:45:203
19364991811,5cyclictest14806-21date08:30:003
1936499181,13cyclictest25265-21apt-get07:25:193
1936499181,12cyclictest31157-21apt-get12:30:123
1936499181,12cyclictest2700-40apt-check11:05:093
1936499180,6cyclictest31-21ksoftirqd/310:05:303
1936299183,2cyclictest111rcu_preempt07:14:302
1936299182,2cyclictest111rcu_preempt12:35:272
1936299182,2cyclictest111rcu_preempt09:45:252
1936299181,5cyclictest1249-1kworker/2:2H09:05:162
1936299181,3cyclictest25-21ksoftirqd/211:41:302
1936299181,12cyclictest5575-21snmp_o5powerctl08:00:222
1936299181,12cyclictest26203-21apt-esm-hook12:15:132
1936299180,5cyclictest0-21swapper/211:20:252
1936299180,5cyclictest0-21swapper/210:00:192
1936199189,7cyclictest22097-21cat07:15:241
1936199182,11cyclictest9859-21apt-esm-hook08:15:171
19361991814,2cyclictest11736-21sh08:20:161
19361991813,3cyclictest26505-21sh10:40:191
19361991811,5cyclictest7190-21sed11:15:251
19361991811,5cyclictest4202-21date11:10:001
19361991811,5cyclictest18062-21cat11:50:161
19361991811,5cyclictest17551-21expr08:35:241
19361991811,5cyclictest14539-21sed11:40:011
19361991811,4cyclictest5703-21grep11:10:271
1936199181,12cyclictest27864-21apt-get12:20:121
1936199181,12cyclictest14577-21apt-get08:30:111
1936199181,12cyclictest12915-21apt-get10:00:111
19361991810,6cyclictest4904-21fw_packets08:00:151
19360991812,3cyclictest6316-21apt-get09:40:160
19360991811,5cyclictest16918-21dpkg08:35:010
19360991810,4cyclictest337-21apt-get12:35:160
1936499177,2cyclictest31-21ksoftirqd/309:40:213
1936499172,3cyclictest111rcu_preempt08:50:163
1936499172,2cyclictest111rcu_preempt07:55:163
19364991711,4cyclictest9555-21mktemp08:15:003
19364991711,4cyclictest30551-21expr07:40:173
19364991711,4cyclictest24690-21cat12:10:133
1936499171,11cyclictest21255-21apt-get12:00:153
19364991710,5cyclictest5777-21unixbench-2d08:00:253
19364991710,5cyclictest21555-21expr10:25:183
19364991710,5cyclictest1832-21basename09:25:183
19364991710,5cyclictest17389-21diskstats08:35:223
1936499170,2cyclictest111rcu_preempt12:05:283
1936499170,2cyclictest111rcu_preempt09:15:233
19362991713,2cyclictest21252-21/usr/sbin/munin12:00:132
1936199179,6cyclictest32007-21grep10:55:221
1936199179,6cyclictest17779-21ls11:50:011
1936199179,6cyclictest17500-21grep10:10:271
1936199172,11cyclictest32100-21apt-get09:20:131
19361991713,2cyclictest28237-21cpuspeed_turbos10:45:151
19361991713,2cyclictest27089-21processes12:15:211
19361991712,3cyclictest20933-21hwlatdetect08:45:171
1936199171,12cyclictest29524-21apt-esm-hook12:25:141
1936199171,12cyclictest2900-21apt-esm-hook07:55:131
1936199171,12cyclictest14639-21apt-get11:40:151
1936199171,12cyclictest11323-21apt-get11:30:141
1936199171,11cyclictest22840-21apt-esm-hook10:30:131
1936199171,11cyclictest22147-21apt-esm-hook08:50:131
19361991710,5cyclictest30480-21cat10:50:231
19361991710,5cyclictest25750-21if_enp2s007:25:241
19361991710,5cyclictest22717-21cat10:30:001
19361991710,5cyclictest20527-21readlink10:20:301
19361991710,5cyclictest15030-21basename10:05:231
19361991710,4cyclictest25183-21grep10:35:261
19364991612,2cyclictest26189-21/usr/sbin/munin12:15:173
19364991611,3cyclictest26320-21sed10:40:013
1936499161,11cyclictest667-21apt-get11:00:163
1936499161,11cyclictest28738-21apt-get09:10:153
1936499161,11cyclictest18830-21apt-get08:40:163
1936299160,3cyclictest111rcu_preempt12:20:242
19361991612,2cyclictest1831-21dump-pmu-power12:40:001
19361991611,3cyclictest19963-21cat08:40:281
1936199161,11cyclictest26932-21apt-get07:30:131
1936199161,11cyclictest23779-21apt-get08:55:141
1936199161,11cyclictest19582-21apt-get11:55:161
19364991510,3cyclictest4491-21expr11:10:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional