You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-08-01 - 20:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Atom(TM) Processor E3940 @ 1.60GHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackaslot8s (updated Sun Aug 01, 2021 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
279722175101,27sleep10-21swapper/107:05:351
282402173101,28sleep00-21swapper/007:09:000
281082163103,28sleep30-21swapper/307:07:193
280842159107,29sleep20-21swapper/207:07:022
2895221430,2sleep30-21swapper/310:08:533
1948621380,1sleep10-21swapper/110:01:401
1364321370,1sleep10-21swapper/111:29:031
211652640,2sleep20-21swapper/209:39:282
211652640,2sleep20-21swapper/209:39:282
232642550,1sleep30-21swapper/311:12:523
307892540,1sleep10-21swapper/107:15:141
84372530,1sleep30-21swapper/309:54:413
55632490,1sleep30-21swapper/311:45:303
149602490,1sleep10-21swapper/109:05:251
151882480,2sleep00-21swapper/011:06:550
307912470,2sleep30-21swapper/311:40:283
103732460,2sleep10-21swapper/108:55:221
57252440,2sleep30-21swapper/309:26:253
122052440,1sleep20-21swapper/211:05:172
321562430,2chrt0-21swapper/109:21:501
280992430,1sleep10-21swapper/109:19:211
252992410,1sleep00-21swapper/010:05:290
166832400,1sleep00-21swapper/012:38:580
242572370,3chrt0-21swapper/211:35:342
298992350,1sleep00-21swapper/009:46:000
88842330,2sleep00-21swapper/010:17:090
2859099329,11cyclictest0-21swapper/010:30:010
2859299302,21cyclictest1295-21ssh10:35:142
2859099303,12cyclictest0-21swapper/008:15:000
62162290,5sleep228-21ksoftirqd/211:00:272
33072290,2sleep00-21swapper/009:25:220
12672290,2chrt0-21swapper/009:23:330
203752280,3sleep225212-1kworker/u9:211:55:312
323732270,3sleep122-21ksoftirqd/110:55:571
2859199279,15cyclictest26548-21fschecks_count11:15:241
2859199272,6cyclictest5365-21apt-get10:15:201
28591992618,6cyclictest29262-21date11:40:001
294072250,2sleep10-21swapper/110:09:201
2859299252,16cyclictest4319-21ssh11:45:152
28592992511,9cyclictest22041-21ssh09:40:152
28591992516,6cyclictest16787-21awk11:30:331
28590992510,13cyclictest29426-21/usr/sbin/munin10:55:000
203352250,2chrt0-21swapper/108:05:001
2859199241,8cyclictest141rcu_preempt11:53:121
28590992414,7cyclictest921-21df_inode07:20:180
28590992414,7cyclictest18905-21apt-get11:10:150
28590992414,6cyclictest7093-21apt-get07:35:150
2859399231,4cyclictest0-21swapper/311:58:403
28593992311,9cyclictest577-21apt-get07:20:153
28592992312,7cyclictest8972-21wc09:55:012
2859199231,9cyclictest291182which07:10:221
2859099231,9cyclictest103872sh11:04:050
28590992313,6cyclictest12249-21cut11:50:250
2859099230,12cyclictest28526-21http12:25:000
150522230,2sleep2271rcuc/212:37:282
2859399221,9cyclictest146012sh11:51:483
28593992211,8cyclictest14099-21apt-get09:05:153
28592992215,4cyclictest7328-21sh08:50:002
28592992212,6cyclictest5016-21grep12:30:292
28592992210,4cyclictest141rcu_preempt10:44:522
2859199228,5cyclictest221-21systemd-journal09:10:201
2859199224,3cyclictest16009-21diskmemload10:12:461
2859199224,3cyclictest141rcu_preempt09:35:031
2859199224,3cyclictest141rcu_preempt09:35:031
2859199222,18cyclictest8573-21cut10:40:151
2859199222,17cyclictest2921350ssh12:25:161
28591992215,5cyclictest4843-21latency_hist07:30:001
28591992214,6cyclictest4027-21sed08:40:241
2859199221,10cyclictest18660-21apt-get11:55:171
2859099221,8cyclictest0-21swapper/010:45:010
2859099221,8cyclictest0-21swapper/010:45:010
2859099221,7cyclictest25825-21apt-get12:00:170
28590992214,4cyclictest8933-21sendmail_mailst12:10:270
28590992211,8cyclictest9733-21df07:40:170
22422220,1sleep10-21swapper/110:35:231
213792220,1sleep00-21swapper/008:05:270
143422220,2sleep00-21swapper/011:29:420
312792210,1sleep30-21swapper/310:33:123
2859399219,9cyclictest28343-21sh11:39:193
28593992111,6cyclictest55350irq/141-enp3s0-09:20:283
28593992110,7cyclictest11833-21apt-get11:05:163
2859299219,9cyclictest30224-21sh10:32:132
2859299217,6cyclictest28-21ksoftirqd/207:50:212
2859299217,12cyclictest15870-21rm09:59:592
2859299213,12cyclictest20889-21kworker/u8:011:15:182
2859299211,8cyclictest28068-21kworker/u8:211:45:002
28592992116,3cyclictest28870-21latency_hist08:25:002
28592992110,9cyclictest3234-21sh10:59:012
2859199214,3cyclictest16009-21diskmemload12:15:561
2859199214,14cyclictest32733-21sed08:30:381
2859199213,3cyclictest16009-21diskmemload12:35:411
2859199213,10cyclictest2244950ssh12:20:211
2859199212,3cyclictest16009-21diskmemload12:34:511
2859199212,3cyclictest16009-21diskmemload11:20:551
2859199211,7cyclictest16009-21diskmemload09:55:081
2859199211,5cyclictest0-21swapper/109:29:021
2859199211,4cyclictest0-21swapper/109:40:201
28591992113,6cyclictest21019-21grep10:25:321
28591992111,6cyclictest55250irq/140-enp3s0-10:20:181
2859099216,13cyclictest13674-21munin-run07:50:000
2859099213,3cyclictest141rcu_preempt10:31:250
28590992113,6cyclictest32645-21users10:10:430
28590992113,6cyclictest15328-21ls07:50:340
143652210,3sleep228588-21cyclictest10:20:362
135232210,2sleep10-21swapper/112:14:201
77912200,8sleep122-21ksoftirqd/107:35:241
2859399208,3cyclictest141rcu_preempt12:10:003
2859399202,7cyclictest13966-21python310:20:333
2859399201,8cyclictest16059-21latency_hist10:00:013
28593992014,4cyclictest32722-21dump-pmu-power07:20:003
28593992014,4cyclictest31153-21sh11:18:363
28593992014,4cyclictest20017-21sh09:38:093
28593992014,4cyclictest20017-21sh09:38:093
28593992013,5cyclictest7099-21http07:35:163
28593992012,6cyclictest30086-21ntpq08:25:263
28593992012,6cyclictest25069-21ssh09:16:013
28593992012,6cyclictest24884-21cut08:15:163
28593992012,6cyclictest11745-21cat07:45:183
28593992010,7cyclictest6354-21memory10:15:293
28593992010,7cyclictest32486-21sensors_temp08:30:363
2859299208,9cyclictest55450irq/142-enp3s0-09:29:422
2859299202,12cyclictest30719-21apt-get10:10:192
28592992014,4cyclictest23818-21sh10:27:522
28592992013,4cyclictest29941-21sh10:09:492
28592992012,6cyclictest8562-21sed12:10:232
28592992012,6cyclictest3332-21awk07:25:232
28592992012,5cyclictest55350irq/141-enp3s0-12:25:162
28592992011,7cyclictest826-21wc08:35:002
28592992011,6cyclictest55450irq/142-enp3s0-10:15:182
28592992011,6cyclictest10520-21grep09:30:312
2859199204,3cyclictest16009-21diskmemload12:06:111
2859199204,3cyclictest16009-21diskmemload10:50:311
2859199203,3cyclictest16009-21diskmemload11:45:431
2859199203,3cyclictest16009-21diskmemload09:31:011
2859199203,2cyclictest141rcu_preempt11:05:051
2859199203,12cyclictest29662-21ssh11:40:171
2859199202,3cyclictest16009-21diskmemload10:34:121
2859199201,3cyclictest141rcu_preempt07:50:211
28591992010,8cyclictest32666-21sh09:48:411
2859099203,3cyclictest16009-21diskmemload10:42:460
2859099202,2cyclictest141rcu_preempt11:30:370
2859099202,12cyclictest9226-21apt-get09:55:120
2859099201,5cyclictest0-21swapper/009:10:240
2859099201,3cyclictest141rcu_preempt12:30:170
2859099201,2cyclictest0-21swapper/012:25:220
28590992012,6cyclictest5853-21perl08:45:170
28590992012,6cyclictest19876-21seq08:01:140
28590992012,6cyclictest17645-21users07:55:510
258232201,1sleep10-21swapper/112:00:171
290382191,2sleep329046-21ssh12:01:513
2859399198,8cyclictest14721-21ssh11:30:013
2859399198,7cyclictest20754-21pmu-power10:25:303
28593991914,3cyclictest8694-21sh12:33:223
28593991914,3cyclictest4180-21sh11:22:003
28593991913,4cyclictest3737-21sh10:59:293
28593991913,4cyclictest1874-21sh12:28:383
28593991913,4cyclictest18720-21sh11:31:593
28593991913,4cyclictest12776-21sh09:32:063
28593991912,5cyclictest27516-21cut09:45:013
28593991912,4cyclictest16009-21diskmemload10:45:013
28593991912,4cyclictest16009-21diskmemload10:45:013
28593991911,6cyclictest28812-21ls07:10:183
28593991911,6cyclictest28156-21df09:45:163
28593991911,6cyclictest21355-21ntpq08:05:263
28593991911,6cyclictest19378-21seq12:18:083
28593991911,6cyclictest13086-21mailstats09:00:283
28593991911,5cyclictest55250irq/140-enp3s0-10:40:213
28593991910,7cyclictest7842-21expr12:10:173
28593991910,7cyclictest2737-21/usr/sbin/munin07:25:293
28593991910,7cyclictest1181-21sh10:35:093
28593991910,7cyclictest11063-21sh11:04:443
28593991910,5cyclictest55350irq/141-enp3s0-10:00:133
2859299197,4cyclictest141rcu_preempt11:11:282
2859299192,13cyclictest22598-21apt-get08:10:162
2859299192,12cyclictest15125-21ssh11:30:042
2859299192,12cyclictest14732-21ssh12:15:142
28592991913,4cyclictest22302-21sh10:04:192
28592991913,4cyclictest14350irq/133-ahci[0008:45:002
28592991913,4cyclictest14350irq/133-ahci[0008:00:012
28592991912,5cyclictest8604-21tune2fs11:25:222
28592991912,5cyclictest30861-21uptime09:20:292
28592991912,5cyclictest29100-21perl07:10:202
2859299191,15cyclictest221-21systemd-journal12:04:552
28592991910,6cyclictest685-21snmpd10:51:392
2859199195,11cyclictest221-21systemd-journal08:05:071
2859199193,3cyclictest16009-21diskmemload11:10:401
2859199191,5cyclictest0-21swapper/111:01:121
28591991912,5cyclictest17121-21mailstats10:45:281
28591991912,5cyclictest17121-21mailstats10:45:281
28591991912,5cyclictest11636-21mktemp07:45:011
2859099199,7cyclictest22843-21apt-get09:15:150
2859099194,3cyclictest16009-21diskmemload10:04:200
2859099193,4cyclictest16009-21diskmemload10:20:170
2859099193,4cyclictest12-21ksoftirqd/009:32:220
2859099193,3cyclictest16009-21diskmemload11:57:160
2859099193,3cyclictest16009-21diskmemload09:53:240
2859099193,3cyclictest16009-21diskmemload09:40:390
2859099192,2cyclictest141rcu_preempt12:05:190
2859099192,2cyclictest141rcu_preempt11:36:460
2859099192,2cyclictest141rcu_preempt10:35:400
28590991913,4cyclictest7057-21sh11:46:230
28590991913,4cyclictest15490-21sh12:15:210
28590991912,5cyclictest28894-21cat07:10:190
28590991912,4cyclictest16009-21diskmemload09:35:080
28590991912,4cyclictest16009-21diskmemload09:35:080
28590991911,6cyclictest4817-21ls07:30:000
183612190,3sleep318359-21ssh09:10:403
2859399187,7cyclictest5413-21apt-get08:45:123
2859399181,8cyclictest22725-21awk10:50:093
2859399181,8cyclictest2069-21ssh10:12:263
28593991813,3cyclictest13184-21sh12:35:433
28593991811,5cyclictest6198-21cat07:30:373
28593991811,5cyclictest27031-21cat08:20:213
28593991811,5cyclictest18944-21cat08:00:243
28593991811,5cyclictest14732-21cat07:50:263
28593991810,6cyclictest9622-21expr07:40:173
28593991810,6cyclictest23652-21which08:10:303
28593991810,6cyclictest22162-21cat12:20:193
2859299182,11cyclictest21906-21apt-get12:20:142
2859299182,11cyclictest15500-21apt-get10:45:032
2859299182,11cyclictest15500-21apt-get10:45:032
2859299182,10cyclictest497-21irqbalance07:15:162
28592991813,3cyclictest30357-21switchtime08:25:292
28592991812,4cyclictest29775-21sh09:45:522
28592991812,4cyclictest2937-21sh12:06:052
28592991812,4cyclictest2664-21sh11:20:362
28592991811,5cyclictest8043-21ntp_states07:35:272
28592991811,5cyclictest2650-21seq08:36:002
28592991811,4cyclictest16009-21diskmemload09:10:312
2859199188,2cyclictest22-21ksoftirqd/109:50:531
28591991811,5cyclictest28153-21grep08:20:431
28591991811,5cyclictest1843-21date08:35:241
28591991810,6cyclictest5413-21apt-get08:45:141
2859099189,5cyclictest0-21swapper/010:55:030
28590991813,3cyclictest9723-21acpi08:55:140
28590991813,3cyclictest4046-21missed_timers08:40:240
28590991811,5cyclictest31370-21unixbench_singl11:40:320
28590991811,4cyclictest25859-21sh11:15:070
28590991810,6cyclictest1858-21memory08:35:250
28590991810,6cyclictest15632-21users09:05:320
28590991810,5cyclictest877-21cat11:20:170
2859399179,6cyclictest16305-21cat07:55:333
28593991711,4cyclictest9805-21ls08:55:183
28593991710,5cyclictest7713-21grep08:50:333
28593991710,5cyclictest1243-21seq08:35:183
2859299179,6cyclictest18651-21expr08:00:202
2859299179,6cyclictest15020-21ntpq09:05:262
2859299179,6cyclictest10529-21ls08:55:242
2859299177,8cyclictest9407-21apt-get07:40:162
28592991712,3cyclictest27245-21sh09:18:252
28592991712,3cyclictest13867-21sh11:51:082
2859299171,12cyclictest20434-21apt-get08:05:142
28592991710,5cyclictest2597-21date07:24:592
2859199178,7cyclictest4937-21apt-get07:30:251
2859199172,10cyclictest577-21apt-get07:20:171
28591991710,5cyclictest29546-21cut08:25:181
2859099179,6cyclictest12000-21cat09:00:170
2859099171,2cyclictest141rcu_preempt08:15:320
28590991710,5cyclictest8255-21cat08:50:380
28590991710,5cyclictest5442-21idleruntime07:30:290
28590991710,5cyclictest32693-21cat08:30:380
28590991710,5cyclictest30980-21cat07:15:180
28590991710,5cyclictest29348-21expr08:25:160
28590991710,5cyclictest26971-21grep08:20:200
2859399169,5cyclictest4613-21cat08:40:303
2859299168,5cyclictest13805-21latency_hist07:50:012
28592991612,2cyclictest4911-21/usr/sbin/munin07:30:342
28592991611,3cyclictest7571-21acpi08:50:252
2859199162,10cyclictest16101-21apt-get07:55:291
2859199161,6cyclictest221-21systemd-journal09:00:181
28591991611,3cyclictest25259-21hwlatdetect08:15:211
2859299151,10cyclictest24763-21apt-get08:15:162
28592991510,3cyclictest13208-21timerandwakeup09:00:302
2859199157,6cyclictest23024-21tune2fs08:10:191
2859199155,8cyclictest9406-21apt07:40:161
2859199151,10cyclictest7601-21apt-get08:50:331
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional