You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-17 - 14:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Atom(TM) Processor E3940 @ 1.60GHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Sun Oct 17, 2021 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
79002183169,8sleep00-21swapper/019:05:530
79382171104,28sleep30-21swapper/319:06:253
80582162105,28sleep20-21swapper/219:07:552
79362160103,29sleep10-21swapper/119:06:231
2208621470,2sleep10-21swapper/120:55:251
776021460,2sleep30-21swapper/323:39:393
745421450,2sleep00-21swapper/000:34:470
323452660,2sleep30-21swapper/322:36:243
205702590,3sleep20-21swapper/223:50:212
234242540,9sleep228-21ksoftirqd/222:00:532
268722460,2sleep30-21swapper/319:50:283
288002440,1sleep00-21swapper/022:34:120
232072440,2sleep227647-21diskmemload00:20:182
110552410,1sleep011057-21expr19:15:180
8490994017,7cyclictest12-21ksoftirqd/000:00:170
130502370,2sleep012-21ksoftirqd/023:43:570
107432370,2chrt0-21swapper/200:36:422
107432370,2chrt0-21swapper/200:36:422
8490993612,12cyclictest8197-21munin-run20:25:000
293982360,8sleep334-21ksoftirqd/322:34:553
8490993416,7cyclictest12-21ksoftirqd/022:45:560
8490993414,3cyclictest141rcu_preempt00:16:070
8493993211,3cyclictest34-21ksoftirqd/321:59:013
849299323,10cyclictest6915-21latency_hist21:19:592
849099327,12cyclictest29880-21munin-run19:59:590
849099323,14cyclictest2936-21perf22:40:000
36482320,2chrt0-21swapper/022:12:050
155912320,9sleep228-21ksoftirqd/200:13:472
849399319,11cyclictest34-21ksoftirqd/322:15:223
849399318,10cyclictest34-21ksoftirqd/300:25:203
849099315,10cyclictest12-21ksoftirqd/021:32:220
849099315,10cyclictest12-21ksoftirqd/021:32:210
8490993112,5cyclictest12-21ksoftirqd/021:11:330
8490993112,10cyclictest12-21ksoftirqd/023:50:240
8490993111,10cyclictest12-21ksoftirqd/000:07:350
849399308,10cyclictest34-21ksoftirqd/322:28:343
849399308,10cyclictest34-21ksoftirqd/321:05:213
8493993010,13cyclictest34-21ksoftirqd/321:31:163
8493993010,13cyclictest34-21ksoftirqd/321:31:163
849299306,9cyclictest28-21ksoftirqd/221:20:242
849299302,6cyclictest28-21ksoftirqd/221:50:312
8490993013,3cyclictest141rcu_preempt22:08:230
8490993013,3cyclictest141rcu_preempt22:08:230
8490993012,4cyclictest12-21ksoftirqd/021:50:040
849099301,14cyclictest26803-21munin-run22:05:000
8490993011,10cyclictest12-21ksoftirqd/022:52:040
849399297,10cyclictest34-21ksoftirqd/323:04:593
849399290,6cyclictest27647-21diskmemload22:50:013
849299296,7cyclictest28-21ksoftirqd/222:25:202
849299293,7cyclictest28-21ksoftirqd/223:40:342
849299293,7cyclictest28-21ksoftirqd/222:31:442
849299291,7cyclictest28-21ksoftirqd/200:15:442
849099299,10cyclictest12-21ksoftirqd/000:11:230
849099296,11cyclictest12-21ksoftirqd/023:23:200
849099296,11cyclictest12-21ksoftirqd/023:23:190
8490992912,3cyclictest141rcu_preempt00:29:170
8490992911,10cyclictest12-21ksoftirqd/022:44:440
8490992910,8cyclictest12-21ksoftirqd/023:15:320
320352290,2sleep10-21swapper/100:27:301
849399289,10cyclictest34-21ksoftirqd/300:21:363
8493992812,5cyclictest34-21ksoftirqd/321:18:523
8493992810,9cyclictest34-21ksoftirqd/321:36:163
8492992810,3cyclictest28-21ksoftirqd/220:45:302
849099289,9cyclictest12-21ksoftirqd/023:45:030
849099289,6cyclictest12-21ksoftirqd/023:00:360
849099287,10cyclictest12-21ksoftirqd/021:29:020
8490992811,4cyclictest12-21ksoftirqd/022:22:000
220532280,4sleep00-21swapper/022:28:030
849399279,9cyclictest34-21ksoftirqd/300:17:283
849399278,11cyclictest34-21ksoftirqd/300:35:003
849399278,10cyclictest34-21ksoftirqd/322:02:153
849399275,11cyclictest34-21ksoftirqd/323:05:583
8493992718,3cyclictest34-21ksoftirqd/323:49:413
8493992715,2cyclictest27647-21diskmemload23:20:293
8493992715,2cyclictest27647-21diskmemload23:20:283
849299275,8cyclictest28-21ksoftirqd/220:50:292
849299275,7cyclictest28-21ksoftirqd/223:20:302
849299275,7cyclictest28-21ksoftirqd/223:20:302
849299274,6cyclictest28-21ksoftirqd/221:43:262
849299273,9cyclictest28-21ksoftirqd/221:31:162
849299273,9cyclictest28-21ksoftirqd/221:31:152
849299273,8cyclictest28-21ksoftirqd/221:05:222
849299273,7cyclictest28-21ksoftirqd/223:25:292
849299273,7cyclictest28-21ksoftirqd/222:37:222
849299273,5cyclictest28-21ksoftirqd/222:20:232
849299272,8cyclictest28-21ksoftirqd/222:40:352
849299272,6cyclictest28-21ksoftirqd/223:05:192
849299272,6cyclictest28-21ksoftirqd/200:30:142
849299271,8cyclictest30251-21taskset22:07:072
849299271,8cyclictest30251-21taskset22:07:062
849299271,7cyclictest28-21ksoftirqd/222:15:452
849099279,5cyclictest12-21ksoftirqd/000:38:070
849099279,5cyclictest12-21ksoftirqd/000:38:060
849099278,9cyclictest141rcu_preempt23:10:140
849099276,13cyclictest12-21ksoftirqd/023:37:410
849099273,11cyclictest12-21ksoftirqd/019:20:310
8490992714,8cyclictest30568-21apt-get21:40:160
8490992713,4cyclictest12-21ksoftirqd/023:25:230
8490992712,4cyclictest12-21ksoftirqd/021:59:280
8490992711,4cyclictest12-21ksoftirqd/022:55:080
8490992710,9cyclictest12-21ksoftirqd/023:34:380
8490992710,5cyclictest12-21ksoftirqd/021:16:190
8490992710,5cyclictest12-21ksoftirqd/000:23:310
849399269,4cyclictest34-21ksoftirqd/300:11:073
849399268,9cyclictest34-21ksoftirqd/323:12:103
849399268,8cyclictest34-21ksoftirqd/323:57:343
849399267,11cyclictest34-21ksoftirqd/323:40:333
849399266,10cyclictest34-21ksoftirqd/323:15:363
849399265,9cyclictest34-21ksoftirqd/321:20:213
8493992611,4cyclictest34-21ksoftirqd/323:30:113
8493992611,4cyclictest34-21ksoftirqd/322:10:083
8493992610,5cyclictest34-21ksoftirqd/320:24:593
849399260,7cyclictest34-21ksoftirqd/323:50:293
849399260,16cyclictest34-21ksoftirqd/300:00:163
849299269,3cyclictest141rcu_preempt22:46:222
849299265,6cyclictest28-21ksoftirqd/221:13:302
849299264,8cyclictest28-21ksoftirqd/221:29:542
849299264,6cyclictest28-21ksoftirqd/223:02:032
849299264,6cyclictest0-21swapper/200:05:242
849299263,8cyclictest28-21ksoftirqd/219:45:322
849299263,7cyclictest28-21ksoftirqd/221:59:502
849299262,7cyclictest28-21ksoftirqd/223:35:302
849299261,9cyclictest28-21ksoftirqd/220:30:292
849299261,7cyclictest28-21ksoftirqd/223:18:282
849299261,6cyclictest10257-21chrt19:13:092
849299261,5cyclictest232202sleep219:44:262
849299261,5cyclictest231452sleep220:59:502
849299260,9cyclictest31386-21chrt23:30:592
849099269,5cyclictest12-21ksoftirqd/022:17:530
849099267,17cyclictest3924-21perf20:15:000
8490992613,5cyclictest12-21ksoftirqd/021:35:220
8490992612,5cyclictest12-21ksoftirqd/023:06:040
8490992612,3cyclictest141rcu_preempt23:59:390
849399259,4cyclictest34-21ksoftirqd/323:25:063
849399259,3cyclictest141rcu_preempt21:40:393
849399258,9cyclictest34-21ksoftirqd/322:07:073
849399258,9cyclictest34-21ksoftirqd/322:07:073
849399258,9cyclictest34-21ksoftirqd/321:14:503
849399258,4cyclictest34-21ksoftirqd/300:07:263
849399257,9cyclictest34-21ksoftirqd/321:25:393
849399257,11cyclictest34-21ksoftirqd/321:50:303
849399257,10cyclictest34-21ksoftirqd/320:45:233
849399255,9cyclictest34-21ksoftirqd/322:52:013
8493992510,3cyclictest141rcu_preempt22:21:313
849399250,6cyclictest3351-21apt-get22:40:153
849399250,10cyclictest9742-21rm00:35:283
849399250,10cyclictest9742-21rm00:35:273
849299254,7cyclictest28-21ksoftirqd/221:35:212
849299254,6cyclictest28-21ksoftirqd/222:55:582
849299253,7cyclictest28-21ksoftirqd/220:05:342
849299253,6cyclictest28-21ksoftirqd/200:27:162
849299253,6cyclictest28-21ksoftirqd/200:02:102
849299253,5cyclictest28-21ksoftirqd/222:50:142
849299252,7cyclictest28-21ksoftirqd/221:46:272
849299252,6cyclictest28-21ksoftirqd/223:56:382
849299252,3cyclictest28-21ksoftirqd/220:10:202
849299251,7cyclictest28-21ksoftirqd/223:49:032
849299251,6cyclictest59212sleep220:18:392
849299251,5cyclictest144192sleep219:21:222
849299250,6cyclictest7247-21chrt23:10:332
849299250,6cyclictest236012sleep221:00:112
8490992511,3cyclictest12-21ksoftirqd/021:45:360
8490992510,3cyclictest141rcu_preempt21:20:140
849099250,3cyclictest141rcu_preempt19:35:000
18792250,3chrt621-21/usr/sbin/munin22:10:312
139062250,2sleep00-21swapper/020:35:300
849399247,8cyclictest34-21ksoftirqd/321:46:553
849399246,3cyclictest141rcu_preempt19:22:403
849399240,8cyclictest20888-21sh22:55:173
849399240,7cyclictest34-21ksoftirqd/319:15:193
849299241,7cyclictest28-21ksoftirqd/219:55:142
849299240,5cyclictest188192sleep219:32:572
849299240,5cyclictest100762sleep220:26:502
8490992418,4cyclictest12-21ksoftirqd/019:40:000
8490992418,4cyclictest12-21ksoftirqd/019:40:000
280082240,2sleep30-21swapper/319:55:173
849399239,4cyclictest34-21ksoftirqd/320:30:323
849399238,4cyclictest34-21ksoftirqd/320:07:423
849399237,4cyclictest34-21ksoftirqd/320:10:483
849399236,5cyclictest141rcu_preempt19:44:563
849399236,11cyclictest34-21ksoftirqd/320:41:203
849399233,13cyclictest34-21ksoftirqd/320:03:353
849299237,2cyclictest141rcu_preempt20:20:282
849299231,5cyclictest0-21swapper/219:25:432
849299230,7cyclictest28-21ksoftirqd/219:50:062
8491992316,4cyclictest9636-21grep00:35:271
8491992316,4cyclictest9636-21grep00:35:271
8491992315,6cyclictest17265-21cstates19:30:191
8491992314,7cyclictest30588-21perl20:00:211
849099239,4cyclictest12-21ksoftirqd/019:40:120
849099239,3cyclictest141rcu_preempt20:17:010
849099233,9cyclictest12-21ksoftirqd/019:45:310
849099233,10cyclictest12-21ksoftirqd/021:00:100
8490992313,2cyclictest141rcu_preempt20:42:550
8490992310,3cyclictest141rcu_preempt20:02:350
197822232,2sleep00-21swapper/020:50:230
849399229,4cyclictest34-21ksoftirqd/320:51:393
849399229,4cyclictest34-21ksoftirqd/320:37:123
849399228,4cyclictest34-21ksoftirqd/319:36:513
849399228,4cyclictest34-21ksoftirqd/319:36:503
849399225,4cyclictest141rcu_preempt21:01:143
849399225,10cyclictest1-21systemd20:59:383
8493992213,6cyclictest9687-21python319:10:263
849299223,6cyclictest28-21ksoftirqd/220:40:362
849299220,7cyclictest28-21ksoftirqd/219:15:092
849299220,6cyclictest317882chrt20:02:062
849299220,6cyclictest19210-21chrt19:35:072
849299220,6cyclictest19210-21chrt19:35:062
849199223,11cyclictest29967-21ssh22:35:191
8491992214,6cyclictest7467-21sensors_temp20:20:291
8491992213,6cyclictest24242-21sed23:25:231
8491992211,8cyclictest5489-21ssh00:05:221
849099229,7cyclictest141rcu_preempt19:12:150
849099229,6cyclictest141rcu_preempt20:49:190
8490992213,2cyclictest141rcu_preempt19:25:100
8490992210,2cyclictest141rcu_preempt20:07:240
326782220,2sleep10-21swapper/123:32:341
849399218,4cyclictest331rcuc/319:45:203
849399217,4cyclictest34-21ksoftirqd/320:16:263
849399216,2cyclictest141rcu_preempt19:29:453
849399214,3cyclictest141rcu_preempt19:32:103
849399213,8cyclictest34-21ksoftirqd/320:25:303
849299211,5cyclictest14375-21chrt20:37:092
849199219,9cyclictest0-21swapper/123:44:591
849199219,8cyclictest24702-21apt-get21:35:131
8491992115,5cyclictest22-21ksoftirqd/123:00:011
8491992114,4cyclictest641-21runrttasks21:32:121
8491992114,4cyclictest641-21runrttasks21:32:111
8491992113,6cyclictest2570-21seq23:35:061
8491992111,6cyclictest55150irq/140-enp3s0-00:30:231
8491992110,7cyclictest12268-21apt-get22:20:151
849099217,7cyclictest141rcu_preempt20:25:410
849099211,12cyclictest656-21snmpd20:55:530
8490992110,2cyclictest141rcu_preempt20:34:270
89362200,2sleep10-21swapper/122:16:271
849199201,8cyclictest8935-21unixbench_multi21:20:301
8491992012,6cyclictest28746-21ls19:55:241
849099208,2cyclictest141rcu_preempt21:05:210
849099206,4cyclictest12-21ksoftirqd/019:54:160
849199198,9cyclictest18382-21ssh23:48:221
849199197,10cyclictest641-21runrttasks23:05:301
849199192,12cyclictest31809-21apt-get00:00:161
849199191,9cyclictest725-21cpu22:10:201
8491991913,4cyclictest26715-21sh22:31:391
8491991913,4cyclictest18245-21sh21:56:401
8491991913,4cyclictest17136-21sh22:51:041
8491991912,5cyclictest27066-21cat19:50:301
8491991912,5cyclictest20143-21ntp_kernel_pll_19:35:231
8491991912,5cyclictest20143-21ntp_kernel_pll_19:35:221
8491991912,5cyclictest20060-21ntpq20:50:241
8491991912,5cyclictest12232-21seq19:16:001
8491991912,4cyclictest20112-21/usr/sbin/munin23:50:221
8491991911,6cyclictest4923-21sed22:40:331
8491991911,6cyclictest27604-21cat22:05:191
8491991911,6cyclictest27604-21cat22:05:181
8491991911,6cyclictest24418-21ls00:20:301
8491991911,6cyclictest1122-21date20:05:331
849199189,7cyclictest21385-21find22:00:141
849199189,7cyclictest10919-21df20:30:181
849199189,5cyclictest27647-21diskmemload21:10:201
849199186,10cyclictest0-21swapper/121:50:331
849199181,8cyclictest16474-21users19:25:321
8491991813,3cyclictest11020-21sh23:14:441
8491991812,4cyclictest3765-21sh21:45:021
8491991812,4cyclictest32542-21sh21:40:501
8491991812,4cyclictest22340-21sh22:28:231
8491991812,4cyclictest18690-21proc_pri23:20:291
8491991812,4cyclictest18690-21proc_pri23:20:291
8491991812,4cyclictest12511-21sh22:47:311
8491991811,5cyclictest26110-21entropy21:05:201
849199181,12cyclictest13166-21ssh21:25:181
8491991810,6cyclictest3456-21sed20:10:331
8491991810,6cyclictest17263-21expr20:45:181
8491991810,6cyclictest15572-21cut20:40:291
8491991810,6cyclictest10117-21seq19:11:211
8491991810,5cyclictest0-21swapper/100:10:201
849199179,6cyclictest21740-21diskstats19:40:191
849199178,7cyclictest12034-21expr23:15:181
849199178,6cyclictest13027-21cat19:20:211
849199172,10cyclictest23179-21apt-get21:00:051
8491991712,3cyclictest17962-21needreboot00:15:231
8491991711,4cyclictest4813-21kernelversion20:15:231
8491991711,4cyclictest26922-21expr23:00:181
8491991710,5cyclictest23803-21wc19:45:201
8491991710,5cyclictest12723-21basename20:35:151
849199161,6cyclictest26715-21irqstats23:55:241
8491991611,3cyclictest9619-21swap20:25:291
8491991610,4cyclictest1398-21sh21:15:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional