You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-28 - 14:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 4.16.15-rt7 (Profile)

Latency plot of shadow in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 28, 2023 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
184302239163,24sleep10-21swapper/107:08:171
182782233160,24sleep00-21swapper/007:06:190
184162224193,21sleep20-21swapper/207:08:062
182512209138,58sleep30-21swapper/307:05:593
901021840,4sleep21866499cyclictest10:25:192
2369621700,3sleep11865799cyclictest08:30:261
1866499106102,3cyclictest0-21swapper/211:14:212
18664998480,3cyclictest0-21swapper/208:52:542
133522690,1sleep20-21swapper/211:50:132
179632650,4sleep121-21rcuc/108:20:141
94092630,4sleep01864899cyclictest08:00:140
94092630,4sleep01864899cyclictest08:00:140
18664996360,2cyclictest0-21swapper/207:56:042
18664996157,3cyclictest0-21swapper/209:34:102
94182570,4sleep339-21rcuc/311:40:193
203372560,4sleep11865799cyclictest09:37:331
205912530,2sleep09-21rcu_preempt09:40:010
8502490,1sleep10-21swapper/110:05:271
8502490,1sleep10-21swapper/110:05:271
278692470,2sleep30-21swapper/309:55:233
18664993835,2cyclictest0-21swapper/207:37:322
1866999361,26cyclictest780-21rngd09:08:003
1864899369,13cyclictest667-21munin-run08:55:000
1864899356,14cyclictest7366-21munin-run09:10:010
18669993412,9cyclictest16717-21taskset10:44:433
1866999341,25cyclictest847-21grep10:05:283
1866999341,25cyclictest847-21grep10:05:273
1865799341,15cyclictest0-21swapper/112:11:251
1865799341,15cyclictest0-21swapper/112:11:251
1864899345,13cyclictest27137-21munin-run09:55:010
168032340,2sleep10-21swapper/111:55:301
1866999323,22cyclictest0-21swapper/308:56:163
1866999320,22cyclictest0-21swapper/311:31:263
1865799321,3cyclictest0-21swapper/107:45:331
1865799320,13cyclictest12432-21ntp_states08:05:231
1866999315,17cyclictest780-21rngd10:36:033
1864899314,13cyclictest2355-21idleruntime-cro07:45:000
1866999306,15cyclictest780-21rngd11:15:523
1866999305,18cyclictest86550irq/126-enp1s0-10:30:483
1866999304,10cyclictest22816-21/usr/sbin/munin07:15:213
1866999301,19cyclictest780-21rngd08:05:003
1866999301,19cyclictest780-21rngd08:05:003
1865799301,12cyclictest0-21swapper/111:46:311
1865799300,19cyclictest0-21swapper/108:04:171
1865799300,19cyclictest0-21swapper/108:04:171
18648993010,16cyclictest5161-21munin-run09:05:010
18648993010,16cyclictest5161-21munin-run09:05:010
199412290,4sleep20-21swapper/208:25:022
1866999293,3cyclictest30772-21cstates07:35:163
1866999290,18cyclictest0-21swapper/308:55:023
1865799291,12cyclictest0-21swapper/107:53:171
1865799291,10cyclictest0-21swapper/111:40:191
1866999286,11cyclictest780-21rngd12:35:133
1866999285,13cyclictest780-21rngd07:59:123
1866999282,11cyclictest0-21swapper/312:32:363
1866999281,8cyclictest4868-21ntp_states10:15:233
1865799281,12cyclictest0-21swapper/112:38:131
1865799281,12cyclictest0-21swapper/110:27:571
1865799281,12cyclictest0-21swapper/109:54:291
1864899281,13cyclictest19939-21munin-run08:25:010
1866999278,9cyclictest0-21swapper/309:36:323
1866999278,5cyclictest22968-21/usr/sbin/munin09:45:213
1866999275,19cyclictest0-21swapper/311:35:283
1866999273,17cyclictest780-21rngd09:40:403
1866999272,10cyclictest0-21swapper/308:13:283
1866999271,15cyclictest17372-21perf12:00:013
1866999271,11cyclictest0-21swapper/309:26:163
1866999271,10cyclictest0-21swapper/309:51:543
1866999270,22cyclictest0-21swapper/311:08:163
1865799271,3cyclictest0-21swapper/110:55:571
1865799271,2cyclictest0-21swapper/110:37:331
1865799271,13cyclictest0-21swapper/110:49:091
1865799271,13cyclictest0-21swapper/107:29:571
1865799271,12cyclictest0-21swapper/107:33:491
1865799271,12cyclictest0-21swapper/107:13:251
1865799271,10cyclictest0-21swapper/112:08:291
1865799270,3cyclictest0-21swapper/109:07:571
1864899274,11cyclictest21173-21idleruntime-cro10:55:010
1864899272,13cyclictest17373-21munin-run12:00:010
1864899271,12cyclictest9569-21perf09:15:010
1866999265,10cyclictest0-21swapper/307:30:043
1866999262,22cyclictest86850irq/129-enp1s0-08:45:543
1866999262,11cyclictest780-21rngd08:31:143
1866999261,7cyclictest0-21swapper/308:20:343
1866999261,12cyclictest780-21rngd10:58:393
1866999260,18cyclictest0-21swapper/309:32:283
1866999260,12cyclictest26408-21chrt12:20:133
1865799266,3cyclictest0-21swapper/112:26:471
1865799262,2cyclictest86550irq/126-enp1s0-09:31:211
1865799261,9cyclictest0-21swapper/111:37:391
1865799261,2cyclictest0-21swapper/111:26:371
1865799261,13cyclictest0-21swapper/112:20:391
1864899261,13cyclictest6748-21idleruntime-cro07:55:000
1864899261,13cyclictest15538-21munin-run08:15:010
1864899260,12cyclictest23993-21basename12:15:020
1864899260,12cyclictest23993-21basename12:15:020
1866999259,8cyclictest0-21swapper/311:00:563
1866999259,6cyclictest0-21swapper/311:13:033
1866999255,18cyclictest780-21rngd11:26:423
1866999252,9cyclictest0-21swapper/312:09:463
1866999252,9cyclictest0-21swapper/312:00:523
1866999251,15cyclictest13569-21users09:20:313
1866999251,12cyclictest0-21swapper/310:25:233
1866999250,9cyclictest0-21swapper/310:54:313
1866999250,10cyclictest0-21swapper/307:53:203
1865799251,9cyclictest7357-21diskmemload11:16:031
1865799251,2cyclictest0-21swapper/112:01:411
1865799251,12cyclictest0-21swapper/110:32:451
1865799251,12cyclictest0-21swapper/108:55:571
1865799251,10cyclictest0-21swapper/111:12:291
1865799250,11cyclictest0-21swapper/112:30:051
1864899259,14cyclictest0-21swapper/011:45:020
1864899251,11cyclictest6388-21munin-run11:35:020
1866999247,7cyclictest86850irq/129-enp1s0-08:36:563
1866999244,8cyclictest0-21swapper/312:15:033
1866999243,8cyclictest780-21rngd07:48:163
1866999243,13cyclictest0-21swapper/310:45:243
1866999241,12cyclictest0-21swapper/307:30:003
1866999241,10cyclictest0-21swapper/307:43:383
18669992410,2cyclictest86850irq/129-enp1s0-08:42:003
1866999240,15cyclictest17134-21kworker/u8:212:26:513
1866999240,12cyclictest0-21swapper/308:09:143
1866999240,11cyclictest0-21swapper/310:00:203
1865799248,3cyclictest0-21swapper/109:41:551
1865799246,3cyclictest0-21swapper/108:10:151
1865799241,12cyclictest0-21swapper/111:53:011
1865799241,10cyclictest0-21swapper/110:00:251
1865799241,10cyclictest0-21swapper/107:59:091
1864899241,12cyclictest8598-21perf11:40:020
1864899241,12cyclictest28391-21idleruntime-cro12:25:020
1866999239,3cyclictest0-21swapper/309:15:123
1866999237,3cyclictest0-21swapper/308:15:223
1866999231,10cyclictest0-21swapper/307:10:513
1866999230,7cyclictest0-21swapper/309:05:023
1866999230,7cyclictest0-21swapper/309:05:023
1866999230,15cyclictest0-21swapper/311:49:533
1866999230,13cyclictest0-21swapper/308:30:023
1865799237,4cyclictest0-21swapper/111:03:191
1865799237,3cyclictest7357-21diskmemload09:55:191
1865799234,3cyclictest0-21swapper/110:13:211
1865799233,2cyclictest0-21swapper/110:41:491
1865799231,6cyclictest0-21swapper/111:09:521
1865799231,3cyclictest0-21swapper/109:18:131
18657992312,4cyclictest4797-21cut10:15:211
1865799231,16cyclictest0-21swapper/109:04:361
1865799231,16cyclictest0-21swapper/109:04:361
1865799231,14cyclictest0-21swapper/108:51:131
1865799231,11cyclictest0-21swapper/109:47:091
1864899237,14cyclictest23366-21perf11:00:010
1866999224,9cyclictest780-21rngd10:11:153
1866999221,10cyclictest14270-21ntp_states11:50:233
1866999221,10cyclictest0-21swapper/312:11:113
1866999221,10cyclictest0-21swapper/312:11:113
1866999221,10cyclictest0-21swapper/309:11:323
1866999220,16cyclictest86550irq/126-enp1s0-11:24:233
1866499221,4cyclictest0-21swapper/212:32:402
1865799225,6cyclictest86750irq/128-enp1s0-09:25:331
1865799224,3cyclictest0-21swapper/109:24:431
1865799222,11cyclictest27821-21ntp_states08:40:231
1865799221,12cyclictest0-21swapper/107:40:231
1865799221,10cyclictest0-21swapper/108:45:391
1864899228,13cyclictest0-21swapper/010:00:010
1864899221,7cyclictest0-21swapper/010:44:010
18648992210,3cyclictest4153-21users07:45:270
1866999212,6cyclictest0-21swapper/310:20:323
1866999211,10cyclictest0-21swapper/307:24:153
1866499215,7cyclictest0-21swapper/211:50:022
1866499211,6cyclictest0-21swapper/209:27:372
1865799217,3cyclictest0-21swapper/112:15:051
1865799215,3cyclictest0-21swapper/109:12:351
18657992114,4cyclictest21159-21munin-run10:55:001
1865799211,3cyclictest0-21swapper/107:35:171
18657992111,5cyclictest0-21swapper/107:23:261
18657992111,3cyclictest22126-21latency_hist08:30:021
1865799211,12cyclictest0-21swapper/111:32:171
1865799211,11cyclictest5439-21kworker/u8:210:20:191
1864899211,11cyclictest2505-21sshd12:38:550
235862200,3sleep230-21rcuc/211:00:162
1866499202,6cyclictest19589-21munin-run12:05:002
1866499202,15cyclictest12933-21grep09:20:232
18664992014,4cyclictest16225-21wc10:40:272
1866499200,5cyclictest0-21swapper/210:14:092
1865799209,4cyclictest0-21swapper/111:20:261
1865799209,2cyclictest0-21swapper/107:15:041
18657992014,2cyclictest0-21swapper/108:35:571
1865799201,3cyclictest0-21swapper/108:15:531
1864899209,3cyclictest29708-21sshd11:11:550
1864899208,10cyclictest759-21dbus-daemon07:40:000
1864899204,14cyclictest28706-21crond08:45:000
1864899202,10cyclictest15760-21/usr/sbin/munin08:15:170
261002191,2sleep20-21swapper/212:19:132
199632191,15sleep220087-21/usr/sbin/munin07:10:242
1866499191,6cyclictest0-21swapper/211:43:112
1866499191,6cyclictest0-21swapper/209:55:472
1866499191,5cyclictest0-21swapper/210:16:002
18664991912,4cyclictest22539-21cut12:10:212
18664991912,4cyclictest22539-21cut12:10:212
1866499191,15cyclictest20325-21tr08:25:142
1864899192,6cyclictest26945-21grep11:05:310
1864899191,9cyclictest0-21swapper/010:48:590
1864899191,6cyclictest5315-21tail10:15:280
1864899191,6cyclictest0-21swapper/011:22:250
18648991912,4cyclictest28102-21idleruntime-cro07:30:010
18648991912,4cyclictest25026-21sort08:35:160
1864899191,11cyclictest19814-21/usr/sbin/munin12:05:260
1864899191,10cyclictest940-21grep10:05:290
1864899191,10cyclictest940-21grep10:05:280
1864899191,10cyclictest9190-21needreboot10:25:230
1864899191,10cyclictest823-21snmpd11:54:030
113972190,3sleep211401-21needreboot10:30:252
1866499188,3cyclictest86850irq/129-enp1s0-07:30:252
1866499185,2cyclictest86750irq/128-enp1s0-09:15:212
1866499181,7cyclictest581-21systemd-journal10:55:002
1866499181,5cyclictest0-21swapper/211:26:002
1866499181,5cyclictest0-21swapper/209:51:072
1866499181,3cyclictest0-21swapper/210:55:272
18664991813,3cyclictest27177-21if_err_enp1s012:20:192
18664991813,3cyclictest21327-21tune2fs09:40:182
18664991813,3cyclictest17705-21runrttasks08:19:542
18664991812,3cyclictest5671-21sendmail_mailtr11:30:272
1866499181,10cyclictest0-21swapper/210:08:092
1866499181,10cyclictest0-21swapper/210:08:092
1866499180,15cyclictest23504-21tr08:30:242
1864899187,3cyclictest10400-21/usr/sbin/munin10:30:330
1864899183,3cyclictest86750irq/128-enp1s0-09:45:160
1864899181,9cyclictest29206-21tail12:25:180
1864899181,8cyclictest7357-21diskmemload11:25:160
1864899181,8cyclictest12589-21/usr/sbin/munin10:35:300
1864899181,7cyclictest24937-21needreboot07:20:230
1864899181,15cyclictest0-21swapper/008:27:120
235822171,2sleep20-21swapper/207:18:302
1866499175,4cyclictest0-21swapper/211:15:322
1866499173,6cyclictest2295-21chrt12:36:102
1866499172,6cyclictest27388-21users11:05:342
1866499171,7cyclictest6921-21memory10:20:252
1866499171,7cyclictest30564-21latency_hist12:30:012
1866499171,6cyclictest0-21swapper/210:45:462
1866499171,6cyclictest0-21swapper/210:00:012
1866499171,4cyclictest0-21swapper/209:37:412
1866499171,3cyclictest0-21swapper/212:09:502
18664991713,2cyclictest27679-21unixbench_singl07:25:282
18664991712,3cyclictest15566-21date08:15:012
18664991712,3cyclictest12699-21threads08:05:262
1866499171,14cyclictest23112-21grep09:45:132
1866499171,14cyclictest0-21swapper/211:20:092
1866499171,13cyclictest13745-21pmu-power10:35:262
1866499170,7cyclictest7555-21latency_hist09:10:012
1864899174,3cyclictest0-21swapper/009:26:570
1864899172,12cyclictest30197-21/usr/sbin/munin11:15:240
1864899171,7cyclictest0-21swapper/010:13:330
1864899171,6cyclictest0-21swapper/012:02:340
1864899171,6cyclictest0-21swapper/009:33:000
18648991713,3cyclictest0-21swapper/008:45:140
18648991712,3cyclictest0-21swapper/010:01:330
1864899171,15cyclictest0-21swapper/009:43:570
1864899171,14cyclictest0-21swapper/011:45:140
18648991711,3cyclictest7212-21expr07:55:160
1864899171,13cyclictest1124-21tr08:55:170
18648991710,4cyclictest0-21swapper/007:30:210
1866499163,4cyclictest29283-21df08:45:142
1866499161,9cyclictest0-21swapper/208:35:212
1866499161,6cyclictest0-21swapper/208:40:282
1866499161,5cyclictest0-21swapper/207:20:322
1866499161,14cyclictest0-21swapper/211:56:412
1866499161,14cyclictest0-21swapper/207:43:472
18664991611,3cyclictest5490-21if_enp1s007:50:182
18664991611,3cyclictest4033-21sed07:45:262
18664991611,3cyclictest3497-21expr09:00:152
18664991611,3cyclictest3497-21expr09:00:142
18664991611,2cyclictest836-21runrttasks09:12:572
1866499161,12cyclictest8185-21grep11:35:282
1864899165,3cyclictest86750irq/128-enp1s0-08:30:240
1864899165,3cyclictest2237-1kworker/u9:012:16:190
1864899163,3cyclictest86750irq/128-enp1s0-09:21:020
1864899161,7cyclictest0-21swapper/009:15:050
1864899161,6cyclictest31074-21cat12:30:160
1864899161,6cyclictest0-21swapper/011:01:250
1864899161,14cyclictest0-21swapper/010:23:450
1864899161,14cyclictest0-21swapper/008:05:540
18648991611,3cyclictest18857-21/usr/sbin/munin07:10:170
108462161,2sleep20-21swapper/208:01:302
108462161,2sleep20-21swapper/208:01:302
1866499151,5cyclictest823-21snmpd08:58:262
1864899158,4cyclictest21516-21munin-node07:15:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional