You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-06 - 17:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 4.16.15-rt7 (Profile)

Latency plot of shadow in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 06, 2022 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17242243164,24sleep10-21swapper/107:09:091
17242243164,24sleep10-21swapper/107:09:091
15172238163,24sleep30-21swapper/307:06:463
15172238163,24sleep30-21swapper/307:06:463
14992237163,60sleep00-21swapper/007:06:320
14992237163,60sleep00-21swapper/007:06:310
15722203165,25sleep20-21swapper/207:07:172
15722203165,25sleep20-21swapper/207:07:172
2603821620,3sleep0188499cyclictest09:15:160
2603821620,3sleep0188499cyclictest09:15:160
1123721570,4sleep3190299cyclictest08:40:243
189699104101,2cyclictest0-21swapper/212:32:002
65742560,4sleep3190299cyclictest10:59:573
1884993912,13cyclictest22950-21munin-run09:10:010
72722370,1sleep10-21swapper/111:00:141
1902993713,21cyclictest3053-21/usr/sbin/munin12:05:243
190299371,11cyclictest22541-21/usr/sbin/munin07:55:263
190299371,11cyclictest14370-21df12:30:133
1902993613,20cyclictest13419-21/usr/sbin/munin11:15:243
1884993611,12cyclictest12569-21munin-run10:00:010
1902993512,3cyclictest4007-21/usr/sbin/munin09:40:233
1902993512,3cyclictest11839-21/usr/sbin/munin12:25:153
1902993511,5cyclictest28805-21gltestperf10:35:173
190299351,12cyclictest18069-21sshd12:38:303
190299349,4cyclictest15184-21/usr/sbin/munin07:35:263
190299343,4cyclictest13400-40dnf07:33:353
190299342,12cyclictest23836-21grep09:10:193
1902993411,3cyclictest18150-21/usr/sbin/munin07:45:213
1902993411,2cyclictest18764-21/usr/sbin/munin09:00:263
190299340,32cyclictest0-21swapper/311:05:243
190299331,7cyclictest25372-21/usr/sbin/munin09:15:193
190299331,7cyclictest25372-21/usr/sbin/munin09:15:193
1902993311,3cyclictest30384-21/usr/sbin/munin10:40:253
1902993310,4cyclictest9752-21turbostat08:40:023
189099331,17cyclictest0-21swapper/110:50:151
188499338,12cyclictest19783-21munin-run11:30:010
190299329,2cyclictest10602-21/usr/sbin/munin09:55:313
190299324,3cyclictest86550irq/126-enp1s0-10:10:133
1902993211,3cyclictest8363-21memory12:15:203
1902993211,3cyclictest24124-21sshd11:39:303
190299321,13cyclictest0-21swapper/310:05:173
1902993210,3cyclictest25352-21grep08:00:163
1902993210,3cyclictest19391-21/usr/sbin/munin10:15:133
190299317,4cyclictest30204-21timerwakeupswit11:50:263
190299317,3cyclictest0-21swapper/311:41:423
190299314,4cyclictest0-21swapper/310:21:213
190299314,4cyclictest0-21swapper/310:21:213
190299311,6cyclictest27950-21latency_hist10:35:013
190299311,3cyclictest11648-21latency_hist12:25:013
1902993111,3cyclictest28601-21cat11:50:013
1902993111,3cyclictest11480-21cat11:10:133
1902993110,3cyclictest30676-21load09:25:213
190299310,13cyclictest0-21swapper/311:56:313
1890993115,4cyclictest20921-21memory11:30:191
189099311,21cyclictest0-21swapper/108:51:311
188499314,12cyclictest22944-21diskmemload10:50:010
190299308,3cyclictest7866-21sort07:20:163
190299301,9cyclictest571-21ntp_states09:30:223
190299301,9cyclictest0-21swapper/311:00:223
1902993012,3cyclictest15399-21nfs4_client08:50:203
190299301,15cyclictest0-21swapper/308:34:173
1902993011,2cyclictest6504-21unixbench_multi07:15:433
1902993010,3cyclictest836-21runrttasks09:53:543
1902993010,3cyclictest5802-21cut12:10:143
1902993010,3cyclictest32489-21grep08:15:223
1902993010,3cyclictest18564-21awk11:25:163
190299299,3cyclictest2598-21wc08:20:253
190299299,3cyclictest21575-21expr09:05:163
190299299,3cyclictest12789-21gltestperf08:45:163
190299299,2cyclictest24314-21fschecks_time10:25:153
190299298,3cyclictest21071-21if_enp1s007:50:193
190299294,3cyclictest86550irq/126-enp1s0-09:45:133
190299293,5cyclictest15962-21diskstats11:20:153
1902992910,3cyclictest3572-21cstates08:25:143
189099292,10cyclictest3777-21latency_hist09:40:011
188499293,13cyclictest22318-21munin-run07:55:000
190299289,3cyclictest2694-21latency09:35:183
190299288,3cyclictest20421-21grep11:30:153
1902992811,3cyclictest3287-21sed10:50:183
190299281,12cyclictest32579-21/usr/sbin/munin10:45:203
190299281,11cyclictest86550irq/126-enp1s0-08:10:243
190299281,11cyclictest86550irq/126-enp1s0-08:10:243
190299281,11cyclictest0-21swapper/309:23:153
1902992810,3cyclictest2760-21sh07:10:163
1902992810,3cyclictest17871-21switchtime08:55:253
189099286,3cyclictest0-21swapper/107:58:351
1890992813,5cyclictest780-21rngd11:05:151
189099281,12cyclictest0-21swapper/108:13:111
189099281,12cyclictest0-21swapper/108:13:111
1890992811,10cyclictest86750irq/128-enp1s0-12:05:371
190299275,4cyclictest0-21swapper/312:00:033
190299271,6cyclictest0-21swapper/310:01:473
190299271,10cyclictest11328-21latency_hist07:30:033
190299268,3cyclictest16800-21latency07:40:193
189099261,18cyclictest0-21swapper/111:35:021
189099261,18cyclictest0-21swapper/111:11:511
189099261,10cyclictest780-21rngd08:05:231
188499261,13cyclictest7235-21awk12:15:000
188499261,12cyclictest30788-21awk11:55:010
189699250,10cyclictest0-21swapper/209:12:552
189099259,7cyclictest158750irq/12-ttyS009:55:201
189099251,10cyclictest0-21swapper/107:32:151
188499251,6cyclictest0-21swapper/009:26:330
188499251,13cyclictest29553-21awk09:25:010
188499251,11cyclictest13820-21awk12:30:020
77832241,19sleep27796-21diskstats12:15:152
251382240,4sleep121-21rcuc/108:00:151
189699245,14cyclictest87950irq/132-enp2s0-09:43:432
189099247,7cyclictest1-21systemd12:40:011
189099246,8cyclictest0-21swapper/107:19:551
189099241,9cyclictest0-21swapper/107:37:051
189099241,17cyclictest0-21swapper/112:11:391
189099241,17cyclictest0-21swapper/110:10:201
189099241,16cyclictest0-21swapper/109:21:411
189099241,12cyclictest8566-21cut12:15:231
189099241,10cyclictest780-21rngd08:21:051
189099241,10cyclictest0-21swapper/110:19:351
189099241,10cyclictest0-21swapper/109:32:481
189099241,10cyclictest0-21swapper/108:38:031
1896992314,3cyclictest6849-21kworker/u8:208:08:362
189099239,3cyclictest0-21swapper/110:58:091
189099238,4cyclictest0-21swapper/110:29:111
189099237,2cyclictest0-21swapper/110:40:151
189099236,9cyclictest0-21swapper/112:00:251
189099236,7cyclictest86750irq/128-enp1s0-07:25:251
189099234,9cyclictest7072-21grep08:30:261
189099233,17cyclictest221ktimersoftd/111:55:141
189099232,6cyclictest0-21swapper/110:08:491
189099231,9cyclictest22311-21latency_hist07:55:011
189099231,9cyclictest0-21swapper/112:29:071
189099231,21cyclictest0-21swapper/109:00:281
189099231,20cyclictest0-21swapper/109:45:141
189099231,16cyclictest0-21swapper/111:19:241
189099231,15cyclictest0-21swapper/111:20:221
189099231,15cyclictest0-21swapper/109:45:011
189099231,12cyclictest29149-21diskstats11:50:141
189099231,12cyclictest0-21swapper/109:11:061
189099231,12cyclictest0-21swapper/108:15:511
189099231,10cyclictest0-21swapper/112:23:271
188499232,8cyclictest13400-40dnf07:33:330
1884992316,4cyclictest17229-21cut08:55:150
189699228,4cyclictest0-21swapper/210:40:112
189699221,4cyclictest19392-21kworker/u8:010:20:282
189699221,4cyclictest19392-21kworker/u8:010:20:282
189699221,4cyclictest0-21swapper/211:25:162
189699221,15cyclictest0-21swapper/211:52:322
189699221,11cyclictest0-21swapper/210:18:382
189099228,3cyclictest0-21swapper/110:02:431
189099226,8cyclictest0-21swapper/108:42:391
189099226,2cyclictest86750irq/128-enp1s0-11:30:011
189099225,7cyclictest0-21swapper/108:45:071
189099225,10cyclictest23-21ksoftirqd/109:25:321
189099221,8cyclictest14112-21cat12:30:121
189099221,15cyclictest0-21swapper/108:55:481
1890992211,4cyclictest22233-21smart_sda09:05:251
189099221,12cyclictest29034-21grep10:35:201
189099221,11cyclictest19619-21unixbench-2d07:45:251
189099221,11cyclictest0-21swapper/110:24:071
189099221,11cyclictest0-21swapper/110:24:071
188499223,2cyclictest0-21swapper/012:17:370
189699211,7cyclictest245332cat08:00:012
189699211,11cyclictest26530-21latency_hist11:45:012
189699211,10cyclictest0-21swapper/208:48:392
189699210,10cyclictest0-21swapper/212:07:292
189099212,17cyclictest0-21swapper/109:51:351
189099211,2cyclictest0-21swapper/109:15:511
189099211,2cyclictest0-21swapper/109:15:511
189099211,18cyclictest0-21swapper/110:32:251
189099211,11cyclictest5355-21latency_hist08:30:011
189099211,11cyclictest0-21swapper/111:48:081
189099211,10cyclictest2238-21tr07:10:131
189099211,10cyclictest0-21swapper/110:45:591
188499218,4cyclictest25255-21grep11:40:190
188499211,9cyclictest0-21swapper/008:35:110
1884992111,3cyclictest86750irq/128-enp1s0-10:54:330
188499211,11cyclictest14036-21/usr/sbin/munin12:30:160
188499211,10cyclictest17739-21tail12:35:260
144252211,2sleep20-21swapper/210:00:512
190299202,11cyclictest0-21swapper/308:05:083
189699205,3cyclictest12050irq/123-ahci[0008:40:012
189699203,7cyclictest17658-21/usr/sbin/munin12:35:252
1896992016,2cyclictest30-21rcuc/212:30:012
189262201,3sleep20-21swapper/210:12:072
189099201,17cyclictest0-21swapper/111:40:041
188499209,3cyclictest823-21snmpd10:42:340
188499209,3cyclictest29757-21unixbench_singl10:35:280
188499201,5cyclictest25364-21grep10:25:250
112512201,2sleep00-21swapper/012:20:320
189699193,7cyclictest32-21ksoftirqd/211:04:572
189699192,7cyclictest0-21swapper/208:27:532
189699191,8cyclictest0-21swapper/211:14:232
189699191,6cyclictest0-21swapper/209:20:282
1896991915,3cyclictest32-21ksoftirqd/209:30:232
189699191,12cyclictest0-21swapper/209:29:072
189699191,11cyclictest780-21rngd07:23:272
189699191,11cyclictest0-21swapper/208:31:272
189699191,10cyclictest24679-21sed10:25:182
189099197,3cyclictest9143-21perf07:25:011
189099191,16cyclictest0-21swapper/107:41:241
188499198,3cyclictest823-21snmpd11:12:400
188499195,5cyclictest10805-21kernelversion08:40:170
188499193,13cyclictest24687-21unixbench_multi09:10:280
188499191,6cyclictest0-21swapper/011:35:400
188499191,6cyclictest0-21swapper/011:03:040
188499191,5cyclictest0-21swapper/012:09:350
188499191,5cyclictest0-21swapper/010:22:340
188499191,5cyclictest0-21swapper/010:22:340
188499191,5cyclictest0-21swapper/010:13:240
188499191,5cyclictest0-21swapper/010:02:580
1884991914,3cyclictest28567-21idleruntime-cro11:50:010
188499191,15cyclictest6724-21cut09:45:140
188499191,11cyclictest12177-21/usr/sbin/munin08:45:230
188499191,11cyclictest0-21swapper/009:50:240
188499191,10cyclictest30998-21/usr/sbin/munin11:55:150
188499191,10cyclictest18764-21/usr/sbin/munin09:00:160
15132191,3sleep00-21swapper/008:20:140
69912181,3sleep20-21swapper/212:12:142
6852181,2sleep20-21swapper/208:17:512
43072180,3sleep20-21swapper/210:54:022
309262181,3sleep20-21swapper/208:12:412
309262181,3sleep20-21swapper/208:12:412
28112180,3sleep00-21swapper/012:04:580
189699186,3cyclictest86550irq/126-enp1s0-11:55:232
189699183,9cyclictest0-21swapper/212:04:122
189699182,5cyclictest30-21rcuc/208:40:112
189699181,9cyclictest14747-21strings07:35:192
1896991813,3cyclictest21055-21cpuspeed_turbos09:05:112
1896991812,3cyclictest19409-21sed09:00:152
189699181,15cyclictest0-21swapper/207:45:082
189699181,11cyclictest0-21swapper/210:05:492
189699181,11cyclictest0-21swapper/209:51:572
189699181,10cyclictest0-21swapper/211:30:022
189699180,6cyclictest32-21ksoftirqd/208:00:182
188499188,8cyclictest18681-21fschecks_time07:45:150
188499186,3cyclictest0-21swapper/010:30:080
188499181,9cyclictest5686-21ntp_states10:55:220
188499181,5cyclictest0-21swapper/009:36:170
188499181,4cyclictest0-21swapper/011:20:260
1884991813,3cyclictest9842-21kernelversion11:05:180
276282171,2sleep20-21swapper/210:30:582
217192170,2sleep00-21swapper/011:31:530
189699177,2cyclictest86750irq/128-enp1s0-11:07:472
189699171,9cyclictest0-21swapper/211:15:182
189699171,8cyclictest22944-21diskmemload09:35:372
189699171,7cyclictest0-21swapper/209:45:112
189699171,5cyclictest0-21swapper/209:19:272
189699171,5cyclictest0-21swapper/209:19:272
189699171,4cyclictest10326-21hwlatdetect12:20:162
189699171,4cyclictest0-21swapper/211:45:142
189699171,2cyclictest86550irq/126-enp1s0-11:35:042
1896991712,3cyclictest5234-21head10:55:162
1896991712,3cyclictest4976-21/usr/sbin/munin07:15:212
1896991712,3cyclictest30-21rcuc/210:35:132
1896991712,3cyclictest1416-21cat08:20:122
1896991712,3cyclictest11264-21tail09:55:212
189699170,5cyclictest1814-21vmstat10:45:272
188499172,12cyclictest4976-21users08:25:260
188499171,6cyclictest0-21swapper/009:30:130
188499171,3cyclictest32337-21head08:15:180
188499171,2cyclictest16130-21runrttasks08:52:280
188499171,14cyclictest0-21swapper/007:25:040
1884991711,3cyclictest4265-21expr09:40:120
1884991711,3cyclictest22825-21cat07:55:120
1884991710,4cyclictest26948-21/usr/sbin/munin08:05:200
188499170,9cyclictest0-21swapper/007:20:250
188499170,5cyclictest0-21swapper/011:15:180
189699161,6cyclictest17483-21memory08:55:182
189699161,5cyclictest0-21swapper/211:20:142
189699161,3cyclictest0-21swapper/207:25:142
189699161,14cyclictest0-21swapper/207:10:582
1896991611,3cyclictest20414-21ls07:50:132
1896991611,3cyclictest15956-21/usr/sbin/munin07:40:262
189699161,11cyclictest14418-21cpu08:50:112
188499161,8cyclictest823-21snmpd08:12:060
188499161,8cyclictest823-21snmpd08:12:060
188499161,6cyclictest19391-21/usr/sbin/munin10:15:140
188499161,14cyclictest0-21swapper/007:12:560
1884991611,3cyclictest5404-21grep07:15:280
1884991611,3cyclictest15956-21/usr/sbin/munin07:40:150
1884991611,3cyclictest13844-21ls07:35:110
188499161,12cyclictest6188-21grep08:30:160
1884991610,3cyclictest25399-21tail08:00:160
189699151,7cyclictest32-21ksoftirqd/207:30:152
188499159,3cyclictest15099-21grep10:05:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional