You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-28 - 03:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Sep 28, 2023 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
155642239164,24sleep30-21swapper/319:08:083
156182237164,23sleep00-21swapper/019:08:500
157092232163,55sleep10-21swapper/119:10:001
155032215185,20sleep20-21swapper/219:07:232
2291221630,4sleep31581399cyclictest21:55:033
2888821430,4sleep21580699cyclictest22:05:302
2133821060,4sleep21580699cyclictest20:35:302
182221000,2sleep20-21swapper/223:33:292
146822590,4sleep121-21rcuc/100:00:371
43872540,3sleep21580699cyclictest22:25:212
1579699389,13cyclictest6353-21munin-run23:45:010
15796993620,13cyclictest6712-21basename20:05:010
1579699355,14cyclictest23971-21awk00:25:010
1579699314,12cyclictest29935-21basename23:25:010
1579699304,13cyclictest11101-21munin-run20:15:010
1579699302,13cyclictest28355-21munin-run00:35:010
1580199291,26cyclictest0-21swapper/121:43:271
15796992910,16cyclictest12332-21awk22:45:010
1580199281,17cyclictest0-21swapper/120:03:081
1579699281,14cyclictest7930-21munin-run22:35:010
1580199271,3cyclictest0-21swapper/122:48:521
1580199271,3cyclictest0-21swapper/121:08:231
1580199271,2cyclictest0-21swapper/121:47:151
1580199261,4cyclictest10131-21munin-run22:40:021
1580199261,2cyclictest0-21swapper/122:11:061
1580199260,2cyclictest0-21swapper/122:15:101
1580199253,14cyclictest23-21ksoftirqd/122:30:021
1580199251,4cyclictest25877-21perf19:35:011
1580199251,3cyclictest0-21swapper/123:07:491
1580199251,11cyclictest29295-21latency_hist22:10:021
15796992519,3cyclictest25865-21latency_hist19:35:000
1581399249,12cyclictest86550irq/126-enp1s0-19:50:303
1581399249,12cyclictest86550irq/126-enp1s0-19:50:303
1581399244,3cyclictest27670-21df22:05:173
1580199242,16cyclictest0-21swapper/119:41:551
1580199241,5cyclictest0-21swapper/123:37:541
1580199241,3cyclictest0-21swapper/121:52:481
1580199241,16cyclictest0-21swapper/121:29:401
1579699246,15cyclictest10136-21awk22:40:010
15813992316,4cyclictest25074-21grep23:10:363
1580699231,16cyclictest0-21swapper/223:14:222
1580699231,11cyclictest15159-21munin-run00:05:012
1580199232,16cyclictest0-21swapper/119:15:411
1580199231,20cyclictest0-21swapper/121:22:101
1580199231,15cyclictest0-21swapper/120:06:371
1580199230,2cyclictest0-21swapper/120:56:271
1580199230,2cyclictest0-21swapper/120:56:271
1580199230,21cyclictest0-21swapper/121:55:321
1580199230,17cyclictest0-21swapper/100:26:301
188562221,2sleep10-21swapper/122:59:251
1580699229,5cyclictest0-21swapper/200:25:182
1580699225,9cyclictest3516-21needreboot19:55:262
1580699221,5cyclictest0-21swapper/223:55:492
1580699221,12cyclictest8160-21/usr/sbin/munin22:35:262
1580199227,2cyclictest0-21swapper/120:10:221
1580199222,18cyclictest0-21swapper/123:22:551
1580199222,18cyclictest0-21swapper/121:17:141
1580199222,18cyclictest0-21swapper/120:31:131
1580199221,18cyclictest2913-21diskmemload22:21:191
1580199221,18cyclictest15736-21diskstats00:05:221
1580199221,15cyclictest0-21swapper/123:55:171
1580199221,15cyclictest0-21swapper/123:19:231
1580199221,15cyclictest0-21swapper/119:57:501
1580199221,15cyclictest0-21swapper/119:53:591
1580199221,15cyclictest0-21swapper/119:53:591
1580199221,15cyclictest0-21swapper/119:35:121
1580199221,15cyclictest0-21swapper/119:28:161
1580199221,15cyclictest0-21swapper/100:36:471
1580199221,15cyclictest0-21swapper/100:16:551
1580199221,14cyclictest0-21swapper/100:10:231
1580199221,13cyclictest0-21swapper/121:32:031
1579699229,4cyclictest13996-21irqstats00:00:290
133162220,4sleep339-21rcuc/320:20:013
1581399213,2cyclictest0-21swapper/321:22:313
1581399212,10cyclictest28301-21/usr/sbin/munin19:40:203
1580699211,7cyclictest0-21swapper/221:59:242
15806992116,3cyclictest16934-21latency_hist22:55:012
1580699211,5cyclictest18106-21munin-node19:15:022
15806992115,3cyclictest20406-21head00:15:232
1580699211,10cyclictest0-21swapper/221:10:462
1580199212,18cyclictest0-21swapper/122:44:111
1580199212,18cyclictest0-21swapper/120:22:201
15801992117,3cyclictest0-21swapper/121:40:021
15801992115,3cyclictest24036-21grep23:10:241
15801992113,3cyclictest0-21swapper/100:24:501
15801992113,2cyclictest0-21swapper/123:41:391
15801992113,2cyclictest0-21swapper/119:21:001
15801992112,3cyclictest0-21swapper/119:10:501
1580199211,19cyclictest0-21swapper/120:42:551
1580199211,19cyclictest0-21swapper/120:36:111
1580199211,18cyclictest0-21swapper/120:50:141
1580199211,18cyclictest0-21swapper/120:19:261
1580199211,15cyclictest0-21swapper/123:29:001
1580199211,11cyclictest1821-21dnf23:33:281
1580199211,11cyclictest10274-21unixbench_multi23:50:331
1580199210,15cyclictest0-21swapper/100:33:251
1579699216,14cyclictest0-21swapper/021:00:010
1579699216,14cyclictest0-21swapper/021:00:010
1579699213,5cyclictest2913-21diskmemload21:50:230
1579699211,11cyclictest0-21swapper/022:53:390
87772201,3sleep00-21swapper/023:50:180
1581399208,3cyclictest823-21snmpd21:45:013
1581399202,10cyclictest5345-21/usr/sbin/munin21:15:283
1581399202,10cyclictest20738-21/usr/sbin/munin21:50:223
1581399201,6cyclictest0-21swapper/322:51:553
1581399201,6cyclictest0-21swapper/300:14:133
1581399201,2cyclictest0-21swapper/322:04:153
1581399201,2cyclictest0-21swapper/322:04:153
1581399201,15cyclictest823-21snmpd23:30:223
1581399201,13cyclictest0-21swapper/321:08:373
1581399201,11cyclictest6581-21/usr/sbin/munin23:45:273
1581399201,11cyclictest2189-21/usr/sbin/munin23:35:343
1581399201,10cyclictest19002-40dnf20:30:313
1580699205,3cyclictest2913-21diskmemload23:40:012
1580699204,9cyclictest0-21swapper/223:42:262
1580699203,4cyclictest581-21systemd-journal21:05:012
1580699203,10cyclictest0-21swapper/200:31:102
1580699202,7cyclictest24516-21/usr/sbin/munin20:45:272
1580699201,5cyclictest0-21swapper/200:10:252
1580699201,13cyclictest0-21swapper/221:39:482
1580699201,11cyclictest28735-21if_err_enp1s023:20:232
1580699201,10cyclictest28671-21latency_hist20:55:012
1580699201,10cyclictest12565-21find20:15:322
1580699201,10cyclictest11092-21latency_hist20:15:012
1580699201,10cyclictest0-21swapper/223:01:312
1580699201,10cyclictest0-21swapper/222:13:042
1580699201,10cyclictest0-21swapper/221:49:412
15801992016,3cyclictest0-21swapper/120:29:581
15801992013,4cyclictest7716-40dnf22:32:311
1580199201,2cyclictest0-21swapper/121:10:251
1580199201,18cyclictest0-21swapper/120:47:441
1580199201,10cyclictest7266-21sed23:45:231
1580199200,18cyclictest0-21swapper/121:00:351
1580199200,14cyclictest86850irq/129-enp1s0-23:04:041
1579699207,3cyclictest0-21swapper/000:13:470
15796992015,3cyclictest24830-21smartctl23:10:340
1579699201,11cyclictest9741-21/usr/sbin/munin21:25:270
1579699201,11cyclictest0-21swapper/022:47:510
114932200,4sleep230-21rcuc/222:40:302
39852191,3sleep00-21swapper/023:38:010
319372191,2sleep00-21swapper/023:27:540
246852190,5sleep01579699cyclictest00:25:210
242332191,3sleep00-21swapper/020:44:320
1581399198,3cyclictest19800-21/usr/sbin/munin00:15:343
1581399193,9cyclictest0-21swapper/300:04:343
1581399192,7cyclictest30874-21munin-run21:00:013
1581399192,7cyclictest30874-21munin-run21:00:013
1581399191,8cyclictest21466-21sh19:25:013
1581399191,7cyclictest0-21swapper/321:29:113
1581399191,6cyclictest0-21swapper/322:15:053
1581399191,5cyclictest0-21swapper/322:40:343
15813991913,3cyclictest16113-21perf21:40:013
1581399191,11cyclictest0-21swapper/323:50:413
1581399191,10cyclictest13528-21/usr/sbin/munin20:20:223
1580699195,4cyclictest21467-21sh19:25:012
1580699193,9cyclictest0-21swapper/221:21:572
1580699192,5cyclictest30-21rcuc/223:45:342
1580699191,9cyclictest21640-21cat23:05:182
1580699191,8cyclictest16082-21diskstats20:25:242
1580699191,7cyclictest23206-21grep19:25:272
1580699191,6cyclictest16788-21timerandwakeup00:05:372
1580699191,4cyclictest3076-21grep22:20:322
15806991914,3cyclictest7524-21tail20:05:222
15806991913,3cyclictest9433-21gltestperf23:50:232
1580699191,10cyclictest4711-21taskset20:00:092
1580699191,10cyclictest28771-21cstates00:35:192
1580699191,10cyclictest0-21swapper/223:29:102
1580699191,10cyclictest0-21swapper/222:18:512
1580699191,10cyclictest0-21swapper/221:51:582
1580699191,10cyclictest0-21swapper/221:26:012
1580699191,10cyclictest0-21swapper/200:20:172
15801991915,3cyclictest0-21swapper/122:00:331
15801991915,3cyclictest0-21swapper/122:00:331
15801991914,3cyclictest0-21swapper/122:50:241
1580199191,16cyclictest0-21swapper/119:45:111
1579699197,3cyclictest0-21swapper/023:56:250
1579699192,3cyclictest18246-21ntp_states19:15:020
1579699192,14cyclictest31774-21cut21:00:220
1579699191,9cyclictest21360-21/usr/sbin/munin23:05:290
1579699191,5cyclictest19611-21ntp_states21:45:300
15796991913,3cyclictest15116-21users20:20:330
1579699191,10cyclictest8776-21/usr/sbin/munin21:20:310
1579699191,10cyclictest32352-21/usr/sbin/munin23:30:260
1579699191,10cyclictest0-21swapper/022:58:020
1579699191,10cyclictest0-21swapper/022:26:250
1579699190,8cyclictest0-21swapper/021:15:310
44432181,2sleep00-21swapper/019:59:360
186612180,2sleep30-21swapper/322:56:543
1581399188,3cyclictest823-21snmpd00:21:233
1581399187,3cyclictest20182-21sshd19:15:303
1581399182,6cyclictest4493-21munin-run20:00:013
1581399181,9cyclictest26104-21/usr/sbin/munin19:35:253
1581399181,8cyclictest823-21snmpd22:10:053
1581399181,7cyclictest0-21swapper/322:45:083
1581399181,7cyclictest0-21swapper/322:32:313
15813991813,3cyclictest11939-21/usr/sbin/munin21:30:203
15813991812,3cyclictest29216-21sensors_temp23:20:303
1581399181,10cyclictest0-21swapper/300:27:323
1580699189,3cyclictest86550irq/126-enp1s0-22:46:552
1580699186,4cyclictest32-21ksoftirqd/222:05:012
1580699186,4cyclictest32-21ksoftirqd/222:05:012
1580699185,3cyclictest30389-21sed20:55:332
1580699185,3cyclictest30389-21sed20:55:332
1580699182,9cyclictest0-21swapper/221:30:312
1580699182,4cyclictest13876-21diskstats20:20:202
1580699181,9cyclictest7557-21users22:30:372
1580699181,9cyclictest0-21swapper/223:18:032
1580699181,9cyclictest0-21swapper/221:40:412
1580699181,10cyclictest0-21swapper/222:56:252
1580699181,10cyclictest0-21swapper/220:34:472
1579699187,4cyclictest0-21swapper/022:23:100
1579699187,3cyclictest6931-21/usr/sbin/munin20:05:310
1579699186,3cyclictest0-21swapper/021:41:410
1579699186,2cyclictest29287-21kworker/u8:122:15:010
1579699182,5cyclictest0-21swapper/000:35:210
1579699181,9cyclictest0-21swapper/023:19:260
1579699181,9cyclictest0-21swapper/023:02:150
1579699181,6cyclictest0-21swapper/021:37:450
15796991813,3cyclictest970-21gltestperf19:50:210
15796991813,3cyclictest970-21gltestperf19:50:210
15796991812,3cyclictest27773-21seq19:36:270
1579699181,11cyclictest0-21swapper/021:58:050
1581399173,3cyclictest86550irq/126-enp1s0-19:30:113
1581399171,9cyclictest0-21swapper/322:20:093
1581399171,9cyclictest0-21swapper/319:45:213
1581399171,9cyclictest0-21swapper/319:10:323
1581399171,8cyclictest0-21swapper/322:25:143
1581399171,7cyclictest12648-21sshd23:56:293
1581399171,7cyclictest0-21swapper/300:35:483
1581399171,6cyclictest9483-21diskstats20:10:233
1581399171,6cyclictest0-21swapper/323:40:383
1581399171,6cyclictest0-21swapper/323:15:273
1581399171,6cyclictest0-21swapper/300:05:543
15813991712,3cyclictest3154-21/usr/sbin/munin21:10:333
15813991712,3cyclictest30798-21gltestperf23:25:223
15813991712,3cyclictest22390-21cpuspeed_turbos20:40:223
15813991712,3cyclictest20317-21cat20:35:183
15813991711,3cyclictest8237-21cat22:35:183
1581399171,10cyclictest0-21swapper/323:00:323
1580699174,4cyclictest31537-21sshd19:45:302
1580699173,3cyclictest0-21swapper/220:40:262
1580699172,4cyclictest19230-21df19:15:182
1580699171,9cyclictest0-21swapper/221:07:542
1580699171,6cyclictest0-21swapper/221:15:032
15806991711,3cyclictest26068-21cat19:35:012
1579699177,3cyclictest15729-21/usr/sbin/munin20:25:220
1579699176,4cyclictest3744-21gltestperf21:10:220
1579699172,6cyclictest842-21lldpd20:38:420
1579699172,3cyclictest0-21swapper/021:34:570
1579699171,9cyclictest86550irq/126-enp1s0-00:15:290
1579699171,8cyclictest0-21swapper/020:34:420
1579699171,4cyclictest25135-21/usr/sbin/munin22:00:270
1579699171,4cyclictest25135-21/usr/sbin/munin22:00:270
15796991711,3cyclictest31532-21ntpq19:45:300
15796991711,3cyclictest305-21ntpq22:15:280
15796991711,3cyclictest27062-21diskstats20:50:200
1579699171,10cyclictest0-21swapper/000:06:260
1581399162,11cyclictest4827-21tr20:00:193
1581399161,9cyclictest0-21swapper/321:00:283
1581399161,6cyclictest26715-21/usr/sbin/munin20:50:283
1581399161,6cyclictest0-21swapper/300:30:053
15813991612,3cyclictest0-21swapper/320:45:373
1581399161,14cyclictest0-21swapper/319:26:573
15813991611,3cyclictest7920-21needreboot20:05:273
15813991611,3cyclictest18539-21/usr/sbin/munin21:45:343
15813991611,3cyclictest17826-21wc20:30:023
15813991610,4cyclictest21960-21gltestperf23:05:223
1580699161,14cyclictest0-21swapper/219:52:352
1580699161,14cyclictest0-21swapper/219:52:352
15806991611,3cyclictest26573-21fschecks_time19:35:222
15806991610,4cyclictest28966-21gltestperf19:40:232
1579699166,3cyclictest0-21swapper/023:45:300
1579699161,3cyclictest28698-21sshd22:05:290
15796991612,3cyclictest0-21swapper/019:28:210
15796991611,3cyclictest29019-21basename19:40:240
15796991611,3cyclictest24594-21cat20:45:180
15796991611,3cyclictest16262-21expr19:10:180
15796991611,3cyclictest12049-21basename20:15:250
1579699159,3cyclictest836-21runrttasks19:21:570
15796991511,2cyclictest927-21/usr/sbin/munin21:05:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional