You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-22 - 01:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Sep 21, 2023 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
77712239163,24sleep10-21swapper/107:09:531
76072239162,24sleep00-21swapper/007:07:450
74792232164,55sleep30-21swapper/307:06:073
76782231162,56sleep20-21swapper/207:08:412
241092580,5sleep224114-21unixbench_singl08:55:342
142452580,1sleep00-21swapper/008:35:200
787699411,29cyclictest2490-21kworker/u8:110:47:071
7882993514,6cyclictest30089-21kworker/2:110:56:282
7882993514,6cyclictest30089-21kworker/2:110:56:282
7870993510,12cyclictest25831-21awk11:30:010
7882993419,4cyclictest26354-21/usr/sbin/munin07:50:302
7870993410,11cyclictest4450-21munin-run11:55:000
788799331,22cyclictest0-21swapper/308:05:113
788299331,11cyclictest30089-21kworker/2:109:43:402
787699331,6cyclictest0-21swapper/107:59:081
787699331,12cyclictest0-21swapper/111:44:421
787099327,12cyclictest5388-21munin-run09:30:000
787099326,13cyclictest3830-21munin-run10:40:000
788799316,8cyclictest86550irq/126-enp1s0-09:55:293
788799314,11cyclictest4441-21latency_hist11:55:013
787699311,14cyclictest0-21swapper/111:36:041
787699311,13cyclictest0-21swapper/112:14:431
787699311,12cyclictest0-21swapper/109:54:511
787099317,11cyclictest21435-21basename11:20:010
788799301,14cyclictest14835-21perf11:05:023
788799301,13cyclictest780-21rngd09:20:013
7887993010,12cyclictest1832-21tail09:20:223
788299301,6cyclictest86650irq/127-enp1s0-10:52:082
788299301,14cyclictest0-21swapper/209:24:542
788299301,13cyclictest30089-21kworker/2:109:12:182
7882993010,7cyclictest30089-21kworker/2:107:15:032
788299300,23cyclictest0-21swapper/211:48:542
788799297,11cyclictest20798-21latency_hist10:05:013
788799295,15cyclictest25175-21taskset10:14:513
788799293,18cyclictest15938-21unixbench_multi09:50:363
788299295,12cyclictest32-21ksoftirqd/209:09:442
788299291,13cyclictest17687-21ntpq09:55:262
788299291,13cyclictest0-21swapper/211:37:582
788299291,13cyclictest0-21swapper/207:35:262
788299291,13cyclictest0-21swapper/207:35:252
788299291,12cyclictest0-21swapper/212:10:142
788299291,12cyclictest0-21swapper/208:04:462
788299290,24cyclictest0-21swapper/208:17:022
788299290,14cyclictest0-21swapper/211:57:262
787699291,11cyclictest0-21swapper/110:42:311
7876992910,4cyclictest0-21swapper/108:58:251
787099291,11cyclictest28857-21head11:35:250
788799281,19cyclictest0-21swapper/312:09:513
788299281,2cyclictest32-21ksoftirqd/207:12:102
788299281,13cyclictest0-21swapper/210:00:302
788299281,11cyclictest0-21swapper/211:53:502
788299280,13cyclictest0-21swapper/212:05:422
787699289,3cyclictest27267-21memory07:50:261
787699281,12cyclictest0-21swapper/111:52:011
787699281,11cyclictest0-21swapper/112:06:551
787699281,11cyclictest0-21swapper/110:20:351
787699281,11cyclictest0-21swapper/107:11:351
788799278,5cyclictest0-21swapper/311:30:273
788299272,10cyclictest30-21rcuc/210:38:002
788299271,13cyclictest0-21swapper/208:10:222
788299271,12cyclictest0-21swapper/210:08:142
788299271,11cyclictest0-21swapper/207:57:342
788299271,10cyclictest0-21swapper/209:33:582
788299271,10cyclictest0-21swapper/208:51:422
7882992710,3cyclictest30089-21kworker/2:107:25:112
787699277,2cyclictest0-21swapper/111:30:181
787699271,12cyclictest0-21swapper/111:13:171
787699271,12cyclictest0-21swapper/110:39:511
787699271,12cyclictest0-21swapper/110:16:491
787699271,11cyclictest0-21swapper/110:10:251
787699271,11cyclictest0-21swapper/108:51:431
787699271,10cyclictest0-21swapper/111:16:571
787699271,10cyclictest0-21swapper/111:07:151
787699271,10cyclictest0-21swapper/111:00:571
787699271,10cyclictest0-21swapper/109:58:471
787699271,10cyclictest0-21swapper/109:22:561
787699270,10cyclictest0-21swapper/112:22:231
231212270,3sleep3788799cyclictest08:55:223
788799266,7cyclictest21713-21turbostat07:40:003
788799266,7cyclictest21713-21turbostat07:40:003
788799261,8cyclictest2333-21sed11:50:013
7887992617,6cyclictest5971-40dnf10:44:323
788799261,18cyclictest0-21swapper/311:41:593
788799261,11cyclictest0-21swapper/309:12:193
788299269,3cyclictest24666-21irqstats11:25:262
788299269,3cyclictest16313-21df08:40:222
788299269,3cyclictest16313-21df08:40:222
788299261,12cyclictest0-21swapper/212:23:262
788299261,11cyclictest1148-21perl08:05:242
788299261,11cyclictest0-21swapper/208:21:022
788299261,10cyclictest0-21swapper/210:29:582
788299260,12cyclictest0-21swapper/210:30:062
788299260,10cyclictest0-21swapper/208:49:022
787699266,3cyclictest0-21swapper/109:48:231
787699266,3cyclictest0-21swapper/109:48:231
787699261,9cyclictest0-21swapper/109:39:471
787699261,8cyclictest12700-21kworker/u8:109:29:331
787699261,15cyclictest21963-21ls10:05:251
7876992611,2cyclictest218602chrt12:32:491
787699261,11cyclictest0-21swapper/110:00:591
787699261,10cyclictest0-21swapper/112:26:391
787699260,11cyclictest0-21swapper/107:27:221
787099261,12cyclictest27398-21munin-run10:20:010
787099261,12cyclictest20802-21munin-run10:05:010
788799258,7cyclictest8811-21diskstats10:50:203
788799254,9cyclictest0-21swapper/308:12:313
788799253,13cyclictest21034-21/usr/sbin/munin10:05:323
788799251,22cyclictest6478-21kworker/u8:212:35:243
788799251,17cyclictest0-21swapper/310:17:463
788799251,13cyclictest0-21swapper/307:29:393
788299258,4cyclictest0-21swapper/211:10:302
788299254,5cyclictest30-21rcuc/208:25:162
788299252,8cyclictest30089-21kworker/2:111:20:402
788299252,7cyclictest86750irq/128-enp1s0-07:41:482
788299251,9cyclictest0-21swapper/212:01:482
788299251,9cyclictest0-21swapper/212:01:482
788299251,12cyclictest0-21swapper/208:39:422
788299251,11cyclictest0-21swapper/207:24:272
788299251,10cyclictest0-21swapper/209:45:022
788299251,10cyclictest0-21swapper/209:45:022
788299251,10cyclictest0-21swapper/209:39:502
788299250,11cyclictest0-21swapper/211:07:422
787699253,3cyclictest0-21swapper/108:30:251
787699252,3cyclictest0-21swapper/108:43:551
787699252,3cyclictest0-21swapper/108:43:551
787699251,9cyclictest0-21swapper/109:10:121
787699251,16cyclictest0-21swapper/107:15:211
787699250,13cyclictest0-21swapper/112:18:011
787099251,13cyclictest29598-21munin-run10:25:010
787099251,12cyclictest9163-21idleruntime-cro08:25:000
787099251,12cyclictest25205-21munin-run10:15:010
788799248,7cyclictest24919-21ntp_states11:25:303
788799244,8cyclictest0-21swapper/309:05:273
788799243,12cyclictest5293-21grep11:55:203
788799241,17cyclictest0-21swapper/311:10:193
788799241,14cyclictest0-21swapper/311:08:423
788799241,11cyclictest0-21swapper/308:50:173
788299241,7cyclictest0-21swapper/212:32:302
788299241,11cyclictest0-21swapper/210:47:582
788299241,11cyclictest0-21swapper/210:12:542
788299241,10cyclictest0-21swapper/209:17:102
788299240,12cyclictest0-21swapper/212:26:382
788299240,11cyclictest0-21swapper/211:03:582
787699246,2cyclictest0-21swapper/107:40:021
787699244,3cyclictest0-21swapper/107:30:401
787699242,11cyclictest25818-21munin-run11:30:011
787699240,10cyclictest0-21swapper/111:21:231
788799235,9cyclictest0-21swapper/312:19:163
788799232,8cyclictest0-21swapper/310:20:303
788799231,9cyclictest0-21swapper/312:34:043
788799231,10cyclictest0-21swapper/309:00:433
788299233,3cyclictest32-21ksoftirqd/207:33:152
788299233,3cyclictest30-21rcuc/210:20:402
788299231,9cyclictest0-21swapper/209:28:302
788299230,9cyclictest0-21swapper/207:45:262
787699236,3cyclictest0-21swapper/108:18:031
787699231,9cyclictest0-21swapper/108:10:281
7876992315,3cyclictest316872sleep110:28:491
787699231,2cyclictest0-21swapper/108:47:131
787699231,13cyclictest22033-21cpuspeed_turbos07:40:201
787699231,13cyclictest0-21swapper/112:01:211
787699231,13cyclictest0-21swapper/112:01:211
787699231,12cyclictest0-21swapper/107:38:581
787699231,11cyclictest1442-21/usr/sbin/munin11:45:311
787699231,10cyclictest0-21swapper/109:43:591
787699230,16cyclictest0-21swapper/109:05:241
787099235,15cyclictest22042-21munin-run12:35:010
52352230,4sleep1787699cyclictest11:55:201
788799228,3cyclictest0-21swapper/308:41:593
788799228,3cyclictest0-21swapper/308:41:593
788799227,9cyclictest0-21swapper/307:23:553
788799226,3cyclictest0-21swapper/307:55:593
788799226,2cyclictest0-21swapper/311:15:083
788799224,6cyclictest17558-21/usr/sbin/munin07:30:303
788799224,5cyclictest10270-21munin-node07:15:013
788799224,10cyclictest0-21swapper/310:55:243
788799224,10cyclictest0-21swapper/310:55:243
788799223,9cyclictest0-21swapper/310:45:303
788799223,3cyclictest0-21swapper/312:03:033
788799223,3cyclictest0-21swapper/312:03:033
788799223,12cyclictest0-21swapper/311:35:133
788799223,12cyclictest0-21swapper/309:49:173
788799223,12cyclictest0-21swapper/309:49:173
788799221,9cyclictest0-21swapper/312:20:403
788799221,7cyclictest18934-21awk12:25:303
7887992212,3cyclictest86550irq/126-enp1s0-08:25:303
788799221,10cyclictest780-21rngd08:03:553
788799221,10cyclictest0-21swapper/310:27:173
7887992210,3cyclictest0-21swapper/307:12:403
788299228,3cyclictest0-21swapper/209:52:062
788299228,2cyclictest0-21swapper/209:00:222
788299221,15cyclictest0-21swapper/211:33:022
788299220,8cyclictest0-21swapper/212:15:262
787699228,3cyclictest25323-21proc_pri07:45:321
787699222,17cyclictest32637-21grep10:30:191
7876992213,3cyclictest0-21swapper/108:25:141
787699221,14cyclictest0-21swapper/110:52:191
787699220,9cyclictest0-21swapper/108:04:141
787099221,13cyclictest0-21swapper/010:57:350
787099221,13cyclictest0-21swapper/010:57:350
787099221,12cyclictest14675-21grep07:20:340
787099221,10cyclictest21659-21/usr/sbin/munin11:20:260
788799213,15cyclictest15354-21head08:35:313
788799211,4cyclictest1160-21kworker/u8:009:36:163
788299210,12cyclictest0-21swapper/211:44:542
788299210,11cyclictest0-21swapper/208:32:382
787699211,9cyclictest0-21swapper/109:34:191
787699211,15cyclictest0-21swapper/112:36:441
787699210,15cyclictest0-21swapper/109:15:051
787699210,15cyclictest0-21swapper/109:01:001
787699210,12cyclictest0-21swapper/110:58:171
787699210,12cyclictest0-21swapper/110:58:171
787699210,12cyclictest0-21swapper/108:05:301
787099216,4cyclictest167062wc07:25:300
787099214,3cyclictest0-21swapper/007:40:230
787099211,7cyclictest26344-21kworker/0:108:02:010
7870992115,4cyclictest18308-21cpuspeed_turbos08:45:180
7870992114,4cyclictest1416-40dnf11:45:330
788799209,3cyclictest28944-21diskmemload09:26:123
788799204,8cyclictest88250irq/135-enp2s0-08:30:153
788799204,8cyclictest86550irq/126-enp1s0-09:40:213
788799203,9cyclictest32365-21df_inode10:30:173
788799203,8cyclictest0-21swapper/308:17:233
788799203,7cyclictest86850irq/129-enp1s0-09:30:233
788799203,7cyclictest0-21swapper/312:10:593
788799201,9cyclictest0-21swapper/308:20:033
788799201,7cyclictest0-21swapper/307:45:233
788799201,4cyclictest0-21swapper/310:35:063
788299207,3cyclictest0-21swapper/211:19:102
788299203,3cyclictest32-21ksoftirqd/210:15:202
788299201,7cyclictest30089-21kworker/2:110:40:022
788299201,11cyclictest0-21swapper/212:35:332
787699201,11cyclictest0-21swapper/108:20:311
787099203,10cyclictest2790-21/usr/sbin/munin08:10:270
787099202,10cyclictest26354-21/usr/sbin/munin07:50:270
787099201,9cyclictest0-21swapper/010:07:440
787099201,6cyclictest0-21swapper/012:16:400
787099201,4cyclictest7036-21grep12:00:190
787099201,4cyclictest7036-21grep12:00:190
787099201,3cyclictest13558-21idleruntime-cro08:35:000
787099201,12cyclictest0-21swapper/009:50:290
787099201,11cyclictest9383-21/usr/sbin/munin08:25:310
787099201,10cyclictest18256-21needreboot11:10:250
787099201,10cyclictest13244-21fschecks_count11:00:220
788799196,2cyclictest0-21swapper/307:50:213
788799194,13cyclictest16670-21kworker/u8:011:21:293
787699198,3cyclictest13955-21cat08:35:171
787699191,17cyclictest0-21swapper/107:23:171
787099199,2cyclictest26280-21grep09:00:310
787099196,3cyclictest0-21swapper/010:26:390
787099194,12cyclictest11-21rcuc/009:20:350
787099191,6cyclictest784-21NetworkManager08:17:280
7870991912,4cyclictest18826-21/usr/sbin/munin12:25:270
787099191,16cyclictest0-21swapper/009:05:310
787099191,12cyclictest0-21swapper/012:11:580
787099191,10cyclictest8081-21/usr/sbin/munin07:10:220
787099191,10cyclictest15080-21cpu11:05:190
787099191,10cyclictest1171-21grep08:05:230
787099190,9cyclictest29153-21grep07:55:260
788799186,9cyclictest21953-21/usr/sbin/munin07:40:343
788799181,8cyclictest0-21swapper/308:45:133
787099185,3cyclictest0-21swapper/007:30:450
787099184,12cyclictest11-21rcuc/009:35:330
787099182,3cyclictest241132unixbench_singl08:55:330
787099182,13cyclictest6538-21memory09:30:250
787099181,9cyclictest823-21snmpd12:38:510
787099181,9cyclictest10048-21/usr/sbin/munin09:40:230
787099181,8cyclictest0-21swapper/011:55:310
787099181,7cyclictest0-21swapper/011:34:170
787099181,7cyclictest0-21swapper/010:48:180
787099181,7cyclictest0-21swapper/007:45:340
787099181,6cyclictest0-21swapper/009:47:060
787099181,6cyclictest0-21swapper/009:47:060
7870991812,3cyclictest10430-21munin-run10:55:000
787099180,10cyclictest17475-21kernelversion09:55:220
12832181,2sleep00-21swapper/010:30:580
787099176,3cyclictest86750irq/128-enp1s0-08:43:300
787099176,3cyclictest86750irq/128-enp1s0-08:43:300
7870991713,2cyclictest12483-21grep07:15:380
7870991712,3cyclictest404-21sed09:15:400
787099171,13cyclictest9698-21gltestperf12:05:230
787099171,13cyclictest30320-21pmu-power09:10:280
787099169,4cyclictest31709-21sensors_temp11:40:320
787099166,3cyclictest0-21swapper/012:20:420
787099161,6cyclictest19750-21/usr/sbin/munin07:35:160
787099161,6cyclictest19750-21/usr/sbin/munin07:35:160
787099161,5cyclictest0-21swapper/010:42:580
787099151,13cyclictest0-21swapper/008:50:560
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional