You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-21 - 08:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 100 highest latencies:
System rack5slot3s.osadl.org (updated Fri Mar 21, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34322241165,24sleep30-21swapper/319:06:313
35502238165,60sleep00-21swapper/019:08:010
36212236163,24sleep20-21swapper/219:08:552
36152235163,58sleep10-21swapper/119:08:511
1199121710,4sleep0379899cyclictest22:10:130
3808998885,2cyclictest0-21swapper/222:32:502
3808996865,2cyclictest0-21swapper/223:55:412
25182620,1sleep10-21swapper/123:15:011
295962610,1sleep30-21swapper/300:30:113
127832550,1sleep00-21swapper/019:25:280
127832550,1sleep00-21swapper/019:25:280
105672550,2sleep00-21swapper/019:20:250
84312510,1sleep30-21swapper/319:15:273
3808995148,2cyclictest0-21swapper/222:25:082
3813994114,9cyclictest30012-21/usr/sbin/munin23:00:243
3813994114,20cyclictest0-21swapper/321:45:013
3813994014,8cyclictest836-21runrttasks00:38:143
3813993915,11cyclictest21699-21diskmemload21:15:193
3813993912,11cyclictest15445-21sh19:35:013
379899389,13cyclictest10080-21munin-run20:40:010
3813993715,10cyclictest9881-21/usr/sbin/munin23:35:173
3813993713,10cyclictest25400-21ntp_states00:15:253
3813993611,8cyclictest11319-21grep22:10:093
3813993611,15cyclictest24844-21latency_hist20:00:033
3813993611,14cyclictest31675-21/usr/sbin/munin20:15:203
3813993611,12cyclictest22801-21/usr/sbin/munin22:40:213
381399359,13cyclictest7652-21kworker/3:119:25:153
381399359,13cyclictest7652-21kworker/3:119:25:153
3813993514,11cyclictest14634-21snmpd23:51:043
3813993513,9cyclictest28502-21munin-plugin-st00:25:023
3813993512,7cyclictest20540-21sendmail_mailst00:00:253
379899358,13cyclictest28295-21munin-run21:25:000
379899355,14cyclictest7458-21munin-run23:30:000
3813993414,10cyclictest6998-21cat23:25:023
3813993413,10cyclictest8601-21expr22:00:163
3813993412,10cyclictest21994-21cat00:10:023
128522340,4sleep2380899cyclictest20:45:132
381399339,12cyclictest27445-21taskset22:53:253
381399339,12cyclictest21699-21diskmemload23:58:213
381399339,10cyclictest0-21swapper/321:05:023
381399338,15cyclictest3997-21/usr/sbin/munin19:10:173
3813993314,15cyclictest4711-21cut20:25:193
3813993311,19cyclictest8149-21cut23:30:123
3813993311,11cyclictest836-21runrttasks22:16:283
3813993310,10cyclictest18400-21/usr/sbin/munin22:30:203
381399329,11cyclictest0-21swapper/319:40:203
3808993225,3cyclictest0-21swapper/200:13:342
379899324,13cyclictest32614-21awk23:10:010
3813993113,10cyclictest22993-21sed00:10:153
3813993111,7cyclictest6184-21/usr/sbin/munin21:55:183
3813993110,9cyclictest20951-21diskstats22:35:143
3813993110,9cyclictest20951-21diskstats22:35:143
3813993110,9cyclictest13955-1kworker/u9:122:26:563
3813993110,12cyclictest30787-21sed20:10:283
379899316,12cyclictest3138-21munin-run21:45:010
3798993112,16cyclictest29114-21munin-run00:30:000
381399309,9cyclictest0-21swapper/320:40:023
381399307,14cyclictest0-21swapper/323:20:023
381399307,10cyclictest0-21swapper/319:20:113
381399302,21cyclictest0-21swapper/321:29:483
3813993013,5cyclictest21699-21diskmemload23:10:213
3813993012,14cyclictest14634-21snmpd19:45:073
3813993010,16cyclictest5139-21pmu-power21:50:263
3813993010,12cyclictest20678-21/usr/sbin/munin19:50:233
381399300,12cyclictest0-21swapper/320:52:443
379899305,12cyclictest20468-21awk19:50:010
3798993012,15cyclictest3753-21munin-run21:50:020
381399299,9cyclictest13955-1kworker/u9:123:44:093
381399299,9cyclictest13955-1kworker/u9:123:44:093
381399299,18cyclictest0-21swapper/321:10:223
381399299,12cyclictest21699-21diskmemload00:25:553
381399298,11cyclictest0-21swapper/320:40:193
3813992913,13cyclictest25891-21awk20:00:193
3813992911,14cyclictest21699-21diskmemload22:22:373
3813992911,11cyclictest21699-21diskmemload23:09:113
381399288,12cyclictest0-21swapper/319:35:223
381399282,11cyclictest0-21swapper/323:27:213
3813992819,6cyclictest5919-21/usr/sbin/munin20:30:283
3813992819,6cyclictest16908-21/usr/sbin/munin20:55:193
3813992819,6cyclictest13533-21/usr/sbin/munin20:45:203
3813992818,7cyclictest27286-21/usr/sbin/munin20:05:183
3813992815,10cyclictest86550irq/126-enp1s0-21:35:183
3813992815,10cyclictest86550irq/126-enp1s0-21:35:183
3813992815,10cyclictest27813-21/usr/sbin/munin22:55:293
3813992810,11cyclictest21699-21diskmemload22:46:323
381399278,8cyclictest21699-21diskmemload21:46:013
3813992719,6cyclictest29133-21/usr/sbin/munin21:30:143
3813992719,5cyclictest26330-21/usr/sbin/munin21:20:253
381399269,8cyclictest21699-21diskmemload22:06:153
381399268,7cyclictest21699-21diskmemload23:46:543
3813992618,5cyclictest1500-21/usr/sbin/munin20:20:153
380399260,18cyclictest0-21swapper/100:15:121
3798992610,14cyclictest0-21swapper/022:00:010
380399251,20cyclictest12680-21gltestperf23:40:161
380399251,20cyclictest12680-21gltestperf23:40:161
293322251,2sleep10-21swapper/100:30:091
3813992414,7cyclictest21727-21wc21:10:023
3803992418,3cyclictest17898-21irqstats23:55:171
380399241,16cyclictest0-21swapper/100:05:021
380399240,17cyclictest0-21swapper/123:37:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional