You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-27 - 21:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Feb 27, 2024 12:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
834799622,14cyclictest131rcu_preempt08:10:182
834799572,46cyclictest0-21swapper/207:25:212
833299545,35cyclictest27131-21ssh11:25:221
8360995327,18cyclictest26466irq/58-eth011:45:203
8347995212,17cyclictest27-21ksoftirqd/210:50:192
834799512,35cyclictest3626-21ssh12:30:212
833299511,3cyclictest0-21swapper/108:35:211
833299492,4cyclictest20781-21apt-get08:30:191
833299482,41cyclictest0-21swapper/110:05:191
832099473,40cyclictest3008-21ssh11:30:230
833299462,6cyclictest0-21swapper/110:35:191
833299451,7cyclictest0-21swapper/109:20:191
833299451,4cyclictest0-21swapper/108:05:211
833299451,3cyclictest0-21swapper/108:45:261
833299451,3cyclictest0-21swapper/108:40:201
833299441,3cyclictest0-21swapper/109:10:201
833299441,3cyclictest0-21swapper/107:10:171
833299441,2cyclictest2985-21apt-get11:30:221
8347994315,17cyclictest131rcu_preempt07:45:242
833299432,31cyclictest10285-21ssh11:15:161
833299431,3cyclictest0-21swapper/111:40:221
834799427,17cyclictest131rcu_preempt10:29:572
833299421,3cyclictest0-21swapper/110:25:181
833299412,4cyclictest0-21swapper/112:10:201
833299411,4cyclictest868-21apt-get09:40:151
833299411,4cyclictest868-21apt-get09:40:141
833299411,4cyclictest0-21swapper/110:30:221
833299410,5cyclictest5099-21apt-get07:40:211
836099392,24cyclictest26466irq/58-eth009:25:203
834799383,18cyclictest10795-21sh09:50:142
834799382,19cyclictest737-21ssh11:08:582
833299381,9cyclictest19322-21if_eth010:40:241
833299381,3cyclictest0-21swapper/109:05:211
833299381,2cyclictest25476-21apt-get10:00:211
834799372,18cyclictest23386-21ssh12:00:582
833299371,4cyclictest0-21swapper/111:55:241
833299371,4cyclictest0-21swapper/111:45:191
833299371,3cyclictest2153-21apt-get11:50:221
833299371,3cyclictest2153-21apt-get11:50:221
833299371,3cyclictest0-21swapper/108:50:191
833299371,3cyclictest0-21swapper/108:15:201
833299371,2cyclictest9878-21apt-get07:45:251
833299370,4cyclictest15909-21apt-get08:25:191
833299370,3cyclictest12010-21ssh12:35:201
833299362,31cyclictest12354-21ssh10:11:041
833299361,3cyclictest10785-21apt-get09:50:161
833299361,3cyclictest0-21swapper/110:15:171
833299360,4cyclictest18331-21apt-get12:00:211
833299351,9cyclictest20976-21latency11:00:311
833299351,6cyclictest0-21swapper/108:55:221
833299351,5cyclictest2588-21ssh09:15:341
833299351,3cyclictest0-21swapper/107:55:221
833299351,3cyclictest0-21swapper/107:35:221
833299351,2cyclictest18708-21ssh11:20:231
833299350,9cyclictest22635-21ssh09:29:161
832099352,27cyclictest0-21swapper/008:25:180
8320993514,17cyclictest12-21ksoftirqd/012:00:210
8320993513,8cyclictest12-21ksoftirqd/009:50:140
8360993417,12cyclictest26466irq/58-eth011:40:213
833299343,28cyclictest0-21swapper/112:05:211
833299341,9cyclictest18125-21ssh12:18:361
833299341,5cyclictest19172-21/usr/sbin/munin07:50:391
833299341,3cyclictest18089-21apt-get09:55:231
833299341,2cyclictest1406-21apt-get08:10:171
833299340,5cyclictest17034-21rm11:37:481
8320993412,13cyclictest8308-21cyclictest07:40:210
8360993322,7cyclictest0-21swapper/310:55:233
834799333,20cyclictest24746irq/40-dwc2_hso11:55:252
834799332,18cyclictest27141-21ssh11:25:232
833299338,22cyclictest23474-21switchtime12:20:381
833299331,8cyclictest20594-21ntp_kernel_pll_09:00:341
833299331,7cyclictest3802-21ssh12:30:241
833299331,11cyclictest29298-21grep11:05:311
833299330,5cyclictest26387-21sed07:25:321
832099333,21cyclictest25489-21ssh10:00:200
834799324,18cyclictest24746irq/40-dwc2_hso09:00:212
833299321,9cyclictest1177-21/usr/sbin/munin10:50:311
833299321,8cyclictest29950-21/usr/sbin/munin10:45:341
833299321,8cyclictest23746-21/usr/sbin/munin09:30:241
833299321,5cyclictest18137-21/usr/sbin/munin07:20:321
833299321,3cyclictest0-21swapper/112:25:181
833299321,2cyclictest11115-21apt-get08:20:221
833299320,6cyclictest28562-21cut07:30:291
833299320,5cyclictest15529-21ssh10:57:551
832099323,21cyclictest14699-21ssh10:15:190
834799310,13cyclictest18100-21ssh09:55:252
833299311,9cyclictest27726-21ntp_states08:00:361
833299311,7cyclictest10556-21ssh09:50:001
833299311,3cyclictest13972-21irqstats07:15:291
8332993111,13cyclictest8308-21cyclictest10:20:221
8320993111,14cyclictest12-21ksoftirqd/010:50:200
836099304,15cyclictest26466irq/58-eth011:30:213
8360993012,14cyclictest0-21swapper/311:55:253
834799305,8cyclictest131rcu_preempt11:45:202
834799302,16cyclictest24746irq/40-dwc2_hso09:30:192
834799302,10cyclictest7903-21ssh11:51:572
834799302,10cyclictest7903-21ssh11:51:562
834799301,15cyclictest24746irq/40-dwc2_hso07:55:212
833299301,8cyclictest5582-21/usr/sbin/munin11:10:321
836099295,14cyclictest0-21swapper/309:55:233
834799293,17cyclictest24746irq/40-dwc2_hso08:15:192
834799292,17cyclictest24746irq/40-dwc2_hso09:05:212
833299291,7cyclictest31560-21rm09:37:051
832099292,20cyclictest16703-21ssh10:36:090
832099292,16cyclictest669-21ntpd08:30:210
836099288,12cyclictest0-21swapper/309:20:203
836099283,18cyclictest26466irq/58-eth012:15:193
834799286,16cyclictest0-21swapper/210:05:202
834799283,20cyclictest0-21swapper/211:15:572
834799283,20cyclictest0-21swapper/210:48:582
834799283,20cyclictest0-21swapper/210:40:562
834799283,16cyclictest24746irq/40-dwc2_hso12:20:222
834799283,16cyclictest24746irq/40-dwc2_hso10:00:202
834799282,11cyclictest0-21swapper/210:36:572
8347992811,10cyclictest27-21ksoftirqd/208:40:212
832099283,4cyclictest131rcu_preempt11:20:220
8320992818,7cyclictest0-21swapper/009:40:150
8320992818,7cyclictest0-21swapper/009:40:140
8320992817,3cyclictest9183-21ssh10:55:240
836099279,8cyclictest26466irq/58-eth011:15:163
836099272,18cyclictest0-21swapper/312:00:573
836099272,18cyclictest0-21swapper/311:06:113
834799273,19cyclictest0-21swapper/211:01:572
834799273,18cyclictest0-21swapper/210:56:102
834799272,20cyclictest0-21swapper/210:31:082
834799272,11cyclictest29873-1kworker/u9:109:40:162
834799272,11cyclictest29873-1kworker/u9:109:40:152
8347992711,13cyclictest0-21swapper/210:20:312
8347992710,11cyclictest0-21swapper/208:50:212
832099272,19cyclictest31234-21ssh11:06:120
836099263,15cyclictest0-21swapper/310:36:093
834799262,9cyclictest32477-21ssh12:06:562
834799262,19cyclictest261rcuc/211:10:572
834799261,14cyclictest8353-21sh09:20:212
833299261,5cyclictest8344-21sed07:10:021
832099262,6cyclictest0-21swapper/008:10:180
832099262,19cyclictest17111-21ssh11:58:580
836099252,17cyclictest0-21swapper/311:51:573
836099252,17cyclictest0-21swapper/311:51:563
834799256,16cyclictest14806-21kworker/2:009:25:182
834799254,17cyclictest24746irq/40-dwc2_hso07:50:202
834799252,17cyclictest3443-21munin-node12:35:222
834799252,16cyclictest0-21swapper/208:30:052
8347992511,11cyclictest2783-21df08:45:262
832099255,11cyclictest0-21swapper/008:20:240
832099254,18cyclictest15912-21ssh09:25:190
832099253,7cyclictest131rcu_preempt10:29:570
836099248,10cyclictest0-21swapper/309:10:203
834799246,12cyclictest131rcu_preempt08:35:292
834799244,15cyclictest27798-21ssh12:25:212
834799241,9cyclictest11225-21ssh12:15:182
832099242,15cyclictest12-21ksoftirqd/008:15:040
8320992413,8cyclictest135-21systemd-journal09:00:000
832099241,18cyclictest0-21swapper/010:10:260
836099234,12cyclictest0-21swapper/311:25:223
836099231,16cyclictest0-21swapper/310:25:193
834799233,17cyclictest3443-21munin-node10:10:272
834799233,12cyclictest24746irq/40-dwc2_hso08:20:222
834799232,6cyclictest0-21swapper/211:42:572
834799232,16cyclictest0-21swapper/210:16:042
834799231,17cyclictest18287-21grep07:20:232
836099225,9cyclictest0-21swapper/309:50:163
836099223,12cyclictest0-21swapper/310:51:113
834799229,4cyclictest0-21swapper/212:10:322
834799227,2cyclictest131rcu_preempt09:15:332
834799223,11cyclictest20475-21apt-key08:30:022
834799222,18cyclictest6939-21ssh11:30:382
834799222,15cyclictest0-21swapper/211:20:572
834799221,5cyclictest7260-21kworker/2:208:55:312
832099227,7cyclictest3589-21ssh12:30:180
836099219,9cyclictest0-21swapper/309:46:533
836099212,13cyclictest0-21swapper/310:30:303
836099212,10cyclictest0-21swapper/312:20:213
8360992113,4cyclictest0-21swapper/311:10:203
836099211,17cyclictest0-21swapper/309:00:193
834799217,6cyclictest0-21swapper/208:05:442
834799213,9cyclictest522-21latency_hist09:40:012
834799211,16cyclictest14390-21kthreadcore07:15:292
834799211,11cyclictest4752-21apt-get07:40:042
834799211,10cyclictest27766-21ps08:00:382
834799211,10cyclictest27083-21fschecks_time09:10:242
8320992113,5cyclictest0-21swapper/010:40:190
8320992112,7cyclictest15606-21sed12:35:340
8320992112,6cyclictest0-21swapper/011:50:390
8320992112,6cyclictest0-21swapper/011:50:380
8320992111,7cyclictest23433-21apt-get09:30:190
8320992111,5cyclictest0-21swapper/012:20:220
832099211,12cyclictest13099-21latency_hist07:15:020
836099203,10cyclictest0-21swapper/312:25:213
836099202,9cyclictest26466irq/58-eth011:00:193
836099202,14cyclictest0-21swapper/309:15:263
836099202,11cyclictest0-21swapper/307:55:223
834799207,10cyclictest8887-21sed07:10:242
834799203,14cyclictest29212-21kthreadcore07:30:312
834799202,10cyclictest32588-21/usr/sbin/munin07:35:232
834799201,10cyclictest3443-21munin-node11:35:202
832099202,12cyclictest10682-21ssh10:33:580
832099201,7cyclictest10699-21http12:15:010
8320992011,6cyclictest0-21swapper/011:15:190
832099201,14cyclictest26816-21apt-get11:45:190
832099201,10cyclictest29300-21sort08:35:370
8320992010,7cyclictest26098-21cat10:20:390
832099200,15cyclictest2886-21cut08:45:280
836099193,11cyclictest0-21swapper/311:20:583
836099193,11cyclictest0-21swapper/310:16:573
836099193,11cyclictest0-21swapper/310:09:563
836099192,8cyclictest0-21swapper/312:05:263
836099192,10cyclictest0-21swapper/308:50:203
8360991913,4cyclictest0-21swapper/310:40:193
8360991911,5cyclictest26466irq/58-eth012:35:223
8360991911,5cyclictest0-21swapper/310:20:343
836099191,13cyclictest0-21swapper/310:10:383
836099191,13cyclictest0-21swapper/309:40:193
836099191,13cyclictest0-21swapper/309:40:183
836099191,13cyclictest0-21swapper/307:40:013
834799191,7cyclictest821666ssh09:45:312
834799191,14cyclictest8413-21apt-key07:10:022
832099191,7cyclictest1965-21ssh11:10:190
8320991911,5cyclictest31048-21ssh09:10:370
832099191,15cyclictest18110-21ssh09:55:250
832099191,15cyclictest0-21swapper/008:00:240
8320991911,4cyclictest0-21swapper/009:15:370
832099191,10cyclictest32147-21kthreadcore08:40:290
8320991910,7cyclictest26350-21sshd11:44:580
836099189,6cyclictest0-21swapper/310:47:063
836099189,6cyclictest0-21swapper/309:38:083
836099188,7cyclictest0-21swapper/312:31:133
836099183,11cyclictest0-21swapper/312:12:573
836099182,11cyclictest0-21swapper/310:00:213
836099182,10cyclictest0-21swapper/309:05:223
836099182,10cyclictest0-21swapper/308:30:053
836099182,10cyclictest0-21swapper/307:45:263
836099182,10cyclictest0-21swapper/307:25:213
836099181,13cyclictest0-21swapper/308:00:403
832099183,10cyclictest0-21swapper/010:09:570
832099182,13cyclictest10689-21apt-get12:15:050
832099182,11cyclictest26392-21ssh10:45:230
832099181,9cyclictest10831-21tr07:45:320
832099181,13cyclictest4756-21munin-run07:40:010
832099181,12cyclictest7848-21apt-get09:20:040
832099181,11cyclictest27195-21cpuspeed11:25:220
8320991810,5cyclictest26511-21diskmemload11:00:200
836099172,10cyclictest0-21swapper/311:35:193
836099172,10cyclictest0-21swapper/309:30:203
836099172,10cyclictest0-21swapper/308:15:043
836099172,10cyclictest0-21swapper/308:10:193
836099172,10cyclictest0-21swapper/307:50:203
836099172,10cyclictest0-21swapper/307:20:213
836099172,10cyclictest0-21swapper/307:15:223
836099171,13cyclictest0-21swapper/307:40:283
836099171,10cyclictest0-21swapper/308:05:343
832099175,4cyclictest25647-21timerandwakeup09:05:390
832099172,12cyclictest669-21ntpd07:25:210
832099172,11cyclictest28478-21irqstats12:25:260
832099171,13cyclictest27811-21/usr/sbin/munin07:30:260
832099171,12cyclictest18503-21unixbench_singl07:50:380
832099171,12cyclictest11601-21irqstats11:35:250
8320991710,5cyclictest0-21swapper/007:20:270
832099170,9cyclictest29040-21/usr/sbin/munin08:05:310
8360991611,3cyclictest0-21swapper/308:45:253
832099169,5cyclictest23980-21vmstat07:55:400
832099168,6cyclictest26885-21df12:05:230
832099166,3cyclictest26511-21diskmemload09:36:120
832099165,8cyclictest8417-21apt-key07:10:010
832099162,9cyclictest0-21swapper/007:15:220
832099161,12cyclictest17901-21kthreadcore09:00:300
832099161,11cyclictest10254-21kthreadcore08:50:320
836099159,4cyclictest0-21swapper/308:20:453
836099152,9cyclictest0-21swapper/308:40:213
836099152,9cyclictest0-21swapper/307:10:053
832099151,8cyclictest8928-21ssh09:47:000
836099149,3cyclictest0-21swapper/308:35:363
836099149,3cyclictest0-21swapper/307:34:573
836099145,7cyclictest33-21ksoftirqd/308:25:243
836099121,8cyclictest0-21swapper/309:00:013
836099111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional