You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-12-09 - 21:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Mon Dec 09, 2024 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28437995917,20cyclictest131rcu_preempt12:05:121
2842699544,39cyclictest11656-21ssh10:15:120
2842699501,41cyclictest23712-21sendmail-msp12:00:130
2842699471,35cyclictest18817-21sendmail-msp10:20:140
2842699471,33cyclictest11603-21ssh11:35:170
2845499454,27cyclictest25866irq/58-eth011:45:143
2843799421,32cyclictest19093-21sendmail-msp11:00:151
2845499404,26cyclictest25866irq/58-eth010:30:133
2843799391,26cyclictest20727-21sendmail-msp10:00:121
2842699391,24cyclictest9645-21ssh12:10:130
28454993828,6cyclictest25866irq/58-eth010:05:133
2843799372,5cyclictest131rcu_preempt11:50:171
2843799372,5cyclictest131rcu_preempt11:50:161
2843799360,31cyclictest24817-21sendmail-msp07:40:131
2842699363,22cyclictest12-21ksoftirqd/010:45:120
2843799350,21cyclictest2840-21ssh10:30:131
2842699351,26cyclictest20113-21sendmail-msp11:40:140
2844599341,30cyclictest19053-21ssh10:40:132
2842699341,26cyclictest17492-21ssh12:35:120
2843799330,25cyclictest8500-21ssh12:30:081
2842699331,25cyclictest20727-21sendmail-msp10:00:140
2842699330,30cyclictest6211-21ssh11:50:170
2842699330,30cyclictest6211-21ssh11:50:160
2845499326,17cyclictest25866irq/58-eth011:15:163
2845499322,19cyclictest25866irq/58-eth009:45:133
2845499321,22cyclictest25866irq/58-eth011:50:173
2845499321,22cyclictest25866irq/58-eth011:50:163
28437993218,9cyclictest27282-21sendmail-msp12:20:131
28437993218,9cyclictest27282-21sendmail-msp12:20:121
2845499316,12cyclictest25866irq/58-eth008:05:133
2844599317,11cyclictest22146irq/40-dwc2_hso12:20:122
2844599317,11cyclictest22146irq/40-dwc2_hso12:20:122
2844599311,28cyclictest10939-21ssh10:35:132
2843799311,19cyclictest29626-21sendmail-msp09:20:131
28426993118,10cyclictest19093-21sendmail-msp11:00:160
28426993111,9cyclictest12-21ksoftirqd/009:45:120
2844599305,22cyclictest11799-21sendmail-msp08:00:132
28445993023,4cyclictest27-21ksoftirqd/212:00:142
28445993022,5cyclictest22146irq/40-dwc2_hso09:20:142
28445993021,7cyclictest27-21ksoftirqd/211:10:162
28445993021,7cyclictest27-21ksoftirqd/211:10:152
2842699302,7cyclictest131rcu_preempt07:30:120
2845499294,18cyclictest0-21swapper/309:55:133
2844599292,21cyclictest27599-21ssh11:05:112
2843799299,14cyclictest632-21Async17:44:201
2842699293,16cyclictest12-21ksoftirqd/009:55:110
2845499285,9cyclictest25866irq/58-eth011:30:103
2845499284,13cyclictest25866irq/58-eth012:30:103
2845499284,10cyclictest25866irq/58-eth010:35:123
28454992813,11cyclictest25866irq/58-eth009:15:143
2844599282,15cyclictest22146irq/40-dwc2_hso07:35:132
28437992820,5cyclictest21-21ksoftirqd/109:40:151
2843799281,20cyclictest18070-21sendmail-msp08:40:151
2842699281,18cyclictest0-21swapper/007:45:140
2842699281,17cyclictest12-21ksoftirqd/009:00:150
2842699280,22cyclictest10209-21ssh11:12:410
2842699280,22cyclictest10209-21ssh11:12:400
2845499273,16cyclictest0-21swapper/312:37:423
2845499273,15cyclictest0-21swapper/312:21:413
2845499273,15cyclictest0-21swapper/312:21:403
2845499272,18cyclictest0-21swapper/311:26:413
2844599277,10cyclictest27-21ksoftirqd/208:10:132
2844599275,16cyclictest22146irq/40-dwc2_hso11:15:142
2843799272,19cyclictest15302-21ssh12:11:301
2845499268,12cyclictest25866irq/58-eth008:00:123
2844599267,9cyclictest19093-21sendmail-msp11:00:312
2844599264,16cyclictest22146irq/40-dwc2_hso10:50:152
2844599264,16cyclictest22146irq/40-dwc2_hso10:50:142
2844599264,16cyclictest22146irq/40-dwc2_hso08:05:122
2844599263,19cyclictest22146irq/40-dwc2_hso10:55:142
2843799264,16cyclictest2487-21ssh11:30:101
28437992617,6cyclictest21-21ksoftirqd/108:20:121
2842699262,18cyclictest14614-21diskmemload12:22:420
2842699262,18cyclictest14614-21diskmemload12:22:410
2842699262,17cyclictest15768-21ssh09:35:130
2842699261,19cyclictest5530-21sendmail-msp07:20:130
2845499259,10cyclictest0-21swapper/311:39:403
2845499254,14cyclictest0-21swapper/311:10:153
2845499254,14cyclictest0-21swapper/311:10:153
2845499254,12cyclictest25866irq/58-eth011:55:163
2844599254,14cyclictest22146irq/40-dwc2_hso07:55:132
2844599253,17cyclictest22146irq/40-dwc2_hso07:30:142
2844599252,9cyclictest925066ntp_states07:20:282
2844599252,14cyclictest29521-21ssh11:45:122
2844599252,13cyclictest24150-21ssh12:37:412
28445992514,8cyclictest27303-21sh10:25:142
28445992513,4cyclictest20553-21/usr/sbin/munin11:40:302
2844599251,21cyclictest15155-21ssh11:55:172
2844599251,19cyclictest900-21snmpd08:45:152
2844599251,15cyclictest5771-21ssh09:50:182
2844599251,15cyclictest0-21swapper/212:30:112
28437992515,5cyclictest0-21swapper/110:05:121
2843799250,19cyclictest3925-21ssh10:10:131
2842699252,11cyclictest15017-21ssh09:10:180
2845499246,9cyclictest0-21swapper/312:10:143
2845499243,17cyclictest0-21swapper/310:46:183
2845499242,16cyclictest0-21swapper/312:00:133
2844599247,9cyclictest20195-21sed11:40:012
2844599243,9cyclictest0-21swapper/212:13:422
28445992412,10cyclictest1485-21ssh10:06:082
2844599241,19cyclictest25286-21kthreadcore09:40:252
28445992411,5cyclictest0-21swapper/211:30:142
2844599241,13cyclictest27488-21latency_hist12:20:022
2843799241,4cyclictest131rcu_preempt11:55:281
28437992414,7cyclictest10943-21ssh10:35:121
2843799241,19cyclictest0-21swapper/109:45:131
2842699240,19cyclictest10856-21ssh11:34:400
28454992317,4cyclictest321rcuc/312:29:493
2845499231,14cyclictest0-21swapper/312:05:123
2844599239,10cyclictest0-21swapper/209:35:112
2844599234,4cyclictest131rcu_preempt08:15:182
2844599233,16cyclictest22146irq/40-dwc2_hso12:05:122
2844599233,14cyclictest22146irq/40-dwc2_hso07:25:132
2844599233,10cyclictest20727-21sendmail-msp10:00:192
2844599232,9cyclictest5408-21ssh12:26:412
2844599232,16cyclictest27-21ksoftirqd/210:20:152
28445992316,5cyclictest6177-21ssh11:50:152
28445992316,5cyclictest6177-21ssh11:50:152
2844599231,14cyclictest4003-21grep08:25:172
2844599231,13cyclictest7918-21find09:30:082
28437992313,7cyclictest7375-21sshd09:29:401
2842699231,14cyclictest29626-21sendmail-msp09:20:290
2845499223,13cyclictest0-21swapper/310:25:123
2845499222,13cyclictest25866irq/58-eth007:40:133
28454992214,6cyclictest0-21swapper/311:20:163
28454992210,8cyclictest25866irq/58-eth010:15:133
2844599226,13cyclictest27-21ksoftirqd/211:20:032
2844599224,4cyclictest131rcu_preempt07:55:022
2844599222,8cyclictest131rcu_preempt08:50:142
2844599222,8cyclictest131rcu_preempt08:50:132
2844599222,16cyclictest0-21swapper/209:00:132
2844599221,9cyclictest131rcu_preempt11:25:392
28445992211,8cyclictest0-21swapper/208:40:302
2844599221,17cyclictest1324-21grep07:45:292
2844599221,16cyclictest0-21swapper/209:55:122
28437992213,6cyclictest21-21ksoftirqd/107:55:121
2842699222,5cyclictest52246usb-storage09:40:220
2842699222,16cyclictest11314-21apt-get10:55:130
28426992214,5cyclictest15149-21ssh11:55:170
2842699221,2cyclictest131rcu_preempt07:50:120
2842699221,14cyclictest7942-21ssh09:30:100
28454992116,3cyclictest321rcuc/309:27:313
28454992110,6cyclictest0-21swapper/310:40:313
2844599219,9cyclictest24330-21kthreadcore09:15:212
2844599212,3cyclictest131rcu_preempt09:27:102
2844599212,15cyclictest0-21swapper/210:48:402
28445992113,3cyclictest0-21swapper/208:20:102
28445992111,8cyclictest0-21swapper/209:12:502
2844599211,17cyclictest10699-21grep10:35:012
2844599211,11cyclictest5382-21munin-run09:50:012
2844599211,10cyclictest28455-21cat07:10:012
2844599211,10cyclictest11474-21latency_hist10:15:022
2843799212,6cyclictest14614-21diskmemload09:34:111
2843799211,6cyclictest131rcu_preempt11:40:141
28437992112,7cyclictest11799-21sendmail-msp08:00:181
28437992112,6cyclictest21133-21sendmail-msp11:20:361
28437992111,7cyclictest23712-21sendmail-msp12:00:281
2842699211,8cyclictest13985-21grep08:35:170
2842699211,7cyclictest52246usb-storage10:40:190
2842699211,5cyclictest0-21swapper/009:05:170
28426992113,5cyclictest0-21swapper/008:05:260
2842699211,16cyclictest21133-21sendmail-msp11:20:160
28426992110,4cyclictest10944-21sed07:25:180
28454992015,3cyclictest321rcuc/310:56:083
28454992014,4cyclictest321rcuc/309:31:513
28454992010,7cyclictest0-21swapper/310:10:273
2844599203,13cyclictest22146irq/40-dwc2_hso07:15:122
2844599202,9cyclictest8775-21fschecks_count08:30:192
2844599202,10cyclictest13202-21kthreadcore09:05:232
2844599201,9cyclictest26913-21kthreadcore07:40:252
2844599201,9cyclictest18050-21munin-run08:40:022
2844599201,10cyclictest18794-21sh10:20:012
2843799202,13cyclictest21-21ksoftirqd/107:30:121
2843799201,9cyclictest131rcu_preempt12:15:211
2843799201,5cyclictest52246usb-storage10:40:071
28437992012,6cyclictest15182-21sshd07:28:391
28437992011,6cyclictest0-21swapper/108:30:341
28437992011,6cyclictest0-21swapper/107:35:181
2842699208,8cyclictest27282-21sendmail-msp12:20:030
2842699204,8cyclictest131rcu_preempt08:20:290
2842699203,3cyclictest131rcu_preempt08:10:290
2842699203,15cyclictest12-21ksoftirqd/010:10:120
2842699201,5cyclictest52246usb-storage08:40:230
2842699201,5cyclictest0-21swapper/009:27:210
2842699201,15cyclictest28730-21cut07:10:160
2842699201,15cyclictest12570-21ssh11:15:150
2842699201,10cyclictest4916-21munin-run09:00:010
2842699201,10cyclictest4916-21munin-run09:00:000
2845499193,11cyclictest0-21swapper/310:51:193
2845499193,11cyclictest0-21swapper/310:51:183
28454991913,4cyclictest321rcuc/310:04:593
28454991913,4cyclictest0-21swapper/307:15:153
2845499191,13cyclictest0-21swapper/308:40:183
2845499191,12cyclictest0-21swapper/308:55:253
2845499191,12cyclictest0-21swapper/308:55:243
2845499191,12cyclictest0-21swapper/308:25:023
2845499191,10cyclictest25866irq/58-eth012:15:143
28454991910,6cyclictest25866irq/58-eth009:50:173
28454991910,6cyclictest0-21swapper/311:41:263
28454991910,6cyclictest0-21swapper/309:20:403
2844599191,15cyclictest28702-21grep07:10:162
2844599191,10cyclictest1271-21kthreadcore08:55:212
2844599191,10cyclictest1271-21kthreadcore08:55:202
2843799192,7cyclictest9548irq/52-mmc107:55:011
2843799191,9cyclictest0-21swapper/111:37:511
28437991913,4cyclictest4910-21sh09:00:001
28437991913,4cyclictest4910-21sh09:00:001
28437991912,4cyclictest0-21swapper/107:20:151
2843799191,16cyclictest18122-21ls08:40:011
28437991911,6cyclictest1-21systemd10:16:141
2843799191,15cyclictest22572-21ssh09:15:151
28437991911,4cyclictest18040-21irqrtprio12:35:181
2842699191,8cyclictest0-21swapper/010:27:560
2842699191,4cyclictest0-21swapper/012:30:010
2842699191,4cyclictest0-21swapper/010:09:170
2842699191,3cyclictest131rcu_preempt08:25:140
2842699191,15cyclictest29512-21apt-get11:45:120
2842699191,10cyclictest0-21swapper/010:31:100
2845499189,6cyclictest25866irq/58-eth011:00:213
2845499182,9cyclictest0-21swapper/309:40:363
2845499182,10cyclictest0-21swapper/308:45:143
28454991813,3cyclictest0-21swapper/307:45:273
28454991812,4cyclictest0-21swapper/309:39:583
28454991812,4cyclictest0-21swapper/307:55:223
2845499181,12cyclictest0-21swapper/307:35:133
2843799189,7cyclictest2071-21cron07:50:011
2843799189,6cyclictest0-21swapper/111:05:101
2843799186,2cyclictest93-21pr/ttyS011:30:021
2843799182,13cyclictest11335-21ssh10:55:151
2843799181,8cyclictest17801-21kthreadcore09:35:221
2843799181,2cyclictest131rcu_preempt11:10:311
2843799181,2cyclictest131rcu_preempt11:10:301
28437991812,4cyclictest0-21swapper/108:05:171
2843799181,15cyclictest4937-21sendmail-msp09:00:181
28437991810,5cyclictest0-21swapper/107:10:281
2843799180,11cyclictest5721-21apt-key11:50:011
2842699189,7cyclictest11799-21sendmail-msp08:00:160
2842699186,2cyclictest131rcu_preempt08:30:220
2842699185,4cyclictest131rcu_preempt11:05:200
2842699181,8cyclictest131rcu_preempt10:50:210
2842699181,8cyclictest131rcu_preempt10:50:210
2842699181,7cyclictest7614-21sed07:55:190
2842699181,12cyclictest31788-21ssh11:26:400
28426991810,6cyclictest27695-21sshd08:46:050
28426991810,4cyclictest12-21ksoftirqd/009:15:140
2842699180,6cyclictest27891-21kthreadcore07:40:240
2842699180,15cyclictest14982-21grep10:35:290
2845499173,9cyclictest0-21swapper/310:20:143
28454991712,3cyclictest0-21swapper/309:11:253
28454991712,3cyclictest0-21swapper/308:35:273
28454991710,4cyclictest0-21swapper/311:05:143
2843799178,7cyclictest1006-21diskstats07:15:161
2843799177,8cyclictest18817-21sendmail-msp10:20:221
2843799175,3cyclictest131rcu_preempt09:10:191
2843799173,12cyclictest14614-21diskmemload10:25:111
2843799171,9cyclictest0-21swapper/112:25:391
2843799171,7cyclictest13734-21tr09:55:171
28437991711,4cyclictest8611-21cat08:30:021
28437991711,4cyclictest5994-21cut10:50:241
28437991711,4cyclictest5994-21cut10:50:231
28437991710,5cyclictest27470-21idleruntime10:45:151
2842699177,8cyclictest5760-21diskmemload09:50:160
2842699175,9cyclictest11100-21tr12:30:170
2842699174,4cyclictest3933-21kthreadcore12:05:220
2842699172,11cyclictest20213-21apt-get07:35:140
2842699171,4cyclictest0-21swapper/007:15:250
2842699170,8cyclictest131rcu_preempt08:55:020
2842699170,8cyclictest131rcu_preempt08:55:010
2845499162,10cyclictest0-21swapper/307:50:143
2845499161,9cyclictest0-21swapper/308:10:123
28454991611,3cyclictest0-21swapper/309:05:263
28454991611,3cyclictest0-21swapper/308:15:183
28454991611,3cyclictest0-21swapper/307:25:293
28454991610,4cyclictest321rcuc/308:50:143
28454991610,4cyclictest321rcuc/308:50:133
28454991610,4cyclictest0-21swapper/308:25:233
2843799168,6cyclictest28122-21cut08:50:161
2843799168,6cyclictest28122-21cut08:50:161
2843799168,5cyclictest167-21systemd-network09:50:051
2843799166,7cyclictest12558-21apt-get11:15:161
2843799165,3cyclictest131rcu_preempt08:45:131
2842699161,4cyclictest0-21swapper/008:16:280
2845499157,6cyclictest0-21swapper/307:21:533
28454991511,3cyclictest321rcuc/309:01:013
28454991510,3cyclictest0-21swapper/308:30:223
28454991510,3cyclictest0-21swapper/307:30:173
28454991510,3cyclictest0-21swapper/307:10:253
2843799158,4cyclictest0-21swapper/108:10:191
2843799151,11cyclictest30401-21grep08:15:301
2843799133,8cyclictest28519-21cut07:10:021
2842699131,9cyclictest28417-21cyclictest07:10:000
284549991,5cyclictest25866irq/58-eth007:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional