You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri May 09, 2025 00:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28695996235,19cyclictest24882-21sendmail-msp00:00:181
28710995745,7cyclictest12004-21sh00:35:152
28710995745,7cyclictest12004-21sh00:35:152
28710995423,20cyclictest27-21ksoftirqd/200:10:142
2869599545,38cyclictest20481-21ssh22:35:151
28710995210,26cyclictest27-21ksoftirqd/219:20:102
2869499515,41cyclictest405-21ssh00:05:120
28710995033,15cyclictest27-21ksoftirqd/220:15:132
28710995033,15cyclictest27-21ksoftirqd/220:15:122
2869599481,8cyclictest131rcu_preempt23:15:111
2869499481,39cyclictest27035-21sendmail-msp23:00:150
28695994734,9cyclictest30124-21sendmail-msp21:40:151
2871099461,37cyclictest14537-21ssh21:30:132
2869599463,8cyclictest131rcu_preempt00:20:141
2869599463,32cyclictest5850-21sendmail-msp19:20:101
2869599462,35cyclictest29514-21sendmail-msp22:00:171
2869599462,35cyclictest29514-21sendmail-msp22:00:161
2871099452,35cyclictest17745-21sendmail-msp20:40:122
28694994416,21cyclictest25571-21sendmail-msp23:40:190
2869599414,26cyclictest30709-21sendmail-msp21:20:141
28695994123,8cyclictest21-21ksoftirqd/120:10:131
2869499411,31cyclictest11800-21sendmail-msp20:00:140
2872399391,25cyclictest25866irq/58-eth021:30:143
2869499390,31cyclictest9172-21ssh23:50:120
2871099383,25cyclictest18010-21ssh23:35:132
2871099382,18cyclictest25571-21sendmail-msp23:41:192
2871099380,29cyclictest0-21swapper/219:10:122
28695993822,8cyclictest28321-21ssh22:40:131
28694993817,16cyclictest12-21ksoftirqd/023:55:150
28694993817,16cyclictest12-21ksoftirqd/023:55:140
28694993813,18cyclictest12-21ksoftirqd/021:10:160
28694993813,18cyclictest0-21swapper/023:05:150
2869599372,26cyclictest21-21ksoftirqd/120:45:161
2871099362,27cyclictest29034-21ssh21:59:182
2871099362,17cyclictest6223-21/usr/sbin/munin21:45:192
2871099362,11cyclictest601-21rngd22:15:192
2871099362,11cyclictest601-21rngd22:15:182
2869499361,27cyclictest30979-21sendmail-msp20:20:110
2869499361,27cyclictest30979-21sendmail-msp20:20:100
2872399351,29cyclictest0-21swapper/323:25:183
28710993519,11cyclictest27-21ksoftirqd/220:50:102
28710993512,13cyclictest0-21swapper/222:20:132
2869499357,17cyclictest601-21rngd19:10:130
2871099346,16cyclictest22146irq/40-dwc2_hso21:50:162
2871099344,17cyclictest22146irq/40-dwc2_hso20:55:152
28710993412,16cyclictest28693-21cyclictest23:30:152
2869599343,19cyclictest6656-21ssh21:25:181
2869599342,24cyclictest25571-21sendmail-msp23:40:141
28710993318,13cyclictest0-21swapper/223:15:122
2872399321,15cyclictest25866irq/58-eth023:15:123
2869499324,18cyclictest22970-21ssh21:15:130
2872399312,16cyclictest25866irq/58-eth023:35:143
2871099316,8cyclictest28693-21cyclictest23:10:152
28710993116,9cyclictest28693-21cyclictest19:25:142
2869599313,21cyclictest21-21ksoftirqd/122:25:131
2869499313,17cyclictest25036-21sendmail-msp19:40:130
2869499312,22cyclictest29709-21ssh22:20:190
28694993117,12cyclictest26630-21ssh23:20:190
28694993117,12cyclictest26630-21ssh23:20:180
2871099303,16cyclictest22146irq/40-dwc2_hso22:55:132
2871099302,11cyclictest27035-21sendmail-msp23:01:202
2871099301,22cyclictest27-21ksoftirqd/222:50:192
28695993022,4cyclictest131rcu_preempt20:25:141
28695993022,4cyclictest131rcu_preempt20:25:131
2869599301,20cyclictest21-21ksoftirqd/119:50:111
2869499305,8cyclictest24882-21sendmail-msp00:00:350
28694993015,8cyclictest601-21rngd00:25:140
2872399293,16cyclictest25866irq/58-eth021:45:143
28723992917,9cyclictest25866irq/58-eth000:30:133
28710992915,10cyclictest22459-21sendmail-msp00:20:142
2871099291,16cyclictest26247-21sendmail-msp23:20:202
2871099291,16cyclictest26247-21sendmail-msp23:20:192
2869599292,21cyclictest26635-21ssh23:20:191
2869599292,21cyclictest26635-21ssh23:20:181
2869499293,13cyclictest23552-21uptime22:55:310
2869499292,20cyclictest4723-21sendmail-msp21:00:170
2872399281,20cyclictest0-21swapper/322:20:203
2871099282,20cyclictest0-21swapper/220:10:152
2869599282,22cyclictest21-21ksoftirqd/122:15:141
2869599282,22cyclictest21-21ksoftirqd/122:15:141
2869599282,18cyclictest30277-21ssh00:25:161
2869499285,9cyclictest22459-21sendmail-msp00:21:310
2869499285,8cyclictest601-21rngd22:25:190
2869499285,11cyclictest8323-21kthreadcore00:30:230
2869499282,21cyclictest13785-21sed22:10:180
2869499282,18cyclictest6666-21ssh21:25:200
28694992820,3cyclictest29514-21sendmail-msp22:00:160
28694992820,3cyclictest29514-21sendmail-msp22:00:150
28694992812,12cyclictest12-21ksoftirqd/020:45:160
2872399272,17cyclictest25866irq/58-eth019:30:153
2872399271,14cyclictest25866irq/58-eth022:30:143
2871099273,9cyclictest0-21swapper/222:27:182
2871099273,19cyclictest0-21swapper/200:18:212
2871099272,19cyclictest22146irq/40-dwc2_hso19:35:142
28710992716,6cyclictest5265-21ssh22:05:152
2871099271,15cyclictest29514-21sendmail-msp22:00:322
2871099271,15cyclictest29514-21sendmail-msp22:00:322
2869599274,10cyclictest131rcu_preempt22:30:131
2869599270,3cyclictest131rcu_preempt19:25:131
2869499276,8cyclictest6170-21/usr/sbin/munin19:20:130
2869499276,8cyclictest18660-21latency_hist00:40:030
2869499276,8cyclictest18660-21latency_hist00:40:030
2869499276,7cyclictest601-21rngd22:08:590
28694992720,4cyclictest27686-21apt-get20:50:130
2869499271,19cyclictest1341-21ssh23:45:160
2872399263,18cyclictest0-21swapper/323:44:193
2871099262,14cyclictest30709-21sendmail-msp21:20:182
2869599262,7cyclictest131rcu_preempt00:19:211
2869599262,19cyclictest13782-21irqrtprio22:10:181
2869499266,7cyclictest27973-21sendmail-msp22:40:530
2869499262,18cyclictest12339-21ssh22:30:140
28694992619,3cyclictest601-21rngd22:15:150
28694992619,3cyclictest601-21rngd22:15:140
28694992618,4cyclictest601-21rngd19:50:120
28694992610,12cyclictest0-21swapper/020:25:130
28694992610,12cyclictest0-21swapper/020:25:130
2872399258,4cyclictest0-21swapper/320:10:143
2872399255,13cyclictest0-21swapper/300:20:143
2872399253,15cyclictest25866irq/58-eth000:10:153
2872399253,13cyclictest0-21swapper/322:15:193
2872399253,13cyclictest0-21swapper/322:15:183
2872399252,18cyclictest0-21swapper/321:58:183
2872399252,17cyclictest0-21swapper/300:17:203
2871099258,14cyclictest15263-21ssh22:30:222
2871099252,10cyclictest27-21ksoftirqd/222:40:152
28710992518,4cyclictest30979-21sendmail-msp20:20:162
28710992518,4cyclictest30979-21sendmail-msp20:20:152
2871099251,17cyclictest22146irq/40-dwc2_hso22:45:132
28695992512,9cyclictest28665-21sendmail-msp22:20:291
2869499255,7cyclictest30709-21sendmail-msp21:20:150
2869499254,6cyclictest601-21rngd22:37:370
2869499254,14cyclictest22461-21ssh21:35:140
2869499253,10cyclictest18524-21kthreadcore00:15:260
28694992518,4cyclictest14530-21apt-get21:30:140
2872399249,11cyclictest0-21swapper/323:24:193
2872399249,11cyclictest0-21swapper/323:24:193
2871099242,14cyclictest22146irq/40-dwc2_hso20:05:142
2871099241,8cyclictest601-21rngd00:01:192
2871099241,18cyclictest19630-21ssh22:13:182
2871099241,17cyclictest601-21rngd23:55:152
2871099241,17cyclictest601-21rngd23:55:142
2871099241,16cyclictest6650-21sh21:25:182
2869599245,2cyclictest131rcu_preempt23:30:041
2869499249,3cyclictest131rcu_preempt20:20:010
2869499249,3cyclictest131rcu_preempt20:20:010
2869499244,8cyclictest18634-21kthreadcore20:40:220
2869499244,8cyclictest17045-21ssh22:51:420
2869499244,8cyclictest14326-21diskmemload23:25:230
2869499243,8cyclictest14326-21diskmemload21:45:290
2869499242,16cyclictest845-21munin-node20:55:160
2869499242,11cyclictest710-21rs:main0
28694992418,3cyclictest16856-21apt-get20:05:140
28694992412,10cyclictest18650-21apt-get23:15:120
2872399233,13cyclictest0-21swapper/322:50:193
2872399232,14cyclictest25866irq/58-eth021:25:193
28723992311,5cyclictest25966irq/59-eth000:05:133
2871099232,6cyclictest0-21swapper/223:25:192
2871099232,15cyclictest22146irq/40-dwc2_hso21:10:162
28710992315,5cyclictest7265-21ssh23:05:352
28710992311,9cyclictest27340-21ssh21:35:412
28710992310,10cyclictest20227-21sshd19:34:112
2869599239,10cyclictest27035-21sendmail-msp23:00:331
2869599232,13cyclictest601-21rngd21:59:191
2869499233,11cyclictest601-21rngd23:30:190
2872399229,9cyclictest0-21swapper/321:50:183
2872399226,13cyclictest0-21swapper/323:50:133
2872399225,13cyclictest25866irq/58-eth020:25:153
2872399225,13cyclictest25866irq/58-eth020:25:153
2872399223,13cyclictest0-21swapper/322:10:183
2872399223,12cyclictest0-21swapper/322:06:203
28723992212,7cyclictest0-21swapper/322:35:173
2871099228,7cyclictest11800-21sendmail-msp20:00:202
2871099222,16cyclictest25523-21kthreadcore20:45:252
2871099222,15cyclictest0-21swapper/200:09:192
28710992211,9cyclictest15033-21ssh23:52:112
2871099221,13cyclictest5168-21cut21:00:212
2869599223,8cyclictest131rcu_preempt19:35:131
2869599222,15cyclictest3744-21ssh22:45:161
2869599221,6cyclictest25619-21latency_hist23:40:021
28695992212,7cyclictest10030-21sed21:45:311
2869499229,9cyclictest0-21swapper/021:05:130
28694992211,8cyclictest20469-21apt-get19:35:150
2869499221,17cyclictest13319-21cut20:35:140
2869499221,17cyclictest13319-21cut20:35:140
28694992210,9cyclictest7773-21sed19:55:180
28694992210,9cyclictest15208-21cut21:50:240
28694992210,8cyclictest15881-21fschecks_time19:30:180
2872399214,14cyclictest0-21swapper/320:15:123
2872399214,14cyclictest0-21swapper/320:15:123
2872399213,12cyclictest0-21swapper/322:00:163
2872399213,12cyclictest0-21swapper/322:00:153
2872399212,13cyclictest0-21swapper/323:00:133
2872399212,13cyclictest0-21swapper/321:35:153
28723992111,4cyclictest0-21swapper/323:05:353
2871099219,5cyclictest16524-21needreboot20:35:232
2871099219,5cyclictest16524-21needreboot20:35:222
2871099219,4cyclictest14330-21sh21:10:012
2871099217,11cyclictest30124-21sendmail-msp21:40:072
2871099213,10cyclictest12932-21latency_hist20:35:022
2871099211,6cyclictest3617-21/usr/sbin/munin20:25:172
2871099211,6cyclictest3617-21/usr/sbin/munin20:25:162
2871099211,15cyclictest5815-21cron19:20:022
2869599216,9cyclictest211ksoftirqd/120:50:121
2869599211,5cyclictest131rcu_preempt21:50:171
28695992112,6cyclictest131rcu_preempt23:05:151
2869499216,11cyclictest17286-21sshd23:13:190
2869499214,9cyclictest1012166ssh22:48:230
2869499212,6cyclictest601-21rngd23:38:380
28694992114,3cyclictest30111-21apt-get19:45:130
28694992111,8cyclictest7942-21apt-get00:10:140
2872399209,6cyclictest0-21swapper/323:45:163
2872399202,12cyclictest0-21swapper/300:00:173
2872399201,13cyclictest0-21swapper/320:50:213
28710992015,3cyclictest27-21ksoftirqd/222:35:042
28710992014,4cyclictest0-21swapper/219:45:242
2871099201,10cyclictest1-21systemd00:30:222
2871099200,10cyclictest30709-21sendmail-msp21:20:012
2869599206,3cyclictest131rcu_preempt00:15:021
28695992010,8cyclictest11800-21sendmail-msp20:00:221
2869499208,9cyclictest11763-21netstat20:30:250
2869499207,9cyclictest11605-21sed19:25:200
2869499206,11cyclictest26877-21ssh21:55:400
2872399199,7cyclictest0-21swapper/322:40:533
2872399193,9cyclictest0-21swapper/320:00:153
2872399193,11cyclictest0-21swapper/322:46:183
2872399193,11cyclictest0-21swapper/321:44:183
2872399192,11cyclictest0-21swapper/320:35:123
2872399192,11cyclictest0-21swapper/320:35:123
2872399191,13cyclictest0-21swapper/320:35:013
28723991910,6cyclictest0-21swapper/322:29:023
28723991910,6cyclictest0-21swapper/321:21:493
2871099193,8cyclictest4939-21latency_hist00:30:022
2871099191,9cyclictest4084-21cut19:50:192
2871099191,15cyclictest1543-21ssh23:45:172
2871099191,14cyclictest11830-21latency_hist20:00:012
2869599192,4cyclictest52246usb-storage20:20:481
2869599192,4cyclictest52246usb-storage20:20:471
2869599191,15cyclictest25519-21kthreadcore21:15:231
2869599191,14cyclictest201rcuc/100:30:131
2869599191,14cyclictest14721-21apt-get21:10:141
2869599191,11cyclictest601-21rngd23:25:161
28695991910,7cyclictest17745-21sendmail-msp20:40:291
2869599190,10cyclictest601-21rngd23:10:141
2869499196,10cyclictest2376-21kthreadcore19:15:200
28694991911,6cyclictest21599-21apt-get20:10:140
2872399182,11cyclictest0-21swapper/323:14:193
2872399182,10cyclictest0-21swapper/323:55:053
2872399182,10cyclictest0-21swapper/323:55:043
2872399182,10cyclictest0-21swapper/321:00:173
2872399182,10cyclictest0-21swapper/320:55:043
2872399182,10cyclictest0-21swapper/319:15:143
2872399181,10cyclictest0-21swapper/319:45:033
28723991810,6cyclictest0-21swapper/300:39:063
28723991810,6cyclictest0-21swapper/300:39:063
28723991810,4cyclictest0-21swapper/321:15:143
2871099185,10cyclictest25036-21sendmail-msp19:40:062
2869599187,8cyclictest25036-21sendmail-msp19:40:201
2869599185,11cyclictest601-21rngd20:15:201
2869599185,11cyclictest601-21rngd20:15:191
2869599182,12cyclictest17475-21cut21:30:241
28695991813,3cyclictest0-21swapper/100:35:351
28695991813,3cyclictest0-21swapper/100:35:351
28695991811,4cyclictest29406-21irqrtprio19:10:171
2869599181,12cyclictest16822-21http23:55:001
2869599181,11cyclictest601-21rngd22:50:141
28694991812,3cyclictest601-21rngd19:10:010
2872399179,6cyclictest0-21swapper/322:58:213
2872399179,6cyclictest0-21swapper/321:10:093
2872399179,6cyclictest0-21swapper/300:25:143
2872399173,10cyclictest0-21swapper/319:45:133
2872399172,11cyclictest0-21swapper/320:45:163
2872399172,10cyclictest0-21swapper/320:40:133
2872399172,10cyclictest0-21swapper/320:20:123
2872399172,10cyclictest0-21swapper/320:20:123
2872399172,10cyclictest0-21swapper/320:05:153
2872399172,10cyclictest0-21swapper/319:35:153
28723991711,4cyclictest0-21swapper/323:30:023
2871099171,8cyclictest28741-21cat19:10:022
2869599172,11cyclictest601-21rngd00:09:181
2869599172,11cyclictest11776-21ssh22:08:191
2869599171,9cyclictest601-21rngd19:20:011
2872399161,10cyclictest0-21swapper/319:55:163
2869599163,3cyclictest131rcu_preempt19:55:231
2869599162,11cyclictest2008-21kthreadcore20:55:231
2869599162,10cyclictest16808-21apt-get23:55:031
2869599162,10cyclictest16808-21apt-get23:55:031
2869599161,5cyclictest601-21rngd23:45:231
2869599161,5cyclictest601-21rngd22:55:191
2869599161,4cyclictest601-21rngd20:05:221
2869599161,3cyclictest601-21rngd21:00:171
2869599161,11cyclictest14331-21latency_hist21:10:031
28695991610,4cyclictest23151-21sh21:35:201
28695991610,4cyclictest1138-21ntp_states19:45:281
2872399151,9cyclictest0-21swapper/319:20:103
2872399151,10cyclictest0-21swapper/319:10:193
2869599152,10cyclictest17794-21latency_hist20:40:031
2869599152,10cyclictest17794-21latency_hist20:40:021
2869599151,11cyclictest8549-21fschecks_time20:30:171
2869599151,10cyclictest601-21rngd19:30:151
2872399141,7cyclictest0-21swapper/319:25:173
2872399131,5cyclictest0-21swapper/319:50:183
2869599131,9cyclictest28822-21latency_hist19:10:011
2872399122,6cyclictest0-21swapper/321:05:143
2872399101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional