You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Fri May 02, 2025 00:45:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
791199711,8cyclictest131rcu_preempt19:35:122
7911995516,18cyclictest27-21ksoftirqd/221:10:162
789999542,47cyclictest16228-21sendmail-msp21:00:181
789999532,39cyclictest15389-21sendmail-msp22:40:141
791899523,32cyclictest25866irq/58-eth000:10:143
789499491,36cyclictest2772-21ssh00:30:150
791199471,2cyclictest131rcu_preempt21:00:172
789999473,37cyclictest30146-21ssh22:10:121
7911994619,19cyclictest27-21ksoftirqd/223:35:142
789999461,37cyclictest17165-21sendmail-msp23:20:151
789499461,34cyclictest23437-21sendmail-msp20:00:150
789499461,34cyclictest23437-21sendmail-msp20:00:150
789999459,28cyclictest21-21ksoftirqd/121:20:201
7918994434,4cyclictest25866irq/58-eth000:15:133
7911994417,20cyclictest900-21snmpd20:50:162
7899994425,16cyclictest21-21ksoftirqd/122:35:131
789499442,33cyclictest4160-21sendmail-msp19:40:200
7911994324,10cyclictest22146irq/40-dwc2_hso19:20:152
7899994330,10cyclictest522-21usb-storage22:20:151
791199422,17cyclictest900-21snmpd00:20:392
789999421,31cyclictest11403-21ssh23:55:131
791199414,32cyclictest22146irq/40-dwc2_hso19:25:142
789999412,25cyclictest2286-21ssh21:15:141
789999410,3cyclictest0-21swapper/121:55:141
789999401,24cyclictest29420-21sendmail-msp20:40:181
789999394,29cyclictest18421-21sendmail-msp23:40:151
789999391,2cyclictest2071-21apt-get20:45:181
789999390,6cyclictest25661-21ssh00:02:171
7894993921,12cyclictest601-21rngd19:35:120
789999382,2cyclictest0-21swapper/120:20:161
791199372,2cyclictest131rcu_preempt23:50:162
789999371,31cyclictest25855-21ssh23:25:141
789999371,11cyclictest137066ssh23:48:021
791899367,14cyclictest25866irq/58-eth021:25:163
7911993617,15cyclictest12970-21sendmail-msp22:00:152
791199361,3cyclictest131rcu_preempt20:35:132
789999364,26cyclictest21735-21ssh22:05:161
789999362,3cyclictest601-21rngd20:05:161
789999361,3cyclictest601-21rngd00:25:151
789999360,5cyclictest9247-21apt-get19:45:181
789499361,28cyclictest17367-21sendmail-msp19:20:140
791199357,16cyclictest27-21ksoftirqd/220:15:162
789999353,29cyclictest16630-21sh23:00:141
791899346,14cyclictest0-21swapper/321:15:133
791899342,19cyclictest25866irq/58-eth020:25:153
791899341,19cyclictest25866irq/58-eth022:15:133
791899341,19cyclictest25866irq/58-eth022:15:133
789999341,2cyclictest24835-21apt-get20:35:131
789999341,2cyclictest14114-21apt-get19:50:141
789999341,25cyclictest25868-21diskmemload00:10:141
789999340,5cyclictest5628-21apt-get20:15:171
789999340,2cyclictest0-21swapper/122:15:151
789999340,2cyclictest0-21swapper/122:15:151
789999332,28cyclictest3211-21ssh23:50:161
789999331,6cyclictest23715-21ntp_192.168.11400:20:321
789999331,3cyclictest0-21swapper/121:05:151
789999331,2cyclictest27219-21apt-get19:30:141
789999330,27cyclictest19115-21ssh21:25:161
789499333,21cyclictest17517-21ssh23:20:150
791199322,12cyclictest601-21rngd23:01:392
789999323,21cyclictest9181-21ssh23:15:141
789999322,26cyclictest27945-21ssh00:05:141
789999321,6cyclictest6187-21usage_Anybus23:30:341
789999321,3cyclictest601-21rngd22:55:141
789999321,2cyclictest12370-21apt-get00:15:131
7918993125,4cyclictest0-21swapper/323:30:153
791199315,18cyclictest261rcuc/221:20:182
791199314,17cyclictest22146irq/40-dwc2_hso00:35:142
791199314,17cyclictest22146irq/40-dwc2_hso00:35:142
789999319,19cyclictest11745-21sendmail-msp21:40:401
789999316,22cyclictest23437-21sendmail-msp20:00:321
789999316,22cyclictest23437-21sendmail-msp20:00:321
789999312,25cyclictest13962-21cat23:35:261
789999311,8cyclictest13294-21/usr/sbin/munin22:00:321
789999311,5cyclictest26272-21apt-get21:10:161
789999311,3cyclictest601-21rngd20:25:151
789999311,2cyclictest32001-21apt-get19:35:121
789999311,2cyclictest24070-21apt-get22:45:121
789999311,27cyclictest23144-21cut21:45:241
789999311,26cyclictest17673-21df_inode19:20:161
789999310,9cyclictest30272-21cat21:50:221
789999310,4cyclictest10405-21apt-get00:35:141
789999310,4cyclictest10405-21apt-get00:35:141
789499311,25cyclictest11745-21sendmail-msp21:40:150
7918993016,9cyclictest25866irq/58-eth022:50:163
791199305,15cyclictest22146irq/40-dwc2_hso20:10:172
791199302,18cyclictest22146irq/40-dwc2_hso23:10:142
791199302,17cyclictest22146irq/40-dwc2_hso19:15:142
791199301,16cyclictest24864-21ssh23:05:152
789999301,3cyclictest4378-21apt-get19:40:211
789999301,25cyclictest16212-21cut21:00:011
789999300,4cyclictest8019-21apt-get19:10:141
789499303,19cyclictest15389-21sendmail-msp22:40:120
789499303,18cyclictest27190-21ssh23:45:160
789499302,21cyclictest22175-21ssh23:01:390
791899294,14cyclictest25866irq/58-eth021:55:143
791199293,20cyclictest0-21swapper/219:34:392
791199291,18cyclictest19344-21sendmail-msp00:00:142
789999291,7cyclictest7902-21sed19:10:021
789999291,7cyclictest31625-21rm23:08:221
789999291,6cyclictest23026-21/usr/sbin/munin22:25:201
789999291,4cyclictest18862-21apt-get19:55:161
789999291,3cyclictest22434-21apt-get19:25:141
789999290,3cyclictest2343-21apt-get00:30:051
789999290,2cyclictest3656-21ssh21:35:171
789499292,20cyclictest20608-21ssh21:45:150
789499292,19cyclictest29420-21sendmail-msp20:40:390
789499291,20cyclictest25868-21diskmemload22:50:150
791899285,16cyclictest0-21swapper/321:35:163
791899282,13cyclictest0-21swapper/323:55:143
791199282,21cyclictest22716-21ssh22:43:392
791199282,11cyclictest601-21rngd22:21:392
791199281,17cyclictest22146irq/40-dwc2_hso22:55:142
789999281,5cyclictest7319-21ps20:50:201
789999281,23cyclictest32513-21ssh22:50:161
789999281,11cyclictest30271-21grep21:30:271
789999280,8cyclictest31429-21/usr/sbin/munin22:30:361
789499289,13cyclictest8325-21apt-get22:55:150
789499284,19cyclictest19344-21sendmail-msp00:00:150
791899274,12cyclictest25866irq/58-eth022:10:133
7918992711,10cyclictest0-21swapper/300:01:393
791199273,16cyclictest22146irq/40-dwc2_hso20:55:172
791199272,20cyclictest0-21swapper/200:07:402
791199272,10cyclictest0-21swapper/220:25:162
789999271,6cyclictest4379-21wc20:10:301
789999271,3cyclictest23444-21ntp_192.168.11420:30:251
789999271,10cyclictest701-21apt-get23:10:141
789999270,2cyclictest12804-21apt-get19:15:131
789499277,5cyclictest12-21ksoftirqd/022:20:160
789499274,14cyclictest10427-21ssh00:35:150
789499274,14cyclictest10427-21ssh00:35:150
791899267,9cyclictest0-21swapper/322:40:153
791899263,9cyclictest25866irq/58-eth020:05:163
791899263,16cyclictest0-21swapper/300:20:393
791199263,9cyclictest12382-21ssh00:15:132
791199262,14cyclictest27721-21ssh00:25:142
791199261,18cyclictest22146irq/40-dwc2_hso22:45:142
789499267,10cyclictest601-21rngd21:55:140
789499263,16cyclictest12970-21sendmail-msp22:00:130
791899254,16cyclictest25866irq/58-eth023:15:153
791899254,15cyclictest0-21swapper/322:05:173
791199251,15cyclictest25849-21sh23:25:142
791199251,10cyclictest601-21rngd20:21:392
789499254,6cyclictest19138-21cat23:40:200
7918992414,6cyclictest0-21swapper/300:35:143
7918992414,6cyclictest0-21swapper/300:35:143
791199244,13cyclictest22146irq/40-dwc2_hso19:55:162
791199243,16cyclictest1989-21ssh23:30:152
791199242,18cyclictest601-21rngd21:41:392
791199242,17cyclictest601-21rngd00:30:052
7911992416,5cyclictest25868-21diskmemload23:20:222
7911992413,9cyclictest14554-21ssh23:16:202
791199240,14cyclictest131rcu_preempt21:18:202
789499249,13cyclictest601-21rngd19:10:140
789499244,4cyclictest601-21rngd21:36:040
789499244,14cyclictest665-21ntpd23:10:140
789499243,3cyclictest12-21ksoftirqd/020:30:250
789499241,18cyclictest16228-21sendmail-msp21:00:330
789499241,13cyclictest28594-21cat22:45:330
791899233,11cyclictest0-21swapper/323:50:163
791199233,14cyclictest22146irq/40-dwc2_hso21:50:142
791199232,6cyclictest0-21swapper/219:43:392
791199231,13cyclictest25194-21sort22:05:272
7911992311,10cyclictest13789-21sshd19:48:392
789499237,5cyclictest0-21swapper/020:50:180
789499234,5cyclictest12-21ksoftirqd/022:14:480
789499233,4cyclictest4242-21ntp_192.168.11420:10:270
789499231,18cyclictest26484-21ls22:25:260
791899229,8cyclictest25866irq/58-eth000:25:153
791899227,9cyclictest0-21swapper/323:00:173
791899223,12cyclictest0-21swapper/321:20:393
791899222,14cyclictest0-21swapper/300:30:143
791199228,6cyclictest4373-21irqcore00:10:202
791199224,3cyclictest601-21rngd20:00:162
791199224,3cyclictest601-21rngd20:00:162
791199222,7cyclictest31461-21ssh22:30:142
7911992211,9cyclictest14337-21ssh22:38:362
791199221,12cyclictest22980-21ssh21:45:232
789499222,9cyclictest12-21ksoftirqd/022:07:150
789499220,6cyclictest1969846latency_hist00:20:020
791899213,14cyclictest0-21swapper/323:20:163
791899213,13cyclictest0-21swapper/322:45:153
7918992113,5cyclictest25866irq/58-eth023:45:163
7918992111,8cyclictest0-21swapper/323:45:023
791899211,15cyclictest25866irq/58-eth022:00:143
791899211,13cyclictest25866irq/58-eth020:40:393
791199219,5cyclictest32132-21sed20:05:302
791199212,14cyclictest22146irq/40-dwc2_hso19:10:152
7911992115,4cyclictest27-21ksoftirqd/221:10:012
791199211,18cyclictest601-21rngd22:15:022
791199211,12cyclictest30241-21irqrtprio20:40:262
791199211,11cyclictest18761-21apt-get23:40:152
789499219,6cyclictest601-21rngd20:35:140
789499214,4cyclictest12-21ksoftirqd/023:55:170
789499213,10cyclictest601-21rngd22:31:510
789499212,6cyclictest10223-21sendmail-msp20:21:070
789499212,5cyclictest52246usb-storage00:21:460
789499211,7cyclictest601-21rngd21:15:210
7894992115,4cyclictest601-21rngd19:50:140
789499211,15cyclictest19269-21grep21:25:200
789499210,4cyclictest601-21rngd22:15:150
789499210,4cyclictest601-21rngd22:15:150
791899209,7cyclictest25866irq/58-eth021:10:163
791899203,11cyclictest0-21swapper/319:25:393
791899202,15cyclictest25866irq/58-eth023:05:153
791899202,13cyclictest0-21swapper/323:35:153
791899201,16cyclictest0-21swapper/322:35:133
7918992011,5cyclictest0-21swapper/300:05:223
791199203,14cyclictest14160-21cut23:55:232
791199201,9cyclictest8112-21kthreadcore21:55:252
7911992014,4cyclictest19290-21ssh21:25:202
791199201,14cyclictest22146irq/40-dwc2_hso21:35:172
791199201,11cyclictest2741-21kthreadcore20:45:252
789499209,8cyclictest28515-21apt-get20:05:160
789499205,5cyclictest12-21ksoftirqd/021:20:410
789499204,4cyclictest12-21ksoftirqd/023:35:050
789499204,12cyclictest15570-21ls22:40:020
789499203,4cyclictest12-21ksoftirqd/023:25:120
789499203,3cyclictest12-21ksoftirqd/023:16:020
789499203,3cyclictest12-21ksoftirqd/000:05:150
789499202,4cyclictest12-21ksoftirqd/021:50:310
789499202,3cyclictest12-21ksoftirqd/023:32:230
789499202,3cyclictest12-21ksoftirqd/021:14:320
7894992014,3cyclictest27698-21apt-get00:25:150
7894992010,7cyclictest18572-21memory20:25:260
791899193,9cyclictest0-21swapper/323:25:143
791899192,11cyclictest0-21swapper/319:48:393
791899192,11cyclictest0-21swapper/319:36:393
7918991912,5cyclictest25866irq/58-eth021:45:153
7918991912,4cyclictest0-21swapper/322:20:153
7918991911,5cyclictest0-21swapper/321:50:303
791899191,13cyclictest0-21swapper/320:30:293
791199197,9cyclictest601-21rngd20:30:202
791199193,14cyclictest1675-21kthreadcore22:50:232
791199191,9cyclictest6776-21irqrtprio22:15:202
791199191,9cyclictest6776-21irqrtprio22:15:202
791199191,8cyclictest27372-21fschecks_count23:45:192
791199191,8cyclictest18632-21sh19:55:012
791199190,16cyclictest23296-21fschecks_time22:25:192
789499193,4cyclictest12-21ksoftirqd/019:25:150
789499193,3cyclictest601-21rngd19:55:240
789499192,8cyclictest12-21ksoftirqd/021:31:320
789499191,4cyclictest601-21rngd21:05:140
789499191,3cyclictest601-21rngd23:05:200
789499191,3cyclictest601-21rngd20:15:170
789499191,3cyclictest0-21swapper/000:10:210
7894991910,6cyclictest7686-21cat23:50:350
791899189,6cyclictest25866irq/58-eth022:55:133
791899183,12cyclictest0-21swapper/321:40:243
791899183,11cyclictest0-21swapper/322:30:143
791899182,12cyclictest0-21swapper/320:20:183
791899181,13cyclictest0-21swapper/323:10:153
791899181,12cyclictest0-21swapper/319:15:293
791899181,11cyclictest0-21swapper/319:40:383
791199181,14cyclictest31673-21unixbench_multi21:30:352
789499189,4cyclictest601-21rngd20:45:180
789499180,3cyclictest601-21rngd19:15:140
791899179,6cyclictest0-21swapper/322:25:453
791899179,6cyclictest0-21swapper/321:31:093
791899172,9cyclictest0-21swapper/319:55:163
791899172,8cyclictest0-21swapper/320:00:153
791899172,8cyclictest0-21swapper/320:00:153
791899171,12cyclictest0-21swapper/320:35:173
791899171,11cyclictest25866irq/58-eth019:20:143
791899171,10cyclictest0-21swapper/320:55:053
791199171,8cyclictest7980-21latency_hist19:10:022
789499172,11cyclictest601-21rngd19:31:380
789499171,8cyclictest12-21ksoftirqd/020:55:300
789499171,10cyclictest665-21ntpd19:45:170
791899162,11cyclictest0-21swapper/320:50:173
791899162,11cyclictest0-21swapper/319:33:393
791899161,12cyclictest0-21swapper/321:00:183
791899152,9cyclictest0-21swapper/319:50:043
791899152,10cyclictest25866irq/58-eth020:10:163
791899152,10cyclictest0-21swapper/320:15:173
791899152,10cyclictest0-21swapper/319:10:163
791899136,3cyclictest0-21swapper/320:45:173
791899131,8cyclictest0-21swapper/321:10:023
789499121,8cyclictest7902-21latency_hist19:10:020
791899101,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional