You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-22 - 21:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue Apr 22, 2025 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19429995741,7cyclictest0-21swapper/208:45:132
1941199553,9cyclictest131rcu_preempt11:00:140
1941199541,41cyclictest27762-21sendmail-msp09:00:140
1941999493,38cyclictest5518-21sendmail-msp10:00:131
1941199482,37cyclictest15822-21sendmail-msp07:40:140
1941199471,10cyclictest131rcu_preempt09:15:120
1941999460,10cyclictest131rcu_preempt08:25:141
1941199451,32cyclictest2610-21sendmail-msp08:00:120
1944099442,23cyclictest25866irq/58-eth012:15:123
1941999442,6cyclictest131rcu_preempt11:15:161
1942999431,31cyclictest24257-21ssh10:15:142
19429994312,20cyclictest27-21ksoftirqd/212:35:152
1941199433,6cyclictest131rcu_preempt09:30:090
1941199422,30cyclictest8585-21sendmail-msp08:40:130
1942999413,5cyclictest10061-21ssh10:30:082
1941999411,30cyclictest25092-21sendmail-msp10:40:151
1942999402,16cyclictest601-21rngd11:18:362
19411994016,17cyclictest28981-21sendmail-msp07:20:130
1941199391,29cyclictest25092-21sendmail-msp10:40:160
1942999381,17cyclictest0-21swapper/207:35:142
1941999384,10cyclictest131rcu_preempt07:35:131
19429993722,9cyclictest24806-21ssh10:39:352
1942999372,28cyclictest16583-21sh10:56:352
1942999372,19cyclictest4627-21ssh09:58:352
1942999372,18cyclictest10073-21ssh10:53:352
1941999362,31cyclictest19027-21sendmail-msp09:20:121
1942999352,21cyclictest0-21swapper/207:45:122
19429993512,5cyclictest16600-21ssh10:10:102
1941199351,6cyclictest131rcu_preempt11:10:100
1941199351,25cyclictest12227-21sendmail-msp09:40:140
1942999342,24cyclictest27126-21ssh09:26:342
1941999341,29cyclictest30417-21sh09:55:131
1942999336,16cyclictest0-21swapper/212:30:102
1942999332,25cyclictest4983-21diskmemload09:15:152
1942999332,16cyclictest25250-21ssh11:30:092
19429993314,12cyclictest22146irq/40-dwc2_hso09:00:142
1941999336,17cyclictest201rcuc/112:00:141
1941999334,24cyclictest22625-21ssh09:50:151
1941199336,17cyclictest3918-21kworker/u8:011:15:160
19411993316,12cyclictest21826-21sendmail-msp08:20:160
19411993314,9cyclictest15056-1kworker/0:2H11:20:130
1941199331,23cyclictest24248-21ssh10:15:130
19429993219,8cyclictest27-21ksoftirqd/209:47:362
19429993211,14cyclictest22146irq/40-dwc2_hso08:00:142
1941199322,23cyclictest9550-21sendmail-msp11:40:150
1941199322,23cyclictest9550-21sendmail-msp11:40:140
1944099312,18cyclictest0-21swapper/310:15:133
19440993113,14cyclictest25866irq/58-eth009:15:133
1942999312,17cyclictest22146irq/40-dwc2_hso12:00:132
19429993118,8cyclictest27-21ksoftirqd/210:49:352
1941999312,26cyclictest13276-21sendmail-msp11:20:151
1941999311,26cyclictest601-21rngd10:15:121
1941999311,23cyclictest522-21usb-storage12:15:111
1941999311,22cyclictest29383-21ssh09:30:081
1942999309,12cyclictest22146irq/40-dwc2_hso10:20:132
1942999303,15cyclictest22146irq/40-dwc2_hso11:10:092
1942999302,18cyclictest22146irq/40-dwc2_hso11:35:122
1942999294,16cyclictest22146irq/40-dwc2_hso08:50:162
1942999292,20cyclictest4678-21ssh09:35:122
1942999292,20cyclictest12270-21ssh10:03:342
19429992910,11cyclictest22146irq/40-dwc2_hso11:00:142
19419992910,12cyclictest131rcu_preempt08:10:161
1942999287,10cyclictest27-21ksoftirqd/207:15:162
1942999283,13cyclictest22146irq/40-dwc2_hso11:50:102
1942999283,11cyclictest0-21swapper/211:24:352
1942999282,17cyclictest22146irq/40-dwc2_hso08:05:142
1944099273,11cyclictest0-21swapper/311:05:363
1942999271,20cyclictest32113-21ssh11:55:142
19419992716,8cyclictest12227-21sendmail-msp09:40:181
19419992711,10cyclictest27762-21sendmail-msp09:00:311
1941199275,17cyclictest0-21swapper/008:15:140
1941199271,19cyclictest31592-21sendmail-msp10:20:110
19411992711,11cyclictest5518-21sendmail-msp10:00:140
1944099269,11cyclictest0-21swapper/311:13:343
1944099263,14cyclictest0-21swapper/311:15:353
1942999264,14cyclictest22146irq/40-dwc2_hso09:05:152
1942999261,18cyclictest8927-21ssh10:28:362
1941999263,13cyclictest21-21ksoftirqd/111:55:121
1941199263,16cyclictest17745-21ssh10:35:110
1944099254,15cyclictest0-21swapper/312:30:083
1944099254,13cyclictest0-21swapper/309:30:083
1944099252,16cyclictest25866irq/58-eth011:02:363
1944099252,13cyclictest0-21swapper/310:55:153
1942999252,8cyclictest23315-21ssh11:27:352
1942999252,19cyclictest28634-21ssh09:52:352
1942999252,14cyclictest22146irq/40-dwc2_hso08:10:172
1942999251,19cyclictest6836-21sendmail-msp12:20:162
1942999251,15cyclictest22146irq/40-dwc2_hso07:20:142
1941999253,17cyclictest3918-21kworker/u8:011:28:341
1941999252,17cyclictest601-21rngd10:56:341
1941999251,18cyclictest21669-21ssh12:10:101
1941199257,9cyclictest131rcu_preempt09:55:140
1944099242,10cyclictest0-21swapper/311:44:353
1944099242,10cyclictest0-21swapper/311:44:353
1942999246,8cyclictest27-21ksoftirqd/210:07:392
1942999243,13cyclictest22146irq/40-dwc2_hso07:50:182
1942999242,19cyclictest0-21swapper/209:40:162
19429992413,9cyclictest16357-21ssh12:25:362
1942999241,17cyclictest601-21rngd07:40:142
1941999242,15cyclictest2610-21sendmail-msp08:00:141
1941999242,14cyclictest31592-21sendmail-msp10:20:131
19419992414,8cyclictest15822-21sendmail-msp07:40:351
1941199247,11cyclictest15056-1kworker/0:2H12:20:140
1944099233,13cyclictest0-21swapper/310:53:343
1944099233,13cyclictest0-21swapper/309:26:353
1944099233,12cyclictest0-21swapper/309:35:133
1944099231,14cyclictest25866irq/58-eth009:50:143
1942999239,7cyclictest21826-21sendmail-msp08:20:172
1942999237,10cyclictest5339-21ssh09:10:152
1942999232,6cyclictest0-21swapper/211:05:352
1942999231,9cyclictest0-21swapper/208:30:192
1942999231,9cyclictest0-21swapper/208:30:182
1942999231,4cyclictest601-21rngd12:09:412
1942999230,3cyclictest601-21rngd11:46:522
1941999232,16cyclictest19470-21ssh11:00:211
1941999231,16cyclictest9550-21sendmail-msp11:40:171
1941999231,16cyclictest9550-21sendmail-msp11:40:161
1944099226,11cyclictest0-21swapper/309:55:133
1944099222,10cyclictest0-21swapper/307:45:123
1942999225,11cyclictest27-21ksoftirqd/209:34:262
1942999222,6cyclictest0-21swapper/210:43:362
19429992212,8cyclictest0-21swapper/212:18:482
19429992211,8cyclictest0-21swapper/209:21:142
1942999221,16cyclictest4006-21apt-get08:35:152
1942999220,3cyclictest601-21rngd12:13:422
1941999229,10cyclictest6836-21sendmail-msp12:20:281
1941999228,4cyclictest201rcuc/107:50:171
1941999222,13cyclictest601-21rngd10:53:351
1941199228,4cyclictest131rcu_preempt12:00:290
1941199227,4cyclictest131rcu_preempt11:55:170
1944099214,12cyclictest0-21swapper/311:55:133
19440992113,6cyclictest0-21swapper/312:00:373
1942999217,11cyclictest10445-21irqcore11:40:202
1942999217,11cyclictest10445-21irqcore11:40:192
1942999212,15cyclictest601-21rngd08:55:112
1942999212,14cyclictest17352-21diskstats08:15:172
1942999210,18cyclictest13433-21latency_hist08:45:022
1941999217,4cyclictest131rcu_preempt11:10:211
1941999214,5cyclictest21-21ksoftirqd/108:00:011
1941999211,17cyclictest10202-21cut10:30:121
1941999210,18cyclictest12477-21ls09:15:181
1941199212,8cyclictest0-21swapper/008:25:130
19411992113,5cyclictest26244-21ssh09:50:270
1944099203,12cyclictest0-21swapper/310:29:353
1944099203,11cyclictest0-21swapper/309:45:353
1944099201,13cyclictest0-21swapper/312:20:193
1944099201,11cyclictest25866irq/58-eth008:55:143
19440992010,7cyclictest0-21swapper/309:41:283
1942999208,5cyclictest261rcuc/207:55:142
1942999201,4cyclictest27-21ksoftirqd/207:10:312
1942999201,16cyclictest10908-21idleruntime-cro07:35:012
19429992011,3cyclictest6176-21ls07:30:012
1941999203,3cyclictest131rcu_preempt07:30:181
1941999202,12cyclictest601-21rngd09:26:351
1941999201,9cyclictest25063-21idleruntime-cro10:40:021
19419992013,3cyclictest601-21rngd09:35:141
1941999201,16cyclictest16584-21apt-get10:10:081
1941199202,14cyclictest11135-21apt-get07:35:140
1941199201,7cyclictest24235-21ssh09:21:000
19411992011,7cyclictest4967-21sshd09:06:350
1944099199,7cyclictest0-21swapper/311:22:023
1944099193,11cyclictest0-21swapper/311:27:343
1944099193,11cyclictest0-21swapper/310:39:343
1944099193,11cyclictest0-21swapper/310:07:363
1944099192,11cyclictest0-21swapper/310:45:363
19440991914,3cyclictest33-21ksoftirqd/311:35:173
19440991911,4cyclictest25866irq/58-eth010:00:133
1944099191,12cyclictest0-21swapper/307:30:023
19440991910,7cyclictest0-21swapper/312:05:053
19440991910,7cyclictest0-21swapper/310:32:143
19440991910,6cyclictest25866irq/58-eth010:10:093
19440991910,6cyclictest0-21swapper/311:34:563
1942999192,9cyclictest26877-21/usr/sbin/munin08:25:172
1941999199,8cyclictest31709-21cron11:55:011
1941999199,7cyclictest8585-21sendmail-msp08:40:261
1941999199,7cyclictest18218-21cron08:50:021
1941999196,4cyclictest601-21rngd12:34:351
1941999196,3cyclictest131rcu_preempt07:20:171
1941999196,10cyclictest22737-21cron11:50:021
1941999191,6cyclictest601-21rngd10:30:011
1941999191,4cyclictest131rcu_preempt12:35:151
1941999191,4cyclictest131rcu_preempt07:45:141
19419991913,4cyclictest21803-21latency_hist08:20:011
1941999191,13cyclictest8958-21ssh09:10:311
1941999191,11cyclictest27762-21sendmail-msp09:00:011
1941999191,10cyclictest601-21rngd10:06:451
1941199191,9cyclictest710366ssh11:37:080
19411991913,3cyclictest522-21usb-storage12:25:100
19411991912,4cyclictest16566-21/usr/sbin/munin10:10:250
19411991911,5cyclictest0-21swapper/010:30:360
19411991910,6cyclictest8257-21ssh09:35:250
1944099189,6cyclictest0-21swapper/311:52:503
1944099189,6cyclictest0-21swapper/311:46:153
1944099182,10cyclictest0-21swapper/307:15:043
1944099182,10cyclictest0-21swapper/307:10:133
19440991813,3cyclictest0-21swapper/309:20:333
19440991812,4cyclictest0-21swapper/310:40:193
19440991810,6cyclictest0-21swapper/312:26:253
1941999189,6cyclictest28878-21ntp_kernel_pll_11:30:221
1941999184,11cyclictest19959-21wc12:30:021
1941999182,10cyclictest601-21rngd11:07:361
1941999181,9cyclictest601-21rngd07:10:281
1941999181,8cyclictest18429-21/usr/sbin/munin08:50:321
1941999181,4cyclictest131rcu_preempt11:35:131
1941999181,13cyclictest601-21rngd08:05:301
1941999181,12cyclictest22047-21apt-get08:20:151
1941199188,3cyclictest131rcu_preempt08:00:010
1941199182,12cyclictest30284-21apt-get12:15:020
1941199181,6cyclictest12-21ksoftirqd/010:50:170
19411991811,5cyclictest8549-21cron08:40:010
1941199181,13cyclictest24088-21apt-get07:15:040
1941199181,13cyclictest20172-21apt-get12:30:090
1941199181,12cyclictest32670-21ntp_192.168.11412:35:290
19411991810,6cyclictest7452-21cron08:05:020
19411991810,6cyclictest25345-21ssh12:10:240
1944099179,6cyclictest0-21swapper/312:12:453
1944099179,6cyclictest0-21swapper/309:13:233
1944099179,6cyclictest0-21swapper/308:30:203
1944099179,6cyclictest0-21swapper/308:30:203
1944099179,3cyclictest0-21swapper/312:35:063
1944099172,6cyclictest0-21swapper/307:30:163
1944099172,10cyclictest0-21swapper/308:15:143
1944099171,12cyclictest0-21swapper/309:05:233
1944099171,12cyclictest0-21swapper/308:35:173
1942999171,8cyclictest19433-21cat07:10:012
1941999173,9cyclictest0-21swapper/109:45:341
1941999171,2cyclictest131rcu_preempt08:30:341
1941999171,2cyclictest131rcu_preempt08:30:331
19419991710,5cyclictest601-21rngd08:40:001
1941199174,11cyclictest6153-21basename07:30:010
1941199173,3cyclictest131rcu_preempt07:50:260
19411991711,4cyclictest9765-21sh10:30:020
19411991711,4cyclictest0-21swapper/008:55:320
19411991711,4cyclictest0-21swapper/007:30:270
1941199171,12cyclictest32308-21sed08:30:240
1941199171,12cyclictest32308-21sed08:30:230
1941199171,11cyclictest22677-21ssh11:26:350
1941199170,13cyclictest601-21rngd08:50:170
1944099168,6cyclictest0-21swapper/307:24:003
1944099162,8cyclictest0-21swapper/309:00:133
1944099162,10cyclictest0-21swapper/308:45:143
1944099162,10cyclictest0-21swapper/307:35:193
19440991611,3cyclictest0-21swapper/308:20:203
1944099161,11cyclictest0-21swapper/310:20:033
1944099161,10cyclictest0-21swapper/307:40:293
1941999169,3cyclictest21-21ksoftirqd/107:25:221
1941999162,9cyclictest601-21rngd10:45:361
1941999161,9cyclictest601-21rngd07:15:041
1941999161,11cyclictest21315-21latency_hist12:10:021
1941999161,11cyclictest1080-21tr09:05:201
1941199169,5cyclictest0-21swapper/012:07:230
1941199165,8cyclictest601-21rngd09:10:170
1941199162,4cyclictest3431-21latency_hist10:50:010
1941199162,10cyclictest601-21rngd11:47:340
1941199162,10cyclictest601-21rngd11:30:350
1941199162,10cyclictest601-21rngd11:09:340
1941199162,10cyclictest601-21rngd10:07:350
1941199162,10cyclictest601-21rngd09:49:360
1941199162,10cyclictest601-21rngd09:25:340
19411991610,4cyclictest21424-21irqrtprio07:45:180
19411991610,4cyclictest0-21swapper/007:10:150
1941199160,8cyclictest16286-21kthreadcore08:45:220
1941199160,8cyclictest15539-21cut08:10:250
1944099151,11cyclictest0-21swapper/308:10:353
1944099151,11cyclictest0-21swapper/308:05:233
1941199151,11cyclictest23311-21ssh11:50:130
1941199151,11cyclictest11354-21df10:55:160
1944099149,3cyclictest0-21swapper/307:52:343
1944099141,10cyclictest0-21swapper/308:00:153
1944099138,3cyclictest33-21ksoftirqd/307:55:283
1944099138,3cyclictest0-21swapper/308:40:363
1944099131,5cyclictest0-21swapper/308:30:013
1941199131,9cyclictest19475-21latency_hist07:10:010
1944099117,2cyclictest33-21ksoftirqd/308:50:183
1944099111,7cyclictest0-21swapper/307:10:023
194199991,6cyclictest601-21rngd07:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional