You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 13:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Thu Sep 03, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
432199621,5cyclictest131rcu_preempt20:50:110
432199551,47cyclictest19924-21ssh22:20:110
434499546,28cyclictest0-21swapper/200:25:132
434499541,3cyclictest0-21swapper/223:00:122
434499522,7cyclictest0-21swapper/219:40:132
4321995233,8cyclictest12-21ksoftirqd/000:35:150
434499511,3cyclictest0-21swapper/223:25:152
434499511,3cyclictest0-21swapper/223:25:142
434499503,43cyclictest27346irq/40-dwc2_hso20:15:122
434499501,3cyclictest0-21swapper/219:35:112
434499501,2cyclictest0-21swapper/220:10:112
4321995029,9cyclictest12-21ksoftirqd/023:05:130
434499492,6cyclictest0-21swapper/220:45:132
434499492,6cyclictest0-21swapper/220:45:122
434499492,2cyclictest7091-21apt-get21:55:122
434499492,2cyclictest0-21swapper/220:40:122
432199483,37cyclictest21420-21ssh21:45:120
434499460,42cyclictest0-21swapper/220:25:102
434499460,3cyclictest25200-21ssh22:05:112
434499451,7cyclictest18288-21ssh22:55:122
433499451,34cyclictest20531-21ssh23:50:131
432199453,34cyclictest5975-21ssh00:00:110
434499445,34cyclictest27346irq/40-dwc2_hso00:10:142
434499441,39cyclictest0-21swapper/221:35:132
434499441,36cyclictest0-21swapper/200:05:172
432199441,32cyclictest20524-21rm23:50:140
434499431,4cyclictest0-21swapper/219:20:142
4321994317,12cyclictest131rcu_preempt20:20:110
4354994229,6cyclictest35966irq/58-eth022:30:113
434499421,2cyclictest0-21swapper/221:25:132
434499411,7cyclictest7067-21kthreadcore00:35:232
434499411,3cyclictest0-21swapper/222:30:132
434499411,2cyclictest31888-21apt-get21:15:142
4321994116,12cyclictest131rcu_preempt00:15:130
4354994029,6cyclictest35966irq/58-eth022:15:113
4334994015,19cyclictest21-21ksoftirqd/121:10:121
4321994024,9cyclictest0-21swapper/000:20:110
432199402,28cyclictest2001-21ssh22:10:180
4321994020,12cyclictest4305-21cyclictest22:40:130
434499392,2cyclictest18910-21apt-get19:25:112
434499391,3cyclictest0-21swapper/221:30:132
434499391,3cyclictest0-21swapper/220:30:122
434499391,34cyclictest25813-21/usr/sbin/munin23:35:182
434499391,2cyclictest13128-21apt-get23:10:192
434499390,3cyclictest23695-21apt-get19:30:122
4321993929,7cyclictest0-21swapper/022:15:110
432199391,33cyclictest29475-21ssh23:55:120
434499381,3cyclictest11508-21apt-get23:45:122
434499381,3cyclictest0-21swapper/223:15:102
434499380,5cyclictest5778-21fschecks_count19:45:182
4321993818,16cyclictest12-21ksoftirqd/021:50:190
435499372,22cyclictest35966irq/58-eth023:30:143
434499372,2cyclictest0-21swapper/222:45:132
434499371,3cyclictest0-21swapper/220:35:132
434499371,33cyclictest0-21swapper/200:20:172
434499371,2cyclictest0-21swapper/200:15:142
434499370,9cyclictest23516-21ntp_192.168.11420:00:252
434499370,4cyclictest27346irq/40-dwc2_hso00:30:132
435499364,27cyclictest35966irq/58-eth022:05:133
434499368,25cyclictest27329-21sh23:52:562
434499366,28cyclictest16466-21ls22:00:172
434499361,7cyclictest9404-21ssh22:50:152
434499361,4cyclictest8246-21df20:55:122
434499361,4cyclictest23062-21grep21:10:172
434499361,3cyclictest0-21swapper/223:40:132
434499361,3cyclictest0-21swapper/220:20:122
434499361,31cyclictest9358-21df19:15:142
434499361,31cyclictest9358-21df19:15:132
434499360,3cyclictest4013-21ssh23:05:122
4334993625,7cyclictest4386-21ssh00:35:151
4334993611,19cyclictest21-21ksoftirqd/122:45:131
435499358,18cyclictest0-21swapper/300:10:133
435499355,23cyclictest35966irq/58-eth020:00:123
435499355,16cyclictest35966irq/58-eth021:35:143
4354993518,9cyclictest0-21swapper/319:20:133
434499351,8cyclictest5703-21latency_hist00:00:022
434499351,6cyclictest28528-21switchtime20:05:312
434499351,3cyclictest0-21swapper/222:10:192
434499351,2cyclictest0-21swapper/222:20:122
434499350,6cyclictest17142-21ps19:55:212
433499351,26cyclictest18302-21ssh22:55:141
435499344,13cyclictest35966irq/58-eth000:35:153
434499342,27cyclictest0-21swapper/222:41:362
434499341,4cyclictest21317-21ntpq21:05:242
434499341,3cyclictest0-21swapper/221:45:122
434499340,5cyclictest4700-21fw_packets19:10:162
434499340,5cyclictest18169-21ssh22:18:232
434499340,5cyclictest16945-21cat21:00:312
434499340,4cyclictest930-21snmpd21:50:292
434499340,4cyclictest7877-21cut00:00:192
435499335,19cyclictest0-21swapper/321:55:133
4354993327,4cyclictest35966irq/58-eth000:25:143
434499330,6cyclictest21599-21cat23:30:322
432199333,21cyclictest18298-21ssh22:55:140
432199333,19cyclictest73-21kswapd023:30:120
432199332,25cyclictest28941-21ssh22:25:130
4321993319,11cyclictest0-21swapper/019:25:120
434499321,5cyclictest6816-21ntpq20:50:252
434499321,3cyclictest0-21swapper/222:29:272
434499320,4cyclictest27346irq/40-dwc2_hso22:35:132
4354993125,4cyclictest35966irq/58-eth023:45:113
435499312,16cyclictest35966irq/58-eth022:25:133
434499312,3cyclictest27346irq/40-dwc2_hso19:50:142
434499312,2cyclictest8519-21apt21:20:142
434499311,3cyclictest0-21swapper/221:43:252
432199312,21cyclictest22442-21diskmemload00:25:140
4321993114,6cyclictest0-21swapper/020:25:110
432199311,26cyclictest9303-21sh22:50:130
434499300,7cyclictest30986-21ssh23:20:042
4334993025,3cyclictest13139-21ssh23:10:181
432199302,19cyclictest8535-21ssh21:20:150
4354992813,10cyclictest35966irq/58-eth022:35:143
433499281,23cyclictest27354-21ssh23:00:121
432199287,12cyclictest13149-21ssh23:10:170
432199284,14cyclictest16186-21ssh22:00:140
432199283,21cyclictest22070-21ssh23:15:120
435499273,16cyclictest0-21swapper/322:45:373
433499271,23cyclictest11537-21ssh23:45:141
432199273,18cyclictest17571-21ssh21:25:150
432199272,19cyclictest22094-21sh00:08:050
432199272,19cyclictest12646-21ssh21:56:190
432199271,20cyclictest27700-21ssh00:30:150
435499269,9cyclictest35966irq/58-eth022:20:123
433499262,18cyclictest5641-21ssh22:30:141
433499261,19cyclictest19949-21ssh22:20:141
432199263,16cyclictest11522-21ssh23:45:110
435499253,16cyclictest0-21swapper/300:05:163
435499253,14cyclictest0-21swapper/322:43:363
435499253,11cyclictest0-21swapper/323:14:383
435499252,15cyclictest35966irq/58-eth023:50:143
4334992519,3cyclictest0-21swapper/119:30:131
432199254,15cyclictest14622-21ssh22:35:140
432199252,7cyclictest131rcu_preempt22:09:360
4354992416,4cyclictest0-21swapper/300:00:113
434499241,5cyclictest4305-21cyclictest19:10:022
433499243,17cyclictest25147-1kworker/u9:221:55:131
433499241,17cyclictest16809-21ssh23:30:131
4321992418,3cyclictest12-21ksoftirqd/021:15:150
4354992315,6cyclictest35966irq/58-eth023:05:123
433499232,18cyclictest7774-21ssh23:25:131
433499232,18cyclictest7774-21ssh23:25:131
433499232,18cyclictest31150-21apt-get23:20:151
433499232,17cyclictest3989-21apt-get23:05:121
4334992314,6cyclictest24112-21ssh00:10:131
4334992314,6cyclictest11609-21ssh22:15:161
432199234,11cyclictest0-21swapper/023:25:130
432199234,11cyclictest0-21swapper/023:25:120
432199232,15cyclictest5235-21ssh22:45:370
435499226,11cyclictest0-21swapper/321:30:123
435499224,15cyclictest0-21swapper/321:20:183
435499224,12cyclictest0-21swapper/323:55:133
4354992212,6cyclictest35966irq/58-eth022:55:133
4354992210,9cyclictest0-21swapper/323:00:493
433499222,15cyclictest0-21swapper/100:33:171
4334992214,6cyclictest27530-21ssh22:40:291
432199222,15cyclictest844-21munin-node23:40:140
435499219,9cyclictest0-21swapper/322:00:123
435499215,12cyclictest35966irq/58-eth022:50:133
435499213,15cyclictest35966irq/58-eth000:20:113
433499214,9cyclictest630-21Async09:36:311
4334992115,3cyclictest0-21swapper/121:35:141
4334992113,6cyclictest9461-21ssh22:50:161
4334992111,8cyclictest1972-21apt-get22:10:181
433499211,17cyclictest4629-21uniq19:10:161
433499211,16cyclictest9280-21munin-run00:20:021
433499211,15cyclictest1814-21ntp_states21:50:301
432199219,9cyclictest23167-21/usr/sbin/munin21:10:180
432199211,3cyclictest0-21swapper/019:50:120
432199211,17cyclictest5762-21cut19:45:180
432199211,14cyclictest1003-21cut19:40:150
432199210,18cyclictest3225-21find21:35:130
435499202,10cyclictest35966irq/58-eth021:15:153
435499202,10cyclictest35966irq/58-eth021:10:123
435499201,11cyclictest35966irq/58-eth020:40:133
4354992010,6cyclictest0-21swapper/323:40:273
433499203,6cyclictest18803-21ntp_states00:05:301
433499202,15cyclictest14612-21ssh22:35:131
4334992013,3cyclictest0-21swapper/123:35:141
4334992011,6cyclictest16105-21ssh21:40:271
433499201,16cyclictest0-21swapper/120:15:101
433499201,14cyclictest28937-21ssh22:25:121
433499201,11cyclictest28530-21cut22:05:211
432199202,13cyclictest5726-21df22:30:130
4321992011,7cyclictest25869-21cron20:40:010
432199201,14cyclictest12379-21ssh21:40:140
4321992010,7cyclictest23182-1kworker/u9:023:40:010
435499199,8cyclictest0-21swapper/323:25:113
435499199,8cyclictest0-21swapper/323:25:103
4354991911,6cyclictest35966irq/58-eth000:15:133
435499191,14cyclictest0-21swapper/300:30:143
4354991910,6cyclictest35966irq/58-eth023:35:133
433499192,14cyclictest15274-21apt-get19:55:121
4334991912,5cyclictest28228-21ssh21:47:491
4334991911,6cyclictest29678-21awk20:40:251
4334991911,6cyclictest10051-21ssh00:20:151
4334991911,5cyclictest11645-21ssh21:20:241
433499191,14cyclictest26604-21ssh21:30:121
433499191,10cyclictest7492-21sshd23:40:391
433499191,10cyclictest0-21swapper/120:25:271
4334991910,6cyclictest31897-21ssh21:15:151
433499190,17cyclictest25241-21sed20:05:211
432199199,7cyclictest27468-21entropy23:00:140
432199192,3cyclictest131rcu_preempt23:20:420
432199192,14cyclictest151rcuc/020:10:100
432199192,13cyclictest13668-21sed21:00:200
4321991912,4cyclictest22468-21latency_hist21:10:010
432199191,13cyclictest24795-21sed00:10:200
435499187,6cyclictest0-21swapper/321:53:583
435499186,9cyclictest0-21swapper/321:25:143
435499183,12cyclictest0-21swapper/322:10:183
435499182,10cyclictest35966irq/58-eth023:15:123
435499181,11cyclictest0-21swapper/320:10:023
433499189,7cyclictest5964-21apt-get00:00:101
433499184,12cyclictest10311-21ntp_kernel_pll_20:20:241
433499184,11cyclictest21908-21irqrtprio20:35:191
433499181,9cyclictest16490-21/usr/sbin/munin20:30:171
4334991812,4cyclictest0-21swapper/120:55:131
433499181,15cyclictest714-21latency_hist19:40:021
4334991811,5cyclictest22948-21ssh22:02:471
4334991811,5cyclictest20130-21ssh21:25:231
433499181,14cyclictest5784-21latency_hist00:00:021
433499181,11cyclictest29736-21cpuspeed20:10:131
4334991810,6cyclictest23501-21cat00:25:331
432199184,10cyclictest0-21swapper/020:00:110
432199183,13cyclictest0-21swapper/020:05:180
432199183,11cyclictest0-21swapper/019:15:130
432199183,11cyclictest0-21swapper/019:15:130
432199181,7cyclictest4992-21sed19:10:180
4321991811,5cyclictest28256-21sshd19:31:310
432199181,14cyclictest10859-21tr20:55:200
435499179,6cyclictest0-21swapper/323:21:113
435499179,6cyclictest0-21swapper/321:42:493
435499172,10cyclictest0-21swapper/321:05:053
435499172,10cyclictest0-21swapper/320:55:123
435499172,10cyclictest0-21swapper/320:50:133
435499172,10cyclictest0-21swapper/320:15:133
435499171,12cyclictest0-21swapper/321:45:193
433499179,5cyclictest14204-21cut21:00:191
433499178,7cyclictest6153-21sed19:45:191
433499178,6cyclictest20677-21kthreadcore21:05:221
433499172,10cyclictest0-21swapper/119:20:131
433499171,8cyclictest2304-21kthreadcore19:40:201
433499171,13cyclictest13147-21unixbench_singl19:15:281
433499171,13cyclictest13147-21unixbench_singl19:15:271
433499171,13cyclictest11824-21kthreadcore19:50:201
433499171,13cyclictest0-21swapper/120:00:131
432199172,13cyclictest15623-21kthreadcore19:20:190
432199172,10cyclictest0-21swapper/020:40:140
432199171,13cyclictest2438-21grep20:45:280
432199171,13cyclictest2438-21grep20:45:270
432199171,13cyclictest16729-21grep20:30:160
435499162,9cyclictest0-21swapper/319:10:143
435499162,10cyclictest0-21swapper/320:45:133
435499162,10cyclictest0-21swapper/320:45:123
435499162,10cyclictest0-21swapper/320:35:143
435499162,10cyclictest0-21swapper/319:55:153
435499162,10cyclictest0-21swapper/319:40:133
433499161,11cyclictest5976-21kthreadcore20:50:211
432199168,5cyclictest29976-21cut19:35:200
432199161,12cyclictest17289-21tr19:55:210
432199161,11cyclictest6760-21latency_hist20:20:020
432199161,11cyclictest26689-21df21:30:140
435499152,8cyclictest0-21swapper/320:30:133
435499151,8cyclictest0-21swapper/320:20:133
435499151,11cyclictest0-21swapper/319:45:243
433499151,11cyclictest31111-21grep20:45:151
433499151,11cyclictest31111-21grep20:45:151
433499151,11cyclictest19486-21irqrtprio19:25:181
435499149,3cyclictest0-21swapper/319:25:253
435499149,3cyclictest0-21swapper/319:15:143
435499149,3cyclictest0-21swapper/319:15:133
433499141,9cyclictest4369-21latency_hist19:10:011
435499133,8cyclictest0-21swapper/320:25:133
435499131,7cyclictest0-21swapper/319:30:133
432199131,9cyclictest4322-21latency_hist19:10:020
435499125,4cyclictest0-21swapper/320:10:103
435499125,3cyclictest0-21swapper/319:35:133
435499122,6cyclictest0-21swapper/319:50:133
435499121,6cyclictest0-21swapper/321:00:293
435499101,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional