You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-03 - 04:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Jul 03, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3122499674,3cyclictest27346irq/40-dwc2_hso19:50:142
3122499642,56cyclictest0-21swapper/223:05:122
3122499634,4cyclictest0-21swapper/200:15:122
3120799614,48cyclictest15921-21ssh23:15:140
3122499573,4cyclictest0-21swapper/219:15:132
3122499561,49cyclictest0-21swapper/221:00:142
3122499521,30cyclictest5590-21ssh23:45:152
3122499511,3cyclictest0-21swapper/222:55:112
3120799504,6cyclictest131rcu_preempt19:10:140
3123199493,30cyclictest35966irq/58-eth023:15:143
3122499497,16cyclictest0-21swapper/200:25:182
3122499484,39cyclictest27346irq/40-dwc2_hso19:30:112
3122499482,4cyclictest0-21swapper/221:15:132
3123199474,27cyclictest35966irq/58-eth022:05:123
3120799473,8cyclictest131rcu_preempt21:00:130
3123199453,25cyclictest35966irq/58-eth022:40:123
3122499451,7cyclictest0-21swapper/220:40:112
3122499441,8cyclictest0-21swapper/222:40:142
3121599442,33cyclictest3015-21ssh21:20:141
3122499434,29cyclictest19667-21ssh22:05:122
3122499432,5cyclictest27346irq/40-dwc2_hso22:25:122
3122499422,4cyclictest0-21swapper/220:25:102
3122499422,3cyclictest14271-21ssh22:20:132
3122499414,11cyclictest27346irq/40-dwc2_hso23:00:192
3122499392,31cyclictest17613-21sh00:09:422
3122499391,5cyclictest6817-21ssh21:40:112
3122499382,8cyclictest5866-21ssh22:15:192
3122499382,31cyclictest24946-21ssh00:12:412
3122499381,7cyclictest3466-21sh22:50:122
3122499381,10cyclictest5224-21latency_hist23:45:012
3121599384,30cyclictest13068-21ssh00:25:171
3121599381,28cyclictest24922-21sh21:50:131
3120799384,24cyclictest23214-21ssh22:25:120
3122499376,4cyclictest8921-21fschecks_count22:35:152
3122499376,4cyclictest8921-21fschecks_count22:35:142
3122499371,8cyclictest28475-21idleruntime-cro23:40:022
3122499371,8cyclictest18168-21latency_hist19:30:032
3122499370,8cyclictest1479-21latency_hist23:25:022
3122499370,8cyclictest13585-21kthreadcore21:05:202
3122499370,8cyclictest10422-21apt-config23:30:022
3123199362,19cyclictest35966irq/58-eth019:25:123
31231993617,12cyclictest35966irq/58-eth000:30:143
3122499361,9cyclictest15833-21latency_hist20:35:012
3122499361,8cyclictest2546-21kthreadcore19:45:202
3122499361,7cyclictest32575-21/usr/sbin/munin00:00:262
3122499361,7cyclictest10636-21cpuspeed22:00:152
3122499360,9cyclictest32110-21grep20:50:192
3122499360,10cyclictest23124-21fschecks_count19:35:162
3120799363,23cyclictest3923-21ssh00:20:130
31207993610,8cyclictest131rcu_preempt23:05:130
3123199352,21cyclictest35966irq/58-eth019:45:133
3123199352,20cyclictest35966irq/58-eth000:35:123
3122499355,4cyclictest3891-21unixbench_singl22:30:302
3122499355,4cyclictest3891-21unixbench_singl22:30:292
3122499351,9cyclictest0-21swapper/220:10:122
3122499351,8cyclictest31351-21/usr/sbin/munin19:10:252
3122499351,8cyclictest2686-21/usr/sbin/munin20:55:282
3122499351,8cyclictest2686-21/usr/sbin/munin20:55:272
3122499351,8cyclictest18783-21usage_Anybus23:50:322
3122499351,7cyclictest7072-21ls21:20:302
3122499351,7cyclictest29025-21/usr/sbin/munin20:15:252
3122499351,5cyclictest13368-21grep19:55:252
3122499351,10cyclictest5509-21timerandwakeup21:55:292
3122499350,8cyclictest27213-21ssh23:55:262
3122499350,8cyclictest21929-21tr20:05:202
31215993522,7cyclictest21-21ksoftirqd/120:00:121
3120799356,16cyclictest19652-21ssh22:05:110
3120799354,4cyclictest131rcu_preempt23:25:140
3123199345,19cyclictest35966irq/58-eth023:20:123
3122499342,5cyclictest1979-21grep21:35:322
3122499341,9cyclictest16574-21uname21:45:202
3122499341,9cyclictest15521-21cron23:15:022
3122499341,5cyclictest3889-21/usr/sbin/munin00:20:182
3122499340,9cyclictest30593-21ntpq22:45:272
3122499340,9cyclictest29808-21awk21:35:012
3121599345,23cyclictest30391-21ssh23:05:111
3120799341,21cyclictest29444-21kworker/u8:120:25:100
3122499331,8cyclictest10360-21apt-get23:30:042
3122499331,7cyclictest22057-21ssh00:30:132
3120799337,15cyclictest131rcu_preempt21:50:130
3123199322,14cyclictest35966irq/58-eth000:10:133
3122499327,22cyclictest30098-21sshd20:49:422
3122499323,8cyclictest11009-21kthreadcore19:20:212
3122499321,9cyclictest32374-21ssh22:10:252
3122499321,9cyclictest27222-21kthreadcore21:50:212
3122499321,7cyclictest31108-21entropy00:35:162
3122499321,7cyclictest19926-21timerandwakeup23:15:302
3122499321,5cyclictest23866-21ssh21:12:272
3123199311,17cyclictest35966irq/58-eth021:50:123
3122499310,8cyclictest4556-21kthreadcore20:20:232
3121599313,21cyclictest17339-21ssh21:10:121
31215993121,5cyclictest0-21swapper/119:50:131
3121599311,22cyclictest14291-21ssh22:20:141
31207993111,3cyclictest131rcu_preempt19:40:130
3120799310,3cyclictest131rcu_preempt19:35:110
3123199308,19cyclictest0-21swapper/300:00:133
3123199304,14cyclictest0-21swapper/321:40:133
3123199303,16cyclictest35966irq/58-eth023:10:113
3122499304,6cyclictest19655-21timerandwakeup20:35:292
3122499303,18cyclictest27346irq/40-dwc2_hso21:25:142
3122499300,9cyclictest14681-21cat20:00:122
3121599302,22cyclictest21208-21ssh21:30:151
31215993014,9cyclictest31200-21cyclictest23:25:131
3121599301,26cyclictest14640-21ssh23:50:141
3121599301,23cyclictest9231-21ssh00:05:141
3120799302,22cyclictest2056-21ssh00:18:180
3120799300,4cyclictest131rcu_preempt21:20:130
3123199293,14cyclictest35966irq/58-eth000:20:143
3123199292,16cyclictest35966irq/58-eth000:05:143
3121599294,8cyclictest21-21ksoftirqd/122:40:121
3121599292,21cyclictest31074-21rm23:58:421
3120799291,19cyclictest30238-21ssh21:35:160
3122499281,8cyclictest31313-21proc_pri19:40:252
3120799289,15cyclictest32629-21diskmemload00:00:130
3120799282,18cyclictest1537-21ssh21:55:130
31207992813,3cyclictest131rcu_preempt21:45:120
3123199275,10cyclictest35966irq/58-eth022:30:123
3123199275,10cyclictest35966irq/58-eth022:30:123
3121599277,10cyclictest0-21swapper/123:00:171
3121599272,19cyclictest18287-21ssh00:10:141
3121599272,18cyclictest664-21ntpd23:10:101
31215992713,5cyclictest21-21ksoftirqd/121:10:011
3121599271,19cyclictest10842-21ssh23:30:141
3120799271,20cyclictest12458-21ssh22:55:140
3120799271,17cyclictest32204-21ssh22:30:130
3120799271,17cyclictest32204-21ssh22:30:130
3121599265,11cyclictest1516-21sh21:55:131
31207992610,4cyclictest131rcu_preempt22:10:110
3123199252,18cyclictest35966irq/58-eth022:10:113
3123199252,12cyclictest35966irq/58-eth021:10:103
3123199251,15cyclictest35966irq/58-eth023:05:133
3121599255,4cyclictest131rcu_preempt00:00:131
3121599253,4cyclictest131rcu_preempt23:45:161
3121599252,18cyclictest10559-21ssh22:00:131
31215992510,4cyclictest21-21ksoftirqd/100:40:011
3120799251,18cyclictest6921-21ssh23:10:100
3120799251,17cyclictest21506-21ssh23:00:190
3123199242,15cyclictest35966irq/58-eth021:05:133
3123199241,19cyclictest35966irq/58-eth023:35:143
3123199241,16cyclictest35966irq/58-eth021:45:133
3121599242,18cyclictest5283-21ssh22:15:141
31215992416,5cyclictest21-21ksoftirqd/123:40:131
31215992413,5cyclictest0-21swapper/120:35:131
3120799243,4cyclictest20273-21irqcore23:35:190
3120799242,4cyclictest0-21swapper/023:50:140
3120799241,15cyclictest12-21ksoftirqd/020:00:140
31207992411,3cyclictest131rcu_preempt22:15:180
31207992410,7cyclictest31200-21cyclictest00:05:120
3123199234,15cyclictest35966irq/58-eth023:50:143
3123199233,13cyclictest35966irq/58-eth022:50:133
31231992312,8cyclictest35966irq/58-eth023:00:183
3121599232,3cyclictest131rcu_preempt20:30:251
3121599231,16cyclictest8785-21ssh22:35:131
3121599231,16cyclictest8785-21ssh22:35:121
3120799239,4cyclictest0-21swapper/019:55:220
3120799232,15cyclictest23615-21ssh23:55:140
3120799231,8cyclictest11744-21kthreadcore23:30:190
3123199223,11cyclictest35966irq/58-eth022:45:143
3123199222,14cyclictest0-21swapper/321:15:143
31231992213,6cyclictest35966irq/58-eth023:30:063
3121599222,15cyclictest22058-21ssh00:30:131
31215992217,3cyclictest0-21swapper/120:50:241
31215992216,4cyclictest0-21swapper/122:25:281
31215992212,4cyclictest21-21ksoftirqd/122:50:171
31215992211,9cyclictest0-21swapper/100:22:281
3121599221,18cyclictest26921-21acpi22:45:121
3121599221,17cyclictest4886-21latency_hist22:15:011
3121599221,16cyclictest26401-21ssh21:15:131
31215992211,4cyclictest21-21ksoftirqd/119:20:001
3120799224,11cyclictest0-21swapper/023:40:120
3120799223,9cyclictest11407-21kthreadcore22:35:220
3120799223,9cyclictest11407-21kthreadcore22:35:210
3120799221,5cyclictest131rcu_preempt22:00:130
3123199217,9cyclictest35966irq/58-eth022:25:133
3123199217,6cyclictest0-21swapper/320:50:133
3123199215,9cyclictest0-21swapper/322:15:143
3123199213,9cyclictest0-21swapper/323:25:143
3123199213,13cyclictest35966irq/58-eth019:55:143
3123199213,12cyclictest0-21swapper/322:20:143
3123199213,12cyclictest0-21swapper/300:25:173
31231992113,4cyclictest33-21ksoftirqd/320:55:133
31231992113,4cyclictest33-21ksoftirqd/320:55:123
31231992112,6cyclictest35966irq/58-eth023:45:163
31231992111,7cyclictest35966irq/58-eth023:40:133
3121599217,4cyclictest131rcu_preempt19:35:191
3121599215,7cyclictest21-21ksoftirqd/122:05:291
3121599215,4cyclictest131rcu_preempt20:20:141
3121599212,4cyclictest21-21ksoftirqd/121:45:211
3121599211,17cyclictest8693-21cut19:20:151
3121599211,16cyclictest27294-21ssh00:15:151
3121599211,14cyclictest12109-21ssh21:25:131
3121599210,6cyclictest131rcu_preempt22:32:261
3121599210,6cyclictest131rcu_preempt22:32:251
3120799216,6cyclictest12-21ksoftirqd/020:35:020
3120799214,13cyclictest26950-21ssh22:45:140
3120799213,8cyclictest16142-21grep20:35:160
3120799212,16cyclictest31295-21kthreadcore20:15:180
3120799212,12cyclictest0-21swapper/020:45:120
31207992112,6cyclictest0-21swapper/019:45:260
3120799211,17cyclictest21591-21tr21:30:180
3120799211,13cyclictest12064-21latency_hist22:55:010
3123199209,6cyclictest0-21swapper/321:23:403
3121599209,3cyclictest21-21ksoftirqd/121:38:211
3121599209,3cyclictest0-21swapper/121:44:321
3121599206,3cyclictest131rcu_preempt23:23:161
3121599206,3cyclictest131rcu_preempt23:15:231
31215992012,5cyclictest0-21swapper/119:10:291
3121599201,14cyclictest19880-21ssh23:35:141
31215992010,4cyclictest21-21ksoftirqd/122:58:531
3120799202,4cyclictest53646usb-storage19:30:050
3120799202,15cyclictest26384-21apt-get21:15:130
3120799201,7cyclictest2492-21grep00:35:280
31207992013,5cyclictest0-21swapper/022:20:160
3120799200,6cyclictest131rcu_preempt23:45:140
3123199199,8cyclictest0-21swapper/321:59:073
3123199193,9cyclictest0-21swapper/323:58:423
3123199193,9cyclictest0-21swapper/300:18:193
3123199192,13cyclictest0-21swapper/322:35:123
3123199192,13cyclictest0-21swapper/322:35:113
3123199192,11cyclictest0-21swapper/320:20:143
31231991912,4cyclictest0-21swapper/322:00:223
31231991911,5cyclictest35966irq/58-eth021:35:143
3123199191,11cyclictest35966irq/58-eth020:15:103
3122499191,9cyclictest31258-21latency_hist19:10:022
3121599195,6cyclictest21-21ksoftirqd/120:40:261
3121599194,4cyclictest131rcu_preempt20:45:291
3121599191,2cyclictest131rcu_preempt19:25:111
31215991912,5cyclictest14425-21sshd19:57:361
3120799193,13cyclictest11621-21apt-get21:25:040
3120799193,10cyclictest0-21swapper/000:12:410
3120799192,9cyclictest25002-21ssh23:20:120
3120799191,9cyclictest6323-21latency_hist20:25:030
3120799191,8cyclictest20370-21kthreadcore20:05:180
31207991911,6cyclictest16939-21cron21:10:010
3120799191,15cyclictest6642-21kthreadcore19:15:210
3123199189,6cyclictest0-21swapper/322:55:093
3123199189,6cyclictest0-21swapper/321:31:283
3123199189,6cyclictest0-21swapper/321:25:523
3123199182,10cyclictest0-21swapper/320:35:133
3123199182,10cyclictest0-21swapper/320:25:053
3123199182,10cyclictest0-21swapper/319:30:053
3123199182,10cyclictest0-21swapper/319:15:133
3123199182,10cyclictest0-21swapper/319:10:153
3123199181,13cyclictest0-21swapper/320:00:113
3123199181,13cyclictest0-21swapper/319:35:153
3121599183,3cyclictest131rcu_preempt19:45:191
3121599183,11cyclictest0-21swapper/120:25:061
3121599181,7cyclictest21-21ksoftirqd/119:40:201
31215991813,3cyclictest0-21swapper/121:00:151
31215991811,5cyclictest0-21swapper/120:10:131
31215991810,6cyclictest7254-21sshd20:55:431
31215991810,6cyclictest7254-21sshd20:55:421
31215991810,5cyclictest24027-21sh20:10:011
3120799186,9cyclictest5064-21acpi19:50:130
3120799184,6cyclictest131rcu_preempt20:10:120
3120799181,4cyclictest131rcu_preempt21:10:240
3120799181,4cyclictest10684-21awk21:40:290
31207991813,3cyclictest0-21swapper/020:50:150
3120799181,15cyclictest26866-21users00:30:340
3120799181,12cyclictest16797-21ssh00:25:300
3120799181,10cyclictest23273-21cut20:40:210
3120799180,14cyclictest20479-21cut22:40:200
3123199172,10cyclictest0-21swapper/321:00:143
3123199172,10cyclictest0-21swapper/319:20:153
31215991711,4cyclictest0-21swapper/119:30:111
3120799172,2cyclictest131rcu_preempt19:20:280
3123199162,9cyclictest0-21swapper/320:10:123
3123199161,12cyclictest0-21swapper/319:50:143
3121599161,12cyclictest31315-21latency_hist19:10:011
3121599161,12cyclictest29870-21tr20:15:181
3120799164,9cyclictest31276-21latency_hist19:10:010
3120799161,4cyclictest0-21swapper/019:27:100
3120799160,6cyclictest2752-21grep20:55:140
3120799160,6cyclictest2752-21grep20:55:130
3123199152,9cyclictest0-21swapper/319:45:023
3123199151,12cyclictest0-21swapper/320:05:173
3123199138,3cyclictest0-21swapper/320:30:153
3123199121,9cyclictest0-21swapper/320:45:253
3123199111,7cyclictest0-21swapper/319:10:013
3123199111,5cyclictest0-21swapper/320:40:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional