You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-11-06 - 23:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Fri Nov 06, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19285997922,10cyclictest131rcu_preempt10:30:120
1930599663,39cyclictest19538-21ssh10:30:132
19294996552,8cyclictest31850-21ssh10:55:131
1928599565,41cyclictest29689-21ssh11:30:130
1928599563,40cyclictest17293-21ssh11:05:130
1928599563,40cyclictest17293-21ssh11:05:120
1930599485,29cyclictest14040-21ssh10:45:132
1931499447,25cyclictest33366irq/58-eth011:55:123
1929499444,32cyclictest9505-21ssh11:55:131
1928599441,4cyclictest131rcu_preempt10:15:130
19314994219,13cyclictest33366irq/58-eth010:25:143
1931499421,25cyclictest33366irq/58-eth010:40:113
1931499415,27cyclictest33366irq/58-eth011:30:123
1929499411,34cyclictest8929-21ssh09:30:121
1929499410,32cyclictest11723-21ssh11:20:111
1928599412,31cyclictest31859-21ssh10:55:130
1929499402,31cyclictest1646-21ssh10:20:121
19285994018,13cyclictest131rcu_preempt08:15:130
1930599392,21cyclictest30376-21ssh10:00:132
1930599392,18cyclictest12560-21ntp_192.168.11409:30:242
19285993921,8cyclictest131rcu_preempt07:30:120
19314993831,4cyclictest33366irq/58-eth010:20:133
1930599382,28cyclictest3395-21sh09:45:132
1928599385,24cyclictest12-21ksoftirqd/010:35:140
1931499372,21cyclictest33366irq/58-eth010:05:113
1930599372,25cyclictest29674-21ssh11:30:122
1930599372,18cyclictest6633-21ssh10:58:242
1928599371,26cyclictest21376-21ssh09:55:130
1930599363,25cyclictest0-21swapper/210:25:242
1929499364,24cyclictest6251-21ssh11:35:141
1929499362,18cyclictest29210-21sh11:29:561
1928599366,22cyclictest12-21ksoftirqd/012:00:120
19314993510,18cyclictest0-21swapper/309:23:243
1930599353,17cyclictest28567-21ssh10:35:142
19305993511,15cyclictest0-21swapper/211:55:132
1928599351,30cyclictest22921-21ssh10:50:120
1931499344,17cyclictest33366irq/58-eth011:05:143
1931499344,17cyclictest33366irq/58-eth011:05:143
19314993418,10cyclictest0-21swapper/310:50:123
19294993421,8cyclictest30504-21rm12:25:121
1928599343,20cyclictest3397-21ssh09:45:130
1931499331,16cyclictest33366irq/58-eth011:25:123
1930599334,16cyclictest26885-21ssh09:40:132
1930599331,20cyclictest17920-21ssh09:35:122
1931499329,12cyclictest0-21swapper/312:05:133
1931499329,12cyclictest0-21swapper/312:05:123
1931499324,16cyclictest33366irq/58-eth011:45:133
1931499321,19cyclictest33366irq/58-eth009:25:123
1930599326,16cyclictest22346irq/40-dwc2_hso08:45:122
1928599324,21cyclictest17904-21ssh09:35:120
1928599322,22cyclictest9510-21ssh11:55:140
1928599312,21cyclictest27394-21ssh12:05:140
1928599312,21cyclictest27394-21ssh12:05:140
1930599303,16cyclictest1661-21ssh10:20:142
1928599302,22cyclictest18772-21ssh10:29:350
1930599293,21cyclictest0-21swapper/209:24:242
1930599293,20cyclictest0-21swapper/211:45:042
19305992910,3cyclictest27-21ksoftirqd/212:34:192
19294992914,10cyclictest25992-21ssh10:32:241
1931499283,17cyclictest33366irq/58-eth012:20:133
1931499282,11cyclictest33366irq/58-eth009:10:113
1930599282,21cyclictest15957-21sh12:35:132
1930599282,19cyclictest0-21swapper/209:00:132
19314992712,12cyclictest33366irq/58-eth011:35:143
19314992711,10cyclictest33366irq/58-eth010:32:243
1930599278,4cyclictest27-21ksoftirqd/210:16:122
1930599276,6cyclictest27-21ksoftirqd/212:15:542
1930599273,7cyclictest0-21swapper/211:05:132
1930599273,7cyclictest0-21swapper/211:05:132
1930599273,15cyclictest22346irq/40-dwc2_hso12:25:132
1930599273,14cyclictest22346irq/40-dwc2_hso07:20:132
19305992713,8cyclictest21356-21apt-get09:55:132
19305992713,5cyclictest27-21ksoftirqd/211:40:002
19305992710,13cyclictest8544-21cron08:40:012
19305992710,13cyclictest8544-21cron08:40:002
1929499273,15cyclictest0-21swapper/110:00:121
1929499272,19cyclictest14159-21rm12:33:251
1928599272,19cyclictest14158-21sh12:33:250
19285992714,7cyclictest4981-21diskmemload09:20:140
1928599270,3cyclictest131rcu_preempt12:20:120
19314992616,7cyclictest33366irq/58-eth009:15:133
1931499261,2cyclictest0-21swapper/311:20:123
1931499260,12cyclictest33366irq/58-eth007:45:133
1930599268,8cyclictest27-21ksoftirqd/212:22:262
1930599265,12cyclictest22346irq/40-dwc2_hso07:30:122
1930599263,16cyclictest22346irq/40-dwc2_hso08:20:132
1930599262,19cyclictest0-21swapper/208:40:132
1930599261,16cyclictest17940-21ssh10:10:192
1929499263,13cyclictest21613-21ssh12:20:141
1929499262,19cyclictest1-21systemd11:54:581
1928599267,12cyclictest0-21swapper/011:40:110
1928599265,16cyclictest4981-21diskmemload10:45:130
1928599262,19cyclictest12447-21ssh09:50:120
1931499252,12cyclictest33366irq/58-eth009:50:113
1930599259,3cyclictest0-21swapper/209:10:522
1930599254,4cyclictest27-21ksoftirqd/212:10:162
1930599252,18cyclictest32375-21diskmemload09:25:122
1930599252,18cyclictest23649-21ssh11:44:582
1930599251,21cyclictest645-21ls09:05:172
19305992511,6cyclictest5483-21users11:50:342
1930599251,13cyclictest8377-21ssh11:00:142
1930599251,13cyclictest8377-21ssh11:00:142
19294992512,11cyclictest6063-21cron07:30:011
1929499251,18cyclictest6923-21ssh10:05:131
1928599256,12cyclictest12-21ksoftirqd/011:10:110
1928599256,12cyclictest12-21ksoftirqd/011:10:110
1928599253,17cyclictest24108-21ssh11:45:130
1931499245,9cyclictest33366irq/58-eth008:05:133
1931499241,11cyclictest33366irq/58-eth008:15:133
1930599249,3cyclictest27-21ksoftirqd/211:24:212
1930599247,7cyclictest27-21ksoftirqd/212:03:542
1930599246,11cyclictest6893-21apt-get10:05:122
1930599242,19cyclictest0-21swapper/207:50:142
1930599242,16cyclictest0-21swapper/211:10:062
1930599242,16cyclictest0-21swapper/211:10:052
1930599242,15cyclictest27-21ksoftirqd/209:54:382
1930599242,13cyclictest18141-21mailstats09:15:282
19305992412,9cyclictest0-21swapper/207:40:222
1929499249,10cyclictest19277-21cyclictest12:19:241
1929499243,16cyclictest5374-21ssh09:10:101
1928599242,17cyclictest14334-21rm09:15:140
1931499231,2cyclictest0-21swapper/311:15:113
1931499231,2cyclictest0-21swapper/311:15:103
1931499231,13cyclictest33366irq/58-eth009:35:123
1930599237,8cyclictest27-21ksoftirqd/210:42:282
1930599233,13cyclictest22346irq/40-dwc2_hso08:50:132
1930599232,15cyclictest0-21swapper/210:50:132
19305992312,8cyclictest24869-21grep11:25:292
1929499233,13cyclictest14312-21ssh10:25:261
1929499232,16cyclictest4998-21ssh10:40:101
19294992312,8cyclictest21-21ksoftirqd/111:10:131
19294992312,8cyclictest21-21ksoftirqd/111:10:121
1929499231,16cyclictest5328-21sh11:34:241
1928599232,14cyclictest6078-21ssh11:35:030
1931499223,15cyclictest33366irq/58-eth009:55:133
1931499222,16cyclictest33366irq/58-eth008:30:143
19314992211,7cyclictest0-21swapper/312:15:153
19314992210,9cyclictest0-21swapper/310:48:483
1930599227,12cyclictest27-21ksoftirqd/212:05:022
1930599227,12cyclictest27-21ksoftirqd/212:05:022
1930599226,11cyclictest27-21ksoftirqd/207:35:162
1930599225,4cyclictest27-21ksoftirqd/211:18:532
1930599225,4cyclictest27-21ksoftirqd/211:18:532
1930599222,13cyclictest0-21swapper/208:05:132
1930599221,11cyclictest24373-21cut07:15:162
19305992210,8cyclictest1300-21sshd07:55:232
1929499223,13cyclictest0-21swapper/108:25:161
19294992212,7cyclictest22910-21apt-get10:50:131
1929499221,18cyclictest26653-21ls08:25:021
19294992211,4cyclictest0-21swapper/107:40:101
1928599222,18cyclictest1565-21ntp_192.168.11410:00:230
1928599221,16cyclictest753-21sendmail-mta09:05:130
1931499219,9cyclictest0-21swapper/312:38:333
1931499215,10cyclictest0-21swapper/312:10:133
1931499213,12cyclictest0-21swapper/312:33:253
1931499213,10cyclictest33366irq/58-eth011:40:123
19314992113,5cyclictest0-21swapper/311:00:003
1930599213,12cyclictest27-21ksoftirqd/207:10:252
1930599212,4cyclictest0-21swapper/208:29:232
1930599212,14cyclictest0-21swapper/207:45:142
1930599211,16cyclictest2864-21fschecks_count08:00:142
1930599211,11cyclictest19325-21latency_hist07:10:022
19305992110,8cyclictest0-21swapper/208:15:212
1930599210,15cyclictest28353-21kworker/u8:108:10:112
1929499212,14cyclictest25905-21ssh09:39:211
1929499212,13cyclictest388-21sendmail-mta09:05:141
1929499211,9cyclictest25069-21ntp_states09:55:271
1929499211,13cyclictest24090-21apt-get11:45:121
1929499210,14cyclictest17145-21ls08:15:021
1928599214,11cyclictest31484-21kworker/u8:109:25:130
1928599212,15cyclictest5008-21ssh10:40:110
19285992112,6cyclictest0-21swapper/008:05:180
1928599211,15cyclictest2867-21ssh11:15:130
1928599211,15cyclictest2867-21ssh11:15:130
19314992014,4cyclictest33-21ksoftirqd/310:00:203
19314992014,4cyclictest0-21swapper/307:50:133
1930599202,15cyclictest26544-21missed_timers08:55:212
1930599201,9cyclictest31760-21fschecks_count08:30:172
1930599201,10cyclictest4870-21needreboot07:25:222
1929499209,8cyclictest1914-21cut09:25:201
1929499202,12cyclictest30625-21ssh09:23:231
1929499201,7cyclictest27752-21ps12:05:181
1929499201,7cyclictest27752-21ps12:05:181
1929499201,16cyclictest18647-21ls08:50:171
1929499201,16cyclictest14013-21apt-get10:45:121
19294992011,5cyclictest6390-21netstat11:15:231
19294992011,5cyclictest6390-21netstat11:15:231
1929499201,11cyclictest25276-21latency_hist07:50:021
1928599202,14cyclictest1757-21df_inode10:20:150
1928599201,15cyclictest22007-21apt-get08:20:120
1928599201,13cyclictest32619-21latency_hist09:05:020
19285992010,8cyclictest5478-21users11:50:340
1928599200,13cyclictest16200-21ls10:10:140
1931499199,7cyclictest0-21swapper/311:10:253
1931499199,7cyclictest0-21swapper/311:10:253
1931499192,11cyclictest0-21swapper/311:50:153
19314991913,4cyclictest0-21swapper/310:10:213
1931499191,12cyclictest0-21swapper/308:45:153
1931499191,11cyclictest0-21swapper/312:30:023
1931499191,11cyclictest0-21swapper/307:35:163
19314991910,6cyclictest0-21swapper/309:30:303
1929499197,10cyclictest21-21ksoftirqd/109:15:121
1929499193,6cyclictest131rcu_preempt11:00:131
1929499193,6cyclictest131rcu_preempt11:00:131
1929499193,11cyclictest21-21ksoftirqd/112:14:251
1929499192,12cyclictest11508-21ssh09:49:221
1929499191,8cyclictest25022-21/usr/sbin/munin10:15:261
1929499191,16cyclictest17365-21cpuspeed11:05:141
1929499191,16cyclictest17365-21cpuspeed11:05:141
1929499191,15cyclictest19504-21cut07:10:161
1929499191,14cyclictest17179-21find08:15:111
1929499191,13cyclictest26886-21diskmemload09:40:131
19294991910,4cyclictest15820-21sshd12:35:051
1928599198,7cyclictest12785-21sh12:15:150
1928599191,4cyclictest30485-21apt-get12:25:130
19285991911,5cyclictest0-21swapper/007:40:280
1928599191,10cyclictest23322-21kthreadcore11:25:210
19285991910,6cyclictest0-21swapper/008:35:240
19285991910,6cyclictest0-21swapper/008:35:240
1931499189,6cyclictest0-21swapper/310:37:473
1931499183,10cyclictest0-21swapper/312:03:243
1931499183,10cyclictest0-21swapper/310:15:053
1931499182,11cyclictest33366irq/58-eth009:45:133
1931499182,10cyclictest0-21swapper/308:55:123
1931499182,10cyclictest0-21swapper/308:40:133
1931499182,10cyclictest0-21swapper/307:20:133
19314991812,4cyclictest0-21swapper/311:00:213
19314991812,4cyclictest0-21swapper/311:00:213
1929499189,5cyclictest19216-21kthreadcore12:00:221
1929499188,7cyclictest698-21cat10:35:301
1929499182,9cyclictest0-21swapper/110:10:121
1929499182,14cyclictest14270-21kthreadcore07:35:211
1929499182,11cyclictest16046-21ntp_192.168.11409:50:221
19294991812,4cyclictest7560-21mailstats08:35:271
19294991812,4cyclictest7560-21mailstats08:35:271
19294991811,5cyclictest12091-21sshd08:09:241
1928599189,7cyclictest18015-21cut12:35:210
1928599189,6cyclictest9667-21kthreadcore09:30:180
1928599183,10cyclictest0-21swapper/007:55:140
1928599182,13cyclictest15365-21ntp_192.168.11411:20:240
1928599181,7cyclictest1252-21cron07:25:020
1928599181,4cyclictest0-21swapper/012:10:080
1928599181,3cyclictest131rcu_preempt09:10:260
1928599181,14cyclictest29934-21kthreadcore08:25:240
1928599181,13cyclictest8482-21ssh11:00:150
1928599181,13cyclictest8482-21ssh11:00:140
1928599181,10cyclictest22448-21kthreadcore07:10:220
1931499172,12cyclictest0-21swapper/309:40:153
1931499172,10cyclictest0-21swapper/307:40:113
1929499179,5cyclictest4948-21cut08:00:191
1929499179,5cyclictest11022-21latency_hist07:35:021
1929499177,7cyclictest15861-21irqstats11:40:181
1929499173,11cyclictest23731-21sed08:55:161
1929499172,3cyclictest1407246sed08:45:171
1929499172,12cyclictest31089-21tr09:00:201
1929499172,10cyclictest0-21swapper/108:30:151
1929499171,13cyclictest30380-21ls07:55:141
1928599179,5cyclictest0-21swapper/007:45:180
1928599176,9cyclictest18473-21sort08:50:140
1928599176,8cyclictest28852-21cat07:20:010
1928599174,10cyclictest9131-21tr10:05:200
1928599173,4cyclictest131rcu_preempt08:40:170
1928599172,12cyclictest12100-21apt-get08:10:030
1928599172,10cyclictest0-21swapper/008:55:110
1928599171,3cyclictest0-21swapper/007:37:240
1931499162,9cyclictest0-21swapper/307:30:133
1929499163,11cyclictest24246-21/usr/sbin/munin07:15:291
1929499163,10cyclictest27385-21kthreadcore07:50:191
1929499163,10cyclictest10545-21kthreadcore08:40:201
1928599169,5cyclictest0-21swapper/008:30:130
1928599162,12cyclictest28146-21cut07:50:210
1928599161,7cyclictest26997-21diskstats09:40:150
1928599161,5cyclictest13781-21fschecks_time08:45:160
1928599161,3cyclictest0-21swapper/008:00:160
1928599161,11cyclictest1823-21cut07:25:160
1931499153,8cyclictest0-21swapper/309:05:143
1931499151,11cyclictest0-21swapper/307:25:183
1931499151,10cyclictest0-21swapper/307:10:263
1929499152,10cyclictest19302-21latency_hist07:10:021
1929499151,10cyclictest29822-21kthreadcore07:20:181
1931499149,3cyclictest0-21swapper/308:25:163
1931499149,3cyclictest0-21swapper/308:02:273
1931499141,6cyclictest0-21swapper/309:00:203
1928599141,10cyclictest19277-21cyclictest07:10:010
1931499131,9cyclictest0-21swapper/308:50:163
1931499131,10cyclictest0-21swapper/307:15:293
1931499127,3cyclictest0-21swapper/308:35:233
1931499127,3cyclictest0-21swapper/308:35:223
1931499122,8cyclictest0-21swapper/308:20:123
1931499122,6cyclictest0-21swapper/307:55:143
1931499121,8cyclictest0-21swapper/308:10:103
1931499101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional