You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-17 - 14:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Jun 17, 2026 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2195499693,52cyclictest13694-21ssh12:10:112
21954996521,21cyclictest27-21ksoftirqd/210:30:132
21954995938,15cyclictest0-21swapper/210:15:122
2193799573,45cyclictest13709-21ssh12:10:120
2193799528,6cyclictest131rcu_preempt07:30:130
2195499513,40cyclictest21329-21ssh10:10:102
2193799491,38cyclictest26161-21ssh09:55:110
21964994841,5cyclictest36066irq/59-eth011:30:123
2194699473,35cyclictest17563-21sh09:15:111
2194699465,30cyclictest18782-21rm11:55:121
21937994629,12cyclictest0-21swapper/011:20:110
2194699434,32cyclictest4597-21ssh12:05:121
2193799432,27cyclictest844-21munin-node12:15:130
21946994115,21cyclictest21-21ksoftirqd/110:35:151
2193799411,29cyclictest30526-21ssh10:15:120
2195499401,10cyclictest24922-21fschecks_time08:20:162
2196499393,26cyclictest35966irq/58-eth009:10:133
2195499391,9cyclictest27787-21kthreadcore11:05:222
21964993820,11cyclictest0-21swapper/309:00:133
2193799381,30cyclictest29716-21ssh10:50:130
2196499372,23cyclictest35966irq/58-eth009:35:143
21946993713,12cyclictest21-21ksoftirqd/110:45:141
21964993619,11cyclictest35966irq/58-eth010:30:133
2196499361,22cyclictest35966irq/58-eth009:25:113
2195499368,8cyclictest0-21swapper/211:30:272
2195499361,8cyclictest13367-21tr11:15:212
2195499361,10cyclictest9087-21irqcore12:25:182
2194699362,26cyclictest17815-21ssh12:30:101
2193799361,9cyclictest15712-21diskstats11:00:160
2193799361,9cyclictest15712-21diskstats11:00:160
2196499353,18cyclictest35966irq/58-eth010:05:123
2195499352,13cyclictest4324-21ntp_states11:45:262
2195499351,10cyclictest29569-21kthreadcore07:15:202
2195499350,15cyclictest23671-21sh11:20:262
2194699356,17cyclictest23939-21ssh11:40:111
2194699355,5cyclictest131rcu_preempt09:30:141
2193799351,9cyclictest6008-21cut09:05:210
2193799351,9cyclictest24379-21processes10:45:270
2193799351,9cyclictest11560-21kthreadcore11:15:180
2193799351,8cyclictest14866-21/usr/sbin/munin11:35:290
2193799351,6cyclictest4128-21latency_hist07:25:010
21937993515,14cyclictest16944-21sh09:50:130
2196499344,20cyclictest35966irq/58-eth010:40:103
2195499342,13cyclictest2917-21kthreadcore11:10:212
2195499341,9cyclictest8834-21kthreadcore09:45:202
2195499341,9cyclictest8701-21cut08:35:192
2193799347,5cyclictest21809-21irqcore10:10:170
2193799347,3cyclictest10616-21kthreadcore10:20:230
2193799346,6cyclictest23746-21idleruntime07:45:160
2193799344,6cyclictest8169-21grep07:25:280
2193799341,9cyclictest5389-21ntp_192.168.11408:30:240
2193799341,9cyclictest29687-21cut07:50:180
2193799341,9cyclictest23574-21switchtime08:15:280
2193799341,9cyclictest1377-21kthreadcore07:55:190
2193799341,7cyclictest4533-21switchtime11:45:280
2193799341,7cyclictest28326-21ntp_192.168.11411:05:240
2193799341,7cyclictest18044-21kthreadcore10:25:200
2193799341,10cyclictest3187-21cron09:25:010
2193799340,7cyclictest2034-21nfs4_client09:00:250
2195499332,4cyclictest0-21swapper/209:50:132
2195499331,7cyclictest930-21snmpd12:01:072
2194699332,20cyclictest573566ssh11:30:121
2193799338,4cyclictest31014-21kthreadcore11:25:210
2193799338,4cyclictest31014-21kthreadcore11:25:210
2193799331,9cyclictest4073-21ssh09:25:160
2193799331,9cyclictest26999-21diskstats07:15:140
2193799331,9cyclictest15601-21unixbench_singl08:41:050
2193799331,7cyclictest5490-21/usr/sbin/munin08:00:280
2193799331,7cyclictest4063-21kthreadcore10:00:190
2193799331,7cyclictest16724-21grep08:45:180
2193799331,7cyclictest14053-21tune2fs07:35:170
2193799331,10cyclictest4696-21df12:05:140
2193799331,10cyclictest26349-21grep08:55:140
2193799331,10cyclictest10236-21sh11:15:010
2196499322,14cyclictest35966irq/58-eth011:35:123
2195499322,8cyclictest27704-21open_inodes11:40:272
2193799327,4cyclictest8014-21diskmemload11:58:050
2193799324,5cyclictest15882-21sh10:05:270
2193799323,6cyclictest16416-21/usr/sbin/munin09:30:270
2193799321,9cyclictest532-21kthreadcore12:20:190
2193799321,9cyclictest15209-21ssh09:12:430
2193799321,9cyclictest11954-21irqcore10:40:150
2193799321,8cyclictest29362-21ssh10:30:260
2193799321,8cyclictest26417-21latency_hist09:20:010
2193799321,8cyclictest19364-21irqstats07:40:170
2193799321,7cyclictest7437-21uname08:35:170
2193799321,7cyclictest15608-21irqcore08:10:190
2193799321,6cyclictest27576-21/usr/sbin/munin11:40:230
2193799321,10cyclictest23233-21kthreadcore07:10:200
2196499313,17cyclictest35966irq/58-eth009:45:133
2195499313,15cyclictest27346irq/40-dwc2_hso09:20:132
2195499311,8cyclictest9085-21/usr/sbin/munin07:30:322
2194699312,18cyclictest21-21ksoftirqd/108:40:141
2194699310,23cyclictest7813-21rm09:45:131
2193799318,4cyclictest25356-21tr09:35:230
2193799316,5cyclictest12977-21kthreadcore08:05:230
2193799311,9cyclictest5830-21df11:30:140
2193799311,9cyclictest27201-21df_inode12:35:150
2193799311,9cyclictest1091-21cut09:40:190
2193799311,8cyclictest32045-21kthreadcore08:25:220
2193799311,8cyclictest27952-21/usr/sbin/munin12:00:290
2193799311,6cyclictest6040-21ntp_192.168.11410:35:260
2195499304,18cyclictest27346irq/40-dwc2_hso08:10:132
2195499302,19cyclictest7307-21ssh10:20:122
2195499300,7cyclictest21183-21kthreadcore12:30:192
2194699303,23cyclictest9682-21ssh11:50:131
2194699302,22cyclictest20992-21ssh12:13:171
2193799306,2cyclictest0-21swapper/012:31:530
2193799301,8cyclictest21930-21cyclictest07:10:020
2193799301,7cyclictest28310-21/usr/sbin/munin08:20:270
2193799301,6cyclictest7950-21sort09:45:150
2193799301,6cyclictest25963-21vmstat08:50:310
2193799301,10cyclictest6384-21sh10:55:110
2193799301,10cyclictest16510-21ssh12:28:540
2196499295,10cyclictest35966irq/58-eth011:40:113
2196499292,18cyclictest35966irq/58-eth009:20:133
2196499291,17cyclictest35966irq/58-eth008:25:143
2196499291,16cyclictest35966irq/58-eth012:10:123
2195499293,18cyclictest27346irq/40-dwc2_hso12:20:142
2194699295,17cyclictest21-21ksoftirqd/111:25:121
2194699295,17cyclictest21-21ksoftirqd/111:25:121
2194699293,19cyclictest14920-21ssh11:35:141
2193799291,9cyclictest844-21munin-node11:50:110
2196499283,19cyclictest35966irq/58-eth009:15:113
2196499282,11cyclictest35966irq/58-eth010:20:123
21946992814,10cyclictest21930-21cyclictest07:55:121
2196499272,18cyclictest0-21swapper/312:29:283
21964992710,11cyclictest0-21swapper/312:20:143
2195499273,18cyclictest27346irq/40-dwc2_hso09:25:122
2195499273,16cyclictest27346irq/40-dwc2_hso08:30:132
2196499263,14cyclictest35966irq/58-eth009:50:133
2196499262,13cyclictest35966irq/58-eth009:30:153
2195499265,13cyclictest27346irq/40-dwc2_hso07:20:122
2195499264,14cyclictest27346irq/40-dwc2_hso08:25:142
2195499261,7cyclictest853966ssh10:01:182
21954992613,4cyclictest27-21ksoftirqd/207:25:132
2194699262,17cyclictest8014-21diskmemload09:35:141
21946992611,6cyclictest2681566ssh09:20:121
2196499255,15cyclictest35966irq/58-eth007:25:123
2195499255,12cyclictest27346irq/40-dwc2_hso08:45:162
2195499251,7cyclictest2440766ssh09:17:452
2194699256,11cyclictest27104-21ssh12:35:141
2194699252,17cyclictest8014-21diskmemload12:26:161
21946992510,9cyclictest0-21swapper/109:00:131
2196499243,12cyclictest35966irq/58-eth008:20:123
21964992412,9cyclictest35966irq/58-eth011:45:123
21964992412,7cyclictest0-21swapper/312:15:133
2195499243,16cyclictest0-21swapper/210:50:142
2195499241,7cyclictest594766ssh09:43:202
2195499241,7cyclictest2784866ssh09:55:202
2195499241,7cyclictest2301766ssh12:15:152
2195499241,13cyclictest2750-21ssh10:35:192
2194699245,10cyclictest0-21swapper/107:40:121
2194699242,16cyclictest0-21swapper/110:20:131
2196499233,13cyclictest0-21swapper/312:31:173
2196499232,15cyclictest35966irq/58-eth010:35:153
2196499232,12cyclictest0-21swapper/311:00:143
2196499232,12cyclictest0-21swapper/311:00:143
2196499231,13cyclictest35966irq/58-eth011:25:113
2196499231,13cyclictest35966irq/58-eth011:25:113
2195499231,9cyclictest13966-21kthreadcore10:40:192
21954992319,2cyclictest536-21usb-storage11:25:112
21954992319,2cyclictest536-21usb-storage11:25:112
2195499231,5cyclictest2776266rm09:36:402
2194699231,18cyclictest6824-21latency_hist10:20:031
2196499224,11cyclictest35966irq/58-eth012:05:133
2196499223,12cyclictest0-21swapper/312:35:053
2196499221,13cyclictest0-21swapper/309:55:123
2195499225,15cyclictest8622-21tune2fs09:10:162
2195499223,11cyclictest2342-21kthreadcore07:55:192
2195499222,14cyclictest27346irq/40-dwc2_hso11:00:132
2195499222,14cyclictest27346irq/40-dwc2_hso11:00:132
21954992217,3cyclictest14176-21ssh10:05:212
21954992211,9cyclictest16247-21ssh11:52:302
21954992210,5cyclictest9874-21ssh12:05:592
2194699228,6cyclictest21930-21cyclictest09:25:121
2194699228,6cyclictest0-21swapper/109:10:131
2194699228,6cyclictest0-21swapper/108:20:131
2194699221,18cyclictest384-21users12:00:331
2194699221,17cyclictest14330-21ssh11:15:251
2196499211,18cyclictest0-21swapper/311:50:123
2196499211,10cyclictest35966irq/58-eth008:10:143
2195499212,10cyclictest15580-21irqrtprio11:35:192
2195499211,17cyclictest26294-21ls08:55:122
2195499211,16cyclictest14454-21cut09:30:222
2195499211,11cyclictest135-21systemd-journal08:20:012
2195499211,10cyclictest10550-21grep10:55:272
2195499210,10cyclictest3371-21ssh12:40:022
2194699212,15cyclictest0-21swapper/112:15:171
21946992112,6cyclictest13669-21cron07:35:011
2194699211,13cyclictest11466-21ssh10:40:111
2196499203,11cyclictest0-21swapper/309:40:123
2196499202,11cyclictest0-21swapper/310:10:043
21964992011,6cyclictest0-21swapper/311:20:223
2196499201,12cyclictest0-21swapper/308:45:193
2195499209,9cyclictest426-21cron07:55:022
2195499203,9cyclictest22329-21switchtime07:40:272
2195499203,14cyclictest15228-21kthreadcore07:35:212
2195499202,15cyclictest16465-21apt-get10:25:122
2195499201,9cyclictest29445-21latency_hist10:50:012
2195499201,8cyclictest8046-21latency_hist09:10:012
2195499201,10cyclictest23261-21kthreadcore08:50:202
2195499201,10cyclictest22484-21ntp_states11:55:272
2195499201,10cyclictest14035-21grep08:05:292
2195499200,10cyclictest31221-21cut09:00:162
2194699207,3cyclictest131rcu_preempt11:20:281
2194699202,15cyclictest2948-21apt-get10:00:111
21946992010,7cyclictest19995-21cat10:25:231
2196499199,6cyclictest0-21swapper/311:05:213
2196499198,8cyclictest0-21swapper/310:27:433
2196499198,6cyclictest0-21swapper/311:10:393
2196499192,11cyclictest0-21swapper/312:00:043
2195499191,8cyclictest5638-21grep08:00:162
2195499191,8cyclictest23625-21fschecks_count07:45:152
2195499191,15cyclictest22591-21grep07:10:182
2195499191,10cyclictest22049-21latency_hist07:10:022
2195499191,10cyclictest15074-21missed_timers08:40:232
2194699199,7cyclictest14104-21tune2fs07:35:171
2194699198,8cyclictest15825-21ntpq10:05:261
2194699192,14cyclictest3423-21find09:05:111
21946991912,4cyclictest0-21swapper/108:10:251
21946991911,6cyclictest21131-21sshd08:48:471
2194699191,16cyclictest19101-21ls11:00:251
2194699191,16cyclictest19101-21ls11:00:251
21946991911,5cyclictest28892-21usage_Anybus11:05:301
2194699191,14cyclictest0-21swapper/108:35:131
2194699191,13cyclictest1592-21ssh11:10:161
2194699191,12cyclictest5738-21ls08:00:161
2196499189,6cyclictest0-21swapper/310:56:283
2196499183,10cyclictest35966irq/58-eth011:55:143
2196499182,9cyclictest0-21swapper/311:15:133
2196499182,5cyclictest0-21swapper/310:50:133
2196499182,10cyclictest0-21swapper/308:55:123
2196499182,10cyclictest0-21swapper/308:15:133
2196499182,10cyclictest0-21swapper/308:05:163
2196499182,10cyclictest0-21swapper/307:20:133
2196499181,8cyclictest0-21swapper/309:05:113
2196499181,13cyclictest0-21swapper/308:05:013
2196499181,11cyclictest0-21swapper/310:15:283
21946991812,4cyclictest0-21swapper/109:40:091
21946991811,4cyclictest0-21swapper/110:10:131
2194699181,13cyclictest602-21sh11:45:121
2194699181,11cyclictest29679-21find10:50:111
2194699181,10cyclictest31578-21ls07:50:211
2196499178,5cyclictest35966irq/58-eth010:45:133
2196499172,10cyclictest0-21swapper/310:00:133
2196499172,10cyclictest0-21swapper/307:40:133
2196499172,10cyclictest0-21swapper/307:35:163
2196499172,10cyclictest0-21swapper/307:30:143
2194699175,9cyclictest12940-21kthreadcore08:05:231
2194699174,10cyclictest21546-21fschecks_time08:50:161
2194699173,11cyclictest26566-21irqcore08:55:161
2194699172,10cyclictest0-21swapper/107:45:131
2194699171,6cyclictest27270-21irqcore07:15:161
2194699171,13cyclictest3175-21kthreadcore08:30:211
2194699171,12cyclictest28439-21tr09:55:211
2194699171,11cyclictest3100-21threads07:20:271
21946991710,5cyclictest1342-21cat12:20:211
2196499162,10cyclictest0-21swapper/307:50:123
2194699169,4cyclictest6477-21df_inode10:55:121
2194699169,4cyclictest0-21swapper/110:30:161
2194699163,10cyclictest25518-21ntp_kernel_err07:10:251
21946991611,3cyclictest0-21swapper/108:25:191
2194699161,12cyclictest7917-21grep07:25:271
2194699161,12cyclictest21027-21cut08:15:191
2194699161,12cyclictest17794-21ssh09:50:191
2196499141,7cyclictest0-21swapper/308:35:113
2194699141,10cyclictest22003-21cat07:10:021
2196499138,3cyclictest0-21swapper/308:50:153
2196499138,3cyclictest0-21swapper/307:17:113
2196499135,6cyclictest0-21swapper/308:30:203
2196499135,4cyclictest0-21swapper/308:40:133
2196499131,10cyclictest0-21swapper/307:55:213
2196499122,6cyclictest0-21swapper/307:10:143
2196499121,9cyclictest0-21swapper/307:45:173
2196499111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional