You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 23:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Nov 18, 2025 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2254199633,39cyclictest11884-21ssh09:55:142
22541994526,16cyclictest1-21systemd10:45:142
2254199450,3cyclictest131rcu_preempt10:55:132
2254199421,25cyclictest0-21swapper/207:30:122
2253299427,6cyclictest131rcu_preempt10:25:131
2254199401,21cyclictest8730-21diskmemload10:10:152
2254199391,20cyclictest27-21ksoftirqd/210:35:162
2253299377,9cyclictest131rcu_preempt11:55:131
2253299363,30cyclictest30751-21ssh10:50:141
2252399351,27cyclictest23503-21sshd10:45:140
22523993013,8cyclictest131rcu_preempt10:20:130
2252399301,22cyclictest24215-21ssh09:20:140
2255099293,14cyclictest29566irq/58-eth009:55:143
2254199292,18cyclictest26346irq/40-dwc2_hso11:45:162
2253299291,18cyclictest28281-21ssh11:10:131
2253299281,21cyclictest11314-21ssh12:20:271
2252399283,8cyclictest131rcu_preempt07:35:140
2252399281,19cyclictest21537-21ssh09:40:150
2252399281,16cyclictest1656-21ssh11:35:140
2254199272,21cyclictest0-21swapper/210:00:162
2254199271,21cyclictest9099-21ssh09:10:152
2253299273,20cyclictest1670-21ssh11:35:151
22532992716,6cyclictest0-21swapper/112:25:141
2253299271,18cyclictest22822-21ssh12:10:141
2254199263,4cyclictest0-21swapper/211:05:162
22541992611,12cyclictest29084-21apt-get09:45:152
22532992615,8cyclictest12551-21sshd10:58:341
2254199252,9cyclictest0-21swapper/208:55:172
2254199252,20cyclictest16801-21ssh09:15:152
2254199252,13cyclictest15912-21ssh10:40:182
2253299252,12cyclictest31461-21ssh09:25:151
22532992516,7cyclictest0-21swapper/110:05:151
2252399251,7cyclictest0-21swapper/011:10:120
22523992517,5cyclictest12-21ksoftirqd/011:25:150
22523992510,10cyclictest10116-21kworker/0:109:55:130
2255099242,16cyclictest29566irq/58-eth009:20:143
2254199241,9cyclictest0-21swapper/207:50:132
2254199241,8cyclictest0-21swapper/210:25:132
2254199241,18cyclictest10251-21ssh10:15:142
2254199241,14cyclictest3480-21ssh11:15:142
2254199241,14cyclictest3480-21ssh11:15:142
22541992411,4cyclictest26063-21latency_hist11:30:032
2253299248,6cyclictest131rcu_preempt08:20:141
2253299248,6cyclictest131rcu_preempt08:20:131
2253299244,16cyclictest9032-21kworker/u8:211:30:131
2253299244,16cyclictest9032-21kworker/u8:211:30:131
22523992417,5cyclictest0-21swapper/012:35:180
22523992417,5cyclictest0-21swapper/012:35:170
2252399241,19cyclictest618-21ssh10:30:150
2255099233,10cyclictest29566irq/58-eth011:45:153
2255099231,9cyclictest29566irq/58-eth010:10:153
2254199231,14cyclictest17206-21ls08:45:022
2254199231,12cyclictest0-21swapper/211:55:132
2253299239,4cyclictest131rcu_preempt12:15:221
2253299237,3cyclictest131rcu_preempt07:30:121
2253299232,17cyclictest21578-21ssh10:20:261
2253299231,17cyclictest932-21snmpd07:50:131
2252399232,16cyclictest9098-21ssh09:10:150
22523992313,7cyclictest11552-21sshd11:20:200
22523992311,6cyclictest22507-21cyclictest09:00:130
2252399231,16cyclictest0-21swapper/011:15:140
2252399231,16cyclictest0-21swapper/011:15:140
2254199222,5cyclictest0-21swapper/210:30:152
2254199221,10cyclictest24394-21idleruntime07:45:172
2253299228,5cyclictest21-21ksoftirqd/108:45:141
2253299222,5cyclictest53046usb-storage10:00:041
22532992212,7cyclictest1-21systemd08:44:551
22532992212,7cyclictest10803-21cron08:05:021
22532992212,7cyclictest10803-21cron08:05:011
2253299221,18cyclictest19498-21ls07:40:151
2253299221,15cyclictest8756-21ls09:10:021
2252399225,10cyclictest0-21swapper/009:35:170
2252399223,16cyclictest25557-21ssh10:25:150
2255099217,8cyclictest29566irq/58-eth009:15:153
2255099211,9cyclictest0-21swapper/310:20:133
22550992110,6cyclictest0-21swapper/312:25:333
2254199219,5cyclictest28089-21cut07:15:222
2254199217,7cyclictest1669-21diskmemload11:35:152
2254199216,8cyclictest24310-21df_inode09:20:162
2254199212,12cyclictest26346irq/40-dwc2_hso09:35:172
2254199211,7cyclictest12192-21cron08:40:012
22541992111,7cyclictest1198-21latency_hist12:35:012
2254199211,16cyclictest13346-21kthreadcore11:20:232
2254199211,11cyclictest7682-21ssh12:20:152
2254199211,11cyclictest17995-21mailstats07:35:282
2254199211,11cyclictest135-21systemd-journal09:35:012
2254199211,10cyclictest1087-21kthreadcore12:15:242
2254199210,13cyclictest26346irq/40-dwc2_hso08:30:132
2253299218,10cyclictest16159-21wc10:40:201
2253299212,14cyclictest7673-21df12:00:161
2253299211,8cyclictest18408-21awk11:25:011
2253299211,17cyclictest17157-21missed_timers11:00:251
22532992110,4cyclictest28494-21ntp_kernel_pll_11:50:291
2252399213,15cyclictest22826-21ssh12:10:140
2252399211,7cyclictest14896-21unixbench_singl08:05:310
22523992111,6cyclictest0-21swapper/010:00:040
2252399211,11cyclictest9487-21kthreadcore12:20:230
22550992011,6cyclictest0-21swapper/312:05:333
22550992010,8cyclictest0-21swapper/312:15:223
22550992010,8cyclictest0-21swapper/309:42:343
2254199209,9cyclictest25827-21cat09:40:322
2254199208,9cyclictest31336-21kthreadcore08:25:222
2254199207,9cyclictest32605-21/usr/sbin/munin07:20:172
2254199202,15cyclictest20462-21kthreadcore07:40:182
2254199201,9cyclictest23613-21kthreadcore08:15:242
2254199201,9cyclictest17754-21irqrtprio08:45:202
2254199201,7cyclictest1364566ssh11:00:162
22541992014,4cyclictest3435-21ssh09:25:302
2254199201,15cyclictest26346irq/40-dwc2_hso10:20:142
2254199201,11cyclictest17146-21kthreadcore12:05:242
2254199201,11cyclictest11450-21ssh11:40:222
2254199201,10cyclictest8784-21latency_hist09:10:022
2254199201,10cyclictest30227-21timerwakeupswit11:30:272
2254199201,10cyclictest30227-21timerwakeupswit11:30:272
2254199201,10cyclictest28040-21irqcore10:05:202
2254199201,10cyclictest26679-21switchtime12:10:302
2254199201,10cyclictest25462-21irqrtprio11:50:202
2254199201,10cyclictest20224-21ntp_kernel_err12:25:272
2254199201,10cyclictest1803-21ssh12:35:202
2254199201,10cyclictest1803-21ssh12:35:202
2254199201,10cyclictest11188-21fschecks_count08:05:182
2253299206,10cyclictest30431-21latency_hist10:50:021
2253299203,13cyclictest842-21munin-node10:15:161
2253299202,14cyclictest8300-21ssh10:35:161
2253299201,8cyclictest13110-21grep09:10:301
2253299201,7cyclictest14894-21unixbench_singl08:05:311
2253299201,16cyclictest9350-21ls11:40:151
22532992011,6cyclictest28988-21sshd09:45:051
22532992011,6cyclictest0-21swapper/107:10:271
22532992011,5cyclictest23137-21usage_Anybus11:25:341
2253299201,15cyclictest1682-21kthreadcore07:20:191
2253299200,6cyclictest53046usb-storage07:45:041
2252399202,13cyclictest8999-21irqstats10:35:210
22523992012,6cyclictest4872-21sshd10:53:340
22523992011,7cyclictest30951-21sshd12:31:350
2252399200,15cyclictest14174-21timerandwakeup10:15:300
2255099199,7cyclictest0-21swapper/311:01:273
2255099198,6cyclictest0-21swapper/312:36:573
2255099198,6cyclictest0-21swapper/312:36:573
2255099192,10cyclictest0-21swapper/308:35:063
2255099191,9cyclictest0-21swapper/308:30:013
22550991911,6cyclictest0-21swapper/311:54:023
22550991911,5cyclictest0-21swapper/312:10:303
22550991911,3cyclictest0-21swapper/310:05:173
2255099191,13cyclictest0-21swapper/308:15:253
2255099191,11cyclictest0-21swapper/311:55:233
22550991910,6cyclictest0-21swapper/311:30:223
22550991910,6cyclictest0-21swapper/311:30:213
2254199192,8cyclictest16658-21kthreadcore08:10:212
2254199191,9cyclictest9722-21tr12:00:232
2254199191,9cyclictest31940-21fschecks_time09:00:172
2254199191,9cyclictest2381-21grep10:50:312
2254199191,8cyclictest25884-21irqcore08:20:192
2254199191,8cyclictest25884-21irqcore08:20:182
2254199191,8cyclictest22757-21fschecks_count07:10:192
2254199191,15cyclictest8231-21unixbench_multi09:50:332
2254199191,14cyclictest1509-21fschecks_count07:55:162
2254199191,10cyclictest6472-21kthreadcore07:25:212
2254199191,10cyclictest22201-21cut08:50:182
22541991910,7cyclictest28329-21cpu11:10:142
2253299195,10cyclictest21-21ksoftirqd/108:25:171
2253299192,12cyclictest11840-21/usr/sbin/munin09:55:191
2253299191,15cyclictest2808-21fschecks_count08:30:161
2253299191,15cyclictest15833-21find08:10:141
22532991910,5cyclictest2394-21kthreadcore12:35:251
22532991910,5cyclictest2394-21kthreadcore12:35:251
2252399196,10cyclictest4597-21apt-key07:25:010
2252399196,10cyclictest13396-21grep11:40:310
2252399195,11cyclictest31342-21ssh12:15:160
2252399195,11cyclictest3089-21sed08:30:180
2252399194,12cyclictest6521-21needreboot10:10:250
2252399191,9cyclictest13611-21/usr/sbin/munin11:00:150
2252399191,9cyclictest11500-21cron09:55:010
22523991911,6cyclictest10798-21sshd08:04:480
22523991911,6cyclictest10798-21sshd08:04:480
2252399191,15cyclictest3050-21kthreadcore07:55:200
2252399190,13cyclictest24036-21ls08:15:250
2252399190,11cyclictest23287-21ls10:45:010
2255099189,7cyclictest0-21swapper/309:52:143
2255099189,6cyclictest0-21swapper/310:32:023
2255099189,6cyclictest0-21swapper/309:35:293
2255099189,5cyclictest29566irq/58-eth012:20:143
2255099188,6cyclictest0-21swapper/311:17:093
2255099188,6cyclictest0-21swapper/311:17:083
2255099182,11cyclictest0-21swapper/309:10:153
2255099182,10cyclictest0-21swapper/310:40:173
2255099182,10cyclictest0-21swapper/308:30:143
2255099181,9cyclictest0-21swapper/309:10:023
2255099181,14cyclictest29566irq/58-eth011:25:313
2255099181,13cyclictest0-21swapper/310:15:303
2254199181,8cyclictest6582-21irqcore08:00:202
2254199181,8cyclictest6582-21irqcore08:00:202
2253299189,5cyclictest619-21cpu10:30:151
2253299187,8cyclictest19101-21sshd09:36:341
2253299182,12cyclictest7279-21apt-get08:35:031
2253299181,8cyclictest0-21swapper/109:17:381
2253299181,5cyclictest6953-21irqcore09:30:211
22532991812,3cyclictest530-21usb-storage07:25:051
2253299181,14cyclictest3841-21irqcore11:15:181
2253299181,14cyclictest3841-21irqcore11:15:181
22532991811,4cyclictest23533-21cut08:15:241
2253299181,13cyclictest28863-21memory12:30:281
2253299181,13cyclictest20940-21sshd11:05:171
22532991810,5cyclictest25444-21timerandwakeup09:40:301
22532991810,4cyclictest0-21swapper/110:10:331
2252399189,6cyclictest0-21swapper/010:55:280
2252399189,6cyclictest0-21swapper/007:40:150
2252399188,6cyclictest25962-21cut08:20:180
2252399188,6cyclictest25962-21cut08:20:180
2252399187,3cyclictest8730-21diskmemload12:25:050
2252399185,10cyclictest12-21ksoftirqd/009:05:170
2252399184,7cyclictest0-21swapper/011:45:160
2252399181,9cyclictest26867-21kthreadcore07:45:210
2252399181,6cyclictest6251-21awk09:30:010
2252399181,4cyclictest53046usb-storage08:35:100
2252399181,14cyclictest8534-21kthreadcore09:30:230
2252399181,14cyclictest26209-21apt-get11:30:110
2252399181,14cyclictest26209-21apt-get11:30:110
2252399181,10cyclictest22654-21kthreadcore11:05:240
22523991810,6cyclictest15423-21irqstats12:05:210
2255099179,6cyclictest0-21swapper/311:09:323
2255099179,6cyclictest0-21swapper/310:49:053
2255099179,6cyclictest0-21swapper/310:02:213
2255099179,6cyclictest0-21swapper/309:46:103
2255099179,5cyclictest29566irq/58-eth010:35:153
2255099178,6cyclictest29566irq/58-eth011:35:143
2255099178,6cyclictest0-21swapper/310:59:013
2255099178,6cyclictest0-21swapper/309:30:333
2255099177,6cyclictest0-21swapper/311:40:253
22550991711,4cyclictest0-21swapper/312:30:243
22550991711,4cyclictest0-21swapper/312:00:213
22550991711,4cyclictest0-21swapper/310:50:143
2253299179,5cyclictest2341-21kthreadcore09:00:231
2253299175,9cyclictest18608-21xargs12:05:301
2253299173,11cyclictest17106-21/usr/sbin/munin11:45:211
2253299171,13cyclictest27033-21fschecks_count08:55:191
2253299171,12cyclictest7873-21ntp_states09:50:301
2253299171,10cyclictest1630-21ls07:55:171
22532991710,4cyclictest0-21swapper/107:15:181
2252399179,5cyclictest0-21swapper/010:05:190
2252399178,7cyclictest17103-21if_eth009:15:200
2252399177,8cyclictest29607-21cut09:45:200
2252399172,12cyclictest27083-21sort08:55:200
2252399172,12cyclictest24070-21kthreadcore07:10:240
2252399171,13cyclictest9595-21apt-get07:30:120
2252399171,13cyclictest8606-21kthreadcore12:00:210
2252399171,13cyclictest7209-21tr12:00:020
2252399170,14cyclictest7644-21tr07:25:220
2255099169,4cyclictest0-21swapper/311:10:203
2255099168,6cyclictest0-21swapper/309:28:593
2255099161,9cyclictest0-21swapper/311:20:313
22550991610,4cyclictest0-21swapper/310:25:213
2254199160,8cyclictest22407-21latency_hist07:10:012
2254199160,8cyclictest22407-21latency_hist07:10:012
2253299167,6cyclictest22230-21fschecks_count08:50:181
2253299162,10cyclictest0-21swapper/107:35:161
2253299161,11cyclictest24389-21fschecks_time09:20:181
2252399163,9cyclictest22507-21cyclictest07:10:010
2252399163,9cyclictest22507-21cyclictest07:10:010
2252399162,11cyclictest28389-21kthreadcore07:15:230
2252399161,12cyclictest30339-21apt-get08:25:160
2252399161,11cyclictest25069-21cut11:50:180
2252399161,11cyclictest22278-21fschecks_time08:50:180
2252399161,11cyclictest19684-21sed08:10:300
2252399161,11cyclictest18065-21kthreadcore08:45:210
2252399161,11cyclictest13049-21sed08:40:200
2252399160,12cyclictest32092-21cut07:50:220
2255099151,8cyclictest0-21swapper/307:40:013
2255099151,10cyclictest0-21swapper/309:00:153
22550991510,3cyclictest0-21swapper/308:11:393
2255099150,10cyclictest0-21swapper/308:05:233
2253299151,11cyclictest22597-21wc07:10:021
2253299151,11cyclictest22597-21wc07:10:011
2255099149,3cyclictest0-21swapper/307:40:113
2255099149,3cyclictest0-21swapper/307:23:463
2255099148,4cyclictest0-21swapper/308:20:293
2255099148,4cyclictest0-21swapper/308:20:293
2255099141,10cyclictest0-21swapper/307:55:293
2255099138,3cyclictest0-21swapper/307:10:213
2255099136,5cyclictest0-21swapper/308:55:263
2255099131,9cyclictest0-21swapper/308:40:203
2255099127,3cyclictest0-21swapper/308:45:193
2255099127,3cyclictest0-21swapper/308:00:313
2255099127,3cyclictest0-21swapper/308:00:313
2255099127,3cyclictest0-21swapper/307:26:333
2255099122,7cyclictest0-21swapper/307:20:013
2255099121,9cyclictest0-21swapper/307:50:023
2255099111,8cyclictest0-21swapper/307:30:133
2255099111,7cyclictest0-21swapper/308:50:213
2255099111,6cyclictest0-21swapper/307:50:183
2255099101,6cyclictest0-21swapper/307:10:013
2255099101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional