You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-07 - 19:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Nov 07, 2025 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3016699683,35cyclictest22498-21ssh10:20:130
3016699604,46cyclictest4694-21ssh10:30:140
3018999574,40cyclictest0-21swapper/210:20:132
3018999521,41cyclictest0-21swapper/211:55:152
30166995021,22cyclictest12-21ksoftirqd/009:15:140
3020199481,29cyclictest29566irq/58-eth009:50:123
3020199481,29cyclictest29566irq/58-eth009:50:123
30179994824,10cyclictest131rcu_preempt10:50:131
3020199464,25cyclictest29566irq/58-eth011:15:153
3017999463,34cyclictest29558-21ssh10:25:161
3016699462,28cyclictest9957-21ssh09:50:110
3016699462,28cyclictest9957-21ssh09:50:110
3020199451,15cyclictest0-21swapper/310:20:133
3018999453,34cyclictest0-21swapper/211:10:172
30179994510,8cyclictest131rcu_preempt09:25:131
30166994315,20cyclictest12-21ksoftirqd/008:05:150
3020199423,29cyclictest29566irq/58-eth009:35:153
3016699421,31cyclictest13422-21ssh11:40:130
3017999418,20cyclictest12496-21sshd10:35:161
3017999395,8cyclictest131rcu_preempt11:20:131
3017999382,30cyclictest20835-21ssh11:45:161
3017999373,8cyclictest131rcu_preempt11:25:131
3017999371,25cyclictest30900-21ssh09:20:121
3020199362,19cyclictest29566irq/58-eth010:45:123
3020199362,19cyclictest29566irq/58-eth010:30:143
3017999362,26cyclictest7916-21ssh11:15:151
3017999362,26cyclictest18206-21ssh12:05:151
3020199342,8cyclictest0-21swapper/307:50:123
3020199342,8cyclictest0-21swapper/307:50:123
3020199323,17cyclictest29566irq/58-eth011:20:143
3018999312,20cyclictest23804-21ssh09:15:142
3017999313,22cyclictest20054-21ssh10:40:141
3020199301,8cyclictest0-21swapper/311:35:153
3017999302,22cyclictest10606-21ssh10:55:151
3017999302,20cyclictest5631-21ssh11:35:151
30166993021,4cyclictest0-21swapper/012:25:140
3020199293,16cyclictest29566irq/58-eth010:25:163
3018999293,21cyclictest0-21swapper/209:34:272
3018999292,21cyclictest0-21swapper/207:40:142
3018999292,11cyclictest14151-21ntp_192.168.11410:55:262
3017999292,19cyclictest1227-21ssh12:35:141
30179992915,11cyclictest21-21ksoftirqd/112:30:161
30166992914,7cyclictest12-21ksoftirqd/012:05:130
3020199286,11cyclictest0-21swapper/312:05:133
3018999287,13cyclictest629-21Async14:00:482
3018999283,16cyclictest26346irq/40-dwc2_hso12:25:142
3016699283,21cyclictest3293-21ssh10:50:140
3020199273,16cyclictest0-21swapper/309:45:273
3020199271,3cyclictest0-21swapper/311:05:143
3020199271,3cyclictest0-21swapper/309:56:273
3020199271,2cyclictest0-21swapper/311:30:123
3020199271,2cyclictest0-21swapper/311:30:123
3020199271,2cyclictest0-21swapper/309:40:293
30201992710,13cyclictest29566irq/58-eth010:40:283
30179992715,8cyclictest21-21ksoftirqd/108:00:151
3016699271,21cyclictest0-21swapper/009:30:140
3020199262,17cyclictest29566irq/58-eth012:20:143
3020199261,4cyclictest0-21swapper/310:06:203
3020199261,3cyclictest0-21swapper/311:58:503
3020199261,3cyclictest0-21swapper/311:50:283
3020199261,3cyclictest0-21swapper/311:25:303
3020199261,3cyclictest0-21swapper/310:50:403
3020199261,3cyclictest0-21swapper/310:38:483
3020199261,3cyclictest0-21swapper/309:31:483
3020199261,3cyclictest0-21swapper/308:10:163
3020199261,3cyclictest0-21swapper/307:55:233
3018999264,14cyclictest26346irq/40-dwc2_hso07:20:132
3018999263,7cyclictest0-21swapper/211:40:132
3018999262,19cyclictest1148-21apt-get12:15:122
3018999262,16cyclictest26346irq/40-dwc2_hso07:10:152
3018999262,14cyclictest22035-21kthreadcore10:40:212
3018999261,21cyclictest29667-21cut08:50:162
3018999261,19cyclictest0-21swapper/207:25:172
3018999261,15cyclictest16409-21kthreadcore08:00:232
3017999264,15cyclictest9299-21ssh12:20:141
3017999264,14cyclictest32608-21ssh10:05:141
3017999262,18cyclictest856-21sshd10:45:171
3017999262,18cyclictest6678-21sed09:45:271
3017999262,17cyclictest16560-21ssh09:10:121
3020199256,16cyclictest29566irq/58-eth011:40:153
3020199252,3cyclictest0-21swapper/311:10:163
3020199252,3cyclictest0-21swapper/308:35:143
3020199251,5cyclictest0-21swapper/312:15:423
3020199251,5cyclictest0-21swapper/310:10:113
3020199251,3cyclictest0-21swapper/312:35:293
3020199251,3cyclictest0-21swapper/312:00:513
3020199251,3cyclictest0-21swapper/310:55:193
3020199251,3cyclictest0-21swapper/310:00:203
3020199251,3cyclictest0-21swapper/308:45:213
3020199251,2cyclictest0-21swapper/311:45:143
3018999253,19cyclictest261rcuc/211:50:222
3018999253,14cyclictest11838-21tr12:00:192
3018999252,20cyclictest9134-21latency_hist12:40:012
3018999252,20cyclictest19710-21kthreadcore09:55:212
3018999252,20cyclictest14059-21kthreadcore10:35:222
3018999252,16cyclictest17963-21kthreadcore11:20:222
3018999252,15cyclictest21476-21kthreadcore11:00:252
30189992513,5cyclictest31833-21ssh11:08:392
3018999251,20cyclictest12367-21kthreadcore09:05:202
3018999251,20cyclictest12367-21kthreadcore09:05:202
3018999251,16cyclictest19659-21kthreadcore07:30:232
3016699252,18cyclictest11077-21apt-get12:00:130
3020199241,6cyclictest0-21swapper/309:25:013
3020199241,4cyclictest0-21swapper/308:05:173
3020199241,3cyclictest0-21swapper/312:25:213
3020199241,3cyclictest0-21swapper/309:15:163
3020199241,3cyclictest0-21swapper/309:10:103
3020199241,3cyclictest0-21swapper/308:25:283
3020199241,3cyclictest0-21swapper/307:45:113
3020199241,3cyclictest0-21swapper/307:35:273
3020199241,3cyclictest0-21swapper/307:10:173
3020199241,12cyclictest0-21swapper/312:30:143
3018999242,19cyclictest1173-21kthreadcore09:20:212
3018999241,20cyclictest24199-21kthreadcore11:45:252
3018999241,19cyclictest8036-21sort11:15:172
3018999241,14cyclictest30989-21kthreadcore10:25:222
3017999242,18cyclictest24216-21ssh09:59:261
3017999242,17cyclictest25574-21ssh11:05:131
30179992411,9cyclictest16656-21cut11:40:211
3017999241,16cyclictest25760-21ssh12:10:141
3017999241,16cyclictest25760-21ssh12:10:141
3020199231,6cyclictest0-21swapper/308:40:283
3020199231,6cyclictest0-21swapper/308:40:283
3020199231,5cyclictest0-21swapper/309:00:223
3020199231,5cyclictest0-21swapper/308:55:243
3020199231,4cyclictest0-21swapper/311:00:143
3020199231,4cyclictest0-21swapper/308:00:213
3020199231,4cyclictest0-21swapper/307:30:153
3020199231,4cyclictest0-21swapper/307:20:193
3020199231,3cyclictest0-21swapper/312:10:213
3020199231,3cyclictest0-21swapper/312:10:213
3020199231,3cyclictest0-21swapper/310:15:223
3020199231,3cyclictest0-21swapper/309:25:183
3020199231,3cyclictest0-21swapper/309:10:013
3020199231,3cyclictest0-21swapper/309:10:013
3020199231,3cyclictest0-21swapper/308:20:273
3020199231,3cyclictest0-21swapper/308:15:193
3020199231,3cyclictest0-21swapper/308:15:193
3020199231,3cyclictest0-21swapper/307:40:193
3020199231,3cyclictest0-21swapper/307:15:193
3018999232,19cyclictest20187-21ntp_states09:10:242
3018999231,20cyclictest30407-21kthreadcore09:40:222
3018999231,13cyclictest27113-21kthreadcore10:00:202
3017999238,7cyclictest629-21Async14:00:481
3017999238,7cyclictest629-21Async14:00:481
3016699233,14cyclictest27653-21ssh09:40:150
3016699231,17cyclictest25229-21latency_hist11:05:020
3016699231,16cyclictest32394-21ls10:05:020
3020199221,3cyclictest0-21swapper/308:30:283
3020199221,3cyclictest0-21swapper/307:25:193
3018999228,11cyclictest612-21dbus-daemon12:30:342
3018999222,6cyclictest0-21swapper/208:35:142
3018999222,17cyclictest7922-21kthreadcore09:25:212
3018999222,17cyclictest22173-21ls07:35:142
3018999222,12cyclictest0-21swapper/207:45:112
3018999222,11cyclictest32265-21kthreadcore11:30:202
3018999222,11cyclictest32265-21kthreadcore11:30:202
30189992210,9cyclictest0-21swapper/212:10:232
30189992210,9cyclictest0-21swapper/212:10:232
3017999222,13cyclictest25023-21ssh10:00:131
3017999221,15cyclictest7304-21sh10:10:111
3017999221,15cyclictest28564-21sh11:50:151
30179992210,9cyclictest25201-21cron12:30:011
3016699222,16cyclictest13093-21kthreadcore07:25:220
3016699221,6cyclictest16226-21munin-run09:10:010
3016699221,6cyclictest16226-21munin-run09:10:010
3016699221,17cyclictest23427-21find08:10:120
3016699221,16cyclictest8969-21apt-get07:55:130
3020199211,3cyclictest0-21swapper/308:50:193
3018999212,16cyclictest6057-21kthreadcore10:30:212
3018999212,16cyclictest1145-21irqcore08:20:182
3018999212,14cyclictest27-21ksoftirqd/210:50:142
3018999212,12cyclictest31270-21kthreadcore08:15:212
3018999212,12cyclictest31270-21kthreadcore08:15:212
3018999211,16cyclictest7158-21cat10:10:022
30189992110,9cyclictest29578-21ssh11:29:192
3017999219,9cyclictest11380-21cron09:05:011
3017999215,3cyclictest131rcu_preempt12:20:011
3017999211,14cyclictest22463-21find10:20:131
30179992110,8cyclictest12903-21df_inode09:30:151
3017999210,18cyclictest8960-21find07:55:131
3016699212,16cyclictest3921-21sendmail_mailtr10:05:280
3016699212,16cyclictest32241-21kthreadcore09:20:190
3016699212,16cyclictest21487-21kthreadcore08:40:200
3016699212,16cyclictest21487-21kthreadcore08:40:200
3016699212,16cyclictest20421-21kthreadcore07:30:240
3016699212,16cyclictest14237-21irqcore08:00:200
3018999203,14cyclictest4671-21kthreadcore08:55:222
3018999203,11cyclictest20918-21kthreadcore08:40:192
3018999203,11cyclictest20918-21kthreadcore08:40:192
3018999202,8cyclictest27-21ksoftirqd/207:50:142
3018999202,8cyclictest27-21ksoftirqd/207:50:142
3018999202,16cyclictest25012-21cut08:10:202
3018999202,15cyclictest21275-21kthreadcore08:05:232
3018999202,15cyclictest10571-21kthreadcore10:10:212
3018999202,15cyclictest10457-21kthreadcore12:20:202
3018999202,11cyclictest7067-21usage_EthcTask07:15:292
3018999202,11cyclictest26210-21kthreadcore08:45:192
3018999202,10cyclictest5683-21ssh11:35:162
3018999201,9cyclictest12340-21kthreadcore09:50:192
3018999201,9cyclictest12340-21kthreadcore09:50:192
30189992015,3cyclictest27-21ksoftirqd/209:49:542
3018999201,17cyclictest20954-21kthreadcore09:35:212
3018999201,10cyclictest17744-21apt-get12:05:022
3018999201,10cyclictest15005-21grep08:35:012
3017999207,10cyclictest22319-21grep07:35:161
3017999202,13cyclictest30436-21ssh11:30:131
3017999202,13cyclictest30436-21ssh11:30:131
3016699208,9cyclictest12993-21cron11:40:000
3016699203,14cyclictest698-21cpu08:20:150
3016699203,14cyclictest10341-21kthreadcore11:15:230
3016699203,11cyclictest0-21swapper/010:58:270
3016699202,16cyclictest31124-21kthreadcore11:50:220
3016699202,15cyclictest7734-21kthreadcore09:25:210
3016699202,15cyclictest4306-21fschecks_time07:50:160
3016699202,15cyclictest4306-21fschecks_time07:50:160
3016699202,15cyclictest3243-21irqrtprio07:15:180
3016699202,15cyclictest15104-21latency_hist08:35:020
3016699202,15cyclictest13278-21kthreadcore10:35:210
3016699201,7cyclictest26035-21irqcore11:05:180
3016699201,16cyclictest1335-21cut12:15:160
3016699201,11cyclictest30211-21cut08:50:190
3016699201,10cyclictest12-21ksoftirqd/008:25:150
3016699200,17cyclictest30679-21ls11:30:160
3016699200,17cyclictest30679-21ls11:30:160
3018999198,8cyclictest0-21swapper/210:45:182
3018999192,9cyclictest15001-21grep10:15:162
3018999192,15cyclictest6954-21fschecks_time09:00:172
3018999191,11cyclictest9477-21irqrtprio07:55:192
3017999198,8cyclictest22902-21cut09:35:231
3017999194,11cyclictest201rcuc/108:30:121
3017999193,9cyclictest21-21ksoftirqd/108:20:131
3017999192,13cyclictest11084-21ssh12:00:141
3017999191,16cyclictest18846-21timerandwakeup08:35:291
30179991911,6cyclictest0-21swapper/108:25:191
3017999191,12cyclictest24929-21fschecks_count08:45:151
3016699193,13cyclictest30847-21kthreadcore10:25:220
3016699193,13cyclictest28952-21kthreadcore08:15:190
3016699193,13cyclictest28952-21kthreadcore08:15:190
3016699192,15cyclictest757-21fschecks_count11:10:200
3016699192,15cyclictest29555-21mailstats12:10:270
3016699192,15cyclictest29555-21mailstats12:10:270
3016699192,14cyclictest8087-21kthreadcore09:00:200
3016699192,14cyclictest5525-21mailstats08:55:280
30166991911,6cyclictest21814-21sshd10:19:260
3016699191,10cyclictest23456-21irqrtprio11:25:190
3020199181,3cyclictest0-21swapper/307:10:013
3017999185,10cyclictest23383-21netstat08:40:231
3017999185,10cyclictest23383-21netstat08:40:231
3017999182,13cyclictest7518-21kthreadcore10:30:231
3017999182,13cyclictest25191-21kthreadcore09:15:201
3017999182,13cyclictest14817-21ssh10:15:121
3016699188,7cyclictest22350-21cut10:40:210
3016699183,11cyclictest0-21swapper/008:35:130
3016699182,14cyclictest8624-21kthreadcore07:20:200
3016699182,14cyclictest3745-21grep11:55:170
3016699182,14cyclictest21735-21kthreadcore09:35:220
3016699182,13cyclictest27109-21kthreadcore12:30:220
3016699182,13cyclictest19745-21kthreadcore09:10:200
3016699181,9cyclictest0-21swapper/007:45:110
3016699181,6cyclictest29333-21munin-run08:50:000
30166991811,5cyclictest14502-21cron10:15:010
3016699180,10cyclictest2783866/usr/sbin/munin10:45:250
3018999171,8cyclictest30152-21cyclictest07:10:012
3017999179,5cyclictest683-21in:imuxsock07:45:011
3017999179,5cyclictest24031-21sshd11:02:271
3017999177,8cyclictest7532-21apt-get07:20:131
3017999174,11cyclictest26847-21sh08:10:241
3017999173,11cyclictest4594-21cut11:55:211
3017999173,11cyclictest30432-21grep07:10:171
3017999173,11cyclictest22224-21mailstats08:05:281
3017999172,10cyclictest0-21swapper/107:15:141
3017999171,14cyclictest28368-21tail09:40:201
30179991711,4cyclictest0-21swapper/108:50:151
3017999171,13cyclictest14232-21kthreadcore07:25:241
3017999171,10cyclictest16369-21ls09:10:011
3017999171,10cyclictest16369-21ls09:10:011
30179991710,4cyclictest21-21ksoftirqd/108:55:131
3016699173,11cyclictest28695-21tr07:40:200
3016699172,13cyclictest9356-21cpuspeed12:20:140
3016699172,13cyclictest20952-21fschecks_count11:45:180
3016699171,13cyclictest15832-21sed11:20:180
3017999167,6cyclictest17595-21irqcore07:30:191
3017999165,3cyclictest131rcu_preempt07:55:011
3017999165,3cyclictest131rcu_preempt07:55:011
3017999161,12cyclictest557-21apt-get11:10:161
3017999161,12cyclictest10504-21/usr/sbin/munin09:50:161
3017999161,12cyclictest10504-21/usr/sbin/munin09:50:161
3016699162,11cyclictest30298-21acpi07:10:130
3016699162,11cyclictest22566-21irqcore07:35:170
30166991611,3cyclictest1272-21cpu12:35:150
3016699161,12cyclictest3221-21cut09:45:170
3016699161,12cyclictest18068-21ssh09:55:180
3017999151,11cyclictest31749-21apt-get07:45:121
3017999141,10cyclictest30217-21latency_hist07:10:011
3016699142,9cyclictest30249-21cat07:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional