You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 23:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Sat Jun 28, 2025 12:45:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2875699683,21cyclictest21-21ksoftirqd/109:20:121
2875699631,13cyclictest0-21swapper/112:30:131
28774996246,13cyclictest32466irq/58-eth012:30:143
2877499496,29cyclictest32466irq/58-eth012:20:163
2877499496,29cyclictest32466irq/58-eth012:20:153
2876599487,30cyclictest29566-21ssh09:20:132
2876599481,39cyclictest0-21swapper/208:35:192
2875699482,9cyclictest131rcu_preempt12:05:161
2875699461,10cyclictest131rcu_preempt11:45:161
28765994527,10cyclictest27-21ksoftirqd/208:45:162
2877499442,24cyclictest32466irq/58-eth012:00:153
28765994325,11cyclictest27-21ksoftirqd/208:50:152
28765994312,9cyclictest27-21ksoftirqd/209:05:162
2877499414,25cyclictest32466irq/58-eth010:05:163
2877499402,22cyclictest32466irq/58-eth010:10:143
28765994016,20cyclictest27-21ksoftirqd/208:15:162
28774993918,15cyclictest32466irq/58-eth011:10:163
2875699396,27cyclictest10555-21ssh09:50:161
2874899371,32cyclictest8452-21ssh10:10:140
2876599362,28cyclictest24734-21ssh09:58:412
2874899362,31cyclictest6135-21ssh11:35:160
2874899362,31cyclictest13354-21ssh12:00:150
2876599353,6cyclictest131rcu_preempt11:10:172
2874899354,28cyclictest4608-21ssh09:25:140
2874899354,28cyclictest4608-21ssh09:25:130
28748993529,4cyclictest0-21swapper/011:10:170
2874899351,30cyclictest30897-21ssh12:10:150
2874899342,25cyclictest4565-21ssh11:55:130
2877499325,13cyclictest32466irq/58-eth010:45:153
2877499322,17cyclictest32466irq/58-eth011:30:153
2877499322,17cyclictest32466irq/58-eth011:30:143
2876599325,18cyclictest23346irq/40-dwc2_hso09:15:142
2875699323,20cyclictest30426-21ssh10:25:171
28756993219,11cyclictest25838-21ssh10:00:181
28748993215,12cyclictest662-21ntpd11:40:130
2874899321,23cyclictest841-21sh10:05:150
2874899321,21cyclictest14777-21ssh09:10:150
2877499312,10cyclictest0-21swapper/309:59:413
2876599317,19cyclictest916-21snmpd10:31:262
2876599316,7cyclictest0-21swapper/210:15:172
2875699312,22cyclictest844-21ssh10:05:161
2874899314,4cyclictest131rcu_preempt11:00:190
2876599303,17cyclictest23346irq/40-dwc2_hso07:20:172
28765993014,12cyclictest15287-21cron11:20:012
2875699309,17cyclictest27716-21ssh10:45:141
2874899304,15cyclictest0-21swapper/010:30:140
2874899303,20cyclictest2924-21ssh10:50:170
2876599293,20cyclictest0-21swapper/209:25:152
2876599293,20cyclictest0-21swapper/209:25:142
2876599292,11cyclictest27163-21ssh10:44:422
2875699292,21cyclictest18340-21ssh10:36:411
2875699292,21cyclictest18340-21ssh10:36:411
28756992917,4cyclictest131rcu_preempt11:00:191
2874899292,20cyclictest7274-21ssh12:15:180
2877499283,11cyclictest0-21swapper/310:35:413
2877499283,11cyclictest0-21swapper/310:35:403
2876599286,18cyclictest27-21ksoftirqd/209:39:412
2876599283,18cyclictest23346irq/40-dwc2_hso08:10:142
2876599283,10cyclictest0-21swapper/209:54:402
2876599282,10cyclictest5549-21ssh11:34:422
2876599282,10cyclictest5549-21ssh11:34:412
2875699282,9cyclictest131rcu_preempt10:40:421
2875699282,19cyclictest13284-21ssh11:16:411
28756992817,6cyclictest0-21swapper/110:30:141
2877499273,16cyclictest32466irq/58-eth008:00:153
2877499273,16cyclictest32466irq/58-eth008:00:143
2876599275,15cyclictest23346irq/40-dwc2_hso12:30:162
2876599275,14cyclictest23346irq/40-dwc2_hso07:25:142
2876599274,16cyclictest23346irq/40-dwc2_hso07:35:152
2876599273,16cyclictest23346irq/40-dwc2_hso07:15:142
2876599272,21cyclictest1703-21ssh09:43:422
2876599272,20cyclictest0-21swapper/210:56:312
2876599272,20cyclictest0-21swapper/210:56:302
2876599272,20cyclictest0-21swapper/207:30:152
2876599272,19cyclictest0-21swapper/207:50:202
2876599272,11cyclictest0-21swapper/207:55:152
2876599272,11cyclictest0-21swapper/207:55:142
2876599271,15cyclictest19276-21ssh11:20:302
2875699272,19cyclictest25846-21ssh09:37:401
2875699271,23cyclictest0-21swapper/108:55:141
2874899272,19cyclictest1738-21ssh10:48:410
2877499266,9cyclictest32566irq/59-eth012:25:173
2877499265,12cyclictest0-21swapper/309:50:173
2877499263,10cyclictest0-21swapper/310:31:243
2877499262,10cyclictest0-21swapper/310:41:413
2877499262,10cyclictest0-21swapper/309:30:413
2876599262,19cyclictest8675-21apt-get08:30:142
28756992615,6cyclictest0-21swapper/109:40:191
2875699261,19cyclictest14413-21diskmemload10:20:171
2874899269,12cyclictest12-21ksoftirqd/011:16:410
2874899267,6cyclictest12-21ksoftirqd/010:25:020
2874899263,15cyclictest30993-21ssh11:30:150
2874899263,15cyclictest30993-21ssh11:30:140
2874899262,21cyclictest28786-21ssh11:50:190
2874899261,5cyclictest12-21ksoftirqd/010:25:170
28748992612,8cyclictest151rcuc/012:05:170
2876599254,15cyclictest23346irq/40-dwc2_hso08:25:172
2876599253,14cyclictest23346irq/40-dwc2_hso07:10:182
28756992514,7cyclictest0-21swapper/109:00:161
2874899251,20cyclictest2864-21ssh09:45:170
2874899251,20cyclictest2864-21ssh09:45:160
28748992510,7cyclictest0-21swapper/012:24:030
28748992510,7cyclictest0-21swapper/012:24:030
28748992510,7cyclictest0-21swapper/011:26:530
2877499242,12cyclictest32466irq/58-eth009:25:143
2877499242,12cyclictest32466irq/58-eth009:25:133
2876599242,16cyclictest0-21swapper/210:50:402
28765992413,8cyclictest0-21swapper/212:05:152
2876599241,20cyclictest29148-21ls11:05:352
2876599241,18cyclictest18202-21ssh10:36:282
2876599241,18cyclictest18202-21ssh10:36:272
2875699246,12cyclictest21-21ksoftirqd/109:55:151
2875699244,13cyclictest0-21swapper/111:05:201
2875699242,8cyclictest131rcu_preempt08:30:171
2874899248,12cyclictest12-21ksoftirqd/009:36:410
2874899247,6cyclictest0-21swapper/012:35:070
2876599239,8cyclictest0-21swapper/212:20:172
2876599239,8cyclictest0-21swapper/212:20:162
2876599232,9cyclictest0-21swapper/211:35:172
2876599232,6cyclictest0-21swapper/210:28:422
2876599232,16cyclictest0-21swapper/210:01:242
2876599232,16cyclictest0-21swapper/209:46:152
2876599232,16cyclictest0-21swapper/209:46:142
2876599231,19cyclictest30193-21sed10:25:022
2876599231,12cyclictest9042-21cut12:15:262
2875699234,9cyclictest131rcu_preempt09:15:181
2875699233,12cyclictest21-21ksoftirqd/111:50:181
2875699231,4cyclictest131rcu_preempt07:55:181
2875699231,4cyclictest131rcu_preempt07:55:171
28756992313,7cyclictest211ksoftirqd/112:15:171
2874899237,7cyclictest0-21swapper/008:35:180
2874899234,13cyclictest151rcuc/009:55:130
28748992312,3cyclictest0-21swapper/011:08:370
28748992310,5cyclictest0-21swapper/009:22:280
2877499223,14cyclictest0-21swapper/312:10:163
2877499223,13cyclictest32466irq/58-eth011:00:193
2877499223,12cyclictest32466irq/58-eth012:35:173
28774992211,6cyclictest0-21swapper/311:25:173
2876599222,6cyclictest0-21swapper/212:10:162
2876599221,9cyclictest25326-21cut11:25:252
28765992213,6cyclictest8156-21ssh10:10:012
28765992213,5cyclictest28842-21ssh10:45:222
2876599221,11cyclictest21545-21fschecks_time11:45:192
28765992210,9cyclictest18752-21ssh12:01:112
28765992210,9cyclictest10772-21ssh11:57:182
28765992210,9cyclictest0-21swapper/208:00:192
28765992210,9cyclictest0-21swapper/208:00:182
2875699221,3cyclictest131rcu_preempt08:45:321
2875699221,2cyclictest131rcu_preempt10:50:291
2874899221,18cyclictest15991-21ssh09:30:300
2874899221,18cyclictest15761-21timerwakeupswit08:00:310
2874899221,18cyclictest15761-21timerwakeupswit08:00:300
2874899221,16cyclictest24906-21rm12:25:190
28748992210,9cyclictest14695-21awk09:50:320
28748992210,4cyclictest12-21ksoftirqd/010:36:110
28748992210,4cyclictest12-21ksoftirqd/010:36:100
28748992210,2cyclictest131rcu_preempt10:15:570
2877499211,9cyclictest32466irq/58-eth010:20:173
28774992113,5cyclictest32466irq/58-eth011:20:173
28774992112,6cyclictest0-21swapper/309:35:323
2877499211,15cyclictest0-21swapper/310:00:213
2876599217,11cyclictest27-21ksoftirqd/209:30:252
2876599216,11cyclictest27-21ksoftirqd/208:55:232
2876599215,3cyclictest131rcu_preempt11:50:242
2876599212,10cyclictest30252-21fschecks_time07:45:202
2876599211,9cyclictest14749-21/usr/sbin/munin09:10:142
2876599211,16cyclictest24853-21/usr/sbin/munin12:25:292
2876599211,16cyclictest15806-21kthreadcore11:40:212
2876599211,10cyclictest10017-21acpi12:35:142
28765992110,9cyclictest14142-21ssh10:12:222
2875699219,7cyclictest131rcu_preempt10:58:421
2875699219,7cyclictest131rcu_preempt10:58:411
2875699216,4cyclictest19944-21kworker/1:011:44:581
2875699211,6cyclictest16309-21irqcore10:15:211
2875699211,17cyclictest31523-21cut08:20:201
2875699211,17cyclictest21704-21find08:10:131
2874899214,8cyclictest12-21ksoftirqd/007:20:320
2874899212,15cyclictest14531-21latency_hist09:10:010
28748992113,5cyclictest31101-21missed_timers09:40:280
2874899211,16cyclictest26734-21cut08:15:210
2874899211,13cyclictest5285-21ls09:00:210
28748992110,3cyclictest12-21ksoftirqd/009:15:140
28748992110,3cyclictest12-21ksoftirqd/007:15:140
2874899210,4cyclictest131rcu_preempt10:55:550
2874899210,4cyclictest131rcu_preempt10:55:540
2877499202,11cyclictest0-21swapper/311:55:143
2877499202,10cyclictest32466irq/58-eth010:50:173
2876599207,5cyclictest916-21snmpd08:20:222
2876599206,6cyclictest18021-21hwlatdetect11:00:232
2876599206,5cyclictest0-21swapper/207:40:202
2876599201,10cyclictest9748-21latency_hist09:05:022
2876599201,10cyclictest18689-21kthreadcore08:05:232
2876599201,10cyclictest18381-21fschecks_count08:40:182
2875699204,5cyclictest14746-21sed07:25:311
2875699203,13cyclictest10926-21/usr/sbin/munin11:35:351
2875699202,15cyclictest16076-21ssh12:20:171
2875699202,15cyclictest16076-21ssh12:20:161
28756992012,5cyclictest14411-21sh09:10:011
2875699200,15cyclictest24905-21sh12:25:201
2874899208,4cyclictest12-21ksoftirqd/008:30:280
2874899203,4cyclictest12-21ksoftirqd/008:05:160
28748992012,5cyclictest15696-21sh11:20:190
28748992012,4cyclictest12-21ksoftirqd/008:45:160
2874899201,15cyclictest12-21ksoftirqd/010:40:160
28748992011,5cyclictest0-21swapper/008:10:130
2874899200,4cyclictest131rcu_preempt08:55:170
2874899200,17cyclictest10550-21ls07:55:240
2874899200,17cyclictest10550-21ls07:55:230
2877499193,11cyclictest0-21swapper/311:16:403
2877499193,11cyclictest0-21swapper/310:56:313
2877499193,11cyclictest0-21swapper/310:56:303
2877499193,11cyclictest0-21swapper/309:45:423
2877499193,11cyclictest0-21swapper/309:45:413
2877499193,10cyclictest32466irq/58-eth010:15:313
2877499192,11cyclictest0-21swapper/308:50:173
2877499191,4cyclictest0-21swapper/311:40:133
28774991912,4cyclictest0-21swapper/311:45:343
2877499191,13cyclictest0-21swapper/309:10:193
2877499191,11cyclictest0-21swapper/308:05:193
2875699199,7cyclictest17031-21ntp_kernel_pll_12:00:291
2875699198,3cyclictest131rcu_preempt09:25:211
2875699198,3cyclictest131rcu_preempt09:25:201
2875699191,9cyclictest6205-21kthreadcore08:25:241
2875699191,6cyclictest1128-21irqstats11:10:231
2875699191,5cyclictest15038-21/usr/sbin/munin09:10:191
28756991912,4cyclictest9016-21usage_EthcTask11:55:321
2875699191,13cyclictest0-21swapper/112:35:171
2875699191,10cyclictest5739-21missed_timers07:50:291
2874899199,2cyclictest12-21ksoftirqd/010:01:090
28748991913,4cyclictest0-21swapper/011:45:280
28748991912,5cyclictest22810-21sshd08:41:410
28748991912,5cyclictest0-21swapper/007:45:220
28748991911,6cyclictest32402-21sshd08:50:410
28748991911,6cyclictest29821-21sshd07:40:430
2874899191,15cyclictest15798-21cut07:30:160
2877499189,6cyclictest32466irq/58-eth011:50:193
2877499183,9cyclictest0-21swapper/312:15:173
2877499182,10cyclictest0-21swapper/310:25:043
2877499182,10cyclictest0-21swapper/308:30:043
2877499182,10cyclictest0-21swapper/308:10:153
2877499182,10cyclictest0-21swapper/307:50:203
2877499181,13cyclictest32466irq/58-eth009:15:143
2877499181,13cyclictest0-21swapper/312:05:193
2877499181,13cyclictest0-21swapper/307:25:303
2877499181,11cyclictest0-21swapper/307:15:303
28774991810,6cyclictest0-21swapper/309:40:293
28774991810,6cyclictest0-21swapper/309:20:113
2876599181,13cyclictest28809-21latency_hist07:10:012
2875699188,7cyclictest30991-21df12:10:181
2875699187,9cyclictest0-21swapper/107:10:231
2875699186,3cyclictest131rcu_preempt09:30:281
2875699181,9cyclictest131rcu_preempt08:50:271
2875699181,9cyclictest0-21swapper/110:10:051
2875699181,8cyclictest29119-21unixbench_singl07:40:311
2875699181,13cyclictest15687-21apt-get11:20:171
2875699180,2cyclictest131rcu_preempt08:40:271
2874899189,5cyclictest8355-21cron08:30:020
2874899188,3cyclictest12-21ksoftirqd/007:35:200
2874899181,4cyclictest131rcu_preempt08:20:300
28748991810,6cyclictest4683-21cat12:30:250
2877499179,6cyclictest0-21swapper/311:09:303
2877499178,6cyclictest32466irq/58-eth011:35:153
2877499172,10cyclictest0-21swapper/308:25:173
2877499171,13cyclictest0-21swapper/309:00:203
2877499171,13cyclictest0-21swapper/307:10:023
2875699176,3cyclictest14413-21diskmemload09:45:031
2875699176,3cyclictest14413-21diskmemload09:45:021
2875699175,3cyclictest131rcu_preempt07:15:221
2875699174,4cyclictest131rcu_preempt08:05:181
2875699173,3cyclictest131rcu_preempt07:45:161
2875699173,11cyclictest15593-21mailstats08:00:291
2875699173,11cyclictest15593-21mailstats08:00:291
2875699171,6cyclictest24091-21irqrtprio11:25:231
2875699171,5cyclictest13917-21cut08:35:231
28756991710,5cyclictest0-21swapper/107:10:011
2874899175,9cyclictest6999-21apt07:55:010
2874899173,11cyclictest12312-21kthreadcore07:25:220
28748991712,3cyclictest12-21ksoftirqd/007:10:260
2877499164,10cyclictest0-21swapper/307:55:143
2877499164,10cyclictest0-21swapper/307:55:133
2877499161,10cyclictest0-21swapper/308:40:233
2875699168,6cyclictest8382-21kthreadcore07:20:241
2875699168,6cyclictest19109-21ntp_kernel_pll_07:30:271
2875699161,9cyclictest0-21swapper/111:30:141
2875699161,9cyclictest0-21swapper/111:30:131
2875699161,6cyclictest27053-21irqrtprio08:15:211
2875699161,12cyclictest20474-21apt-get07:35:161
2877499152,9cyclictest0-21swapper/307:30:153
2877499152,10cyclictest0-21swapper/307:35:193
2877499151,10cyclictest0-21swapper/309:05:233
2877499151,10cyclictest0-21swapper/308:15:183
28774991510,3cyclictest0-21swapper/307:10:413
2877499149,3cyclictest0-21swapper/308:20:193
2877499143,7cyclictest0-21swapper/308:45:143
2877499143,6cyclictest0-21swapper/308:35:193
2874899143,9cyclictest28770-21latency_hist07:10:020
2877499138,3cyclictest0-21swapper/307:40:103
2877499131,8cyclictest0-21swapper/307:20:173
2877499121,9cyclictest0-21swapper/308:55:203
2877499121,5cyclictest0-21swapper/307:45:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional