You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 23:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Dec 03, 2025 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1622699575,6cyclictest131rcu_preempt10:00:131
16245995129,17cyclictest29566irq/58-eth009:50:133
1622699513,39cyclictest3398-21ssh11:00:141
1622699503,6cyclictest131rcu_preempt08:55:121
1622699496,40cyclictest6435-21ssh11:45:141
1622699472,37cyclictest7225-21rm09:35:161
1622699472,37cyclictest7225-21rm09:35:151
1622699441,32cyclictest25261-21ssh12:20:141
16234994214,19cyclictest27-21ksoftirqd/209:15:132
1622699423,35cyclictest15201-21ssh10:25:131
1622699421,33cyclictest17992-21ssh12:15:141
1622699421,33cyclictest17992-21ssh12:15:131
1624599414,22cyclictest29566irq/58-eth012:00:123
1623499412,26cyclictest26346irq/40-dwc2_hso11:10:132
1622699402,31cyclictest30413-21ssh10:35:151
1624599382,22cyclictest29566irq/58-eth012:30:133
16234993823,9cyclictest0-21swapper/211:30:142
16226993812,6cyclictest7803-21cron07:35:021
1621499381,27cyclictest32361-21ssh09:30:150
16234993715,13cyclictest27-21ksoftirqd/212:00:132
1623499351,18cyclictest7919-21ssh12:30:132
1622699353,25cyclictest3683-21ssh09:55:171
1622699353,25cyclictest3683-21ssh09:55:161
1622699353,25cyclictest28366-21ssh09:50:131
1623499343,20cyclictest26346irq/40-dwc2_hso07:25:142
1622699345,17cyclictest21-21ksoftirqd/109:10:131
1622699342,24cyclictest23805-21ssh11:35:141
1621499341,25cyclictest0-21swapper/007:10:140
1623499333,8cyclictest0-21swapper/210:04:232
1623499332,21cyclictest26346irq/40-dwc2_hso08:50:142
1622699331,17cyclictest22722-21ls07:50:171
1623499326,15cyclictest27-21ksoftirqd/212:05:172
16226993211,4cyclictest736-21/usr/sbin/munin10:15:301
1624599317,16cyclictest29566irq/58-eth012:35:133
16234993122,7cyclictest27-21ksoftirqd/207:40:112
1623499312,12cyclictest0-21swapper/211:15:142
1623499312,11cyclictest0-21swapper/209:05:122
1623499311,4cyclictest0-21swapper/211:45:122
1622699312,22cyclictest20590-21apt-get10:50:051
1622699311,11cyclictest27195-21apt-get12:00:031
1621499312,21cyclictest27541-21ssh12:00:050
1623499302,18cyclictest26346irq/40-dwc2_hso09:20:142
16245992920,6cyclictest29566irq/58-eth010:55:143
16245992920,6cyclictest29566irq/58-eth010:55:133
1624599291,13cyclictest29566irq/58-eth010:10:133
1623499293,24cyclictest0-21swapper/209:46:232
1623499293,19cyclictest0-21swapper/210:45:152
1623499292,21cyclictest0-21swapper/208:35:142
1623499292,21cyclictest0-21swapper/208:35:132
16234992914,12cyclictest3643-21/usr/sbin/munin09:55:252
16234992914,12cyclictest3643-21/usr/sbin/munin09:55:242
1623499291,18cyclictest26346irq/40-dwc2_hso08:00:152
16226992910,5cyclictest24408-21sshd09:24:491
1621499293,16cyclictest18854-1kworker/u9:111:55:130
1621499292,20cyclictest25906-21ssh10:10:130
1621499292,19cyclictest31973-21ntp_192.168.11409:50:230
1624599284,11cyclictest29566irq/58-eth011:50:123
1623499283,16cyclictest26346irq/40-dwc2_hso11:50:132
1621499282,20cyclictest16930-21ssh11:50:240
1624599271,16cyclictest0-21swapper/310:00:143
1624599271,13cyclictest29566irq/58-eth012:20:143
1623499273,16cyclictest26346irq/40-dwc2_hso10:10:122
1623499272,17cyclictest26346irq/40-dwc2_hso11:25:122
16234992717,7cyclictest0-21swapper/210:35:142
16234992712,12cyclictest0-21swapper/207:30:192
1622699274,19cyclictest15527-21ssh12:35:141
16226992712,3cyclictest131rcu_preempt10:10:151
1622699271,12cyclictest24157-21ntp_192.168.11411:55:251
1621499279,12cyclictest13057-1kworker/u9:009:46:240
1621499275,6cyclictest12-21ksoftirqd/011:26:370
16214992720,4cyclictest23804-21sh11:35:130
1623499263,16cyclictest26346irq/40-dwc2_hso07:15:152
1623499262,9cyclictest0-21swapper/210:15:132
1623499262,17cyclictest0-21swapper/210:50:042
1623499262,10cyclictest0-21swapper/210:55:142
1623499262,10cyclictest0-21swapper/210:55:132
1621499267,13cyclictest12-21ksoftirqd/008:45:130
1621499262,16cyclictest842-21munin-node10:05:140
16214992614,6cyclictest0-21swapper/007:40:130
1621499261,19cyclictest24866-21sh09:25:130
1621499261,15cyclictest17497-21ssh09:20:130
1623499257,12cyclictest24865-21diskmemload09:25:132
1623499256,10cyclictest27194-21cron12:00:002
1623499252,16cyclictest0-21swapper/209:50:142
1623499252,15cyclictest26346irq/40-dwc2_hso08:45:132
1621499256,6cyclictest12-21ksoftirqd/009:17:010
1621499254,5cyclictest12-21ksoftirqd/011:40:300
16245992416,5cyclictest321rcuc/310:39:403
16245992414,4cyclictest0-21swapper/311:55:133
1623499244,16cyclictest26346irq/40-dwc2_hso08:10:132
1623499243,8cyclictest0-21swapper/211:00:142
1623499243,15cyclictest26346irq/40-dwc2_hso07:10:132
1623499241,8cyclictest7228-21ssh09:35:172
1623499241,8cyclictest7228-21ssh09:35:162
16234992417,4cyclictest4811-21ssh09:10:222
1623499241,14cyclictest26353-21unixbench_singl07:50:282
1622699242,17cyclictest2701-21ssh11:40:271
1621499249,8cyclictest12-21ksoftirqd/009:40:440
1621499248,3cyclictest131rcu_preempt10:02:170
1621499248,10cyclictest12-21ksoftirqd/012:31:300
16214992416,5cyclictest151rcuc/008:50:120
16214992410,6cyclictest12-21ksoftirqd/010:29:110
1624599233,10cyclictest29566irq/58-eth008:45:133
1623499236,15cyclictest18811-21cron08:20:012
16234992317,4cyclictest19127-21ssh10:05:192
1623499231,18cyclictest18142-21cut12:15:172
1623499231,18cyclictest18142-21cut12:15:162
1623499231,13cyclictest8908-21kthreadcore10:20:192
1622699231,19cyclictest2864-21find12:05:171
16226992311,9cyclictest25616-21cron11:15:011
1621499239,4cyclictest12-21ksoftirqd/012:29:130
1621499238,7cyclictest12-21ksoftirqd/010:20:270
1621499237,8cyclictest12-21ksoftirqd/011:19:330
1621499234,2cyclictest131rcu_preempt12:17:300
1621499234,2cyclictest131rcu_preempt12:17:300
1621499232,16cyclictest16844-21ssh10:45:240
1621499232,13cyclictest12-21ksoftirqd/011:14:210
16214992317,3cyclictest0-21swapper/007:25:140
1621499231,16cyclictest3476-21ls11:00:150
16214992310,3cyclictest131rcu_preempt12:13:560
1624599224,13cyclictest0-21swapper/311:10:133
1624599223,12cyclictest0-21swapper/309:49:243
16245992213,6cyclictest29566irq/58-eth010:40:253
16245992213,6cyclictest29566irq/58-eth009:35:163
16245992213,6cyclictest29566irq/58-eth009:35:163
16245992212,7cyclictest29566irq/58-eth010:25:133
1623499222,14cyclictest26346irq/40-dwc2_hso08:55:122
16234992210,5cyclictest8118-21ssh11:24:262
1622699223,13cyclictest10198-21ssh12:10:151
1622699222,16cyclictest12633-21cron07:40:001
1622699221,16cyclictest2169-21ls07:26:101
1621499227,6cyclictest12-21ksoftirqd/010:35:010
1621499226,9cyclictest12-21ksoftirqd/012:20:110
1621499225,7cyclictest12-21ksoftirqd/011:31:240
1621499224,8cyclictest12-21ksoftirqd/009:38:320
1621499224,8cyclictest12-21ksoftirqd/009:38:310
1621499222,15cyclictest7088-21sh09:55:250
1621499222,15cyclictest7088-21sh09:55:240
16214992210,9cyclictest29757-21cron09:05:010
16214992210,9cyclictest29757-21cron09:05:000
16214992210,3cyclictest131rcu_preempt10:41:060
1624599213,13cyclictest0-21swapper/312:05:193
16245992113,5cyclictest29566irq/58-eth011:00:143
1623499219,9cyclictest19679-21cat12:35:312
1623499213,14cyclictest9285-21ntpq10:40:272
1623499213,14cyclictest26346irq/40-dwc2_hso08:05:132
1623499212,8cyclictest15210-21sh10:25:142
1623499212,14cyclictest932-21snmpd11:05:132
1623499212,10cyclictest30058-21kthreadcore07:55:222
1623499211,9cyclictest30242-21latency_hist10:35:012
1623499211,16cyclictest30686-21wc07:25:002
1623499211,15cyclictest31818-21cut11:40:192
1623499211,11cyclictest10250-21cpuspeed12:10:172
1623499211,10cyclictest9726-21grep08:40:292
1623499211,10cyclictest28556-21latency_hist08:30:022
1623499211,10cyclictest21514-21grep07:45:292
1623499211,10cyclictest16460-21kthreadcore09:40:222
16234992110,9cyclictest30048-21ssh11:38:072
1622699219,5cyclictest0-21swapper/108:30:131
1622699212,14cyclictest28337-21ssh10:55:141
1622699212,14cyclictest28337-21ssh10:55:131
1622699211,14cyclictest23037-21find10:30:131
1621499218,7cyclictest12-21ksoftirqd/012:05:080
1621499218,4cyclictest12-21ksoftirqd/010:17:410
1621499218,3cyclictest131rcu_preempt08:25:160
1621499216,5cyclictest12-21ksoftirqd/010:51:040
1621499216,3cyclictest131rcu_preempt07:15:280
1621499214,6cyclictest131rcu_preempt09:13:110
1621499211,15cyclictest17850-21cut07:45:160
16214992110,3cyclictest12-21ksoftirqd/010:55:150
16214992110,3cyclictest12-21ksoftirqd/010:55:150
1624599202,11cyclictest0-21swapper/311:40:133
1624599202,10cyclictest0-21swapper/309:25:133
1624599201,9cyclictest29566irq/58-eth007:30:143
16245992015,3cyclictest321rcuc/310:22:433
16245992015,3cyclictest321rcuc/309:16:323
16245992014,4cyclictest321rcuc/312:14:253
16245992011,7cyclictest0-21swapper/311:15:053
1624599201,13cyclictest0-21swapper/311:25:153
1623499205,12cyclictest0-21swapper/208:20:232
1623499201,9cyclictest26753-21kthreadcore09:00:212
1623499201,9cyclictest26753-21kthreadcore09:00:212
1623499201,9cyclictest1047-21latency_hist08:35:012
1623499201,16cyclictest7031-21latency_hist09:35:022
1623499201,15cyclictest25520-21idleruntime12:20:172
1623499201,10cyclictest1944-21kthreadcore12:25:222
1623499200,17cyclictest10057-21kthreadcore07:35:242
1622699208,3cyclictest21-21ksoftirqd/108:35:141
1622699208,3cyclictest21-21ksoftirqd/108:35:131
1622699202,14cyclictest16224-21apt-get11:30:121
1622699201,6cyclictest27137-21kthreadcore07:20:201
16226992013,5cyclictest0-21swapper/108:00:341
16226992012,5cyclictest0-21swapper/111:05:141
1622699200,15cyclictest25577-21ls10:10:021
1622699200,15cyclictest17825-21uniq07:45:171
1621499208,4cyclictest12-21ksoftirqd/008:15:270
1621499207,7cyclictest12-21ksoftirqd/008:05:160
1621499206,6cyclictest12-21ksoftirqd/011:10:010
1621499206,2cyclictest131rcu_preempt11:20:270
1621499205,8cyclictest12-21ksoftirqd/012:35:580
1621499205,8cyclictest12-21ksoftirqd/007:35:250
1621499204,7cyclictest131rcu_preempt08:00:280
1621499204,6cyclictest12-21ksoftirqd/007:35:020
1621499202,15cyclictest27275-21kthreadcore07:20:210
1621499202,14cyclictest29990-21apt-get09:05:110
1621499201,14cyclictest31865-21grep08:00:010
1621499200,16cyclictest1501-21missed_timers10:35:240
1624599192,11cyclictest0-21swapper/310:05:043
16245991913,4cyclictest0-21swapper/310:34:253
16245991911,6cyclictest0-21swapper/311:46:433
16245991911,5cyclictest29566irq/58-eth011:30:153
16245991911,5cyclictest29566irq/58-eth010:45:133
1622699199,8cyclictest31855-21cron08:00:021
1622699197,9cyclictest8810-21diskmemload12:30:201
1622699192,13cyclictest12246-21ntp_states11:25:271
1622699191,7cyclictest13687-21sed10:45:191
1622699191,5cyclictest0-21swapper/109:45:301
1622699191,14cyclictest10049-21ssh09:15:141
1622699191,11cyclictest3930-21memory12:25:241
1621499198,3cyclictest12-21ksoftirqd/008:20:310
1621499198,3cyclictest0-21swapper/011:45:090
1621499191,5cyclictest12-21ksoftirqd/008:30:250
1624599183,9cyclictest0-21swapper/309:55:253
1624599183,9cyclictest0-21swapper/309:55:243
1624599183,12cyclictest0-21swapper/312:15:273
1624599183,12cyclictest0-21swapper/312:15:263
1624599183,10cyclictest0-21swapper/309:44:533
16245991813,4cyclictest321rcuc/310:53:423
16245991813,3cyclictest29666irq/59-eth009:20:283
16245991813,3cyclictest0-21swapper/309:05:273
16245991813,3cyclictest0-21swapper/308:05:233
16245991812,4cyclictest321rcuc/312:29:493
16245991812,4cyclictest321rcuc/311:22:083
16245991812,4cyclictest0-21swapper/311:38:243
16245991812,4cyclictest0-21swapper/309:34:493
1624599181,11cyclictest0-21swapper/308:40:223
1623499181,9cyclictest16134-21latency_hist07:10:022
1622699189,6cyclictest20952-21cron07:15:011
1622699181,9cyclictest27133-21latency08:25:231
1622699181,7cyclictest131rcu_preempt08:15:181
16226991811,4cyclictest26525-21kthreadcore09:25:201
1622699181,12cyclictest7775-21ssh10:20:131
1621499189,3cyclictest12-21ksoftirqd/008:55:270
1621499183,3cyclictest131rcu_preempt07:55:020
16214991810,5cyclictest151rcuc/008:35:120
16214991810,5cyclictest151rcuc/008:35:110
1624599179,3cyclictest0-21swapper/308:55:133
1624599172,9cyclictest0-21swapper/307:45:143
1624599172,9cyclictest0-21swapper/307:15:153
1624599172,10cyclictest0-21swapper/309:00:153
1624599172,10cyclictest0-21swapper/309:00:143
16245991712,3cyclictest0-21swapper/308:10:173
16245991712,3cyclictest0-21swapper/308:00:253
16245991712,3cyclictest0-21swapper/307:20:163
16245991711,4cyclictest321rcuc/309:10:123
16245991711,4cyclictest0-21swapper/311:06:313
16245991711,4cyclictest0-21swapper/310:15:523
16245991711,4cyclictest0-21swapper/307:10:263
1624599171,13cyclictest0-21swapper/308:15:153
16245991710,4cyclictest0-21swapper/308:35:263
16245991710,4cyclictest0-21swapper/308:35:253
1622699179,6cyclictest29753-21cron09:05:011
1622699179,6cyclictest29753-21cron09:05:001
1622699179,5cyclictest1117-21rm11:20:151
1622699177,6cyclictest13328-21sshd11:50:111
1622699172,12cyclictest25619-21apt-get11:15:041
1622699171,8cyclictest7790-21cut08:40:211
1622699171,8cyclictest31170-21kthreadcore09:05:191
1622699171,7cyclictest19259-21fschecks_time08:20:191
1622699171,3cyclictest0-21swapper/109:30:211
1622699171,12cyclictest9117-21ssh10:40:241
1622699171,12cyclictest23657-21cut07:15:221
1621499173,9cyclictest0-21swapper/008:40:130
1621499173,5cyclictest131rcu_preempt08:10:300
1624599162,9cyclictest0-21swapper/307:55:133
16245991611,3cyclictest0-21swapper/308:50:313
16245991611,3cyclictest0-21swapper/308:30:183
16245991611,3cyclictest0-21swapper/308:25:283
1624599161,12cyclictest0-21swapper/307:50:173
16245991610,4cyclictest0-21swapper/308:20:313
16245991610,4cyclictest0-21swapper/307:35:273
1622699163,10cyclictest10092-21kthreadcore08:10:201
1622699162,10cyclictest0-21swapper/109:40:151
1622699161,5cyclictest11141-21irqcore08:45:181
1622699161,12cyclictest12889-21apt-get07:40:121
1622699161,11cyclictest15682-21fschecks_time08:50:171
1624599152,9cyclictest0-21swapper/307:40:143
1624599152,9cyclictest0-21swapper/307:25:143
1622699152,9cyclictest0-21swapper/108:05:131
1622699141,10cyclictest16324-21wc07:10:011
1621499143,9cyclictest16203-21cyclictest07:10:020
1624599111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional