You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 17:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Dec 02, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
167472241165,24sleep10-21swapper/107:07:301
167482235163,58sleep20-21swapper/207:07:312
167222232162,57sleep30-21swapper/307:07:123
167192203165,25sleep00-21swapper/007:07:100
17043991080,106cyclictest0-21swapper/212:36:372
17043998784,2cyclictest0-21swapper/209:46:182
17043997261,10cyclictest0-21swapper/209:15:142
17043995249,2cyclictest0-21swapper/208:32:072
134882520,2sleep121-21rcuc/110:40:131
215942510,1sleep00-21swapper/012:10:260
237252440,3sleep011-21rcuc/007:20:250
318622370,4sleep01703199cyclictest08:55:150
1703799361,16cyclictest14760-21sshd11:55:191
1703799341,15cyclictest0-21swapper/108:52:431
1703799341,12cyclictest0-21swapper/112:30:011
1703799341,12cyclictest0-21swapper/109:45:151
1703799340,14cyclictest5114-21crond07:55:011
1703199347,12cyclictest13634-21idleruntime-cro11:55:010
1703799331,13cyclictest19582-21fschecks_count09:40:151
1703799330,15cyclictest0-21swapper/107:59:211
187892320,5sleep339-21rcuc/310:50:243
1703799321,13cyclictest0-21swapper/109:31:131
1703799311,15cyclictest0-21swapper/110:16:031
1703799311,13cyclictest0-21swapper/111:51:451
1703799311,12cyclictest0-21swapper/112:02:551
1703799311,12cyclictest0-21swapper/111:06:491
1703799311,12cyclictest0-21swapper/108:20:251
1703799309,12cyclictest0-21swapper/112:35:151
1703799301,14cyclictest0-21swapper/110:47:551
1703799301,12cyclictest0-21swapper/108:45:551
1705099291,12cyclictest86550irq/126-enp1s0-09:31:043
1703799291,3cyclictest0-21swapper/110:51:391
17037992913,3cyclictest20936-21irqstats07:15:171
1703799291,14cyclictest0-21swapper/109:56:411
1703799291,13cyclictest0-21swapper/110:20:571
1703799291,12cyclictest0-21swapper/109:28:151
1703799291,11cyclictest0-21swapper/109:00:171
1703799291,10cyclictest0-21swapper/108:12:491
1703799290,13cyclictest0-21swapper/109:12:171
1703199295,11cyclictest5779-21awk09:10:000
1703199295,11cyclictest29329-21idleruntime-cro08:50:010
17031992910,16cyclictest25576-21idleruntime-cro09:55:010
1703799281,9cyclictest0-21swapper/109:39:371
1703799281,19cyclictest0-21swapper/108:28:031
1703799281,13cyclictest0-21swapper/112:08:331
1703799281,13cyclictest0-21swapper/107:31:131
1703799281,12cyclictest0-21swapper/107:49:511
1703799281,11cyclictest0-21swapper/108:00:271
1703799281,10cyclictest0-21swapper/110:39:451
1703799281,10cyclictest0-21swapper/107:25:271
1703199283,12cyclictest21800-21idleruntime-cro11:00:010
1703199281,12cyclictest5135-21basename07:55:010
1703799271,13cyclictest23240-21memory07:20:191
1703799271,12cyclictest0-21swapper/112:28:411
1703799271,12cyclictest0-21swapper/112:11:431
1703799271,12cyclictest0-21swapper/111:42:391
1703799271,12cyclictest0-21swapper/111:03:151
1703799271,11cyclictest0-21swapper/110:32:271
1703799271,10cyclictest0-21swapper/111:20:511
17037992710,3cyclictest30945-21date07:40:011
17037992710,3cyclictest10365-21memory11:45:191
107972270,1sleep210795-21ls10:35:012
1704399261,12cyclictest0-21swapper/212:17:052
1703799261,17cyclictest21329-21perl08:30:151
1703799261,12cyclictest0-21swapper/110:07:551
1703799261,11cyclictest0-21swapper/110:56:131
1703799261,11cyclictest0-21swapper/107:10:041
1703799261,10cyclictest0-21swapper/109:17:131
1703799261,10cyclictest0-21swapper/108:42:111
1703199267,16cyclictest18037-21munin-run12:05:020
1703799251,16cyclictest0-21swapper/108:38:411
1703799251,12cyclictest0-21swapper/111:11:051
1703799251,12cyclictest0-21swapper/110:01:451
1703799251,11cyclictest0-21swapper/109:24:531
1703799251,11cyclictest0-21swapper/109:24:531
1703799251,10cyclictest0-21swapper/110:28:171
1703799251,10cyclictest0-21swapper/110:28:171
1705099241,9cyclictest14634-21snmpd12:29:163
1703799249,3cyclictest31916-21cstates08:55:151
1703799241,7cyclictest0-21swapper/109:51:111
1703799241,13cyclictest0-21swapper/112:15:291
1703799241,13cyclictest0-21swapper/110:14:111
1703799241,13cyclictest0-21swapper/108:09:311
1703799240,11cyclictest0-21swapper/107:40:151
1703199249,3cyclictest154142/usr/sbin/munin10:45:130
1703199241,12cyclictest10158-21munin-run09:20:000
1703799231,9cyclictest0-21swapper/111:38:111
1703199231,11cyclictest31534-21awk08:55:010
45792221,3sleep34666-21grep09:05:203
1705099222,12cyclictest0-21swapper/311:06:213
1705099221,12cyclictest842-21lldpd07:55:333
17043992216,3cyclictest23487-21fschecks_time08:35:162
1703799221,16cyclictest0-21swapper/111:25:081
1703799221,15cyclictest0-21swapper/111:30:441
1703799221,15cyclictest0-21swapper/111:30:441
1703799221,11cyclictest0-21swapper/109:06:031
1703799221,10cyclictest0-21swapper/108:17:071
1703799221,10cyclictest0-21swapper/108:17:071
1703199223,16cyclictest18962-21munin-run09:40:010
107912221,3sleep20-21swapper/208:05:242
66202210,2sleep00-21swapper/007:55:300
1705099214,9cyclictest0-21swapper/309:16:213
17050992114,4cyclictest8916-21tr10:30:133
1705099211,10cyclictest5758-21diskmemload12:37:263
1704399216,6cyclictest32161-21ls10:10:002
1704399211,13cyclictest0-21swapper/211:46:332
1703799215,3cyclictest0-21swapper/111:15:591
1703799210,15cyclictest0-21swapper/112:20:031
1703199211,6cyclictest22095-21munin-run07:20:010
288982200,3sleep30-21swapper/307:35:023
210982200,2sleep00-21swapper/009:44:260
1705099206,7cyclictest0-21swapper/311:56:333
1705099206,3cyclictest0-21swapper/310:10:243
1705099204,6cyclictest0-21swapper/310:40:293
1705099201,6cyclictest0-21swapper/307:53:573
1705099201,3cyclictest2970-21if_enp2s010:15:193
1705099201,17cyclictest0-21swapper/312:19:223
1705099201,11cyclictest19186-21/usr/sbin/munin09:40:173
1705099201,11cyclictest16046-21/usr/sbin/munin12:00:223
1705099201,11cyclictest0-21swapper/309:58:573
17050992010,2cyclictest22437-21/usr/sbin/munin09:45:223
1705099200,5cyclictest0-21swapper/309:26:433
1705099200,3cyclictest0-21swapper/311:01:263
1704399202,15cyclictest21348-21grep10:55:272
1704399202,15cyclictest13167-21grep09:25:162
1704399201,7cyclictest13735-1kworker/u9:011:32:402
1704399201,7cyclictest13735-1kworker/u9:011:32:402
1704399201,7cyclictest0-21swapper/210:49:302
1704399201,6cyclictest0-21swapper/211:38:092
1703199209,3cyclictest28919-21/usr/sbin/munin07:35:210
1703199209,3cyclictest12582-21/usr/sbin/munin09:25:140
1703199202,15cyclictest3571-21idleruntime-cro09:05:000
1703199202,10cyclictest27983-21/usr/sbin/munin10:00:200
1703199202,10cyclictest2227-21/usr/sbin/munin10:15:210
1703199202,10cyclictest13214-21/usr/sbin/munin10:40:160
1703199201,10cyclictest9161-21sshd11:44:200
139742200,4sleep011-21rcuc/008:15:010
264192191,2sleep00-21swapper/007:29:130
1705099198,3cyclictest28149-21cstates10:00:133
1705099196,4cyclictest0-21swapper/312:07:373
1705099191,5cyclictest0-21swapper/309:35:113
1705099191,4cyclictest0-21swapper/310:25:443
1705099191,4cyclictest0-21swapper/310:25:443
1705099191,3cyclictest0-21swapper/310:22:283
1705099191,3cyclictest0-21swapper/307:35:153
1705099191,12cyclictest0-21swapper/312:30:233
1705099191,11cyclictest14634-21snmpd07:47:213
1705099191,11cyclictest0-21swapper/311:35:183
1705099191,10cyclictest28073-21sshd11:10:583
1705099191,10cyclictest23583-21/usr/sbin/munin09:50:153
1705099191,10cyclictest19247-21kworker/u8:208:05:303
1705099191,10cyclictest15414-21/usr/sbin/munin10:45:203
1705099191,10cyclictest0-21swapper/311:30:593
1705099191,10cyclictest0-21swapper/311:30:593
1704399195,5cyclictest24863-21/usr/sbin/munin12:20:262
1704399194,10cyclictest0-21swapper/207:29:182
1704399193,10cyclictest14634-21snmpd12:11:222
1704399191,8cyclictest0-21swapper/212:06:572
1704399191,4cyclictest0-21swapper/210:51:082
17043991913,4cyclictest13937-21kworker/u8:107:17:542
17043991913,3cyclictest12891-21fgrep11:50:252
17043991913,3cyclictest12424-21fschecks_time08:10:162
17043991912,4cyclictest26961-21fschecks_time11:10:182
1704399191,15cyclictest28269-21memory08:45:212
1704399191,10cyclictest27063-21/usr/sbin/munin12:25:242
1703199196,4cyclictest10432-21sshd11:45:200
1703199195,8cyclictest5770-40dnf09:10:020
1703199192,4cyclictest19598-21munin-run10:55:010
1703199192,10cyclictest8295-21sshd10:26:200
1703199192,10cyclictest8295-21sshd10:26:200
1703199191,7cyclictest11018-21/usr/sbin/munin10:35:240
1703199191,5cyclictest31008-21cstates11:20:120
17031991914,3cyclictest17602-21grep07:10:140
1703199191,10cyclictest928-21/usr/sbin/munin07:45:170
1703199191,10cyclictest4423-21/usr/sbin/munin10:20:240
1703199191,10cyclictest31116-21/usr/sbin/munin07:40:230
75322181,2sleep30-21swapper/308:00:083
247072180,3sleep30-21swapper/308:37:263
187882181,3sleep20-21swapper/209:38:012
1705099187,3cyclictest21530-21ntp_states12:10:243
1705099186,4cyclictest0-21swapper/309:11:283
1705099186,3cyclictest0-21swapper/311:25:343
1705099185,4cyclictest0-21swapper/310:56:533
1705099182,13cyclictest28304-21cat08:45:223
1705099182,10cyclictest0-21swapper/311:41:133
1705099181,9cyclictest16349-21/usr/sbin/munin08:20:193
1705099181,7cyclictest25153-21cat12:20:153
1705099181,4cyclictest1582-21/usr/sbin/munin09:00:253
17050991813,3cyclictest10032-21perl11:45:173
1705099181,10cyclictest0-21swapper/310:05:253
1705099180,5cyclictest0-21swapper/307:25:133
1704399185,6cyclictest0-21swapper/209:10:012
1704399183,4cyclictest13735-1kworker/u9:011:00:292
1704399182,13cyclictest31295-21fschecks_time11:20:142
1704399182,12cyclictest31844-21cpuspeed_turbos08:55:152
1704399182,10cyclictest0-21swapper/208:40:542
1704399181,8cyclictest4210-21awk10:20:002
1704399181,7cyclictest0-21swapper/211:44:252
1704399181,6cyclictest0-21swapper/212:00:442
1704399181,6cyclictest0-21swapper/210:38:082
1704399181,5cyclictest0-21swapper/208:20:032
17043991813,3cyclictest14113-21expr11:55:132
17043991812,4cyclictest8207-21sed08:00:152
17043991812,3cyclictest24896-21sh11:05:172
1704399181,10cyclictest0-21swapper/212:30:152
1703199187,3cyclictest32390-21/usr/sbin/munin10:10:250
1703199183,12cyclictest18263-21/usr/sbin/munin12:05:260
1703199182,13cyclictest10281-21grep08:05:180
1703199181,8cyclictest0-21swapper/012:19:260
1703199181,8cyclictest0-21swapper/011:55:230
1703199181,7cyclictest7541-21/usr/sbin/munin08:00:170
17031991815,2cyclictest7-21ksoftirqd/009:50:010
1703199181,14cyclictest16900-21grep08:20:140
1703199181,10cyclictest836-21runrttasks09:20:150
1703199181,10cyclictest836-21runrttasks09:20:150
1703199181,10cyclictest0-21swapper/010:09:390
169162180,4sleep011-21rcuc/009:35:010
110922180,3sleep339-21rcuc/310:35:153
297042170,1sleep20-21swapper/211:15:242
1705099172,12cyclictest14634-21snmpd07:40:213
1705099171,4cyclictest11652-21/usr/sbin/munin11:50:173
17050991713,3cyclictest0-21swapper/308:30:593
17050991712,3cyclictest25416-21expr08:40:153
17050991712,3cyclictest24504-21date07:25:023
17050991711,3cyclictest16271-21latency_hist08:20:023
17050991711,3cyclictest16271-21latency_hist08:20:023
1705099171,10cyclictest13937-21kworker/u8:107:11:213
1705099171,10cyclictest0-21swapper/311:20:123
1705099171,10cyclictest0-21swapper/311:15:513
1705099171,10cyclictest0-21swapper/308:10:153
1705099170,6cyclictest0-21swapper/309:21:333
1705099170,6cyclictest0-21swapper/309:21:333
1704399174,4cyclictest19195-21idleruntime-cro07:15:002
1704399173,4cyclictest20751-21grep09:40:292
1704399172,10cyclictest0-21swapper/207:23:432
1704399171,7cyclictest0-21swapper/210:42:072
1704399171,6cyclictest0-21swapper/207:52:492
1704399171,5cyclictest0-21swapper/211:26:172
1704399171,5cyclictest0-21swapper/210:26:332
1704399171,5cyclictest0-21swapper/210:26:332
1704399171,5cyclictest0-21swapper/208:29:022
1704399171,5cyclictest0-21swapper/207:41:572
1704399171,5cyclictest0-21swapper/207:30:162
17043991713,3cyclictest0-21swapper/210:11:072
17043991712,3cyclictest5915-21unixbench_multi10:20:262
17043991712,3cyclictest29635-21ls08:50:132
17043991712,3cyclictest27983-21/usr/sbin/munin10:00:172
17043991711,3cyclictest836-21runrttasks09:04:142
17043991711,3cyclictest30476-21unixbench_singl07:35:282
17043991710,4cyclictest24112-21fschecks_time09:50:152
1703199179,5cyclictest848-21sssd_nss12:30:240
1703199172,12cyclictest22015-21/usr/sbin/munin11:00:260
1703199171,7cyclictest0-21swapper/011:15:340
1703199171,6cyclictest8819-21/usr/sbin/munin10:30:250
1703199171,6cyclictest0-21swapper/011:30:140
1703199171,6cyclictest0-21swapper/011:30:140
1703199171,4cyclictest26728-21/usr/sbin/munin07:30:170
1703199171,4cyclictest25156-21/usr/sbin/munin08:40:190
17031991712,3cyclictest31460-21/usr/sbin/munin12:35:210
1703199171,15cyclictest0-21swapper/008:33:410
17031991711,3cyclictest26695-21cat11:10:150
17031991711,3cyclictest25289-21sort12:20:170
17031991711,3cyclictest18551-21/usr/sbin/munin08:25:140
1705099161,6cyclictest0-21swapper/308:55:233
1705099161,14cyclictest0-21swapper/308:25:043
17050991611,3cyclictest19341-21munin-node07:15:033
1705099161,12cyclictest14634-21snmpd08:53:373
1704399163,5cyclictest9-21rcu_preempt09:05:122
1704399161,7cyclictest0-21swapper/209:56:022
17043991612,2cyclictest14149-21/usr/sbin/munin08:15:142
17043991612,2cyclictest14149-21/usr/sbin/munin08:15:142
1704399161,14cyclictest0-21swapper/207:49:092
17043991611,3cyclictest6276-21wc07:55:252
17043991611,3cyclictest16392-21vmstat09:30:392
1704399161,13cyclictest10752-21grep09:20:142
1704399161,13cyclictest10752-21grep09:20:142
1703199161,6cyclictest24223-21/usr/sbin/munin11:05:190
1703199161,6cyclictest0-21swapper/012:25:480
17031991611,3cyclictest26085-21expr09:55:120
1703199161,12cyclictest22950-21/usr/sbin/munin08:35:160
17031991610,3cyclictest5744-21hostname11:35:180
17031991610,3cyclictest1725-21uname11:25:200
1703199159,3cyclictest14571-21fschecks_count08:15:140
1703199159,3cyclictest14571-21fschecks_count08:15:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional