You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-13 - 17:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Sep 13, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18492238162,25sleep20-21swapper/207:07:532
17252237160,24sleep00-21swapper/007:06:180
18632236159,25sleep10-21swapper/107:08:041
19332220149,23sleep30-21swapper/307:09:003
170402780,2sleep00-21swapper/011:20:210
298692670,4sleep10-21swapper/111:50:181
191632610,2sleep00-21swapper/010:10:400
83452520,2sleep30-21swapper/308:35:143
144012520,2sleep30-21swapper/311:15:183
211499391,5cyclictest0-21swapper/209:55:282
212199351,20cyclictest26266-21/usr/sbin/munin11:40:353
211499351,16cyclictest0-21swapper/212:07:212
212199341,22cyclictest0-21swapper/311:59:393
2121993310,5cyclictest0-21swapper/308:25:583
212199311,20cyclictest14634-21snmpd09:45:053
211499311,15cyclictest0-21swapper/208:15:102
211499311,11cyclictest2272-21kworker/2:210:40:242
2121993014,8cyclictest0-21swapper/309:36:113
212199301,12cyclictest0-21swapper/311:08:313
2114992910,11cyclictest86750irq/128-enp1s0-08:54:522
210499292,14cyclictest2548-21munin-run10:50:010
212199281,12cyclictest0-21swapper/310:19:113
212199281,11cyclictest0-21swapper/310:20:233
210999282,14cyclictest18272-21cut11:25:151
210999281,11cyclictest0-21swapper/111:06:001
210999280,20cyclictest0-21swapper/108:30:201
196202280,4sleep1210999cyclictest07:45:251
2121992717,7cyclictest31960-21grep10:40:243
2121992711,4cyclictest0-21swapper/309:00:183
2121992711,4cyclictest0-21swapper/309:00:183
212199271,11cyclictest0-21swapper/310:28:393
212199270,9cyclictest0-21swapper/310:39:193
211499275,8cyclictest780-21rngd09:01:442
211499275,8cyclictest780-21rngd09:01:442
211499271,5cyclictest4185-21perl08:25:182
210999271,14cyclictest27776-21irqstats11:45:201
210999270,12cyclictest31646-21memory10:40:201
210499278,16cyclictest10670-21munin-run09:55:000
2104992717,3cyclictest17231-1kworker/u9:012:19:280
164292270,2sleep011-21rcuc/007:40:140
119212270,3sleep3212199cyclictest11:10:123
118002270,3sleep111804-21unixbench_multi08:41:051
211499263,9cyclictest2272-21kworker/2:208:08:562
210999264,13cyclictest86750irq/128-enp1s0-10:10:131
210999261,15cyclictest0-21swapper/107:10:081
210999261,14cyclictest5286-40dnf12:08:561
210999260,7cyclictest0-21swapper/112:31:421
161122260,4sleep1210999cyclictest10:05:201
161122260,4sleep1210999cyclictest10:05:191
212199252,14cyclictest14634-21snmpd10:10:513
212199251,11cyclictest14634-21snmpd09:16:203
212199251,11cyclictest0-21swapper/308:15:203
211499251,11cyclictest0-21swapper/211:24:222
2114992510,4cyclictest0-21swapper/211:08:462
211499250,12cyclictest0-21swapper/208:33:262
210999259,3cyclictest870-21df10:45:151
210999259,3cyclictest7577-21latency_hist12:15:021
2109992512,3cyclictest17452-21smart_sda07:40:251
210999251,11cyclictest24331-21chrt11:37:331
210999251,11cyclictest0-21swapper/109:12:491
210999250,6cyclictest0-21swapper/112:23:281
210999250,6cyclictest0-21swapper/112:23:281
249462241,3sleep025047-21memory10:25:180
212199249,3cyclictest0-21swapper/307:18:593
212199245,7cyclictest0-21swapper/312:23:393
212199245,7cyclictest0-21swapper/312:23:393
212199245,10cyclictest9943-21diskstats07:25:153
212199241,16cyclictest0-21swapper/309:59:513
212199241,15cyclictest16315-21df_inode11:20:133
2121992411,4cyclictest0-21swapper/309:12:323
212199240,2cyclictest86550irq/126-enp1s0-12:38:353
211499248,3cyclictest0-21swapper/210:55:102
210999242,8cyclictest22777-21/usr/sbin/munin07:55:171
210999242,8cyclictest22777-21/usr/sbin/munin07:55:171
210999241,10cyclictest24553-21fschecks_count10:25:151
210999240,6cyclictest0-21swapper/110:18:241
210999240,5cyclictest0-21swapper/111:40:441
212199239,3cyclictest0-21swapper/308:50:483
212199238,4cyclictest86550irq/126-enp1s0-08:07:333
212199237,11cyclictest0-21swapper/307:13:473
212199231,12cyclictest0-21swapper/309:53:153
212199231,12cyclictest0-21swapper/308:04:073
211499237,3cyclictest0-21swapper/209:37:262
211499233,2cyclictest86750irq/128-enp1s0-10:14:022
211499231,5cyclictest0-21swapper/211:26:182
211499231,12cyclictest0-21swapper/207:38:062
211499231,11cyclictest0-21swapper/210:19:582
211499231,10cyclictest0-21swapper/209:45:282
211499230,2cyclictest86750irq/128-enp1s0-07:55:142
211499230,2cyclictest86750irq/128-enp1s0-07:55:142
210999232,18cyclictest29159-21munin-run08:10:011
210999231,3cyclictest22343-21/usr/sbin/munin11:35:021
210999231,3cyclictest15041-21smartctl11:15:281
210999231,3cyclictest0-21swapper/109:46:301
210999231,20cyclictest0-21swapper/107:15:431
2109992311,3cyclictest12839-21smart_sda11:10:241
210999230,7cyclictest0-21swapper/110:53:071
210999230,6cyclictest0-21swapper/111:03:501
210999230,6cyclictest0-21swapper/109:57:011
210999230,5cyclictest0-21swapper/110:36:201
210999230,5cyclictest0-21swapper/109:19:251
210999230,16cyclictest0-21swapper/109:43:351
210499236,15cyclictest23252-21awk09:10:010
210499232,6cyclictest0-21swapper/011:08:480
210499231,15cyclictest30238-21kworker/0:311:10:370
66912220,5sleep3401ktimersoftd/308:30:203
212199228,3cyclictest0-21swapper/310:09:273
212199228,3cyclictest0-21swapper/310:09:273
212199228,3cyclictest0-21swapper/307:38:003
212199223,11cyclictest4976-21/usr/sbin/munin10:55:183
212199221,8cyclictest0-21swapper/312:09:283
212199221,10cyclictest6869-21kworker/u8:211:03:023
2121992210,3cyclictest0-21swapper/310:01:593
212199220,13cyclictest0-21swapper/312:30:153
211499228,4cyclictest2272-21kworker/2:207:42:582
211499221,6cyclictest0-21swapper/209:52:282
211499221,12cyclictest2272-21kworker/2:207:15:272
211499220,11cyclictest0-21swapper/207:48:462
2109992218,3cyclictest0-21swapper/107:24:061
210999221,20cyclictest0-21swapper/111:55:201
210999220,7cyclictest0-21swapper/112:18:021
210999220,6cyclictest0-21swapper/112:04:471
210499221,3cyclictest0-21swapper/012:15:010
139372221,3sleep110125-21kworker/1:212:27:081
212199217,3cyclictest0-21swapper/308:41:023
212199216,3cyclictest0-21swapper/310:47:073
212199216,3cyclictest0-21swapper/310:30:413
2121992115,3cyclictest21920-21sed09:05:173
212199211,14cyclictest0-21swapper/311:35:013
212199211,11cyclictest0-21swapper/308:14:583
212199210,13cyclictest0-21swapper/309:30:113
212199210,12cyclictest0-21swapper/308:55:163
212199210,12cyclictest0-21swapper/307:40:503
211499214,2cyclictest2272-21kworker/2:209:16:202
211499212,3cyclictest32-21ksoftirqd/210:37:542
211499211,5cyclictest9-21rcu_preempt08:35:202
211499211,5cyclictest0-21swapper/212:18:462
211499211,5cyclictest0-21swapper/211:48:332
210999211,7cyclictest14156-21ntp_states10:00:231
210999211,6cyclictest0-21swapper/111:21:411
2109992116,2cyclictest8698-21/usr/sbin/munin09:50:251
210999211,3cyclictest0-21swapper/109:24:061
210999211,2cyclictest0-21swapper/107:50:221
210999211,18cyclictest19475-21/usr/sbin/munin09:00:181
210999211,18cyclictest19475-21/usr/sbin/munin09:00:181
210999210,3cyclictest0-21swapper/110:56:071
210999210,3cyclictest0-21swapper/109:27:351
210499218,5cyclictest17231-1kworker/u9:009:15:020
210499217,5cyclictest17231-1kworker/u9:011:49:400
210499215,8cyclictest6502-21/usr/sbin/munin09:45:260
210499211,9cyclictest0-21swapper/011:30:080
210499211,5cyclictest0-21swapper/011:29:490
2104992114,4cyclictest24552-21basename11:40:010
210499211,12cyclictest0-21swapper/010:59:330
77432201,2sleep07801-21fschecks_count07:20:160
309072200,3sleep20-21swapper/211:52:152
212199208,4cyclictest0-21swapper/312:17:253
212199208,3cyclictest29836-21hddtemp_smartct11:50:163
212199204,7cyclictest0-21swapper/310:50:053
212199204,6cyclictest0-21swapper/311:38:423
2121992013,4cyclictest28936-21ntpq09:20:243
2121992013,4cyclictest12548-21df_inode12:25:143
212199201,12cyclictest0-21swapper/309:41:353
212199201,11cyclictest0-21swapper/311:48:153
212199201,11cyclictest0-21swapper/311:29:213
212199201,11cyclictest0-21swapper/309:26:263
2121992010,3cyclictest1474-21expr12:00:133
212199200,12cyclictest0-21swapper/307:21:593
211499204,3cyclictest86750irq/128-enp1s0-12:30:432
211499202,2cyclictest86550irq/126-enp1s0-11:03:522
211499202,11cyclictest2272-21kworker/2:210:45:202
211499201,7cyclictest0-21swapper/209:27:572
211499201,5cyclictest8350-21kworker/u8:110:06:052
211499201,5cyclictest8350-21kworker/u8:110:06:052
211499201,3cyclictest2272-21kworker/2:209:34:402
211499201,2cyclictest0-21swapper/209:21:302
211499201,15cyclictest0-21swapper/211:55:252
211499201,12cyclictest0-21swapper/210:22:302
211499201,12cyclictest0-21swapper/208:49:022
211499200,13cyclictest0-21swapper/210:25:102
210999202,3cyclictest0-21swapper/108:47:591
210999202,17cyclictest0-21swapper/110:34:171
210999202,17cyclictest0-21swapper/109:38:201
210999202,17cyclictest0-21swapper/108:15:331
210999202,17cyclictest0-21swapper/107:29:121
210999201,9cyclictest23103-21proc_pri10:20:251
2109992016,3cyclictest0-21swapper/107:30:181
210999201,18cyclictest0-21swapper/108:29:031
210499205,7cyclictest86750irq/128-enp1s0-09:40:180
210499205,2cyclictest17231-1kworker/u9:012:06:530
210499204,7cyclictest0-21swapper/012:35:250
210499204,14cyclictest4288-21awk07:15:020
210499201,9cyclictest0-21swapper/012:32:110
210499201,8cyclictest0-21swapper/011:01:350
210499201,5cyclictest21998-21cpu10:20:130
210499201,5cyclictest0-21swapper/010:50:240
2104992013,4cyclictest8139-21cpuspeed_turbos08:35:120
210499201,11cyclictest1402-21/usr/sbin/munin08:20:260
210499201,10cyclictest1172-21/usr/sbin/munin12:00:200
210499201,10cyclictest10008-21/usr/sbin/munin12:20:160
210499201,10cyclictest10008-21/usr/sbin/munin12:20:160
174782200,3sleep20-21swapper/208:55:202
5722190,2sleep00-21swapper/011:56:200
212199194,8cyclictest0-21swapper/307:53:323
2121991912,4cyclictest5958-21df_inode12:10:153
212199191,10cyclictest22777-21/usr/sbin/munin07:55:263
212199191,10cyclictest22777-21/usr/sbin/munin07:55:263
212199190,12cyclictest0-21swapper/308:20:123
212199190,11cyclictest0-21swapper/308:49:063
212199190,11cyclictest0-21swapper/307:46:033
211499191,8cyclictest30-21rcuc/207:25:202
211499191,7cyclictest0-21swapper/209:42:352
211499191,6cyclictest0-21swapper/208:40:212
211499191,3cyclictest0-21swapper/208:00:312
211499191,2cyclictest2272-21kworker/2:211:15:162
211499191,2cyclictest2272-21kworker/2:208:20:122
2114991911,2cyclictest86750irq/128-enp1s0-07:33:162
211499191,11cyclictest0-21swapper/211:30:222
210999191,8cyclictest23-21ksoftirqd/108:20:171
210999191,3cyclictest0-21swapper/112:35:371
2109991913,4cyclictest22480-21rt-features09:05:241
2109991912,4cyclictest25318-21df_inode08:00:141
210999191,17cyclictest0-21swapper/108:14:471
210999191,17cyclictest0-21swapper/107:36:431
210999191,11cyclictest0-21swapper/109:30:241
210999190,17cyclictest0-21swapper/108:50:211
210499196,8cyclictest0-21swapper/007:15:160
210499196,3cyclictest3871-21sshd09:37:340
210499195,7cyclictest9594-21/usr/sbin/munin07:25:150
210499192,14cyclictest25188-21grep11:40:170
210499191,8cyclictest0-21swapper/012:26:310
210499191,4cyclictest25054-21/usr/sbin/munin09:10:300
2104991914,3cyclictest17291-21munin-run10:10:000
2104991914,3cyclictest17291-21munin-run10:10:000
210499191,10cyclictest29709-21sshd09:23:350
210499191,10cyclictest14634-21snmpd10:30:320
210499191,10cyclictest0-21swapper/011:51:560
317902181,2sleep00-21swapper/009:27:050
212199183,3cyclictest0-21swapper/307:33:003
211499183,6cyclictest86750irq/128-enp1s0-10:30:182
211499183,5cyclictest18032-21cat12:35:272
211499181,7cyclictest9365-21latency_hist07:25:022
211499181,7cyclictest4382-21kworker/u8:009:14:542
211499181,7cyclictest0-21swapper/211:40:112
211499181,6cyclictest0-21swapper/210:51:102
211499181,3cyclictest12050irq/123-ahci[0012:13:312
211499181,2cyclictest86650irq/127-enp1s0-11:35:312
2114991812,4cyclictest2084-21latency12:00:192
211499181,15cyclictest14634-21snmpd11:10:272
211499181,13cyclictest0-21swapper/212:20:132
211499181,13cyclictest0-21swapper/212:20:132
211499181,13cyclictest0-21swapper/209:05:172
211499181,12cyclictest20816-21cat07:50:142
2109991811,4cyclictest16999-21tr08:55:161
210999180,17cyclictest0-21swapper/108:35:331
210499184,6cyclictest13093-21/usr/sbin/munin10:00:180
210499184,3cyclictest86550irq/126-enp1s0-08:30:230
210499183,12cyclictest16861-21/usr/sbin/munin08:55:230
210499181,9cyclictest836-21runrttasks08:05:230
210499181,8cyclictest0-21swapper/010:36:530
210499181,7cyclictest0-21swapper/011:15:140
210499181,5cyclictest1288-21sed08:20:010
210499181,5cyclictest0-21swapper/010:41:510
210499181,5cyclictest0-21swapper/007:55:230
210499181,5cyclictest0-21swapper/007:55:230
210499181,5cyclictest0-21swapper/007:51:170
210499181,4cyclictest12659-21sshd09:57:340
210499181,4cyclictest0-21swapper/007:33:260
2104991813,3cyclictest25580-21tail08:00:160
2104991812,3cyclictest776-21wc09:30:200
141702181,2sleep00-21swapper/008:46:470
211499172,3cyclictest30-21rcuc/207:13:472
211499171,5cyclictest0-21swapper/208:12:302
2114991712,3cyclictest13332-21pmu-power12:25:232
210499172,3cyclictest5323-21sshd08:26:340
210499171,6cyclictest18379-21sshd07:45:080
210499171,6cyclictest0-21swapper/008:40:230
210499171,5cyclictest0-21swapper/010:15:010
210499171,4cyclictest0-21swapper/009:01:110
210499171,4cyclictest0-21swapper/009:01:110
210499171,3cyclictest0-21swapper/008:51:290
2104991712,3cyclictest13990-21/usr/sbin/munin07:35:230
147852171,2sleep20-21swapper/210:01:292
210499151,4cyclictest29406-21cpu08:10:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional