You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-19 - 20:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Fri Jun 19, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10322251163,22sleep20-21swapper/207:06:212
317372238163,23sleep30-21swapper/307:05:103
11822235165,57sleep00-21swapper/007:08:160
9492200163,24sleep10-21swapper/107:05:281
1793121350,3sleep1140899cyclictest07:45:131
141499105102,2cyclictest0-21swapper/208:18:262
1414996864,3cyclictest0-21swapper/209:37:262
1404993310,11cyclictest31908-21idleruntime-cro10:45:010
209942280,4sleep230-21rcuc/209:05:162
140899281,24cyclictest0-21swapper/110:29:381
140499281,12cyclictest24700-21awk09:15:010
142199267,10cyclictest0-21swapper/310:00:213
140499263,11cyclictest14537-21/usr/sbin/munin10:05:220
140499262,12cyclictest24914-21/usr/sbin/munin09:15:190
142199249,6cyclictest0-21swapper/312:15:203
1421992417,4cyclictest31697-21/usr/sbin/munin08:15:173
1421992414,3cyclictest0-21swapper/310:15:093
140899246,9cyclictest21-21rcuc/111:55:141
140899243,2cyclictest30250-21sshd11:54:211
140899243,2cyclictest30250-21sshd11:54:211
140899241,16cyclictest0-21swapper/110:43:221
140899241,16cyclictest0-21swapper/109:29:071
140899241,16cyclictest0-21swapper/108:40:221
140899240,17cyclictest0-21swapper/109:46:311
140499241,5cyclictest28401-21hddtemp_smartct10:35:150
140899236,5cyclictest24366-21taskset09:11:011
140899231,6cyclictest0-21swapper/112:30:161
1408992316,4cyclictest6295-21chrt07:17:451
140899231,16cyclictest0-21swapper/111:32:261
140899230,6cyclictest0-21swapper/111:15:281
140899230,17cyclictest87550irq/128-enp1s0-09:38:111
140499231,6cyclictest0-21swapper/010:15:080
142199228,4cyclictest0-21swapper/309:54:313
142199227,6cyclictest0-21swapper/311:04:133
142199226,7cyclictest0-21swapper/310:10:223
142199226,6cyclictest0-21swapper/311:35:353
142199222,11cyclictest1318-21/usr/sbin/munin09:35:113
1421992212,3cyclictest0-21swapper/310:48:323
1421992212,3cyclictest0-21swapper/310:30:143
142199220,6cyclictest0-21swapper/309:12:073
141499223,6cyclictest0-21swapper/212:19:562
1408992217,3cyclictest0-21swapper/111:25:491
1408992213,3cyclictest0-21swapper/112:26:241
1408992213,3cyclictest0-21swapper/112:21:191
140899221,16cyclictest0-21swapper/110:58:461
140899221,14cyclictest0-21swapper/111:36:431
140899221,12cyclictest17886-21latency_hist12:40:011
140899221,12cyclictest11046-21memory09:55:181
140899221,12cyclictest0-21swapper/110:16:221
140899221,11cyclictest21228-21grep10:20:131
140899220,16cyclictest0-21swapper/112:10:161
140499227,13cyclictest11362-21grep11:10:150
140499221,5cyclictest0-21swapper/010:30:140
142199219,3cyclictest28690-21/usr/sbin/munin08:10:273
142199217,6cyclictest0-21swapper/311:54:253
142199217,6cyclictest0-21swapper/311:54:253
142199217,4cyclictest0-21swapper/309:48:103
142199215,6cyclictest0-21swapper/310:35:583
142199215,2cyclictest87350irq/126-enp1s0-09:57:183
142199211,6cyclictest0-21swapper/311:25:433
1421992116,3cyclictest23997-21grep10:25:163
1421992116,3cyclictest11750-21cut07:30:173
142199211,11cyclictest808-21snmpd10:08:553
142199211,11cyclictest30045-21expr10:40:123
141499219,11cyclictest17324-21kworker/u8:007:50:322
141499216,11cyclictest87550irq/128-enp1s0-09:15:232
141499215,3cyclictest12050irq/123-ahci[0008:35:012
141499211,5cyclictest0-21swapper/211:28:172
140899219,3cyclictest27663-21perl09:20:161
140899218,3cyclictest9275-21uniq07:25:121
140899218,11cyclictest17324-21kworker/u8:008:35:071
140899215,10cyclictest0-21swapper/109:15:121
140899212,14cyclictest0-21swapper/108:47:421
140899212,10cyclictest25916-21latency_hist11:45:011
1408992117,3cyclictest0-21swapper/112:00:131
1408992117,3cyclictest0-21swapper/110:10:181
140899211,3cyclictest8959-21munin-run12:20:011
140899211,18cyclictest0-21swapper/107:22:021
140899211,17cyclictest9390-21cat09:50:261
140899211,15cyclictest0-21swapper/108:30:361
140899211,14cyclictest0-21swapper/108:50:111
140899211,11cyclictest589-21sed08:20:011
140899211,11cyclictest14308-21latency_hist10:05:021
140899211,11cyclictest0-21swapper/109:43:461
140899211,10cyclictest0-21swapper/110:52:461
140899210,19cyclictest0-21swapper/110:32:471
140899210,15cyclictest0-21swapper/111:10:461
140899210,15cyclictest0-21swapper/110:49:171
140499218,3cyclictest6883-21grep09:45:220
140499212,10cyclictest23950-21/usr/sbin/munin11:40:170
140499211,5cyclictest0-21swapper/012:29:120
140499211,5cyclictest0-21swapper/010:48:330
140499211,4cyclictest24184-21grep10:25:180
140499211,11cyclictest26496-21/usr/sbin/munin08:05:140
140499211,10cyclictest16918-21grep12:35:230
140499210,18cyclictest11-21rcuc/008:30:020
142199207,5cyclictest0-21swapper/311:22:353
142199207,5cyclictest0-21swapper/310:23:313
142199205,6cyclictest0-21swapper/312:35:513
142199205,6cyclictest0-21swapper/311:33:513
142199205,6cyclictest0-21swapper/310:50:353
142199205,6cyclictest0-21swapper/309:21:233
142199205,2cyclictest87350irq/126-enp1s0-12:25:373
142199201,6cyclictest0-21swapper/309:30:573
142199201,13cyclictest0-21swapper/309:29:393
142199201,11cyclictest19895-21/usr/sbin/munin07:50:153
142199201,10cyclictest1617-21/usr/sbin/munin07:10:173
142199201,10cyclictest0-21swapper/312:03:593
142199201,10cyclictest0-21swapper/310:55:113
142199200,7cyclictest0-21swapper/311:59:533
142199200,11cyclictest87350irq/126-enp1s0-11:40:123
141499203,4cyclictest12737-21munin-run11:15:012
141499201,6cyclictest12291-21memory12:25:172
1414992016,2cyclictest32-21ksoftirqd/208:30:012
141499201,5cyclictest29673-21uniq09:25:142
141499201,10cyclictest2539-21fschecks_time10:50:142
141499201,10cyclictest23726-21munin-run11:40:022
140899208,2cyclictest13069-21crond07:35:011
140899202,16cyclictest0-21swapper/109:00:181
140899201,17cyclictest0-21swapper/112:08:561
140899201,17cyclictest0-21swapper/111:23:221
140899201,17cyclictest0-21swapper/108:06:361
140899201,13cyclictest0-21swapper/108:20:271
140899201,11cyclictest691-21chrt09:30:521
140899201,10cyclictest909-21sssd_nss07:10:241
140499202,10cyclictest26147-21/usr/sbin/munin11:45:110
140499202,10cyclictest15154-21/usr/sbin/munin11:20:160
140499201,8cyclictest0-21swapper/007:25:120
140499201,12cyclictest0-21swapper/011:06:380
140499201,11cyclictest27108-21/usr/sbin/munin09:20:170
140499201,10cyclictest24289-21/usr/sbin/munin08:00:200
53742191,2sleep00-21swapper/012:10:160
273442190,3sleep2141499cyclictest09:20:132
242852191,3sleep30-21swapper/308:00:093
142199196,4cyclictest12451-21sshd11:11:323
142199196,4cyclictest0-21swapper/312:23:293
142199191,8cyclictest0-21swapper/312:35:013
142199191,6cyclictest0-21swapper/309:05:133
141499194,8cyclictest87350irq/126-enp1s0-09:55:162
141499193,7cyclictest14072-21kworker/u8:110:14:412
141499191,7cyclictest0-21swapper/212:21:212
141499191,7cyclictest0-21swapper/211:17:032
1414991912,5cyclictest17324-21kworker/u8:007:32:452
1414991912,4cyclictest32-21ksoftirqd/208:00:002
141499191,10cyclictest8354-21awk11:05:012
140899195,6cyclictest21-21rcuc/111:45:131
140899192,16cyclictest0-21swapper/108:55:271
1408991915,3cyclictest0-21swapper/110:06:421
1408991914,3cyclictest17043-21grep07:40:251
1408991913,2cyclictest0-21swapper/111:08:511
140899191,16cyclictest0-21swapper/108:10:421
140899191,14cyclictest203062sleep107:50:151
140899191,13cyclictest0-21swapper/110:35:531
1408991911,2cyclictest0-21swapper/111:03:311
140899190,14cyclictest0-21swapper/108:27:541
140499198,3cyclictest17697-21/usr/sbin/munin07:45:210
140499194,13cyclictest0-21swapper/012:05:010
140499193,7cyclictest13303-21/usr/sbin/munin07:35:240
140499191,8cyclictest0-21swapper/010:12:370
1404991913,3cyclictest21526-21munin-plugin-st11:35:010
140499191,11cyclictest0-21swapper/011:53:080
140499191,11cyclictest0-21swapper/011:53:080
140499191,10cyclictest9182-21/usr/sbin/munin12:20:140
81872180,4sleep011-21rcuc/009:50:130
142199187,5cyclictest0-21swapper/311:05:263
142199185,4cyclictest18774-21ntpq07:45:223
142199185,4cyclictest11843-21grep08:45:123
142199185,4cyclictest0-21swapper/312:12:063
142199184,4cyclictest26496-21/usr/sbin/munin08:05:153
142199181,6cyclictest17350-21wc08:55:233
142199181,10cyclictest808-21snmpd09:02:203
142199181,10cyclictest22487-21diskmemload09:42:303
142199181,10cyclictest0-21swapper/309:18:123
141499185,5cyclictest32328-21grep09:30:172
141499185,4cyclictest32-21ksoftirqd/210:15:572
141499183,9cyclictest0-21swapper/210:45:402
141499183,4cyclictest5912-21wc09:45:132
141499182,5cyclictest4569-21munin-run12:10:022
141499182,10cyclictest87350irq/126-enp1s0-11:53:012
141499182,10cyclictest87350irq/126-enp1s0-11:53:012
141499181,7cyclictest0-21swapper/207:28:162
141499181,5cyclictest25421-21pmu-power08:00:232
141499181,5cyclictest0-21swapper/211:45:152
141499181,5cyclictest0-21swapper/209:42:172
1414991813,3cyclictest13189-21grep08:45:252
141499181,11cyclictest0-21swapper/210:30:402
1408991815,2cyclictest0-21swapper/107:55:511
1408991814,2cyclictest0-21swapper/107:35:161
140899181,3cyclictest0-21swapper/109:06:141
140899181,13cyclictest24385-21grep08:00:111
140499187,3cyclictest15339-21sshd12:32:320
140499186,3cyclictest808-21snmpd11:17:460
140499185,4cyclictest3106-21sshd09:37:310
140499182,7cyclictest30541-21/usr/sbin/munin11:55:130
140499181,9cyclictest808-21snmpd09:42:420
140499181,7cyclictest0-21swapper/010:53:140
140499181,5cyclictest0-21swapper/009:27:310
140499181,4cyclictest808-21snmpd12:17:570
140499181,2cyclictest0-21swapper/011:35:440
1404991812,3cyclictest22395-21smart_sda10:20:240
140499181,11cyclictest0-21swapper/008:35:160
140499181,10cyclictest0-21swapper/007:40:220
83832171,2sleep30-21swapper/307:21:203
59012170,2sleep01391-21cyclictest10:57:120
57892170,2sleep20-21swapper/210:55:462
24942170,3sleep00-21swapper/008:22:300
172342170,3sleep20-21swapper/207:42:102
142199179,5cyclictest909-21sssd_nss07:35:173
142199175,3cyclictest12050irq/123-ahci[0008:35:013
142199173,5cyclictest14910-21needreboot08:50:193
142199171,9cyclictest0-21swapper/311:45:063
142199171,6cyclictest24181-21cat08:00:013
142199171,4cyclictest9522-21/usr/sbin/munin08:40:123
1421991711,4cyclictest0-21swapper/312:05:083
142199170,11cyclictest0-21swapper/307:15:223
141499175,3cyclictest20185-21sshd09:03:492
141499174,3cyclictest5750irq/9-acpi10:20:162
141499173,3cyclictest87650irq/129-enp1s0-12:10:292
141499172,9cyclictest0-21swapper/211:59:102
141499172,4cyclictest2709-21munin-run08:25:012
141499172,12cyclictest13450-21cpuspeed_turbos07:35:132
141499171,7cyclictest0-21swapper/209:54:582
141499171,6cyclictest18047-21/usr/sbin/munin07:45:142
141499171,6cyclictest0-21swapper/210:36:062
141499171,5cyclictest0-21swapper/212:01:422
1414991713,3cyclictest0-21swapper/210:43:352
1414991713,3cyclictest0-21swapper/208:59:192
1414991712,3cyclictest23950-21/usr/sbin/munin11:40:252
1414991712,3cyclictest20820-21swap11:30:232
1414991712,3cyclictest16755-21vmstat11:20:272
141499171,15cyclictest0-21swapper/211:08:172
141499171,15cyclictest0-21swapper/209:13:402
141499171,15cyclictest0-21swapper/208:39:212
141499171,15cyclictest0-21swapper/207:23:472
1414991711,4cyclictest12050irq/123-ahci[0008:10:242
1414991711,3cyclictest13944-21grep12:30:142
141499171,10cyclictest0-21swapper/210:06:382
1414991710,4cyclictest12913-21/usr/sbin/munin10:00:152
140499178,3cyclictest0-21swapper/007:10:140
140499174,3cyclictest598-21latency_hist08:20:020
140499171,7cyclictest10474-21df_inode09:55:130
140499171,5cyclictest0-21swapper/012:08:560
1404991711,4cyclictest22247-21cut07:55:120
1404991711,3cyclictest6709-21diskstats11:00:130
1404991711,3cyclictest18418-21cpuspeed_turbos09:00:140
1404991711,3cyclictest17262-21tr08:55:220
140499171,10cyclictest0-21swapper/007:20:130
140499170,6cyclictest0-21swapper/008:10:200
142199161,9cyclictest0-21swapper/311:15:183
142199161,6cyclictest0-21swapper/308:28:213
1421991611,3cyclictest7379-21cpu08:35:133
142199161,13cyclictest635-21/usr/sbin/munin08:20:253
1421991611,3cyclictest17093-21vmstat07:40:273
142199161,12cyclictest8904-21/usr/sbin/munin07:25:133
141499164,2cyclictest87550irq/128-enp1s0-08:53:262
141499162,5cyclictest27162-21sed08:05:152
1414991611,3cyclictest23331-21/usr/sbin/munin10:25:112
1414991610,3cyclictest827-21runrttasks12:35:112
1414991610,3cyclictest1826-21cat07:10:142
140499169,4cyclictest14287-21fschecks_count08:50:140
140499165,3cyclictest87350irq/126-enp1s0-07:50:220
140499162,12cyclictest4515-21/usr/sbin/munin07:15:200
140499161,6cyclictest18161-21irqstats11:25:160
140499161,2cyclictest0-21swapper/008:30:070
1404991612,3cyclictest0-21swapper/009:30:450
1404991611,3cyclictest20827-21expr09:05:140
140499161,13cyclictest11081-21grep08:40:280
140499161,12cyclictest12927-21grep10:00:150
140499161,12cyclictest11716-21/usr/sbin/munin08:45:140
127892160,2sleep00-21swapper/007:31:290
141499159,3cyclictest9737-21cat08:40:132
1414991512,2cyclictest0-21swapper/207:16:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional