You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-15 - 02:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Sep 15, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
289692240163,25sleep30-21swapper/319:06:343
289082235168,55sleep20-21swapper/219:05:442
292062222191,20sleep00-21swapper/019:09:370
272832201166,23sleep10-21swapper/119:05:111
311222920,2sleep30-21swapper/322:50:233
29337997368,3cyclictest0-21swapper/223:19:162
150702490,1sleep10-21swapper/122:15:141
29392460,1sleep20-21swapper/200:15:142
307192450,4sleep10-21swapper/120:25:021
22602420,4sleep011-21rcuc/021:45:190
2934199385,7cyclictest19330-40dnf19:56:413
234002340,4sleep22933799cyclictest21:20:172
151862340,1sleep20-21swapper/221:00:262
29341993312,5cyclictest16804-21grep22:20:113
29341993312,5cyclictest16804-21grep22:20:113
2934199331,10cyclictest18873-21/usr/sbin/munin22:25:103
29341993213,12cyclictest0-21swapper/321:44:553
2934199321,13cyclictest0-21swapper/322:02:493
2932799327,12cyclictest18048-21awk21:10:000
2934199311,4cyclictest0-21swapper/321:59:403
2934199311,14cyclictest0-21swapper/321:01:043
2934199311,13cyclictest6974-21hddtemp_smartct23:10:153
2933199311,2cyclictest0-21swapper/120:19:021
2934199301,6cyclictest21984-21/usr/sbin/munin22:30:183
2934199291,14cyclictest0-21swapper/322:48:083
2934199291,13cyclictest0-21swapper/323:58:043
2934199291,13cyclictest0-21swapper/320:12:123
2933199291,14cyclictest0-21swapper/119:29:531
2933199290,3cyclictest0-21swapper/122:31:271
2933199290,2cyclictest0-21swapper/122:09:301
2933199290,2cyclictest0-21swapper/100:22:021
29327992912,15cyclictest19277-21perf23:40:010
29327992912,15cyclictest19277-21perf23:40:010
325632280,4sleep22933799cyclictest22:55:162
2934199281,12cyclictest18030-21diskmemload00:08:533
2933199281,13cyclictest0-21swapper/119:35:531
2933199281,12cyclictest0-21swapper/120:46:181
2933199281,12cyclictest0-21swapper/120:33:421
2933199280,2cyclictest0-21swapper/123:55:581
2933199280,2cyclictest0-21swapper/122:38:471
2933199280,2cyclictest0-21swapper/122:14:181
2933199280,2cyclictest0-21swapper/100:01:581
2934199271,5cyclictest19965-21cpuspeed_turbos20:00:143
2934199271,14cyclictest0-21swapper/322:09:293
2934199271,13cyclictest0-21swapper/321:45:443
2934199271,12cyclictest0-21swapper/323:04:413
2934199271,12cyclictest0-21swapper/321:11:003
2934199271,12cyclictest0-21swapper/321:11:003
2934199271,12cyclictest0-21swapper/300:17:573
2933199271,12cyclictest0-21swapper/121:08:101
2933199271,12cyclictest0-21swapper/121:01:061
2933199271,11cyclictest0-21swapper/119:30:291
2933199270,12cyclictest0-21swapper/123:46:181
2932799271,12cyclictest7-21ksoftirqd/021:00:000
277892270,4sleep07-21ksoftirqd/021:30:180
118052270,4sleep121-21rcuc/120:55:121
2934199269,3cyclictest0-21swapper/319:29:123
2934199262,9cyclictest17124-21ntp_states21:05:203
2934199261,5cyclictest15095-21/usr/sbin/munin23:30:203
2934199261,13cyclictest0-21swapper/320:52:033
2934199261,13cyclictest0-21swapper/300:22:373
2934199261,12cyclictest0-21swapper/320:09:363
2934199261,12cyclictest0-21swapper/320:09:363
2934199261,12cyclictest0-21swapper/319:48:243
2934199261,12cyclictest0-21swapper/319:43:283
2934199261,12cyclictest0-21swapper/319:35:323
2934199261,12cyclictest0-21swapper/300:26:493
2934199261,11cyclictest0-21swapper/319:11:373
29341992610,3cyclictest0-21swapper/322:17:313
2934199260,14cyclictest0-21swapper/323:15:293
2934199260,12cyclictest0-21swapper/321:23:323
2933199260,2cyclictest0-21swapper/121:41:531
2934199259,3cyclictest0-21swapper/320:33:363
2934199259,3cyclictest0-21swapper/319:52:003
2934199251,16cyclictest0-21swapper/319:21:083
2934199251,12cyclictest0-21swapper/321:37:563
2934199251,12cyclictest0-21swapper/319:17:443
29341992510,3cyclictest0-21swapper/319:30:203
2933199251,20cyclictest24627-21munin-run21:25:011
2933199250,11cyclictest0-21swapper/119:53:411
2933199250,10cyclictest0-21swapper/119:18:261
2932799252,11cyclictest8288-21awk23:15:000
2934199249,3cyclictest0-21swapper/323:46:173
2934199249,3cyclictest0-21swapper/323:42:203
2934199241,11cyclictest0-21swapper/323:52:013
29341992410,7cyclictest0-21swapper/300:13:313
29341992410,3cyclictest0-21swapper/323:29:243
2934199240,12cyclictest0-21swapper/300:33:013
2933199241,9cyclictest0-21swapper/121:53:591
2933199241,16cyclictest19311-21ntp_kernel_pll_21:10:221
2933199241,16cyclictest19311-21ntp_kernel_pll_21:10:221
2933199240,9cyclictest0-21swapper/121:16:181
2932799249,14cyclictest24023-21awk20:10:010
2932799249,14cyclictest24023-21awk20:10:010
248452240,4sleep124846-21grep20:10:171
2934199239,3cyclictest0-21swapper/323:06:333
2934199239,3cyclictest0-21swapper/320:18:233
2934199239,3cyclictest0-21swapper/300:02:093
2934199238,3cyclictest0-21swapper/321:15:083
2934199231,14cyclictest0-21swapper/320:46:193
2934199231,11cyclictest0-21swapper/300:37:573
2933199238,12cyclictest2435-21awk21:45:211
29331992316,4cyclictest31212-21idleruntime-cro21:40:011
29331992312,3cyclictest9793-21df20:50:131
2933199231,16cyclictest0-21swapper/119:55:171
2932799232,7cyclictest1453-40dnf22:57:390
2934199228,3cyclictest0-21swapper/322:38:353
2934199228,3cyclictest0-21swapper/320:55:313
2934199227,3cyclictest0-21swapper/322:56:093
2934199227,3cyclictest0-21swapper/321:25:123
2934199221,14cyclictest0-21swapper/321:30:043
2934199221,14cyclictest0-21swapper/320:37:513
2934199221,11cyclictest0-21swapper/321:50:043
29337992215,4cyclictest21743-21hddtemp_smartct22:30:172
2933199221,9cyclictest0-21swapper/119:42:051
2933199221,10cyclictest0-21swapper/121:27:211
2933199221,10cyclictest0-21swapper/119:46:371
2932799226,6cyclictest0-21swapper/000:17:520
2932799224,10cyclictest0-21swapper/022:02:180
2932799220,10cyclictest0-21swapper/023:32:500
2932799220,10cyclictest0-21swapper/021:51:580
2934199211,14cyclictest0-21swapper/322:41:283
2934199211,12cyclictest0-21swapper/323:20:093
2933799216,3cyclictest21818-21perf20:05:002
2933799214,3cyclictest30-21rcuc/220:35:012
2933799211,6cyclictest0-21swapper/221:14:152
2933799211,6cyclictest0-21swapper/221:14:152
29337992110,3cyclictest27440-21awk22:45:012
2933199216,12cyclictest0-21swapper/121:57:281
2933199211,9cyclictest0-21swapper/100:34:381
2933199211,7cyclictest18030-21diskmemload00:05:131
2933199211,6cyclictest0-21swapper/123:54:341
2933199211,3cyclictest2897-21diskstats00:15:131
2933199211,3cyclictest20190-21/usr/sbin/munin23:40:151
2933199211,3cyclictest0-21swapper/123:09:021
2933199211,15cyclictest0-21swapper/123:37:271
2933199211,15cyclictest0-21swapper/123:37:271
2933199210,9cyclictest0-21swapper/123:15:311
2932799217,3cyclictest0-21swapper/023:44:300
2932799214,7cyclictest0-21swapper/023:19:380
2932799211,6cyclictest1004-21hddtemp_smartct00:10:170
29327992111,6cyclictest32302-21systemd-journal00:07:510
29327992110,3cyclictest4785-21hddtemp_smartct23:05:150
123642210,5sleep112366-21ntp_kernel_err00:35:201
2934199201,11cyclictest12279-21/usr/sbin/munin22:10:233
2934199201,10cyclictest28627-21/usr/sbin/munin20:20:113
2933799202,7cyclictest10499-21tr20:50:192
2933799201,14cyclictest0-21swapper/221:37:062
29337992010,3cyclictest87350irq/126-enp1s0-21:55:122
2933199209,3cyclictest6886-21gltestperf23:10:141
2933199201,14cyclictest0-21swapper/123:28:381
2933199201,14cyclictest0-21swapper/123:23:311
2933199201,14cyclictest0-21swapper/122:21:291
2933199201,14cyclictest0-21swapper/122:21:291
2933199201,14cyclictest0-21swapper/122:00:291
2933199201,14cyclictest0-21swapper/121:31:201
2933199201,14cyclictest0-21swapper/120:43:021
2933199201,14cyclictest0-21swapper/120:06:211
2933199201,14cyclictest0-21swapper/120:06:211
2933199201,14cyclictest0-21swapper/120:00:081
2933199201,11cyclictest1453-40dnf22:57:391
2933199201,10cyclictest0-21swapper/123:04:071
2932799206,5cyclictest0-21swapper/023:04:130
2932799204,3cyclictest87350irq/126-enp1s0-21:59:120
2932799202,10cyclictest11669-21cat23:20:210
2932799201,11cyclictest0-21swapper/021:35:190
2932799201,10cyclictest27143-21ntp_states23:55:200
2932799200,9cyclictest4733-21kworker/0:000:22:230
2932799200,9cyclictest0-21swapper/022:13:130
205872201,2sleep10-21swapper/122:26:551
2934199191,2cyclictest0-21swapper/323:37:283
2934199191,2cyclictest0-21swapper/323:37:283
2934199191,15cyclictest31247-21/usr/sbin/munin20:25:133
2934199191,14cyclictest0-21swapper/320:42:033
2933799198,3cyclictest30-21rcuc/219:56:382
2933799193,6cyclictest17684-21fschecks_count23:35:152
2933799193,6cyclictest17684-21fschecks_count23:35:152
2933799191,9cyclictest4840-21munin-run20:40:012
2933799191,9cyclictest27187-21grep21:30:142
2933799191,8cyclictest0-21swapper/200:34:432
29337991913,3cyclictest16009-21memory23:30:192
2933199196,2cyclictest31542-21munin-node19:15:021
2933199191,8cyclictest0-21swapper/100:25:141
2933199191,2cyclictest0-21swapper/119:20:181
2933199191,17cyclictest0-21swapper/122:52:161
2933199191,14cyclictest0-21swapper/123:30:391
2933199191,13cyclictest0-21swapper/120:37:341
2933199191,12cyclictest0-21swapper/122:44:451
2933199191,10cyclictest31067-21expr20:25:121
2933199190,12cyclictest0-21swapper/122:45:511
2932799197,4cyclictest0-21swapper/022:36:450
2932799197,3cyclictest0-21swapper/000:37:360
2932799196,6cyclictest0-21swapper/023:52:180
2932799196,6cyclictest0-21swapper/000:30:200
2932799195,8cyclictest0-21swapper/022:21:110
2932799195,8cyclictest0-21swapper/022:21:110
2932799195,3cyclictest0-21swapper/022:31:040
2932799194,3cyclictest0-21swapper/021:16:520
2932799194,2cyclictest87350irq/126-enp1s0-21:29:210
29327991912,4cyclictest26965-21grep20:15:160
2932799191,11cyclictest0-21swapper/000:25:490
2932799191,10cyclictest9457-21/usr/sbin/munin20:50:130
2932799191,10cyclictest0-21swapper/022:45:190
2932799191,10cyclictest0-21swapper/022:15:280
2932799191,10cyclictest0-21swapper/021:10:360
2932799191,10cyclictest0-21swapper/021:10:360
2932799190,8cyclictest0-21swapper/022:54:330
2932799190,6cyclictest5145-21kworker/u8:123:26:320
271422190,2sleep20-21swapper/223:55:212
239662191,3sleep20-21swapper/220:09:382
239662191,3sleep20-21swapper/220:09:382
95562180,3sleep20-21swapper/222:01:182
6002181,2sleep00-21swapper/021:40:350
2933799188,3cyclictest87350irq/126-enp1s0-20:21:452
2933799187,3cyclictest87650irq/129-enp1s0-23:05:192
2933799186,3cyclictest32-21ksoftirqd/222:45:232
2933799185,5cyclictest755-21rngd20:25:142
2933799184,3cyclictest87350irq/126-enp1s0-22:38:382
2933799182,6cyclictest20847-21munin-run22:30:012
2933799181,5cyclictest6649-21/usr/sbin/munin19:30:112
29337991813,3cyclictest30-21rcuc/219:15:022
29337991813,3cyclictest12748-21grep22:10:142
2933799181,11cyclictest0-21swapper/223:45:172
29331991815,2cyclictest0-21swapper/100:10:341
2932799184,11cyclictest0-21swapper/020:25:380
2932799181,8cyclictest23690-21date23:50:020
29327991811,4cyclictest22811-21tr21:20:110
2932799181,10cyclictest13116-21kworker/0:121:02:140
2932799180,8cyclictest0-21swapper/022:09:110
22492181,2sleep00-21swapper/019:20:070
2933799175,4cyclictest12007-21switchtime23:20:272
2933799175,3cyclictest30-21rcuc/221:45:022
2933799174,4cyclictest30-21rcuc/220:45:132
2933799173,7cyclictest0-21swapper/223:13:362
2933799172,6cyclictest0-21swapper/200:10:142
2933799171,8cyclictest87350irq/126-enp1s0-22:50:202
2933799171,7cyclictest31643-21munin-node20:40:302
2933799171,7cyclictest29237-21missed_timers00:00:202
2933799171,4cyclictest0-21swapper/223:50:162
29337991714,2cyclictest32-21ksoftirqd/200:30:002
29337991713,3cyclictest0-21swapper/222:20:052
29337991713,3cyclictest0-21swapper/222:20:042
29337991713,3cyclictest0-21swapper/221:28:102
29337991713,3cyclictest0-21swapper/200:37:092
29337991713,2cyclictest909-21sssd_nss23:25:212
29337991712,3cyclictest5417-21hddtemp_smartct00:20:172
29337991712,3cyclictest4505-21netstat21:50:202
2933799171,15cyclictest0-21swapper/220:56:312
2933799171,14cyclictest0-21swapper/219:41:052
29337991711,3cyclictest827-21runrttasks22:09:342
29337991711,3cyclictest2356-21ntpq21:45:202
29337991711,3cyclictest21520-21seq21:15:212
2933799171,10cyclictest0-21swapper/200:07:482
2932799175,5cyclictest0-21swapper/000:02:190
2932799175,11cyclictest0-21swapper/020:45:020
2932799172,12cyclictest15908-21grep19:50:150
2932799171,9cyclictest0-21swapper/022:25:040
2932799171,4cyclictest7261-21/usr/sbin/munin20:45:170
29327991712,3cyclictest8616-21turbostat.cron19:35:020
29327991712,3cyclictest13425-21ls19:45:120
2932799171,15cyclictest0-21swapper/020:10:170
2932799171,14cyclictest0-21swapper/020:02:180
2932799171,12cyclictest9480-21grep19:35:150
2932799171,10cyclictest20600-21kworker/u8:022:40:380
29327991710,4cyclictest31491-21latency_hist19:15:020
2933799161,9cyclictest12050irq/123-ahci[0023:00:142
2933799161,7cyclictest87350irq/126-enp1s0-22:15:232
2933799161,6cyclictest0-21swapper/219:25:152
2933799161,4cyclictest0-21swapper/219:45:202
29337991612,3cyclictest0-21swapper/223:40:142
29337991612,3cyclictest0-21swapper/220:19:052
29337991612,3cyclictest0-21swapper/219:18:192
2933799161,14cyclictest0-21swapper/219:20:142
29337991611,3cyclictest9485-21expr19:35:152
29337991611,3cyclictest26411-21sed20:15:012
29337991610,3cyclictest16507-21fschecks_count21:05:142
2932799163,4cyclictest30146-21sed20:20:240
2932799161,6cyclictest0-21swapper/019:40:290
29327991611,3cyclictest4979-21fschecks_time19:25:160
29327991611,3cyclictest17636-21/usr/sbin/munin19:55:210
29327991610,3cyclictest943-21load19:15:190
2933799151,5cyclictest15786-21diskstats19:50:132
2932799152,4cyclictest1815-21pmu-power20:30:230
2932799151,12cyclictest2868-21/usr/sbin/munin20:35:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional