You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-25 - 16:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Apr 25, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
262382232164,55sleep10-21swapper/107:05:351
265732231161,58sleep00-21swapper/007:09:560
246272202167,23sleep30-21swapper/307:05:123
246262201164,24sleep20-21swapper/207:05:112
1449821800,4sleep32667499cyclictest11:35:013
503321680,5sleep22667099cyclictest07:30:202
2230321430,2sleep20-21swapper/208:10:172
26670991101,107cyclictest0-21swapper/212:28:432
26670997572,2cyclictest0-21swapper/210:57:052
26670997571,3cyclictest0-21swapper/209:28:572
26670996259,2cyclictest0-21swapper/211:46:082
89592540,2sleep08961-21unixbench_multi10:05:250
26670993935,3cyclictest0-21swapper/208:05:042
26670993935,3cyclictest0-21swapper/208:05:042
2665499345,13cyclictest6592-21awk08:50:010
2665499316,12cyclictest21362-21awk08:10:010
2665499316,12cyclictest21362-21awk08:10:010
2665499303,17cyclictest29323-21latency_hist10:55:020
2665499302,13cyclictest15385-21munin-run09:10:010
291052290,4sleep12666299cyclictest09:40:131
2666299291,5cyclictest0-21swapper/111:15:491
2665499295,14cyclictest4281-21/usr/sbin/munin12:25:170
2665499294,12cyclictest13817-21idleruntime-cro10:20:010
2667499283,14cyclictest21692-40dnf09:21:363
2665499281,12cyclictest30785-21awk09:45:010
2667099274,7cyclictest3912-40dnf12:23:252
2665499279,15cyclictest5976-21munin-run07:35:010
2665499271,13cyclictest0-21swapper/012:34:440
2667099268,4cyclictest0-21swapper/210:30:232
26662992617,2cyclictest0-21swapper/112:23:351
2665499269,14cyclictest12572-21awk07:50:010
2665499261,12cyclictest18211-21basename10:30:010
2665499261,12cyclictest18211-21basename10:30:000
2667499254,11cyclictest0-21swapper/312:23:243
2667099255,6cyclictest0-21swapper/209:35:132
2667099250,6cyclictest26989-21crond10:50:012
2666299251,16cyclictest0-21swapper/111:54:151
2666299251,13cyclictest2304-21unixbench-2d09:50:291
2665499251,4cyclictest0-21swapper/012:15:240
44652240,2sleep10-21swapper/107:30:151
2667499244,2cyclictest0-21swapper/311:11:493
2667499241,5cyclictest0-21swapper/309:43:463
2666299242,3cyclictest27624-21munin-run12:05:011
2666299241,17cyclictest30039-21kworker/u8:109:47:191
2666299241,12cyclictest0-21swapper/110:49:291
2665499248,7cyclictest29763-21/usr/sbin/munin07:15:150
2665499247,7cyclictest30045-21/usr/sbin/munin12:10:180
2665499247,2cyclictest0-21swapper/011:35:200
2665499247,2cyclictest0-21swapper/011:35:200
2665499243,8cyclictest26492-21cut09:35:010
2667499231,7cyclictest0-21swapper/310:25:233
2667499231,7cyclictest0-21swapper/310:25:233
2666299239,3cyclictest23-21ksoftirqd/112:15:241
2666299239,3cyclictest0-21swapper/109:22:511
2666299233,9cyclictest27973-21cpuspeed_turbos12:05:121
2666299231,5cyclictest31967-21users08:30:261
26662992314,3cyclictest0-21swapper/111:42:171
2666299231,17cyclictest0-21swapper/109:12:251
2666299231,12cyclictest13758-21diskstats09:05:161
2666299231,11cyclictest0-21swapper/110:44:241
2666299230,17cyclictest0-21swapper/112:27:281
2665499237,13cyclictest26996-21munin-run10:50:010
2665499236,14cyclictest23564-21idleruntime-cro08:15:010
2665499234,8cyclictest19410-21runrttasks09:15:300
2665499234,8cyclictest19410-21runrttasks09:15:300
2665499233,12cyclictest15377-21diskmemload12:20:260
2665499231,15cyclictest0-21swapper/009:54:390
254762230,4sleep19-21rcu_preempt09:30:221
2667499225,9cyclictest23015-21kworker/u8:212:32:573
2666299228,3cyclictest0-21swapper/107:16:311
2666299227,3cyclictest0-21swapper/110:20:011
2666299221,6cyclictest0-21swapper/109:19:241
2666299221,6cyclictest0-21swapper/109:19:241
2666299221,15cyclictest0-21swapper/111:24:171
2666299221,14cyclictest0-21swapper/111:25:271
2666299221,11cyclictest0-21swapper/110:32:361
2666299220,15cyclictest0-21swapper/111:35:181
2666299220,15cyclictest0-21swapper/111:35:181
2665499228,9cyclictest12463-21smart_sda09:00:240
2665499228,5cyclictest27655-40dnf08:21:230
2665499225,5cyclictest2411-21/usr/sbin/munin08:40:140
2665499224,6cyclictest4731-21ntp_states11:10:220
2665499223,16cyclictest29188-21timerandwakeup12:05:240
2665499221,12cyclictest14982-21/usr/sbin/munin07:55:240
276142210,1sleep10-21swapper/109:35:201
2667499216,3cyclictest87350irq/126-enp1s0-12:00:013
2667499216,3cyclictest23490-21hddtemp_smartct10:40:173
2667499214,9cyclictest5149-21latency12:25:193
2667499214,9cyclictest41-21ksoftirqd/310:45:153
2667499214,6cyclictest10270-21cpu11:25:123
2667499213,6cyclictest0-21swapper/309:13:383
2667499213,10cyclictest21663-21cpuspeed_turbos08:10:133
2667499211,9cyclictest0-21swapper/310:20:163
2667499211,3cyclictest0-21swapper/309:18:253
2667499211,3cyclictest0-21swapper/309:18:253
2667499211,14cyclictest0-21swapper/310:56:433
2667499211,11cyclictest1655-21irqstats09:50:213
26670992114,4cyclictest20531-21fschecks_time09:20:142
2666299219,3cyclictest417-21smart_sda11:00:241
26662992115,3cyclictest14061-21sshd11:30:341
2666299211,2cyclictest31099-21ntpq12:10:201
2666299211,19cyclictest0-21swapper/107:28:001
2666299211,11cyclictest27933-21cat08:24:561
2666299211,11cyclictest23117-21seq09:25:201
2666299211,11cyclictest0-21swapper/112:31:011
2665499217,3cyclictest15377-21diskmemload10:38:210
2665499211,6cyclictest0-21swapper/011:32:430
2665499211,6cyclictest0-21swapper/010:04:190
2665499211,12cyclictest12785-21/usr/sbin/munin07:50:260
2665499211,11cyclictest31960-21/usr/sbin/munin07:20:140
2665499210,6cyclictest87550irq/128-enp1s0-11:05:010
2667499209,2cyclictest0-21swapper/310:36:513
2667499208,3cyclictest15377-21diskmemload12:16:383
2667499207,1cyclictest0-21swapper/309:25:153
2667499206,5cyclictest0-21swapper/309:31:553
2667499203,9cyclictest17198-21entropy11:40:153
2667499203,10cyclictest39-21rcuc/310:15:163
2667499202,10cyclictest39-21rcuc/307:40:153
2667499202,10cyclictest25333-21kworker/u8:112:07:203
2667499201,16cyclictest28980-21munin-node09:07:413
2667499201,11cyclictest8056-21/usr/sbin/munin11:20:143
2667499201,11cyclictest5859-21/usr/sbin/munin11:15:193
2667499201,11cyclictest23777-21/usr/sbin/munin08:15:253
2667499201,10cyclictest5951-21basename10:00:173
2667499201,10cyclictest19570-21pmu-power10:30:233
2667499201,10cyclictest19426-21fschecks_count11:45:153
2667099204,4cyclictest755-21rngd09:58:212
26670992014,4cyclictest32-21ksoftirqd/209:05:232
26670992014,3cyclictest25891-21unixbench_multi09:30:262
2667099201,10cyclictest4341-21smartctl11:10:172
2666299208,3cyclictest0-21swapper/107:35:291
2666299205,6cyclictest0-21swapper/112:35:071
26662992017,2cyclictest0-21swapper/108:12:401
2666299201,18cyclictest0-21swapper/110:04:251
2666299201,11cyclictest0-21swapper/111:07:281
2666299201,11cyclictest0-21swapper/110:28:101
2666299201,11cyclictest0-21swapper/110:28:101
2666299201,11cyclictest0-21swapper/110:14:471
2666299201,10cyclictest579-21systemd-journal08:45:011
2666299201,10cyclictest21507-21sed10:35:181
2666299201,10cyclictest21344-21latency_hist08:10:011
2666299201,10cyclictest0-21swapper/110:59:131
26662992010,2cyclictest19102-21egrep11:45:121
2666299200,5cyclictest0-21swapper/110:20:031
2666299200,18cyclictest0-21swapper/109:55:341
2665499208,3cyclictest9458-21ntp_states07:40:210
2665499204,8cyclictest87550irq/128-enp1s0-11:05:220
2665499204,8cyclictest0-21swapper/008:15:170
2665499204,14cyclictest0-21swapper/007:35:150
2665499203,2cyclictest87550irq/128-enp1s0-09:20:120
2665499203,14cyclictest26868-21/usr/sbin/munin07:10:130
2665499202,7cyclictest0-21swapper/010:20:070
2665499202,10cyclictest25633-21/usr/sbin/munin12:00:180
2665499202,10cyclictest11233-21sed11:25:190
26654992011,5cyclictest2150-21diskstats07:25:150
26654992010,7cyclictest24696-21smart_sda11:55:250
2665499200,10cyclictest81ktimersoftd/009:10:220
2667499194,8cyclictest87350irq/126-enp1s0-10:05:143
2667499194,13cyclictest401ktimersoftd/308:45:173
2667499193,3cyclictest1-21systemd08:30:013
2667499192,3cyclictest26856-21expr09:35:133
2667499191,9cyclictest28839-21munin-run07:15:013
2667499191,5cyclictest0-21swapper/312:11:503
2667499191,10cyclictest3328-21runrttasks11:08:503
2667499191,10cyclictest15377-21diskmemload10:50:253
2667499191,10cyclictest0-21swapper/311:35:193
2667499191,10cyclictest0-21swapper/311:35:193
2667099196,3cyclictest87350irq/126-enp1s0-09:00:012
2667099195,4cyclictest755-21rngd08:31:392
2667099194,12cyclictest32-21ksoftirqd/208:55:012
2667099192,7cyclictest28797-21/usr/sbin/munin09:40:122
2667099192,3cyclictest87350irq/126-enp1s0-08:25:012
2667099191,8cyclictest13552-21pluginstate11:30:212
26670991915,2cyclictest32-21ksoftirqd/207:25:012
26670991914,3cyclictest24247-21grep08:15:152
26670991913,3cyclictest18923-21pmu-power09:15:222
26670991913,3cyclictest18923-21pmu-power09:15:222
2667099191,15cyclictest0-21swapper/211:37:132
2667099191,15cyclictest0-21swapper/211:37:132
2666299198,3cyclictest0-21swapper/110:07:391
26662991913,3cyclictest12270-21ntpq09:00:211
2666299191,17cyclictest0-21swapper/108:52:431
2666299191,16cyclictest0-21swapper/111:55:061
2666299191,16cyclictest0-21swapper/108:05:021
2666299191,11cyclictest23-21ksoftirqd/108:00:261
2666299191,10cyclictest9354-21diskstats08:55:151
2666299191,10cyclictest0-21swapper/111:10:161
2666299190,18cyclictest0-21swapper/108:45:221
2666299190,10cyclictest0-21swapper/110:54:001
2665499196,9cyclictest3829-21/usr/sbin/munin09:55:190
2665499196,4cyclictest0-21swapper/009:45:120
2665499194,3cyclictest21238-21/usr/sbin/munin11:50:280
2665499193,2cyclictest32577-21/usr/sbin/munin08:35:250
2665499193,2cyclictest0-21swapper/009:35:180
2665499191,8cyclictest0-21swapper/011:45:250
2665499191,4cyclictest0-21swapper/011:16:350
2665499191,12cyclictest0-21swapper/012:35:490
2665499191,11cyclictest0-21swapper/010:55:450
2665499191,11cyclictest0-21swapper/010:33:440
2665499191,10cyclictest30380-21/usr/sbin/munin08:30:220
193272190,4sleep011-21rcuc/008:05:010
3102181,3sleep20-21swapper/209:47:422
2667499189,2cyclictest13845-21ntpq07:50:213
2667499184,2cyclictest0-21swapper/311:03:023
2667499184,12cyclictest39-21rcuc/307:55:273
2667499184,11cyclictest41-21ksoftirqd/312:00:163
2667499184,11cyclictest41-21ksoftirqd/308:55:273
2667499183,9cyclictest28886-21kworker/3:008:04:373
2667499183,1cyclictest0-21swapper/312:37:123
2667499182,7cyclictest39-21rcuc/308:05:133
2667499182,7cyclictest39-21rcuc/308:05:133
2667499181,6cyclictest30993-21/usr/sbin/munin07:15:243
2667499181,6cyclictest0-21swapper/310:10:193
2667499180,7cyclictest0-21swapper/309:48:543
2667099188,2cyclictest30-21rcuc/211:53:442
2667099186,5cyclictest26788-21grep12:00:222
2667099186,3cyclictest6202-21df_inode11:15:152
2667099183,3cyclictest0-21swapper/212:36:342
2667099182,7cyclictest17179-21/usr/sbin/munin08:00:242
2667099181,7cyclictest12892-21ntp_offset10:15:222
2667099181,7cyclictest12218-21latency_hist11:30:012
2667099181,6cyclictest0-21swapper/211:21:402
2667099181,6cyclictest0-21swapper/209:13:392
2667099181,5cyclictest0-21swapper/212:32:332
2667099181,5cyclictest0-21swapper/210:00:212
2667099181,4cyclictest0-21swapper/210:29:082
2667099181,4cyclictest0-21swapper/210:29:082
2667099181,3cyclictest12817-21sshd09:00:352
2667099181,2cyclictest0-21swapper/212:10:162
26670991812,3cyclictest8159-21latency_hist07:40:012
26670991812,3cyclictest2333-21fschecks_time07:25:162
2666299184,2cyclictest32293-21expr07:20:141
26662991815,2cyclictest0-21swapper/108:17:331
26662991815,2cyclictest0-21swapper/107:54:051
26662991814,3cyclictest0-21swapper/108:25:341
26662991814,3cyclictest0-21swapper/107:56:231
2666299181,16cyclictest0-21swapper/108:37:571
2666299181,16cyclictest0-21swapper/107:48:201
2666299181,16cyclictest0-21swapper/107:40:511
2665499189,5cyclictest10697-21ntp_states10:10:210
2665499187,8cyclictest9537-21fschecks_time08:55:160
2665499187,3cyclictest6804-21/usr/sbin/munin08:50:230
2665499184,7cyclictest0-21swapper/011:22:330
2665499181,5cyclictest0-21swapper/009:25:090
2665499181,10cyclictest18613-21sshd11:42:340
2667499175,3cyclictest87350irq/126-enp1s0-09:55:173
2667499173,3cyclictest5750irq/9-acpi08:32:353
2667499172,5cyclictest6196-21/usr/sbin/munin07:35:193
2667499171,6cyclictest0-21swapper/311:51:403
26674991712,3cyclictest27655-40dnf08:21:233
26674991712,3cyclictest2484-21smartctl07:25:173
26674991711,3cyclictest11506-21cat09:00:133
2667499170,15cyclictest0-21swapper/308:52:363
2667099175,4cyclictest311ktimersoftd/207:59:392
2667099174,4cyclictest2256-21cat09:50:282
2667099171,7cyclictest0-21swapper/207:50:262
2667099171,6cyclictest28755-21fschecks_time08:25:162
2667099171,6cyclictest11954-21sed07:45:262
2667099171,6cyclictest0-21swapper/211:55:172
2667099171,5cyclictest0-21swapper/212:18:342
2667099171,3cyclictest0-21swapper/211:09:142
26670991713,3cyclictest30-21rcuc/210:20:172
26670991712,3cyclictest7438-21/usr/sbin/munin10:05:252
2667099171,15cyclictest0-21swapper/210:35:142
2667099171,15cyclictest0-21swapper/208:49:472
26670991711,3cyclictest18121-21switchtime11:40:242
26670991710,4cyclictest0-21swapper/208:35:162
2666299170,15cyclictest0-21swapper/107:10:071
2665499173,7cyclictest28178-21/usr/sbin/munin08:25:130
2665499171,10cyclictest0-21swapper/010:40:260
327172160,1sleep30-21swapper/307:20:183
2667499163,11cyclictest39-21rcuc/307:30:123
2667499162,12cyclictest41-21ksoftirqd/308:35:263
2667499162,11cyclictest10615-21cpu07:45:133
2667499161,6cyclictest2445-21cpu08:40:123
2667099164,2cyclictest87350irq/126-enp1s0-10:41:232
2667099163,10cyclictest28492-21sed12:05:152
2667099162,6cyclictest32-21ksoftirqd/207:15:052
2667099161,6cyclictest0-21swapper/210:50:242
2667099161,4cyclictest755-21rngd10:13:462
2667099161,2cyclictest0-21swapper/208:44:192
26670991612,3cyclictest0-21swapper/207:40:072
26670991612,3cyclictest0-21swapper/207:10:242
26670991610,3cyclictest32606-21needreboot11:00:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional