You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-15 - 14:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Sep 15, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
289562240162,25sleep00-21swapper/007:07:000
289572237162,24sleep10-21swapper/107:07:021
291252203166,25sleep20-21swapper/207:09:122
289492202164,25sleep30-21swapper/307:06:563
320521430,4sleep32928899cyclictest12:15:163
76232590,3sleep20-21swapper/208:45:152
175292510,2sleep10-21swapper/111:35:131
198742490,1sleep30-21swapper/310:25:203
2927599391,11cyclictest12903-21hddtemp_smartct10:10:151
2927599391,10cyclictest17354-21latency_hist07:55:021
2927599374,6cyclictest3642-40dnf11:02:401
29275993713,3cyclictest29476-21/usr/sbin/munin07:10:161
29275993712,22cyclictest9410-21/usr/sbin/munin08:50:251
2927599361,3cyclictest2217-21/usr/sbin/munin08:30:281
2927599352,7cyclictest18829-21/usr/sbin/munin10:25:261
2927599351,10cyclictest27067-21/usr/sbin/munin11:55:191
2928199341,21cyclictest0-21swapper/207:15:322
29275993417,14cyclictest11337-21diskstats07:40:151
29275993411,3cyclictest32362-21df_inode10:55:141
2927599341,11cyclictest27334-21df09:30:141
2927599341,11cyclictest13998-21irqstats07:45:161
29288993318,10cyclictest0-21swapper/310:56:453
2927599337,7cyclictest27047-21gltestperf08:15:161
2927599337,4cyclictest11686-21/usr/sbin/munin12:35:151
2927599331,12cyclictest29262-21cyclictest10:04:281
2927599331,10cyclictest29285-21/usr/sbin/munin12:00:201
2927599330,15cyclictest0-21swapper/107:18:541
2927599329,3cyclictest32254-21sort09:40:181
2927599329,3cyclictest31334-21grep10:50:251
2927599329,3cyclictest2484-21/usr/sbin/munin12:15:261
2927599329,3cyclictest23222-21/usr/sbin/munin10:35:211
2927599329,3cyclictest20410-21/usr/sbin/munin09:15:201
2927599329,21cyclictest2606-21kworker/u8:010:33:101
2927599321,7cyclictest0-21swapper/109:14:301
2927599321,6cyclictest0-21swapper/110:07:141
2927599321,4cyclictest579-21systemd-journal11:20:001
29275993211,3cyclictest29853-21smart_sda08:20:261
29275993210,3cyclictest6520-21grep08:40:261
29275993210,3cyclictest11609-21/usr/sbin/munin08:55:211
2927599320,14cyclictest0-21swapper/107:57:101
2927599320,14cyclictest0-21swapper/107:21:541
2926499328,11cyclictest32431-21awk12:10:010
2927599318,3cyclictest18108-21grep10:20:261
2927599315,3cyclictest87550irq/128-enp1s0-11:50:221
2927599311,11cyclictest29599-21perf10:50:001
2927599310,14cyclictest0-21swapper/107:32:091
2928899301,21cyclictest0-21swapper/310:36:483
2928899301,12cyclictest0-21swapper/310:09:483
2927599309,3cyclictest827-21runrttasks12:08:251
2927599309,3cyclictest827-21runrttasks09:00:581
2927599301,3cyclictest23955-21latency_hist08:10:011
2927599301,3cyclictest14589-21tr10:15:121
29275993011,3cyclictest19792-21diskstats11:40:141
2927599301,12cyclictest7855-21users11:10:281
2927599301,11cyclictest0-21swapper/109:35:211
2927599301,10cyclictest24760-21ls09:25:011
29275993010,3cyclictest827-21runrttasks09:06:441
29275993010,3cyclictest5735-21ntp_offset12:20:211
29275993010,3cyclictest5735-21ntp_offset12:20:211
29275993010,3cyclictest26317-21tail09:25:251
2927599300,14cyclictest0-21swapper/108:01:311
2926499302,13cyclictest14829-21munin-run11:30:020
29288992913,3cyclictest27845-21expr10:45:133
2928899291,23cyclictest0-21swapper/310:12:043
2928899291,13cyclictest0-21swapper/307:40:243
2927599298,3cyclictest7547-21/usr/sbin/munin12:25:151
2927599298,3cyclictest6077-21date11:10:011
2927599295,3cyclictest21643-21/usr/sbin/munin11:45:251
2927599291,6cyclictest30785-21/usr/sbin/munin08:25:231
2927599291,4cyclictest32649-21/usr/sbin/munin12:10:201
2927599291,3cyclictest12608-21cat11:24:561
2927599291,11cyclictest11036-21latency_hist12:35:021
29275992910,3cyclictest827-21runrttasks11:32:441
29275992910,3cyclictest25420-21/usr/sbin/munin10:40:151
29275992910,3cyclictest13383-21grep11:25:141
2926499295,12cyclictest22397-21perf09:20:020
2928899287,3cyclictest0-21swapper/309:44:243
29288992815,3cyclictest0-21swapper/307:33:323
2928899281,14cyclictest0-21swapper/312:00:173
2927599283,3cyclictest87550irq/128-enp1s0-09:50:141
2927599281,11cyclictest4583-21cstates07:25:131
2927599281,10cyclictest0-21swapper/108:35:181
29275992810,3cyclictest10289-21unixbench_multi07:35:271
29275992810,3cyclictest10289-21unixbench_multi07:35:271
29275992810,2cyclictest0-21swapper/108:10:201
2928899277,3cyclictest0-21swapper/311:45:453
29288992716,3cyclictest0-21swapper/308:10:293
29288992714,8cyclictest0-21swapper/308:50:333
2928899271,12cyclictest17982-21diskmemload11:06:213
2928899271,12cyclictest0-21swapper/311:10:333
2928899271,12cyclictest0-21swapper/307:15:063
2927599271,12cyclictest9315-21cat08:50:011
2926499273,12cyclictest15175-21awk07:50:010
2928899268,3cyclictest0-21swapper/309:24:213
29288992612,8cyclictest0-21swapper/312:34:403
2928899261,10cyclictest0-21swapper/311:35:493
2928899261,10cyclictest0-21swapper/310:16:523
29275992612,2cyclictest0-21swapper/109:56:461
29275992612,2cyclictest0-21swapper/109:56:461
2927599260,14cyclictest0-21swapper/109:48:171
2926499264,10cyclictest1644-21basename11:00:000
260812260,3sleep29-21rcu_preempt09:25:232
2928899259,3cyclictest0-21swapper/309:54:003
2928899258,3cyclictest0-21swapper/308:59:053
2928899256,3cyclictest0-21swapper/309:17:133
2928899251,12cyclictest5356-21df_inode08:40:153
2928899251,11cyclictest0-21swapper/311:04:493
2928899251,11cyclictest0-21swapper/310:34:483
2928899251,10cyclictest16644-21cpu10:20:123
2928899251,10cyclictest0-21swapper/307:39:523
2928899251,10cyclictest0-21swapper/307:39:523
2928899250,9cyclictest0-21swapper/309:35:443
2928899250,10cyclictest0-21swapper/308:45:413
2928899249,4cyclictest808-21snmpd07:21:003
2928899246,3cyclictest0-21swapper/312:22:083
2928899246,3cyclictest0-21swapper/312:22:083
2928899246,3cyclictest0-21swapper/311:54:533
2928899246,3cyclictest0-21swapper/311:40:533
2928899246,3cyclictest0-21swapper/308:27:573
29288992415,3cyclictest0-21swapper/311:25:013
29288992415,3cyclictest0-21swapper/308:01:013
2928899241,14cyclictest0-21swapper/309:56:403
2928899241,14cyclictest0-21swapper/309:56:403
2928899241,12cyclictest0-21swapper/312:10:253
2926499241,3cyclictest0-21swapper/012:02:340
189532240,4sleep30-21swapper/307:55:273
2928899239,3cyclictest0-21swapper/308:21:493
2928899238,3cyclictest17982-21diskmemload12:08:453
2928899238,3cyclictest0-21swapper/309:08:093
2928899238,3cyclictest0-21swapper/307:11:213
2928899237,3cyclictest0-21swapper/309:01:333
2928899231,9cyclictest0-21swapper/311:55:453
2928899231,9cyclictest0-21swapper/311:29:053
2928899231,11cyclictest30505-21/usr/sbin/munin10:50:173
2928899231,10cyclictest0-21swapper/311:33:213
2928899230,9cyclictest0-21swapper/309:48:363
2928899229,3cyclictest0-21swapper/312:27:363
2928899227,3cyclictest0-21swapper/312:39:443
2928899227,3cyclictest0-21swapper/307:51:203
2928899226,3cyclictest0-21swapper/309:25:403
2928899226,3cyclictest0-21swapper/307:27:123
2928899224,3cyclictest0-21swapper/310:44:493
29281992215,4cyclictest3793-21grep12:15:242
2926499221,12cyclictest766-21ntpd10:10:010
2926499221,12cyclictest23239-21chrt10:35:120
101482221,3sleep210196-21pmu-power12:30:262
2928899218,3cyclictest0-21swapper/310:04:203
2928899217,3cyclictest0-21swapper/308:05:453
2928899216,3cyclictest0-21swapper/308:38:493
2928899216,3cyclictest0-21swapper/308:33:413
2928899214,3cyclictest0-21swapper/309:34:123
2928899211,14cyclictest0-21swapper/308:16:373
2928899211,10cyclictest0-21swapper/311:15:133
2928199211,6cyclictest0-21swapper/211:29:002
2928199211,13cyclictest0-21swapper/210:15:242
2926499218,4cyclictest0-21swapper/009:30:190
2928899201,4cyclictest0-21swapper/309:11:413
2928899201,10cyclictest0-21swapper/307:45:163
2926499209,3cyclictest11-21rcuc/011:20:230
2926499207,2cyclictest7869-21needreboot12:25:190
2926499205,9cyclictest0-21swapper/009:35:120
2926499205,3cyclictest21624-21gltestperf10:30:150
2926499204,8cyclictest87350irq/126-enp1s0-10:45:180
2926499204,13cyclictest7-21ksoftirqd/009:10:170
2926499202,10cyclictest31397-21/usr/sbin/munin09:40:190
2926499201,8cyclictest8795-21/usr/sbin/munin07:35:150
2926499201,8cyclictest8795-21/usr/sbin/munin07:35:150
2926499201,6cyclictest0-21swapper/011:34:170
2926499201,5cyclictest0-21swapper/009:49:520
2926499201,10cyclictest5578-21sed11:05:260
2928199193,5cyclictest25147-21sshd10:39:392
2928199191,6cyclictest0-21swapper/211:35:292
29281991910,2cyclictest87550irq/128-enp1s0-10:53:502
2926499199,2cyclictest10170-21if_enp2s008:50:180
2926499193,8cyclictest0-21swapper/012:14:020
2926499192,15cyclictest9120-21sed11:15:170
2926499191,5cyclictest827-21runrttasks10:18:410
29264991914,3cyclictest17097-21grep10:20:150
2926499191,11cyclictest4680-21/usr/sbin/munin12:20:180
2926499191,11cyclictest4680-21/usr/sbin/munin12:20:180
2926499191,10cyclictest8265-21ntpq08:45:200
2926499191,10cyclictest808-21snmpd12:37:510
235112191,3sleep00-21swapper/011:48:540
192042191,2sleep00-21swapper/007:55:410
172562192,2sleep017528-21cstates11:35:140
56312181,2sleep00-21swapper/009:55:130
56312181,2sleep00-21swapper/009:55:130
2928199188,2cyclictest4155-21cat09:50:162
2928199187,3cyclictest87350irq/126-enp1s0-08:05:152
2928199185,10cyclictest8854-21perf12:30:002
2928199183,9cyclictest0-21swapper/209:21:472
2928199182,6cyclictest21670-21smart_sda09:15:242
2928199182,5cyclictest8564-1kworker/u9:109:12:212
2928199182,15cyclictest0-21swapper/211:07:252
2928199182,14cyclictest0-21swapper/207:37:072
2928199182,14cyclictest0-21swapper/207:37:072
2928199181,7cyclictest0-21swapper/210:44:082
2928199181,5cyclictest0-21swapper/211:47:322
2928199181,4cyclictest0-21swapper/211:54:452
29281991814,3cyclictest0-21swapper/211:55:102
29281991813,3cyclictest8898-21ntp_states10:00:232
29281991813,3cyclictest13223-21grep10:10:182
2928199181,11cyclictest0-21swapper/212:10:162
2928199181,10cyclictest0-21swapper/210:22:432
2928199180,1cyclictest0-21swapper/210:06:052
2926499187,3cyclictest808-21snmpd10:27:280
2926499185,2cyclictest26912-21fschecks_time08:15:150
2926499182,7cyclictest0-21swapper/011:56:210
2926499182,13cyclictest11-21rcuc/007:25:180
2926499181,8cyclictest0-21swapper/011:03:420
2926499181,6cyclictest0-21swapper/012:15:380
2926499181,6cyclictest0-21swapper/009:21:010
2926499181,5cyclictest0-21swapper/012:34:210
2926499181,3cyclictest20803-21timerandwakeup11:40:250
29264991812,3cyclictest23974-21awk08:10:010
2926499181,11cyclictest0-21swapper/008:55:280
2926499181,10cyclictest808-21snmpd10:02:240
2926499181,10cyclictest0-21swapper/009:25:260
14772181,2sleep20-21swapper/210:58:322
63082171,2sleep20-21swapper/212:20:462
63082171,2sleep20-21swapper/212:20:462
314742170,3sleep00-21swapper/010:51:060
307822171,2sleep00-21swapper/008:25:100
2928199178,3cyclictest0-21swapper/208:11:492
2928199177,3cyclictest5750irq/9-acpi09:30:572
2928199174,4cyclictest6585-21missed_timers09:55:222
2928199174,4cyclictest6585-21missed_timers09:55:212
2928199172,7cyclictest27616-21/usr/sbin/munin10:45:172
2928199172,10cyclictest0-21swapper/207:56:512
2928199171,9cyclictest0-21swapper/211:40:242
2928199171,7cyclictest0-21swapper/208:36:142
2928199171,6cyclictest32256-21grep08:25:262
2928199171,6cyclictest31068-21vmstat07:10:262
2928199171,6cyclictest0-21swapper/211:17:322
2928199171,6cyclictest0-21swapper/211:03:482
2928199171,6cyclictest0-21swapper/209:35:142
2928199171,3cyclictest0-21swapper/210:28:532
2928199171,3cyclictest0-21swapper/209:41:462
29281991713,3cyclictest0-21swapper/208:51:112
29281991712,3cyclictest6614-21cpu07:30:112
2928199171,14cyclictest0-21swapper/208:40:072
2926499173,12cyclictest11-21rcuc/009:00:270
2926499172,3cyclictest21768-21perf08:05:010
2926499171,8cyclictest17982-21diskmemload10:44:310
2926499171,6cyclictest17365-21perf07:55:010
2926499171,6cyclictest1612-21grep08:30:210
2926499171,6cyclictest13090-21sort10:10:160
29264991712,3cyclictest24185-21/usr/sbin/munin08:10:190
2926499171,15cyclictest0-21swapper/011:52:400
2926499171,14cyclictest0-21swapper/008:37:080
2926499171,13cyclictest11985-21needreboot07:40:190
2928199169,4cyclictest15898-21/usr/sbin/munin11:30:202
2928199165,3cyclictest87350irq/126-enp1s0-11:10:162
2928199163,10cyclictest2187-21grep09:45:172
2928199161,7cyclictest0-21swapper/208:02:442
2928199161,6cyclictest2589-21latency_hist08:35:022
2928199161,6cyclictest0-21swapper/212:05:162
2928199161,6cyclictest0-21swapper/211:20:402
2928199161,5cyclictest0-21swapper/212:00:122
2928199161,3cyclictest0-21swapper/210:31:212
29281991612,3cyclictest0-21swapper/208:57:432
29281991612,3cyclictest0-21swapper/208:18:402
29281991611,3cyclictest16539-21fschecks_time09:05:142
29281991611,3cyclictest11410-21cpuspeed_turbos12:35:132
29281991611,2cyclictest14573-21if_enp2s009:00:182
29281991610,3cyclictest827-21runrttasks08:20:572
29281991610,3cyclictest13274-21cpuspeed_turbos07:45:102
2926499166,3cyclictest87550irq/128-enp1s0-07:20:260
2926499165,3cyclictest87350irq/126-enp1s0-08:40:140
29264991612,3cyclictest0-21swapper/007:10:280
2926499161,13cyclictest7588-21needreboot07:30:180
29264991611,3cyclictest6265-21/usr/sbin/munin11:10:180
29264991611,3cyclictest4944-21grep09:50:260
29264991610,3cyclictest30762-21date08:25:010
16932161,2sleep00-21swapper/007:16:330
2928199151,6cyclictest0-21swapper/207:29:032
29281991510,3cyclictest2205-21/usr/sbin/munin07:20:162
29281991510,3cyclictest15395-21acpi07:50:122
29281991510,3cyclictest11406-21fschecks_count07:40:162
2926499151,5cyclictest16014-21/usr/sbin/munin09:05:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional