You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-04 - 01:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jun 03, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
123012237162,24sleep30-21swapper/307:05:343
106762235163,24sleep10-21swapper/107:05:031
124782230161,57sleep00-21swapper/007:07:510
126432203166,24sleep20-21swapper/207:09:592
635821620,2sleep11273399cyclictest08:05:191
2178821410,4sleep01272699cyclictest09:55:140
1273899107103,3cyclictest0-21swapper/211:09:562
184302660,4sleep20-21swapper/208:35:012
134722550,4sleep011-21rcuc/010:50:120
319882500,3sleep10-21swapper/107:50:181
32502490,1sleep30-21swapper/308:00:113
320932450,1sleep132096-21sendmail_mailqu11:30:221
186162430,2sleep19-21rcu_preempt12:15:121
5282410,1sleep30-21swapper/311:35:013
1273899371,32cyclictest24932-21gltestperf11:15:162
1272699377,14cyclictest19666-21awk11:05:000
1274499349,3cyclictest20490-21/usr/sbin/munin12:20:173
1274499344,9cyclictest10558-40dnf10:41:313
1274499344,3cyclictest27267-21cstates12:35:123
1274499341,30cyclictest18482-21irqstats11:00:183
1273899348,4cyclictest8554-21missed_timers08:10:192
12738993411,20cyclictest29154-21tr07:45:122
1272699348,12cyclictest0-21swapper/009:35:010
1274499332,28cyclictest12454-21sort08:20:163
1274499331,5cyclictest0-21swapper/311:54:483
1273899336,23cyclictest24047-21munin-plugin-st11:15:012
12738993312,18cyclictest23023-21/usr/sbin/munin08:45:232
12738993311,19cyclictest28079-21sshd11:20:322
12738993311,19cyclictest16164-21df_inode07:15:132
1274499328,4cyclictest19282-21smart_sda07:20:253
1274499328,3cyclictest827-21runrttasks09:58:033
12744993212,17cyclictest32258-21ntp_offset07:50:223
1274499321,12cyclictest6673-21/usr/sbin/munin10:35:203
12744993210,3cyclictest24201-21unixbench_multi12:25:273
12744993210,3cyclictest11414-21df_inode10:45:133
1274499320,3cyclictest7194-21taskset08:07:223
1273899326,23cyclictest0-21swapper/208:00:242
1273899323,4cyclictest10739-21awk11:55:232
12738993211,17cyclictest13292-21tail09:35:152
1273899320,5cyclictest0-21swapper/209:25:152
1274499318,4cyclictest0-21swapper/311:45:133
1274499318,3cyclictest0-21swapper/307:43:403
1274499312,8cyclictest24112-21sort11:15:013
1274499311,5cyclictest29003-21/usr/sbin/munin07:45:143
12744993113,15cyclictest15659-21irqstats09:40:173
12744993112,16cyclictest8695-21ntp_offset08:10:213
12744993112,16cyclictest3135-21sed09:10:253
12744993112,16cyclictest26467-21/usr/sbin/munin11:20:243
12744993111,17cyclictest28668-21/usr/sbin/munin11:25:143
12744993111,17cyclictest28668-21/usr/sbin/munin11:25:143
1273899319,19cyclictest15341-21pluginstate08:25:232
1273899318,19cyclictest25830-21gltestperf08:50:172
1273899317,3cyclictest12632-21/usr/sbin/munin12:00:182
1273899313,25cyclictest18718-21grep12:15:152
1273899311,7cyclictest0-21swapper/210:40:112
1273899311,12cyclictest21380-21proc_pri07:25:242
12738993111,17cyclictest827-21runrttasks09:50:232
12738993110,18cyclictest2283-21/usr/sbin/munin10:25:252
12738993110,18cyclictest17766-21grep11:00:132
253082300,1sleep10-21swapper/108:50:141
1274499309,3cyclictest827-21runrttasks09:46:183
1274499309,3cyclictest24674-21ntp_kernel_err10:00:223
1274499309,3cyclictest13432-21cut07:10:163
1274499308,3cyclictest827-21runrttasks12:32:323
1274499308,3cyclictest827-21runrttasks12:32:323
1274499307,5cyclictest0-21swapper/312:15:243
1274499305,22cyclictest0-21swapper/312:10:133
1274499304,4cyclictest3267-21/usr/sbin/munin11:40:143
1274499301,6cyclictest0-21swapper/311:37:333
1274499301,19cyclictest31308-21ntp_states10:15:213
12744993010,3cyclictest3863-21/usr/sbin/munin09:15:273
1273899309,18cyclictest827-21runrttasks12:26:192
1273899309,18cyclictest827-21runrttasks12:13:432
1273899308,19cyclictest23405-21crond10:00:002
1273899307,3cyclictest4476-21/usr/sbin/munin10:30:222
1273899306,2cyclictest0-21swapper/209:45:122
1273899306,21cyclictest0-21swapper/210:05:122
1273899305,22cyclictest0-21swapper/209:30:122
1273899302,4cyclictest2135-21/usr/sbin/munin07:55:232
1273899302,4cyclictest2135-21/usr/sbin/munin07:55:232
1273899302,25cyclictest8029-21awk09:25:022
1273899301,5cyclictest25237-21diskstats12:30:142
1273899301,5cyclictest25237-21diskstats12:30:142
12738993011,16cyclictest827-21runrttasks08:19:142
12738993010,17cyclictest9061-21sshd11:52:312
1272699301,14cyclictest0-21swapper/007:35:000
273122290,6sleep3401ktimersoftd/308:55:013
1274499299,4cyclictest27419-21/usr/sbin/munin08:55:193
1274499299,3cyclictest28154-21grep10:10:123
1274499299,3cyclictest28154-21grep10:10:123
1274499298,4cyclictest19626-21uname08:35:203
1274499298,4cyclictest0-21swapper/309:25:173
1274499298,2cyclictest827-21runrttasks10:06:213
1274499296,3cyclictest15886-21cpuspeed_turbos07:15:123
1274499291,5cyclictest26247-21munin-run11:20:013
1274499291,3cyclictest15472-21/usr/sbin/munin10:55:163
12744992911,15cyclictest14236-21/usr/sbin/munin08:25:123
12744992910,4cyclictest15124-21chrt10:53:363
12744992910,3cyclictest22263-21sort08:40:253
12744992910,15cyclictest763-21dbus-daemon09:50:213
1273899299,17cyclictest6472-21ntp_kernel_err08:05:202
1273899299,17cyclictest23328-21missed_timers07:30:192
1273899298,3cyclictest17481-21cut11:00:012
1273899297,3cyclictest0-21swapper/208:42:192
1273899297,19cyclictest28229-21cstates10:10:132
1273899297,19cyclictest28229-21cstates10:10:132
1273899295,5cyclictest0-21swapper/210:35:142
1273899294,2cyclictest0-21swapper/212:35:162
1273899292,5cyclictest15164-21smart_sda12:05:232
1273899292,4cyclictest3248-21diskstats11:40:132
1273899292,4cyclictest12955-21memory08:20:202
1273899291,8cyclictest15510-21tail09:40:152
1273899291,5cyclictest31194-21df11:30:122
1273899291,5cyclictest29433-21if_enp2s011:25:192
1273899291,5cyclictest29433-21if_enp2s011:25:182
1273899291,5cyclictest0-21swapper/210:15:122
1273899291,4cyclictest19293-21tail08:35:172
1273899291,4cyclictest18487-21/usr/sbin/munin07:20:152
12738992911,15cyclictest13281-21dnf07:10:152
1273899290,15cyclictest0-21swapper/207:38:192
1274499289,3cyclictest827-21runrttasks07:28:333
1274499289,3cyclictest6052-21/usr/sbin/munin09:20:123
1274499289,2cyclictest827-21runrttasks09:01:073
1274499288,3cyclictest12644-21/usr/sbin/munin09:35:163
1274499287,5cyclictest32767-21missed_timers09:05:193
1274499287,4cyclictest10388-21tune2fs08:15:153
1274499286,3cyclictest0-21swapper/309:33:543
1274499285,5cyclictest1564-21fschecks_time07:55:163
1274499285,5cyclictest1564-21fschecks_time07:55:163
1274499285,3cyclictest0-21swapper/312:06:313
1274499283,6cyclictest1323-21smart_sda10:20:233
12744992811,3cyclictest3127-21latency10:25:183
1273899289,16cyclictest7138-21latency_hist11:50:012
1273899289,16cyclictest302-21unixbench_singl07:50:272
1273899289,15cyclictest11262-21expr10:45:122
1273899287,2cyclictest28562-21grep08:55:222
1273899284,2cyclictest0-21swapper/212:20:242
1273899282,4cyclictest2577-21memory09:10:182
1273899282,4cyclictest24450-21irqstats10:00:182
1273899281,5cyclictest30298-21/usr/sbin/munin09:00:162
1273899281,4cyclictest32158-21df09:05:132
1273899281,3cyclictest0-21swapper/207:40:222
1274499279,4cyclictest22551-21cpuspeed_turbos07:30:133
1274499277,3cyclictest10278-21if_err_enp2s011:55:183
1274499276,3cyclictest12276-21tr12:00:163
1274499273,5cyclictest17695-21smart_sda08:30:243
1273899272,4cyclictest1706-21/usr/sbin/munin11:35:182
1273899272,4cyclictest14189-21memory10:50:182
1273899271,4cyclictest4931-21ntp_states09:15:252
1273399271,9cyclictest0-21swapper/108:45:281
12733992716,7cyclictest28145-21grep07:40:251
1272699271,12cyclictest15876-21munin-run12:10:000
1274499268,4cyclictest20430-21tune2fs11:05:153
1274499266,3cyclictest23752-21if_enp1s008:45:173
12744992611,12cyclictest4842-21fschecks_count10:30:143
12744992610,3cyclictest26583-21perf07:40:003
1273899260,4cyclictest32547-21tr10:20:122
1273399263,14cyclictest23-21ksoftirqd/110:18:021
1273399261,9cyclictest0-21swapper/109:14:221
1272699264,10cyclictest30988-21perf07:50:010
1272699260,12cyclictest7425-21perf08:10:020
1273399251,3cyclictest0-21swapper/111:14:571
1272699259,14cyclictest30651-21perf11:30:010
1272699259,14cyclictest30651-21perf11:30:010
1272699251,6cyclictest0-21swapper/012:26:110
1272699251,5cyclictest0-21swapper/009:15:220
12726992512,3cyclictest1415-21diskmemload10:20:120
1273399244,12cyclictest16224-21kworker/u8:010:07:301
12733992418,3cyclictest28001-21memory12:35:181
1273399241,13cyclictest0-21swapper/112:26:461
1272699241,6cyclictest0-21swapper/009:11:380
1272699241,10cyclictest14078-21pmu-power07:10:230
1273399238,4cyclictest0-21swapper/110:58:571
1273399232,13cyclictest0-21swapper/110:31:541
1273399232,12cyclictest2059-21munin-run10:25:011
12733992317,3cyclictest21195-21basename12:20:161
1273399231,5cyclictest0-21swapper/112:14:341
12733992315,2cyclictest0-21swapper/111:15:551
1273399231,17cyclictest0-21swapper/111:07:191
1273399231,16cyclictest0-21swapper/110:54:011
1273399231,12cyclictest0-21swapper/111:59:091
1273399231,12cyclictest0-21swapper/109:28:461
1273399231,11cyclictest0-21swapper/111:54:351
1273399229,8cyclictest0-21swapper/112:09:441
1273399229,8cyclictest0-21swapper/111:03:061
1273399227,4cyclictest0-21swapper/111:28:061
1273399227,4cyclictest0-21swapper/111:28:061
1273399221,6cyclictest0-21swapper/109:24:411
1273399221,19cyclictest0-21swapper/110:00:441
1273399221,16cyclictest0-21swapper/108:58:321
1273399221,16cyclictest0-21swapper/108:27:221
1273399221,11cyclictest0-21swapper/110:43:051
12733992210,3cyclictest0-21swapper/111:47:241
1273399220,15cyclictest0-21swapper/112:00:341
1272699226,4cyclictest0-21swapper/010:35:030
1272699223,8cyclictest27876-40dnf07:40:240
1272699221,11cyclictest9513-21smart_sda09:25:250
1272699221,10cyclictest8285-21grep11:50:190
1272699221,10cyclictest0-21swapper/010:15:240
153782211,16sleep115578-21hddtemp_smartct07:15:031
1273399218,4cyclictest0-21swapper/110:11:131
1273399218,4cyclictest0-21swapper/110:11:131
1273399215,4cyclictest0-21swapper/111:23:411
1273399211,3cyclictest18813-21cstates08:35:141
12733992113,3cyclictest0-21swapper/108:19:281
12733992113,2cyclictest0-21swapper/107:55:261
12733992113,2cyclictest0-21swapper/107:55:261
1273399211,17cyclictest0-21swapper/108:00:161
1273399211,15cyclictest0-21swapper/110:26:101
1273399211,15cyclictest0-21swapper/109:55:071
1273399211,14cyclictest0-21swapper/111:37:251
1273399211,14cyclictest0-21swapper/107:26:451
1273399211,11cyclictest4741-21cat09:15:201
1273399211,11cyclictest10612-21cpuspeed_turbos09:30:121
1273399210,6cyclictest0-21swapper/109:40:341
1273399210,5cyclictest0-21swapper/109:54:521
1273399210,16cyclictest0-21swapper/112:34:501
1273399210,16cyclictest0-21swapper/112:34:501
1273399210,15cyclictest0-21swapper/109:00:361
1272699218,3cyclictest1612-21memory11:35:170
1272699214,6cyclictest21582-21if_enp2s008:40:170
1272699211,7cyclictest0-21swapper/009:22:190
1272699211,5cyclictest0-21swapper/010:03:100
1272699211,11cyclictest6244-21grep11:45:220
256112200,4sleep081ktimersoftd/011:15:240
1273399207,2cyclictest17051-21/usr/sbin/munin09:45:241
12733992014,2cyclictest4055-21kworker/u8:110:45:111
12733992013,2cyclictest0-21swapper/109:07:091
1273399201,17cyclictest0-21swapper/110:35:191
1273399201,16cyclictest0-21swapper/108:21:341
1273399201,14cyclictest0-21swapper/111:41:311
1273399201,14cyclictest0-21swapper/107:38:111
1273399201,10cyclictest21999-21ps08:40:221
1273399200,14cyclictest0-21swapper/108:30:261
1272699209,3cyclictest4886-21awk11:45:010
1272699204,14cyclictest5230-21awk08:05:020
1272699201,6cyclictest21815-21awk07:25:270
1272699201,5cyclictest0-21swapper/011:06:120
1272699201,10cyclictest17606-21grep08:30:240
1273399197,4cyclictest7864-21expr08:10:131
1273399194,9cyclictest16224-21kworker/u8:009:35:401
1273399191,3cyclictest0-21swapper/107:49:051
1273399190,14cyclictest0-21swapper/107:21:091
1272699198,3cyclictest26659-21irqstats10:05:180
1272699198,3cyclictest16571-21if_enp2s007:15:170
1272699191,9cyclictest0-21swapper/012:10:420
1272699191,8cyclictest13690-21awk12:05:010
1272699191,7cyclictest0-21swapper/011:55:120
1272699191,4cyclictest0-21swapper/012:18:350
1272699191,3cyclictest3758-21grep10:25:270
1272699191,3cyclictest24531-21tail08:45:260
1272699191,10cyclictest24889-21/usr/sbin/munin12:30:180
1272699191,10cyclictest24889-21/usr/sbin/munin12:30:180
1272699191,10cyclictest15380-21grep08:25:240
1272699191,10cyclictest0-21swapper/010:40:140
79372180,15sleep00-21swapper/010:35:250
1273399181,9cyclictest13608-21hddtemp_smartct07:10:171
12733991812,2cyclictest0-21swapper/107:31:151
1272699188,3cyclictest808-21snmpd08:16:400
1272699188,2cyclictest0-21swapper/008:50:170
1272699181,8cyclictest22454-21cut11:10:150
1272699181,8cyclictest0-21swapper/011:21:300
1272699181,6cyclictest0-21swapper/011:32:380
1272699181,6cyclictest0-21swapper/010:14:210
1272699181,6cyclictest0-21swapper/010:14:210
1272699181,6cyclictest0-21swapper/009:36:120
1272699181,5cyclictest24655-1kworker/u9:009:48:260
1272699181,15cyclictest1415-21diskmemload09:55:000
1272699175,3cyclictest28895-21sed08:55:250
1272699171,9cyclictest0-21swapper/007:20:170
1272699171,8cyclictest839-21lldpd08:13:100
1272699171,7cyclictest32761-21sed07:50:270
1272699171,7cyclictest0-21swapper/012:38:570
1272699171,7cyclictest0-21swapper/012:20:500
1272699171,6cyclictest15762-21memory09:40:180
12726991712,3cyclictest899-21grep09:05:250
12726991711,3cyclictest29704-21ls09:00:120
12726991710,4cyclictest16575-21/usr/sbin/munin10:55:210
1272699165,3cyclictest87550irq/128-enp1s0-07:36:310
1272699161,3cyclictest11070-21/usr/sbin/munin10:45:200
12726991612,3cyclictest0-21swapper/008:24:030
12726991611,3cyclictest18632-21/usr/sbin/munin08:35:140
12726991510,3cyclictest1564-21fschecks_time07:55:160
12726991510,3cyclictest1564-21fschecks_time07:55:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional