You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-24 - 08:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Jul 24, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
58812237163,25sleep20-21swapper/219:07:402
57562237163,24sleep30-21swapper/319:06:043
57282236163,25sleep00-21swapper/019:05:400
58802234162,24sleep10-21swapper/119:07:391
315732790,4sleep20-21swapper/220:05:152
255602680,2sleep00-21swapper/019:50:190
6154996460,3cyclictest0-21swapper/220:48:012
183352570,1sleep20-21swapper/219:35:122
327302470,1sleep20-21swapper/222:35:122
275962440,2sleep30-21swapper/323:35:203
614499329,11cyclictest20313-21perf00:35:010
616099301,13cyclictest9478-21cstates19:15:143
616099301,12cyclictest0-21swapper/320:45:543
24532290,4sleep339-21rcuc/321:25:183
615499281,14cyclictest0-21swapper/221:15:172
615099282,4cyclictest9313-21munin-run00:10:011
615099282,4cyclictest9313-21munin-run00:10:001
614499280,13cyclictest22504-21idleruntime-cro00:40:010
99772270,4sleep121-21rcuc/119:15:171
616099270,12cyclictest0-21swapper/319:58:553
614499272,12cyclictest7106-21munin-run00:05:000
6154992610,3cyclictest19058-21awk22:05:022
56962260,3sleep24486-21/usr/sbin/munin22:45:242
6150992511,13cyclictest1440-21kworker/u8:121:02:481
615099250,17cyclictest0-21swapper/119:35:171
614499251,12cyclictest6464-21munin-run22:50:000
197652250,4sleep121-21rcuc/123:20:021
616099241,9cyclictest12802-21ntp_states00:15:243
616099241,13cyclictest0-21swapper/323:45:253
616099241,13cyclictest0-21swapper/321:40:013
616099241,10cyclictest0-21swapper/321:57:533
615099244,7cyclictest0-21swapper/119:28:251
615099244,7cyclictest0-21swapper/119:28:251
615099241,3cyclictest13763-21wc00:20:011
615099241,20cyclictest12448-21munin-run21:50:011
615099241,20cyclictest12448-21munin-run21:50:011
616099239,3cyclictest0-21swapper/321:00:123
616099231,14cyclictest0-21swapper/321:08:283
616099230,14cyclictest0-21swapper/322:45:143
615499233,9cyclictest31560-21sed23:45:162
615499231,10cyclictest11368-21cat19:20:012
615099232,17cyclictest28260-21irqstats21:10:191
615099231,19cyclictest327562sleep123:49:041
615099231,16cyclictest0-21swapper/121:15:371
615099231,12cyclictest10480-21missed_timers00:10:201
615099231,12cyclictest0-21swapper/119:50:181
616099228,4cyclictest29641-21/usr/sbin/munin21:15:133
616099227,10cyclictest0-21swapper/322:35:063
616099222,11cyclictest24076-21fschecks_count22:15:153
615499227,3cyclictest0-21swapper/220:35:322
615499227,3cyclictest0-21swapper/220:35:322
615499223,9cyclictest32-21ksoftirqd/223:35:012
615499221,10cyclictest17781-21sort23:15:112
615499221,10cyclictest10240-21latency_hist21:45:012
615099222,18cyclictest0-21swapper/119:33:201
615099221,19cyclictest0-21swapper/120:10:181
615099221,11cyclictest7106-21munin-run00:05:011
614499228,6cyclictest0-21swapper/023:36:510
614499227,3cyclictest16669-21kworker/0:023:26:080
614499224,7cyclictest15466-40dnf19:28:250
614499224,7cyclictest15466-40dnf19:28:250
614499220,11cyclictest0-21swapper/023:55:260
614499220,10cyclictest0-21swapper/022:25:040
616099218,3cyclictest0-21swapper/322:00:533
616099218,1cyclictest0-21swapper/320:04:073
616099212,4cyclictest21608-21if_enp2s020:55:183
616099211,9cyclictest0-21swapper/319:47:333
616099211,6cyclictest0-21swapper/321:20:133
616099211,5cyclictest0-21swapper/323:44:443
6160992115,4cyclictest16983-21sed23:10:273
615499214,8cyclictest23073-21needreboot23:25:182
615499213,9cyclictest25015-21crond21:05:012
615499213,9cyclictest12050irq/123-ahci[0022:00:022
615499213,5cyclictest1-21systemd19:30:002
615499213,5cyclictest1-21systemd19:30:002
615499211,9cyclictest27213-21diskmemload22:15:022
615499211,6cyclictest0-21swapper/221:37:382
615499211,10cyclictest7265-21cut19:10:192
615499211,10cyclictest579-21systemd-journal00:40:022
615099211,9cyclictest0-21swapper/100:31:561
6150992117,2cyclictest0-21swapper/119:10:211
615099211,3cyclictest773-21NetworkManager23:23:091
615099211,3cyclictest2481-21sshd23:52:371
615099211,18cyclictest0-21swapper/123:28:181
615099211,18cyclictest0-21swapper/123:04:081
615099211,18cyclictest0-21swapper/119:45:141
615099211,11cyclictest8725-21ntp_states20:25:201
615099211,11cyclictest5676-21processes22:45:231
615099211,11cyclictest26823-21grep21:05:261
615099211,11cyclictest15809-21cstates23:10:131
615099211,10cyclictest13549-21latency21:50:211
6150992110,3cyclictest27541-21ntp_states23:35:201
6150992110,2cyclictest21-21rcuc/120:00:171
615099210,19cyclictest0-21swapper/121:20:261
615099210,11cyclictest0-21swapper/119:55:201
614499216,2cyclictest87350irq/126-enp1s0-22:42:550
614499215,8cyclictest27213-21diskmemload23:13:350
614499214,4cyclictest9320-21awk00:10:000
614499214,4cyclictest9320-21awk00:10:000
614499211,17cyclictest21251-21idleruntime-cro22:10:010
614499211,10cyclictest661-21expr22:35:140
614499210,1cyclictest0-21swapper/023:50:130
614499210,14cyclictest0-21swapper/021:18:310
616099208,2cyclictest0-21swapper/319:10:263
616099202,10cyclictest9857-21/usr/sbin/munin20:30:123
616099201,6cyclictest0-21swapper/323:03:383
616099201,14cyclictest0-21swapper/323:55:193
616099201,11cyclictest13929-21/usr/sbin/munin00:20:173
616099201,11cyclictest1036-21/usr/sbin/munin20:10:113
615499204,10cyclictest0-21swapper/222:43:172
615499203,14cyclictest19882-21grep22:05:172
615499203,10cyclictest0-21swapper/200:10:272
615499201,7cyclictest0-21swapper/222:30:052
615499201,5cyclictest14011-21grep21:50:262
615499201,10cyclictest26702-21cut22:20:192
615099208,3cyclictest20286-21egrep19:40:131
615099208,2cyclictest10991-21head20:30:201
615099206,4cyclictest0-21swapper/122:29:321
615099205,9cyclictest0-21swapper/122:58:011
615099205,9cyclictest0-21swapper/122:58:011
615099202,17cyclictest0-21swapper/122:01:141
615099202,17cyclictest0-21swapper/121:39:411
615099202,10cyclictest3881-21users22:40:271
615099201,9cyclictest0-21swapper/122:11:141
615099201,8cyclictest0-21swapper/123:57:231
6150992016,3cyclictest0-21swapper/120:40:081
615099201,3cyclictest0-21swapper/122:21:501
615099201,17cyclictest0-21swapper/122:30:301
615099201,17cyclictest0-21swapper/120:51:011
615099201,17cyclictest0-21swapper/120:48:211
615099201,17cyclictest0-21swapper/120:15:071
615099200,19cyclictest0-21swapper/121:59:141
615099200,18cyclictest0-21swapper/119:22:151
615099200,18cyclictest0-21swapper/100:25:181
615099200,11cyclictest0-21swapper/120:05:211
614499207,4cyclictest0-21swapper/022:58:550
614499207,4cyclictest0-21swapper/022:58:540
614499206,5cyclictest0-21swapper/021:50:070
614499206,5cyclictest0-21swapper/021:30:430
614499204,8cyclictest87550irq/128-enp1s0-20:40:280
614499201,8cyclictest0-21swapper/019:32:270
614499201,6cyclictest0-21swapper/021:38:270
6144992015,3cyclictest19153-21grep23:15:250
614499201,16cyclictest28804-21basename20:00:010
614499201,16cyclictest10247-21munin-run21:45:010
614499201,11cyclictest21646-21sshd23:22:290
614499201,11cyclictest21464-21/usr/sbin/munin22:10:200
614499201,11cyclictest1673-21/usr/sbin/munin21:25:240
614499201,11cyclictest0-21swapper/023:05:220
114382201,2sleep311499-21cpu19:20:133
616099196,7cyclictest0-21swapper/323:23:063
616099196,4cyclictest790-21in:imjournal22:42:173
616099193,3cyclictest87650irq/129-enp1s0-23:05:503
616099191,8cyclictest27453-21users22:20:283
616099191,4cyclictest579-21systemd-journal21:40:163
616099191,4cyclictest11092-21grep00:10:283
616099191,4cyclictest0-21swapper/322:31:273
616099191,3cyclictest0-21swapper/321:33:593
6160991913,4cyclictest22098-21cut22:10:153
616099191,14cyclictest0-21swapper/300:34:083
616099191,11cyclictest0-21swapper/319:26:133
616099191,11cyclictest0-21swapper/319:26:133
616099191,10cyclictest7331-21/usr/sbin/munin00:05:263
616099191,10cyclictest7331-21/usr/sbin/munin00:05:263
616099191,10cyclictest19272-21/usr/sbin/munin22:05:253
616099190,15cyclictest0-21swapper/323:52:273
616099190,15cyclictest0-21swapper/320:22:363
615499195,3cyclictest87350irq/126-enp1s0-19:45:162
615499194,3cyclictest8324-21grep00:05:232
615499194,3cyclictest8324-21grep00:05:232
615499193,4cyclictest7276-21sshd22:50:172
615499191,9cyclictest0-21swapper/220:15:062
615499191,8cyclictest0-21swapper/223:05:232
615499191,6cyclictest0-21swapper/200:02:032
615499191,5cyclictest827-21runrttasks23:50:192
615499191,5cyclictest27226-21idleruntime-cro21:10:012
615499191,2cyclictest87550irq/128-enp1s0-20:00:162
615499191,11cyclictest21015-21grep23:20:222
615499191,11cyclictest0-21swapper/223:40:352
615499191,10cyclictest11950-21latency_hist20:35:022
615099193,7cyclictest23-21ksoftirqd/123:40:241
615099191,8cyclictest0-21swapper/122:35:511
6150991915,3cyclictest0-21swapper/123:06:181
6150991914,3cyclictest0-21swapper/121:29:391
615099191,17cyclictest0-21swapper/100:35:351
615099191,16cyclictest0-21swapper/121:42:521
615099191,16cyclictest0-21swapper/120:37:521
615099191,16cyclictest0-21swapper/120:37:521
615099191,16cyclictest0-21swapper/120:23:561
615099191,16cyclictest0-21swapper/100:20:091
615099191,10cyclictest0-21swapper/123:30:201
615099190,17cyclictest0-21swapper/121:32:081
615099190,14cyclictest0-21swapper/120:56:461
6144991915,2cyclictest250032sleep022:15:280
614499191,11cyclictest15215-21diskstats21:55:150
614499191,11cyclictest11735-21/usr/sbin/munin00:15:280
614499191,11cyclictest0-21swapper/023:44:050
614499191,11cyclictest0-21swapper/000:20:120
614499191,10cyclictest18029-21/usr/sbin/munin19:35:210
614499191,10cyclictest0-21swapper/022:53:380
614499191,10cyclictest0-21swapper/022:01:410
74012181,3sleep00-21swapper/020:24:470
616099187,3cyclictest0-21swapper/320:15:143
616099183,5cyclictest87650irq/129-enp1s0-00:35:173
616099181,9cyclictest0-21swapper/320:41:303
616099181,4cyclictest27444-21/usr/sbin/munin21:10:273
616099181,4cyclictest0-21swapper/300:25:583
616099181,3cyclictest0-21swapper/323:15:083
616099181,14cyclictest0-21swapper/300:01:553
6160991811,2cyclictest87350irq/126-enp1s0-23:34:163
616099181,11cyclictest0-21swapper/321:45:133
616099181,11cyclictest0-21swapper/321:45:133
616099181,10cyclictest0-21swapper/323:25:213
616099181,10cyclictest0-21swapper/322:55:093
616099181,10cyclictest0-21swapper/322:55:093
615499186,7cyclictest0-21swapper/221:48:032
615499186,7cyclictest0-21swapper/221:48:032
615499186,3cyclictest87350irq/126-enp1s0-00:15:232
615499183,3cyclictest22806-21turbostat.cron21:00:012
615499182,6cyclictest14511-21fw_packets00:20:162
615499182,5cyclictest18245-1kworker/u9:221:25:142
615499181,9cyclictest0-21swapper/222:56:322
615499181,9cyclictest0-21swapper/222:56:322
615499181,5cyclictest0-21swapper/200:25:232
615499181,4cyclictest21490-21smart_sda19:40:242
615499181,4cyclictest15508-21mailstats20:40:262
6154991813,3cyclictest15955-21tune2fs23:10:152
6154991812,3cyclictest31837-21/usr/sbin/munin21:20:232
6154991811,4cyclictest909-21sssd_nss22:25:192
6154991810,4cyclictest19504-21processes00:30:242
615099181,16cyclictest0-21swapper/122:52:111
615099181,10cyclictest0-21swapper/122:15:091
615099180,16cyclictest0-21swapper/122:05:061
614499187,8cyclictest30630-21fschecks_count22:30:140
614499187,3cyclictest16448-21/usr/sbin/munin20:45:220
614499187,3cyclictest10687-21tr00:10:230
614499186,3cyclictest0-21swapper/023:02:310
614499183,12cyclictest26239-21needreboot21:05:210
614499182,6cyclictest0-21swapper/021:11:180
614499182,4cyclictest24952-40dnf23:30:170
614499181,7cyclictest0-21swapper/020:31:370
6144991814,2cyclictest32214-21strings20:05:190
6144991812,3cyclictest1753-21sshd20:10:150
614499181,15cyclictest11258-21if_err_enp2s021:45:180
614499181,15cyclictest11258-21if_err_enp2s021:45:180
614499181,11cyclictest0-21swapper/000:25:170
614499181,10cyclictest31837-21/usr/sbin/munin21:20:210
291632181,2sleep20-21swapper/221:11:572
616099171,7cyclictest30042-21awk22:30:023
616099171,7cyclictest0-21swapper/320:50:103
6160991712,3cyclictest18029-21/usr/sbin/munin19:35:233
616099171,15cyclictest0-21swapper/320:35:513
616099171,15cyclictest0-21swapper/320:35:513
616099171,13cyclictest808-21snmpd19:50:273
6160991711,3cyclictest12837-21cstates21:50:133
616099171,10cyclictest0-21swapper/322:50:153
616099171,10cyclictest0-21swapper/319:30:213
615499173,8cyclictest87650irq/129-enp1s0-23:00:172
615499171,8cyclictest27816-21needreboot19:55:212
615499171,8cyclictest0-21swapper/220:23:262
615499171,4cyclictest26657-21cat23:35:102
615499171,4cyclictest12050irq/123-ahci[0019:25:012
6154991713,3cyclictest0-21swapper/220:54:002
6154991712,2cyclictest2905-21/usr/sbin/munin23:55:182
6154991711,3cyclictest3871-21/usr/sbin/munin21:30:132
6154991711,3cyclictest26166-21unixbench_singl19:50:272
614499171,4cyclictest12053-21/usr/sbin/munin20:35:250
614499171,4cyclictest12053-21/usr/sbin/munin20:35:240
6144991712,3cyclictest27324-21cat22:20:260
6144991711,4cyclictest29019-21/usr/sbin/munin20:00:170
6144991710,4cyclictest7999-21sh20:25:120
6144991710,4cyclictest4500-21/usr/sbin/munin20:15:220
614499170,6cyclictest0-21swapper/023:45:250
616099161,12cyclictest0-21swapper/320:08:273
616099161,11cyclictest0-21swapper/320:25:123
6160991610,3cyclictest20224-21/usr/sbin/munin19:40:133
615499161,6cyclictest7659-21/usr/sbin/munin20:25:212
6154991612,3cyclictest0-21swapper/222:16:062
6154991610,3cyclictest1036-21/usr/sbin/munin20:10:222
614499167,6cyclictest6477-21cpuspeed_turbos19:10:110
614499161,6cyclictest18643-21/usr/sbin/munin20:50:230
614499161,5cyclictest9043-21diskstats19:15:040
6144991612,3cyclictest0-21swapper/021:04:120
614499161,14cyclictest0-21swapper/019:23:110
614499160,10cyclictest0-21swapper/020:55:160
6154991510,3cyclictest16532-21idleruntime19:30:172
6144991510,3cyclictest22756-21expr19:45:120
6144991510,3cyclictest20901-21hddtemp_smartct19:40:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional