You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-23 - 20:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Dec 23, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
200885421100,6sleep00-21swapper/007:55:120
200885421100,6sleep00-21swapper/007:55:110
210472721060,8sleep3411rcuc/312:27:353
197399930,27rtkit-daemon4497-21PK-Backend11:27:161
197399930,24rtkit-daemon4497-21PK-Backend09:44:251
197399840,10rtkit-daemon2072531-21ntp_states10:55:271
197399710,61rtkit-daemon0-21swapper/007:07:450
199178327038,26sleep20-21swapper/207:06:292
20224862650,6sleep2331rcuc/208:35:022
197399650,5rtkit-daemon1994399-21munin-node08:06:243
20411262640,5sleep20-21swapper/209:25:282
199057826242,14sleep30-21swapper/307:05:093
197399620,9rtkit-daemon1972-21rtkit-daemon08:01:142
197399610,10rtkit-daemon1972-21rtkit-daemon10:39:352
199057126041,14sleep10-21swapper/107:05:041
197399600,9rtkit-daemon1972-21rtkit-daemon10:51:492
21041202590,6sleep14497-21PK-Backend12:25:281
20104412580,15sleep3421ksoftirqd/308:00:013
20104412580,15sleep3421ksoftirqd/308:00:013
197399580,8rtkit-daemon1972-21rtkit-daemon10:32:282
199227599570,52cyclictest1972-21rtkit-daemon08:12:282
197399560,9rtkit-daemon1972-21rtkit-daemon08:15:102
197399550,9rtkit-daemon1972-21rtkit-daemon08:40:182
197399540,9rtkit-daemon1972-21rtkit-daemon08:08:172
197399530,8rtkit-daemon1972-21rtkit-daemon08:37:492
20391402520,7sleep14497-21PK-Backend09:20:191
1992275995117,29cyclictest2042695-21cat09:30:242
1992275995117,29cyclictest2042695-21cat09:30:242
1992275995114,32cyclictest2086203-21cat11:35:252
1992275995018,28cyclictest2005860-21uname07:45:222
1992275995016,30cyclictest2002859-21wc07:35:292
1992275995014,31cyclictest2079509-21processes11:15:252
199227599497,12cyclictest1074-21lldpd11:21:102
1992275994917,27cyclictest2074981-21latency_hist11:05:022
1992275994913,31cyclictest2077105-21expr11:10:132
1992275994912,32cyclictest2031991-21cat09:00:232
1992275994818,26cyclictest1137-21runrttasks10:04:172
1992275994818,25cyclictest2090538-21cron11:50:002
1992275994816,27cyclictest2068318-21sh10:45:102
1992275994815,28cyclictest1-21systemd12:15:002
1992275994814,31cyclictest2048713-21sh09:50:012
197399480,39rtkit-daemon1074-21lldpd08:25:512
197399480,24rtkit-daemon0-21swapper/007:45:080
1992275994716,26cyclictest2107878-21cut12:35:212
1992275994712,31cyclictest2097472-21cron12:10:012
20459262460,6sleep30-21swapper/309:40:183
1992275994619,23cyclictest2046973-21latency_hist09:45:012
1992275994616,26cyclictest2104555-21apt-key12:27:352
1992275994615,26cyclictest2076100-21gzip11:05:262
1992275994615,26cyclictest2049084-21cpuspeed_turbos09:50:132
1992263994628,6cyclictest13-21ksoftirqd/008:30:140
1992263994624,11cyclictest13-21ksoftirqd/007:35:140
197399460,31rtkit-daemon2087722-21df11:40:161
21009932450,5sleep3411rcuc/312:17:043
1992275994517,24cyclictest4497-21PK-Backend07:51:522
1992275994514,27cyclictest2050696-21acpi09:55:122
1992275994514,27cyclictest2050696-21acpi09:55:112
199227599451,39cyclictest2005108-21awk07:44:592
1992275994512,28cyclictest2087555-21cstates11:40:152
1992275994510,30cyclictest2082297-21tr11:25:132
1992263994519,14cyclictest13-21ksoftirqd/007:15:060
199227599449,10cyclictest2058575-21uname10:15:232
1992275994418,22cyclictest1137-21runrttasks08:57:452
1992275994415,25cyclictest1998590-21cstates07:25:142
1992275994414,26cyclictest2038797-21cstates09:20:132
1992275994414,26cyclictest1997148-21kernelversion07:20:222
1992275994410,29cyclictest2084260-21df11:30:162
1992263994427,5cyclictest13-21ksoftirqd/009:00:010
1992263994427,13cyclictest13-21ksoftirqd/009:05:140
1992263994425,7cyclictest13-21ksoftirqd/007:50:230
197399440,31rtkit-daemon0-21swapper/010:05:270
1992279994324,16cyclictest1054-21snmpd10:28:483
1992275994316,23cyclictest2092713-21cpuspeed_turbos11:55:122
1992275994313,26cyclictest2055627-21latency_hist10:10:012
1992275994312,26cyclictest2057654-21latency_hist10:15:032
1992275994310,29cyclictest2067386-21proc_pri10:40:262
1992275994310,29cyclictest2062918-21date10:30:012
1992263994326,6cyclictest13-21ksoftirqd/011:40:200
1992263994325,6cyclictest13-21ksoftirqd/012:27:480
1992263994325,6cyclictest13-21ksoftirqd/012:20:110
1992263994325,6cyclictest13-21ksoftirqd/012:20:110
1992263994325,6cyclictest13-21ksoftirqd/011:35:480
1992263994325,6cyclictest13-21ksoftirqd/010:40:090
1992279994215,22cyclictest42-21ksoftirqd/312:35:293
1992275994216,22cyclictest1137-21runrttasks12:31:062
1992275994216,22cyclictest1137-21runrttasks09:18:502
1992270994219,19cyclictest1-21systemd12:10:001
199227099421,36cyclictest2025601-21sshd08:41:171
1992263994224,6cyclictest13-21ksoftirqd/012:30:150
1992263994224,6cyclictest13-21ksoftirqd/012:15:150
1992263994224,6cyclictest13-21ksoftirqd/009:55:020
1992263994224,6cyclictest13-21ksoftirqd/009:05:010
1992263994224,6cyclictest13-21ksoftirqd/008:55:020
1992263994224,6cyclictest13-21ksoftirqd/008:45:250
1992263994224,6cyclictest13-21ksoftirqd/008:35:110
1992263994224,6cyclictest13-21ksoftirqd/008:25:300
1992263994224,6cyclictest13-21ksoftirqd/007:40:140
1992263994224,6cyclictest13-21ksoftirqd/007:10:180
1992263994223,6cyclictest13-21ksoftirqd/010:00:250
1992263994223,6cyclictest13-21ksoftirqd/009:30:000
1992263994223,6cyclictest13-21ksoftirqd/008:40:300
1992263994220,13cyclictest13-21ksoftirqd/012:05:010
197399420,28rtkit-daemon4497-21PK-Backend11:20:150
199227599418,6cyclictest0-21swapper/207:58:132
199227599418,6cyclictest0-21swapper/207:58:122
199227599415,11cyclictest2060592-21cat10:20:302
199227599415,11cyclictest2060592-21cat10:20:292
1992275994116,22cyclictest1994255-21latency_hist07:15:012
1992275994111,26cyclictest20189302chrt08:24:342
1992275994111,26cyclictest20189302chrt08:24:342
199227599411,11cyclictest2095751-21sh12:05:002
199227599411,11cyclictest2043759-21/usr/sbin/munin09:35:232
1992275994110,27cyclictest0-21swapper/212:20:562
1992275994110,27cyclictest0-21swapper/212:20:562
1992270994121,16cyclictest1-21systemd07:50:001
1992270994116,21cyclictest1993785-40apt-check07:11:011
1992263994124,5cyclictest13-21ksoftirqd/010:10:120
1992263994123,6cyclictest13-21ksoftirqd/010:30:200
1992263994123,6cyclictest13-21ksoftirqd/009:15:180
1992263994123,6cyclictest13-21ksoftirqd/008:25:020
1992263994123,6cyclictest13-21ksoftirqd/008:25:020
1992263994123,6cyclictest13-21ksoftirqd/008:05:100
1992263994123,6cyclictest13-21ksoftirqd/008:00:200
1992263994123,6cyclictest13-21ksoftirqd/007:30:160
1992263994123,6cyclictest13-21ksoftirqd/007:30:150
1992263994123,5cyclictest13-21ksoftirqd/009:15:000
20741102400,8sleep30-21swapper/311:00:233
20710242400,16sleep342-21ksoftirqd/310:50:333
20294912400,8sleep20-21swapper/208:55:012
1992279994025,10cyclictest438-20systemd-journal11:08:173
199227999401,16cyclictest20785622cut11:15:013
1992279994010,11cyclictest141rcu_preempt09:25:323
199227599402,15cyclictest1994597-21df07:15:042
199227599401,12cyclictest2036279-21apt09:14:592
1992275994011,25cyclictest2027667-21systemd08:50:012
1992263994023,6cyclictest13-21ksoftirqd/010:35:100
1992263994023,6cyclictest13-21ksoftirqd/008:10:190
1992263994023,5cyclictest13-21ksoftirqd/012:15:010
1992263994022,6cyclictest13-21ksoftirqd/011:50:280
1992263994022,6cyclictest13-21ksoftirqd/011:25:160
1992263994022,6cyclictest13-21ksoftirqd/007:25:260
1992263994022,5cyclictest13-21ksoftirqd/012:05:290
20229862390,7sleep34497-21PK-Backend08:35:153
20150172390,4sleep10-21swapper/108:10:291
1992279993924,11cyclictest923-21dbus-daemon09:49:573
199227599397,6cyclictest0-21swapper/212:15:022
199227599391,12cyclictest2001491-21taskset07:34:072
199227599391,12cyclictest2001491-21taskset07:34:062
199227599391,11cyclictest2033470-21sshd09:05:182
1992263993923,5cyclictest13-21ksoftirqd/011:45:200
1992263993922,5cyclictest13-21ksoftirqd/011:55:200
1992263993922,5cyclictest13-21ksoftirqd/010:25:310
1992263993922,5cyclictest13-21ksoftirqd/008:15:160
1992263993921,5cyclictest13-21ksoftirqd/009:25:010
197399390,27rtkit-daemon4497-21PK-Backend12:23:171
197399390,27rtkit-daemon4497-21PK-Backend12:23:171
197399390,27rtkit-daemon4497-21PK-Backend11:15:581
197399390,25rtkit-daemon0-21swapper/011:15:500
197399390,25rtkit-daemon0-21swapper/010:56:180
1992279993821,13cyclictest2091768-21threads11:50:303
1992279993814,20cyclictest42-21ksoftirqd/311:55:283
199227099385,29cyclictest1137-21runrttasks09:17:051
199227099382,32cyclictest2044369-21memory09:35:231
1992270993821,12cyclictest438-20systemd-journal10:35:171
1992263993821,6cyclictest13-21ksoftirqd/012:35:130
1992263993821,6cyclictest13-21ksoftirqd/007:20:160
1992263993821,5cyclictest13-21ksoftirqd/011:30:280
1992263993818,13cyclictest13-21ksoftirqd/010:45:120
197399380,27rtkit-daemon4497-21PK-Backend11:12:191
197399380,27rtkit-daemon4497-21PK-Backend11:09:551
197399380,26rtkit-daemon4497-21PK-Backend07:20:511
197399380,26rtkit-daemon0-21swapper/010:16:040
1992279993723,10cyclictest2055623-21apt-get10:10:013
1992279993720,12cyclictest438-20systemd-journal09:40:013
1992279993710,12cyclictest141rcu_preempt08:00:253
199227599371,11cyclictest2091995-21chrt11:51:282
199227099372,7cyclictest2000495-21diskstats07:30:181
199227099372,7cyclictest2000495-21diskstats07:30:181
199227099372,31cyclictest2023193-21irqstats08:35:201
1992270993721,12cyclictest2011465-21python308:00:271
199227099371,10cyclictest0-21swapper/108:46:511
1992263993720,6cyclictest13-21ksoftirqd/011:01:120
1992263993720,5cyclictest13-21ksoftirqd/011:10:090
1992263993720,5cyclictest13-21ksoftirqd/011:10:010
1992263993720,5cyclictest13-21ksoftirqd/009:55:020
1992263993720,5cyclictest13-21ksoftirqd/009:55:020
1992263993720,5cyclictest13-21ksoftirqd/009:35:200
197399370,26rtkit-daemon0-21swapper/107:15:521
197399370,25rtkit-daemon0-21swapper/109:50:371
197399370,25rtkit-daemon0-21swapper/109:34:471
197399370,25rtkit-daemon0-21swapper/109:34:471
197399370,25rtkit-daemon0-21swapper/109:07:381
197399370,25rtkit-daemon0-21swapper/107:44:391
197399370,25rtkit-daemon0-21swapper/010:53:360
199227999367,25cyclictest2038038-21chrt09:17:003
199227999362,30cyclictest2084768-21python311:30:283
1992279993620,11cyclictest2058680-21ntpq10:15:253
1992279993610,6cyclictest141rcu_preempt09:30:153
1992279993610,6cyclictest141rcu_preempt09:30:153
199227599361,11cyclictest2072935-21chrt10:56:142
199227099362,6cyclictest2015756-21/usr/sbin/munin08:15:201
199227099362,11cyclictest2052167-21munin-run10:00:011
199227099362,11cyclictest2052167-21munin-run10:00:011
1992270993621,11cyclictest2090987-21cpuspeed_turbos11:50:141
1992270993621,11cyclictest2041004-21cut09:25:241
1992270993620,12cyclictest2090217-21seq11:45:561
1992270993620,11cyclictest438-20systemd-journal12:10:181
1992270993620,11cyclictest438-20systemd-journal10:34:541
1992270993620,11cyclictest438-20systemd-journal07:40:001
1992263993620,5cyclictest13-21ksoftirqd/009:30:230
1992263993620,5cyclictest13-21ksoftirqd/009:30:230
1992263993619,5cyclictest13-21ksoftirqd/010:25:000
1992263993619,5cyclictest13-21ksoftirqd/010:24:590
197399360,24rtkit-daemon0-21swapper/111:59:261
197399360,24rtkit-daemon0-21swapper/111:01:291
197399360,24rtkit-daemon0-21swapper/108:54:101
197399360,23rtkit-daemon0-21swapper/009:41:030
199227999358,10cyclictest141rcu_preempt09:02:233
1992279993523,8cyclictest2052804-21df_inode10:00:163
1992279993520,11cyclictest923-21dbus-daemon10:10:063
1992279993520,11cyclictest42-21ksoftirqd/312:10:263
1992279993518,13cyclictest42-21ksoftirqd/307:50:013
1992279993518,12cyclictest438-20systemd-journal09:00:003
199227999351,31cyclictest4497-21PK-Backend12:05:503
1992279993512,20cyclictest976-21rs:main3
1992279993512,18cyclictest42-21ksoftirqd/307:40:103
1992270993523,7cyclictest2008372-21systemd-udevd07:52:491
1992270993519,12cyclictest2053401-21sed10:00:281
1992270993519,12cyclictest2019610-21sshd08:25:161
1992270993515,16cyclictest4497-21PK-Backend10:10:001
199227099351,15cyclictest4497-21PK-Backend10:18:051
197399350,23rtkit-daemon0-21swapper/112:38:221
199227999348,7cyclictest2086249-21missed_timers11:35:253
199227999348,10cyclictest141rcu_preempt07:51:563
1992279993421,9cyclictest1998101-21cron07:25:013
1992279993420,10cyclictest2081672-21sshd11:22:173
1992279993419,11cyclictest42-21ksoftirqd/307:35:233
1992279993419,11cyclictest2068619-21expr10:45:133
1992279993419,11cyclictest2026819-21/usr/sbin/munin08:45:223
1992279993419,11cyclictest2014925-21python308:10:283
1992279993419,10cyclictest2067035-21awk10:40:173
1992279993418,11cyclictest2051414-21ntpq09:55:233
1992279993418,11cyclictest2051414-21ntpq09:55:233
1992279993417,13cyclictest42-21ksoftirqd/310:20:233
1992279993417,13cyclictest42-21ksoftirqd/310:20:233
1992279993417,13cyclictest42-21ksoftirqd/308:20:223
1992279993417,13cyclictest42-21ksoftirqd/308:20:213
1992279993417,13cyclictest42-21ksoftirqd/307:20:023
1992279993417,13cyclictest2038951-21df09:20:143
1992279993416,13cyclictest42-21ksoftirqd/308:30:243
1992279993411,20cyclictest1137-21runrttasks11:30:023
1992270993427,4cyclictest4497-21PK-Backend11:39:311
1992270993423,7cyclictest2062847-21turbostat10:29:591
199227099341,9cyclictest4497-21PK-Backend12:01:161
1992270993416,13cyclictest2013084-21awk08:05:251
1992263993421,9cyclictest2048700-21sh09:50:010
197399340,25rtkit-daemon1972-21rtkit-daemon08:18:003
199227999337,10cyclictest141rcu_preempt10:34:433
1992279993323,5cyclictest42-21ksoftirqd/312:00:233
1992279993323,5cyclictest42-21ksoftirqd/308:26:503
1992279993322,7cyclictest2049167-21cstates09:50:143
199227999331,5cyclictest4497-21PK-Backend12:31:283
199227999331,4cyclictest4497-21PK-Backend10:57:033
1992279993314,14cyclictest438-20systemd-journal12:25:003
1992279993314,14cyclictest438-20systemd-journal12:25:003
199227999331,11cyclictest1998453-21cat07:25:123
199227099331,9cyclictest4497-21PK-Backend10:12:261
199227099331,9cyclictest4497-21PK-Backend08:57:241
1992270993318,11cyclictest2021694-21ntpq08:30:221
1992270993317,12cyclictest2066893-21expr10:40:131
199227099331,29cyclictest0-21swapper/112:17:041
197399330,22rtkit-daemon0-21swapper/110:24:171
197399330,22rtkit-daemon0-21swapper/110:24:171
1992279993221,7cyclictest2088816-21turbostat.cron11:44:593
199227999321,9cyclictest4497-21PK-Backend09:06:453
1992279993218,11cyclictest42-21ksoftirqd/308:40:243
1992279993218,10cyclictest2065446-21grep10:35:213
1992279993217,11cyclictest42-21ksoftirqd/307:10:143
199227999321,5cyclictest4497-21PK-Backend11:45:303
1992279993215,13cyclictest42-21ksoftirqd/308:55:003
199227999321,4cyclictest4497-21PK-Backend09:10:053
1992270993217,11cyclictest2048262-21grep09:45:271
1992270993214,13cyclictest438-20systemd-journal07:25:021
199227099321,28cyclictest0-21swapper/112:33:561
199227099321,28cyclictest0-21swapper/107:55:571
199227099321,28cyclictest0-21swapper/107:55:571
199227999311,9cyclictest4497-21PK-Backend07:34:403
199227999311,9cyclictest4497-21PK-Backend07:34:403
1992270993121,5cyclictest26-21ksoftirqd/111:21:581
1992270993118,9cyclictest2069939-21latency_hist10:50:021
1992270993117,10cyclictest2017571-21cpuspeed08:20:141
1992270993117,10cyclictest2017571-21cpuspeed08:20:141
199227099311,27cyclictest4497-21PK-Backend11:32:201
199227099311,27cyclictest0-21swapper/109:11:561
1992270993015,10cyclictest2071686-21cat10:55:021
1992270992914,11cyclictest2032872-21sed09:05:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional