You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-16 - 22:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Wed Sep 16, 2026 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
216662623460,12sleep3411rcuc/307:05:153
224445821460,9sleep013-21ksoftirqd/010:45:120
218970621350,4sleep30-21swapper/308:10:003
218970621350,4sleep30-21swapper/308:10:003
225096321320,3sleep03056-21PK-Backend11:00:230
225096321320,3sleep03056-21PK-Backend11:00:220
218687321230,3sleep00-21swapper/008:00:190
216743927940,33sleep20-21swapper/207:06:112
216758427739,14sleep00-21swapper/007:08:020
216707226645,14sleep10-21swapper/107:05:301
22499162470,7sleep1261ksoftirqd/111:00:121
22499162470,7sleep1261ksoftirqd/111:00:121
22395232440,3sleep20-21swapper/210:30:182
139099390,3rtkit-daemon1389-21rtkit-daemon11:56:162
139099390,3rtkit-daemon1389-21rtkit-daemon11:56:162
139099280,4rtkit-daemon1389-21rtkit-daemon11:53:383
139099280,4rtkit-daemon1389-21rtkit-daemon07:45:413
139099280,4rtkit-daemon1389-21rtkit-daemon07:35:103
139099270,4rtkit-daemon1389-21rtkit-daemon07:14:593
139099270,3rtkit-daemon1389-21rtkit-daemon10:28:013
22785462260,2chrt0-21swapper/312:20:183
139099260,4rtkit-daemon1389-21rtkit-daemon12:37:223
139099260,4rtkit-daemon1389-21rtkit-daemon09:36:252
139099260,3rtkit-daemon1389-21rtkit-daemon11:13:243
22822592250,9sleep342-21ksoftirqd/312:30:203
216796599250,23cyclictest1389-21rtkit-daemon07:32:553
216796599240,22cyclictest1389-21rtkit-daemon07:28:173
2167961992412,10cyclictest2281766-21df_inode12:30:142
139099240,4rtkit-daemon1389-21rtkit-daemon09:27:593
139099240,4rtkit-daemon1389-21rtkit-daemon07:43:523
139099240,3rtkit-daemon1389-21rtkit-daemon11:01:413
139099240,3rtkit-daemon1389-21rtkit-daemon11:01:413
2167961992321,1cyclictest0-21swapper/208:36:172
2167956992315,6cyclictest2168709-21df_inode07:10:141
216796599220,19cyclictest1389-21rtkit-daemon11:56:163
216796599220,19cyclictest1389-21rtkit-daemon11:56:153
2167961992220,1cyclictest0-21swapper/212:03:012
2167961992220,1cyclictest0-21swapper/211:37:052
2167961992220,1cyclictest0-21swapper/211:27:182
2167961992220,1cyclictest0-21swapper/209:55:452
2167961992220,1cyclictest0-21swapper/208:20:282
2167956992220,1cyclictest3056-21PK-Backend07:35:231
2167956992220,1cyclictest0-21swapper/108:15:111
2167956992219,2cyclictest3056-21PK-Backend11:39:591
216795699220,19cyclictest0-21swapper/108:50:131
216795199224,16cyclictest1-21systemd08:35:000
2167951992220,1cyclictest0-21swapper/008:36:140
2167951992216,5cyclictest2243977-21systemd-run10:45:000
2167951992215,5cyclictest2275315-21python312:10:190
216795199220,20cyclictest2204082-21apt08:50:130
216796599210,20cyclictest3056-21PK-Backend07:58:573
216796599210,20cyclictest0-21swapper/309:30:093
216796599210,19cyclictest3056-21PK-Backend09:20:013
216796599210,18cyclictest2192405-21/usr/sbin/munin08:15:193
2167961992119,1cyclictest0-21swapper/211:14:212
2167961992119,1cyclictest0-21swapper/211:09:342
2167961992119,1cyclictest0-21swapper/209:24:562
2167961992119,1cyclictest0-21swapper/209:07:192
2167961992119,1cyclictest0-21swapper/209:07:192
2167961992119,1cyclictest0-21swapper/208:30:562
2167961992119,1cyclictest0-21swapper/208:11:572
2167961992119,1cyclictest0-21swapper/207:58:452
2167961992119,1cyclictest0-21swapper/207:16:542
2167961992114,6cyclictest2251193-21sshd11:02:382
2167961992114,6cyclictest2251193-21sshd11:02:382
216796199210,20cyclictest0-21swapper/211:41:032
216796199210,20cyclictest0-21swapper/207:50:132
216796199210,20cyclictest0-21swapper/207:13:502
216796199210,19cyclictest3056-21PK-Backend12:10:242
2167956992119,1cyclictest3056-21PK-Backend10:35:441
2167956992119,1cyclictest3056-21PK-Backend07:28:131
2167956992119,1cyclictest0-21swapper/108:00:581
2167956992119,1cyclictest0-21swapper/107:19:481
2167956992118,2cyclictest3056-21PK-Backend08:56:551
2167956992118,2cyclictest3056-21PK-Backend08:21:201
216795699210,20cyclictest0-21swapper/110:30:241
216795699210,20cyclictest0-21swapper/108:35:191
216795699210,19cyclictest3056-21PK-Backend08:07:071
216795699210,19cyclictest3056-21PK-Backend08:07:071
216795699210,19cyclictest3056-21PK-Backend07:42:331
216795699210,19cyclictest3056-21PK-Backend07:33:201
2167951992113,6cyclictest2252660-21sed11:05:220
216795199210,20cyclictest0-21swapper/011:46:080
139099210,4rtkit-daemon1389-21rtkit-daemon10:35:432
139099210,3rtkit-daemon1389-21rtkit-daemon07:53:383
2167965992018,1cyclictest0-21swapper/307:23:413
2167965992017,2cyclictest3056-21PK-Backend08:34:253
2167965992016,3cyclictest1095-21runrttasks08:04:083
2167965992012,6cyclictest436-20systemd-journal10:19:593
2167965992012,6cyclictest2280414-21awk12:25:183
2167965992012,6cyclictest2219934-21find09:35:123
216796599200,1cyclictest3056-21PK-Backend12:06:103
216796599200,1cyclictest3056-21PK-Backend10:08:013
216796599200,19cyclictest3056-21PK-Backend11:48:253
216796599200,19cyclictest3056-21PK-Backend11:08:363
216796599200,19cyclictest3056-21PK-Backend10:44:163
216796599200,19cyclictest0-21swapper/311:44:273
216796599200,19cyclictest0-21swapper/308:59:203
216796599200,18cyclictest3056-21PK-Backend08:52:233
216796599200,18cyclictest3056-21PK-Backend08:41:283
2167961992018,1cyclictest0-21swapper/212:17:482
2167961992018,1cyclictest0-21swapper/210:53:452
2167961992018,1cyclictest0-21swapper/210:13:172
2167961992018,1cyclictest0-21swapper/208:50:012
2167961992018,1cyclictest0-21swapper/208:40:162
216796199200,19cyclictest0-21swapper/212:25:112
216796199200,19cyclictest0-21swapper/211:17:312
216796199200,19cyclictest0-21swapper/210:23:092
216796199200,19cyclictest0-21swapper/210:03:302
216796199200,19cyclictest0-21swapper/208:58:072
216796199200,19cyclictest0-21swapper/208:25:002
216796199200,19cyclictest0-21swapper/208:08:112
216796199200,19cyclictest0-21swapper/208:08:112
216796199200,19cyclictest0-21swapper/207:44:252
216796199200,19cyclictest0-21swapper/207:28:372
2167956992018,1cyclictest3056-21PK-Backend11:40:471
2167956992018,1cyclictest3056-21PK-Backend10:55:241
2167956992018,1cyclictest3056-21PK-Backend10:24:291
2167956992018,1cyclictest3056-21PK-Backend10:12:261
2167956992018,1cyclictest3056-21PK-Backend07:45:511
2167956992018,1cyclictest0-21swapper/111:16:271
2167956992018,1cyclictest0-21swapper/111:09:101
2167956992018,1cyclictest0-21swapper/109:40:281
2167956992018,1cyclictest0-21swapper/108:31:341
2167956992018,1cyclictest0-21swapper/107:58:281
216795699201,17cyclictest2244602-21cpuspeed_turbos10:45:131
216795699200,1cyclictest3056-21PK-Backend12:24:341
216795699200,1cyclictest3056-21PK-Backend10:29:361
216795699200,1cyclictest3056-21PK-Backend07:51:251
216795699200,19cyclictest3056-21PK-Backend11:54:521
216795699200,19cyclictest3056-21PK-Backend11:23:151
216795699200,19cyclictest3056-21PK-Backend11:11:341
216795699200,19cyclictest0-21swapper/110:06:031
216795699200,19cyclictest0-21swapper/110:04:271
216795699200,19cyclictest0-21swapper/109:58:201
216795699200,19cyclictest0-21swapper/109:16:421
216795699200,18cyclictest3056-21PK-Backend09:45:561
2167951992018,1cyclictest3056-21PK-Backend07:20:080
2167951992018,1cyclictest0-21swapper/010:02:320
2167951992018,1cyclictest0-21swapper/009:06:350
2167951992018,1cyclictest0-21swapper/009:06:340
2167951992018,1cyclictest0-21swapper/008:16:470
2167951992018,1cyclictest0-21swapper/008:14:030
2167951992012,6cyclictest2253375-21find11:10:110
216795199200,19cyclictest0-21swapper/012:22:370
216795199200,18cyclictest3056-21PK-Backend08:26:270
2167965991917,1cyclictest0-21swapper/310:30:253
2167965991917,1cyclictest0-21swapper/309:11:423
2167965991914,3cyclictest2208280-21ntp_states09:00:183
2167965991913,4cyclictest2156-21gnome-shell11:29:023
2167965991912,5cyclictest2222613-21cat09:40:233
2167965991911,6cyclictest2190336-21awk08:10:153
216796599191,14cyclictest2195415-21apt08:25:123
216796599190,7cyclictest2210339-21/usr/sbin/munin09:05:213
216796599190,7cyclictest2210339-21/usr/sbin/munin09:05:213
216796599190,1cyclictest3056-21PK-Backend12:12:323
216796599190,1cyclictest3056-21PK-Backend09:50:373
216796599190,1cyclictest3056-21PK-Backend07:16:323
216796599190,1cyclictest0-21swapper/310:50:393
216796599190,18cyclictest3056-21PK-Backend09:15:423
216796599190,18cyclictest0-21swapper/310:22:073
2167961991917,1cyclictest0-21swapper/210:18:032
216796199190,1cyclictest0-21swapper/209:41:132
216796199190,1cyclictest0-21swapper/207:34:442
216796199190,18cyclictest0-21swapper/209:51:132
2167956991917,1cyclictest3056-21PK-Backend09:00:131
2167956991917,1cyclictest0-21swapper/110:41:531
2167956991917,1cyclictest0-21swapper/109:50:151
2167956991917,1cyclictest0-21swapper/108:28:361
2167956991912,5cyclictest2272405-21sshd12:04:381
216795699190,1cyclictest3056-21PK-Backend11:58:521
216795699190,1cyclictest3056-21PK-Backend11:58:521
216795699190,1cyclictest3056-21PK-Backend09:08:381
216795699190,1cyclictest3056-21PK-Backend09:08:371
216795699190,1cyclictest3056-21PK-Backend08:11:261
216795699190,18cyclictest0-21swapper/112:06:091
216795199193,14cyclictest1-21systemd07:40:010
216795199191,5cyclictest2249669-21turbostat11:00:000
2167951991913,4cyclictest2189676-21sh08:10:000
2167951991913,4cyclictest2189676-21sh08:10:000
2167951991912,5cyclictest2273162-21awk12:05:160
2167951991912,5cyclictest2264619-21sed11:40:170
2167951991912,5cyclictest2218806-21expr09:30:190
2167951991911,6cyclictest2268258-21ntp_states11:50:190
216795199190,1cyclictest3056-21PK-Backend12:26:090
216795199190,1cyclictest0-21swapper/009:00:130
216795199190,1cyclictest0-21swapper/007:25:230
216795199190,18cyclictest0-21swapper/010:18:000
216795199190,18cyclictest0-21swapper/009:54:550
139099190,15rtkit-daemon2167944-21cyclictest11:39:083
2167965991816,1cyclictest3056-21PK-Backend10:01:043
2167965991816,1cyclictest3056-21PK-Backend09:56:323
2167965991816,1cyclictest0-21swapper/310:47:283
2167965991813,3cyclictest2255287-21grep11:15:123
2167965991813,3cyclictest2156-21gnome-shell10:36:093
216796599181,15cyclictest2223388-21/usr/sbin/munin09:45:143
2167961991816,1cyclictest0-21swapper/209:34:232
2167961991816,1cyclictest0-21swapper/209:28:012
2167961991816,1cyclictest0-21swapper/209:16:592
2167961991816,1cyclictest0-21swapper/208:46:032
2167961991816,1cyclictest0-21swapper/208:18:052
2167961991816,1cyclictest0-21swapper/208:03:582
2167961991814,3cyclictest2181684-21tail07:45:172
2167961991812,5cyclictest2268887-21seq11:53:572
2167961991812,5cyclictest2261066-21irqstats11:30:172
2167961991811,6cyclictest2223756-21ls09:45:142
2167961991811,5cyclictest2237655-21entropy10:25:142
2167961991811,5cyclictest2178433-21cat07:35:232
216795699189,7cyclictest2202349-21apt08:45:131
2167956991816,1cyclictest3056-21PK-Backend09:31:561
2167956991816,1cyclictest0-21swapper/110:50:381
2167956991811,5cyclictest2274658-21cat12:10:111
2167956991811,5cyclictest2200874-21expr08:40:141
216795199189,7cyclictest883-21dbus-daemon10:30:010
2167951991812,4cyclictest2168957-21cat07:10:200
2167951991811,5cyclictest1095-21runrttasks08:45:130
2167951991810,6cyclictest2185490-21sed07:55:220
216795199180,1cyclictest0-21swapper/010:36:260
216795199180,1cyclictest0-21swapper/010:34:090
216796599172,13cyclictest2257100-21/usr/sbin/munin11:20:193
2167965991711,5cyclictest42-21ksoftirqd/310:10:223
216796599170,15cyclictest2260564-21seq11:30:103
216796199179,6cyclictest2245935-21apt-get10:50:012
2167961991715,1cyclictest0-21swapper/209:04:222
2167961991711,4cyclictest2248919-21ntp_kernel_pll_10:55:182
2167961991710,5cyclictest2242989-21cut10:40:182
2167956991715,1cyclictest3056-21PK-Backend09:35:491
2167956991715,1cyclictest3056-21PK-Backend09:25:481
2167956991715,1cyclictest3056-21PK-Backend09:11:011
216795199179,6cyclictest2224121-21ntpq09:45:190
2167951991715,1cyclictest0-21swapper/012:15:350
2167951991715,1cyclictest0-21swapper/011:27:220
2167951991715,1cyclictest0-21swapper/010:23:480
2167951991715,1cyclictest0-21swapper/007:41:180
2167951991712,4cyclictest2271431-21cat12:00:150
2167951991712,3cyclictest2230924-21missed_timers10:05:170
2167951991711,5cyclictest13-21ksoftirqd/009:13:070
2167951991711,4cyclictest2255918-21ps11:15:200
2167951991710,6cyclictest2194192-21awk08:20:210
216795199170,16cyclictest0-21swapper/011:57:300
216795199170,16cyclictest0-21swapper/011:57:290
216796599169,5cyclictest2276519-21cat12:15:143
216796599169,5cyclictest2195224-21date08:25:013
2167965991613,2cyclictest2270986-21apt12:00:093
2167965991612,3cyclictest2248174-21/usr/sbin/munin10:55:133
216796199168,6cyclictest2211248-21cat09:10:122
216796199168,5cyclictest2173030-21ntp_offset07:20:182
2167961991611,4cyclictest2231925-21grep10:10:012
216795699169,5cyclictest2267132-21sshd11:48:391
216795699163,11cyclictest2260333-21munin-plugin-st11:29:591
2167956991610,4cyclictest2234316-21cut10:15:171
216795199169,5cyclictest2213501-21awk09:15:200
216795199167,7cyclictest1-21systemd09:25:010
216795199166,8cyclictest2200789-21cpuspeed_turbos08:40:120
216795199164,11cyclictest1095-21runrttasks10:14:590
2167951991614,1cyclictest0-21swapper/012:33:130
2167951991614,1cyclictest0-21swapper/009:42:100
2167951991614,1cyclictest0-21swapper/007:52:270
2167951991614,1cyclictest0-21swapper/007:48:350
2167951991614,1cyclictest0-21swapper/007:34:150
2167951991614,1cyclictest0-21swapper/007:17:320
2167951991611,3cyclictest2283267-21tr12:35:130
216795199160,15cyclictest0-21swapper/011:37:090
139099160,13rtkit-daemon1317-21systemd12:22:202
22730402150,4sleep234-21ksoftirqd/212:05:122
2167965991513,1cyclictest3056-21PK-Backend08:36:013
2167965991512,2cyclictest3056-21PK-Backend08:47:153
216796199159,4cyclictest2156-21gnome-shell11:21:092
216796199158,5cyclictest2283366-21cat12:35:122
2167961991511,3cyclictest2266247-21if_err_enp2s011:45:172
216795699159,4cyclictest2172487-21tr07:20:121
216795699158,6cyclictest2214705-21mktemp09:20:011
2167956991511,3cyclictest1095-21runrttasks12:25:141
2167956991510,4cyclictest3056-21PK-Backend12:35:161
216795199159,5cyclictest2220899-21grep09:35:230
216795199158,5cyclictest2257283-21expr11:20:130
216795199157,7cyclictest2085813-21snapd08:59:130
216795199154,2cyclictest141rcu_preempt10:50:160
216795199152,11cyclictest2216669-21grep09:25:120
2167951991510,3cyclictest1095-21runrttasks09:55:030
216795199150,14cyclictest0-21swapper/011:30:530
2167956991410,3cyclictest3056-21PK-Backend12:15:101
2167956991410,3cyclictest3056-21PK-Backend11:34:451
216795699127,3cyclictest2281126-21sh12:34:561
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional