You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 13:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat May 02, 2026 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
413185722400,8sleep30-21swapper/319:05:193
3756621200,2sleep00-21swapper/023:45:180
414344021170,3sleep13056-21PK-Backend19:36:381
413126027340,9sleep00-21swapper/019:05:090
413165627244,20sleep20-21swapper/219:05:162
413255726445,14sleep10-21swapper/119:07:431
234292520,2sleep10-21swapper/123:05:131
41886722490,2sleep33056-21PK-Backend21:45:123
41374852340,2sleep30-21swapper/319:20:133
413288499301,5cyclictest0-21swapper/222:55:132
4132884993012,16cyclictest4174348-21date21:05:002
139099300,4rtkit-daemon1389-21rtkit-daemon22:29:413
139099290,3rtkit-daemon1389-21rtkit-daemon00:15:533
4132884992810,16cyclictest30571-21tr23:25:182
413287899281,25cyclictest4191897-21systemd-run21:55:010
413288499278,5cyclictest4158572-21cat20:20:152
413288499278,5cyclictest4143856-21cut19:40:012
4132884992712,13cyclictest37060-21cstates23:45:132
413288499271,11cyclictest4151789-21python320:00:222
4132884992711,14cyclictest54982-21latency00:35:152
4132884992711,14cyclictest4162538-21timerandwakeup20:30:212
4132884992710,6cyclictest4153651-21ntp_states20:05:192
4132884992710,15cyclictest9493-21sh22:25:132
413288099270,25cyclictest49274-21apt00:20:121
139099270,4rtkit-daemon1389-21rtkit-daemon19:56:383
413288499268,6cyclictest4194255-21awk22:00:152
413288499268,3cyclictest42287-21cstates00:00:122
413288499268,3cyclictest42287-21cstates00:00:122
413288499266,5cyclictest45277-21apt-get00:10:012
413288499262,4cyclictest13453-21python322:35:202
139099260,4rtkit-daemon1389-21rtkit-daemon20:26:323
4132889992517,6cyclictest883-21dbus-daemon23:00:003
413288499258,6cyclictest883-21dbus-daemon20:20:002
413288499258,5cyclictest4173983-21cron21:04:592
413288499255,5cyclictest4145016-21sed19:40:202
413288499255,4cyclictest0-21swapper/221:39:412
413288499251,8cyclictest38598-21apt23:50:102
413288499251,11cyclictest3679-20unin-run22:10:002
413288999248,14cyclictest45851-21cpu00:10:113
413288499249,13cyclictest4160501-21needreboot20:25:172
413288499248,3cyclictest4165968-21switchtime20:40:202
413288499248,15cyclictest0-21swapper/220:40:002
413288499247,4cyclictest4193154-21sed21:55:212
413288499246,4cyclictest4189206-21processes21:45:182
413288499246,4cyclictest36759-21latency_hist23:45:012
413288499246,4cyclictest0-21swapper/220:50:102
413288499244,5cyclictest4149458-21expr19:55:122
413288499242,7cyclictest48291-21awk00:15:212
413288499242,5cyclictest21255-21apt23:00:132
4132884992419,4cyclictest0-21swapper/200:22:152
413288499241,8cyclictest4288-21df_inode22:10:142
413288499240,2cyclictest12319-21latency_hist22:35:012
413288499238,2cyclictest18195-21idleruntime22:50:142
413288499237,2cyclictest4147331-21latency_hist19:50:012
413288499236,3cyclictest44659-21processes00:05:262
413288499236,3cyclictest4178757-21open_inodes21:15:192
413288499236,3cyclictest41406-21vmstat23:55:232
413288499234,5cyclictest7657-21expr22:20:122
413288499232,7cyclictest82332-21update-manager19:11:362
4132884992321,1cyclictest0-21swapper/200:26:542
413288499231,8cyclictest4154789-21cpuspeed_turbos20:10:112
413288499231,8cyclictest4147431-21/usr/sbin/munin19:50:232
413288499231,10cyclictest14295-21/usr/sbin/munin22:40:222
413288499230,7cyclictest4136171-21python319:15:192
413288499230,11cyclictest4172271-21munin-plugin-st21:00:002
4132880992320,2cyclictest3056-21PK-Backend21:42:081
413288099230,22cyclictest16562-21/usr/sbin/munin22:45:151
413288099230,21cyclictest3056-21PK-Backend20:00:281
413287899231,20cyclictest4165802-21ntp_states20:40:180
413287899230,4cyclictest4141059-21ls19:30:150
4132889992216,4cyclictest4182923-21cron21:30:003
413288999220,20cyclictest3056-21PK-Backend22:21:343
413288499226,2cyclictest33359-21taskset23:35:072
413288499225,5cyclictest4180043-21expr21:20:112
413288499221,8cyclictest4137841-21/usr/sbin/munin19:20:162
413288499221,7cyclictest4141091-21memory19:30:172
413288499221,4cyclictest33134-21/usr/sbin/munin23:35:012
413288499220,9cyclictest4140303-21cat19:30:002
413288499220,8cyclictest23415-21/usr/sbin/munin23:05:132
413288499220,7cyclictest25581-21python323:10:192
4132880992220,1cyclictest3056-21PK-Backend22:32:581
4132880992220,1cyclictest0-21swapper/121:49:111
4132880992220,1cyclictest0-21swapper/120:45:231
4132880992219,2cyclictest3056-21PK-Backend19:58:051
4132878992220,1cyclictest0-21swapper/021:55:490
4132878992220,1cyclictest0-21swapper/021:03:370
4132878992216,4cyclictest50748-21munin-run00:25:000
4132889992114,5cyclictest4133796-21ntp_kernel_pll_19:10:193
413288999210,20cyclictest3056-21PK-Backend22:48:313
413288999210,20cyclictest3056-21PK-Backend21:52:473
413288999210,19cyclictest3056-21PK-Backend21:30:303
413288999210,19cyclictest3056-21PK-Backend20:52:043
413288999210,19cyclictest3056-21PK-Backend19:28:223
413288999210,19cyclictest3056-21PK-Backend00:30:313
413288499215,3cyclictest1095-21runrttasks21:43:002
4132884992119,1cyclictest0-21swapper/223:24:292
4132884992119,1cyclictest0-21swapper/222:18:382
413288499211,6cyclictest4184669-21/usr/sbin/munin21:35:012
4132884992112,7cyclictest4181831-21df_inode21:25:142
413288499210,8cyclictest4176258-21apt21:10:102
413288499210,20cyclictest0-21swapper/223:16:512
4132880992119,1cyclictest3056-21PK-Backend20:25:221
4132880992119,1cyclictest3056-21PK-Backend00:17:351
4132880992119,1cyclictest0-21swapper/123:23:001
4132880992119,1cyclictest0-21swapper/123:17:121
4132880992119,1cyclictest0-21swapper/123:13:001
4132880992119,1cyclictest0-21swapper/121:37:161
4132880992119,1cyclictest0-21swapper/121:23:221
4132880992119,1cyclictest0-21swapper/120:55:511
4132880992118,2cyclictest3056-21PK-Backend22:52:111
4132880992116,3cyclictest14059-21latency_hist22:40:001
413288099210,20cyclictest3056-21PK-Backend23:51:121
413288099210,20cyclictest3056-21PK-Backend23:40:311
413288099210,20cyclictest3056-21PK-Backend20:21:241
413288099210,20cyclictest3056-21PK-Backend19:23:171
413288099210,20cyclictest0-21swapper/121:28:161
4132878992119,1cyclictest0-21swapper/023:53:510
4132878992119,1cyclictest0-21swapper/023:38:090
4132878992119,1cyclictest0-21swapper/023:10:520
4132878992119,1cyclictest0-21swapper/022:48:070
4132878992119,1cyclictest0-21swapper/022:41:590
4132878992119,1cyclictest0-21swapper/022:20:030
4132878992119,1cyclictest0-21swapper/021:47:110
4132878992119,1cyclictest0-21swapper/021:28:240
4132878992119,1cyclictest0-21swapper/020:51:060
4132878992119,1cyclictest0-21swapper/020:16:250
4132878992119,1cyclictest0-21swapper/020:13:140
4132878992119,1cyclictest0-21swapper/019:57:050
4132878992119,1cyclictest0-21swapper/019:39:140
4132878992119,1cyclictest0-21swapper/000:28:520
4132878992119,1cyclictest0-21swapper/000:12:570
4132878992114,5cyclictest26429-21/usr/sbin/munin23:15:220
413287899210,20cyclictest0-21swapper/000:38:460
413287899210,1cyclictest4144655-21strings19:40:160
413287899210,1cyclictest0-21swapper/019:45:210
510542200,3sleep33056-21PK-Backend00:25:113
41870112200,4sleep00-21swapper/021:40:110
41515642200,2sleep30-21swapper/320:00:183
4132889992018,1cyclictest3056-21PK-Backend23:29:203
4132889992018,1cyclictest0-21swapper/323:32:583
4132889992015,3cyclictest4136173-21python319:15:193
4132889992013,6cyclictest4144514-21sed19:40:153
4132889992013,5cyclictest4154626-21find20:10:103
413288999200,1cyclictest3056-21PK-Backend23:10:073
413288999200,19cyclictest3056-21PK-Backend22:43:103
413288999200,19cyclictest3056-21PK-Backend21:13:133
413288999200,19cyclictest3056-21PK-Backend20:44:063
413288999200,19cyclictest0-21swapper/323:51:583
413288999200,19cyclictest0-21swapper/323:36:373
413288999200,19cyclictest0-21swapper/323:01:423
413288999200,19cyclictest0-21swapper/321:35:573
413288999200,19cyclictest0-21swapper/319:49:043
413288999200,19cyclictest0-21swapper/300:01:513
413288999200,19cyclictest0-21swapper/300:01:513
413288499205,3cyclictest0-21swapper/221:50:172
413288499200,9cyclictest4168525-21apt-get20:50:002
413288499200,19cyclictest0-21swapper/200:34:112
4132880992018,1cyclictest3056-21PK-Backend21:16:401
4132880992018,1cyclictest0-21swapper/123:28:001
4132880992018,1cyclictest0-21swapper/122:40:431
4132880992018,1cyclictest0-21swapper/121:53:301
4132880992018,1cyclictest0-21swapper/121:04:161
4132880992018,1cyclictest0-21swapper/100:31:061
4132880992017,2cyclictest3056-21PK-Backend23:36:051
4132880992017,2cyclictest3056-21PK-Backend19:41:161
4132880992017,2cyclictest3056-21PK-Backend00:36:341
4132880992013,6cyclictest51885-21awk00:25:201
4132880992013,5cyclictest4192108-21/usr/sbin/munin21:55:151
4132880992013,5cyclictest3038-21python322:05:201
4132880992012,6cyclictest4156123-21cron20:15:011
413288099200,1cyclictest3056-21PK-Backend19:45:211
413288099200,1cyclictest3056-21PK-Backend19:33:041
413288099200,19cyclictest3056-21PK-Backend23:58:151
413288099200,19cyclictest3056-21PK-Backend23:47:341
413288099200,19cyclictest0-21swapper/120:43:351
413288099200,19cyclictest0-21swapper/119:55:011
413288099200,18cyclictest3056-21PK-Backend22:55:221
4132878992018,1cyclictest0-21swapper/023:43:230
4132878992018,1cyclictest0-21swapper/023:26:400
4132878992018,1cyclictest0-21swapper/023:21:400
4132878992018,1cyclictest0-21swapper/022:53:280
4132878992018,1cyclictest0-21swapper/022:36:380
4132878992018,1cyclictest0-21swapper/021:23:310
4132878992018,1cyclictest0-21swapper/020:32:070
4132878992018,1cyclictest0-21swapper/000:18:590
4132878992015,3cyclictest23424-21if_enp2s023:05:130
413287899201,17cyclictest4161158-21pool-udisksd20:28:030
413287899200,1cyclictest0-21swapper/020:21:460
413287899200,1cyclictest0-21swapper/019:51:440
413287899200,19cyclictest0-21swapper/020:57:560
413287899200,18cyclictest3056-21PK-Backend22:05:150
413287899200,18cyclictest0-21swapper/022:13:580
4132889991917,1cyclictest3056-21PK-Backend21:57:073
4132889991917,1cyclictest3056-21PK-Backend21:19:093
4132889991917,1cyclictest3056-21PK-Backend21:06:443
4132889991917,1cyclictest0-21swapper/323:55:363
4132889991917,1cyclictest0-21swapper/322:06:333
4132889991917,1cyclictest0-21swapper/321:42:133
4132889991917,1cyclictest0-21swapper/320:36:293
4132889991917,1cyclictest0-21swapper/320:33:523
4132889991917,1cyclictest0-21swapper/320:16:083
4132889991917,1cyclictest0-21swapper/320:06:553
4132889991917,1cyclictest0-21swapper/319:51:213
4132889991916,2cyclictest3056-21PK-Backend22:17:483
4132889991916,2cyclictest3056-21PK-Backend19:30:253
4132889991916,2cyclictest3056-21PK-Backend00:22:263
4132889991914,3cyclictest10857-21apt22:30:103
413288999191,3cyclictest0-21swapper/322:10:133
4132889991913,5cyclictest42-21ksoftirqd/323:41:423
4132889991912,6cyclictest55194-21ntpq00:35:173
4132889991912,6cyclictest4159086-21sed20:20:223
413288999190,6cyclictest4143733-21perf19:40:003
413288999190,1cyclictest0-21swapper/322:50:063
413288999190,1cyclictest0-21swapper/300:05:303
413288999190,18cyclictest0-21swapper/323:05:073
413288499195,5cyclictest34-21ksoftirqd/222:45:142
413288099193,15cyclictest32416-21/usr/sbin/munin23:30:191
4132880991917,1cyclictest0-21swapper/123:03:471
4132880991917,1cyclictest0-21swapper/121:13:421
4132880991917,1cyclictest0-21swapper/121:08:361
4132880991917,1cyclictest0-21swapper/120:50:231
4132880991917,1cyclictest0-21swapper/120:15:151
413288099190,1cyclictest3056-21PK-Backend22:13:591
413288099190,1cyclictest3056-21PK-Backend20:35:571
413288099190,1cyclictest3056-21PK-Backend19:16:541
413288099190,1cyclictest3056-21PK-Backend00:05:181
413288099190,1cyclictest0-21swapper/122:00:081
413288099190,1cyclictest0-21swapper/121:32:211
413288099190,18cyclictest0-21swapper/119:27:091
4132878991917,1cyclictest0-21swapper/023:58:170
4132878991913,5cyclictest4175556-21sed21:05:210
4132878991913,5cyclictest4151366-21awk20:00:160
4132878991912,5cyclictest4183432-21expr21:30:120
4132878991911,6cyclictest9119-21apt22:25:100
413287899190,1cyclictest0-21swapper/021:18:580
413287899190,1cyclictest0-21swapper/021:13:040
413287899190,1cyclictest0-21swapper/020:47:090
413287899190,1cyclictest0-21swapper/019:29:140
413287899190,1cyclictest0-21swapper/019:16:530
413287899190,1cyclictest0-21swapper/019:13:120
413287899190,18cyclictest0-21swapper/023:31:540
413287899190,18cyclictest0-21swapper/023:00:040
413287899190,18cyclictest0-21swapper/020:35:570
4132889991816,1cyclictest3056-21PK-Backend23:24:203
4132889991816,1cyclictest3056-21PK-Backend22:01:263
4132889991816,1cyclictest3056-21PK-Backend21:22:533
4132889991816,1cyclictest3056-21PK-Backend20:49:203
4132889991815,2cyclictest3056-21PK-Backend22:39:253
4132880991816,1cyclictest3056-21PK-Backend22:28:251
4132880991816,1cyclictest3056-21PK-Backend00:12:211
4132880991810,6cyclictest8391-21cat22:20:211
4132878991816,1cyclictest0-21swapper/000:00:060
4132878991816,1cyclictest0-21swapper/000:00:060
4132878991814,3cyclictest6191-21latency22:15:150
413288999172,13cyclictest36841-21/usr/sbin/munin23:45:173
4132889991715,1cyclictest3056-21PK-Backend23:19:203
4132880991714,2cyclictest3056-21PK-Backend19:13:441
4132880991712,3cyclictest6737-21vmstat22:15:211
4132878991715,1cyclictest0-21swapper/000:06:080
4132878991712,4cyclictest809-21sed22:00:210
4132878991711,4cyclictest53148-21basename00:30:150
4132878991711,4cyclictest4186618-21cat21:40:010
4132878991711,4cyclictest10979-21cpuspeed_turbos22:30:110
4132889991614,1cyclictest3056-21PK-Backend21:03:123
4132878991610,5cyclictest21062-21grep23:00:000
413287899160,3cyclictest41374842awk19:20:130
4132889991511,3cyclictest42-21ksoftirqd/320:58:403
413288099151,2cyclictest141rcu_preempt20:30:101
413288099142,2cyclictest141rcu_preempt20:09:131
4132880991410,3cyclictest3056-21PK-Backend00:00:231
4132880991410,3cyclictest3056-21PK-Backend00:00:231
413287899126,5cyclictest4153469-21cat20:05:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional