You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 04:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Thu Jun 25, 2026 00:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
511221130,3sleep00-21swapper/020:53:190
511221130,3sleep00-21swapper/020:53:190
511221130,3sleep00-21swapper/020:53:190
122452890,3chrt0-21swapper/322:38:233
122452890,3chrt0-21swapper/322:38:233
122452890,3chrt0-21swapper/322:38:233
249242880,8sleep024012-21kworker/0:0-events23:13:170
249242880,8sleep024012-21kworker/0:0-events23:13:170
107542840,3chrt0-21swapper/321:08:183
107542840,3chrt0-21swapper/321:08:183
107542840,3chrt0-21swapper/321:08:183
225682780,3chrt0-21swapper/000:33:170
181602760,2chrt0-21swapper/322:53:263
181602760,2chrt0-21swapper/322:53:263
181602760,2chrt0-21swapper/322:53:263
208852730,2sleep30-21swapper/323:03:153
208852730,2sleep30-21swapper/323:03:153
99982710,2chrt9999-21grep19:38:271
99982710,2chrt9999-21grep19:38:271
212202640,2chrt0-21swapper/320:08:243
212202640,2chrt0-21swapper/320:08:243
188302550,3chrt0-21swapper/100:23:171
188302550,3chrt0-21swapper/100:23:171
41472460,4sleep30-21swapper/323:43:233
41472460,4sleep30-21swapper/323:43:233
121462460,2sleep010682-40mandb00:03:310
121462460,2sleep010682-40mandb00:03:310
248822450,3sleep30-21swapper/321:48:033
248822450,3sleep30-21swapper/321:48:033
205962440,4sleep0171ktimers/020:08:170
205962440,4sleep0171ktimers/020:08:170
130262440,9sleep013029-21timerandwakeup21:13:250
130262440,9sleep013029-21timerandwakeup21:13:250
32155994136,3cyclictest0-21swapper/000:03:100
32155994136,3cyclictest0-21swapper/000:03:100
32155994136,3cyclictest0-21swapper/000:03:100
190212410,1sleep00-21swapper/020:03:230
190212410,1sleep00-21swapper/020:03:230
190212410,1sleep00-21swapper/020:03:230
3215799382,28cyclictest11615-21ntp_states00:03:231
3215799382,28cyclictest11615-21ntp_states00:03:231
293182380,3sleep30-21swapper/321:58:193
293182380,3sleep30-21swapper/321:58:193
293182380,3sleep30-21swapper/321:58:193
244282370,3sleep20-21swapper/200:38:182
244282370,3sleep20-21swapper/200:38:182
88542360,3sleep30-21swapper/323:58:103
88542360,3sleep30-21swapper/323:58:103
88542360,3sleep30-21swapper/323:58:103
55552360,3sleep032154-21cyclictest19:28:170
55552360,3sleep032154-21cyclictest19:28:170
55552360,3sleep032154-21cyclictest19:28:170
32155993629,5cyclictest550-21cron21:43:030
32155993629,5cyclictest550-21cron21:43:030
229062360,3sleep00-21swapper/023:08:150
229062360,3sleep00-21swapper/023:08:150
229062360,3sleep00-21swapper/023:08:150
134372360,2sleep013440-21fschecks_count22:43:140
134372360,2sleep013440-21fschecks_count22:43:140
134372360,2sleep013440-21fschecks_count22:43:140
179092350,5sleep2351rcuc/221:28:152
32155993428,4cyclictest2762-21fschecks_count20:48:160
32155993428,4cyclictest2762-21fschecks_count20:48:160
32157993324,7cyclictest25817-21fschecks_count20:23:161
32157993324,7cyclictest25817-21fschecks_count20:23:161
32155993327,5cyclictest20739-21unixbench_singl21:33:250
32155993327,5cyclictest20739-21unixbench_singl21:33:250
32155993326,5cyclictest7219-21fschecks_count23:53:150
32155993325,6cyclictest642-21kernelversion22:08:170
16732330,3sleep20-21swapper/223:38:162
16732330,3sleep20-21swapper/223:38:162
16732330,3sleep20-21swapper/223:38:162
3217799322,24cyclictest10739-21chrt00:03:093
3217799322,24cyclictest10739-21chrt00:03:093
3217799322,24cyclictest10739-21chrt00:03:093
32157993228,3cyclictest32154-21cyclictest22:43:021
32157993228,3cyclictest32154-21cyclictest22:43:021
32157993228,3cyclictest32154-21cyclictest22:43:021
32155993225,5cyclictest11903-21kernelversion22:38:170
263312320,2sleep026363-21latency_hist23:18:130
263312320,2sleep026363-21latency_hist23:18:130
32157993122,7cyclictest18374-21fschecks_count20:03:151
32157993122,7cyclictest18374-21fschecks_count20:03:151
32157993122,7cyclictest18374-21fschecks_count20:03:151
32155993125,5cyclictest22047-21/usr/sbin/munin00:33:210
32155993125,5cyclictest22047-21/usr/sbin/munin00:33:210
32155993125,4cyclictest31377-21cut20:38:150
32155993124,5cyclictest32154-21cyclictest23:58:040
32155993124,5cyclictest32154-21cyclictest23:58:040
32155993124,5cyclictest32154-21cyclictest23:58:040
32155993123,5cyclictest674-21gpm20:23:090
32155993123,5cyclictest674-21gpm20:23:090
299142310,3sleep10-21swapper/120:33:181
299142310,3sleep10-21swapper/120:33:181
3216799302,19cyclictest12163-21chrt00:03:442
3216799302,19cyclictest12163-21chrt00:03:442
32157993022,6cyclictest11992-21sshd19:43:441
32157993022,6cyclictest11992-21sshd19:43:441
32157993021,7cyclictest23426-21fschecks_count21:43:151
32157993021,7cyclictest23426-21fschecks_count21:43:151
32155993024,4cyclictest19298-21latency_hist21:33:030
32155993024,4cyclictest19298-21latency_hist21:33:030
32155993024,4cyclictest1346-21avahi-daemon22:13:240
32155993024,4cyclictest1346-21avahi-daemon22:13:240
32155993024,4cyclictest1346-21avahi-daemon22:13:240
32155993023,5cyclictest31736-21threads22:03:260
32155993023,5cyclictest31736-21threads22:03:260
32155993023,5cyclictest29892-21kernelversion20:33:180
32155993022,6cyclictest21999-21latency_hist00:33:100
32155993022,6cyclictest21999-21latency_hist00:33:100
315202300,3sleep30-21swapper/323:29:443
315202300,3sleep30-21swapper/323:29:443
32157992923,4cyclictest25417-21cron20:23:031
32157992923,4cyclictest25417-21cron20:23:031
32157992922,6cyclictest32154-21cyclictest19:28:131
32157992922,6cyclictest32154-21cyclictest19:28:131
32155992923,4cyclictest17967-21sendmail20:03:020
32155992923,4cyclictest17967-21sendmail20:03:020
32155992922,5cyclictest8413-21cut21:03:150
32155992922,5cyclictest8413-21cut21:03:150
32155992912,5cyclictest60-21kcompactd023:26:500
32155992912,5cyclictest60-21kcompactd023:26:500
32155992912,5cyclictest60-21kcompactd023:26:500
170492290,1sleep117085-21proc_pri19:58:221
170492290,1sleep117085-21proc_pri19:58:221
10372290,3sleep20-21swapper/220:43:162
10372290,3sleep20-21swapper/220:43:162
10372290,3sleep20-21swapper/220:43:162
64132280,2sleep30-21swapper/322:23:193
64132280,2sleep30-21swapper/322:23:193
32157992824,3cyclictest6701-21latency_hist19:33:031
32157992824,3cyclictest6701-21latency_hist19:33:031
32157992824,3cyclictest6701-21latency_hist19:33:031
32157992823,4cyclictest6152-21gltestperf22:23:171
32157992823,4cyclictest6152-21gltestperf22:23:171
32157992822,4cyclictest7969-21cron21:03:031
32157992821,5cyclictest14041-21fschecks_count21:18:171
32157992821,5cyclictest14041-21fschecks_count21:18:171
32157992819,6cyclictest2192-21ntp_offset23:38:221
32157992819,6cyclictest2192-21ntp_offset23:38:221
32157992819,6cyclictest2192-21ntp_offset23:38:221
32155992822,4cyclictest32154-21cyclictest19:48:390
32155992822,4cyclictest32154-21cyclictest19:48:390
32155992821,5cyclictest11770-21sh21:13:040
32155992821,5cyclictest11770-21sh21:13:040
32155992821,5cyclictest11770-21sh21:13:040
32155992820,6cyclictest4903-21threads22:18:270
32155992820,6cyclictest4903-21threads22:18:270
32155992820,6cyclictest4903-21threads22:18:270
32155992819,7cyclictest5104-21latency_hist19:28:130
32155992819,7cyclictest5104-21latency_hist19:28:130
320542280,2sleep20-21swapper/223:33:132
320542280,2sleep20-21swapper/223:33:132
231052280,3sleep30-21swapper/300:33:243
231052280,3sleep30-21swapper/300:33:243
32157992722,4cyclictest749-21runrttasks22:05:331
32157992722,4cyclictest749-21runrttasks22:05:331
32157992722,4cyclictest749-21runrttasks22:05:331
32157992722,3cyclictest0-21swapper/121:06:011
32157992722,3cyclictest0-21swapper/121:06:011
32157992721,4cyclictest22611-21unixbench_singl21:38:261
32157992721,4cyclictest22611-21unixbench_singl21:38:261
32157992720,5cyclictest24148-21cut00:38:161
32157992718,7cyclictest27686-21tr21:53:211
32157992718,7cyclictest27686-21tr21:53:211
32155992723,2cyclictest0-21swapper/000:23:020
32155992723,2cyclictest0-21swapper/000:23:020
32155992723,2cyclictest0-21swapper/000:23:020
32155992721,5cyclictest27711-21fschecks_count20:28:140
32155992721,4cyclictest25497-21sshd00:41:530
32155992721,4cyclictest25497-21sshd00:41:530
32155992721,4cyclictest0-21swapper/019:38:160
32155992721,4cyclictest0-21swapper/019:38:160
32155992721,3cyclictest0-21swapper/000:13:230
32155992721,3cyclictest0-21swapper/000:13:230
32155992719,6cyclictest446-21grep23:33:230
32155992719,6cyclictest446-21grep23:33:230
32155992719,6cyclictest446-21grep23:33:230
32155992719,6cyclictest23252-21/usr/sbin/munin21:43:210
32155992719,6cyclictest23252-21/usr/sbin/munin21:43:210
32155992719,6cyclictest11632-21fschecks_time22:38:150
32155992719,6cyclictest11632-21fschecks_time22:38:150
32155992718,7cyclictest7505-21latency_hist22:28:060
32155992718,7cyclictest7505-21latency_hist22:28:060
32177992621,3cyclictest431rcuc/319:28:163
32177992621,3cyclictest431rcuc/319:28:163
32177992621,3cyclictest431rcuc/319:28:163
32157992621,4cyclictest23401-21ntp_offset23:08:201
32157992621,4cyclictest23401-21ntp_offset23:08:201
32157992621,4cyclictest23401-21ntp_offset23:08:201
32157992621,4cyclictest23401-21ntp_offset23:08:201
32157992621,4cyclictest21044-21fschecks_time23:03:161
32157992620,5cyclictest3321-21ntp_offset20:48:221
32157992620,5cyclictest3321-21ntp_offset20:48:221
32157992620,5cyclictest10843-21ntp_offset21:08:211
32157992620,5cyclictest10843-21ntp_offset21:08:211
32157992620,5cyclictest10843-21ntp_offset21:08:211
32157992620,4cyclictest3892-21latency23:43:181
32157992620,4cyclictest3892-21latency23:43:181
32157992620,4cyclictest20658-21if_eth000:28:171
32157992620,4cyclictest20658-21if_eth000:28:171
32157992620,4cyclictest20658-21if_eth000:28:171
32157992619,6cyclictest31591-21gltestperf20:38:161
32157992619,6cyclictest31591-21gltestperf20:38:161
32155992622,2cyclictest0-21swapper/022:08:230
32155992622,2cyclictest0-21swapper/022:08:230
32155992622,2cyclictest0-21swapper/022:08:230
32155992621,3cyclictest0-21swapper/021:28:130
32155992621,3cyclictest0-21swapper/021:28:130
32155992621,3cyclictest0-21swapper/000:08:210
32155992621,3cyclictest0-21swapper/000:08:210
32155992619,5cyclictest24513-21cpuspeed23:13:130
32155992619,5cyclictest24513-21cpuspeed23:13:130
32155992619,5cyclictest21960-21sshd20:13:140
32155992619,5cyclictest21960-21sshd20:13:140
32155992619,5cyclictest21960-21sshd20:13:140
32155992615,9cyclictest1833-21idleruntime23:38:170
32155992615,9cyclictest1833-21idleruntime23:38:170
32155992615,9cyclictest1833-21idleruntime23:38:170
32155992614,11cyclictest60-21kcompactd022:53:160
32155992614,11cyclictest60-21kcompactd022:53:160
32155992614,11cyclictest60-21kcompactd022:53:160
32155992613,3cyclictest181rcu_preempt22:54:590
32155992613,3cyclictest181rcu_preempt22:54:590
32155992613,3cyclictest181rcu_preempt22:54:590
32157992521,2cyclictest0-21swapper/120:43:151
32157992521,2cyclictest0-21swapper/120:43:151
32157992521,2cyclictest0-21swapper/120:43:151
32157992520,4cyclictest10234-21needreboot22:33:211
32157992520,4cyclictest10234-21needreboot22:33:211
32157992520,4cyclictest10234-21needreboot22:33:211
32157992520,3cyclictest0-21swapper/122:48:151
32157992520,3cyclictest0-21swapper/122:48:151
32157992520,3cyclictest0-21swapper/122:48:151
32157992520,3cyclictest0-21swapper/100:03:031
32157992520,3cyclictest0-21swapper/100:03:031
32157992520,3cyclictest0-21swapper/100:03:031
32157992520,2cyclictest0-21swapper/123:33:151
32157992520,2cyclictest0-21swapper/123:33:151
32157992519,5cyclictest16976-21kernelversion00:18:181
32157992519,5cyclictest16976-21kernelversion00:18:181
32157992519,5cyclictest16976-21kernelversion00:18:181
32157992518,5cyclictest19642-21ntp_kernel_pll_22:58:211
32157992517,6cyclictest7911-21cut22:28:161
32157992517,6cyclictest7911-21cut22:28:161
32157992516,7cyclictest17838-21fschecks_time21:28:141
32155992520,4cyclictest27302-21sh20:28:140
32155992520,4cyclictest27302-21sh20:28:140
32155992520,4cyclictest27302-21sh20:28:140
32155992520,3cyclictest0-21swapper/021:53:190
32155992520,3cyclictest0-21swapper/021:53:190
32155992519,4cyclictest25433-21grep21:48:170
32155992519,4cyclictest25433-21grep21:48:170
32155992519,4cyclictest0-21swapper/020:58:210
32155992519,4cyclictest0-21swapper/020:38:230
32155992519,4cyclictest0-21swapper/020:38:230
32155992519,3cyclictest0-21swapper/023:43:230
32155992519,3cyclictest0-21swapper/023:43:230
32155992519,3cyclictest0-21swapper/023:18:190
32155992519,3cyclictest0-21swapper/023:18:190
32155992518,5cyclictest29328-21kernelversion21:58:190
32155992518,5cyclictest29328-21kernelversion21:58:190
32155992518,5cyclictest29328-21kernelversion21:58:190
32155992517,6cyclictest7812-21grep19:33:240
32155992517,6cyclictest7812-21grep19:33:240
32155992517,6cyclictest7812-21grep19:33:240
32155992512,10cyclictest171ktimers/020:28:000
112182250,4sleep30-21swapper/322:38:033
112182250,4sleep30-21swapper/322:38:033
112182250,4sleep30-21swapper/322:38:033
94752240,2sleep30-21swapper/319:38:213
94752240,2sleep30-21swapper/319:38:213
32157992419,3cyclictest0-21swapper/123:53:211
32157992419,3cyclictest0-21swapper/123:53:211
32157992419,3cyclictest0-21swapper/123:53:211
32157992419,3cyclictest0-21swapper/119:53:031
32157992419,3cyclictest0-21swapper/119:53:031
32157992418,5cyclictest24295-21gltestperf00:38:171
32157992418,5cyclictest24295-21gltestperf00:38:171
32157992418,5cyclictest18496-21rt-features21:28:221
32157992418,5cyclictest18496-21rt-features21:28:221
32157992418,4cyclictest0-21swapper/100:33:231
32157992417,5cyclictest872-21ntp_kernel_pll_22:08:201
32157992417,5cyclictest872-21ntp_kernel_pll_22:08:201
32157992417,5cyclictest872-21ntp_kernel_pll_22:08:201
32155992418,3cyclictest0-21swapper/019:43:180
32155992418,3cyclictest0-21swapper/019:43:180
32155992415,7cyclictest14678-21systemd-journal23:48:160
32155992415,7cyclictest14678-21systemd-journal23:48:160
32177992319,3cyclictest431rcuc/323:12:333
32177992319,3cyclictest431rcuc/323:12:333
32177992319,3cyclictest431rcuc/323:12:333
32177992319,3cyclictest431rcuc/323:12:333
32177992319,3cyclictest431rcuc/321:26:123
32177992319,3cyclictest431rcuc/321:26:123
32177992319,3cyclictest431rcuc/320:04:513
32177992319,3cyclictest431rcuc/320:04:513
32177992319,3cyclictest431rcuc/320:04:513
32157992318,2cyclictest0-21swapper/120:13:231
32157992318,2cyclictest0-21swapper/120:13:231
32157992318,2cyclictest0-21swapper/120:13:231
32157992317,5cyclictest16628-21fschecks_count00:18:151
32157992317,4cyclictest14588-21cpu00:13:131
32157992317,4cyclictest14588-21cpu00:13:131
32157992317,4cyclictest0-21swapper/121:33:141
32157992317,4cyclictest0-21swapper/121:33:141
32157992316,5cyclictest22865-21cut23:08:141
32157992316,5cyclictest22865-21cut23:08:141
32157992316,5cyclictest22865-21cut23:08:141
32157992316,5cyclictest0-21swapper/123:13:191
32157992316,5cyclictest0-21swapper/123:13:191
3215799231,13cyclictest4759-21grep20:53:161
3215799231,13cyclictest4759-21grep20:53:161
3215799231,13cyclictest4759-21grep20:53:161
32155992318,3cyclictest0-21swapper/023:28:190
32155992318,3cyclictest0-21swapper/023:28:190
32155992317,4cyclictest0-21swapper/021:18:270
32155992317,4cyclictest0-21swapper/021:18:270
32155992317,3cyclictest0-21swapper/022:28:230
32155992317,3cyclictest0-21swapper/022:28:230
272032230,3chrt0-21swapper/221:53:152
272032230,3chrt0-21swapper/221:53:152
32177992218,3cyclictest431rcuc/321:29:543
32177992218,3cyclictest431rcuc/321:29:543
32177992218,3cyclictest431rcuc/320:45:523
32177992218,3cyclictest431rcuc/320:45:523
32177992218,3cyclictest431rcuc/320:45:523
32177992218,3cyclictest431rcuc/320:23:493
32177992218,3cyclictest431rcuc/320:23:493
32177992218,3cyclictest431rcuc/319:59:573
32177992218,3cyclictest431rcuc/319:59:573
32177992218,3cyclictest431rcuc/300:18:253
32177992218,3cyclictest431rcuc/300:18:253
32177992218,3cyclictest431rcuc/300:18:253
32157992218,2cyclictest0-21swapper/123:33:221
32157992218,2cyclictest0-21swapper/123:33:221
32157992218,2cyclictest0-21swapper/123:33:221
32157992218,2cyclictest0-21swapper/122:53:161
32157992218,2cyclictest0-21swapper/122:53:161
32157992218,2cyclictest0-21swapper/122:53:161
32157992218,2cyclictest0-21swapper/122:18:031
32157992218,2cyclictest0-21swapper/122:18:031
32157992218,2cyclictest0-21swapper/122:18:031
32157992217,3cyclictest0-21swapper/121:13:231
32157992217,3cyclictest0-21swapper/121:13:231
32157992216,4cyclictest26268-21grep21:48:261
32157992216,4cyclictest26268-21grep21:48:261
32157992216,4cyclictest15346-21tr00:13:211
32157992216,4cyclictest0-21swapper/123:18:161
32157992216,4cyclictest0-21swapper/123:18:161
32157992216,4cyclictest0-21swapper/122:03:141
32157992216,4cyclictest0-21swapper/122:03:141
32157992216,4cyclictest0-21swapper/122:03:141
32157992216,3cyclictest0-21swapper/123:48:231
32157992216,3cyclictest0-21swapper/123:48:231
32157992216,3cyclictest0-21swapper/123:48:231
3215799221,12cyclictest27839-21grep20:28:141
3215799221,12cyclictest27839-21grep20:28:141
3215799221,12cyclictest27839-21grep20:28:141
32155992217,3cyclictest14789-21tail19:53:160
32155992217,3cyclictest14789-21tail19:53:160
32155992217,3cyclictest0-21swapper/022:48:150
32155992217,3cyclictest0-21swapper/022:48:150
32155992217,3cyclictest0-21swapper/022:48:150
32155992216,5cyclictest19925-21threads22:58:240
32155992216,5cyclictest19925-21threads22:58:240
32155992214,5cyclictest19388-21timerandwakeup00:23:250
32155992214,5cyclictest19388-21timerandwakeup00:23:250
155352220,3sleep30-21swapper/322:48:163
155352220,3sleep30-21swapper/322:48:163
155352220,3sleep30-21swapper/322:48:163
145542220,3sleep20-21swapper/222:43:242
145542220,3sleep20-21swapper/222:43:242
145542220,3sleep20-21swapper/222:43:242
3217799212,15cyclictest0-21swapper/300:03:193
3217799212,15cyclictest0-21swapper/300:03:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional