You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 05:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Sun Jun 28, 2026 00:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2267921200,2sleep122683-21gltestperf23:43:191
2267921200,2sleep122683-21gltestperf23:43:191
2351021100,2chrt23508-21memory22:18:200
2351021100,2chrt23508-21memory22:18:200
1800921020,2chrt18004-21strings22:03:240
1800921020,2chrt18004-21strings22:03:240
284002780,2sleep128401-21/usr/sbin/munin19:38:211
284002780,2sleep128401-21/usr/sbin/munin19:38:211
284002780,2sleep128401-21/usr/sbin/munin19:38:211
223542750,3chrt0-21swapper/119:23:181
223542750,3chrt0-21swapper/119:23:181
174312650,1sleep116721-21/usr/sbin/munin23:28:241
174312650,1sleep116721-21/usr/sbin/munin23:28:241
174312650,1sleep116721-21/usr/sbin/munin23:28:241
51012610,3chrt5100-21sed20:03:181
51012610,3chrt5100-21sed20:03:181
52722590,2sleep20-21swapper/222:58:052
52722590,2sleep20-21swapper/222:58:052
205592560,1sleep1271rcuc/120:43:261
205592560,1sleep1271rcuc/120:43:261
205592560,1sleep1271rcuc/120:43:261
29132530,3sleep20-21swapper/222:48:292
29132530,3sleep20-21swapper/222:48:292
29132530,3sleep20-21swapper/222:48:292
234272510,3sleep20-21swapper/223:43:282
234272510,3sleep20-21swapper/223:43:282
18592995033,14cyclictest0-21swapper/000:03:120
18592995033,14cyclictest0-21swapper/000:03:120
18592995033,14cyclictest0-21swapper/000:03:120
20492480,3sleep20-21swapper/221:22:502
20492480,3sleep20-21swapper/221:22:502
100252470,4sleep19209-21/usr/sbin/munin23:08:251
100252470,4sleep19209-21/usr/sbin/munin23:08:251
4922450,3sleep20-21swapper/222:43:222
4922450,3sleep20-21swapper/222:43:222
4922450,3sleep20-21swapper/222:43:222
106272450,3sleep20-21swapper/220:18:182
273442430,3sleep20-21swapper/222:28:232
273442430,3sleep20-21swapper/222:28:232
160902430,1chrt15799-21cron20:33:161
160902430,1chrt15799-21cron20:33:161
18593993831,5cyclictest18586-21cyclictest00:03:191
18593993831,5cyclictest18586-21cyclictest00:03:191
18593993831,5cyclictest18586-21cyclictest00:03:191
18593993831,5cyclictest10787-21fschecks_time00:38:181
18593993831,5cyclictest10787-21fschecks_time00:38:181
1859299382,28cyclictest30340-21irqstats00:03:220
1859299382,28cyclictest30340-21irqstats00:03:220
1859299382,28cyclictest30340-21irqstats00:03:220
115622380,4sleep20-21swapper/200:38:272
115622380,4sleep20-21swapper/200:38:272
48492370,2sleep10-21swapper/121:28:251
48492370,2sleep10-21swapper/121:28:251
48492370,2sleep10-21swapper/121:28:251
252702370,3sleep20-21swapper/222:23:202
252702370,3sleep20-21swapper/222:23:202
67142360,3sleep30-21swapper/300:28:153
67142360,3sleep30-21swapper/300:28:153
233952360,2sleep30-21swapper/322:18:193
233952360,2sleep30-21swapper/322:18:193
233952360,2sleep30-21swapper/322:18:193
18592993629,5cyclictest29487-21sendmail19:43:050
18592993629,5cyclictest29487-21sendmail19:43:050
18592993629,5cyclictest29487-21sendmail19:43:050
41012350,4chrt4104-21sed22:53:221
41012350,4chrt4104-21sed22:53:221
148572350,2sleep2351rcuc/221:53:292
148572350,2sleep2351rcuc/221:53:292
50212340,2sleep05099-21gltestperf20:03:180
50212340,2sleep05099-21gltestperf20:03:180
18593993327,4cyclictest547-21rs:main1
18593993327,4cyclictest547-21rs:main1
18593993325,6cyclictest15341-21unixbench_multi20:28:281
18593993325,6cyclictest15341-21unixbench_multi20:28:281
18593993322,9cyclictest30849-21latency_hist21:13:091
18593993322,9cyclictest30849-21latency_hist21:13:091
18592993327,4cyclictest8254-21gltestperf21:38:190
18593993227,3cyclictest0-21swapper/100:03:261
18593993227,3cyclictest0-21swapper/100:03:261
18593993227,3cyclictest0-21swapper/100:03:261
18593993223,7cyclictest10543-21netstat21:43:241
18593993223,7cyclictest10543-21netstat21:43:241
18593993223,7cyclictest10543-21netstat21:43:241
143612320,5sleep20-21swapper/223:19:282
143612320,5sleep20-21swapper/223:19:282
18593993126,3cyclictest10746-21gltestperf20:18:191
1859499301,23cyclictest29755-21taskset00:03:142
1859499301,23cyclictest29755-21taskset00:03:142
1859499301,23cyclictest29755-21taskset00:03:142
18593993026,3cyclictest3364-21cut00:18:201
18593993026,3cyclictest3364-21cut00:18:201
18593993025,3cyclictest15227-21cron21:58:051
18593993025,3cyclictest15227-21cron21:58:051
18593993023,5cyclictest896-21latency_hist19:53:121
18593993023,5cyclictest896-21latency_hist19:53:121
18593993023,5cyclictest896-21latency_hist19:53:121
18593993023,5cyclictest16918-21sshd22:00:531
18593993023,5cyclictest16918-21sshd22:00:531
18593993022,6cyclictest7080-21threads21:33:271
18592993025,3cyclictest18586-21cyclictest21:58:160
18592993025,3cyclictest18586-21cyclictest21:58:160
18592993023,5cyclictest29850-21ntp_offset21:08:250
18592993023,5cyclictest29850-21ntp_offset21:08:250
18592993023,5cyclictest0-21swapper/020:03:060
18592993023,5cyclictest0-21swapper/020:03:060
18592993022,6cyclictest2747-21threads22:48:280
18592993022,6cyclictest2747-21threads22:48:280
18592993022,6cyclictest2747-21threads22:48:280
18592993021,7cyclictest12524-21fschecks_count00:43:180
18592993021,7cyclictest12524-21fschecks_count00:43:180
18593992924,3cyclictest19593-21idleruntime22:08:201
18593992924,3cyclictest19593-21idleruntime22:08:201
18593992924,3cyclictest19593-21idleruntime22:08:201
18593992923,5cyclictest27388-21netstat22:28:241
18593992923,5cyclictest27388-21netstat22:28:241
18593992922,5cyclictest6445-21switchtime22:58:281
18593992922,5cyclictest6445-21switchtime22:58:281
18593992922,5cyclictest6445-21switchtime22:58:281
18593992920,2cyclictest181rcu_preempt23:13:271
18593992920,2cyclictest181rcu_preempt23:13:271
18593992915,11cyclictest12745-21timerandwakeup21:48:281
18593992915,11cyclictest12745-21timerandwakeup21:48:281
18592992924,3cyclictest18586-21cyclictest21:24:460
18592992924,3cyclictest18586-21cyclictest21:24:460
18592992924,3cyclictest18586-21cyclictest21:24:460
18592992920,7cyclictest24726-21latency_hist22:23:120
18592992920,7cyclictest24726-21latency_hist22:23:120
18592992920,7cyclictest21312-21fschecks_time22:13:180
18592992920,7cyclictest21312-21fschecks_time22:13:180
18593992824,3cyclictest18586-21cyclictest00:08:281
18593992820,6cyclictest7644-21threads20:08:281
18593992820,6cyclictest7644-21threads20:08:281
18593992820,6cyclictest7644-21threads20:08:281
18593992820,6cyclictest15829-21grep23:23:261
18593992820,6cyclictest15829-21grep23:23:261
18593992820,6cyclictest15829-21grep23:23:261
18593992820,6cyclictest1148-21/usr/sbin/munin00:13:261
18593992820,6cyclictest1148-21/usr/sbin/munin00:13:261
18592992823,3cyclictest10317-21cron00:38:050
18592992823,3cyclictest10317-21cron00:38:050
18592992822,4cyclictest0-21swapper/023:48:300
18592992822,4cyclictest0-21swapper/023:48:300
18592992822,4cyclictest0-21swapper/023:48:300
18592992821,5cyclictest30925-21unixbench_singl19:43:290
18592992821,5cyclictest30925-21unixbench_singl19:43:290
18592992820,6cyclictest3869-21fschecks_time22:53:200
18592992820,6cyclictest3869-21fschecks_time22:53:200
18592992816,10cyclictest0-21swapper/020:43:190
18592992816,10cyclictest0-21swapper/020:43:190
18592992816,10cyclictest0-21swapper/020:43:190
18593992722,3cyclictest23411-21kworker/u9:1-xprtiod19:33:171
18593992722,3cyclictest0-21swapper/121:38:291
18593992722,3cyclictest0-21swapper/120:53:051
18593992722,3cyclictest0-21swapper/120:53:051
18593992722,3cyclictest0-21swapper/119:43:281
18593992722,3cyclictest0-21swapper/119:43:281
18593992722,2cyclictest0-21swapper/100:43:051
18593992722,2cyclictest0-21swapper/100:43:051
18593992720,5cyclictest18586-21cyclictest23:35:451
18593992720,5cyclictest18586-21cyclictest23:35:451
18593992720,5cyclictest18586-21cyclictest23:35:451
18593992720,5cyclictest18586-21cyclictest22:18:151
18593992720,5cyclictest18586-21cyclictest22:18:151
18593992720,5cyclictest18586-21cyclictest22:18:151
18593992719,6cyclictest29042-21kernelversion22:33:181
18593992719,6cyclictest29042-21kernelversion22:33:181
18592992722,3cyclictest32083-21sendmail22:43:050
18592992721,5cyclictest7149-21sendmail23:03:050
18592992721,5cyclictest7149-21sendmail23:03:050
18592992721,5cyclictest7149-21sendmail23:03:050
18592992721,4cyclictest8608-21unixbench_singl23:03:290
18592992721,4cyclictest8608-21unixbench_singl23:03:290
1859299271,10cyclictest11692ntp_states21:18:270
1859299271,10cyclictest11692ntp_states21:18:270
18593992620,5cyclictest31727-21ntp_offset21:13:231
18593992620,5cyclictest31727-21ntp_offset21:13:231
18593992620,5cyclictest31727-21ntp_offset21:13:231
18593992620,5cyclictest18904-21threads20:38:271
18593992620,4cyclictest0-21swapper/122:23:271
18593992620,4cyclictest0-21swapper/122:23:271
18593992619,5cyclictest3749-21sshd21:25:281
18593992619,5cyclictest3749-21sshd21:25:281
18593992619,5cyclictest3749-21sshd21:25:281
18593992619,5cyclictest30934-21latency22:38:201
18592992621,4cyclictest1-21systemd23:08:280
18592992621,4cyclictest1-21systemd23:08:280
18592992620,4cyclictest15055-21grep23:23:180
18592992620,4cyclictest15055-21grep23:23:180
18592992620,4cyclictest0-21swapper/021:13:230
18592992620,4cyclictest0-21swapper/021:13:230
18592992620,4cyclictest0-21swapper/021:13:230
18592992619,5cyclictest23445-21sort20:53:140
18592992619,5cyclictest23445-21sort20:53:140
18592992619,5cyclictest23099-21fschecks_time22:18:160
18592992618,6cyclictest9023-21apache200:26:300
18592992618,6cyclictest9023-21apache200:26:300
18592992612,4cyclictest181rcu_preempt20:13:050
18592992612,4cyclictest181rcu_preempt20:13:050
18592992612,4cyclictest181rcu_preempt20:13:050
11702260,2sleep11169-21ntp_states21:18:271
11702260,2sleep11169-21ntp_states21:18:271
305802250,2sleep20-21swapper/200:03:262
305802250,2sleep20-21swapper/200:03:262
305802250,2sleep20-21swapper/200:03:262
18593992521,2cyclictest0-21swapper/122:18:241
18593992521,2cyclictest0-21swapper/122:18:241
18593992520,3cyclictest0-21swapper/120:23:241
18593992520,3cyclictest0-21swapper/120:23:241
18593992519,5cyclictest21200-21netstat23:38:241
18593992519,5cyclictest21200-21netstat23:38:241
18593992518,5cyclictest8655-21cut20:13:171
18593992518,5cyclictest8655-21cut20:13:171
18593992518,5cyclictest8655-21cut20:13:171
18593992518,5cyclictest7548-21fschecks_count23:03:191
18593992518,5cyclictest7548-21fschecks_count23:03:191
18592992521,3cyclictest6311-21fschecks_time21:33:170
18592992521,3cyclictest6311-21fschecks_time21:33:170
18592992521,2cyclictest0-21swapper/000:13:240
18592992521,2cyclictest0-21swapper/000:13:240
18592992520,3cyclictest0-21swapper/023:33:240
18592992520,3cyclictest0-21swapper/023:33:240
18592992520,3cyclictest0-21swapper/023:33:240
18592992520,3cyclictest0-21swapper/020:33:060
18592992520,3cyclictest0-21swapper/020:33:060
18592992519,4cyclictest17776-21xargs23:28:280
18592992519,4cyclictest17776-21xargs23:28:280
18592992519,4cyclictest17776-21xargs23:28:280
18592992519,4cyclictest0-21swapper/023:43:190
18592992519,4cyclictest0-21swapper/023:43:190
18592992519,4cyclictest0-21swapper/020:43:050
18592992518,3cyclictest0-21swapper/023:53:290
18592992518,3cyclictest0-21swapper/023:53:290
18592992518,3cyclictest0-21swapper/023:53:290
58722240,3sleep20-21swapper/220:03:272
58722240,3sleep20-21swapper/220:03:272
18593992420,2cyclictest0-21swapper/123:53:181
18593992420,2cyclictest0-21swapper/123:53:181
18593992420,2cyclictest0-21swapper/123:53:181
18593992419,3cyclictest0-21swapper/122:48:241
18593992419,3cyclictest0-21swapper/122:48:241
18593992419,3cyclictest0-21swapper/122:48:241
18593992418,4cyclictest0-21swapper/120:00:331
18593992418,4cyclictest0-21swapper/120:00:331
18593992418,3cyclictest0-21swapper/120:53:261
18593992418,3cyclictest0-21swapper/120:53:261
18593992418,3cyclictest0-21swapper/120:53:261
18593992412,10cyclictest14678-21systemd-journal00:28:061
18593992412,10cyclictest14678-21systemd-journal00:28:061
1859299248,3cyclictest700-21192.168.115.40-manager00:09:110
18592992420,3cyclictest5138-21cat21:28:290
18592992419,3cyclictest25995-21cat19:33:150
18592992419,3cyclictest25995-21cat19:33:150
18592992419,3cyclictest15751-21grep21:58:190
18592992419,3cyclictest15751-21grep21:58:190
18592992418,5cyclictest12852-21latency_hist23:18:110
18592992418,5cyclictest12852-21latency_hist23:18:110
18592992418,4cyclictest7395-21missed_timers00:28:230
18592992418,4cyclictest7395-21missed_timers00:28:230
18592992418,4cyclictest7395-21missed_timers00:28:230
18592992418,4cyclictest0-21swapper/020:13:230
18592992418,4cyclictest0-21swapper/020:13:230
18592992418,4cyclictest0-21swapper/020:13:230
18592992418,4cyclictest0-21swapper/019:35:290
18592992418,4cyclictest0-21swapper/019:35:290
18592992418,3cyclictest0-21swapper/020:58:190
18592992418,3cyclictest0-21swapper/020:58:190
18592992418,3cyclictest0-21swapper/020:33:260
18592992418,3cyclictest0-21swapper/020:33:260
18592992418,3cyclictest0-21swapper/020:23:210
18592992418,3cyclictest0-21swapper/020:23:210
18592992417,5cyclictest0-21swapper/022:43:190
18592992417,5cyclictest0-21swapper/022:43:190
18592992417,5cyclictest0-21swapper/022:43:190
18592992417,4cyclictest0-21swapper/022:08:280
1859799235,10cyclictest0-21swapper/300:03:143
1859799235,10cyclictest0-21swapper/300:03:143
1859799235,10cyclictest0-21swapper/300:03:143
18594992319,3cyclictest351rcuc/220:37:042
18594992319,3cyclictest351rcuc/220:37:042
18594992319,3cyclictest351rcuc/219:28:302
18594992319,3cyclictest351rcuc/219:28:302
18593992318,3cyclictest0-21swapper/122:03:281
18593992318,3cyclictest0-21swapper/122:03:281
18593992318,3cyclictest0-21swapper/119:53:281
18593992318,3cyclictest0-21swapper/119:53:281
18593992317,4cyclictest18586-21cyclictest23:53:071
18593992317,4cyclictest18586-21cyclictest23:53:071
18593992317,4cyclictest18586-21cyclictest23:53:071
18593992315,6cyclictest26908-21ls19:33:251
18593992315,6cyclictest26908-21ls19:33:251
18593992314,7cyclictest508-21cut22:43:231
18593992314,7cyclictest508-21cut22:43:231
18593992314,7cyclictest508-21cut22:43:231
18593992314,7cyclictest24895-21processes19:28:261
18592992319,2cyclictest0-21swapper/021:03:280
18592992319,2cyclictest0-21swapper/021:03:280
18592992319,2cyclictest0-21swapper/021:03:280
18592992318,4cyclictest6012-21/usr/sbin/munin21:33:210
18592992318,3cyclictest21686-21tail23:38:300
18592992318,3cyclictest21686-21tail23:38:300
18592992317,4cyclictest29450-21mailstats22:33:240
18592992317,4cyclictest29450-21mailstats22:33:240
18592992317,4cyclictest0-21swapper/020:53:260
18592992317,4cyclictest0-21swapper/020:53:260
18592992317,4cyclictest0-21swapper/020:53:260
18592992317,3cyclictest0-21swapper/022:28:250
18592992317,3cyclictest0-21swapper/022:28:250
18592992316,4cyclictest0-21swapper/021:48:280
18592992316,4cyclictest0-21swapper/021:48:280
18592992312,3cyclictest0-21swapper/000:18:300
18592992312,3cyclictest0-21swapper/000:18:300
1859799221,12cyclictest0-21swapper/321:38:293
18593992218,2cyclictest0-21swapper/123:18:241
18593992218,2cyclictest0-21swapper/123:18:241
18593992217,3cyclictest0-21swapper/121:03:261
18593992217,3cyclictest0-21swapper/121:03:261
18593992217,3cyclictest0-21swapper/121:03:261
18593992217,3cyclictest0-21swapper/100:28:231
18593992217,3cyclictest0-21swapper/100:28:231
18593992217,3cyclictest0-21swapper/100:28:231
18592992218,3cyclictest26472-21basename22:28:060
18592992218,3cyclictest26472-21basename22:28:060
18592992217,3cyclictest0-21swapper/019:48:280
18592992217,3cyclictest0-21swapper/019:48:280
18592992217,3cyclictest0-21swapper/019:48:280
18592992216,4cyclictest0-21swapper/019:53:180
18592992216,4cyclictest0-21swapper/019:53:180
11872220,2chrt0-21swapper/200:13:192
11872220,2chrt0-21swapper/200:13:192
322542210,3sleep20-21swapper/221:13:292
322542210,3sleep20-21swapper/221:13:292
322542210,3sleep20-21swapper/221:13:292
18592992116,3cyclictest0-21swapper/023:23:280
18592992116,3cyclictest0-21swapper/023:23:280
18592992116,3cyclictest0-21swapper/023:23:280
18592992112,7cyclictest9978-21fschecks_count21:43:180
18592992112,7cyclictest9978-21fschecks_count21:43:180
18592992112,7cyclictest9978-21fschecks_count21:43:180
1859299211,10cyclictest22786-21ls19:23:210
1859299211,10cyclictest22786-21ls19:23:210
1859799203,9cyclictest0-21swapper/320:18:283
18594992015,3cyclictest351rcuc/220:58:232
18594992015,3cyclictest351rcuc/220:58:232
18592992013,5cyclictest11434-21ls20:18:280
1859799193,9cyclictest0-21swapper/321:35:293
1859799183,9cyclictest0-21swapper/320:15:283
1859799183,9cyclictest0-21swapper/320:15:283
1859799183,9cyclictest0-21swapper/320:15:283
1859799182,11cyclictest13530-21chrt23:18:203
1859799182,11cyclictest13530-21chrt23:18:203
18597991811,4cyclictest146742sleep320:28:203
18597991811,4cyclictest146742sleep320:28:203
1859799181,12cyclictest30791-21chrt21:13:063
1859799181,12cyclictest30791-21chrt21:13:063
18594991814,3cyclictest351rcuc/223:02:282
18594991814,3cyclictest351rcuc/223:02:282
18594991814,3cyclictest351rcuc/223:02:282
18594991814,3cyclictest351rcuc/221:13:102
18594991814,3cyclictest351rcuc/221:13:102
18594991814,3cyclictest351rcuc/219:42:362
18594991814,3cyclictest351rcuc/219:42:362
18594991814,3cyclictest351rcuc/219:42:362
18594991814,3cyclictest351rcuc/200:10:182
1859799173,10cyclictest0-21swapper/319:53:283
1859799173,10cyclictest0-21swapper/319:53:283
1859799172,9cyclictest0-21swapper/320:44:283
1859799172,9cyclictest0-21swapper/320:44:283
1859799172,9cyclictest0-21swapper/320:44:283
1859799171,12cyclictest0-21swapper/323:43:223
1859799171,12cyclictest0-21swapper/323:43:223
1859799171,11cyclictest32461-21taskset00:08:253
1859799171,11cyclictest15605-21chrt23:23:233
1859799171,11cyclictest15605-21chrt23:23:233
1859799171,11cyclictest15605-21chrt23:23:233
1859799171,10cyclictest0-21swapper/320:58:213
1859799171,10cyclictest0-21swapper/320:58:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional