You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 19:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Sun Jun 28, 2026 12:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1396621000,5sleep113967-21latency_hist10:53:111
1396621000,5sleep113967-21latency_hist10:53:111
154032870,3sleep015404-21grep08:03:170
154032870,3sleep015404-21grep08:03:170
241362820,2sleep10-21swapper/111:18:231
241362820,2sleep10-21swapper/111:18:231
292532790,3chrt0-21swapper/311:33:213
292532790,3chrt0-21swapper/311:33:213
244232760,2sleep00-21swapper/008:28:100
244232760,2sleep00-21swapper/008:28:100
129982730,2chrt12996-21fschecks_time09:23:170
129982730,2chrt12996-21fschecks_time09:23:170
216482720,4sleep10-21swapper/112:38:231
216482720,4sleep10-21swapper/112:38:231
216482720,4sleep10-21swapper/112:38:231
270922690,3chrt0-21swapper/308:33:233
270922690,3chrt0-21swapper/308:33:233
247302640,3sleep30-21swapper/309:53:233
247302640,3sleep30-21swapper/309:53:233
247302640,3sleep30-21swapper/309:53:233
260362620,3sleep30-21swapper/311:23:243
260362620,3sleep30-21swapper/311:23:243
34922610,2sleep30-21swapper/310:23:223
34922610,2sleep30-21swapper/310:23:223
34922610,2sleep30-21swapper/310:23:223
309112600,6sleep030914-21ntp_kernel_pll_08:43:250
309112600,6sleep030914-21ntp_kernel_pll_08:43:250
39722580,3sleep30-21swapper/307:33:183
31002550,2sleep20-21swapper/211:48:272
31002550,2sleep20-21swapper/211:48:272
219222510,3chrt0-21swapper/308:18:283
194712500,1sleep3431rcuc/308:13:213
194712500,1sleep3431rcuc/308:13:213
40832490,2sleep14082-21ntp_kernel_pll_08:58:261
40832490,2sleep14082-21ntp_kernel_pll_08:58:261
40832490,2sleep14082-21ntp_kernel_pll_08:58:261
39832460,3sleep30-21swapper/308:58:243
39832460,3sleep30-21swapper/308:58:243
39832460,3sleep30-21swapper/308:58:243
23212460,1sleep10-21swapper/111:48:181
297592440,1sleep10-21swapper/111:33:271
297592440,1sleep10-21swapper/111:33:271
290542440,3sleep30-21swapper/308:38:303
273102420,3sleep30-21swapper/310:00:313
273102420,3sleep30-21swapper/310:00:313
274102400,2sleep026955-21latency_hist11:28:200
232732400,3sleep30-21swapper/311:18:093
232732400,3sleep30-21swapper/311:18:093
232732400,3sleep30-21swapper/311:18:093
209492390,2sleep10-21swapper/108:18:181
28922993831,5cyclictest27478-21gltestperf11:28:201
188982370,1sleep018999-21memory09:38:220
188982370,1sleep018999-21memory09:38:220
2892299357,5cyclictest281ktimers/108:40:321
28922993531,3cyclictest90-1pr/ttyS011:30:371
28922993531,3cyclictest90-1pr/ttyS011:30:371
28922993531,3cyclictest90-1pr/ttyS009:14:331
28922993531,3cyclictest90-1pr/ttyS009:14:331
28922993530,3cyclictest90-1pr/ttyS008:55:331
28922993530,3cyclictest90-1pr/ttyS008:55:331
28922993530,3cyclictest90-1pr/ttyS008:55:331
28922993527,6cyclictest11955-21kernelversion07:53:201
28922993527,6cyclictest11955-21kernelversion07:53:201
151402350,2sleep115344-21unixbench_singl10:53:281
151402350,2sleep115344-21unixbench_singl10:53:281
151402350,2sleep115344-21unixbench_singl10:53:281
139042350,3sleep30-21swapper/312:18:193
139042350,3sleep30-21swapper/312:18:193
139042350,3sleep30-21swapper/312:18:193
28922993430,3cyclictest90-1pr/ttyS011:01:061
28922993430,3cyclictest90-1pr/ttyS011:01:061
28922993430,3cyclictest90-1pr/ttyS011:01:061
275302340,1sleep10-21swapper/108:33:291
275302340,1sleep10-21swapper/108:33:291
267412340,2sleep026745-21tail08:33:200
267412340,2sleep026745-21tail08:33:200
28922993330,2cyclictest90-1pr/ttyS009:56:341
28922993330,2cyclictest90-1pr/ttyS009:56:341
28922993330,2cyclictest90-1pr/ttyS009:56:341
28922993328,3cyclictest90-1pr/ttyS009:52:341
28922993328,3cyclictest90-1pr/ttyS009:52:341
28922993328,3cyclictest90-1pr/ttyS009:52:341
28922993324,7cyclictest31613-21netstat11:38:231
28922993324,7cyclictest31613-21netstat11:38:231
2892299332,3cyclictest281ktimers/108:31:021
2892299332,3cyclictest281ktimers/108:31:021
183772330,5sleep30-21swapper/308:08:313
183772330,5sleep30-21swapper/308:08:313
28922993226,4cyclictest18905-21latency_hist12:33:061
28922993226,4cyclictest18905-21latency_hist12:33:061
28922993226,4cyclictest18905-21latency_hist12:33:061
28922993226,4cyclictest17248-21fschecks_count08:08:181
28922993226,4cyclictest17248-21fschecks_count08:08:181
28922993224,3cyclictest181rcu_preempt10:12:081
28922993224,3cyclictest181rcu_preempt10:12:081
28922993224,3cyclictest181rcu_preempt10:12:081
2892299317,5cyclictest271rcuc/111:57:371
2892299317,5cyclictest271rcuc/111:57:371
2892299315,24cyclictest281ktimers/111:26:371
28922993125,4cyclictest5184-21apache209:43:271
28922993125,4cyclictest5184-21apache209:43:271
28922993124,5cyclictest28896-21cyclictest09:07:551
28922993124,5cyclictest28896-21cyclictest09:07:551
28922993124,5cyclictest28896-21cyclictest09:07:551
28922993123,6cyclictest281ktimers/110:33:351
28922993123,6cyclictest281ktimers/110:33:351
28922993118,3cyclictest181rcu_preempt09:42:181
28922993118,3cyclictest181rcu_preempt09:42:181
28922993114,3cyclictest90-1pr/ttyS009:18:181
28910993114,4cyclictest90-1pr/ttyS009:35:040
28910993114,4cyclictest90-1pr/ttyS009:35:040
2892299303,5cyclictest281ktimers/112:07:381
2892299303,5cyclictest281ktimers/112:07:381
28922993022,6cyclictest281ktimers/111:17:061
28922993022,6cyclictest281ktimers/111:17:061
28922993022,6cyclictest281ktimers/111:17:061
28922993022,6cyclictest13981-21sort07:58:221
28922993021,7cyclictest27691-21latency_hist10:03:161
28922993021,7cyclictest27691-21latency_hist10:03:161
28922993016,2cyclictest181rcu_preempt07:41:311
28910993026,3cyclictest4468-21idleruntime-cro10:28:060
28910993026,3cyclictest4468-21idleruntime-cro10:28:060
28910993026,3cyclictest4468-21idleruntime-cro10:28:060
28910993022,6cyclictest9394-21cut10:38:280
28910993022,6cyclictest9394-21cut10:38:280
28910993022,6cyclictest28115-21head10:03:200
28910993022,6cyclictest11157-21fschecks_time09:18:180
28910993021,7cyclictest30065-21threads11:33:310
28910993021,7cyclictest30065-21threads11:33:310
115242300,3sleep30-21swapper/309:18:223
2892299298,10cyclictest29-21ksoftirqd/109:23:551
2892299298,10cyclictest29-21ksoftirqd/109:23:551
28922992925,3cyclictest28896-21cyclictest10:03:271
28922992925,3cyclictest28896-21cyclictest10:03:271
28922992925,2cyclictest0-21swapper/108:17:321
28922992925,2cyclictest0-21swapper/108:17:321
28922992922,5cyclictest281ktimers/108:44:321
28922992922,5cyclictest281ktimers/108:44:321
28922992922,5cyclictest1792-21sshd07:28:011
28922992922,5cyclictest1792-21sshd07:28:011
28922992920,7cyclictest20073-21switchtime12:33:251
28922992920,7cyclictest20073-21switchtime12:33:251
28922992920,7cyclictest20073-21switchtime12:33:251
28922992913,4cyclictest181rcu_preempt09:36:271
28922992913,4cyclictest181rcu_preempt09:36:271
28910992923,4cyclictest749-21runrttasks10:18:170
28910992923,4cyclictest749-21runrttasks10:18:170
28910992923,4cyclictest749-21runrttasks10:18:170
28910992921,6cyclictest9945-21gltestperf07:48:190
28910992921,6cyclictest9945-21gltestperf07:48:190
28910992921,6cyclictest9945-21gltestperf07:48:190
28910992921,6cyclictest3043-21processes11:48:270
28910992921,6cyclictest3043-21processes11:48:270
28910992917,4cyclictest28896-21cyclictest11:07:300
28910992917,4cyclictest28896-21cyclictest11:07:300
28910992917,4cyclictest28896-21cyclictest11:07:300
2892299287,9cyclictest29-21ksoftirqd/110:43:151
2892299287,9cyclictest29-21ksoftirqd/110:43:151
2892299287,18cyclictest281ktimers/111:52:231
2892299287,18cyclictest281ktimers/111:52:231
28922992818,3cyclictest181rcu_preempt07:52:311
28922992818,3cyclictest181rcu_preempt07:52:311
28922992818,3cyclictest181rcu_preempt07:52:311
28922992811,4cyclictest181rcu_preempt12:25:101
28922992811,4cyclictest181rcu_preempt12:25:101
28922992811,4cyclictest181rcu_preempt12:25:101
28910992823,3cyclictest0-21swapper/012:23:060
28910992823,3cyclictest0-21swapper/012:23:060
28910992823,3cyclictest0-21swapper/012:23:060
28910992822,5cyclictest10119-21gltestperf12:08:190
28910992822,5cyclictest10119-21gltestperf12:08:190
28910992822,5cyclictest10119-21gltestperf12:08:190
28910992821,5cyclictest1740-21fschecks_time08:53:170
28910992821,5cyclictest1740-21fschecks_time08:53:170
28910992820,6cyclictest9023-21apache210:58:440
28910992820,6cyclictest9023-21apache210:58:440
28910992820,6cyclictest9023-21apache210:58:440
28910992818,8cyclictest8361-21kernelversion12:03:210
28910992818,8cyclictest8361-21kernelversion12:03:210
2892299278,10cyclictest29-21ksoftirqd/111:10:401
2892299278,10cyclictest29-21ksoftirqd/111:10:401
2892299278,10cyclictest29-21ksoftirqd/111:10:401
28922992723,3cyclictest749-21runrttasks11:03:201
28922992723,3cyclictest749-21runrttasks11:03:201
28922992723,3cyclictest749-21runrttasks11:03:201
28922992723,2cyclictest0-21swapper/108:23:271
28922992723,2cyclictest0-21swapper/108:23:271
28922992721,5cyclictest627-21fschecks_time11:43:181
28922992721,3cyclictest0-21swapper/109:28:201
28922992721,3cyclictest0-21swapper/109:28:201
28922992720,5cyclictest2755-21ntp_kernel_pll_07:28:251
28922992720,5cyclictest2755-21ntp_kernel_pll_07:28:251
28922992720,4cyclictest281ktimers/110:13:351
28922992720,4cyclictest281ktimers/110:13:351
28922992720,4cyclictest281ktimers/110:13:351
28922992712,4cyclictest181rcu_preempt12:11:231
28922992712,4cyclictest181rcu_preempt12:11:231
28922992712,4cyclictest181rcu_preempt12:11:231
2891099278,5cyclictest201rcuc/008:55:330
2891099278,5cyclictest201rcuc/008:55:330
2891099278,5cyclictest201rcuc/008:55:330
28910992721,5cyclictest16873-21gltestperf09:33:170
28910992721,5cyclictest16873-21gltestperf09:33:170
28910992721,4cyclictest5535-21gltestperf09:03:200
28910992721,4cyclictest5535-21gltestperf09:03:200
28910992721,4cyclictest5535-21gltestperf09:03:200
28910992720,5cyclictest17835-21ntp_offset08:08:230
28910992720,5cyclictest17835-21ntp_offset08:08:230
32912260,3sleep20-21swapper/207:28:302
32912260,3sleep20-21swapper/207:28:302
28922992622,2cyclictest0-21swapper/108:48:281
28922992622,2cyclictest0-21swapper/108:48:281
28922992618,6cyclictest9566-21sort09:13:221
28922992618,6cyclictest9566-21sort09:13:221
28922992618,6cyclictest9566-21sort09:13:221
28922992618,2cyclictest181rcu_preempt12:23:151
28922992618,2cyclictest181rcu_preempt12:23:151
28922992618,2cyclictest181rcu_preempt12:23:151
28922992616,4cyclictest16449-21kworker/u9:0-xprtiod10:28:171
28922992616,4cyclictest16449-21kworker/u9:0-xprtiod10:28:171
28922992614,2cyclictest181rcu_preempt07:33:451
28922992614,2cyclictest181rcu_preempt07:33:451
28922992614,2cyclictest181rcu_preempt07:33:451
28922992610,4cyclictest181rcu_preempt10:31:571
28922992610,4cyclictest181rcu_preempt10:31:571
28922992610,4cyclictest181rcu_preempt10:31:571
28910992622,3cyclictest90-1pr/ttyS009:04:330
28910992622,3cyclictest90-1pr/ttyS009:04:330
28910992622,3cyclictest90-1pr/ttyS009:04:330
28910992621,3cyclictest0-21swapper/011:38:160
28910992621,3cyclictest0-21swapper/011:38:160
28910992620,5cyclictest1011-21rt-features07:23:260
28910992620,5cyclictest1011-21rt-features07:23:260
28910992619,5cyclictest4895-21fschecks_time10:28:200
28910992619,5cyclictest4895-21fschecks_time10:28:200
28910992619,5cyclictest4895-21fschecks_time10:28:200
28910992619,5cyclictest25655-21latency_hist09:58:190
28910992619,5cyclictest25655-21latency_hist09:58:190
28910992618,6cyclictest24901-21gltestperf08:28:190
28910992618,6cyclictest24901-21gltestperf08:28:190
67082250,1sleep10-21swapper/111:58:231
67082250,1sleep10-21swapper/111:58:231
28922992522,2cyclictest90-1pr/ttyS010:22:051
28922992522,2cyclictest90-1pr/ttyS010:22:051
28922992522,2cyclictest90-1pr/ttyS010:22:051
28922992522,2cyclictest90-1pr/ttyS010:22:051
28922992519,4cyclictest547-21rs:main1
28922992519,4cyclictest547-21rs:main1
28922992517,2cyclictest181rcu_preempt12:16:081
28922992517,2cyclictest181rcu_preempt12:16:081
28922992517,2cyclictest181rcu_preempt12:16:081
28910992520,4cyclictest15009-21ls08:03:060
28910992520,3cyclictest547-21rs:main0
28910992520,3cyclictest547-21rs:main0
28910992520,3cyclictest0-21swapper/007:28:200
28910992520,3cyclictest0-21swapper/007:28:200
28910992519,4cyclictest28896-21cyclictest07:43:240
28910992519,4cyclictest28896-21cyclictest07:43:240
28910992517,5cyclictest0-21swapper/008:18:270
2891099251,21cyclictest27096-21grep09:58:310
28931992417,4cyclictest13582sleep307:23:293
28931992417,4cyclictest13582sleep307:23:293
2892299242,8cyclictest90-1pr/ttyS010:25:501
28910992420,2cyclictest0-21swapper/007:38:040
28910992420,2cyclictest0-21swapper/007:38:040
28910992420,2cyclictest0-21swapper/007:38:040
28910992419,3cyclictest0-21swapper/012:13:190
28910992419,3cyclictest0-21swapper/012:13:190
28910992419,3cyclictest0-21swapper/012:13:190
28910992419,3cyclictest0-21swapper/011:18:190
28910992419,3cyclictest0-21swapper/011:18:190
28910992419,3cyclictest0-21swapper/010:48:280
28910992419,3cyclictest0-21swapper/010:48:280
28910992419,3cyclictest0-21swapper/009:58:060
28910992418,5cyclictest1479-21latency10:18:210
28910992418,5cyclictest1479-21latency10:18:210
28910992418,5cyclictest1479-21latency10:18:210
28910992418,5cyclictest1479-21latency10:18:210
28910992417,5cyclictest22652-21kernelversion09:48:210
28910992417,5cyclictest22652-21kernelversion09:48:210
28910992417,5cyclictest22652-21kernelversion09:48:210
28910992416,6cyclictest19130-21uniq08:13:190
28910992416,6cyclictest19130-21uniq08:13:190
28910992415,7cyclictest14678-21systemd-journal11:13:060
28910992415,7cyclictest14678-21systemd-journal11:13:060
28910992415,7cyclictest14678-21systemd-journal11:13:060
28910992410,12cyclictest8177-21grep09:08:280
28910992410,12cyclictest8177-21grep09:08:280
28910992410,12cyclictest8177-21grep09:08:280
28922992318,3cyclictest0-21swapper/107:46:311
28922992318,3cyclictest0-21swapper/107:46:311
28922992316,2cyclictest181rcu_preempt08:03:211
28922992316,2cyclictest181rcu_preempt08:03:211
2891099238,5cyclictest181rcu_preempt10:13:180
2891099238,5cyclictest181rcu_preempt10:13:180
2891099238,5cyclictest181rcu_preempt10:13:180
28910992319,2cyclictest0-21swapper/011:23:180
28910992319,2cyclictest0-21swapper/011:23:180
28910992318,4cyclictest28896-21cyclictest12:31:310
28910992318,4cyclictest28896-21cyclictest12:31:310
28910992318,4cyclictest28896-21cyclictest12:31:310
28910992318,3cyclictest0-21swapper/010:43:190
28910992318,3cyclictest0-21swapper/010:43:190
28910992318,3cyclictest0-21swapper/009:43:240
28910992318,3cyclictest0-21swapper/009:43:240
28910992317,4cyclictest0-21swapper/012:33:270
28910992317,4cyclictest0-21swapper/012:33:270
28910992317,4cyclictest0-21swapper/012:33:270
28910992317,3cyclictest0-21swapper/007:53:260
28910992317,3cyclictest0-21swapper/007:53:260
28910992218,2cyclictest0-21swapper/010:03:290
28910992218,2cyclictest0-21swapper/010:03:290
28910992217,3cyclictest23155-21grep12:43:200
28910992217,3cyclictest23155-21grep12:43:200
28910992217,3cyclictest0-21swapper/012:23:250
28910992217,3cyclictest0-21swapper/012:23:250
28910992217,3cyclictest0-21swapper/012:23:250
28910992217,3cyclictest0-21swapper/008:43:050
28910992216,5cyclictest4770-21needreboot11:53:220
28910992216,5cyclictest4770-21needreboot11:53:220
28910992215,4cyclictest0-21swapper/010:53:270
28910992215,4cyclictest0-21swapper/010:53:270
28910992215,4cyclictest0-21swapper/010:53:270
128662220,3sleep30-21swapper/310:48:223
128662220,3sleep30-21swapper/310:48:223
106942220,3sleep30-21swapper/309:18:063
106942220,3sleep30-21swapper/309:18:063
28910992117,3cyclictest6029-21grep07:38:180
28910992116,3cyclictest0-21swapper/011:43:260
28910992116,3cyclictest0-21swapper/011:43:260
28910992116,3cyclictest0-21swapper/010:33:190
28910992116,3cyclictest0-21swapper/010:33:190
28910992115,4cyclictest0-21swapper/011:23:270
2891099211,11cyclictest23440-21cpuspeed11:18:160
28910992015,3cyclictest0-21swapper/011:28:250
28910992015,3cyclictest0-21swapper/011:28:250
28910992012,6cyclictest10076-21sort09:13:280
28910992012,6cyclictest10076-21sort09:13:280
126882200,2chrt0-21swapper/209:23:112
28910991914,3cyclictest0-21swapper/012:38:300
61772180,3sleep20-21swapper/207:38:192
2893199183,8cyclictest0-21swapper/307:52:303
2893199183,8cyclictest0-21swapper/307:52:303
2893199183,8cyclictest0-21swapper/307:52:303
28929991813,3cyclictest265382chrt08:33:182
28929991813,3cyclictest265382chrt08:33:182
2892999181,12cyclictest22539-21taskset09:48:202
2892999181,12cyclictest22539-21taskset09:48:202
2892999181,12cyclictest22539-21taskset09:48:202
2893199174,7cyclictest0-21swapper/309:53:183
2893199174,7cyclictest0-21swapper/309:53:183
2893199172,9cyclictest0-21swapper/311:10:313
2893199172,9cyclictest0-21swapper/311:10:313
2893199172,9cyclictest0-21swapper/311:10:313
2893199172,10cyclictest0-21swapper/311:53:203
2893199172,10cyclictest0-21swapper/311:53:203
2893199171,12cyclictest10789-21taskset10:43:203
2893199171,12cyclictest10789-21taskset10:43:203
2893199171,11cyclictest22101-21taskset12:38:293
2893199171,11cyclictest22101-21taskset12:38:293
2893199171,11cyclictest22101-21taskset12:38:293
2893199171,11cyclictest16221-21chrt09:31:483
2893199171,11cyclictest16221-21chrt09:31:483
2893199171,11cyclictest0-21swapper/309:38:163
2893199171,11cyclictest0-21swapper/309:38:163
2892999174,11cyclictest0-21swapper/209:58:312
2892999174,11cyclictest0-21swapper/209:58:312
28929991712,3cyclictest200812sleep212:33:252
28929991712,3cyclictest200812sleep212:33:252
28929991712,3cyclictest200812sleep212:33:252
2892999171,13cyclictest16750-21chrt09:33:162
2892999171,13cyclictest16750-21chrt09:33:162
2892999171,12cyclictest9552-21chrt07:48:162
2892999171,12cyclictest9552-21chrt07:48:162
2892999171,11cyclictest17310-21chrt08:08:182
2892999171,11cyclictest17310-21chrt08:08:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional