You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 16:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Mon Dec 29, 2025 12:44:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1640321660,2sleep30-21swapper/307:50:313
1640321660,2sleep30-21swapper/307:50:313
76042930,2chrt7599-21threads07:25:390
76042930,2chrt7599-21threads07:25:390
186032670,2sleep118477-21/usr/sbin/munin10:50:281
186032670,2sleep118477-21/usr/sbin/munin10:50:281
54882660,3sleep20-21swapper/211:40:382
54882660,3sleep20-21swapper/211:40:382
65332650,1sleep06364-21latency_hist07:25:240
8812590,3sleep01050-21cyclictest11:30:180
8812590,3sleep01050-21cyclictest11:30:180
40132580,2chrt0-21swapper/208:45:212
40132580,2chrt0-21swapper/208:45:212
96192530,2chrt0-21swapper/207:30:402
96192530,2chrt0-21swapper/207:30:402
166282530,2chrt0-21swapper/207:50:342
166282530,2chrt0-21swapper/207:50:342
15602520,3sleep20-21swapper/208:35:402
15602520,3sleep20-21swapper/208:35:402
137542520,4sleep20-21swapper/209:10:302
137542520,4sleep20-21swapper/209:10:302
194022500,3sleep20-21swapper/209:25:292
194022500,3sleep20-21swapper/209:25:292
170052480,3sleep20-21swapper/210:45:302
84202470,3sleep20-21swapper/210:20:372
84202470,3sleep20-21swapper/210:20:372
294392460,2sleep30-21swapper/309:50:393
116992460,3chrt0-21swapper/209:05:272
116992460,3chrt0-21swapper/209:05:272
95462450,3sleep20-21swapper/210:25:322
95462450,3sleep20-21swapper/210:25:322
47932450,3sleep00-21swapper/011:40:270
47932450,3sleep00-21swapper/011:40:270
149392450,1sleep00-21swapper/012:05:360
149392450,1sleep00-21swapper/012:05:360
1051994520,23cyclictest0-21swapper/007:50:170
1051994520,23cyclictest0-21swapper/007:50:170
273122430,3sleep20-21swapper/209:45:362
1051994318,3cyclictest0-21swapper/010:20:290
1051994318,23cyclictest12089-21kernelversion09:05:310
1051994318,23cyclictest12089-21kernelversion09:05:310
93292410,3sleep20-21swapper/211:50:392
138372400,3sleep30-21swapper/310:35:383
138372400,3sleep30-21swapper/310:35:383
1051993933,4cyclictest31942-21latency_hist10:00:160
1051993933,4cyclictest31942-21latency_hist10:00:160
201522380,4sleep20-21swapper/208:00:312
201522380,4sleep20-21swapper/208:00:312
79302370,3chrt0-21swapper/207:26:302
79302370,3chrt0-21swapper/207:26:302
1052993728,7cyclictest18866-21cron09:25:161
1052993728,7cyclictest18866-21cron09:25:161
1051993614,3cyclictest0-21swapper/010:30:410
1051993614,3cyclictest0-21swapper/010:30:410
1051993528,5cyclictest23297-21timerandwakeup11:00:420
1051993528,5cyclictest23297-21timerandwakeup11:00:420
1051993528,5cyclictest23297-21timerandwakeup11:00:420
279172340,2sleep027959-21fschecks_count11:15:280
279172340,2sleep027959-21fschecks_count11:15:280
279172340,2sleep027959-21fschecks_count11:15:280
1052993420,4cyclictest181rcu_preempt08:54:591
1052993420,4cyclictest181rcu_preempt08:54:591
1051993428,5cyclictest15770-21threads10:40:410
1051993428,5cyclictest15770-21threads10:40:410
235632330,3sleep00-21swapper/009:35:370
235632330,3sleep00-21swapper/009:35:370
1052993324,7cyclictest11474-21if_wlan010:30:331
1052993324,7cyclictest11474-21if_wlan010:30:331
1052993225,2cyclictest181rcu_preempt11:50:031
1052993225,2cyclictest181rcu_preempt11:50:031
1052993224,6cyclictest23636-21fschecks_count12:30:291
1052993224,6cyclictest23636-21fschecks_count12:30:291
1051993226,3cyclictest0-21swapper/012:00:160
1051993226,3cyclictest0-21swapper/012:00:160
33072310,2sleep13309-21grep08:40:421
33072310,2sleep13309-21grep08:40:421
105299319,5cyclictest281ktimers/108:28:571
1052993127,3cyclictest25991-21ntp_offset08:15:371
1052993127,3cyclictest25991-21ntp_offset08:15:371
1052993125,4cyclictest271rcuc/111:17:441
1052993125,4cyclictest271rcuc/111:17:441
1052993125,4cyclictest271rcuc/111:17:441
1052993124,5cyclictest9242-21netstat11:50:361
1052993124,5cyclictest5773-21latency_hist08:50:171
1052993124,5cyclictest5773-21latency_hist08:50:171
1052993119,4cyclictest181rcu_preempt12:08:341
1052993119,4cyclictest181rcu_preempt12:08:341
1052993116,8cyclictest90-1pr/ttyS007:43:171
1052993116,8cyclictest90-1pr/ttyS007:43:171
1052993112,7cyclictest29-21ksoftirqd/111:35:371
1052993112,7cyclictest29-21ksoftirqd/111:35:371
1051993124,5cyclictest28157-21threads08:20:380
105299306,6cyclictest281ktimers/111:57:141
105299306,6cyclictest281ktimers/111:57:141
1052993023,5cyclictest2925-21uptime10:05:401
1052993022,6cyclictest742-21threads10:00:391
1052993022,6cyclictest742-21threads10:00:391
1052993016,4cyclictest90-1pr/ttyS012:30:171
1052993016,4cyclictest90-1pr/ttyS012:30:171
1051993025,4cyclictest1928-21fschecks_time10:05:300
1051993024,5cyclictest12696-21kernelversion07:40:330
1051993024,5cyclictest12696-21kernelversion07:40:330
1051993024,4cyclictest1050-21cyclictest10:30:170
1051993024,4cyclictest1050-21cyclictest10:30:170
1051993023,5cyclictest8343-21kernelversion08:55:330
1051993023,5cyclictest8343-21kernelversion08:55:330
1051993022,6cyclictest6989-21threads08:50:390
1051993022,6cyclictest6989-21threads08:50:390
105199301,9cyclictest30710-21ls08:30:180
105299299,3cyclictest281ktimers/110:29:161
105299299,3cyclictest281ktimers/110:29:161
105299295,6cyclictest281ktimers/107:46:561
105299295,6cyclictest281ktimers/107:46:561
105299295,5cyclictest271rcuc/109:53:191
105299294,6cyclictest281ktimers/111:03:591
105299294,6cyclictest281ktimers/111:03:591
105299294,6cyclictest281ktimers/111:03:591
1052992922,5cyclictest6462-21latency_hist11:45:171
1052992922,5cyclictest6462-21latency_hist11:45:171
1052992921,6cyclictest31007-21tr09:55:351
1052992921,6cyclictest31007-21tr09:55:351
1052992921,6cyclictest26054-21netstat12:35:351
1052992921,6cyclictest26054-21netstat12:35:351
1052992921,6cyclictest26054-21netstat12:35:351
1052992921,6cyclictest18795-21threads07:55:391
1052992921,6cyclictest18795-21threads07:55:391
1052992921,6cyclictest18795-21threads07:55:391
1052992920,2cyclictest181rcu_preempt11:10:231
1052992920,2cyclictest181rcu_preempt11:10:231
1052992916,3cyclictest90-1pr/ttyS011:00:231
1052992916,3cyclictest90-1pr/ttyS011:00:231
1052992916,3cyclictest90-1pr/ttyS011:00:231
1052992915,7cyclictest90-1pr/ttyS009:40:211
1052992915,7cyclictest90-1pr/ttyS009:40:211
1052992914,2cyclictest181rcu_preempt09:10:281
1052992914,2cyclictest181rcu_preempt09:10:281
1051992925,2cyclictest0-21swapper/007:25:160
1051992924,3cyclictest0-21swapper/009:20:290
1051992924,3cyclictest0-21swapper/009:20:290
1051992923,4cyclictest1050-21cyclictest11:51:380
1051992922,5cyclictest6579-21latency_hist11:45:230
1051992922,5cyclictest6579-21latency_hist11:45:230
210142280,2sleep30-21swapper/310:55:363
210142280,2sleep30-21swapper/310:55:363
210142280,2sleep30-21swapper/310:55:363
105299285,5cyclictest281ktimers/109:46:361
1052992817,3cyclictest181rcu_preempt10:19:151
1052992817,3cyclictest181rcu_preempt10:19:151
1052992817,3cyclictest181rcu_preempt10:19:151
1052992814,7cyclictest463-21kworker/u9:1-rpciod08:40:191
1052992814,7cyclictest463-21kworker/u9:1-rpciod08:40:191
1051992821,5cyclictest11477-21latency_hist09:05:210
1051992820,6cyclictest19930-21cut12:20:300
1051992820,6cyclictest19930-21cut12:20:300
1051992820,6cyclictest16867-21rt-features12:10:380
1051992820,6cyclictest16867-21rt-features12:10:380
1051992819,7cyclictest11159-21processes07:35:360
1051992819,7cyclictest11159-21processes07:35:360
1051992819,7cyclictest11159-21processes07:35:360
105299276,4cyclictest281ktimers/110:39:311
105299276,4cyclictest281ktimers/110:39:311
1052992722,4cyclictest1050-21cyclictest08:55:311
1052992722,4cyclictest1050-21cyclictest08:55:311
1052992720,5cyclictest12521-21cut12:00:321
1052992720,5cyclictest12521-21cut12:00:321
1052992718,4cyclictest181rcu_preempt11:13:361
1052992718,4cyclictest181rcu_preempt11:13:361
1052992717,2cyclictest181rcu_preempt09:35:111
1052992717,2cyclictest181rcu_preempt09:35:111
1052992715,3cyclictest90-1pr/ttyS012:17:321
1052992715,3cyclictest90-1pr/ttyS012:17:321
1052992715,3cyclictest90-1pr/ttyS012:17:321
1052992712,2cyclictest181rcu_preempt08:25:231
1052992711,3cyclictest181rcu_preempt07:55:231
1052992711,3cyclictest181rcu_preempt07:55:231
1051992722,3cyclictest0-21swapper/011:10:380
1051992722,3cyclictest0-21swapper/011:10:380
1051992721,4cyclictest16630-21ntp_kernel_err07:50:340
1051992721,4cyclictest16630-21ntp_kernel_err07:50:340
1051992721,3cyclictest0-21swapper/009:50:390
1051992719,7cyclictest20847-21kernelversion10:55:340
1051992719,7cyclictest20847-21kernelversion10:55:340
1051992719,7cyclictest20847-21kernelversion10:55:340
105299265,9cyclictest21531-21/usr/sbin/munin08:05:371
105299264,8cyclictest29-21ksoftirqd/109:15:361
105299264,8cyclictest29-21ksoftirqd/109:15:361
105299264,8cyclictest29-21ksoftirqd/109:15:361
1052992622,3cyclictest1050-21cyclictest07:30:351
1052992622,3cyclictest1050-21cyclictest07:30:351
1052992620,5cyclictest5350-21missed_timers07:20:331
1052992620,5cyclictest5350-21missed_timers07:20:331
1052992619,5cyclictest16319-21gltestperf12:10:311
1052992619,5cyclictest16319-21gltestperf12:10:311
1052992616,9cyclictest12083-21kernelversion09:05:311
1052992616,9cyclictest12083-21kernelversion09:05:311
1052992616,2cyclictest181rcu_preempt11:33:561
105299261,5cyclictest281ktimers/110:42:281
105299261,5cyclictest281ktimers/110:42:281
1052992613,4cyclictest90-1pr/ttyS007:26:591
1052992613,4cyclictest90-1pr/ttyS007:26:591
1052992613,3cyclictest181rcu_preempt11:25:311
1052992613,3cyclictest181rcu_preempt11:25:311
1052992613,2cyclictest181rcu_preempt10:45:351
1052992610,2cyclictest181rcu_preempt10:10:381
1052992610,2cyclictest181rcu_preempt10:10:381
1051992620,3cyclictest0-21swapper/011:45:310
1051992620,3cyclictest0-21swapper/011:45:310
1051992619,5cyclictest7041-21latency_hist10:20:180
1051992619,5cyclictest7041-21latency_hist10:20:180
105299259,3cyclictest181rcu_preempt08:00:311
105299259,3cyclictest181rcu_preempt08:00:311
105299257,3cyclictest271rcuc/107:38:141
105299257,3cyclictest271rcuc/107:38:141
105299257,3cyclictest271rcuc/107:38:141
1052992520,3cyclictest281ktimers/109:25:341
1052992520,3cyclictest281ktimers/109:25:341
1052992518,2cyclictest181rcu_preempt09:01:181
1052992517,2cyclictest181rcu_preempt12:20:371
1052992517,2cyclictest181rcu_preempt12:20:371
1052992512,2cyclictest181rcu_preempt09:40:381
1052992512,2cyclictest181rcu_preempt09:40:381
1051992521,2cyclictest0-21swapper/009:25:300
1051992521,2cyclictest0-21swapper/009:25:300
1051992521,2cyclictest0-21swapper/008:35:310
1051992521,2cyclictest0-21swapper/008:35:310
1051992520,3cyclictest0-21swapper/011:30:340
1051992520,3cyclictest0-21swapper/009:15:310
1051992520,3cyclictest0-21swapper/009:15:310
1051992520,3cyclictest0-21swapper/009:15:310
1051992520,3cyclictest0-21swapper/007:30:300
1051992520,3cyclictest0-21swapper/007:30:300
1051992519,5cyclictest4026-21kernelversion10:10:320
1051992519,5cyclictest4026-21kernelversion10:10:320
1051992519,4cyclictest21665-21ntp_kernel_err09:30:360
1051992519,4cyclictest21665-21ntp_kernel_err09:30:360
1051992519,3cyclictest0-21swapper/010:00:400
1051992519,3cyclictest0-21swapper/010:00:400
1051992517,6cyclictest674-21gpm12:30:340
1051992517,6cyclictest674-21gpm12:30:340
1051992517,6cyclictest18808-21grep12:15:390
1051992517,6cyclictest18808-21grep12:15:390
1051992517,6cyclictest18808-21grep12:15:390
1052992419,3cyclictest31217-21grep08:30:321
1052992419,3cyclictest31217-21grep08:30:321
1052992419,3cyclictest24071-21needreboot08:10:351
1052992419,3cyclictest24071-21needreboot08:10:351
1052992419,3cyclictest0-21swapper/111:20:321
1052992419,3cyclictest0-21swapper/111:20:321
1052992411,2cyclictest181rcu_preempt10:20:311
1052992411,2cyclictest181rcu_preempt10:20:311
1051992418,4cyclictest749-21runrttasks11:20:300
1051992418,4cyclictest749-21runrttasks11:20:300
1051992418,3cyclictest0-21swapper/008:40:350
1051992418,3cyclictest0-21swapper/008:40:350
1051992417,5cyclictest21695-21cpuspeed12:25:270
1051992417,5cyclictest21695-21cpuspeed12:25:270
1051992416,6cyclictest23244-21latency_hist08:10:190
148952230,3chrt0-21swapper/312:05:353
148952230,3chrt0-21swapper/312:05:353
1062992317,4cyclictest431rcuc/310:11:473
1062992317,4cyclictest431rcuc/310:11:473
1051992318,3cyclictest0-21swapper/008:15:400
1051992318,3cyclictest0-21swapper/008:15:400
1051992317,4cyclictest13748-21sendmail_mailst10:35:370
1051992317,4cyclictest13748-21sendmail_mailst10:35:370
1051992317,3cyclictest0-21swapper/009:10:320
1051992317,3cyclictest0-21swapper/009:10:320
1051992315,6cyclictest18437-21needreboot07:55:350
1051992315,6cyclictest18437-21needreboot07:55:350
1051992315,6cyclictest18437-21needreboot07:55:350
114512220,4chrt0-21swapper/207:35:382
114512220,4chrt0-21swapper/207:35:382
114512220,4chrt0-21swapper/207:35:382
1062992218,3cyclictest431rcuc/309:18:323
1062992218,3cyclictest431rcuc/309:18:323
1062992218,3cyclictest431rcuc/309:18:323
1062992217,4cyclictest431rcuc/312:40:013
1062992217,4cyclictest431rcuc/312:40:013
1062992217,4cyclictest431rcuc/312:40:013
1062992217,3cyclictest431rcuc/308:45:073
1062992217,3cyclictest431rcuc/308:45:073
1062992217,3cyclictest195542chrt09:25:303
1062992217,3cyclictest195542chrt09:25:303
1062992216,4cyclictest431rcuc/307:55:503
1062992216,4cyclictest431rcuc/307:55:503
1062992216,4cyclictest431rcuc/307:55:503
1051992217,3cyclictest0-21swapper/008:10:320
1051992217,3cyclictest0-21swapper/008:10:320
1051992216,4cyclictest749-21runrttasks11:05:380
1051992216,4cyclictest749-21runrttasks11:05:380
1051992216,4cyclictest0-21swapper/009:45:370
1051992216,3cyclictest0-21swapper/012:35:290
1051992216,3cyclictest0-21swapper/012:35:290
1051992216,3cyclictest0-21swapper/012:35:290
1051992214,6cyclictest5789-21cat08:50:170
1051992214,6cyclictest5789-21cat08:50:170
1051992214,6cyclictest18665-21fschecks_count10:50:280
1051992214,6cyclictest18665-21fschecks_count10:50:280
1051992214,6cyclictest12789-21kernelversion12:00:350
1051992214,6cyclictest12789-21kernelversion12:00:350
1051992213,8cyclictest90-1pr/ttyS010:20:350
1051992213,8cyclictest90-1pr/ttyS010:20:350
1062992117,3cyclictest431rcuc/312:35:153
1062992117,3cyclictest431rcuc/312:35:153
1062992117,3cyclictest431rcuc/312:29:083
1062992117,3cyclictest431rcuc/312:29:083
1062992117,3cyclictest431rcuc/308:08:093
1062992116,3cyclictest431rcuc/307:47:023
1062992116,3cyclictest431rcuc/307:47:023
1062992113,6cyclictest431rcuc/307:21:373
1062992113,6cyclictest431rcuc/307:21:373
1051992116,3cyclictest0-21swapper/009:45:250
1051992116,3cyclictest0-21swapper/009:45:250
1062992016,3cyclictest431rcuc/310:24:173
1062992016,3cyclictest431rcuc/310:24:173
1062992016,3cyclictest431rcuc/310:17:543
1062992016,3cyclictest431rcuc/310:17:543
1062992016,3cyclictest431rcuc/310:17:543
1062992016,3cyclictest431rcuc/309:58:043
1062992016,3cyclictest431rcuc/309:58:043
1062992016,3cyclictest431rcuc/309:39:143
1062992016,3cyclictest431rcuc/309:39:143
1062992015,4cyclictest431rcuc/310:08:253
1062992015,3cyclictest431rcuc/312:13:083
1062992015,3cyclictest431rcuc/312:13:083
1062992015,3cyclictest431rcuc/312:01:383
1062992015,3cyclictest431rcuc/312:01:383
1062992014,4cyclictest431rcuc/311:08:433
1062992014,4cyclictest431rcuc/311:08:433
1062992013,5cyclictest431rcuc/307:31:303
1062992013,5cyclictest431rcuc/307:31:303
1051992015,3cyclictest0-21swapper/008:00:290
1051992015,3cyclictest0-21swapper/008:00:290
105199201,10cyclictest749-21runrttasks08:32:330
105199201,10cyclictest749-21runrttasks08:32:330
1062991915,3cyclictest431rcuc/312:24:583
1062991915,3cyclictest431rcuc/312:24:583
1062991915,3cyclictest431rcuc/310:46:243
1062991915,3cyclictest431rcuc/309:04:453
1062991915,3cyclictest431rcuc/308:48:293
1062991915,3cyclictest431rcuc/308:48:293
1062991915,3cyclictest431rcuc/308:38:003
1062991915,3cyclictest431rcuc/308:38:003
1062991915,3cyclictest431rcuc/308:32:573
1062991915,3cyclictest431rcuc/308:32:573
1062991915,3cyclictest431rcuc/308:18:263
1062991915,3cyclictest431rcuc/308:18:263
1062991914,4cyclictest431rcuc/311:58:083
1062991914,4cyclictest431rcuc/311:58:083
1062991914,4cyclictest431rcuc/309:35:203
1062991914,4cyclictest431rcuc/309:35:203
1062991914,4cyclictest431rcuc/308:01:093
1062991914,4cyclictest431rcuc/308:01:093
1062991914,3cyclictest431rcuc/311:42:413
1062991914,3cyclictest431rcuc/311:42:413
1062991914,3cyclictest431rcuc/311:33:563
1062991914,3cyclictest431rcuc/311:25:283
1062991914,3cyclictest431rcuc/311:25:283
1062991914,3cyclictest431rcuc/310:01:343
1062991914,3cyclictest431rcuc/310:01:343
1062991913,4cyclictest431rcuc/308:56:213
1062991913,4cyclictest431rcuc/308:56:213
105899193,9cyclictest0-21swapper/211:45:472
105899193,9cyclictest0-21swapper/211:45:472
1058991913,4cyclictest32045-21chrt08:30:422
1058991913,4cyclictest32045-21chrt08:30:422
1051991911,6cyclictest17607-21sort10:45:380
1062991814,3cyclictest431rcuc/311:52:213
1062991814,3cyclictest431rcuc/310:51:553
1062991814,3cyclictest431rcuc/310:51:553
1062991814,3cyclictest431rcuc/310:45:203
1062991814,3cyclictest431rcuc/310:45:203
1062991814,3cyclictest431rcuc/309:41:153
1062991814,3cyclictest431rcuc/309:41:153
1062991814,3cyclictest431rcuc/309:23:303
1062991814,3cyclictest431rcuc/309:23:303
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional