You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-20 - 03:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Mon Apr 20, 2026 00:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1856521440,2sleep20-21swapper/200:02:172
1856521440,2sleep20-21swapper/200:02:172
1856521440,2sleep20-21swapper/200:02:172
888121290,2sleep20-21swapper/222:07:252
888121290,2sleep20-21swapper/222:07:252
888121290,2sleep20-21swapper/222:07:252
3000921190,4sleep20-21swapper/223:07:102
3000921190,4sleep20-21swapper/223:07:102
234462990,2sleep0171ktimers/022:47:220
234462990,2sleep0171ktimers/022:47:220
248502980,2sleep00-21swapper/019:57:260
248502980,2sleep00-21swapper/019:57:260
282922910,2sleep20-21swapper/220:07:212
282922910,2sleep20-21swapper/220:07:212
63872900,1sleep00-21swapper/023:27:260
63872900,1sleep00-21swapper/023:27:260
196332820,2chrt18116-40mandb00:03:011
196332820,2chrt18116-40mandb00:03:011
196332820,2chrt18116-40mandb00:03:011
221442750,1sleep0171ktimers/021:17:220
221442750,1sleep0171ktimers/021:17:220
105742670,4chrt0-21swapper/323:41:563
105742670,4chrt0-21swapper/323:41:563
105742670,4chrt0-21swapper/323:41:563
105742670,4chrt0-21swapper/323:41:563
80272580,1sleep18029-21sed22:07:151
214382560,2sleep00-21swapper/022:42:200
214382560,2sleep00-21swapper/022:42:200
214382560,2sleep00-21swapper/022:42:200
57512500,3sleep10-21swapper/122:02:091
57512500,3sleep10-21swapper/122:02:091
321052500,3sleep20-21swapper/221:43:452
321052500,3sleep20-21swapper/221:43:452
321052500,3sleep20-21swapper/221:43:452
742599483,11cyclictest19536-21unixbench_singl00:02:290
742599483,11cyclictest19536-21unixbench_singl00:02:290
742599483,11cyclictest19536-21unixbench_singl00:02:290
82792440,1sleep08254-21unixbench_multi23:32:250
20672430,2chrt0-21swapper/023:17:170
20672430,2chrt0-21swapper/023:17:170
20672430,2chrt0-21swapper/023:17:170
14882430,2sleep30-21swapper/320:22:213
14882430,2sleep30-21swapper/320:22:213
75362420,4sleep30-21swapper/322:07:053
75362420,4sleep30-21swapper/322:07:053
75362420,4sleep30-21swapper/322:07:053
742599423,6cyclictest550-21cron22:41:020
742599423,6cyclictest550-21cron22:41:020
742599411,11cyclictest16767-21sed23:57:160
742599411,11cyclictest16767-21sed23:57:160
742599411,11cyclictest16767-21sed23:57:160
742599411,11cyclictest15679-21latency_hist21:02:090
742599411,11cyclictest15679-21latency_hist21:02:090
742599411,11cyclictest15679-21latency_hist21:02:090
7425994016,22cyclictest0-21swapper/000:36:280
7425994016,22cyclictest0-21swapper/000:36:280
276992400,3sleep0171ktimers/020:07:150
276992400,3sleep0171ktimers/020:07:150
7425993915,22cyclictest7413-21cyclictest21:27:210
7425993915,22cyclictest7413-21cyclictest21:27:210
742599391,10cyclictest749-21runrttasks19:33:020
742599391,10cyclictest749-21runrttasks19:33:020
743099382,21cyclictest18191-21grep00:02:071
743099382,21cyclictest18191-21grep00:02:071
743099382,21cyclictest18191-21grep00:02:071
7425993812,24cyclictest6172-21cron20:37:030
7425993812,24cyclictest6172-21cron20:37:030
742599381,11cyclictest22760-21ps19:52:230
742599381,10cyclictest27207-21fschecks_time21:32:160
742599381,10cyclictest27207-21fschecks_time21:32:160
742599381,10cyclictest19507-21/usr/sbin/munin21:12:250
742599381,10cyclictest19507-21/usr/sbin/munin21:12:250
31532380,2sleep30-21swapper/323:22:023
31532380,2sleep30-21swapper/323:22:023
31532380,2sleep30-21swapper/323:22:023
303742380,3sleep30-21swapper/320:12:253
303742380,3sleep30-21swapper/320:12:253
742599371,7cyclictest17458-21latency_hist21:07:050
742599371,7cyclictest17458-21latency_hist21:07:050
7425993715,20cyclictest4090-21ntp_offset23:22:210
7425993715,20cyclictest4090-21ntp_offset23:22:210
742599371,12cyclictest18765-21latency_hist22:37:060
742599371,12cyclictest18765-21latency_hist22:37:060
742599371,12cyclictest11359-21memory19:22:180
742599371,12cyclictest11359-21memory19:22:180
742599371,11cyclictest12275-21fschecks_time20:52:160
742599371,10cyclictest749-21runrttasks19:28:480
742599371,10cyclictest749-21runrttasks19:28:480
742599371,10cyclictest28842-21/usr/sbin/munin21:37:140
742599371,10cyclictest28842-21/usr/sbin/munin21:37:140
742599371,10cyclictest28842-21/usr/sbin/munin21:37:140
742599370,11cyclictest3820-21latency_hist21:57:070
742599370,11cyclictest3820-21latency_hist21:57:070
742599361,9cyclictest14919-21taskset22:26:430
742599361,9cyclictest14919-21taskset22:26:430
742599361,9cyclictest14919-21taskset22:26:430
742599361,8cyclictest17731-21df_inode21:07:150
742599361,7cyclictest7655-21acpi22:07:130
742599361,7cyclictest7655-21acpi22:07:130
742599361,7cyclictest7655-21acpi22:07:130
742599361,7cyclictest23106-21seq00:12:250
742599361,7cyclictest23106-21seq00:12:250
742599361,13cyclictest1857-21cron21:52:050
742599361,13cyclictest1857-21cron21:52:050
742599361,13cyclictest1857-21cron21:52:050
742599361,11cyclictest9867-21head22:12:160
742599361,11cyclictest749-21runrttasks20:20:340
742599361,11cyclictest749-21runrttasks20:20:340
742599361,11cyclictest31480-21ntp_states21:42:230
742599361,11cyclictest31480-21ntp_states21:42:230
742599361,11cyclictest31480-21ntp_states21:42:230
742599361,11cyclictest2872-21tune2fs20:27:160
742599361,10cyclictest749-21runrttasks00:29:440
742599361,10cyclictest749-21runrttasks00:29:440
742599361,10cyclictest749-21runrttasks00:29:440
178012360,3sleep30-21swapper/321:07:163
743099359,7cyclictest281ktimers/120:12:281
743099359,7cyclictest281ktimers/120:12:281
743099358,7cyclictest281ktimers/120:38:591
743099358,7cyclictest281ktimers/120:38:591
743099354,4cyclictest281ktimers/123:27:041
743099354,4cyclictest281ktimers/123:27:041
7430993528,4cyclictest281ktimers/119:59:281
7430993528,4cyclictest281ktimers/119:59:281
742599355,8cyclictest20340-21fschecks_count00:07:160
742599355,8cyclictest20340-21fschecks_count00:07:160
742599355,7cyclictest938-21cut20:22:160
742599355,7cyclictest938-21cut20:22:160
7425993528,5cyclictest15054-21kernelversion23:52:180
7425993528,5cyclictest15054-21kernelversion23:52:180
742599352,10cyclictest8498-21fschecks_time20:42:150
742599352,10cyclictest8498-21fschecks_time20:42:150
742599351,9cyclictest25968-21fschecks_count00:22:160
742599351,9cyclictest25968-21fschecks_count00:22:160
742599351,9cyclictest19897-21latency_hist19:47:030
742599351,9cyclictest19897-21latency_hist19:47:030
742599351,8cyclictest9047-21fschecks_count19:17:140
742599351,8cyclictest8966-21cpu23:37:140
742599351,8cyclictest8966-21cpu23:37:140
742599351,8cyclictest8966-21cpu23:37:140
742599351,8cyclictest8966-21cpu23:37:140
742599351,8cyclictest16537-21taskset22:28:330
742599351,8cyclictest16537-21taskset22:28:330
742599351,8cyclictest16537-21taskset22:28:330
7425993518,15cyclictest0-21swapper/000:17:180
7425993518,15cyclictest0-21swapper/000:17:180
7425993518,15cyclictest0-21swapper/000:17:180
742599351,7cyclictest17162-21processes19:37:210
742599351,7cyclictest17162-21processes19:37:210
742599351,7cyclictest10050-21kernelversion22:12:170
742599351,7cyclictest10050-21kernelversion22:12:170
742599351,12cyclictest9927-21latency_hist20:47:150
742599351,12cyclictest9927-21latency_hist20:47:150
742599351,10cyclictest8264-21latency22:07:170
742599351,10cyclictest8264-21latency22:07:170
742599351,10cyclictest749-21runrttasks20:37:460
742599351,10cyclictest749-21runrttasks20:37:460
742599351,10cyclictest30067-21/usr/sbin/munin23:07:170
742599351,10cyclictest30067-21/usr/sbin/munin23:07:170
742599351,10cyclictest30067-21/usr/sbin/munin23:07:170
742599351,10cyclictest24460-21/usr/sbin/munin22:52:170
742599351,10cyclictest24460-21/usr/sbin/munin22:52:170
743399341,23cyclictest18227-21chrt00:02:092
743399341,23cyclictest18227-21chrt00:02:092
743399341,23cyclictest18227-21chrt00:02:092
743099347,4cyclictest281ktimers/119:52:281
7430993427,5cyclictest281ktimers/123:07:331
7430993427,5cyclictest281ktimers/123:07:331
7430993427,5cyclictest281ktimers/123:07:331
7430993424,8cyclictest11641-21processes23:42:231
7430993424,8cyclictest11641-21processes23:42:231
7430993424,8cyclictest11641-21processes23:42:231
7430993424,8cyclictest11641-21processes23:42:231
7425993427,5cyclictest11536-21netstat23:42:200
7425993427,5cyclictest11536-21netstat23:42:200
7425993427,5cyclictest11536-21netstat23:42:200
7425993427,5cyclictest11536-21netstat23:42:200
742599341,9cyclictest749-21runrttasks22:00:390
742599341,9cyclictest749-21runrttasks22:00:390
742599341,9cyclictest29426-21/usr/sbin/munin20:12:210
742599341,9cyclictest29426-21/usr/sbin/munin20:12:210
742599341,9cyclictest26651-21grep22:57:160
742599341,9cyclictest26651-21grep22:57:160
742599341,11cyclictest737-21grep23:12:250
742599341,11cyclictest737-21grep23:12:250
742599341,11cyclictest13654-21switchtime23:47:240
742599341,11cyclictest13654-21switchtime23:47:240
742599341,10cyclictest31421-21/usr/sbin/munin00:37:170
742599341,10cyclictest31421-21/usr/sbin/munin00:37:170
742599341,10cyclictest31421-21/usr/sbin/munin00:37:170
742599341,10cyclictest29893-21grep23:07:020
742599341,10cyclictest29893-21grep23:07:020
742599341,10cyclictest26420-21ntpq20:02:210
742599341,10cyclictest26420-21ntpq20:02:210
742599341,10cyclictest26420-21ntpq20:02:210
742599341,10cyclictest20849-21ntp_states19:47:200
7430993310,4cyclictest281ktimers/121:31:301
7430993310,4cyclictest281ktimers/121:31:301
742599331,10cyclictest23618-21gltestperf21:22:170
742599331,10cyclictest23618-21gltestperf21:22:170
742599331,10cyclictest12346-21mailstats22:17:230
742599331,10cyclictest12346-21mailstats22:17:230
742599331,10cyclictest12346-21mailstats22:17:230
743099327,4cyclictest281ktimers/100:27:361
743099327,4cyclictest281ktimers/100:27:361
743099327,4cyclictest281ktimers/100:27:361
7430993125,4cyclictest281ktimers/122:26:021
7430993125,4cyclictest281ktimers/122:26:021
7430993125,4cyclictest281ktimers/122:26:021
7430993118,3cyclictest181rcu_preempt19:22:181
7430993118,3cyclictest181rcu_preempt19:22:181
7430993117,8cyclictest90-1pr/ttyS021:11:001
42282310,3sleep30-21swapper/321:57:163
42282310,3sleep30-21swapper/321:57:163
743099308,9cyclictest29-21ksoftirqd/121:13:021
743099308,9cyclictest29-21ksoftirqd/121:13:021
743099308,5cyclictest281ktimers/121:34:311
743099308,5cyclictest281ktimers/121:34:311
7430993022,6cyclictest13372-21needreboot23:47:201
7430993022,6cyclictest13372-21needreboot23:47:201
7430993022,6cyclictest12839-21fschecks_time19:27:161
7430993022,6cyclictest12839-21fschecks_time19:27:161
7430993022,5cyclictest281ktimers/122:10:321
7430993022,5cyclictest281ktimers/122:10:321
7430993022,5cyclictest281ktimers/122:10:321
7430993020,3cyclictest181rcu_preempt20:48:261
7430993020,3cyclictest181rcu_preempt20:48:261
743099299,4cyclictest281ktimers/123:40:351
743099299,4cyclictest281ktimers/123:40:351
743099299,4cyclictest281ktimers/123:40:351
743099299,4cyclictest281ktimers/123:40:351
743099297,5cyclictest281ktimers/122:52:331
743099297,5cyclictest281ktimers/122:52:331
743099296,5cyclictest281ktimers/119:21:441
7430992925,3cyclictest18978-21ntp_offset19:42:211
7430992925,3cyclictest18978-21ntp_offset19:42:211
7430992924,3cyclictest90-1pr/ttyS022:29:021
7430992924,3cyclictest90-1pr/ttyS022:29:021
7430992924,3cyclictest90-1pr/ttyS022:29:021
7430992921,6cyclictest19822-21cut22:37:231
7430992921,6cyclictest19822-21cut22:37:231
7430992921,3cyclictest181rcu_preempt23:53:261
7430992921,3cyclictest181rcu_preempt23:53:261
7430992919,4cyclictest181rcu_preempt20:22:251
7430992919,4cyclictest181rcu_preempt20:22:251
7430992919,2cyclictest181rcu_preempt20:42:171
7430992919,2cyclictest181rcu_preempt20:42:171
246122290,3sleep20-21swapper/219:57:242
246122290,3sleep20-21swapper/219:57:242
100962290,3sleep20-21swapper/219:17:242
743099286,9cyclictest29-21ksoftirqd/122:48:091
743099286,9cyclictest29-21ksoftirqd/122:48:091
7430992825,2cyclictest90-1pr/ttyS023:29:041
7430992825,2cyclictest90-1pr/ttyS023:29:041
7430992823,4cyclictest90-1pr/ttyS023:16:041
7430992823,4cyclictest90-1pr/ttyS023:16:041
7430992822,5cyclictest11701-21cut22:17:161
7430992822,5cyclictest11701-21cut22:17:161
7430992822,5cyclictest11701-21cut22:17:161
7430992820,3cyclictest181rcu_preempt21:37:241
7430992820,3cyclictest181rcu_preempt21:37:241
7430992820,3cyclictest181rcu_preempt21:37:241
7430992816,3cyclictest90-1pr/ttyS021:44:561
7430992816,3cyclictest90-1pr/ttyS021:44:561
7430992816,3cyclictest90-1pr/ttyS021:44:561
33102280,3sleep30-21swapper/321:52:263
33102280,3sleep30-21swapper/321:52:263
743099278,17cyclictest271rcuc/123:17:151
743099278,17cyclictest271rcuc/123:17:151
743099278,17cyclictest271rcuc/123:17:151
743099277,4cyclictest281ktimers/121:23:001
743099277,4cyclictest281ktimers/121:23:001
743099274,3cyclictest271rcuc/122:13:561
743099274,3cyclictest271rcuc/122:13:561
7430992723,2cyclictest0-21swapper/120:32:191
7430992723,2cyclictest0-21swapper/120:32:191
7430992721,4cyclictest21122-21head22:42:171
7430992721,4cyclictest21122-21head22:42:171
7430992721,4cyclictest21122-21head22:42:171
7430992721,3cyclictest0-21swapper/122:57:151
7430992721,3cyclictest0-21swapper/122:57:151
7430992720,2cyclictest181rcu_preempt23:02:201
7430992720,2cyclictest181rcu_preempt23:02:201
7430992720,2cyclictest181rcu_preempt20:02:241
7430992720,2cyclictest181rcu_preempt20:02:241
7430992720,2cyclictest181rcu_preempt20:02:241
7430992720,2cyclictest181rcu_preempt00:37:251
7430992720,2cyclictest181rcu_preempt00:37:251
7430992720,2cyclictest181rcu_preempt00:37:251
7430992719,3cyclictest181rcu_preempt21:02:201
7430992719,3cyclictest181rcu_preempt21:02:201
743099271,8cyclictest29-21ksoftirqd/121:57:011
743099271,8cyclictest29-21ksoftirqd/121:57:011
7430992714,4cyclictest181rcu_preempt20:07:461
7430992714,4cyclictest181rcu_preempt20:07:461
7430992714,3cyclictest90-1pr/ttyS022:03:561
7430992714,3cyclictest90-1pr/ttyS022:03:561
743099264,3cyclictest271rcuc/119:50:491
7430992619,2cyclictest181rcu_preempt19:32:161
7430992619,2cyclictest181rcu_preempt19:32:161
743099260,8cyclictest281ktimers/100:33:131
743099260,8cyclictest281ktimers/100:33:131
743399252,19cyclictest13291-21chrt20:52:272
7430992520,3cyclictest271rcuc/100:07:401
7430992520,3cyclictest271rcuc/100:07:401
7430992520,3cyclictest0-21swapper/121:17:151
7430992520,3cyclictest0-21swapper/121:17:151
7430992519,4cyclictest930-21ntp_kernel_pll_21:47:211
7430992519,4cyclictest930-21ntp_kernel_pll_21:47:211
7430992519,4cyclictest930-21ntp_kernel_pll_21:47:211
7430992519,3cyclictest0-21swapper/120:17:251
7430992519,3cyclictest0-21swapper/120:17:251
7430992518,5cyclictest23075-21timerandwakeup00:12:251
7430992518,5cyclictest23075-21timerandwakeup00:12:251
7430992518,2cyclictest181rcu_preempt20:57:251
7430992518,2cyclictest181rcu_preempt20:57:251
7430992518,2cyclictest181rcu_preempt20:57:251
7430992518,2cyclictest181rcu_preempt20:27:231
7430992512,3cyclictest90-1pr/ttyS000:18:561
7430992512,3cyclictest90-1pr/ttyS000:18:561
7430992512,3cyclictest90-1pr/ttyS000:18:561
7430992512,2cyclictest181rcu_preempt22:32:161
7430992512,2cyclictest181rcu_preempt22:32:161
743099242,7cyclictest29-21ksoftirqd/123:33:041
7430992417,2cyclictest181rcu_preempt00:22:191
7430992417,2cyclictest181rcu_preempt00:22:191
7430992415,2cyclictest181rcu_preempt20:53:001
7433992319,3cyclictest351rcuc/223:44:262
7433992319,3cyclictest351rcuc/223:44:262
7433992319,3cyclictest351rcuc/223:44:262
7433992319,3cyclictest351rcuc/223:44:262
7433992319,3cyclictest351rcuc/221:04:302
7433992319,3cyclictest351rcuc/221:04:302
7433992319,3cyclictest351rcuc/200:35:032
7433992319,3cyclictest351rcuc/200:35:032
7433992319,3cyclictest351rcuc/200:14:202
7433992319,3cyclictest351rcuc/200:14:202
7433992317,5cyclictest351rcuc/220:29:212
7430992310,4cyclictest181rcu_preempt19:38:491
7430992310,4cyclictest181rcu_preempt19:38:491
271872230,3sleep30-21swapper/321:32:163
271872230,3sleep30-21swapper/321:32:163
7433992218,3cyclictest351rcuc/221:28:142
7433992218,3cyclictest351rcuc/221:28:142
7433992218,3cyclictest351rcuc/200:11:342
7433992218,3cyclictest351rcuc/200:11:342
7433992217,3cyclictest351rcuc/223:52:182
7433992217,3cyclictest351rcuc/223:52:182
7433992217,3cyclictest351rcuc/221:07:552
743499213,11cyclictest0-21swapper/300:02:083
743499213,11cyclictest0-21swapper/300:02:083
743499213,11cyclictest0-21swapper/300:02:083
7433992116,3cyclictest351rcuc/222:15:462
7433992116,3cyclictest351rcuc/222:15:462
7433992016,3cyclictest351rcuc/223:47:552
7433992016,3cyclictest351rcuc/223:47:552
7433992016,3cyclictest351rcuc/223:34:042
7433992016,3cyclictest351rcuc/222:56:582
7433992016,3cyclictest351rcuc/222:56:582
7433992016,3cyclictest351rcuc/220:26:402
7433992016,3cyclictest351rcuc/220:26:402
7433992016,3cyclictest351rcuc/200:20:562
7433992016,3cyclictest351rcuc/200:20:562
7433992016,3cyclictest351rcuc/200:20:562
7433992015,4cyclictest351rcuc/221:12:462
7433992015,4cyclictest351rcuc/221:12:462
7433992015,3cyclictest351rcuc/221:56:032
7433992015,3cyclictest351rcuc/221:56:032
7433992015,3cyclictest351rcuc/220:52:042
7433992015,3cyclictest351rcuc/220:52:042
7433992014,4cyclictest351rcuc/223:30:272
7433992014,4cyclictest351rcuc/223:30:272
743499193,12cyclictest0-21swapper/322:33:553
743499193,12cyclictest0-21swapper/322:33:553
7434991910,6cyclictest357-21taskset21:47:163
7434991910,6cyclictest357-21taskset21:47:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional