You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 09:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Tue Jul 07, 2026 00:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1151421170,2sleep20-21swapper/223:08:312
1151421170,2sleep20-21swapper/223:08:312
156472930,2chrt0-21swapper/320:28:253
156472930,2chrt0-21swapper/320:28:253
128582720,2sleep20-21swapper/200:43:222
174222690,3sleep30-21swapper/320:33:193
174222690,3sleep30-21swapper/320:33:193
14562650,2chrt20341-21munin-node22:43:250
14562650,2chrt20341-21munin-node22:43:250
210562630,3sleep30-21swapper/322:08:263
210562630,3sleep30-21swapper/322:08:263
210562630,3sleep30-21swapper/322:08:263
312762620,1sleep122700-21kworker/u9:0+xprtiod00:03:181
312762620,1sleep122700-21kworker/u9:0+xprtiod00:03:181
312762620,1sleep122700-21kworker/u9:0+xprtiod00:03:181
127762590,2chrt12779-21cpuspeed00:43:211
127762590,2chrt12779-21cpuspeed00:43:211
310692580,3sleep20-21swapper/219:43:172
310692580,3sleep20-21swapper/219:43:172
85192560,1sleep0595-21sshd23:00:460
85192560,1sleep0595-21sshd23:00:460
106142550,2chrt10610-21sed00:33:281
106142550,2chrt10610-21sed00:33:281
68452540,3chrt0-21swapper/321:28:343
68452540,3chrt0-21swapper/321:28:343
202052540,3sleep10-21swapper/123:33:221
132522540,3sleep20-21swapper/221:48:232
132522540,3sleep20-21swapper/221:48:232
70102520,3sleep30-21swapper/320:03:323
70102520,3sleep30-21swapper/320:03:323
278892500,10sleep1281ktimers/119:33:271
278892500,10sleep1281ktimers/119:33:271
88862450,1sleep10-21swapper/121:33:391
88862450,1sleep10-21swapper/121:33:391
304922450,3sleep30-21swapper/321:08:163
304922450,3sleep30-21swapper/321:08:163
304922450,3sleep30-21swapper/321:08:163
292912450,2chrt29288-21cat19:38:210
21239994527,16cyclictest3668-21head22:48:271
21239994527,16cyclictest3668-21head22:48:271
21239994527,16cyclictest3668-21head22:48:271
135452450,1chrt0-21swapper/121:48:261
21239994318,23cyclictest23524-21kworker/u9:2-xprtiod00:03:321
21239994318,23cyclictest23524-21kworker/u9:2-xprtiod00:03:321
137432430,2sleep013913-21grep23:13:370
137432430,2sleep013913-21grep23:13:370
104332420,2sleep010436-21fschecks_count00:33:270
104332420,2sleep010436-21fschecks_count00:33:270
314722410,5sleep031468-21df_inode00:03:240
314722410,5sleep031468-21df_inode00:03:240
314722410,5sleep031468-21df_inode00:03:240
21239994115,24cyclictest0-21swapper/123:38:361
21239994020,18cyclictest2441-21latency_hist19:53:211
21239994020,18cyclictest2441-21latency_hist19:53:211
21239994017,21cyclictest0-21swapper/119:53:361
21239994017,21cyclictest0-21swapper/119:53:361
21239994012,25cyclictest0-21swapper/120:43:351
21239994012,25cyclictest0-21swapper/120:43:351
2123899404,29cyclictest32309-21find00:03:350
2123899404,29cyclictest32309-21find00:03:350
21239993916,21cyclictest0-21swapper/120:13:341
21239993916,21cyclictest0-21swapper/120:13:341
2123999391,13cyclictest11560-21ntpq23:08:321
2123999391,13cyclictest11560-21ntpq23:08:321
2123999391,12cyclictest16399-21latency_hist23:23:221
2123999391,12cyclictest16399-21latency_hist23:23:221
2123999391,11cyclictest7134-21grep20:03:341
2123999391,11cyclictest7134-21grep20:03:341
109612390,2sleep00-21swapper/020:13:350
109612390,2sleep00-21swapper/020:13:350
85332380,2chrt0-21swapper/300:28:283
85332380,2chrt0-21swapper/300:28:283
21239993820,16cyclictest0-21swapper/122:28:371
21239993820,16cyclictest0-21swapper/122:28:371
21239993816,20cyclictest0-21swapper/120:38:341
21239993816,20cyclictest0-21swapper/120:38:341
2123999381,12cyclictest8372-21/usr/sbin/munin00:28:261
2123999381,12cyclictest8372-21/usr/sbin/munin00:28:261
2123999381,11cyclictest9920-21mailstats23:03:361
2123999381,11cyclictest9920-21mailstats23:03:361
247902370,3sleep30-21swapper/322:18:273
247902370,3sleep30-21swapper/322:18:273
2123999379,5cyclictest10049-21cron00:33:141
2123999379,5cyclictest10049-21cron00:33:141
2123999378,6cyclictest20877-21head23:33:311
2123999378,6cyclictest20877-21head23:33:311
2123999378,5cyclictest5939-21fschecks_count21:28:251
2123999378,5cyclictest5939-21fschecks_count21:28:251
2123999378,5cyclictest5939-21fschecks_count21:28:251
2123999377,7cyclictest7460-21sensors00:23:341
21239993727,8cyclictest32213-21threads19:43:351
21239993727,8cyclictest32213-21threads19:43:351
21239993727,8cyclictest32213-21threads19:43:351
21239993712,4cyclictest281ktimers/122:43:351
21239993712,4cyclictest281ktimers/122:43:351
2123999371,14cyclictest17412-21ls20:33:181
2123999371,14cyclictest17412-21ls20:33:181
2123999371,13cyclictest22978-21unixbench-2d19:18:351
2123999371,13cyclictest22978-21unixbench-2d19:18:351
2123999371,12cyclictest25367-21tail20:53:281
2123999371,11cyclictest28062-21ls22:28:131
2123999371,11cyclictest28062-21ls22:28:131
2123999371,11cyclictest28062-21ls22:28:131
21238993730,5cyclictest28554-21latency_hist21:03:140
21238993730,5cyclictest28554-21latency_hist21:03:140
2123999367,7cyclictest6751-21cut00:23:251
2123999367,7cyclictest6751-21cut00:23:251
21239993615,19cyclictest0-21swapper/123:28:391
21239993614,2cyclictest0-21swapper/120:18:271
21239993614,2cyclictest0-21swapper/120:18:271
2123999361,15cyclictest20047-21ls22:03:361
2123999361,15cyclictest20047-21ls22:03:361
2123999361,15cyclictest20047-21ls22:03:361
2123999361,13cyclictest30462-21date21:08:141
2123999361,13cyclictest30462-21date21:08:141
2123999361,13cyclictest30462-21date21:08:141
2123999361,12cyclictest24474-21/usr/sbin/munin22:18:361
2123999361,12cyclictest24474-21/usr/sbin/munin22:18:361
2123999361,12cyclictest24474-21/usr/sbin/munin22:18:361
2123999361,11cyclictest29582-21fschecks_time19:38:261
2123999361,11cyclictest29582-21fschecks_time19:38:261
2123999361,11cyclictest21854-21timerandwakeup22:08:351
2123999361,11cyclictest21854-21timerandwakeup22:08:351
2123999361,11cyclictest21854-21timerandwakeup22:08:351
2123999361,10cyclictest749-21runrttasks22:43:211
2123999361,10cyclictest749-21runrttasks22:43:211
21238993627,7cyclictest23451-21fschecks_time20:48:280
21238993627,7cyclictest23451-21fschecks_time20:48:280
2123999351,13cyclictest5297-21grep19:58:381
2123999351,13cyclictest5297-21grep19:58:381
21239993511,3cyclictest0-21swapper/122:53:361
21239993511,3cyclictest0-21swapper/122:53:361
21239993511,3cyclictest0-21swapper/122:53:361
2123999351,11cyclictest27551-21latency_hist23:53:191
2123999351,11cyclictest27551-21latency_hist23:53:191
2123999351,11cyclictest27551-21latency_hist23:53:191
2123999351,11cyclictest17765-21tune2fs20:33:261
2123999351,11cyclictest17765-21tune2fs20:33:261
2123999351,11cyclictest17237-21ntp_states23:23:341
2123999351,11cyclictest17237-21ntp_states23:23:341
2123999350,12cyclictest14069-21fschecks_time20:23:281
2123999350,12cyclictest14069-21fschecks_time20:23:281
21239993415,16cyclictest0-21swapper/122:18:281
21239993414,2cyclictest0-21swapper/121:18:271
21239993414,2cyclictest0-21swapper/121:18:271
21239993414,2cyclictest0-21swapper/121:18:271
2123999341,13cyclictest8124-21/usr/sbin/munin20:08:321
2123999341,13cyclictest8124-21/usr/sbin/munin20:08:321
2123999341,13cyclictest26300-21awk19:28:331
2123999341,11cyclictest955-21ps21:13:351
2123999341,11cyclictest955-21ps21:13:351
2123999341,11cyclictest749-21runrttasks22:36:361
2123999341,11cyclictest749-21runrttasks22:36:361
2123999341,11cyclictest749-21runrttasks00:12:431
2123999341,11cyclictest749-21runrttasks00:12:431
2123999341,11cyclictest749-21runrttasks00:12:431
2123999341,10cyclictest749-21runrttasks23:16:371
2123999341,10cyclictest749-21runrttasks23:16:371
21238993416,10cyclictest90-1pr/ttyS021:49:210
21238993416,10cyclictest90-1pr/ttyS021:49:210
21238993416,10cyclictest90-1pr/ttyS021:49:210
2123999331,12cyclictest7082-21/usr/sbin/munin22:58:361
2123999331,12cyclictest7082-21/usr/sbin/munin22:58:361
2123999331,12cyclictest24691-21fschecks_count22:18:271
2123999331,12cyclictest24691-21fschecks_count22:18:271
2123999331,11cyclictest28427-21sshd21:01:401
2123999331,11cyclictest28427-21sshd21:01:401
2123999331,11cyclictest24025-21grep19:23:291
2123999331,11cyclictest24025-21grep19:23:291
2123999331,11cyclictest24025-21grep19:23:291
2123999331,11cyclictest23192-21/usr/sbin/munin20:48:341
2123999331,11cyclictest23192-21/usr/sbin/munin20:48:341
21238993326,5cyclictest6226-21threads22:53:350
21238993326,5cyclictest6226-21threads22:53:350
21238993326,5cyclictest6226-21threads22:53:350
21238993325,6cyclictest773-21apache221:13:330
21238993325,6cyclictest773-21apache221:13:330
21239993225,5cyclictest11764-21gltestperf21:43:271
2123999321,12cyclictest30659-21/usr/sbin/munin21:08:331
2123999321,12cyclictest30659-21/usr/sbin/munin21:08:331
2123999321,11cyclictest18299-21unixbench_singl21:58:391
2123999321,11cyclictest18299-21unixbench_singl21:58:391
2123999321,11cyclictest18299-21unixbench_singl21:58:391
21238993225,5cyclictest29100-21latency_hist19:38:130
2123999311,11cyclictest749-21runrttasks23:47:461
2123999311,11cyclictest749-21runrttasks23:47:461
2123999311,11cyclictest749-21runrttasks23:47:461
2124099301,20cyclictest32097-21chrt00:03:302
21239993022,6cyclictest13826-21kernelversion21:48:291
21239993022,6cyclictest13826-21kernelversion21:48:291
21239993022,6cyclictest13826-21kernelversion21:48:291
21238993025,3cyclictest0-21swapper/020:43:130
21238993025,3cyclictest0-21swapper/020:43:130
21238993023,5cyclictest572-21cups-browsed23:08:350
21238993023,5cyclictest572-21cups-browsed23:08:350
265272290,4sleep30-21swapper/322:23:263
265272290,4sleep30-21swapper/322:23:263
265272290,4sleep30-21swapper/322:23:263
21238992923,5cyclictest19588-21needreboot22:03:320
21238992923,5cyclictest19588-21needreboot22:03:320
21238992923,5cyclictest19588-21needreboot22:03:320
21238992923,5cyclictest1213-21sendmail22:43:130
21238992923,5cyclictest1213-21sendmail22:43:130
21238992922,5cyclictest29604-21cpuspeed23:58:260
21238992922,5cyclictest29604-21cpuspeed23:58:260
21238992920,7cyclictest2890-21sendmail_mailqu21:18:340
21238992920,7cyclictest2890-21sendmail_mailqu21:18:340
21238992920,7cyclictest2890-21sendmail_mailqu21:18:340
21238992915,12cyclictest0-21swapper/000:43:220
288642280,2sleep128865-21awk23:53:361
288642280,2sleep128865-21awk23:53:361
21240992822,2cyclictest0-21swapper/220:51:032
21240992822,2cyclictest0-21swapper/220:51:032
21238992823,3cyclictest21237-21cyclictest21:43:340
21238992823,3cyclictest21237-21cyclictest21:43:340
21238992822,5cyclictest9487-21kernelversion23:03:300
21238992822,5cyclictest9487-21kernelversion23:03:300
21238992822,4cyclictest9573-21sshd20:10:200
21238992822,4cyclictest9573-21sshd20:10:200
21238992821,5cyclictest7130-21rt-features20:03:340
21238992821,5cyclictest7130-21rt-features20:03:340
21238992819,7cyclictest2640-21fschecks_count19:53:260
21238992819,7cyclictest2640-21fschecks_count19:53:260
244222270,3chrt0-21swapper/022:18:200
244222270,3chrt0-21swapper/022:18:200
244222270,3chrt0-21swapper/022:18:200
21239992722,3cyclictest0-21swapper/121:38:391
21239992722,3cyclictest0-21swapper/121:38:391
21239992721,4cyclictest15543-21cut21:53:271
21239992721,4cyclictest15543-21cut21:53:271
21239992721,4cyclictest15543-21cut21:53:271
21238992723,3cyclictest29164-21gltestperf21:03:280
21238992723,3cyclictest29164-21gltestperf21:03:280
21238992723,3cyclictest29164-21gltestperf21:03:280
21238992722,4cyclictest1346-21avahi-daemon19:28:360
21238992721,4cyclictest24786-21timerandwakeup19:23:380
21238992721,4cyclictest24786-21timerandwakeup19:23:380
21238992721,4cyclictest24786-21timerandwakeup19:23:380
21238992721,4cyclictest22963-21systemctl23:38:340
21238992720,5cyclictest6356-21latency21:28:280
21238992720,5cyclictest6356-21latency21:28:280
21238992720,2cyclictest181rcu_preempt21:40:360
21238992720,2cyclictest181rcu_preempt21:40:360
21238992716,10cyclictest749-21runrttasks23:43:270
21238992716,10cyclictest749-21runrttasks23:43:270
21238992716,10cyclictest749-21runrttasks23:43:270
198642270,3sleep20-21swapper/220:38:292
198642270,3sleep20-21swapper/220:38:292
2124199266,16cyclictest90612chrt21:34:463
2124199266,16cyclictest90612chrt21:34:463
2124199261,8cyclictest27084-21chrt20:58:283
2124199261,8cyclictest27084-21chrt20:58:283
2124199261,8cyclictest14660-21chrt21:49:213
2124199261,8cyclictest14660-21chrt21:49:213
2124199261,8cyclictest14660-21chrt21:49:213
21240992622,2cyclictest0-21swapper/222:30:132
21240992622,2cyclictest0-21swapper/222:30:132
2123899267,3cyclictest90-1pr/ttyS019:48:500
2123899267,3cyclictest90-1pr/ttyS019:48:500
21238992621,4cyclictest25553-21kernelversion20:53:300
21238992620,5cyclictest26778-21switchtime23:48:350
21238992620,5cyclictest26778-21switchtime23:48:350
21238992620,5cyclictest26778-21switchtime23:48:350
21238992620,5cyclictest17576-21latency21:58:310
21238992620,5cyclictest17576-21latency21:58:310
21238992620,5cyclictest17576-21latency21:58:310
21238992620,4cyclictest5921-21fschecks_count21:28:250
21238992620,4cyclictest5921-21fschecks_count21:28:250
21238992620,4cyclictest5921-21fschecks_count21:28:250
2123899261,21cyclictest201rcuc/019:47:530
2123899261,21cyclictest201rcuc/019:47:530
2123899261,21cyclictest201rcuc/019:47:530
47252250,3sleep20-21swapper/219:58:292
47252250,3sleep20-21swapper/219:58:292
2124199257,12cyclictest30044-21chrt19:38:323
2124199257,12cyclictest30044-21chrt19:38:323
2124199251,9cyclictest2065-21chrt19:50:303
2124199251,9cyclictest2065-21chrt19:50:303
2124199251,9cyclictest12940-21chrt21:46:313
2124199251,9cyclictest12940-21chrt21:46:313
2124199251,8cyclictest23521-21chrt23:40:443
2124199251,8cyclictest21651-21chrt23:35:373
2124199251,8cyclictest21651-21chrt23:35:373
2124199251,8cyclictest1765-21chrt21:17:533
2124199251,8cyclictest1765-21chrt21:17:533
2124199251,7cyclictest29978-21chrt23:58:293
2124199251,7cyclictest29978-21chrt23:58:293
2124199251,7cyclictest29978-21chrt23:58:293
2124199251,7cyclictest1624-21chrt00:08:343
2124199251,7cyclictest1624-21chrt00:08:343
2124199251,7cyclictest1624-21chrt00:08:343
2124199251,4cyclictest8939-21chrt23:03:253
2124199251,4cyclictest8939-21chrt23:03:253
2124199251,4cyclictest5675-21chrt19:58:423
2124199251,4cyclictest5675-21chrt19:58:423
2124199251,4cyclictest221292sleep322:09:153
2124199251,4cyclictest221292sleep322:09:153
2124199251,4cyclictest221292sleep322:09:153
2124199251,4cyclictest19023-21chrt20:36:253
2124199251,4cyclictest19023-21chrt20:36:253
2124199251,4cyclictest11470-21chrt20:15:463
2124199251,4cyclictest11470-21chrt20:15:463
2124199251,4cyclictest0-21swapper/321:08:263
2124199251,4cyclictest0-21swapper/321:08:263
2124199251,2cyclictest0-21swapper/300:18:593
2124199251,2cyclictest0-21swapper/300:18:593
2124199250,4cyclictest241882sleep322:16:463
21240992515,6cyclictest6548-21chrt22:54:202
21240992515,6cyclictest6548-21chrt22:54:202
21240992515,6cyclictest6548-21chrt22:54:202
21238992519,5cyclictest14678-21systemd-journal22:30:390
21238992519,5cyclictest14678-21systemd-journal22:30:390
21238992519,4cyclictest0-21swapper/020:43:270
21238992519,4cyclictest0-21swapper/020:43:270
21238992518,5cyclictest8690-21timerandwakeup21:33:380
21238992518,5cyclictest8690-21timerandwakeup21:33:380
21238992518,5cyclictest12513-21latency_hist00:43:210
21238992515,7cyclictest201rcuc/000:21:330
21238992515,7cyclictest201rcuc/000:21:330
2124199249,12cyclictest0-21swapper/321:53:273
2124199249,12cyclictest0-21swapper/321:53:273
2124199249,12cyclictest0-21swapper/321:53:273
2124199246,14cyclictest121752chrt23:09:163
2124199246,14cyclictest121752chrt23:09:163
2124199242,6cyclictest0-21swapper/323:18:313
2124199242,6cyclictest0-21swapper/323:18:313
2124199241,9cyclictest26447-21chrt20:55:063
2124199241,7cyclictest5259-21chrt21:24:083
2124199241,7cyclictest5259-21chrt21:24:083
2124199241,7cyclictest5259-21chrt21:24:083
2124199241,7cyclictest4869-21chrt22:52:043
2124199241,7cyclictest4869-21chrt22:52:043
2124199241,7cyclictest4869-21chrt22:52:043
2124199241,7cyclictest28988-21chrt23:53:553
2124199241,7cyclictest28988-21chrt23:53:553
2124199241,7cyclictest26920-21chrt19:29:153
2124199241,7cyclictest18059-21chrt21:58:373
2124199241,7cyclictest18059-21chrt21:58:373
2124199241,7cyclictest18059-21chrt21:58:373
2124199241,6cyclictest4077-21chrt00:14:213
2124199241,6cyclictest4077-21chrt00:14:213
2124199241,6cyclictest0-21swapper/321:18:313
2124199241,6cyclictest0-21swapper/321:18:313
2124199241,6cyclictest0-21swapper/321:18:313
2124199241,6cyclictest0-21swapper/319:33:333
2124199241,6cyclictest0-21swapper/319:33:333
2124199241,4cyclictest9527-21chrt20:09:433
2124199241,4cyclictest9527-21chrt20:09:433
2124199241,4cyclictest8712sleep322:38:573
2124199241,4cyclictest8712sleep322:38:573
2124199241,4cyclictest6777-21chrt22:57:183
2124199241,4cyclictest6777-21chrt22:57:183
2124199241,4cyclictest6777-21chrt22:57:183
2124199241,4cyclictest3861-21chrt19:54:403
2124199241,4cyclictest3861-21chrt19:54:403
2124199241,4cyclictest296882sleep322:30:213
2124199241,4cyclictest296882sleep322:30:213
2124199241,4cyclictest25447-21chrt23:46:153
2124199241,4cyclictest25447-21chrt23:46:153
2124199241,4cyclictest25447-21chrt23:46:153
2124199241,4cyclictest25065-21chrt19:24:213
2124199241,4cyclictest25065-21chrt19:24:213
2124199241,4cyclictest25065-21chrt19:24:213
2124199241,4cyclictest23132-21chrt19:18:503
2124199241,4cyclictest23132-21chrt19:18:503
2124199241,4cyclictest207062sleep320:39:063
2124199241,4cyclictest207062sleep320:39:063
2124199241,4cyclictest19851-21chrt23:31:313
2124199241,4cyclictest19851-21chrt23:31:313
2124199241,4cyclictest133452sleep320:20:533
2124199241,4cyclictest133452sleep320:20:533
2124199241,4cyclictest103012sleep323:04:063
2124199241,4cyclictest103012sleep323:04:063
2124199241,4cyclictest0-21swapper/322:43:293
2124199241,4cyclictest0-21swapper/322:43:293
2124199241,3cyclictest277992chrt22:24:583
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional