You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-15 - 01:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Sun Jun 14, 2026 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31482790,2chrt3149-21grep12:38:160
31482790,2chrt3149-21grep12:38:160
31482790,2chrt3149-21grep12:38:160
154312710,2sleep20-21swapper/211:47:532
154312710,2sleep20-21swapper/211:47:532
154312710,2sleep20-21swapper/211:47:532
273702620,2sleep027480-21df10:53:040
273702620,2sleep027480-21df10:53:040
273702620,2sleep027480-21df10:53:040
166482590,2chrt0-21swapper/008:58:020
166482590,2chrt0-21swapper/008:58:020
141462570,2chrt14135-21awk10:17:540
231352560,2sleep00-21swapper/007:48:070
231352560,2sleep00-21swapper/007:48:070
231352560,2sleep00-21swapper/007:48:070
56622550,7sleep0171ktimers/008:28:070
56622550,7sleep0171ktimers/008:28:070
56622550,7sleep0171ktimers/008:28:070
215162540,2sleep021619-21missed_timers07:43:090
215162540,2sleep021619-21missed_timers07:43:090
38062530,1sleep13008-21/usr/sbin/munin09:48:131
38062530,1sleep13008-21/usr/sbin/munin09:48:131
182822530,2sleep018286-21sed07:33:150
182822530,2sleep018286-21sed07:33:150
214592520,7sleep021463-21fschecks_time12:03:060
214592520,7sleep021463-21fschecks_time12:03:060
301472480,2sleep030152-21gltestperf09:33:070
301472480,2sleep030152-21gltestperf09:33:070
258982440,3sleep20-21swapper/207:53:152
258982440,3sleep20-21swapper/207:53:152
130042440,3sleep30-21swapper/308:48:043
60302430,4sleep30-21swapper/308:28:103
60302430,4sleep30-21swapper/308:28:103
60302430,4sleep30-21swapper/308:28:103
14752430,2sleep10-21swapper/108:17:561
14752430,2sleep10-21swapper/108:17:561
72102410,2sleep07213-21grep11:23:140
72102410,2sleep07213-21grep11:23:140
58472380,2sleep15963-21find09:53:151
58472380,2sleep15963-21find09:53:151
236022380,6chrt23600-21head09:13:170
236022380,6chrt23600-21head09:13:170
139072380,3sleep10-21swapper/108:48:141
139072380,3sleep10-21swapper/108:48:141
10262993830,6cyclictest14983-21unixbench_multi11:43:161
10262993830,6cyclictest14983-21unixbench_multi11:43:161
10262993830,6cyclictest14983-21unixbench_multi11:43:161
57312370,2sleep00-21swapper/009:53:140
57312370,2sleep00-21swapper/009:53:140
224892370,1sleep122485-21strings10:38:091
224892370,1sleep122485-21strings10:38:091
18162360,1sleep11864-21awk09:43:101
18162360,1sleep11864-21awk09:43:101
18162360,1sleep11864-21awk09:43:101
10261993628,6cyclictest27116-21threads09:23:150
10261993628,6cyclictest27116-21threads09:23:150
10261993628,6cyclictest27116-21threads09:23:150
204292350,1sleep019873-21/usr/sbin/munin10:33:060
10261993527,6cyclictest550-21cron10:47:530
10261993527,6cyclictest550-21cron10:47:530
10261993527,6cyclictest550-21cron10:47:530
10261993527,6cyclictest550-21cron10:47:530
10262993428,4cyclictest547-21rs:main1
10262993428,4cyclictest547-21rs:main1
10262993425,7cyclictest25432-21latency_hist10:47:571
10262993425,7cyclictest25432-21latency_hist10:47:571
10262993425,7cyclictest25432-21latency_hist10:47:571
10262993425,7cyclictest25432-21latency_hist10:47:571
10262993423,3cyclictest181rcu_preempt07:54:141
10262993423,3cyclictest181rcu_preempt07:54:141
10262993421,3cyclictest181rcu_preempt12:03:451
10262993421,3cyclictest181rcu_preempt12:03:451
10262993421,3cyclictest181rcu_preempt12:03:451
10261993427,5cyclictest4566-21fschecks_time11:18:050
263252330,2sleep026495-21if_eth009:23:070
263252330,2sleep026495-21if_eth009:23:070
263252330,2sleep026495-21if_eth009:23:070
10262993327,5cyclictest19386-21processes09:03:131
10262993327,5cyclictest19386-21processes09:03:131
10262993326,5cyclictest18777-21needreboot10:28:131
10262993326,5cyclictest18777-21needreboot10:28:131
10262993326,5cyclictest18777-21needreboot10:28:131
10262993324,7cyclictest2567-21fschecks_count11:13:041
10262993324,7cyclictest2567-21fschecks_count11:13:041
10261993319,11cyclictest0-21swapper/012:23:060
10261993319,11cyclictest0-21swapper/012:23:060
10261993319,11cyclictest0-21swapper/012:23:060
94492320,1sleep19476-21ntp_states10:03:121
94492320,1sleep19476-21ntp_states10:03:121
84182320,3sleep20-21swapper/211:28:072
84182320,3sleep20-21swapper/211:28:072
84182320,3sleep20-21swapper/211:28:072
10262993228,3cyclictest17499-21processes08:58:121
10262993228,3cyclictest17499-21processes08:58:121
10262993228,3cyclictest10344-21head11:33:051
10262993228,3cyclictest10344-21head11:33:051
10262993225,5cyclictest4506-21fschecks_count11:18:051
10262993225,5cyclictest30740-21cut12:28:041
10262993225,5cyclictest30740-21cut12:28:041
10262993211,5cyclictest281ktimers/108:40:211
10262993211,5cyclictest281ktimers/108:40:211
10261993226,5cyclictest31268-21needreboot12:28:100
10261993226,5cyclictest31268-21needreboot12:28:100
10261993226,4cyclictest9944-21cat11:33:020
10261993226,4cyclictest9944-21cat11:33:020
10261993226,4cyclictest9944-21cat11:33:020
10261993222,8cyclictest11303-21netstat10:08:140
10261993222,8cyclictest11303-21netstat10:08:140
10261993222,8cyclictest11303-21netstat10:08:140
10261993222,7cyclictest2262-21ls09:43:150
10261993222,7cyclictest2262-21ls09:43:150
10261993222,7cyclictest2262-21ls09:43:150
244572310,3sleep30-21swapper/310:43:143
244572310,3sleep30-21swapper/310:43:143
244572310,3sleep30-21swapper/310:43:143
244572310,3sleep30-21swapper/310:43:143
10262993125,4cyclictest30198-21cut09:33:071
10262993125,4cyclictest30198-21cut09:33:071
10262993124,5cyclictest1739-21latency_hist12:37:571
10262993124,5cyclictest1739-21latency_hist12:37:571
10262993124,5cyclictest1739-21latency_hist12:37:571
10262993123,6cyclictest28976-21threads09:28:151
10262993123,6cyclictest28976-21threads09:28:151
132102300,2sleep10-21swapper/110:13:131
10262993022,2cyclictest181rcu_preempt07:18:131
10262993021,2cyclictest181rcu_preempt11:21:031
10262993021,2cyclictest181rcu_preempt11:21:031
10262993019,3cyclictest16316-21kworker/u10:2+rpciod12:02:571
10262993019,3cyclictest16316-21kworker/u10:2+rpciod12:02:571
10262993019,3cyclictest16316-21kworker/u10:2+rpciod12:02:571
10262993017,4cyclictest181rcu_preempt08:12:071
10262993017,4cyclictest181rcu_preempt08:12:071
10262993014,4cyclictest181rcu_preempt07:23:121
10262993014,4cyclictest181rcu_preempt07:23:121
10261993022,6cyclictest16429-21fschecks_count10:23:050
10261993022,6cyclictest16429-21fschecks_count10:23:050
1026299294,4cyclictest281ktimers/110:25:511
1026299294,4cyclictest281ktimers/110:25:511
10262992924,3cyclictest0-21swapper/112:42:541
10262992924,3cyclictest0-21swapper/112:42:541
10262992924,3cyclictest0-21swapper/112:42:541
10262992923,4cyclictest10259-21cyclictest08:33:001
10262992923,4cyclictest10259-21cyclictest08:33:001
10262992923,4cyclictest10259-21cyclictest08:33:001
10262992922,5cyclictest31973-21gltestperf09:38:061
10262992920,7cyclictest27102-21kernelversion07:58:091
10262992920,7cyclictest15246-21cut10:18:141
10262992920,7cyclictest15246-21cut10:18:141
10262992918,4cyclictest181rcu_preempt09:21:011
10262992918,4cyclictest181rcu_preempt09:21:011
10262992918,4cyclictest181rcu_preempt09:21:011
1026299291,7cyclictest700-21192.168.115.40-manager12:29:351
1026299291,7cyclictest700-21192.168.115.40-manager12:29:351
10262992916,4cyclictest181rcu_preempt11:49:071
10262992916,4cyclictest181rcu_preempt11:49:071
10262992916,4cyclictest181rcu_preempt11:49:071
10261992924,4cyclictest10259-21cyclictest12:33:140
10261992924,4cyclictest10259-21cyclictest12:33:140
10261992924,4cyclictest10259-21cyclictest12:33:140
10261992923,5cyclictest20509-21sshd07:39:260
10261992923,4cyclictest24562-21sshd12:10:250
10261992923,4cyclictest24562-21sshd12:10:250
10261992923,4cyclictest24562-21sshd12:10:250
10261992921,6cyclictest17018-21sshd07:32:260
10261992921,6cyclictest17018-21sshd07:32:260
10262992823,2cyclictest0-21swapper/112:27:541
10262992822,3cyclictest0-21swapper/107:43:081
10262992822,3cyclictest0-21swapper/107:43:081
10262992820,6cyclictest3523-21latency_hist08:23:001
10262992820,6cyclictest3523-21latency_hist08:23:001
10262992820,6cyclictest23941-21sort07:48:161
10262992820,6cyclictest23941-21sort07:48:161
10262992820,6cyclictest23941-21sort07:48:161
10262992819,2cyclictest181rcu_preempt11:15:491
10262992818,4cyclictest181rcu_preempt09:15:021
10262992818,4cyclictest181rcu_preempt09:15:021
10262992814,2cyclictest181rcu_preempt08:53:051
10262992814,2cyclictest181rcu_preempt08:53:051
10262992813,4cyclictest181rcu_preempt08:25:451
10262992813,4cyclictest181rcu_preempt08:25:451
10262992813,4cyclictest181rcu_preempt08:25:451
10261992822,4cyclictest14678-21systemd-journal10:42:580
10261992822,4cyclictest14678-21systemd-journal10:42:580
10261992821,5cyclictest20768-21processes10:33:110
10261992821,5cyclictest20768-21processes10:33:110
10261992820,5cyclictest0-21swapper/008:13:050
10261992813,13cyclictest0-21swapper/008:18:060
10261992813,13cyclictest0-21swapper/008:18:060
10262992717,2cyclictest181rcu_preempt09:28:091
10262992717,2cyclictest181rcu_preempt09:28:091
10262992717,2cyclictest181rcu_preempt09:28:091
10262992716,3cyclictest181rcu_preempt11:01:151
10262992716,3cyclictest181rcu_preempt11:01:151
10262992715,4cyclictest181rcu_preempt10:12:221
10262992715,4cyclictest181rcu_preempt10:12:221
10262992715,4cyclictest181rcu_preempt10:12:221
10262992714,2cyclictest181rcu_preempt11:38:171
10262992714,2cyclictest181rcu_preempt11:38:171
10262992713,2cyclictest181rcu_preempt11:32:541
10262992712,4cyclictest181rcu_preempt07:29:171
10262992712,4cyclictest181rcu_preempt07:29:171
10262992712,2cyclictest181rcu_preempt10:53:041
10262992712,2cyclictest181rcu_preempt10:53:041
10262992712,2cyclictest181rcu_preempt10:53:041
10261992720,5cyclictest8312-21threads08:33:170
10261992720,5cyclictest8312-21threads08:33:170
10261992720,5cyclictest18355-21fschecks_time10:28:080
10261992720,5cyclictest18355-21fschecks_time10:28:080
75372260,1sleep17540-21fschecks_time08:33:061
75372260,1sleep17540-21fschecks_time08:33:061
10263992611,11cyclictest2110-21chrt12:38:062
1026299268,4cyclictest281ktimers/110:55:541
1026299268,4cyclictest281ktimers/110:55:541
1026299268,4cyclictest281ktimers/110:55:541
1026299267,17cyclictest281ktimers/112:13:121
1026299266,10cyclictest29-21ksoftirqd/111:26:141
1026299266,10cyclictest29-21ksoftirqd/111:26:141
10262992620,5cyclictest31449-21cut11:03:051
10262992620,5cyclictest31449-21cut11:03:051
10262992620,5cyclictest31449-21cut11:03:051
10262992619,5cyclictest11359-21cut08:43:061
10262992619,5cyclictest11359-21cut08:43:061
10262992619,2cyclictest181rcu_preempt07:33:071
10262992619,2cyclictest181rcu_preempt07:33:071
10262992618,2cyclictest181rcu_preempt11:37:321
10262992618,2cyclictest181rcu_preempt11:37:321
1026299261,5cyclictest281ktimers/112:08:241
1026299261,5cyclictest281ktimers/112:08:241
1026299261,5cyclictest281ktimers/112:08:241
1026299261,5cyclictest281ktimers/109:08:481
1026299261,5cyclictest281ktimers/109:08:481
1026299261,5cyclictest281ktimers/109:08:481
10262992615,8cyclictest28729-21cut08:03:071
10262992615,8cyclictest28729-21cut08:03:071
10262992612,3cyclictest181rcu_preempt10:33:121
10262992612,3cyclictest181rcu_preempt10:33:121
10262992611,3cyclictest181rcu_preempt07:38:051
10261992622,3cyclictest10259-21cyclictest08:13:060
10261992622,3cyclictest10259-21cyclictest08:13:060
10261992621,3cyclictest0-21swapper/011:22:530
10261992621,3cyclictest0-21swapper/011:22:530
10261992620,5cyclictest18800-21sshd11:53:250
10261992620,5cyclictest18800-21sshd11:53:250
10261992620,4cyclictest0-21swapper/008:38:150
10261992620,4cyclictest0-21swapper/008:38:150
10261992620,3cyclictest0-21swapper/009:58:160
10261992620,3cyclictest0-21swapper/009:58:160
10261992619,5cyclictest13652-21ntp_kernel_err08:48:110
10261992619,5cyclictest13652-21ntp_kernel_err08:48:110
10261992618,6cyclictest11179-21sort08:43:040
10261992618,6cyclictest11179-21sort08:43:040
307102250,5sleep030711-21gltestperf08:08:090
1026299257,8cyclictest29-21ksoftirqd/112:18:071
1026299257,8cyclictest29-21ksoftirqd/112:18:071
10262992510,3cyclictest0-21swapper/111:56:441
10262992510,3cyclictest0-21swapper/111:56:441
10261992521,2cyclictest0-21swapper/011:13:130
10261992519,5cyclictest13395-21sshd11:41:250
10261992519,5cyclictest13395-21sshd11:41:250
10261992519,4cyclictest10259-21cyclictest07:19:220
10261992519,4cyclictest0-21swapper/007:57:530
10261992519,4cyclictest0-21swapper/007:57:530
10261992518,5cyclictest0-21swapper/011:12:530
10261992518,5cyclictest0-21swapper/011:12:530
10261992517,6cyclictest19732-21unixbench-2d09:03:150
10261992517,6cyclictest19732-21unixbench-2d09:03:150
10261992516,7cyclictest29421-21fschecks_count10:58:040
10261992516,7cyclictest29421-21fschecks_count10:58:040
10261992516,7cyclictest29421-21fschecks_count10:58:040
10263992417,4cyclictest351rcuc/212:34:242
10263992417,4cyclictest351rcuc/212:34:242
10261992420,2cyclictest0-21swapper/011:07:530
10261992420,2cyclictest0-21swapper/011:07:530
10261992420,2cyclictest0-21swapper/011:07:530
10261992419,3cyclictest0-21swapper/011:43:060
10261992419,3cyclictest0-21swapper/011:43:060
10261992419,3cyclictest0-21swapper/011:43:060
10261992419,3cyclictest0-21swapper/010:58:110
10261992419,3cyclictest0-21swapper/010:58:110
10261992418,4cyclictest0-21swapper/007:23:150
10261992418,4cyclictest0-21swapper/007:23:150
10261992418,3cyclictest0-21swapper/010:18:130
10261992418,3cyclictest0-21swapper/010:18:130
10261992417,5cyclictest547-21rs:main0
10261992417,5cyclictest547-21rs:main0
10261992417,4cyclictest0-21swapper/012:03:140
10261992417,4cyclictest0-21swapper/012:03:140
10261992417,4cyclictest0-21swapper/012:03:140
10261992415,7cyclictest3649-21sort08:23:040
10261992415,7cyclictest3649-21sort08:23:040
10261992415,7cyclictest3649-21sort08:23:040
10261992415,7cyclictest25472-21kernelversion12:13:070
302832230,3sleep30-21swapper/310:58:133
302832230,3sleep30-21swapper/310:58:133
10261992318,4cyclictest29051-21timerwakeupswit09:28:150
10261992318,4cyclictest29051-21timerwakeupswit09:28:150
10261992318,3cyclictest0-21swapper/011:33:130
10261992318,3cyclictest0-21swapper/011:33:130
10261992313,8cyclictest14678-21systemd-journal09:52:530
10261992313,8cyclictest14678-21systemd-journal09:52:530
10261992313,7cyclictest21164-21tr09:08:120
10261992313,7cyclictest21164-21tr09:08:120
10261992313,7cyclictest21164-21tr09:08:120
193252220,4sleep30-21swapper/309:03:113
193252220,4sleep30-21swapper/309:03:113
1026199228,11cyclictest729-21dbus-daemon09:02:530
1026199228,11cyclictest729-21dbus-daemon09:02:530
10261992218,3cyclictest10259-21cyclictest09:38:130
10261992218,2cyclictest0-21swapper/012:03:040
10261992216,4cyclictest0-21swapper/010:03:060
10261992216,4cyclictest0-21swapper/010:03:060
10261992216,4cyclictest0-21swapper/007:58:090
10263992116,3cyclictest351rcuc/211:07:332
10263992116,3cyclictest351rcuc/211:07:332
10263992116,3cyclictest351rcuc/211:07:332
10263992116,3cyclictest351rcuc/210:44:212
10263992116,3cyclictest351rcuc/210:44:212
10263992116,3cyclictest351rcuc/210:44:212
10263992116,3cyclictest351rcuc/210:44:212
10263992115,4cyclictest351rcuc/208:17:492
10263992115,4cyclictest351rcuc/208:17:492
10263992114,5cyclictest351rcuc/208:24:042
10263992114,5cyclictest351rcuc/208:24:042
10263992114,5cyclictest351rcuc/208:24:042
10261992115,5cyclictest28987-21kernelversion08:03:100
10261992115,5cyclictest28987-21kernelversion08:03:100
10261992115,5cyclictest16147-21kernelversion11:48:090
10261992115,5cyclictest16147-21kernelversion11:48:090
10261992115,5cyclictest16147-21kernelversion11:48:090
10263992015,3cyclictest351rcuc/212:40:392
10263992015,3cyclictest351rcuc/212:40:392
10263992015,3cyclictest351rcuc/212:40:392
10263992015,3cyclictest351rcuc/212:09:252
10263992015,3cyclictest351rcuc/212:09:252
10263992015,3cyclictest351rcuc/212:09:252
10263992015,3cyclictest351rcuc/210:53:022
10263992015,3cyclictest351rcuc/210:53:022
10263992015,3cyclictest351rcuc/210:53:022
10263992014,4cyclictest351rcuc/212:13:382
143252190,3sleep30-21swapper/307:23:133
143252190,3sleep30-21swapper/307:23:133
1026499191,11cyclictest22303-21chrt10:38:073
1026399198,7cyclictest31046-21taskset11:02:582
1026399198,7cyclictest31046-21taskset11:02:582
10263991915,3cyclictest351rcuc/209:22:562
10263991915,3cyclictest351rcuc/209:22:562
10263991915,3cyclictest351rcuc/209:22:562
10263991914,3cyclictest351rcuc/212:21:382
10263991914,3cyclictest351rcuc/212:21:382
10263991914,3cyclictest351rcuc/211:19:182
10263991914,3cyclictest351rcuc/211:19:182
10263991914,3cyclictest351rcuc/209:55:292
10263991914,3cyclictest351rcuc/209:55:292
10263991914,3cyclictest351rcuc/209:34:142
10263991914,3cyclictest351rcuc/209:34:142
10263991914,3cyclictest351rcuc/207:32:212
10263991914,3cyclictest351rcuc/207:32:212
10263991913,4cyclictest351rcuc/211:17:462
10263991913,4cyclictest351rcuc/211:17:462
10263991913,4cyclictest351rcuc/210:08:392
10263991913,4cyclictest351rcuc/210:08:392
10263991913,4cyclictest351rcuc/210:08:392
10263991912,5cyclictest351rcuc/210:43:012
10263991912,5cyclictest351rcuc/210:43:012
10263991912,5cyclictest351rcuc/208:58:452
10263991912,5cyclictest351rcuc/208:58:452
10263991813,3cyclictest351rcuc/210:25:322
10263991813,3cyclictest351rcuc/210:25:322
10263991813,3cyclictest351rcuc/208:40:192
10263991813,3cyclictest351rcuc/208:40:192
10263991813,3cyclictest351rcuc/208:04:492
10263991813,3cyclictest351rcuc/208:04:492
10263991813,3cyclictest351rcuc/208:01:152
10263991812,4cyclictest351rcuc/212:05:222
10263991812,4cyclictest351rcuc/212:05:222
10263991812,4cyclictest351rcuc/212:05:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional