You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-20 - 16:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Mon Apr 20, 2026 12:44:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
101352700,2chrt0-21swapper/310:02:023
101352700,2chrt0-21swapper/310:02:023
81412680,3chrt0-21swapper/308:27:233
81412680,3chrt0-21swapper/308:27:233
203962660,2chrt0-21swapper/309:02:133
303682650,3chrt0-21swapper/309:27:213
303682650,3chrt0-21swapper/309:27:213
303682650,3chrt0-21swapper/309:27:213
296792590,1chrt29677-21unixbench_singl12:17:260
296792590,1chrt29677-21unixbench_singl12:17:260
296792590,1chrt29677-21unixbench_singl12:17:260
208052580,2sleep020804-21ls11:57:040
143942560,2sleep014551-21/usr/sbin/munin10:12:170
77012550,1sleep17803-21missed_timers08:27:181
77012550,1sleep17803-21missed_timers08:27:181
259882550,2sleep111754-21cyclictest12:07:271
259882550,2sleep111754-21cyclictest12:07:271
259882550,2sleep111754-21cyclictest12:07:271
282962520,3sleep0201rcuc/012:17:060
282962520,3sleep0201rcuc/012:17:060
220102510,3chrt0-21swapper/111:57:261
220102510,3chrt0-21swapper/111:57:261
220102510,3chrt0-21swapper/111:57:261
4092500,3sleep132150-21/usr/sbin/munin12:27:201
4092500,3sleep132150-21/usr/sbin/munin12:27:201
289632490,2sleep30-21swapper/309:22:273
289632490,2sleep30-21swapper/309:22:273
289632490,2sleep30-21swapper/309:22:273
221562490,3sleep30-21swapper/307:37:273
221562490,3sleep30-21swapper/307:37:273
36062480,2sleep30-21swapper/309:42:213
36062480,2sleep30-21swapper/309:42:213
36062480,2sleep30-21swapper/309:42:213
35112480,2sleep13512-21cat09:42:191
35112480,2sleep13512-21cat09:42:191
35112480,2sleep13512-21cat09:42:191
94712450,2sleep09470-21switchtime09:57:250
94712450,2sleep09470-21switchtime09:57:250
287692440,3sleep00-21swapper/010:51:250
287692440,3sleep00-21swapper/010:51:250
301552420,1sleep030272-21missed_timers09:27:190
301552420,1sleep030272-21missed_timers09:27:190
301552420,1sleep030272-21missed_timers09:27:190
11776994018,9cyclictest29-21ksoftirqd/108:07:361
11776994018,9cyclictest29-21ksoftirqd/108:07:361
11774993932,5cyclictest5055-21mailstats11:12:250
11774993932,5cyclictest5055-21mailstats11:12:250
65372380,2sleep06540-21sed09:52:100
65372380,2sleep06540-21sed09:52:100
254532380,3sleep30-21swapper/312:07:213
254532380,3sleep30-21swapper/312:07:213
254532380,3sleep30-21swapper/312:07:213
129402370,2sleep013102-21tune2fs08:42:170
129402370,2sleep013102-21tune2fs08:42:170
11776993715,5cyclictest700-21192.168.115.40-manager08:39:201
11776993715,5cyclictest700-21192.168.115.40-manager08:39:201
11774993628,6cyclictest20542-21fschecks_time09:02:140
11774993628,6cyclictest20542-21fschecks_time09:02:140
300052350,2sleep30-21swapper/308:02:033
300052350,2sleep30-21swapper/308:02:033
300052350,2sleep30-21swapper/308:02:033
11776993429,3cyclictest11754-21cyclictest11:30:111
11776993429,3cyclictest11754-21cyclictest11:30:111
11776993425,7cyclictest14602-21kernelversion10:12:181
11776993326,5cyclictest20590-21gltestperf09:02:141
11776993326,5cyclictest20590-21gltestperf09:02:141
11776993320,4cyclictest181rcu_preempt08:18:111
11776993320,4cyclictest181rcu_preempt08:18:111
11776993314,6cyclictest29-21ksoftirqd/112:42:031
11776993314,6cyclictest29-21ksoftirqd/112:42:031
11776993314,6cyclictest29-21ksoftirqd/112:42:031
152862320,3chrt14663-21/usr/sbin/munin08:47:191
152862320,3chrt14663-21/usr/sbin/munin08:47:191
11776993225,5cyclictest734-21latency_hist09:37:031
11776993224,6cyclictest28492-21fschecks_count07:57:151
11776993224,6cyclictest28492-21fschecks_count07:57:151
11776993224,6cyclictest28492-21fschecks_count07:57:151
11776993224,6cyclictest19473-21kernelversion07:32:181
11776993224,6cyclictest19473-21kernelversion07:32:181
11776993223,7cyclictest17608-21kernelversion07:27:181
11776993222,2cyclictest181rcu_preempt08:02:361
11776993222,2cyclictest181rcu_preempt08:02:361
11776993216,4cyclictest90-1pr/ttyS009:07:191
11774993223,7cyclictest7382-21fschecks_count08:27:150
11774993223,7cyclictest7382-21fschecks_count08:27:150
183542310,2sleep30-21swapper/308:57:113
169822310,3sleep00-21swapper/007:27:070
169822310,3sleep00-21swapper/007:27:070
11776993125,4cyclictest24716-21needreboot09:12:201
11776993125,4cyclictest24716-21needreboot09:12:201
11776993124,5cyclictest26408-21kernelversion09:17:191
11776993124,5cyclictest26408-21kernelversion09:17:191
11776993124,5cyclictest26408-21kernelversion09:17:191
11774993124,5cyclictest27920-21sh09:22:160
11774993124,5cyclictest27920-21sh09:22:160
11774993124,5cyclictest27920-21sh09:22:160
11774993123,6cyclictest1188-21fschecks_count09:37:170
11774993123,6cyclictest1188-21fschecks_count09:37:170
60442300,8sleep1281ktimers/109:47:291
60442300,8sleep1281ktimers/109:47:291
24592300,3sleep111754-21cyclictest09:39:291
24592300,3sleep111754-21cyclictest09:39:291
142632300,2sleep014368-21/usr/sbin/munin11:37:240
142632300,2sleep014368-21/usr/sbin/munin11:37:240
1177699308,10cyclictest29-21ksoftirqd/109:30:251
1177699308,10cyclictest29-21ksoftirqd/109:30:251
1177699308,10cyclictest29-21ksoftirqd/109:30:251
11776993023,5cyclictest8143-21cut11:22:161
11776993023,5cyclictest8143-21cut11:22:161
11776993023,5cyclictest8143-21cut11:22:161
11776993023,5cyclictest281ktimers/108:13:031
11776993023,5cyclictest281ktimers/108:13:031
11776993023,5cyclictest21351-21cron10:32:161
11776993023,5cyclictest21351-21cron10:32:161
11776993015,2cyclictest181rcu_preempt08:57:251
11776993015,2cyclictest181rcu_preempt08:57:251
11774993023,5cyclictest25001-21gltestperf12:07:150
11774993023,5cyclictest17964-21tr11:47:220
11774993023,5cyclictest17964-21tr11:47:220
228872290,4sleep20-21swapper/207:42:162
228872290,4sleep20-21swapper/207:42:162
1177699296,5cyclictest281ktimers/107:37:571
1177699296,5cyclictest281ktimers/107:37:571
11776992923,4cyclictest27678-21kernelversion10:47:171
11776992923,4cyclictest27678-21kernelversion10:47:171
11776992922,5cyclictest21455-21latency_hist10:32:091
11776992922,5cyclictest21455-21latency_hist10:32:091
11776992920,2cyclictest181rcu_preempt07:25:301
11776992920,2cyclictest181rcu_preempt07:25:301
11776992915,2cyclictest181rcu_preempt07:42:191
11776992915,2cyclictest181rcu_preempt07:42:191
11774992923,5cyclictest2812-21latency11:07:190
11774992923,5cyclictest2812-21latency11:07:190
11774992923,5cyclictest16455-21kernelversion10:17:180
11774992923,5cyclictest16455-21kernelversion10:17:180
11776992822,5cyclictest9346-21rt-features09:57:241
11776992822,5cyclictest9346-21rt-features09:57:241
11776992821,5cyclictest24493-21timerandwakeup10:37:271
11776992821,5cyclictest24493-21timerandwakeup10:37:271
11776992821,2cyclictest181rcu_preempt10:26:391
11776992821,2cyclictest181rcu_preempt10:26:391
11776992821,2cyclictest181rcu_preempt10:26:391
11776992819,7cyclictest13837-21threads08:42:261
11776992819,7cyclictest13837-21threads08:42:261
11776992815,3cyclictest60-21kcompactd010:10:321
11776992815,3cyclictest60-21kcompactd010:10:321
11776992815,3cyclictest181rcu_preempt12:07:141
11776992815,3cyclictest181rcu_preempt12:07:141
11776992814,2cyclictest181rcu_preempt11:22:071
11776992814,2cyclictest181rcu_preempt11:22:071
11776992814,2cyclictest181rcu_preempt11:07:251
11776992814,2cyclictest181rcu_preempt11:07:251
11776992814,2cyclictest181rcu_preempt10:57:141
11776992814,2cyclictest181rcu_preempt10:57:141
11774992823,4cyclictest11754-21cyclictest07:29:570
11774992821,5cyclictest28066-21sshd07:56:160
11774992821,5cyclictest28066-21sshd07:56:160
11774992821,5cyclictest24267-21sshd12:04:160
11774992818,8cyclictest4539-21cron09:47:030
11774992818,8cyclictest4539-21cron09:47:030
11774992818,8cyclictest4539-21cron09:47:030
225712270,3sleep00-21swapper/010:32:220
225712270,3sleep00-21swapper/010:32:220
1177699279,3cyclictest271rcuc/110:54:101
1177699279,3cyclictest271rcuc/110:54:101
1177699278,16cyclictest281ktimers/112:18:151
1177699278,16cyclictest281ktimers/112:18:151
1177699278,16cyclictest281ktimers/112:18:151
11776992721,5cyclictest13985-21kernelversion11:37:181
11776992721,5cyclictest13985-21kernelversion11:37:181
11776992720,6cyclictest27028-21latency_hist10:47:051
11776992720,6cyclictest27028-21latency_hist10:47:051
11776992720,4cyclictest281ktimers/111:33:111
11776992720,4cyclictest281ktimers/111:33:111
11776992714,3cyclictest181rcu_preempt11:12:241
11776992714,3cyclictest181rcu_preempt11:12:241
11776992714,2cyclictest181rcu_preempt08:52:191
11776992714,2cyclictest181rcu_preempt08:52:191
11774992721,5cyclictest13051-21processes10:07:250
11774992721,5cyclictest13051-21processes10:07:250
11774992721,4cyclictest11754-21cyclictest08:37:030
11774992721,4cyclictest11754-21cyclictest08:37:030
11774992721,4cyclictest0-21swapper/012:32:170
11774992721,4cyclictest0-21swapper/012:32:170
11774992720,5cyclictest6373-21timerandwakeup08:22:240
11774992720,5cyclictest6373-21timerandwakeup08:22:240
11774992720,5cyclictest550-21cron10:07:040
11774992720,5cyclictest550-21cron10:07:040
11774992720,5cyclictest550-21cron10:07:040
11774992720,5cyclictest26346-21latency_hist12:12:140
11774992720,5cyclictest26346-21latency_hist12:12:140
11774992720,5cyclictest1346-21avahi-daemon11:06:560
11774992720,5cyclictest1346-21avahi-daemon11:06:560
11774992720,5cyclictest1346-21avahi-daemon11:06:560
1177699267,7cyclictest29-21ksoftirqd/112:32:191
1177699267,7cyclictest29-21ksoftirqd/112:32:191
1177699267,6cyclictest29-21ksoftirqd/111:02:221
1177699267,6cyclictest29-21ksoftirqd/111:02:221
1177699267,6cyclictest29-21ksoftirqd/111:02:221
1177699267,17cyclictest281ktimers/111:48:471
1177699267,17cyclictest281ktimers/111:48:471
11776992619,5cyclictest6023-21needreboot08:22:201
11776992619,5cyclictest6023-21needreboot08:22:201
11776992618,6cyclictest7590-21sort09:52:251
11776992617,2cyclictest181rcu_preempt12:13:421
11776992617,2cyclictest181rcu_preempt12:13:421
11776992614,3cyclictest90-1pr/ttyS010:04:291
11776992614,3cyclictest90-1pr/ttyS010:04:291
11776992614,3cyclictest90-1pr/ttyS010:04:291
11776992612,4cyclictest90-1pr/ttyS008:33:431
11776992612,4cyclictest90-1pr/ttyS008:33:431
11776992612,3cyclictest181rcu_preempt11:42:191
11776992612,3cyclictest181rcu_preempt11:42:191
11776992612,3cyclictest181rcu_preempt11:42:191
11776992612,3cyclictest181rcu_preempt07:52:161
11776992612,3cyclictest181rcu_preempt07:52:161
11776992611,2cyclictest181rcu_preempt10:17:181
11776992611,2cyclictest181rcu_preempt10:17:181
11774992621,3cyclictest11754-21cyclictest07:34:070
11774992621,3cyclictest11754-21cyclictest07:34:070
11774992621,3cyclictest0-21swapper/008:02:030
11774992621,3cyclictest0-21swapper/008:02:030
11774992621,3cyclictest0-21swapper/008:02:030
11774992620,4cyclictest0-21swapper/009:56:080
11774992619,6cyclictest24547-21kernelversion09:12:190
11774992619,6cyclictest24547-21kernelversion09:12:190
11774992619,5cyclictest4869-21sshd12:38:020
11774992619,5cyclictest4869-21sshd12:38:020
11774992619,5cyclictest4869-21sshd12:38:020
11774992618,6cyclictest22875-21sort07:42:160
11774992618,6cyclictest22875-21sort07:42:160
1177699255,5cyclictest281ktimers/107:48:011
11776992518,5cyclictest550-21cron11:52:161
11776992512,2cyclictest181rcu_preempt09:22:171
11776992512,2cyclictest181rcu_preempt09:22:171
11776992512,2cyclictest181rcu_preempt09:22:171
11776992511,2cyclictest181rcu_preempt12:22:211
11776992511,2cyclictest181rcu_preempt12:22:211
11776992511,2cyclictest181rcu_preempt12:22:211
1177499256,3cyclictest181rcu_preempt09:33:060
11774992521,2cyclictest0-21swapper/010:47:140
11774992520,3cyclictest0-21swapper/011:17:170
11774992520,3cyclictest0-21swapper/011:17:170
11774992519,4cyclictest12858-21unixbench_singl11:32:250
11774992519,4cyclictest12858-21unixbench_singl11:32:250
11774992517,6cyclictest20708-21ls10:27:250
11774992517,6cyclictest20708-21ls10:27:250
11774992517,5cyclictest0-21swapper/008:17:270
11774992517,5cyclictest0-21swapper/008:17:270
11774992516,7cyclictest24888-21fschecks_time07:47:160
1177499251,3cyclictest0-21swapper/012:23:470
1177499251,3cyclictest0-21swapper/012:23:470
1177499251,3cyclictest0-21swapper/012:23:470
11774992420,2cyclictest0-21swapper/010:22:240
11774992420,2cyclictest0-21swapper/010:22:240
11774992420,2cyclictest0-21swapper/010:22:240
11774992419,3cyclictest12578-21latency_hist08:42:020
11774992419,3cyclictest12578-21latency_hist08:42:020
11774992419,3cyclictest0-21swapper/008:57:160
11774992419,3cyclictest0-21swapper/008:57:160
11774992418,4cyclictest0-21swapper/009:07:190
11774992418,4cyclictest0-21swapper/008:47:160
11774992418,4cyclictest0-21swapper/008:47:160
11774992417,5cyclictest18119-21sshd08:56:150
11774992417,5cyclictest18119-21sshd08:56:150
11774992417,5cyclictest1229-21sshd12:29:160
11774992417,5cyclictest1229-21sshd12:29:160
11774992416,6cyclictest1758-21grep08:12:150
11774992416,6cyclictest1758-21grep08:12:150
11774992415,7cyclictest29900-21rt-features10:52:230
11774992415,7cyclictest29900-21rt-features10:52:230
217152231,4sleep20-21swapper/211:57:242
217152231,4sleep20-21swapper/211:57:242
217152231,4sleep20-21swapper/211:57:242
11774992319,2cyclictest0-21swapper/011:22:160
11774992319,2cyclictest0-21swapper/011:22:160
11774992319,2cyclictest0-21swapper/011:22:160
11774992318,3cyclictest0-21swapper/011:57:180
11774992318,3cyclictest0-21swapper/011:57:180
11774992318,3cyclictest0-21swapper/011:57:180
11774992318,3cyclictest0-21swapper/011:27:240
11774992318,3cyclictest0-21swapper/011:27:240
11774992318,3cyclictest0-21swapper/010:57:160
11774992318,3cyclictest0-21swapper/010:57:160
11774992316,5cyclictest15578-21cut11:42:160
11774992316,5cyclictest15578-21cut11:42:160
11774992316,5cyclictest15578-21cut11:42:160
11774992314,7cyclictest24188-21ntp_states10:37:240
11774992314,7cyclictest24188-21ntp_states10:37:240
11774992314,7cyclictest142-21systemd-journal09:17:120
11774992314,7cyclictest142-21systemd-journal09:17:120
11774992314,7cyclictest142-21systemd-journal09:17:120
11774992314,6cyclictest21878-21ls07:37:250
11774992314,6cyclictest21878-21ls07:37:250
11777992216,4cyclictest195792sleep208:57:262
11777992216,4cyclictest195792sleep208:57:262
11774992214,6cyclictest25521-21cut10:42:150
11774992116,3cyclictest0-21swapper/008:07:170
11774992116,3cyclictest0-21swapper/008:07:170
11774992115,4cyclictest0-21swapper/012:07:240
11774992112,7cyclictest142-21systemd-journal08:02:140
11774992112,7cyclictest142-21systemd-journal08:02:140
244882201,2sleep20-21swapper/210:37:282
244882201,2sleep20-21swapper/210:37:282
11777992013,5cyclictest351rcuc/209:12:382
11777992013,5cyclictest351rcuc/209:12:382
1177899193,8cyclictest0-21swapper/307:27:163
221002180,2sleep20-21swapper/210:32:162
221002180,2sleep20-21swapper/210:32:162
1177899183,9cyclictest0-21swapper/308:18:163
1177899183,9cyclictest0-21swapper/308:18:163
1177899183,8cyclictest0-21swapper/308:54:153
1177899181,12cyclictest27146-21chrt10:47:133
1177899181,12cyclictest27146-21chrt10:47:133
1177899181,12cyclictest24665-21taskset12:07:123
1177899181,12cyclictest24665-21taskset12:07:123
1177899181,12cyclictest24466-21chrt09:12:183
1177899181,12cyclictest24466-21chrt09:12:183
1177899181,12cyclictest19496-21chrt11:52:173
1177899181,12cyclictest1502-21chrt12:32:053
1177899181,12cyclictest1502-21chrt12:32:053
11777991812,5cyclictest351rcuc/208:39:202
11777991812,5cyclictest351rcuc/208:39:202
151272171,2sleep20-21swapper/211:42:042
151272171,2sleep20-21swapper/211:42:042
1177899172,11cyclictest0-21swapper/309:37:203
1177899172,11cyclictest0-21swapper/309:37:203
1177899171,12cyclictest30014-21chrt12:21:153
1177899171,12cyclictest30014-21chrt12:21:153
1177899171,12cyclictest30014-21chrt12:21:153
1177899171,12cyclictest18733-21chrt10:22:243
1177899171,12cyclictest18733-21chrt10:22:243
1177899171,12cyclictest18733-21chrt10:22:243
11777991712,3cyclictest351rcuc/210:53:192
11777991712,3cyclictest351rcuc/210:53:192
11777991712,3cyclictest351rcuc/210:26:252
11777991712,3cyclictest351rcuc/210:26:252
11777991712,3cyclictest351rcuc/210:26:252
11777991712,3cyclictest351rcuc/208:18:112
11777991712,3cyclictest351rcuc/208:18:112
1177799171,13cyclictest22074-21chrt07:37:262
1177799171,13cyclictest22074-21chrt07:37:262
1177799171,12cyclictest5098-21chrt09:47:182
1177799171,12cyclictest5098-21chrt09:47:182
1177799171,12cyclictest31619-21chrt09:32:152
1177799171,12cyclictest31619-21chrt09:32:152
11777991710,5cyclictest351rcuc/212:33:542
11777991710,5cyclictest351rcuc/212:33:542
11777991710,5cyclictest351rcuc/207:51:542
1177899161,10cyclictest9827-21chrt08:32:223
1177899161,10cyclictest9827-21chrt08:32:223
1177899161,10cyclictest21079-21chrt11:57:153
1177899161,10cyclictest21079-21chrt11:57:153
1177899161,10cyclictest21079-21chrt11:57:153
1177899161,10cyclictest0-21swapper/311:02:223
1177899161,10cyclictest0-21swapper/311:02:223
1177899161,10cyclictest0-21swapper/311:02:223
1177799163,9cyclictest0-21swapper/208:48:152
1177799163,9cyclictest0-21swapper/208:48:152
11777991611,3cyclictest351rcuc/208:03:352
11777991611,3cyclictest351rcuc/208:03:352
1177799161,12cyclictest25797-21chrt10:42:172
1177799161,12cyclictest25797-21chrt10:42:172
1177799161,11cyclictest1203-21chrt09:37:172
1177799161,11cyclictest1203-21chrt09:37:172
1177799161,10cyclictest3401-21chrt08:17:092
1177799161,10cyclictest3401-21chrt08:17:092
1177799161,10cyclictest30570-21chrt12:22:152
1177799161,10cyclictest30570-21chrt12:22:152
1177799161,10cyclictest30570-21chrt12:22:152
11777991610,4cyclictest351rcuc/211:03:442
11777991610,4cyclictest351rcuc/211:03:442
11777991610,4cyclictest351rcuc/211:03:442
229762151,2sleep30-21swapper/310:33:593
229762151,2sleep30-21swapper/310:33:593
155542150,3sleep30-21swapper/311:42:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional