You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-03 - 00:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Thu Jul 02, 2026 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3039535996050,7cyclictest3156620-40mandb11:21:521
3039534995145,4cyclictest3081698-21ssh09:20:010
3039536994836,5cyclictest0-21swapper/207:45:362
3039535994739,5cyclictest3075580-21idleruntime-cro09:10:011
3039535994739,5cyclictest3039540-21systemd-run07:10:021
3039536994535,5cyclictest0-21swapper/210:10:412
303953699442,31cyclictest3167746-21ssh11:40:352
3039535994436,5cyclictest3100154-21systemd-run09:50:021
3039535994436,5cyclictest3048556-21systemd-run07:40:011
3039534994436,6cyclictest3182427-21/usr/sbin/munin12:05:280
3039534994431,11cyclictest421-21dbus-daemon11:45:340
3039536994333,6cyclictest265-21systemd-journal09:55:472
3039536994331,9cyclictest3161946-21mailstats11:30:422
3039535994335,5cyclictest3106208-21ssh10:00:001
3039535994335,3cyclictest0-21swapper/111:05:401
3039535994335,3cyclictest0-21swapper/111:05:401
3039536994234,5cyclictest3160634-21cut11:30:012
3039535994234,5cyclictest3158284-21fschecks_time11:25:261
3039534994229,11cyclictest3156620-40mandb11:21:530
3039536994133,5cyclictest3178715-21ssh12:00:012
3039536994133,5cyclictest3109287-21ls10:05:012
3039536994131,6cyclictest421-21dbus-daemon12:25:022
3039536994131,6cyclictest421-21dbus-daemon12:25:022
3039535994133,5cyclictest3056974-21sshd08:05:351
3039535994133,5cyclictest3056974-21sshd08:05:351
3039535994132,6cyclictest1003-21mta-sts-daemon09:45:021
3039534994132,6cyclictest12650irq/32-eth011:10:380
3039536994031,5cyclictest3101033-21missed_timers09:50:302
3039536994031,5cyclictest3101033-21missed_timers09:50:302
3039535994033,4cyclictest3166663-21munin-run11:40:011
3039535994030,5cyclictest3089307-21wc09:30:441
3039534994030,7cyclictest3096355-21sshd09:42:460
3039536993931,5cyclictest3109809-21aten_r2power_vo10:05:272
3039536993929,8cyclictest3166673-21ssh11:40:012
303953699392,31cyclictest3140663-21ssh10:55:392
3039535993928,8cyclictest3188616-21latency12:15:271
3039534993928,8cyclictest12650irq/32-eth009:35:340
3039536993830,5cyclictest1-21systemd10:40:022
3039535993830,6cyclictest3202936-21cat12:40:021
3039535993822,3cyclictest171rcu_preempt08:00:431
3039535993822,3cyclictest171rcu_preempt08:00:431
3039534993832,4cyclictest3066568-21idleruntime-cro08:40:010
3039536993729,5cyclictest3151575-21cron11:15:002
3039536993729,5cyclictest1002-21mta-sts-daemon10:53:372
3039536993727,7cyclictest694-21cron12:35:002
3039536993727,7cyclictest694-21cron12:35:002
3039535993729,5cyclictest3175873-21latency_hist11:55:031
3039535993726,8cyclictest3086133-21mailstats09:25:441
3039535993717,12cyclictest3125521-21ntp_kernel_err10:30:321
3039534993729,5cyclictest623-21sshd11:01:040
3039534993729,5cyclictest623-21sshd11:01:040
3039534993729,5cyclictest2244171-21cups-browsed12:04:100
3039534993710,20cyclictest3131652-21ssh10:40:390
3039536993629,4cyclictest3145568-21cron11:05:002
3039536993629,4cyclictest3145568-21cron11:05:002
3039536993628,5cyclictest3094019-21sendmail09:40:012
3039536993628,5cyclictest1002-21mta-sts-daemon08:35:242
3039536993627,6cyclictest265-21systemd-journal09:30:022
3039536993627,5cyclictest3060564-21sendmail08:20:002
3039535993627,5cyclictest420-21avahi-daemon10:13:101
3039535993615,5cyclictest265-21systemd-journal11:03:471
3039535993615,5cyclictest265-21systemd-journal11:03:471
303953599360,5cyclictest3167846-21ssh11:40:391
3039534993629,5cyclictest3053794-21sshd07:55:290
3039536993530,2cyclictest0-21swapper/212:05:012
3039536993529,3cyclictest0-21swapper/212:40:002
3039536993527,5cyclictest3124642-21latency_hist10:30:022
3039536993527,5cyclictest3072556-21grep09:00:012
3039536993526,6cyclictest420-21avahi-daemon11:15:242
3039536993524,7cyclictest420-21avahi-daemon12:26:252
3039536993524,7cyclictest420-21avahi-daemon12:26:252
3039535993530,2cyclictest0-21swapper/110:10:011
3039535993528,4cyclictest3142547-21grep11:00:001
3039535993528,4cyclictest3079177-21cpu09:15:251
3039535993527,5cyclictest439-21systemd-logind09:00:011
3039535993526,6cyclictest3172722-21true11:50:011
3039535993519,3cyclictest171rcu_preempt10:15:241
3039535993517,13cyclictest265-21systemd-journal10:50:021
303953499358,5cyclictest14-21ksoftirqd/007:45:340
303953499357,8cyclictest3056986-21ntpq08:05:360
303953499357,8cyclictest3056986-21ntpq08:05:360
3039534993529,3cyclictest0-21swapper/011:25:450
3039534993528,3cyclictest0-21swapper/009:20:370
3039534993525,7cyclictest3148591-21grep11:10:010
3039534993525,7cyclictest3148591-21grep11:10:010
3039534993522,8cyclictest12650irq/32-eth007:50:360
3039536993425,5cyclictest3171297-21unixbench_multi11:45:472
3039535993428,3cyclictest0-21swapper/110:25:011
3039535993427,4cyclictest3191849-21ntp_kernel_pll_12:20:341
3039535993427,4cyclictest3191849-21ntp_kernel_pll_12:20:341
3039535993425,6cyclictest421-21dbus-daemon12:00:401
3039535993420,3cyclictest171rcu_preempt12:10:321
3039534993430,2cyclictest0-21swapper/012:20:000
3039534993428,4cyclictest3124859-21/usr/sbin/munin10:30:340
3039534993427,4cyclictest3093279-21kworker/u8:7-rpciod10:10:470
3039534993424,7cyclictest0-21swapper/009:10:420
3039534993411,6cyclictest14-21ksoftirqd/010:20:380
3039536993327,3cyclictest0-21swapper/208:00:012
3039536993326,4cyclictest3121489-21cron10:25:012
3039536993326,4cyclictest3100153-21systemctl09:50:022
3039536993326,4cyclictest1-21systemd08:55:022
3039536993325,5cyclictest3127692-21sh10:35:022
3039536993325,5cyclictest3083358-21grep09:20:452
3039536993325,5cyclictest3041034-21systemd-run07:15:022
3039535993327,3cyclictest0-21swapper/111:20:011
3039535993321,3cyclictest171rcu_preempt07:30:361
3039535993315,3cyclictest171rcu_preempt08:25:031
3039535993314,2cyclictest171rcu_preempt12:06:251
303953499339,5cyclictest14-21ksoftirqd/012:28:020
303953499339,5cyclictest14-21ksoftirqd/012:28:020
3039534993328,3cyclictest3192423-21unixbench_multi12:20:500
3039534993328,3cyclictest3192423-21unixbench_multi12:20:500
3039534993328,2cyclictest0-21swapper/009:10:010
3039534993326,5cyclictest439-21systemd-logind08:20:010
3039534993324,7cyclictest3152224-21sort11:15:270
3039534993322,9cyclictest3130604-21idleruntime-cro10:40:010
3039534993316,10cyclictest421-21dbus-daemon12:35:380
303953799322,11cyclictest0-21swapper/311:10:383
3039536993226,3cyclictest0-21swapper/212:10:012
3039536993225,4cyclictest1002-21mta-sts-daemon08:10:382
3039536993224,5cyclictest3156642-21ssh11:21:512
3039536993224,5cyclictest3065053-21cron08:35:012
3039536993224,5cyclictest0-21swapper/210:20:012
3039536993223,6cyclictest3047551-21fschecks_time07:35:262
3039536993222,6cyclictest3188309-21cpu12:15:222
303953599328,8cyclictest28-21ksoftirqd/110:40:011
303953599325,20cyclictest271ktimers/111:55:421
303953599324,10cyclictest28-21ksoftirqd/108:30:371
3039535993225,4cyclictest3196858-21latency_hist12:30:011
3039535993225,4cyclictest3196858-21latency_hist12:30:011
3039535993225,4cyclictest3101716-21unixbench_multi09:50:451
3039535993225,4cyclictest3101716-21unixbench_multi09:50:451
3039535993224,5cyclictest3107800-21unixbench-2d10:00:441
3039535993224,5cyclictest3060565-21munin-run08:20:001
3039535993224,4cyclictest3067631-21sendmail_mailqu08:40:421
3039535993223,3cyclictest171rcu_preempt07:55:291
3039535993222,3cyclictest171rcu_preempt09:35:401
3039535993221,8cyclictest3041886-21ntp_kernel_pll_07:15:351
3039535993219,10cyclictest3148824-21sh11:10:041
303953499329,4cyclictest14-21ksoftirqd/011:37:100
303953499328,5cyclictest14-21ksoftirqd/011:30:330
303953499328,4cyclictest14-21ksoftirqd/012:31:390
303953499328,4cyclictest14-21ksoftirqd/012:31:390
303953499326,10cyclictest14-21ksoftirqd/009:55:320
3039534993227,3cyclictest0-21swapper/009:50:370
3039534993227,3cyclictest0-21swapper/009:50:370
3039534993220,9cyclictest0-21swapper/007:30:020
3039534993213,11cyclictest3167697-21ssh11:40:330
3039534993211,6cyclictest14-21ksoftirqd/010:25:370
3039534993211,5cyclictest14-21ksoftirqd/008:40:390
3039534993210,6cyclictest14-21ksoftirqd/008:50:350
3039536993123,5cyclictest3056083-21munin-plugin-st08:05:012
3039536993123,5cyclictest3056083-21munin-plugin-st08:05:012
303953699312,24cyclictest3146709-21ssh11:05:372
303953699312,24cyclictest3146709-21ssh11:05:372
303953599318,2cyclictest171rcu_preempt07:10:331
3039535993125,3cyclictest0-21swapper/112:30:501
3039535993125,3cyclictest0-21swapper/112:30:501
3039535993121,3cyclictest171rcu_preempt09:10:331
3039535993118,3cyclictest171rcu_preempt09:00:401
3039535993118,3cyclictest171rcu_preempt07:40:361
3039534993125,4cyclictest3116027-21fschecks_time10:15:270
3039534993125,4cyclictest3100123-21latency_hist09:50:010
3039534993125,4cyclictest3088523-21fschecks_count09:30:270
3039534993125,3cyclictest0-21swapper/010:00:350
3039534993125,3cyclictest0-21swapper/008:25:390
3039534993118,6cyclictest3042533-21awk07:20:010
3039534993118,11cyclictest3109727-21aten_r2power_fr10:05:250
3039534993113,4cyclictest14-21ksoftirqd/008:00:430
3039534993113,4cyclictest14-21ksoftirqd/008:00:430
303953699302,6cyclictest171rcu_preempt10:45:342
3039536993024,4cyclictest3039533-21cyclictest12:10:292
3039536993023,4cyclictest3172723-21kworker/u8:8-xprtiod11:50:412
3039536993018,9cyclictest3079467-21latency09:15:292
303953599304,12cyclictest271ktimers/107:25:341
3039535993021,5cyclictest3139525-21ssh10:55:011
3039535993021,5cyclictest271ktimers/108:50:341
3039535993020,5cyclictest271ktimers/107:45:371
3039535993018,2cyclictest171rcu_preempt09:20:331
303953499308,5cyclictest14-21ksoftirqd/010:45:350
303953499307,4cyclictest14-21ksoftirqd/010:53:100
303953499307,4cyclictest14-21ksoftirqd/008:30:390
3039534993024,4cyclictest3062073-21sh08:25:010
3039534993021,7cyclictest12650irq/32-eth011:50:330
3039534993021,6cyclictest3049944-21head07:40:460
3039534993020,7cyclictest3185236-21aten_r2power_po12:10:240
3039534993011,5cyclictest14-21ksoftirqd/007:10:330
3039534993010,15cyclictest12650irq/32-eth010:55:350
303953699293,17cyclictest3131686-21ssh10:40:402
3039536992923,4cyclictest3039533-21cyclictest09:05:012
3039536992921,5cyclictest3075571-21diskmemload09:10:422
3039536992921,5cyclictest3042528-21sendmail07:20:012
3039536992919,6cyclictest3056375-21aten_r2power_en08:05:242
3039536992919,6cyclictest3056375-21aten_r2power_en08:05:242
3039535992922,4cyclictest3124531-21munin-plugin-st10:30:011
3039535992920,5cyclictest271ktimers/110:40:401
3039535992920,5cyclictest271ktimers/108:45:331
3039535992915,6cyclictest171rcu_preempt08:15:011
3039535992915,3cyclictest171rcu_preempt11:30:331
3039535992911,3cyclictest0-21swapper/108:35:321
303953499297,5cyclictest14-21ksoftirqd/009:00:360
3039534992924,3cyclictest151ktimers/007:35:350
3039534992910,15cyclictest420-21avahi-daemon07:30:360
303953699285,4cyclictest171rcu_preempt08:20:362
3039536992822,3cyclictest0-21swapper/208:30:022
303953699282,20cyclictest3088909-21ntpq09:30:342
3039536992821,4cyclictest3068072-21munin-run08:45:012
3039536992821,2cyclictest0-21swapper/209:10:012
3039536992815,5cyclictest1003-21mta-sts-daemon09:44:372
303953599288,6cyclictest271ktimers/107:20:461
3039535992820,3cyclictest171rcu_preempt07:50:411
3039535992816,3cyclictest171rcu_preempt08:25:361
303953499287,5cyclictest14-21ksoftirqd/011:55:550
303953499286,4cyclictest14-21ksoftirqd/009:25:370
303953699275,15cyclictest428-21mta-sts-daemon07:20:382
3039536992719,4cyclictest3069712-21latency_hist08:50:022
303953499276,5cyclictest14-21ksoftirqd/008:55:350
303953499275,4cyclictest14-21ksoftirqd/008:10:230
303953499274,7cyclictest2244171-21cups-browsed07:21:360
3039536992618,5cyclictest3053066-21munin-plugin-st07:55:012
3039536992618,5cyclictest3049971-21unixbench_singl07:40:472
3039536992611,3cyclictest171rcu_preempt07:30:012
3039534992622,2cyclictest0-21swapper/008:45:320
303953799252,12cyclictest0-21swapper/309:35:343
3039536992517,5cyclictest2244171-21cups-browsed07:31:532
303953799223,11cyclictest0-21swapper/311:15:373
303953799214,9cyclictest0-21swapper/311:35:403
303953799214,9cyclictest0-21swapper/311:20:383
303953799214,8cyclictest0-21swapper/312:10:353
303953799212,12cyclictest0-21swapper/309:40:363
303953799212,11cyclictest0-21swapper/311:55:383
3039534992116,3cyclictest151ktimers/007:10:020
303953799203,9cyclictest0-21swapper/312:30:393
303953799203,9cyclictest0-21swapper/312:30:393
303953799203,9cyclictest0-21swapper/312:05:343
303953799203,10cyclictest0-21swapper/311:00:323
303953799203,10cyclictest0-21swapper/311:00:323
303953799203,10cyclictest0-21swapper/310:05:433
303953799202,9cyclictest0-21swapper/310:20:353
303953799202,11cyclictest0-21swapper/309:45:373
303953799202,10cyclictest0-21swapper/309:25:403
303953799201,8cyclictest0-21swapper/307:10:373
303953799201,12cyclictest0-21swapper/309:15:373
303953799201,11cyclictest0-21swapper/311:40:333
303953799194,9cyclictest0-21swapper/312:20:453
303953799194,9cyclictest0-21swapper/312:20:453
303953799194,8cyclictest0-21swapper/310:50:433
303953799194,10cyclictest0-21swapper/310:15:013
303953799193,9cyclictest0-21swapper/312:35:263
303953799193,10cyclictest0-21swapper/312:00:323
303953799193,10cyclictest0-21swapper/311:50:383
303953799193,10cyclictest0-21swapper/311:05:383
303953799193,10cyclictest0-21swapper/311:05:383
303953799192,10cyclictest0-21swapper/311:30:423
303953799192,10cyclictest0-21swapper/311:25:353
303953799191,12cyclictest0-21swapper/312:15:343
303953799191,11cyclictest0-21swapper/310:30:343
3039536991911,2cyclictest0-21swapper/207:10:022
303953799183,7cyclictest0-21swapper/309:00:363
303953799183,7cyclictest0-21swapper/308:35:353
303953799183,10cyclictest0-21swapper/310:00:303
303953799182,9cyclictest0-21swapper/310:15:413
303953799182,9cyclictest0-21swapper/309:50:303
303953799182,9cyclictest0-21swapper/309:50:303
303953799182,10cyclictest0-21swapper/310:25:343
303953799181,11cyclictest0-21swapper/310:45:263
303953799181,10cyclictest0-21swapper/309:55:263
303953799181,10cyclictest0-21swapper/309:30:323
303953799173,9cyclictest0-21swapper/311:45:313
303953799173,8cyclictest0-21swapper/309:20:333
303953799172,9cyclictest0-21swapper/312:25:333
303953799172,9cyclictest0-21swapper/312:25:333
303953799172,9cyclictest0-21swapper/310:40:303
303953799172,8cyclictest0-21swapper/308:00:473
303953799172,8cyclictest0-21swapper/308:00:473
303953799172,7cyclictest0-21swapper/308:15:333
303953799171,9cyclictest0-21swapper/309:10:023
303953799162,9cyclictest0-21swapper/308:30:373
303953799162,9cyclictest0-21swapper/307:50:363
303953799162,8cyclictest0-21swapper/307:40:353
303953799162,6cyclictest0-21swapper/308:40:333
303953799162,10cyclictest0-21swapper/310:35:453
303953799161,10cyclictest0-21swapper/309:10:283
303953799153,8cyclictest0-21swapper/310:55:343
303953799153,5cyclictest0-21swapper/308:10:333
303953799152,8cyclictest0-21swapper/307:35:393
303953799152,7cyclictest0-21swapper/308:50:383
303953799152,7cyclictest0-21swapper/307:55:343
303953799152,7cyclictest0-21swapper/307:15:353
303953799151,9cyclictest0-21swapper/307:45:373
303953799143,7cyclictest0-21swapper/307:30:363
303953799142,7cyclictest0-21swapper/307:20:403
303953799141,8cyclictest0-21swapper/308:20:343
303953799133,7cyclictest0-21swapper/308:30:033
303953799132,7cyclictest0-21swapper/307:25:383
303953799132,6cyclictest0-21swapper/308:55:333
303953799132,6cyclictest0-21swapper/308:05:333
303953799132,6cyclictest0-21swapper/308:05:333
303953799111,7cyclictest0-21swapper/308:45:313
30395379991,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional