You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-29 - 18:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Fri May 29, 2026 12:42:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3279499996656,6cyclictest1-21systemd12:21:002
3279499996656,6cyclictest1-21systemd12:21:002
3279498994737,7cyclictest3356269-21awk12:00:011
3279498994638,5cyclictest3304371-21munin-plugin-st09:10:011
3279498994535,4cyclictest0-21swapper/112:21:001
3279498994535,4cyclictest0-21swapper/112:21:001
3279499994331,8cyclictest2555081-21cups-browsed07:40:552
3279499994331,8cyclictest2555081-21cups-browsed07:40:552
3279498994335,5cyclictest3282779-21systemd-run07:25:011
3279499994234,4cyclictest0-21swapper/210:40:002
3279498994232,7cyclictest3349325-21/usr/sbin/munin11:35:531
3279499994131,6cyclictest3326305-21systemd-run10:20:012
3279498994133,5cyclictest3344276-21cut11:20:011
3279499994030,6cyclictest261-21systemd-journal09:47:422
3279499994029,6cyclictest0-21swapper/211:00:012
3279499993932,4cyclictest0-21swapper/212:00:012
3279499993917,13cyclictest3364682-21sshd12:25:582
3279498993932,3cyclictest0-21swapper/109:40:001
3279498993931,5cyclictest601-21gdbus09:47:421
3279497993921,10cyclictest151ktimers/011:51:010
327949999384,24cyclictest3289207-21run-parts08:01:002
327949999384,24cyclictest3289207-21run-parts08:01:002
3279499993827,7cyclictest439-21systemd-logind09:10:012
3279497993831,4cyclictest555-21ModemManager09:46:570
3279499993731,3cyclictest0-21swapper/212:10:582
3279499993731,3cyclictest0-21swapper/212:10:582
3279499993727,6cyclictest420-21avahi-daemon09:41:072
3279499993719,4cyclictest171rcu_preempt08:05:592
3279498993729,5cyclictest3281001-21systemd-run07:15:011
3279497993731,4cyclictest3342789-21systemd-run11:15:020
3279497993721,11cyclictest151ktimers/010:36:000
3279499993629,4cyclictest3318709-21sh09:55:012
3279498993628,5cyclictest3295090-21cron08:35:011
3279497993628,5cyclictest3323245-21systemd-run10:10:010
3279499993525,7cyclictest1-21systemd08:45:022
3279498993529,3cyclictest0-21swapper/109:00:011
3279498993529,3cyclictest0-21swapper/107:40:011
3279498993528,4cyclictest0-21swapper/107:30:011
3279498993525,8cyclictest3333794-21sh10:45:011
3279497993528,4cyclictest3366696-21grep12:31:120
3279497993528,4cyclictest3357771-21cron12:05:000
3279497993528,4cyclictest3296900-21systemd08:45:020
3279499993426,5cyclictest3347296-21systemd-run11:30:012
3279499993425,6cyclictest3339774-21latency_hist11:05:022
3279499993425,6cyclictest1054-21mta-sts-daemon08:50:592
3279499993423,8cyclictest3280380-21ntp_kernel_pll_07:10:592
3279499993422,9cyclictest3321745-21sh10:05:012
3279499993414,7cyclictest36-21ksoftirqd/211:10:592
3279499993413,4cyclictest171rcu_preempt10:26:012
327949899346,9cyclictest171rcu_preempt11:41:051
3279498993426,5cyclictest3347675-21aten_r2power_vo11:30:521
3279498993426,5cyclictest3318737-21systemd-run09:55:021
3279498993426,5cyclictest2555081-21cups-browsed09:41:001
3279497993429,3cyclictest0-21swapper/008:40:010
3279497993427,4cyclictest1054-21mta-sts-daemon11:55:590
3279497993426,5cyclictest2555081-21cups-browsed08:15:490
3279497993426,5cyclictest1-21systemd10:10:170
3279499993326,3cyclictest341rcuc/208:15:012
3279499993323,6cyclictest3293109-21systemd-run08:20:022
3279499993321,3cyclictest171rcu_preempt11:21:022
3279499993319,3cyclictest171rcu_preempt11:31:012
3279499993318,11cyclictest421-21dbus-daemon07:51:022
3279499993318,11cyclictest421-21dbus-daemon07:51:022
3279499993312,16cyclictest421-21dbus-daemon09:56:022
3279498993327,3cyclictest0-21swapper/110:20:011
3279497993327,4cyclictest421-21dbus-daemon10:00:020
3279497993326,4cyclictest3301354-21cut09:00:010
3279497993325,6cyclictest420-21avahi-daemon11:46:070
3279497993324,6cyclictest3295101-21munin-plugin-st08:35:010
3279497993321,9cyclictest3303338-21cpuspeed09:05:510
3279499993224,5cyclictest3344260-21cron11:20:012
3279498993226,3cyclictest0-21swapper/108:00:001
3279498993226,3cyclictest0-21swapper/108:00:001
3279498993224,5cyclictest3364352-21fschecks_time12:25:531
3279498993224,5cyclictest3324771-21idleruntime-cro10:15:011
3279498993223,6cyclictest2555081-21cups-browsed11:01:221
3279498993221,7cyclictest1056-21mta-sts-daemon07:50:591
3279498993221,7cyclictest1056-21mta-sts-daemon07:50:591
3279497993226,4cyclictest3344814-21fschecks_time11:20:530
3279497993224,6cyclictest3326787-21sshd10:20:530
3279497993220,9cyclictest0-21swapper/009:35:520
3279497993220,10cyclictest261-21systemd-journal11:10:000
327949999319,9cyclictest3308328-21ntpq09:21:022
3279499993125,3cyclictest0-21swapper/212:35:022
3279499993123,6cyclictest0-21swapper/207:40:012
3279499993122,4cyclictest3302752-21unixbench_multi09:01:102
3279499993121,6cyclictest3301404-21latency_hist09:00:022
3279499993112,3cyclictest171rcu_preempt07:21:012
3279497993124,4cyclictest3336835-21latency_hist10:55:010
3279497993124,4cyclictest3321802-21latency_hist10:05:020
3279497993122,5cyclictest191rcuc/011:31:020
3279497993121,8cyclictest3310277-21unixbench_multi09:26:110
3279497993121,8cyclictest3310277-21unixbench_multi09:26:110
3279497993117,11cyclictest3363823-21cron12:25:000
3279497993117,11cyclictest3363823-21cron12:25:000
3279499993024,3cyclictest0-21swapper/208:40:012
3279499993023,3cyclictest3311787-21unixbench_singl09:31:122
3279499993023,3cyclictest3311787-21unixbench_singl09:31:122
3279499993021,6cyclictest1-21systemd07:35:012
3279499993021,5cyclictest3368232-21tail12:36:142
3279499993021,5cyclictest1051-21mta-sts-daemon10:51:032
3279499993012,8cyclictest171rcu_preempt09:16:042
327949899305,4cyclictest171rcu_preempt11:26:021
327949899302,5cyclictest171rcu_preempt10:31:011
3279498993022,5cyclictest439-21systemd-logind11:25:021
3279498993022,5cyclictest3305044-21latency09:10:571
3279498993022,5cyclictest1056-21mta-sts-daemon12:36:011
3279498993020,7cyclictest1-21systemd07:45:011
3279498993020,7cyclictest1-21systemd07:45:011
3279498993020,6cyclictest3320262-21latency_hist10:00:021
3279497993022,6cyclictest3305184-21needreboot09:10:590
327949799300,2cyclictest171rcu_preempt10:46:020
327949799300,2cyclictest171rcu_preempt10:46:020
327949999298,8cyclictest3330223-21ntpq10:31:012
3279499992921,5cyclictest3293337-21cron08:25:012
3279499992919,3cyclictest171rcu_preempt10:05:572
3279499992915,7cyclictest3345765-21kworker/u8:0+xprtiod11:54:592
3279499992911,3cyclictest171rcu_preempt08:45:582
3279499992910,3cyclictest171rcu_preempt08:25:582
3279498992923,3cyclictest0-21swapper/108:51:051
3279498992922,4cyclictest0-21swapper/111:54:541
3279497992922,4cyclictest3330661-21tail10:31:090
3279497992922,4cyclictest0-21swapper/012:25:580
3279497992921,5cyclictest3307282-21unixbench_singl09:16:100
327949799291,3cyclictest171rcu_preempt09:31:050
327949799291,3cyclictest171rcu_preempt09:31:050
327949999288,3cyclictest171rcu_preempt11:06:052
327949999287,3cyclictest171rcu_preempt10:46:032
327949999287,3cyclictest171rcu_preempt10:46:032
327949999282,17cyclictest2555081-21cups-browsed10:41:002
3279499992820,3cyclictest171rcu_preempt11:40:502
3279499992816,3cyclictest171rcu_preempt11:35:522
3279499992813,6cyclictest171rcu_preempt12:05:592
3279499992811,2cyclictest171rcu_preempt10:11:052
3279498992823,2cyclictest0-21swapper/110:25:011
3279498992822,3cyclictest0-21swapper/109:01:011
3279498992822,3cyclictest0-21swapper/108:46:071
3279498992821,4cyclictest3328634-21ntp_kernel_err10:26:001
3279498992821,4cyclictest3311743-21unixbench_multi09:31:121
3279498992821,4cyclictest3311743-21unixbench_multi09:31:121
3279498992821,3cyclictest0-21swapper/108:06:051
3279498992814,4cyclictest171rcu_preempt08:36:121
3279497992821,5cyclictest3338296-21systemd-run11:00:020
3279497992820,5cyclictest420-21avahi-daemon08:15:020
3279497992817,8cyclictest12350irq/32-eth012:11:000
3279497992817,8cyclictest12350irq/32-eth012:11:000
3279497992813,9cyclictest2555081-21cups-browsed11:01:110
3279499992721,3cyclictest0-21swapper/209:30:012
3279499992721,3cyclictest0-21swapper/209:30:012
3279499992720,4cyclictest3288503-21latency_hist08:00:022
3279499992720,4cyclictest3288503-21latency_hist08:00:022
3279499992719,5cyclictest1055-21mta-sts-daemon11:45:552
3279499992716,8cyclictest3312732-21ntp_kernel_pll_09:35:592
3279499992711,3cyclictest171rcu_preempt12:01:002
3279498992722,2cyclictest0-21swapper/111:50:021
3279498992721,3cyclictest0-21swapper/109:25:001
3279498992721,3cyclictest0-21swapper/109:20:001
3279498992720,4cyclictest3331305-21fschecks_time10:35:531
3279498992720,4cyclictest3309214-21aten_r2power_fr09:25:501
3279498992720,4cyclictest3309214-21aten_r2power_fr09:25:501
3279498992719,5cyclictest3321617-21tail10:01:091
3279498992719,5cyclictest1056-21mta-sts-daemon11:06:031
3279498992718,7cyclictest3293998-21kernelversion08:25:551
3279498992718,6cyclictest428-21mta-sts-daemon10:54:531
3279497992719,6cyclictest1054-21mta-sts-daemon11:44:490
3279497992714,10cyclictest3359279-21munin-plugin-st12:10:000
3279497992713,11cyclictest151ktimers/007:40:330
3279497992713,11cyclictest151ktimers/007:40:330
327949999268,7cyclictest36-21ksoftirqd/209:10:562
3279499992613,4cyclictest171rcu_preempt07:49:422
3279499992613,4cyclictest171rcu_preempt07:49:422
3279498992618,5cyclictest0-21swapper/108:20:011
327949799269,4cyclictest171rcu_preempt08:45:590
3279497992621,3cyclictest3338236-21kworker/u8:8-writeback12:20:010
3279497992621,3cyclictest3338236-21kworker/u8:8-writeback12:20:010
3279497992620,4cyclictest3279496-21cyclictest09:05:010
3279497992619,5cyclictest1051-21mta-sts-daemon08:26:020
3279497992619,4cyclictest2555081-21cups-browsed07:21:390
3279497992617,5cyclictest1054-21mta-sts-daemon07:10:560
327949799261,21cyclictest151ktimers/011:16:010
3279499992517,4cyclictest0-21swapper/208:35:012
3279499992515,3cyclictest171rcu_preempt10:21:012
3279499992514,4cyclictest171rcu_preempt12:15:532
3279499992514,4cyclictest171rcu_preempt12:15:532
327949899252,17cyclictest2555081-21cups-browsed12:11:021
327949899252,17cyclictest2555081-21cups-browsed12:11:021
3279498992519,3cyclictest0-21swapper/112:20:001
3279498992519,3cyclictest0-21swapper/112:20:001
3279498992517,5cyclictest3366841-21cron12:35:011
3279498992517,5cyclictest3342165-21ntpq11:11:001
3279498992517,5cyclictest2555081-21cups-browsed07:30:271
327949799253,13cyclictest171rcu_preempt11:26:020
3279497992520,2cyclictest0-21swapper/007:40:010
3279497992519,4cyclictest3288025-21cron07:55:000
3279497992519,4cyclictest3288025-21cron07:55:000
3279497992517,5cyclictest547-21gmain08:50:490
327949799251,16cyclictest151ktimers/009:41:030
3279497992511,10cyclictest0-21swapper/012:36:140
327949999246,4cyclictest351ktimers/207:20:012
3279498992418,3cyclictest3328008-21kworker/u8:0-xprtiod10:45:541
3279498992418,3cyclictest3328008-21kworker/u8:0-xprtiod10:45:541
3279498992417,4cyclictest3293336-21cron08:25:011
3279498992416,5cyclictest2555081-21cups-browsed08:44:491
3279497992418,4cyclictest1055-21mta-sts-daemon10:25:560
3279497992418,4cyclictest1055-21mta-sts-daemon09:20:510
3279497992417,4cyclictest3318705-21cron09:55:010
327949799241,3cyclictest171rcu_preempt07:31:040
3279497992413,2cyclictest171rcu_preempt08:25:020
3279498992318,2cyclictest0-21swapper/110:56:031
3279498992317,4cyclictest3279496-21cyclictest12:05:511
3279498992317,4cyclictest3279496-21cyclictest07:19:171
3279498992316,4cyclictest3322134-21aten2.4-expect10:05:511
3279498992315,5cyclictest1054-21mta-sts-daemon08:04:051
3279498992315,5cyclictest1054-21mta-sts-daemon08:04:051
3279497992318,3cyclictest1055-21mta-sts-daemon07:15:140
3279497992315,6cyclictest3288297-21sendmail08:00:000
3279497992315,6cyclictest3288297-21sendmail08:00:000
3279497992311,8cyclictest2555081-21cups-browsed10:41:110
327949999229,2cyclictest171rcu_preempt07:10:022
3279499992217,3cyclictest1051-21mta-sts-daemon07:25:432
327949899227,3cyclictest171rcu_preempt12:00:571
327949899221,6cyclictest2614761-21munin-node07:10:021
327949799228,11cyclictest3283084-21latency_hist07:30:020
327949799222,15cyclictest3289241-21ntpq08:01:020
327949799222,15cyclictest3289241-21ntpq08:01:020
327949799221,18cyclictest3325624-21ntpq10:16:030
327949799220,11cyclictest3350109-21unixbench-2d11:36:120
3279498992115,3cyclictest0-21swapper/107:50:011
3279498992115,3cyclictest0-21swapper/107:50:011
327949799211,14cyclictest151ktimers/008:06:040
3279498992010,7cyclictest3291543-21sshd08:12:421
327949799201,8cyclictest1-21systemd07:50:010
327949799201,8cyclictest1-21systemd07:50:010
3279497991913,4cyclictest3279449-21latency_hist07:10:020
327950099162,5cyclictest0-21swapper/309:41:033
327950099161,9cyclictest0-21swapper/311:54:583
327950099141,6cyclictest0-21swapper/309:26:053
327950099141,6cyclictest0-21swapper/309:26:053
327950099141,6cyclictest0-21swapper/308:51:013
327950099132,4cyclictest0-21swapper/312:11:003
327950099132,4cyclictest0-21swapper/312:11:003
327950099132,4cyclictest0-21swapper/309:46:063
327950099131,7cyclictest0-21swapper/311:06:043
327950099131,6cyclictest0-21swapper/310:46:013
327950099131,6cyclictest0-21swapper/310:46:013
327950099131,6cyclictest0-21swapper/310:20:593
327950099131,6cyclictest0-21swapper/309:01:043
327950099131,5cyclictest0-21swapper/311:35:593
327950099131,5cyclictest0-21swapper/309:55:593
327950099122,5cyclictest0-21swapper/310:00:593
327950099121,6cyclictest0-21swapper/312:36:023
327950099121,6cyclictest0-21swapper/312:21:003
327950099121,6cyclictest0-21swapper/312:21:003
327950099121,6cyclictest0-21swapper/312:15:593
327950099121,6cyclictest0-21swapper/312:15:593
327950099121,6cyclictest0-21swapper/311:55:593
327950099121,6cyclictest0-21swapper/309:51:073
327950099121,6cyclictest0-21swapper/309:15:013
327950099121,5cyclictest0-21swapper/310:40:593
327950099121,5cyclictest0-21swapper/310:30:013
327950099121,5cyclictest0-21swapper/310:16:043
327950099121,5cyclictest0-21swapper/309:20:593
327950099121,5cyclictest0-21swapper/307:41:003
327950099121,5cyclictest0-21swapper/307:41:003
327950099121,4cyclictest0-21swapper/312:31:003
327950099120,6cyclictest0-21swapper/307:11:003
327950099114,4cyclictest0-21swapper/308:45:543
327950099112,3cyclictest0-21swapper/308:06:043
327950099111,7cyclictest0-21swapper/311:40:533
327950099111,6cyclictest0-21swapper/310:11:113
327950099111,5cyclictest0-21swapper/312:01:073
327950099111,5cyclictest0-21swapper/311:01:053
327950099111,5cyclictest0-21swapper/310:50:593
327950099111,5cyclictest0-21swapper/310:05:533
327950099111,5cyclictest0-21swapper/309:15:503
327950099110,6cyclictest0-21swapper/309:35:583
327950099102,4cyclictest0-21swapper/312:26:103
327950099101,6cyclictest0-21swapper/311:25:533
327950099101,6cyclictest0-21swapper/308:45:013
327950099101,5cyclictest0-21swapper/312:06:033
327950099101,5cyclictest0-21swapper/311:20:573
327950099101,5cyclictest0-21swapper/310:35:583
327950099101,5cyclictest0-21swapper/310:30:563
327950099101,5cyclictest0-21swapper/309:31:003
327950099101,5cyclictest0-21swapper/309:31:003
327950099101,5cyclictest0-21swapper/308:35:593
327950099101,5cyclictest0-21swapper/308:25:583
327950099101,5cyclictest0-21swapper/307:21:053
327950099100,5cyclictest0-21swapper/308:01:063
327950099100,5cyclictest0-21swapper/308:01:063
32795009993,4cyclictest0-21swapper/311:15:583
32795009991,5cyclictest0-21swapper/311:31:003
32795009991,5cyclictest0-21swapper/311:10:523
32795009991,5cyclictest0-21swapper/308:15:543
32795009991,5cyclictest0-21swapper/308:00:013
32795009991,5cyclictest0-21swapper/308:00:013
32795009991,5cyclictest0-21swapper/307:50:513
32795009991,5cyclictest0-21swapper/307:50:513
32795009991,5cyclictest0-21swapper/307:40:013
32795009991,5cyclictest0-21swapper/307:20:023
32795009991,4cyclictest0-21swapper/311:00:033
32795009991,4cyclictest0-21swapper/309:05:513
32795009991,4cyclictest0-21swapper/308:35:033
32795009991,4cyclictest0-21swapper/307:30:023
32795009990,5cyclictest0-21swapper/308:15:013
32795009990,5cyclictest0-21swapper/307:30:583
32795009981,4cyclictest0-21swapper/308:56:033
32795009981,4cyclictest0-21swapper/308:25:013
32795009981,4cyclictest0-21swapper/307:50:023
32795009981,4cyclictest0-21swapper/307:50:023
32795009981,4cyclictest0-21swapper/307:10:033
32795009980,3cyclictest0-21swapper/311:45:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional