You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 01:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Wed Jun 03, 2026 00:42:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3267904996253,6cyclictest0-21swapper/223:15:522
326790399615,41cyclictest3415785-21rm00:11:051
3267902996125,7cyclictest0-21swapper/022:56:060
3267904996050,6cyclictest1055-21mta-sts-daemon20:41:032
326790499545,38cyclictest3399876-21ssh23:46:022
3267902995236,7cyclictest12450irq/33-eth023:41:050
3267902995236,7cyclictest12450irq/33-eth023:41:050
3267903995038,7cyclictest0-21swapper/121:26:021
326790399503,38cyclictest3380810-21ssh23:16:031
326790299502,44cyclictest3380276-21/usr/sbin/munin23:15:520
3267902994935,11cyclictest3292690-21ls20:45:000
3267904994735,9cyclictest0-21swapper/200:21:062
3267904994537,5cyclictest3433561-21munin-run00:40:002
3267904994537,5cyclictest3433561-21munin-run00:40:002
326790499453,31cyclictest3403005-21ssh23:51:032
326790499452,37cyclictest3374529-21ssh23:06:112
326790399453,31cyclictest3329969-21ssh21:56:021
3267902994537,5cyclictest3269402-21systemd-run19:15:010
3267902994532,10cyclictest3377975-21timerandwakeup23:11:140
3267904994435,6cyclictest0-21swapper/221:45:022
3267904994432,9cyclictest421-21dbus-daemon21:35:002
3267903994433,7cyclictest420-21avahi-daemon21:40:091
3267902994425,11cyclictest151ktimers/000:26:020
3267902994425,11cyclictest151ktimers/000:26:020
3267902994416,21cyclictest151ktimers/023:56:010
326790399427,25cyclictest1054-21mta-sts-daemon00:01:041
3267903994229,7cyclictest3304696-21sh21:16:071
3267903994216,23cyclictest0-21swapper/123:31:071
326790299424,30cyclictest3387231-21ssh23:26:060
3267902994224,4cyclictest0-21swapper/000:06:100
3267904994131,7cyclictest1525705-21cups-browsed22:29:282
326790399419,9cyclictest3290554-21ntpq20:36:061
326790399412,30cyclictest3355401-21ssh22:36:051
3267903994115,8cyclictest3338084-21ssh22:09:241
3267902994135,3cyclictest0-21swapper/022:51:080
3267902994134,4cyclictest3273183-21grep19:40:000
3267902994134,3cyclictest0-21swapper/021:31:160
3267904994034,4cyclictest0-21swapper/221:55:022
3267903994032,3cyclictest0-21swapper/100:36:091
3267903994032,3cyclictest0-21swapper/100:36:091
3267904993929,6cyclictest3408200-21systemd-run00:00:012
3267904993929,4cyclictest0-21swapper/219:11:042
326790399398,22cyclictest0-21swapper/123:21:071
3267903993932,4cyclictest3335154-21systemd-run22:05:011
3267902993932,4cyclictest3420901-21systemd-run00:20:010
3267902993931,6cyclictest3345686-21sh22:21:010
3267902993930,5cyclictest3336378-21ntp_kernel_pll_22:06:040
3267902993929,7cyclictest1-21systemd21:10:010
3267904993830,5cyclictest3385955-21systemd-run23:25:022
3267904993830,5cyclictest0-21swapper/221:00:002
3267904993825,9cyclictest3364874-21sh22:51:032
3267903993830,6cyclictest3297194-21sendmail21:00:001
3267903993830,5cyclictest3326136-21head21:50:531
3267903993827,4cyclictest0-21swapper/123:26:091
3267903993811,8cyclictest3428438-21ntpq00:31:041
3267903993811,8cyclictest3428438-21ntpq00:31:041
3267902993822,10cyclictest12450irq/33-eth023:31:010
3267904993730,4cyclictest3283663-21kworker/u8:6-rpciod21:25:582
3267904993728,6cyclictest3411505-21latency_hist00:05:022
3267904993727,7cyclictest3408135-21cups-browsed00:30:362
3267904993727,7cyclictest3408135-21cups-browsed00:30:362
3267904993726,8cyclictest3287538-21ntp_kernel_err20:26:002
3267904993726,6cyclictest0-21swapper/220:21:062
326790399374,25cyclictest3403167-21ssh23:51:101
3267903993731,3cyclictest0-21swapper/121:50:011
3267903993728,6cyclictest1525705-21cups-browsed20:05:021
3267903993713,4cyclictest3371293-21ntpq23:01:051
326790299377,24cyclictest3428538-21ssh00:31:070
326790299377,24cyclictest3428538-21ssh00:31:070
326790299373,27cyclictest3431671-21ssh00:36:060
326790299373,27cyclictest3431671-21ssh00:36:060
3267902993730,5cyclictest3269690-21systemd-run19:20:010
3267904993628,5cyclictest3358138-21fschecks_count22:40:552
3267904993628,5cyclictest0-21swapper/220:20:012
326790399369,7cyclictest3427238-21munin-run00:30:011
326790399369,7cyclictest3427238-21munin-run00:30:011
3267903993630,4cyclictest3267901-21cyclictest20:06:071
3267903993628,5cyclictest3319228-21cron21:40:011
3267903993611,8cyclictest3361734-21ntpq22:46:041
3267903993610,9cyclictest3268798-21ntpq19:11:021
326790299363,2cyclictest0-21swapper/023:46:030
3267902993628,5cyclictest3339317-21latency22:10:580
3267904993529,3cyclictest0-21swapper/221:40:012
3267904993528,5cyclictest1055-21mta-sts-daemon23:11:482
3267904993528,4cyclictest0-21swapper/200:16:102
3267904993526,6cyclictest351ktimers/223:41:042
3267904993526,6cyclictest351ktimers/223:41:042
3267904993526,4cyclictest0-21swapper/220:31:052
3267904993524,7cyclictest1525705-21cups-browsed22:39:022
3267904993523,6cyclictest547-21gmain21:15:032
326790399359,10cyclictest3418316-21aten_r2power_fr00:15:551
326790399353,25cyclictest3393476-21ssh23:36:011
326790399353,25cyclictest3393476-21ssh23:36:011
326790399353,24cyclictest3345980-21ssh22:21:091
3267903993527,5cyclictest3406809-21grep23:56:141
3267903993524,8cyclictest439-21systemd-logind19:50:011
3267903993524,7cyclictest3274040-21ntpq19:40:561
3267903993517,12cyclictest3276682-21diskstats19:50:541
326790399351,29cyclictest3377553-21ssh23:11:021
3267902993529,4cyclictest3357289-21cron22:40:010
3267902993529,3cyclictest0-21swapper/022:00:520
3267902993527,5cyclictest3319763-21/usr/sbin/munin21:40:580
3267904993428,3cyclictest0-21swapper/221:10:002
3267904993428,3cyclictest0-21swapper/220:40:012
3267904993426,4cyclictest0-21swapper/222:56:092
326790399343,24cyclictest3349125-21ssh22:26:051
3267903993429,3cyclictest0-21swapper/123:45:001
3267903993429,3cyclictest0-21swapper/123:45:001
3267903993427,4cyclictest0-21swapper/121:11:111
3267903993427,3cyclictest0-21swapper/122:20:011
3267903993427,3cyclictest0-21swapper/100:25:011
3267903993426,5cyclictest3354326-21latency_hist22:35:031
3267903993425,5cyclictest3413188-21ssh00:06:181
3267903993423,7cyclictest1525705-21cups-browsed20:48:351
326790399342,24cyclictest3374412-21ssh23:06:061
3267903993420,8cyclictest3284647-21ntpq20:16:071
3267903993419,10cyclictest3291423-21/usr/sbin/munin20:41:161
326790299343,23cyclictest3310971-21ssh21:26:030
3267902993427,4cyclictest3344709-21latency_hist22:20:020
326790299342,25cyclictest3300186-21diskmemload23:36:040
326790299342,25cyclictest3300186-21diskmemload23:36:040
3267902993414,14cyclictest12350irq/32-eth022:41:010
326790499336,20cyclictest3390535-21awk23:31:102
3267904993325,5cyclictest1056-21mta-sts-daemon23:36:022
3267904993325,5cyclictest1056-21mta-sts-daemon23:36:022
3267904993325,4cyclictest0-21swapper/222:01:032
3267904993324,6cyclictest420-21avahi-daemon00:10:012
326790499332,23cyclictest3415796-21ssh00:11:052
3267903993322,7cyclictest548-21gdbus20:00:011
3267903993321,8cyclictest3286119-21ntpq20:21:041
3267903993321,8cyclictest3270560-21ntpq19:20:581
3267903993320,8cyclictest3299580-21ntpq21:06:031
3267902993327,4cyclictest3277681-21munin-plugin-st19:55:010
3267902993326,4cyclictest3279170-21sendmail20:00:010
3267902993325,5cyclictest0-21swapper/020:20:010
3267902993317,9cyclictest151ktimers/020:01:030
326790299331,29cyclictest3317235-21ntpq21:36:000
326790299331,24cyclictest3422082-21ssh00:21:030
3267904993226,4cyclictest3267901-21cyclictest22:05:592
3267904993225,4cyclictest3329767-21latency21:55:572
3267904993224,5cyclictest3303553-21latency_hist21:15:022
326790499322,24cyclictest3425278-21ssh00:26:032
326790499322,24cyclictest3425278-21ssh00:26:032
326790499322,24cyclictest3361708-21ssh22:46:032
326790399322,21cyclictest3306813-21sshd21:20:081
3267903993221,7cyclictest3272322-21ntpq19:30:591
3267903993220,8cyclictest3298104-21ntpq21:01:041
3267903993220,7cyclictest3314153-21ntpq21:31:151
3267903993218,11cyclictest3295101-21ntpq20:51:081
3267903993215,11cyclictest3287549-21ntpq20:26:001
3267903993210,7cyclictest3283705-21munin-run20:15:021
3267902993225,3cyclictest151ktimers/021:01:020
3267902993224,3cyclictest0-21swapper/022:46:070
326790299321,25cyclictest1525705-21cups-browsed19:21:000
3267904993124,4cyclictest3279159-21cron20:00:012
3267904993121,7cyclictest3324228-21unixbench_multi21:46:182
3267903993124,4cyclictest0-21swapper/119:20:011
3267903993120,7cyclictest3368058-21ntpq22:56:041
3267903993119,7cyclictest3289038-21ntpq20:31:021
3267903993117,11cyclictest3339581-21ntpq22:11:051
3267903993110,5cyclictest3300186-21diskmemload22:43:191
3267902993125,4cyclictest3283682-21/usr/sbin/munin20:15:020
3267902993124,5cyclictest1051-21mta-sts-daemon23:23:090
3267902993124,3cyclictest0-21swapper/021:48:240
3267902993122,6cyclictest3354123-21cron22:35:010
3267902993117,10cyclictest12350irq/32-eth021:11:100
3267904993023,4cyclictest1051-21mta-sts-daemon19:46:072
3267904993022,5cyclictest3294076-21unixbench_multi20:46:142
3267904993022,3cyclictest0-21swapper/223:01:022
326790499302,20cyclictest3339490-21ssh22:11:022
326790399304,22cyclictest3399969-21ssh23:46:061
326790399302,23cyclictest3364960-21ssh22:51:061
326790299303,22cyclictest1017-21sendmail-mta23:01:070
3267902993024,4cyclictest3376544-21latency_hist23:10:020
326790299302,23cyclictest3409522-21sh00:01:090
3267902993020,7cyclictest3279416-21kworker/u8:11+xprtiod20:10:010
3267902993016,11cyclictest151ktimers/000:11:080
3267904992923,3cyclictest0-21swapper/219:20:012
3267904992921,5cyclictest3274706-21systemd-run19:45:012
3267904992920,5cyclictest1056-21mta-sts-daemon22:31:482
3267904992918,7cyclictest3344744-21latency_hist22:20:022
3267903992916,10cyclictest3271400-21sshd19:27:071
3267903992916,10cyclictest3271400-21sshd19:27:071
3267902992923,3cyclictest3289699-21kworker/u8:9-xprtiod20:35:020
326790299292,19cyclictest3329947-21ssh21:56:010
3267902992920,7cyclictest3288049-21unixbench-2d20:26:120
3267902992911,15cyclictest3348995-21ntpq22:26:010
3267904992822,3cyclictest0-21swapper/222:21:162
3267904992821,4cyclictest3309764-21cat21:25:022
326790299281,22cyclictest3304731-21rm21:16:090
326790299281,13cyclictest3326728-21cut21:51:020
3267904992720,3cyclictest0-21swapper/220:01:012
3267903992713,10cyclictest3273182-21/usr/sbin/munin19:40:011
3267902992721,4cyclictest3307316-21cpuspeed21:20:540
3267902992717,3cyclictest0-21swapper/019:45:550
326790299271,22cyclictest3403015-21sh23:51:040
3267904992620,3cyclictest0-21swapper/219:40:012
3267904992619,4cyclictest3387183-21ssh23:26:042
3267904992618,5cyclictest3283258-21systemctl20:11:092
3267904992618,5cyclictest1525705-21cups-browsed19:20:492
3267902992621,2cyclictest0-21swapper/020:46:140
3267902992619,5cyclictest3274393-21switchtime19:41:090
3267902992618,6cyclictest695-21cron21:00:000
3267902992616,7cyclictest3294990-21needreboot20:51:040
3267904992519,3cyclictest0-21swapper/219:30:002
3267904992519,3cyclictest0-21swapper/219:30:002
3267904992517,5cyclictest3294515-21aten_r2power_fr20:50:542
3267904992517,5cyclictest3271690-21aten_r2power_cu19:30:482
3267904992416,5cyclictest1525705-21cups-browsed19:51:392
3267904992415,6cyclictest3298725-21latency_hist21:05:012
326790299239,11cyclictest3271571-21latency_hist19:30:020
326790299239,11cyclictest3271571-21latency_hist19:30:020
3267902992317,4cyclictest3291177-21sendmail20:40:010
3267902992215,5cyclictest1051-21mta-sts-daemon19:31:000
3267904992112,6cyclictest3281595-21ntpq20:06:062
3267903992112,6cyclictest3267913-21systemd-run19:10:011
3267902992110,9cyclictest0-21swapper/020:20:510
326790599192,8cyclictest0-21swapper/323:51:063
326790599191,8cyclictest0-21swapper/322:41:013
326790299192,14cyclictest3267858-21munin-run19:10:000
326790599173,8cyclictest0-21swapper/323:01:113
3267904991710,4cyclictest3267852-21latency_hist19:10:022
326790599165,8cyclictest0-21swapper/321:26:023
326790599164,5cyclictest0-21swapper/322:11:023
326790599164,5cyclictest0-21swapper/322:01:153
326790599163,7cyclictest0-21swapper/300:21:073
326790599155,6cyclictest0-21swapper/321:21:053
326790599152,9cyclictest0-21swapper/300:06:103
326790599152,8cyclictest0-21swapper/319:11:063
326790599152,7cyclictest0-21swapper/323:36:013
326790599152,7cyclictest0-21swapper/323:36:013
326790599152,7cyclictest0-21swapper/322:46:023
326790599152,7cyclictest0-21swapper/300:16:043
326790599151,6cyclictest0-21swapper/323:56:013
326790599151,6cyclictest0-21swapper/323:21:023
326790599151,6cyclictest0-21swapper/321:16:023
326790599151,6cyclictest0-21swapper/300:11:023
326790599151,6cyclictest0-21swapper/300:01:063
326790599146,4cyclictest0-21swapper/322:26:043
326790599143,7cyclictest0-21swapper/300:31:103
326790599143,7cyclictest0-21swapper/300:31:103
326790599143,5cyclictest0-21swapper/323:46:053
326790599143,4cyclictest0-21swapper/321:06:033
326790599142,8cyclictest0-21swapper/323:41:093
326790599142,8cyclictest0-21swapper/323:41:093
326790599142,7cyclictest0-21swapper/323:05:583
326790599141,7cyclictest0-21swapper/322:51:093
326790599141,7cyclictest0-21swapper/322:16:133
326790599141,7cyclictest0-21swapper/321:41:023
326790599141,7cyclictest0-21swapper/321:36:073
326790599141,7cyclictest0-21swapper/300:26:053
326790599141,7cyclictest0-21swapper/300:26:053
326790599141,6cyclictest0-21swapper/323:10:573
326790599141,6cyclictest0-21swapper/320:36:053
326790599137,4cyclictest0-21swapper/322:40:013
326790599135,5cyclictest0-21swapper/323:16:043
326790599134,6cyclictest0-21swapper/323:31:063
326790599132,7cyclictest0-21swapper/321:55:553
326790599132,6cyclictest0-21swapper/320:11:073
326790599132,5cyclictest0-21swapper/320:41:033
326790599132,5cyclictest0-21swapper/319:56:043
326790599131,7cyclictest0-21swapper/321:11:043
326790599131,7cyclictest0-21swapper/300:36:063
326790599131,7cyclictest0-21swapper/300:36:063
326790599131,6cyclictest0-21swapper/322:56:033
326790599131,6cyclictest0-21swapper/321:31:163
326790599131,6cyclictest0-21swapper/320:06:093
326790599122,7cyclictest0-21swapper/323:26:093
326790599122,6cyclictest0-21swapper/321:00:003
326790599122,5cyclictest0-21swapper/321:45:523
326790599122,5cyclictest0-21swapper/319:30:003
326790599122,5cyclictest0-21swapper/319:30:003
326790599121,7cyclictest0-21swapper/322:21:063
326790599121,6cyclictest0-21swapper/320:21:013
326790599120,6cyclictest0-21swapper/320:51:073
326790599112,6cyclictest0-21swapper/320:01:013
326790599112,4cyclictest0-21swapper/319:20:563
326790599111,6cyclictest0-21swapper/322:05:533
326790599111,6cyclictest0-21swapper/321:51:043
326790599111,6cyclictest0-21swapper/321:01:063
326790599111,6cyclictest0-21swapper/320:46:023
326790599111,6cyclictest0-21swapper/319:41:043
326790599111,5cyclictest0-21swapper/322:30:523
326790599107,2cyclictest421rcuc/319:10:013
326790599103,4cyclictest0-21swapper/319:46:023
326790599102,5cyclictest0-21swapper/320:31:053
326790599101,5cyclictest0-21swapper/320:25:553
326790599101,5cyclictest0-21swapper/320:15:563
326790599101,5cyclictest0-21swapper/319:35:023
32679059991,5cyclictest0-21swapper/319:18:063
32679059991,4cyclictest0-21swapper/319:51:013
32679059982,4cyclictest0-21swapper/319:36:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional