You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 01:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Sat May 23, 2026 00:43:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3286592998862,22cyclictest3467296-40mandb00:11:060
3286593998772,9cyclictest0-21swapper/100:11:001
3286592998566,13cyclictest3467533-21latency_hist00:10:030
3286592998566,13cyclictest3467533-21latency_hist00:10:030
3286594996960,5cyclictest0-21swapper/200:10:552
3286592996331,20cyclictest12350irq/32-eth021:40:530
3286594995949,4cyclictest0-21swapper/200:10:022
3286594995949,4cyclictest0-21swapper/200:10:022
3286593995546,4cyclictest0-21swapper/100:10:031
3286593995546,4cyclictest0-21swapper/100:10:031
3286593995243,6cyclictest3455402-21latency_hist23:55:021
3286593995140,6cyclictest0-21swapper/121:05:571
3286592995039,8cyclictest3309109-21systemd-run20:25:010
3286592995023,3cyclictest0-21swapper/021:56:080
3286592995023,3cyclictest0-21swapper/021:56:080
3286592995017,7cyclictest0-21swapper/023:40:580
3286594994936,8cyclictest420-21avahi-daemon22:15:582
3286593994738,5cyclictest3427172-21ssh23:20:011
3286593994736,7cyclictest3435192-21latency_hist23:30:011
3286592994735,8cyclictest1511301-21cups-browsed22:31:000
3286592994730,13cyclictest151ktimers/000:31:020
3286592994714,10cyclictest12350irq/32-eth023:31:020
3286592994713,11cyclictest3440605-21rm23:35:560
3286592994713,11cyclictest3440605-21rm23:35:560
3286592994628,10cyclictest12350irq/32-eth022:40:590
3286592994628,10cyclictest12350irq/32-eth022:40:590
3286594994537,5cyclictest0-21swapper/223:00:012
3286592994537,5cyclictest3411084-21munin-run23:00:010
328659299453,11cyclictest3415142-21kworker/u8:18+rpciod23:05:560
3286594994436,5cyclictest3326795-21latency_hist21:15:022
3286594994436,5cyclictest1051-21mta-sts-daemon22:50:582
3286592994432,4cyclictest0-21swapper/023:10:580
328659299443,21cyclictest3339989-21ssh21:31:030
328659299443,19cyclictest3364062-21ntpq22:00:590
328659299443,17cyclictest12350irq/32-eth022:11:030
3286592994423,17cyclictest0-21swapper/022:35:250
3286592994417,9cyclictest0-21swapper/022:25:010
3286594994335,5cyclictest3383484-21sh22:25:462
3286593994335,4cyclictest0-21swapper/121:35:571
328659299433,13cyclictest1511301-21cups-browsed22:06:030
3286593994232,7cyclictest420-21avahi-daemon22:55:581
3286592994234,3cyclictest0-21swapper/023:20:010
3286592994233,6cyclictest3455952-21tail23:55:450
3286592994233,6cyclictest3455952-21tail23:55:450
3286594994134,4cyclictest3439817-21acpi23:35:432
3286594994134,4cyclictest3439817-21acpi23:35:432
3286594994134,3cyclictest0-21swapper/221:45:012
3286593994134,4cyclictest3321102-21kworker/u8:0-rpciod22:45:021
3286593994134,4cyclictest3321102-21kworker/u8:0-rpciod22:45:021
3286593994129,8cyclictest3392037-21latency22:35:521
328659499402,32cyclictest3372150-21ssh22:11:032
328659499402,32cyclictest3331907-21ssh21:20:562
3286593994033,3cyclictest0-21swapper/122:55:011
3286593994032,5cyclictest3412348-21needreboot23:00:531
3286593994032,5cyclictest3412348-21needreboot23:00:531
3286593994028,9cyclictest3484947-21switchtime00:31:071
3286592994030,7cyclictest3313743-21latency_hist20:40:020
3286592994017,4cyclictest3475248-21idleruntime-cro00:20:010
3286594993931,5cyclictest3411743-21aten_r2power_cu23:00:452
3286594993931,5cyclictest3411743-21aten_r2power_cu23:00:452
3286594993931,5cyclictest3364599-21unixbench_multi22:01:092
3286593993931,5cyclictest3382814-21cut22:25:011
3286593993931,5cyclictest3319698-21latency_hist21:00:021
3286593993930,6cyclictest439-21systemd-logind19:10:011
3286593993930,6cyclictest1-21systemd20:45:011
3286593993930,5cyclictest3439250-21ssh23:35:021
3286592993933,3cyclictest0-21swapper/021:25:490
3286592993918,5cyclictest3459987-21/usr/sbin/munin00:01:110
328659299391,31cyclictest3356068-21ssh21:51:050
3286594993830,5cyclictest3362676-21/usr/sbin/munin22:00:022
3286594993830,5cyclictest3362676-21/usr/sbin/munin22:00:022
3286594993830,5cyclictest3330597-21idleruntime-cro21:20:012
3286594993830,4cyclictest0-21swapper/223:30:572
3286594993820,13cyclictest351ktimers/220:15:562
3286593993830,4cyclictest0-21swapper/120:01:011
3286593993829,6cyclictest421-21dbus-daemon20:20:021
3286593993829,6cyclictest420-21avahi-daemon20:26:061
3286593993829,6cyclictest0-21swapper/100:00:011
3286593993829,6cyclictest0-21swapper/100:00:011
3286593993829,5cyclictest1056-21mta-sts-daemon21:33:161
3286593993828,7cyclictest1-21systemd19:30:011
328659299383,13cyclictest3400434-21ssh22:46:000
3286594993729,5cyclictest0-21swapper/200:00:012
3286594993729,5cyclictest0-21swapper/200:00:012
3286594993728,6cyclictest3388307-21ssh22:31:002
3286593993730,4cyclictest0-21swapper/121:20:011
3286593993729,5cyclictest3423133-21true23:15:011
3286593993729,5cyclictest3300098-21systemd-run19:55:011
3286593993729,5cyclictest3300098-21systemd-run19:55:011
3286593993726,8cyclictest3459337-21cups-browsed00:26:011
3286593993722,7cyclictest3440510-21sh23:35:531
3286593993722,7cyclictest3440510-21sh23:35:531
3286592993729,5cyclictest12350irq/32-eth023:01:000
3286592993729,5cyclictest12350irq/32-eth023:01:000
3286592993727,7cyclictest3477555-21sshd00:21:410
3286592993719,15cyclictest3435215-21systemd-run23:30:010
3286594993628,5cyclictest3448053-21aten_r2power_po23:45:462
3286594993626,7cyclictest1511301-21cups-browsed21:25:052
328659499362,27cyclictest3355915-21ssh21:50:592
3286594993614,12cyclictest351ktimers/221:31:042
3286593993628,5cyclictest3463578-21latency_hist00:05:021
3286593993627,6cyclictest0-21swapper/120:40:011
3286593993627,6cyclictest0-21swapper/100:40:011
3286593993627,6cyclictest0-21swapper/100:40:011
3286593993627,5cyclictest0-21swapper/121:45:011
3286593993619,12cyclictest3315075-21kworker/u8:2+nfsiod21:55:581
3286593993619,12cyclictest3315075-21kworker/u8:2+nfsiod21:55:581
3286592993629,5cyclictest3377092-21sshd22:16:480
3286592993628,5cyclictest3322583-21diskmemload21:35:500
3286592993611,3cyclictest171rcu_preempt21:45:570
3286594993527,5cyclictest3483217-21cron00:30:012
3286594993527,5cyclictest3346518-21cron21:40:012
3286594993527,5cyclictest3289592-21sendmail19:20:002
3286593993529,3cyclictest0-21swapper/119:40:011
3286593993525,7cyclictest3371838-21latency22:10:561
3286593993525,6cyclictest3431176-21idleruntime-cro23:25:011
3286593993524,8cyclictest3363835-21missed_timers22:00:531
3286592993527,5cyclictest3448222-21sort23:45:490
3286594993426,5cyclictest1511301-21cups-browsed21:46:352
3286594993426,4cyclictest0-21swapper/219:31:022
3286594993425,6cyclictest3319558-21cron21:00:002
3286594993422,8cyclictest3367805-21latency22:05:522
3286594993419,4cyclictest0-21swapper/223:21:592
3286593993426,5cyclictest3477240-21grep00:21:091
3286593993426,5cyclictest3368704-21/usr/sbin/munin22:06:091
328659299345,15cyclictest3452654-21ssh23:50:580
3286592993426,4cyclictest1056-21mta-sts-daemon22:52:080
3286592993425,6cyclictest3431181-21munin-plugin-st23:25:010
3286594993327,3cyclictest0-21swapper/223:55:012
3286594993325,5cyclictest3322576-21cron21:10:002
3286594993324,6cyclictest1-21systemd22:50:012
328659499332,24cyclictest3424480-21ssh23:15:582
3286593993327,3cyclictest0-21swapper/122:50:011
3286593993327,3cyclictest0-21swapper/122:17:011
3286593993326,4cyclictest3475361-21ssh00:20:021
3286593993326,4cyclictest0-21swapper/119:20:011
3286592993324,7cyclictest3491171-21munin-plugin-st00:40:020
3286592993324,7cyclictest3491171-21munin-plugin-st00:40:020
3286594993227,2cyclictest0-21swapper/200:21:072
3286594993226,3cyclictest0-21swapper/219:55:002
3286594993226,3cyclictest0-21swapper/219:55:002
3286594993225,4cyclictest3487190-21cron00:35:012
3286594993224,5cyclictest3394946-21cron22:40:012
3286594993224,5cyclictest3298598-21systemd-run19:50:012
3286594993224,5cyclictest3291121-21systemd-run19:25:012
3286594993222,5cyclictest0-21swapper/219:35:542
3286594993220,9cyclictest3432765-21mailstats23:26:042
328659499321,25cyclictest3472600-21ssh00:15:572
3286593993224,5cyclictest3386857-21cron22:30:011
3286592993226,3cyclictest0-21swapper/022:25:550
3286592993224,6cyclictest1056-21mta-sts-daemon21:10:460
3286594993124,4cyclictest3463458-21latency_hist00:05:012
3286594993122,4cyclictest0-21swapper/220:05:572
3286594993120,8cyclictest3488514-21ntp_kernel_pll_00:35:562
3286594993120,8cyclictest3488514-21ntp_kernel_pll_00:35:562
3286594993120,8cyclictest3312977-21ntp_kernel_pll_20:35:552
328659499311,5cyclictest171rcu_preempt20:26:012
3286593993123,5cyclictest3291089-21cron19:25:011
3286593993123,4cyclictest0-21swapper/123:05:571
328659399312,21cyclictest1055-21mta-sts-daemon21:46:021
3286593993119,9cyclictest420-21avahi-daemon19:45:481
3286593993119,9cyclictest3311481-21ntp_kernel_pll_20:30:581
3286592993123,5cyclictest1511301-21cups-browsed21:18:050
3286592993123,5cyclictest1511301-21cups-browsed20:00:500
3286592993121,7cyclictest0-21swapper/020:15:020
3286592993120,8cyclictest3334722-21latency_hist21:25:020
328659499302,22cyclictest441-21gpm20:41:002
3286594993021,6cyclictest3447316-21systemd-run23:45:012
3286594993021,6cyclictest3291431-21aten_r2power_fr19:25:462
328659499301,22cyclictest3396505-21grep22:41:032
328659499301,22cyclictest3396505-21grep22:41:032
3286593993024,3cyclictest0-21swapper/121:25:551
3286593993019,8cyclictest624-21lldpd21:11:041
3286593993019,8cyclictest3296437-21ntp_kernel_err19:40:551
3286594992924,2cyclictest0-21swapper/219:44:422
3286594992923,3cyclictest0-21swapper/223:06:072
3286594992922,4cyclictest3423112-21munin-run23:15:002
3286594992921,5cyclictest3379718-21fschecks_count22:20:502
3286594992921,5cyclictest3288113-21grep19:15:022
3286594992921,5cyclictest3288113-21grep19:15:022
328659399293,18cyclictest3444734-21rm23:41:001
3286593992922,3cyclictest0-21swapper/121:50:041
3286593992922,3cyclictest0-21swapper/121:20:581
3286593992920,6cyclictest1-21systemd20:25:011
3286592992923,4cyclictest0-21swapper/020:35:010
3286592992922,4cyclictest3294099-21systemd-run19:35:010
3286592992921,5cyclictest3287976-21tail19:11:120
3286592992921,5cyclictest3287976-21tail19:11:120
3286592992919,5cyclictest12350irq/32-eth000:25:570
3286594992821,4cyclictest0-21swapper/220:00:002
3286594992819,6cyclictest3309110-21true20:25:012
3286593992820,5cyclictest3294005-21unixbench_singl19:31:121
3286592992819,7cyclictest3288094-21kworker/u8:13-xprtiod19:26:060
3286594992721,3cyclictest3303318-21kworker/u8:0-xprtiod20:10:422
3286594992721,3cyclictest0-21swapper/220:55:002
3286594992720,4cyclictest0-21swapper/221:00:592
3286593992721,3cyclictest0-21swapper/122:30:471
3286593992720,4cyclictest1054-21mta-sts-daemon20:05:361
3286593992718,5cyclictest0-21swapper/123:45:581
3286592992722,2cyclictest0-21swapper/021:10:000
3286592992721,3cyclictest0-21swapper/019:40:010
328659499267,5cyclictest62450lldpd20:48:172
3286594992619,2cyclictest0-21swapper/220:00:542
3286593992620,3cyclictest0-21swapper/120:55:001
3286593992614,9cyclictest3300975-21ntp_kernel_pll_19:55:571
328659299265,4cyclictest171rcu_preempt20:16:000
328659299264,13cyclictest428-21mta-sts-daemon21:00:580
328659299263,13cyclictest428-21mta-sts-daemon20:45:560
3286592992620,4cyclictest3289626-21systemd-run19:20:010
3286592992619,4cyclictest3317784-21threads20:51:220
3286592992617,6cyclictest3290936-21unixbench-2d19:21:090
3286592992614,9cyclictest0-21swapper/020:56:070
3286594992519,3cyclictest0-21swapper/219:10:012
3286594992517,6cyclictest351ktimers/220:30:552
3286593992519,4cyclictest3286591-21cyclictest20:47:371
3286593992519,3cyclictest0-21swapper/119:15:011
3286593992519,3cyclictest0-21swapper/119:15:011
3286593992516,6cyclictest3304851-21aten_r2power_cu20:10:451
3286592992519,4cyclictest3309458-21aten_r2power_po20:25:470
3286592992518,5cyclictest1511301-21cups-browsed19:56:030
3286592992418,3cyclictest3314956-21sed20:41:070
3286593992318,2cyclictest0-21swapper/121:01:101
328659299232,7cyclictest3298305-21wc19:46:080
3286592992317,3cyclictest3304598-21sh20:10:010
3286592992316,4cyclictest3296269-21latency19:40:520
328659599214,11cyclictest0-21swapper/300:10:543
328659599203,9cyclictest0-21swapper/300:09:563
328659599203,9cyclictest0-21swapper/300:09:563
328659299202,15cyclictest3286548-21munin-run19:10:010
3286592992013,6cyclictest3297115-21kworker/u8:2-xprtiod19:50:460
3286592992013,6cyclictest3297115-21kworker/u8:2-xprtiod19:50:460
328659599191,8cyclictest0-21swapper/321:20:583
328659599183,9cyclictest0-21swapper/323:56:003
328659599183,9cyclictest0-21swapper/323:56:003
328659599183,9cyclictest0-21swapper/322:55:563
328659599183,9cyclictest0-21swapper/322:15:583
328659599182,8cyclictest0-21swapper/321:05:573
328659599181,9cyclictest0-21swapper/322:45:593
328659599181,8cyclictest0-21swapper/321:25:563
328659599175,7cyclictest0-21swapper/322:11:023
328659599174,8cyclictest0-21swapper/323:40:593
328659599174,4cyclictest0-21swapper/320:45:553
328659599173,8cyclictest0-21swapper/323:30:593
328659599173,8cyclictest0-21swapper/323:25:583
328659599173,7cyclictest0-21swapper/321:15:563
328659599173,5cyclictest0-21swapper/320:10:583
328659599172,9cyclictest0-21swapper/321:10:553
328659599172,9cyclictest0-21swapper/300:26:013
328659599171,9cyclictest0-21swapper/319:45:593
328659599163,8cyclictest0-21swapper/323:06:003
328659599163,7cyclictest0-21swapper/300:00:593
328659599163,5cyclictest0-21swapper/319:25:573
328659599162,9cyclictest0-21swapper/322:50:553
328659599162,9cyclictest0-21swapper/321:36:023
328659599162,8cyclictest0-21swapper/323:20:563
328659599162,8cyclictest0-21swapper/322:25:583
328659599162,8cyclictest0-21swapper/321:30:563
328659599162,8cyclictest0-21swapper/300:35:583
328659599162,8cyclictest0-21swapper/300:35:583
328659599162,7cyclictest0-21swapper/321:40:473
328659599161,8cyclictest0-21swapper/323:11:033
328659599161,7cyclictest0-21swapper/320:30:573
328659599161,7cyclictest0-21swapper/319:55:563
328659599161,6cyclictest0-21swapper/321:01:003
328659599160,9cyclictest0-21swapper/322:00:573
328659599153,8cyclictest0-21swapper/320:05:583
328659599152,8cyclictest0-21swapper/323:50:413
328659599152,8cyclictest0-21swapper/323:50:013
328659599152,8cyclictest0-21swapper/323:35:533
328659599152,8cyclictest0-21swapper/323:35:533
328659599152,8cyclictest0-21swapper/322:30:563
328659599152,6cyclictest0-21swapper/319:16:013
328659599152,5cyclictest0-21swapper/319:40:583
328659599151,8cyclictest0-21swapper/323:00:513
328659599151,8cyclictest0-21swapper/323:00:513
328659599151,8cyclictest0-21swapper/322:35:573
328659599151,8cyclictest0-21swapper/321:45:513
328659599151,8cyclictest0-21swapper/300:20:473
328659599151,7cyclictest0-21swapper/320:40:543
328659599151,6cyclictest0-21swapper/320:35:543
328659599151,6cyclictest0-21swapper/320:25:593
328659599142,8cyclictest0-21swapper/323:15:543
328659599142,8cyclictest0-21swapper/322:20:463
328659599142,7cyclictest0-21swapper/300:30:563
328659599142,7cyclictest0-21swapper/300:15:573
328659599142,6cyclictest0-21swapper/319:30:593
328659599141,8cyclictest0-21swapper/322:05:563
328659599141,8cyclictest0-21swapper/321:50:593
328659599141,7cyclictest0-21swapper/319:36:013
328659599133,6cyclictest0-21swapper/321:55:563
328659599133,6cyclictest0-21swapper/321:55:563
328659599132,8cyclictest0-21swapper/322:40:483
328659599132,8cyclictest0-21swapper/322:40:483
328659599132,6cyclictest0-21swapper/319:25:013
328659599132,5cyclictest0-21swapper/319:10:533
328659599132,5cyclictest0-21swapper/319:10:533
328659599131,5cyclictest0-21swapper/320:15:573
328659599123,5cyclictest0-21swapper/319:50:563
328659599123,5cyclictest0-21swapper/319:50:563
328659599121,7cyclictest0-21swapper/320:00:553
328659599121,6cyclictest0-21swapper/320:20:543
328659599121,5cyclictest0-21swapper/320:51:123
328659599101,6cyclictest0-21swapper/320:55:523
32865959991,4cyclictest0-21swapper/319:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional