You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 04:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Thu May 21, 2026 00:43:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3565323996048,8cyclictest261-21systemd-journal00:18:442
3565323996048,8cyclictest261-21systemd-journal00:18:442
3565323995745,8cyclictest1056-21mta-sts-daemon21:11:012
3565323995645,7cyclictest3663384-21unixbench_singl22:26:142
3565323995645,7cyclictest3663384-21unixbench_singl22:26:142
3565323995244,5cyclictest3685516-21/usr/sbin/munin22:55:012
3565323994938,5cyclictest0-21swapper/220:30:592
3565321994939,6cyclictest695-21cron19:35:000
3565323994737,7cyclictest3598387-21latency_hist21:00:022
3565321994733,7cyclictest12350irq/32-eth021:40:590
3565321994733,7cyclictest12350irq/32-eth021:40:590
3565323994637,5cyclictest0-21swapper/222:20:012
3565322994636,6cyclictest3609103-21basename21:20:011
3565323994535,7cyclictest1051-21mta-sts-daemon23:11:052
3565323994535,7cyclictest1051-21mta-sts-daemon23:11:052
3565323994535,6cyclictest1-21systemd22:25:022
3565323994535,6cyclictest1-21systemd22:25:022
3565323994438,3cyclictest0-21swapper/200:10:552
3565322994433,4cyclictest0-21swapper/121:41:031
3565322994433,4cyclictest0-21swapper/121:41:031
3565322994431,9cyclictest0-21swapper/122:10:011
3565321994426,15cyclictest3630200-21if_wlan021:45:510
3565321994426,15cyclictest3630200-21if_wlan021:45:510
3565322994334,6cyclictest3641251-21munin-run22:00:011
356532199433,30cyclictest3719310-21ssh23:36:000
3565323994232,6cyclictest3621215-21latency_hist21:35:022
356532299424,32cyclictest3678905-21ssh22:46:001
356532299422,35cyclictest3695063-21ssh23:05:591
356532199424,30cyclictest3650721-21ssh22:10:580
3565323994133,5cyclictest3717878-21ssh23:35:012
3565323994132,6cyclictest1744621-21cups-browsed22:06:042
3565322994130,8cyclictest3750187-21ssh00:15:011
3565321994134,4cyclictest421-21dbus-daemon22:35:010
3565323994032,5cyclictest3592347-21latency_hist20:40:022
3565323994032,4cyclictest3578816-21kworker/u8:5+xprtiod19:55:012
3565323994031,5cyclictest3713903-21latency_hist23:30:022
3565323994029,7cyclictest1744621-21cups-browsed22:45:242
3565323994026,3cyclictest171rcu_preempt00:00:002
3565322994034,3cyclictest0-21swapper/100:05:011
3565322994032,5cyclictest3725972-21ssh23:45:021
3565322994032,5cyclictest3598370-21systemd-run21:00:011
3565322994031,6cyclictest1-21systemd21:35:011
3565322994027,7cyclictest3690963-21sh23:00:581
3565321994032,5cyclictest3682163-21ssh22:50:450
3565321994023,11cyclictest3686955-21ssh22:55:590
3565323993932,4cyclictest1054-21mta-sts-daemon00:36:052
3565323993928,8cyclictest3633221-21sh21:50:012
3565323993928,8cyclictest3633221-21sh21:50:012
3565323993928,7cyclictest1056-21mta-sts-daemon21:57:062
3565322993931,5cyclictest1051-21mta-sts-daemon23:56:191
3565322993930,6cyclictest3617076-21munin-plugin-st21:30:011
3565321993933,3cyclictest0-21swapper/000:40:010
3565321993930,4cyclictest0-21swapper/023:25:580
3565321993927,9cyclictest3766214-21ssh00:35:010
3565323993831,3cyclictest0-21swapper/221:50:502
3565323993830,4cyclictest0-21swapper/200:22:072
3565323993829,5cyclictest3627142-21unixbench_multi21:41:132
3565323993829,5cyclictest3627142-21unixbench_multi21:41:132
3565322993830,5cyclictest1051-21mta-sts-daemon00:20:021
3565322993830,5cyclictest1051-21mta-sts-daemon00:20:021
3565322993830,4cyclictest3762044-21kworker/u8:8-xprtiod00:35:011
3565322993828,6cyclictest0-21swapper/123:20:451
3565323993730,3cyclictest0-21swapper/200:30:452
3565323993729,5cyclictest3625146-21idleruntime-cro21:40:012
3565323993724,10cyclictest3651041-21switchtime22:11:062
356532399372,26cyclictest3670858-21ssh22:36:002
356532399372,26cyclictest3670858-21ssh22:36:002
3565322993727,7cyclictest3682123-21aten_r2power_cu22:50:441
3565322993726,8cyclictest3687260-21switchtime22:56:061
3565321993729,5cyclictest3609073-21munin-run21:20:010
3565321993728,6cyclictest420-21avahi-daemon23:18:410
3565321993723,11cyclictest3619002-21unixbench-2d21:31:120
356532399363,26cyclictest3759589-21ssh00:25:592
3565323993629,4cyclictest1056-21mta-sts-daemon23:01:142
3565323993628,5cyclictest3595319-21systemd-run20:50:012
3565323993628,5cyclictest3565320-21cyclictest00:06:212
3565323993627,6cyclictest3575762-21grep19:41:072
3565323993625,5cyclictest0-21swapper/221:26:592
356532399362,26cyclictest3723388-21ssh23:41:022
3565322993629,4cyclictest3657382-21ssh22:20:011
3565322993628,5cyclictest3630520-21ntp_kernel_pll_21:45:571
3565322993628,5cyclictest3630520-21ntp_kernel_pll_21:45:571
3565322993628,5cyclictest3588585-21grep20:25:541
3565322993625,8cyclictest3717859-21munin-plugin-st23:35:021
356532199363,28cyclictest3739419-21ssh00:00:540
3565321993631,3cyclictest3657392-21idleruntime-cro22:20:010
3565321993629,5cyclictest1056-21mta-sts-daemon23:07:360
3565321993629,3cyclictest0-21swapper/000:20:010
3565321993629,3cyclictest0-21swapper/000:20:010
356532399357,9cyclictest0-21swapper/219:20:542
3565323993528,4cyclictest3578816-21kworker/u8:5-rpciod20:05:012
3565323993528,4cyclictest0-21swapper/219:55:552
3565323993523,9cyclictest3610170-21kernelversion21:20:522
3565323993523,8cyclictest3694214-21/usr/sbin/munin23:05:482
3565322993527,5cyclictest3734047-21ssh23:55:011
3565322993527,5cyclictest3734047-21ssh23:55:011
3565322993527,5cyclictest3625147-21munin-run21:40:011
3565322993527,3cyclictest0-21swapper/122:01:061
3565322993526,6cyclictest421-21dbus-daemon20:10:011
3565322993525,7cyclictest3565343-21latency_hist19:10:021
3565322993525,6cyclictest3601343-21cron21:10:011
3565321993528,4cyclictest3605106-21systemd-run21:15:010
3565321993526,6cyclictest1744621-21cups-browsed23:41:390
3565321993524,8cyclictest3670401-21if_eth022:35:500
3565321993524,8cyclictest3670401-21if_eth022:35:500
3565321993524,8cyclictest0-21swapper/022:40:480
3565321993521,12cyclictest3701706-21idleruntime-cro23:15:010
3565321993521,12cyclictest3701706-21idleruntime-cro23:15:010
3565323993426,3cyclictest0-21swapper/221:15:582
3565323993425,5cyclictest3705736-21grep23:20:002
3565323993424,6cyclictest421-21dbus-daemon20:25:012
3565322993427,4cyclictest3673406-21munin-run22:40:001
3565322993427,4cyclictest3673406-21munin-run22:40:001
3565322993425,5cyclictest0-21swapper/123:46:001
356532299342,24cyclictest3759614-21ssh00:26:001
356532199345,23cyclictest3622482-21ssh21:36:000
3565321993425,7cyclictest3568309-21/usr/sbin/munin19:20:020
3565321993425,6cyclictest3738109-21cups-browsed00:06:080
3565321993425,6cyclictest3738081-21cupsd00:10:330
3565321993425,6cyclictest1744618-21cupsd23:47:540
3565323993327,3cyclictest0-21swapper/219:20:012
3565323993325,3cyclictest0-21swapper/222:31:042
3565323993314,4cyclictest171rcu_preempt19:46:012
3565323993314,4cyclictest171rcu_preempt19:46:012
3565322993325,5cyclictest3719799-21unixbench_multi23:36:131
3565322993324,4cyclictest0-21swapper/122:40:541
3565322993319,10cyclictest271ktimers/120:45:571
356532299331,26cyclictest548-21gdbus21:20:561
3565321993327,3cyclictest0-21swapper/021:25:560
3565321993326,4cyclictest3635106-21unixbench-2d21:51:070
3565321993326,4cyclictest0-21swapper/021:25:000
3565321993325,5cyclictest3638870-21systemctl21:56:070
3565323993226,3cyclictest0-21swapper/222:03:442
3565323993226,3cyclictest0-21swapper/220:20:012
3565323993225,5cyclictest0-21swapper/223:51:012
3565323993225,5cyclictest0-21swapper/223:51:012
3565323993224,5cyclictest3742184-21systemd-run00:05:012
3565323993223,5cyclictest3601255-21unixbench_multi21:06:072
356532399321,5cyclictest0-21swapper/222:42:032
3565322993224,4cyclictest0-21swapper/122:30:591
3565321993223,6cyclictest3646367-21kernelversion22:05:520
3565321993222,7cyclictest3715556-21switchtime23:31:070
3565323993124,4cyclictest1054-21mta-sts-daemon20:10:592
3565322993125,3cyclictest0-21swapper/123:26:071
3565322993125,3cyclictest0-21swapper/120:20:011
3565322993123,5cyclictest3754937-21aten_r2power_fr00:20:451
3565322993122,7cyclictest3596109-21needreboot20:50:501
3565322993121,7cyclictest3743961-21timerandwakeup00:06:131
3565322993120,8cyclictest3650027-21aten_r2power_fr22:10:451
3565321993125,4cyclictest3738097-21munin-run00:00:000
3565321993123,5cyclictest3576038-21/usr/sbin/munin19:45:480
3565321993123,5cyclictest3576038-21/usr/sbin/munin19:45:480
3565323993023,4cyclictest1056-21mta-sts-daemon19:25:592
3565323993023,3cyclictest0-21swapper/223:25:012
3565323993022,3cyclictest0-21swapper/223:45:462
3565323993020,5cyclictest0-21swapper/223:35:582
3565323993020,3cyclictest0-21swapper/222:55:562
3565322993024,3cyclictest0-21swapper/123:15:011
3565322993024,3cyclictest0-21swapper/123:15:011
3565322993023,4cyclictest3602659-21ntp_kernel_pll_21:10:561
3565322993023,4cyclictest1056-21mta-sts-daemon23:20:021
3565322993020,8cyclictest3567705-21ntp_kernel_pll_19:15:561
3565322993020,7cyclictest3663157-21xargs22:26:131
3565322993020,7cyclictest3663157-21xargs22:26:131
356532199303,9cyclictest428-21mta-sts-daemon23:51:420
356532199303,9cyclictest428-21mta-sts-daemon23:51:420
3565321993025,2cyclictest0-21swapper/020:25:010
3565321993023,4cyclictest3590706-21unixbench_multi20:31:070
3565321993021,6cyclictest3568848-21fschecks_count19:20:490
3565321993020,7cyclictest3598377-21systemd-run21:00:010
3565321993018,9cyclictest0-21swapper/000:20:490
356532399290,4cyclictest171rcu_preempt20:06:042
3565322992923,3cyclictest0-21swapper/100:40:011
3565322992922,3cyclictest0-21swapper/121:51:011
3565322992913,11cyclictest3658281-21df22:20:481
3565322992913,11cyclictest3658281-21df22:20:481
3565321992923,4cyclictest3575764-21unixbench_singl19:41:070
3565321992923,3cyclictest0-21swapper/019:55:010
3565321992921,5cyclictest3690947-21sh23:00:580
3565321992921,4cyclictest151ktimers/000:25:580
3565323992821,4cyclictest3574170-21unixbench_multi19:36:072
3565323992821,3cyclictest0-21swapper/220:41:042
3565323992820,4cyclictest0-21swapper/220:26:022
356532199282,8cyclictest3707143-21ntpq23:20:570
3565321992822,3cyclictest0-21swapper/022:26:040
3565321992822,3cyclictest0-21swapper/022:26:040
3565321992820,5cyclictest1-21systemd21:10:010
3565321992816,9cyclictest3569837-21kworker/u8:10-xprtiod19:40:000
356532399275,5cyclictest171rcu_preempt20:50:532
3565323992722,2cyclictest0-21swapper/219:35:002
3565322992719,5cyclictest439-21systemd-logind20:45:011
3565322992719,5cyclictest3599389-21processes21:01:031
3565322992718,4cyclictest0-21swapper/119:30:561
3565322992710,7cyclictest3581652-21unixbench-2d20:01:081
356532199273,11cyclictest3601351-21diskmemload22:02:580
3565321992721,4cyclictest3586339-21systemd-run20:20:010
3565321992721,3cyclictest3588338-21cut20:25:510
3565323992621,2cyclictest0-21swapper/221:05:012
3565323992613,8cyclictest341rcuc/219:10:512
3565322992622,2cyclictest0-21swapper/120:15:011
3565322992621,2cyclictest0-21swapper/119:40:011
3565322992620,3cyclictest0-21swapper/120:36:001
3565322992618,4cyclictest0-21swapper/119:25:551
3565322992617,6cyclictest421-21dbus-daemon20:35:011
356532199262,18cyclictest3678877-21ssh22:45:590
3565321992621,2cyclictest0-21swapper/020:35:560
3565321992618,5cyclictest3658436-21iwlist22:20:520
3565321992618,5cyclictest3658436-21iwlist22:20:520
3565322992517,4cyclictest0-21swapper/119:41:001
3565322992516,7cyclictest3576038-21/usr/sbin/munin19:45:471
3565322992516,7cyclictest3576038-21/usr/sbin/munin19:45:471
3565322992516,7cyclictest3569849-21latency_hist19:25:021
3565322992516,7cyclictest1056-21mta-sts-daemon19:55:521
356532299251,19cyclictest659-21snmpd19:10:501
3565321992520,3cyclictest3596606-21timerwakeupswit20:51:040
3565321992518,5cyclictest1744621-21cups-browsed20:10:130
3565321992517,5cyclictest0-21swapper/019:30:010
3565321992515,7cyclictest12350irq/32-eth019:55:550
3565321992515,6cyclictest12350irq/32-eth020:45:590
3565322992316,4cyclictest3586337-21kworker/u8:14+xprtiod20:25:011
3565322992315,5cyclictest547-21gmain19:50:471
3565321992317,3cyclictest1056-21mta-sts-daemon21:00:510
3565321992217,3cyclictest1051-21mta-sts-daemon20:05:020
3565321992216,3cyclictest3593695-21sed20:41:130
3565321992215,5cyclictest1056-21mta-sts-daemon20:09:440
3565321992115,4cyclictest2884192-21kworker/u8:10+xprtiod19:10:570
3565321992114,4cyclictest3565421-21latency_hist19:10:020
356532499183,9cyclictest0-21swapper/322:15:463
356532499182,9cyclictest0-21swapper/322:11:033
356532499182,8cyclictest0-21swapper/322:20:573
356532499182,8cyclictest0-21swapper/322:20:573
356532499173,8cyclictest0-21swapper/300:20:563
356532499172,8cyclictest0-21swapper/321:55:573
3565323991711,3cyclictest3565277-21munin-run19:10:012
356532499163,8cyclictest0-21swapper/322:55:553
356532499162,9cyclictest0-21swapper/323:11:023
356532499162,9cyclictest0-21swapper/323:11:023
356532499162,9cyclictest0-21swapper/321:41:063
356532499162,9cyclictest0-21swapper/321:41:063
356532499162,8cyclictest0-21swapper/321:30:593
356532499153,8cyclictest0-21swapper/323:51:003
356532499153,8cyclictest0-21swapper/323:51:003
356532499153,5cyclictest0-21swapper/320:55:563
356532499152,9cyclictest0-21swapper/321:11:003
356532499152,8cyclictest0-21swapper/323:45:593
356532499152,8cyclictest0-21swapper/323:35:583
356532499152,8cyclictest0-21swapper/322:05:593
356532499152,7cyclictest0-21swapper/323:55:543
356532499152,7cyclictest0-21swapper/321:15:583
356532499152,6cyclictest0-21swapper/320:50:513
356532499152,5cyclictest0-21swapper/319:45:593
356532499152,5cyclictest0-21swapper/319:45:593
356532499151,9cyclictest0-21swapper/322:26:093
356532499151,9cyclictest0-21swapper/322:26:093
356532499151,8cyclictest0-21swapper/323:25:583
356532499151,8cyclictest0-21swapper/323:20:543
356532499151,8cyclictest0-21swapper/321:45:563
356532499151,8cyclictest0-21swapper/321:45:563
356532499151,7cyclictest0-21swapper/322:50:553
356532499151,7cyclictest0-21swapper/320:35:543
356532499151,6cyclictest0-21swapper/321:05:583
356532499147,4cyclictest0-21swapper/323:15:533
356532499144,6cyclictest0-21swapper/321:20:533
356532499144,5cyclictest0-21swapper/322:36:023
356532499144,5cyclictest0-21swapper/322:36:023
356532499143,7cyclictest0-21swapper/300:16:013
356532499143,7cyclictest0-21swapper/300:16:013
356532499143,6cyclictest0-21swapper/322:31:013
356532499142,8cyclictest0-21swapper/323:00:453
356532499142,8cyclictest0-21swapper/320:45:023
356532499142,7cyclictest0-21swapper/319:25:553
356532499142,6cyclictest0-21swapper/319:10:503
356532499142,6cyclictest0-21swapper/300:15:033
356532499142,6cyclictest0-21swapper/300:00:493
356532499141,8cyclictest0-21swapper/323:41:013
356532499141,8cyclictest0-21swapper/322:45:583
356532499141,8cyclictest0-21swapper/321:25:593
356532499141,8cyclictest0-21swapper/300:35:543
356532499141,7cyclictest0-21swapper/319:50:513
356532499141,6cyclictest0-21swapper/320:20:543
356532499141,6cyclictest0-21swapper/320:10:573
356532499141,6cyclictest0-21swapper/319:15:533
356532499134,5cyclictest0-21swapper/322:00:593
356532499133,6cyclictest0-21swapper/319:30:563
356532499133,6cyclictest0-21swapper/300:30:523
356532499132,7cyclictest0-21swapper/321:50:553
356532499132,7cyclictest0-21swapper/300:05:593
356532499132,6cyclictest0-21swapper/319:35:593
356532499131,8cyclictest0-21swapper/321:35:553
356532499131,8cyclictest0-21swapper/300:25:553
356532499131,6cyclictest0-21swapper/320:26:023
356532499131,6cyclictest0-21swapper/319:40:543
356532499131,6cyclictest0-21swapper/319:21:023
356532499126,4cyclictest0-21swapper/320:45:523
356532499122,6cyclictest0-21swapper/320:30:553
356532499122,5cyclictest0-21swapper/323:05:453
356532499120,6cyclictest0-21swapper/321:00:573
356532499113,6cyclictest0-21swapper/322:40:543
356532499113,4cyclictest0-21swapper/320:06:033
356532499111,6cyclictest0-21swapper/323:30:563
356532499111,6cyclictest0-21swapper/319:56:013
356532499102,6cyclictest0-21swapper/319:10:013
356532499101,5cyclictest0-21swapper/320:00:463
35653249991,4cyclictest0-21swapper/320:16:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional