You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-30 - 02:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sat May 30, 2026 00:42:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4135768995447,3cyclictest0-21swapper/223:16:002
4135768995344,4cyclictest0-21swapper/223:01:012
4135768995142,5cyclictest4190994-21sh21:40:012
4135768995141,6cyclictest421-21dbus-daemon21:50:012
413576899511,8cyclictest171rcu_preempt22:50:582
413576799515,15cyclictest4145780-21ps19:41:051
4135768994942,4cyclictest86545-21ssh00:00:012
413576799493,34cyclictest36693-21ssh22:41:031
4135767994839,6cyclictest1-21systemd22:25:021
4135767994738,3cyclictest0-21swapper/100:25:011
413576799463,7cyclictest27169-21ntpq22:26:081
4135767994623,20cyclictest0-21swapper/120:00:011
4135768994538,4cyclictest1055-21mta-sts-daemon20:40:592
4135767994536,6cyclictest4152639-21basename20:05:531
413576799443,7cyclictest4147213-21ntpq19:46:031
4135767994435,6cyclictest4187938-21latency_hist21:35:021
413576799441,12cyclictest90962-21ntpq00:06:001
413576799432,7cyclictest4154667-21ntpq20:11:011
413576799432,7cyclictest4154667-21ntpq20:11:011
413576799431,12cyclictest4151627-21ntpq20:00:581
413576799431,11cyclictest4157638-21ntpq20:20:591
4135766994336,4cyclictest4152275-21kworker/u8:3+xprtiod20:05:010
4135766994336,4cyclictest35362-21munin-plugin-st22:40:010
413576699432,38cyclictest151ktimers/021:26:050
413576699432,37cyclictest26993-21ssh22:26:010
4135767994233,6cyclictest2555081-21cups-browsed22:06:031
4135767994220,19cyclictest0-21swapper/123:21:021
4135767994218,21cyclictest2555081-21cups-browsed20:52:501
4135767994212,26cyclictest0-21swapper/121:40:591
413576799421,12cyclictest77735-21diskstats23:45:521
413576799421,11cyclictest4138217-21ntp_states19:16:031
413576799421,11cyclictest4138217-21ntp_states19:16:031
413576799421,10cyclictest4148657-21ntpq19:51:051
4135766994235,3cyclictest0-21swapper/021:55:010
4135766994234,6cyclictest4167262-21systemd-run20:55:010
413576699421,38cyclictest151ktimers/020:30:590
4135768994127,9cyclictest0-21swapper/223:40:592
413576799413,15cyclictest4624-21ssh21:50:581
413576799412,22cyclictest65350-21ssh23:25:591
413576799412,16cyclictest4182659-21ntpq21:26:031
4135767994119,19cyclictest4190938-21kworker/u8:1-xprtiod22:01:171
4135766994136,3cyclictest3446-21grep21:50:010
413576699411,37cyclictest12350irq/32-eth023:26:010
413576699411,31cyclictest103884-21ssh00:26:060
4135768994032,3cyclictest0-21swapper/200:11:062
413576799403,7cyclictest4165233-21ntpq20:46:061
413576799403,16cyclictest548-21gdbus23:40:571
4135767994020,17cyclictest0-21swapper/121:16:011
4135767994019,18cyclictest0-21swapper/100:36:171
4135767994014,2cyclictest0-21swapper/121:46:031
413576799401,12cyclictest89550-21ssh00:04:221
413576799401,12cyclictest4168257-21ps20:56:091
413576799401,10cyclictest94221-21ntpq00:11:041
413576799401,10cyclictest52683-21ssh23:06:051
4135766994013,18cyclictest151ktimers/023:56:020
4135768993933,3cyclictest0-21swapper/223:36:122
4135768993930,6cyclictest56376-21sed23:11:122
4135768993930,6cyclictest2555081-21cups-browsed22:55:592
413576799394,8cyclictest59046-21ssh23:16:021
413576799394,29cyclictest84552-21ssh23:56:011
413576799394,10cyclictest7945-21ntpq21:56:031
413576799393,8cyclictest49376-21ntpq23:00:591
4135767993914,3cyclictest0-21swapper/119:30:571
4135767993912,6cyclictest1056-21mta-sts-daemon22:31:161
4135767993911,5cyclictest4162302-21sendmail_mailqu20:36:061
413576799391,14cyclictest4169636-21fake-hwclock21:01:011
4135766993933,4cyclictest57763-21idleruntime-cro23:15:010
4135768993830,5cyclictest4135777-21systemd-run19:10:012
4135768993830,5cyclictest4135777-21systemd-run19:10:012
4135768993830,5cyclictest20055-21aten_r2power_po22:15:502
4135768993830,4cyclictest0-21swapper/220:01:002
4135768993828,5cyclictest0-21swapper/200:06:022
4135767993830,4cyclictest4161400-21ls20:35:031
413576799382,12cyclictest41778-21/usr/sbin/munin22:50:021
413576799381,9cyclictest55851-21grep23:11:041
4135767993815,20cyclictest103829-21sh00:26:051
413576799381,11cyclictest4139641-21ntpq19:20:581
4135766993829,6cyclictest73726-21ssh23:40:010
4135766993828,7cyclictest94820-21unixbench_singl00:11:150
4135768993729,5cyclictest4171820-21latency_hist21:10:022
4135768993728,5cyclictest0-21swapper/200:40:012
4135768993727,6cyclictest439-21systemd-logind20:40:022
413576799373,6cyclictest4171095-21ntpq21:05:591
413576799372,13cyclictest33516-21ssh22:36:071
4135767993714,20cyclictest0-21swapper/122:15:591
413576799371,12cyclictest4179425-21ntpq21:21:021
413576799371,11cyclictest4172959-21ntpq21:11:001
413576799371,10cyclictest97459-21ntpq00:16:031
413576699376,9cyclictest14-21ksoftirqd/022:50:580
413576699372,27cyclictest20804-21ssh22:16:060
4135766993721,10cyclictest12350irq/32-eth021:36:070
4135768993631,2cyclictest0-21swapper/221:25:012
4135768993629,4cyclictest1055-21mta-sts-daemon22:11:022
4135768993629,3cyclictest0-21swapper/223:20:592
4135768993627,6cyclictest86535-21cups-browsed00:20:532
4135768993625,8cyclictest4157465-21latency20:20:552
413576899362,28cyclictest33554-21ssh22:36:092
4135768993622,3cyclictest0-21swapper/221:30:012
413576799366,8cyclictest421-21dbus-daemon23:00:011
413576799364,8cyclictest4188976-21ssh21:36:001
413576799363,10cyclictest107012-21ssh00:31:021
4135767993616,17cyclictest0-21swapper/119:15:001
4135767993611,2cyclictest0-21swapper/122:10:551
413576799361,11cyclictest42950-21ntpq22:50:581
4135766993627,6cyclictest32155-21ssh22:35:010
413576699362,29cyclictest97324-21ssh00:15:570
413576899354,24cyclictest4173101-21ssh21:11:062
413576899353,26cyclictest14229-21ssh22:05:592
413576799359,3cyclictest0-21swapper/123:36:011
4135766993527,5cyclictest421-21dbus-daemon22:55:580
4135766993525,8cyclictest4170269-21systemd-run21:05:010
4135766993518,10cyclictest151ktimers/021:16:050
413576899344,6cyclictest171rcu_preempt22:26:072
4135768993428,3cyclictest0-21swapper/220:00:012
4135768993428,3cyclictest0-21swapper/200:30:572
4135768993427,4cyclictest4138268-21proc_pri19:16:042
4135768993427,4cyclictest4138268-21proc_pri19:16:042
4135768993427,4cyclictest0-21swapper/221:20:012
4135768993426,5cyclictest89774-21ssh00:05:012
4135768993426,5cyclictest86535-21cups-browsed00:26:462
4135768993422,8cyclictest4185757-21ssh21:30:562
413576799349,3cyclictest0-21swapper/120:45:011
4135767993427,4cyclictest4158910-21kernelversion20:25:551
413576799341,9cyclictest81342-21ntpq23:50:591
413576799341,11cyclictest4156199-21ntpq20:16:041
413576799341,11cyclictest4156199-21ntpq20:16:041
413576799341,11cyclictest4144157-21ntpq19:36:011
4135766993428,4cyclictest36195-21sort22:40:530
4135766993427,4cyclictest76938-21cron23:45:010
4135766993422,10cyclictest4159643-21tail20:26:110
413576699341,31cyclictest0-21swapper/023:15:560
4135768993326,4cyclictest81119-21latency23:50:552
4135768993326,3cyclictest0-21swapper/221:41:042
4135768993325,5cyclictest9837-21munin-run22:00:012
413576899332,24cyclictest97477-21sh00:16:032
4135768993321,5cyclictest1-21systemd22:32:542
413576799339,4cyclictest4141117-21ntpq19:25:571
4135766993326,4cyclictest61514-21aten_r2power_en23:20:480
4135766993326,4cyclictest151ktimers/019:21:040
4135766993325,5cyclictest52208-21fschecks_time23:05:550
4135766993318,12cyclictest41793-21systemd-run22:50:010
4135768993224,5cyclictest421-21dbus-daemon19:35:012
4135768993224,5cyclictest4147745-21cron19:50:012
4135768993223,5cyclictest1-21systemd20:50:022
4135768993222,5cyclictest0-21swapper/222:46:022
413576799326,7cyclictest68529-21ntpq23:30:581
413576699323,22cyclictest4192311-21ssh21:41:030
413576699321,27cyclictest110175-21ssh00:36:060
4135768993124,4cyclictest4170256-21cut21:05:012
4135768993123,5cyclictest67934-21ssh23:30:492
4135768993123,4cyclictest77765-21fschecks_time23:45:522
413576899312,24cyclictest23928-21ssh22:21:032
413576899312,22cyclictest65436-21ssh23:26:032
413576799311,5cyclictest4135723-21/usr/sbin/munin19:10:011
413576799311,5cyclictest4135723-21/usr/sbin/munin19:10:011
4135766993125,3cyclictest0-21swapper/020:20:010
4135766993125,3cyclictest0-21swapper/020:20:010
4135766993112,15cyclictest100567-21ssh00:21:000
4135766993112,13cyclictest421-21dbus-daemon20:46:030
413576899303,21cyclictest4720-21ssh21:51:022
4135768993022,5cyclictest1051-21mta-sts-daemon23:06:142
413576699302,23cyclictest78115-21ntpq23:45:580
4135766993018,4cyclictest4149772-21fschecks_time19:55:540
413576699301,24cyclictest4179001-21sort21:20:530
413576899293,18cyclictest428-21mta-sts-daemon19:10:592
4135768992921,5cyclictest4159633-21unixbench_multi20:26:112
4135768992921,5cyclictest1-21systemd19:25:012
4135768992921,4cyclictest0-21swapper/219:40:022
413576699293,20cyclictest49423-21cron23:01:010
4135766992922,3cyclictest0-21swapper/022:10:500
4135766992919,7cyclictest0-21swapper/022:01:050
4135766992918,8cyclictest12350irq/32-eth021:56:020
413576699291,21cyclictest4185872-21ssh21:31:010
4135766992912,14cyclictest548-21gdbus20:41:060
4135768992822,4cyclictest4135765-21cyclictest21:00:012
4135768992819,5cyclictest4156783-21systemd-run20:20:012
4135768992819,5cyclictest4156783-21systemd-run20:20:012
413576699283,19cyclictest81446-21ssh23:51:030
4135766992822,4cyclictest4143181-21head19:31:110
4135766992822,3cyclictest0-21swapper/000:30:540
4135766992821,4cyclictest87388-21/usr/sbin/munin00:00:540
4135766992819,4cyclictest1054-21mta-sts-daemon21:13:170
4135766992814,11cyclictest0-21swapper/019:40:010
4135768992721,3cyclictest0-21swapper/222:41:112
4135768992717,6cyclictest4155178-21unixbench_singl20:11:112
4135768992717,6cyclictest4155178-21unixbench_singl20:11:112
4135768992717,4cyclictest351ktimers/219:51:032
4135768992716,8cyclictest4145661-21ntp_kernel_pll_19:41:002
413576699274,15cyclictest12350irq/32-eth019:11:010
413576699273,17cyclictest12350irq/32-eth022:06:000
4135766992720,5cyclictest23484-21sshd22:20:540
4135766992718,5cyclictest151ktimers/019:16:030
4135766992718,5cyclictest151ktimers/019:16:030
4135768992620,3cyclictest3679-21kworker/u8:0-xprtiod22:05:012
4135768992620,3cyclictest0-21swapper/220:06:042
4135768992619,4cyclictest1054-21mta-sts-daemon19:26:052
4135768992618,5cyclictest4161264-21systemd-run20:35:012
4135766992621,3cyclictest151ktimers/019:25:590
4135766992619,4cyclictest4147765-21systemd-run19:50:010
4135766992618,5cyclictest1-21systemd00:10:020
4135766992610,12cyclictest548-21gdbus21:06:020
413576899254,15cyclictest428-21mta-sts-daemon20:50:592
413576699251,19cyclictest68517-21sh23:30:570
4135766992511,11cyclictest0-21swapper/020:56:130
4135766992417,5cyclictest4149330-21kworker/u8:2+nfsiod20:11:010
4135766992417,5cyclictest4149330-21kworker/u8:2+nfsiod20:11:010
4135766992417,4cyclictest4135777-21systemd-run19:10:010
4135766992417,4cyclictest4135777-21systemd-run19:10:010
4135766992416,5cyclictest548-21gdbus20:06:030
413576699232,16cyclictest4145716-21ntp_states19:41:030
413576699230,16cyclictest151ktimers/020:35:580
4135766992213,6cyclictest0-21swapper/020:21:110
4135766992212,3cyclictest4135765-21cyclictest19:52:180
413576999211,9cyclictest0-21swapper/323:56:023
413576999183,9cyclictest0-21swapper/321:30:573
413576999171,7cyclictest0-21swapper/320:20:593
413576999162,7cyclictest0-21swapper/322:01:073
413576999161,7cyclictest0-21swapper/322:26:073
413576999153,5cyclictest0-21swapper/300:36:103
413576999152,8cyclictest0-21swapper/322:05:543
413576999152,8cyclictest0-21swapper/300:26:073
413576999152,7cyclictest0-21swapper/322:36:043
413576999152,7cyclictest0-21swapper/321:36:053
413576999152,6cyclictest0-21swapper/322:46:003
413576999152,5cyclictest0-21swapper/322:10:583
413576999151,7cyclictest0-21swapper/320:51:013
413576999151,7cyclictest0-21swapper/319:41:013
413576999151,6cyclictest0-21swapper/320:16:023
413576999151,6cyclictest0-21swapper/320:16:023
413576999144,4cyclictest0-21swapper/320:56:053
413576999143,5cyclictest0-21swapper/321:55:013
413576999142,6cyclictest0-21swapper/321:11:003
413576999142,6cyclictest0-21swapper/300:06:023
413576999142,5cyclictest0-21swapper/319:45:583
413576999141,8cyclictest0-21swapper/321:41:003
413576999141,7cyclictest0-21swapper/323:36:053
413576999141,7cyclictest0-21swapper/322:56:013
413576999141,7cyclictest0-21swapper/300:01:053
413576999141,6cyclictest0-21swapper/323:46:003
413576999141,6cyclictest0-21swapper/320:06:033
413576999141,6cyclictest0-21swapper/319:26:023
413576999141,3cyclictest0-21swapper/300:20:553
413576999133,6cyclictest0-21swapper/322:50:583
413576999133,4cyclictest0-21swapper/319:11:003
413576999132,7cyclictest0-21swapper/323:20:573
413576999132,6cyclictest0-21swapper/300:15:583
413576999132,5cyclictest0-21swapper/321:20:493
413576999132,5cyclictest0-21swapper/319:56:003
413576999131,8cyclictest0-21swapper/321:06:033
413576999131,6cyclictest0-21swapper/323:16:003
413576999131,6cyclictest0-21swapper/323:06:013
413576999131,6cyclictest0-21swapper/320:10:583
413576999131,6cyclictest0-21swapper/320:10:583
413576999131,5cyclictest0-21swapper/320:00:583
413576999130,7cyclictest0-21swapper/322:20:593
413576999122,6cyclictest0-21swapper/323:40:553
413576999122,6cyclictest0-21swapper/321:56:013
413576999122,6cyclictest0-21swapper/300:11:033
413576999121,7cyclictest0-21swapper/323:01:013
413576999121,7cyclictest0-21swapper/300:31:013
413576999121,6cyclictest0-21swapper/323:50:583
413576999121,6cyclictest0-21swapper/322:31:053
413576999121,6cyclictest0-21swapper/322:16:033
413576999121,6cyclictest0-21swapper/321:25:573
413576999121,6cyclictest0-21swapper/320:31:023
413576999121,6cyclictest0-21swapper/319:31:043
413576999121,5cyclictest0-21swapper/321:46:003
413576999121,3cyclictest0-21swapper/319:35:583
413576999113,6cyclictest0-21swapper/323:10:533
413576999112,6cyclictest0-21swapper/323:30:583
413576999112,5cyclictest0-21swapper/319:51:023
413576999111,6cyclictest0-21swapper/323:26:003
413576999111,6cyclictest0-21swapper/322:40:583
413576999111,6cyclictest0-21swapper/321:01:013
413576999111,6cyclictest0-21swapper/320:35:593
413576999111,6cyclictest0-21swapper/319:16:033
413576999111,6cyclictest0-21swapper/319:16:033
413576999102,4cyclictest0-21swapper/321:15:493
413576999101,6cyclictest0-21swapper/320:45:563
413576999101,6cyclictest0-21swapper/319:20:593
413576999101,5cyclictest0-21swapper/320:25:533
41357699992,5cyclictest0-21swapper/320:40:533
41357699991,4cyclictest0-21swapper/319:10:023
41357699991,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional