You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-29 - 10:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sat Aug 29, 2026 00:42:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
202423799153147,4cyclictest351ktimers/200:05:172
202423699118110,3cyclictest2142791-21kworker/u8:4-rpciod00:05:171
2024235999992,5cyclictest2161165-21kworker/u8:12-rpciod00:05:170
2024236996248,9cyclictest177899-21cups-browsed21:51:081
2024236996248,9cyclictest177899-21cups-browsed21:51:081
2024235995432,15cyclictest151ktimers/020:06:020
202423799452,33cyclictest2128709-21ssh23:01:052
202423799451,40cyclictest2141029-21ssh23:21:082
2024237994436,4cyclictest0-21swapper/200:00:012
2024237994436,4cyclictest0-21swapper/200:00:012
2024237994434,6cyclictest439-21systemd-logind20:25:012
2024237994434,6cyclictest439-21systemd-logind20:25:012
2024237994434,6cyclictest2164266-21cups-browsed00:12:492
202423799443,32cyclictest665-21snmpd22:26:052
2024236994336,4cyclictest0-21swapper/100:11:021
2024236994335,5cyclictest2154998-21systemd23:45:021
202423599435,35cyclictest2159335-21ssh23:51:080
2024235994333,6cyclictest2085286-21/usr/sbin/munin21:50:550
2024235994333,6cyclictest2085286-21/usr/sbin/munin21:50:550
2024237994233,3cyclictest0-21swapper/222:11:082
2024237994226,10cyclictest0-21swapper/219:41:012
2024235994226,12cyclictest12650irq/32-eth023:01:060
2024237994133,5cyclictest2071013-21switchtime21:26:152
2024237994131,5cyclictest2064780-21kworker/u8:4-rpciod21:16:122
2024237994131,5cyclictest0-21swapper/223:00:012
2024237994131,5cyclictest0-21swapper/223:00:012
2024237994030,7cyclictest2119424-21missed_timers22:46:012
202423799402,35cyclictest2137856-21ssh23:16:042
202423699403,33cyclictest2095240-21ssh22:06:081
202423699402,28cyclictest2064540-21ssh21:16:041
2024235994033,4cyclictest2115384-21latency_hist22:40:020
202423599403,32cyclictest2119597-21sh22:46:060
2024235994032,6cyclictest2024256-21latency_hist19:10:010
202423599400,36cyclictest2079907-21ssh21:41:030
202423799393,27cyclictest2186943-21ssh00:36:092
2024237993931,4cyclictest2059240-21fschecks_count21:05:552
202423799391,10cyclictest2080021-21ssh21:41:092
202423799383,25cyclictest2067626-21ssh21:21:062
2024237993830,5cyclictest2124452-21sh22:55:012
2024237993830,5cyclictest2049736-21systemd-run20:35:022
2024237993830,5cyclictest1001-21mta-sts-daemon19:51:082
2024237993813,6cyclictest0-21swapper/219:31:032
2024236993831,4cyclictest1001-21mta-sts-daemon21:46:041
2024236993831,4cyclictest1001-21mta-sts-daemon21:46:041
2024236993830,5cyclictest2033210-21sh19:40:011
2024236993829,6cyclictest177899-21cups-browsed20:45:011
202423599384,5cyclictest171rcu_preempt00:11:120
2024235993831,4cyclictest2180543-21missed_timers00:26:010
2024235993822,11cyclictest2147096-21rm23:31:050
2024237993731,3cyclictest0-21swapper/222:41:032
2024237993731,3cyclictest0-21swapper/220:26:122
2024237993729,5cyclictest2106141-21systemd-run22:25:012
2024237993728,6cyclictest2024755-21fschecks_time19:10:552
2024237993728,6cyclictest2024755-21fschecks_time19:10:552
2024237993728,5cyclictest2042226-21cut20:10:012
2024237993728,5cyclictest2033255-21systemd-run19:40:022
2024236993729,5cyclictest2103874-21fschecks_count22:20:561
2024236993729,5cyclictest2057155-21unixbench_singl20:56:171
2024236993727,7cyclictest2148853-21ssh23:35:011
2024236993726,8cyclictest177899-21cups-browsed22:15:551
2024235993729,3cyclictest2164120-21munin-run00:00:010
2024235993729,3cyclictest2164120-21munin-run00:00:010
2024235993724,6cyclictest151ktimers/022:31:030
2024235993718,15cyclictest750-21ntpd21:01:040
2024237993630,3cyclictest0-21swapper/219:10:012
2024237993628,5cyclictest2082711-21latency21:45:582
2024237993628,5cyclictest2082711-21latency21:45:582
2024237993628,4cyclictest1001-21mta-sts-daemon00:31:532
2024236993630,3cyclictest2124990-21aten2.4-expect22:55:531
2024236993630,3cyclictest2124990-21aten2.4-expect22:55:531
2024236993628,5cyclictest421-21dbus-daemon21:45:011
2024236993628,5cyclictest2118362-21munin-plugin-st22:45:011
2024236993628,5cyclictest1000-21mta-sts-daemon21:28:281
2024235993618,3cyclictest171rcu_preempt23:36:090
2024235993614,14cyclictest151ktimers/021:31:030
202423799353,26cyclictest2156285-21ssh23:46:082
202423799353,26cyclictest2089129-21ssh21:56:062
202423799353,25cyclictest2113639-21ssh22:36:102
2024237993528,3cyclictest0-21swapper/221:00:012
2024237993526,5cyclictest2154959-21cron23:45:012
202423799352,20cyclictest177899-21cups-browsed21:51:082
202423799352,20cyclictest177899-21cups-browsed21:51:082
2024236993528,4cyclictest2124441-21ssh22:55:011
2024236993527,5cyclictest2156384-21wc23:46:101
2024236993527,5cyclictest2142767-21cron23:25:011
2024236993527,5cyclictest2039206-21munin-plugin-st20:00:011
2024236993526,4cyclictest0-21swapper/122:46:161
2024235993527,4cyclictest0-21swapper/023:16:030
2024235993523,6cyclictest2125262-21fschecks_count22:56:010
2024235993523,6cyclictest2125262-21fschecks_count22:56:010
2024237993426,5cyclictest2100654-21ssh22:15:552
2024237993426,4cyclictest0-21swapper/223:15:002
2024237993425,6cyclictest421-21dbus-daemon23:30:012
2024237993424,6cyclictest2112219-21cron22:35:012
2024237993420,11cyclictest351ktimers/200:21:022
202423799341,30cyclictest2076883-21ssh21:36:062
202423799341,22cyclictest2165547-21sh00:01:102
202423699343,22cyclictest177899-21cups-browsed23:01:041
2024236993430,2cyclictest0-21swapper/100:40:011
2024236993428,3cyclictest0-21swapper/100:00:021
2024236993428,3cyclictest0-21swapper/100:00:021
2024236993427,4cyclictest2185581-21idleruntime-cro00:35:011
2024236993427,4cyclictest1001-21mta-sts-daemon21:14:211
2024236993426,5cyclictest1-21systemd22:00:011
2024236993426,4cyclictest1000-21mta-sts-daemon22:27:071
2024236993424,7cyclictest2068850-21sshd21:23:111
2024236993420,7cyclictest2058946-21/usr/sbin/munin21:05:541
2024235993428,4cyclictest2028749-21systemd-run19:25:010
2024235993422,10cyclictest2057100-21tail20:56:170
202423799337,16cyclictest428-21mta-sts-daemon23:06:062
2024237993325,5cyclictest2039209-21munin-plugin-st20:00:012
2024237993324,5cyclictest2148861-21latency_hist23:35:032
2024237993323,6cyclictest2058737-21systemd-run21:05:012
202423799331,26cyclictest2180733-21ssh00:26:062
2024236993327,3cyclictest0-21swapper/120:40:011
2024236993326,4cyclictest1001-21mta-sts-daemon23:06:221
2024236993325,5cyclictest2179659-21latency_hist00:25:021
2024236993325,5cyclictest2030143-21grep19:26:151
2024236993323,6cyclictest0-21swapper/121:35:561
2024236993319,6cyclictest28-21ksoftirqd/120:46:071
2024235993326,4cyclictest2066361-21ssh21:20:010
2024235993325,4cyclictest2185572-21ssh00:35:010
2024235993324,7cyclictest2133590-21latency_hist23:10:000
2024235993324,6cyclictest0-21swapper/023:50:020
202423599331,26cyclictest2140923-21ssh23:21:040
202423599331,25cyclictest2067751-21ssh21:21:120
2024235993312,17cyclictest2092063-21sh22:01:040
2024237993222,7cyclictest2053255-21fschecks_time20:45:552
2024237993222,5cyclictest0-21swapper/200:18:302
2024237993220,9cyclictest2149845-21latency23:36:012
2024236993226,3cyclictest0-21swapper/123:51:031
2024236993225,4cyclictest2134588-21latency23:10:581
2024236993225,4cyclictest2091500-21aten_r2power_fr22:00:521
2024236993223,6cyclictest2109934-21ls22:30:541
2024235993225,5cyclictest12650irq/32-eth022:41:030
2024235993224,6cyclictest2179496-21latency_hist00:25:020
2024235993222,8cyclictest2033204-21cron19:40:010
2024235993220,3cyclictest0-21swapper/000:20:020
2024235993219,8cyclictest12750irq/33-eth021:36:080
2024237993123,5cyclictest2036147-21unixbench_singl19:46:142
2024237993119,8cyclictest265-21systemd-journal20:05:012
2024236993124,4cyclictest2144601-21grep23:26:151
2024236993124,4cyclictest2113993-21unixbench_multi22:36:151
2024236993124,4cyclictest2025740-21systemd-run19:15:011
2024236993124,4cyclictest2025740-21systemd-run19:15:011
202423699311,8cyclictest171rcu_preempt21:31:071
2024235993126,3cyclictest2069988-21aten2.4-expect21:25:540
2024235993122,6cyclictest420-21avahi-daemon23:28:460
2024235993118,11cyclictest2051051-21unixbench-2d20:36:130
202423599311,5cyclictest171rcu_preempt22:21:070
2024235993113,14cyclictest2098193-21ssh22:11:060
202423799302,22cyclictest2074675-21ssh21:32:122
202423799302,19cyclictest2092076-21ssh22:01:052
2024237993020,4cyclictest0-21swapper/221:11:132
2024237993012,11cyclictest2095090-21/usr/sbin/munin22:06:022
2024236993024,3cyclictest0-21swapper/100:20:011
2024236993023,4cyclictest2165040-21fschecks_count00:00:561
2024236993023,4cyclictest2042226-21cut20:10:011
2024236993022,5cyclictest2045754-21fschecks_count20:20:551
2024236993022,5cyclictest2045754-21fschecks_count20:20:551
2024235993024,4cyclictest694-21cron20:45:010
2024235993023,4cyclictest2082539-21uniq21:45:550
2024235993023,4cyclictest2082539-21uniq21:45:550
2024235993023,4cyclictest151ktimers/020:26:110
202423599302,22cyclictest2165144-21awk00:00:580
202423599302,22cyclictest2153166-21ssh23:41:060
202423799297,13cyclictest420-21avahi-daemon19:16:042
202423799297,13cyclictest420-21avahi-daemon19:16:042
2024237992921,5cyclictest2051198-21cron20:40:012
2024237992919,6cyclictest2042738-21fschecks_count20:10:562
2024237992910,11cyclictest2158799-21/usr/sbin/munin23:50:562
2024236992922,4cyclictest2039686-21cpuspeed20:00:541
2024236992922,4cyclictest2028332-21mailstats19:21:111
2024236992922,3cyclictest0-21swapper/123:16:021
2024236992922,3cyclictest0-21swapper/100:26:131
202423599296,15cyclictest12650irq/32-eth022:06:100
2024235992924,4cyclictest2024234-21cyclictest20:46:020
2024235992922,4cyclictest2101794-21diskmemload22:16:160
2024235992920,7cyclictest2043706-21cron20:15:010
202423599291,22cyclictest2134769-21ssh23:11:020
2024236992823,2cyclictest0-21swapper/119:55:011
2024236992820,5cyclictest2055697-21cron20:55:011
2024235992816,8cyclictest12650irq/32-eth021:56:080
2024235992815,9cyclictest421-21dbus-daemon20:16:010
202423799274,4cyclictest171rcu_preempt20:16:042
202423799274,14cyclictest428-21mta-sts-daemon19:21:072
202423799272,18cyclictest665-21snmpd20:51:082
2024236992722,2cyclictest0-21swapper/120:20:011
2024236992721,3cyclictest0-21swapper/119:45:011
2024236992720,4cyclictest0-21swapper/122:11:021
2024236992718,6cyclictest1000-21mta-sts-daemon23:35:321
2024235992719,5cyclictest2031709-21cron19:35:010
2024235992719,5cyclictest12750irq/33-eth019:26:060
2024235992715,8cyclictest151ktimers/021:06:020
202423799262,5cyclictest0-21swapper/219:26:062
2024236992621,2cyclictest0-21swapper/121:05:011
2024236992620,3cyclictest0-21swapper/120:31:061
2024236992618,5cyclictest2027092-21unixbench_multi19:16:171
2024236992618,5cyclictest2027092-21unixbench_multi19:16:171
2024235992620,4cyclictest2024234-21cyclictest20:00:010
2024235992620,3cyclictest1002-21mta-sts-daemon19:45:010
2024235992619,4cyclictest2061643-21sendmail_mailst21:11:100
2024235992618,5cyclictest151ktimers/019:16:080
2024235992618,5cyclictest151ktimers/019:16:080
202423599261,20cyclictest2122633-21ssh22:51:070
202423599261,18cyclictest2186866-21ssh00:36:060
202423599252,15cyclictest958-21sendmail-mta20:31:050
2024235992520,3cyclictest1003-21mta-sts-daemon22:29:470
2024235992518,4cyclictest1002-21mta-sts-daemon19:10:560
2024235992518,4cyclictest1002-21mta-sts-daemon19:10:560
2024235992513,9cyclictest151ktimers/019:51:070
2024235992417,5cyclictest177899-21cups-browsed20:51:490
2024237992315,5cyclictest2051746-21fschecks_count20:40:562
2024236992317,3cyclictest0-21swapper/120:11:051
2024236992317,3cyclictest0-21swapper/119:10:011
2024236992316,4cyclictest1003-21mta-sts-daemon19:31:071
2024235992318,2cyclictest0-21swapper/020:25:000
2024235992318,2cyclictest0-21swapper/020:25:000
2024236992216,3cyclictest1000-21mta-sts-daemon19:47:241
2024236992214,5cyclictest2047242-21fschecks_count20:26:001
202423599221,8cyclictest2040204-21/usr/sbin/munin20:01:070
2024235992215,4cyclictest0-21swapper/019:46:090
202423899191,8cyclictest0-21swapper/320:46:023
202423899182,8cyclictest0-21swapper/323:56:033
202423899182,8cyclictest0-21swapper/323:56:033
202423899182,10cyclictest0-21swapper/300:21:073
202423899172,8cyclictest0-21swapper/321:31:033
202423899164,7cyclictest0-21swapper/322:51:083
202423899163,7cyclictest0-21swapper/321:20:513
202423899162,8cyclictest0-21swapper/323:51:013
202423899162,6cyclictest0-21swapper/321:06:043
202423899162,5cyclictest0-21swapper/319:46:043
202423899161,8cyclictest0-21swapper/300:36:013
202423899161,7cyclictest0-21swapper/322:16:063
202423899153,7cyclictest0-21swapper/321:56:053
202423899153,7cyclictest0-21swapper/300:26:063
202423899153,5cyclictest0-21swapper/322:26:053
202423899152,8cyclictest0-21swapper/321:11:063
202423899152,8cyclictest0-21swapper/300:19:133
202423899152,6cyclictest0-21swapper/319:16:073
202423899152,6cyclictest0-21swapper/319:16:073
202423899151,8cyclictest0-21swapper/321:20:013
202423899151,7cyclictest0-21swapper/323:00:553
202423899151,7cyclictest0-21swapper/322:41:013
202423899151,6cyclictest0-21swapper/320:16:043
202423899144,7cyclictest0-21swapper/323:20:553
202423899143,6cyclictest0-21swapper/319:56:023
202423899142,8cyclictest0-21swapper/323:06:123
202423899142,7cyclictest0-21swapper/322:36:013
202423899142,7cyclictest0-21swapper/320:31:063
202423899142,7cyclictest0-21swapper/300:05:573
202423899142,6cyclictest0-21swapper/320:55:013
202423899142,6cyclictest0-21swapper/300:00:583
202423899141,8cyclictest0-21swapper/321:51:013
202423899141,8cyclictest0-21swapper/321:51:013
202423899141,8cyclictest0-21swapper/321:41:023
202423899141,7cyclictest0-21swapper/322:56:073
202423899141,7cyclictest0-21swapper/322:56:073
202423899141,7cyclictest0-21swapper/322:06:063
202423899141,7cyclictest0-21swapper/300:31:013
202423899141,6cyclictest0-21swapper/320:26:053
202423899141,6cyclictest0-21swapper/320:01:053
202423899141,6cyclictest0-21swapper/319:41:033
202423899141,5cyclictest0-21swapper/319:26:013
202423899133,6cyclictest0-21swapper/323:46:013
202423899132,7cyclictest0-21swapper/323:36:003
202423899132,7cyclictest0-21swapper/321:35:563
202423899132,5cyclictest0-21swapper/323:11:013
202423899131,8cyclictest0-21swapper/323:16:033
202423899131,8cyclictest0-21swapper/300:11:093
202423899131,7cyclictest0-21swapper/322:45:593
202423899131,6cyclictest0-21swapper/320:11:053
202423899131,6cyclictest0-21swapper/319:36:013
202423899131,6cyclictest0-21swapper/319:11:173
202423899131,6cyclictest0-21swapper/319:11:173
202423899131,5cyclictest0-21swapper/319:31:033
202423899124,6cyclictest0-21swapper/322:30:533
202423899123,7cyclictest0-21swapper/322:21:093
202423899122,7cyclictest0-21swapper/323:30:553
202423899122,6cyclictest0-21swapper/322:05:023
202423899122,6cyclictest0-21swapper/321:01:013
202423899121,7cyclictest0-21swapper/322:10:513
202423899121,7cyclictest0-21swapper/321:25:553
202423899121,6cyclictest0-21swapper/321:45:113
202423899121,6cyclictest0-21swapper/321:45:113
202423899121,6cyclictest0-21swapper/320:21:083
202423899121,6cyclictest0-21swapper/320:21:083
202423899120,7cyclictest0-21swapper/323:26:133
202423899113,4cyclictest0-21swapper/320:41:253
202423899112,6cyclictest0-21swapper/323:40:533
202423899111,6cyclictest0-21swapper/319:51:043
202423899111,5cyclictest0-21swapper/320:36:003
202423899101,5cyclictest0-21swapper/320:05:113
202423899101,5cyclictest0-21swapper/319:21:023
20242389993,4cyclictest0-21swapper/320:56:113
20242389991,5cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional