You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-14 - 14:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat Mar 14, 2026 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
194626599508499,6cyclictest1960597-21turbostat19:54:551
194626599503496,4cyclictest2039829-21turbostat23:39:541
194627299469444,21cyclictest2029135-21turbostat23:09:553
194627299465457,4cyclictest1997353-21turbostat21:35:003
194627299465456,6cyclictest2002532-21turbostat21:54:563
194627299465456,6cyclictest2002532-21turbostat21:54:563
194627299465456,6cyclictest1988683-21turbostat21:14:553
194626599465442,20cyclictest1950141-21turbostat19:24:541
194627299461454,4cyclictest1990418-21turbostat21:19:553
194626899461454,4cyclictest2018756-21turbostat22:39:552
194626899461454,4cyclictest2011497-21turbostat22:19:552
194627299460452,4cyclictest2025679-21turbostat22:55:003
194627299458440,14cyclictest2060093-21turbostat00:29:593
194627299457449,4cyclictest2022217-21turbostat22:45:003
194626399454445,6cyclictest2060093-21turbostat00:34:560
194627299450443,4cyclictest2018756-21turbostat22:35:003
194627299450443,4cyclictest1988683-21turbostat21:10:003
194627299448439,6cyclictest2013226-21turbostat22:20:013
194627299447438,6cyclictest2027405-21turbostat23:00:003
194627299446439,4cyclictest2043591-21turbostat23:45:013
194627299446437,6cyclictest2029135-21turbostat23:05:003
194627299445438,4cyclictest1966100-21turbostat20:04:593
194626599445436,6cyclictest2034640-21turbostat23:24:541
194627299443436,4cyclictest1955370-21turbostat19:35:003
194627299442435,4cyclictest1978262-21turbostat20:44:563
194627299441432,6cyclictest1974741-21turbostat20:29:593
194627299439431,5cyclictest2000808-21turbostat21:45:003
194627299438430,5cyclictest1978262-21turbostat20:40:013
194627299437430,4cyclictest2006002-21turbostat22:00:003
194626599436429,4cyclictest1995613-21turbostat21:34:561
194627299435426,6cyclictest2020486-21turbostat22:40:003
194627299433426,4cyclictest1957139-21turbostat19:44:553
194626899433426,4cyclictest2023949-21turbostat22:54:552
194626599432423,6cyclictest2054593-21turbostat00:19:551
194627299431424,4cyclictest1957139-21turbostat19:39:593
194627299428420,5cyclictest2061830-21turbostat00:39:563
194626599428421,4cyclictest1992154-21turbostat21:24:561
194626599426419,4cyclictest1967822-21turbostat20:14:561
194627299425418,4cyclictest2032907-21turbostat23:19:553
194626599422415,4cyclictest1973014-21turbostat20:29:551
194627299420413,4cyclictest1979995-21turbostat20:49:543
194627299417410,4cyclictest1947917-21turbostat19:15:003
194627299416409,4cyclictest2058052-21turbostat00:24:593
194627299416396,17cyclictest2023949-21turbostat22:50:003
194626899414406,5cyclictest2022217-21turbostat22:49:552
194627299413406,4cyclictest1985233-21turbostat21:00:003
194626899412405,4cyclictest2049395-21turbostat00:04:542
194627299408401,4cyclictest2008027-21turbostat22:09:543
194626899408401,4cyclictest2045617-21turbostat23:54:562
194626899408401,4cyclictest2045617-21turbostat23:54:552
194627299407400,4cyclictest1969553-21turbostat20:19:563
194627299404397,4cyclictest2036365-21turbostat23:25:013
194627299402395,4cyclictest1983463-21turbostat20:54:593
194626399401394,4cyclictest1951902-21turbostat19:29:550
194626899398390,5cyclictest2027405-21turbostat23:04:562
194627299390381,6cyclictest1993880-21turbostat21:25:003
194627299388370,15cyclictest2014943-21turbostat22:25:003
194627299387381,3cyclictest2051139-21turbostat00:05:003
194627299386379,4cyclictest2032907-21turbostat23:15:003
194626899386379,4cyclictest2043591-21turbostat23:49:562
194627299385377,5cyclictest2006002-21turbostat22:04:543
194626599385377,5cyclictest2036365-21turbostat23:29:561
194627299382376,3cyclictest2045617-21turbostat23:50:003
194627299382374,5cyclictest2000808-21turbostat21:49:553
194627299380373,4cyclictest1997353-21turbostat21:39:553
194626899380373,4cyclictest2016714-21turbostat22:34:552
194627299379373,4cyclictest2047354-21turbostat23:55:003
194627299379373,4cyclictest2047354-21turbostat23:55:003
194627299377371,3cyclictest1971288-21turbostat20:24:563
194627299373367,3cyclictest2011497-21turbostat22:15:003
194626899371348,20cyclictest1999079-21turbostat21:44:552
194627299369361,5cyclictest2049395-21turbostat23:59:593
194627299363356,4cyclictest2039829-21turbostat23:34:593
194627299363356,4cyclictest2039829-21turbostat23:34:593
194626899363340,20cyclictest1946130-21turbostat19:14:552
194627299362354,5cyclictest2014943-21turbostat22:29:543
194626599359353,3cyclictest1962331-21turbostat19:59:551
194627299358352,3cyclictest1953632-21turbostat19:30:013
194626599358336,19cyclictest2025679-21turbostat22:59:551
194627299357349,5cyclictest2054593-21turbostat00:15:013
194626399357350,4cyclictest2004269-21turbostat21:59:550
194627299356348,5cyclictest1976482-21turbostat20:35:013
194626899356349,4cyclictest1953632-21turbostat19:34:562
194626599353342,8cyclictest2009766-21turbostat22:14:561
194627299352347,3cyclictest2041560-21turbostat23:40:003
194627299349341,5cyclictest2052869-21turbostat00:10:013
194627299345338,4cyclictest1962331-21turbostat19:55:013
194627299338332,3cyclictest1947917-21turbostat19:19:553
194627299334328,4cyclictest1995613-21turbostat21:30:003
194626899326320,3cyclictest1958870-21turbostat19:49:542
194627299325319,3cyclictest1951902-21turbostat19:25:013
194626399324316,5cyclictest1976482-21turbostat20:39:560
194626899322315,4cyclictest2058052-21turbostat00:29:542
194627299321313,5cyclictest2061830-21turbostat00:35:013
194626399321313,5cyclictest1985233-21turbostat21:04:560
194626399317310,4cyclictest1993880-21turbostat21:29:550
194626599316310,3cyclictest1974741-21turbostat20:34:551
194627299314307,4cyclictest2056332-21turbostat00:20:003
194626899313306,4cyclictest2056332-21turbostat00:24:562
194626899313305,5cyclictest2031170-21turbostat23:14:552
194627299262239,19cyclictest1967822-21turbostat20:10:003
194627299247233,10cyclictest2038097-21turbostat23:30:003
194627299208194,10cyclictest1986961-21turbostat21:05:013
1945774213770,23sleep20-21swapper/219:05:422
1973991190,97rtkit-daemon0-21swapper/119:06:151
205975921140,6sleep00-21swapper/000:26:230
1945760210875,22sleep30-21swapper/319:05:313
19462689910795,8cyclictest2038097-21turbostat23:34:552
19462689910795,8cyclictest2038097-21turbostat23:34:552
1945757210672,23sleep00-21swapper/019:05:290
19462659910392,8cyclictest1964363-21turbostat20:04:551
19462639910190,8cyclictest1966100-21turbostat20:09:540
19462729910091,6cyclictest1960597-21turbostat19:50:013
1946272999990,6cyclictest1969553-21turbostat20:15:013
1946272999891,4cyclictest1964363-21turbostat20:00:003
19624362950,7sleep01962438-21ls19:55:020
197399730,5rtkit-daemon0-21swapper/223:28:172
197399660,9rtkit-daemon1972-21rtkit-daemon20:20:042
197399620,9rtkit-daemon1972-21rtkit-daemon21:37:322
197399620,10rtkit-daemon1972-21rtkit-daemon22:14:412
197399610,8rtkit-daemon1972-21rtkit-daemon21:31:262
197399600,8rtkit-daemon1972-21rtkit-daemon21:50:262
197399600,8rtkit-daemon1972-21rtkit-daemon21:50:262
197399590,8rtkit-daemon1972-21rtkit-daemon21:00:182
197399580,9rtkit-daemon1972-21rtkit-daemon22:55:412
197399580,8rtkit-daemon1972-21rtkit-daemon21:26:312
197399570,8rtkit-daemon1972-21rtkit-daemon20:18:102
197399560,8rtkit-daemon1972-21rtkit-daemon22:05:512
1946268995620,31cyclictest1980022-21wc20:45:012
1946268995518,32cyclictest2031175-21munin-run23:10:002
1946268995319,30cyclictest1983472-21basename20:55:002
1946268995319,30cyclictest1948956-21seq19:15:132
1946268995220,27cyclictest2014448-21grep22:20:292
1946268995218,30cyclictest2053855-21ntp_kernel_pll_00:10:242
1946268995116,12cyclictest1987781-21sed21:05:172
1946268995019,27cyclictest1137-21runrttasks23:38:542
1946268995016,11cyclictest2007389-21python322:00:272
1946268995013,12cyclictest2035526-21cat23:20:212
1946268994911,12cyclictest2054974-21wc00:15:112
20250102480,5sleep10-21swapper/122:50:251
1946268994812,12cyclictest1033-21gdbus19:39:332
1946268994810,10cyclictest1985154-21systemd-udevd20:59:452
197399470,39rtkit-daemon2212-21gvfs-afc-volume22:25:462
1946268994616,10cyclictest1975183-21tr20:30:122
1946268994612,12cyclictest2020787-21cpu22:40:122
1946268994612,11cyclictest1990964-21expr21:15:142
194626399461,19cyclictest20465002missed_timers23:50:210
194626399461,19cyclictest20465002missed_timers23:50:210
1946268994520,21cyclictest2042697-21missed_timers23:40:222
1946268994512,28cyclictest2006028-21gpgv22:00:012
1946268994511,11cyclictest1952261-21cpuspeed_turbos19:25:142
197399440,37rtkit-daemon1974166-21/usr/sbin/munin20:25:282
194626899449,11cyclictest1981254-21sed20:45:292
1946268994421,19cyclictest1962440-21apt-config19:55:012
1946268994417,7cyclictest1137-21runrttasks21:49:392
194626899441,28cyclictest0-21swapper/220:05:482
194626899441,15cyclictest438-20systemd-journal21:10:202
194626899432,16cyclictest2049033-40apt-check23:56:222
1946268994321,19cyclictest0-21swapper/221:20:142
197399420,33rtkit-daemon34-21ksoftirqd/223:19:592
1946268994211,7cyclictest1137-21runrttasks00:09:492
194626899421,15cyclictest1958888-21sleep19:44:592
1946268994211,12cyclictest1951131-21awk19:20:212
1946265994225,13cyclictest1032-21gdbus00:37:101
194626399426,12cyclictest13-21ksoftirqd/020:49:260
194626399425,10cyclictest13-21ksoftirqd/023:20:170
1946263994224,13cyclictest13-21ksoftirqd/019:40:450
1946268994114,7cyclictest1963755-21switchtime19:55:302
194626899411,15cyclictest2062602-21/usr/sbin/munin00:35:192
1946265994123,13cyclictest2021279-21sort22:40:201
1946265994119,10cyclictest2004282-21basename21:55:011
1946265994119,10cyclictest2004282-21basename21:55:001
1946265994118,19cyclictest2043289-40lsb_release23:41:301
194626399416,11cyclictest13-21ksoftirqd/000:17:420
194626399416,10cyclictest13-21ksoftirqd/022:09:350
194626399415,11cyclictest13-21ksoftirqd/021:51:250
194626399415,11cyclictest13-21ksoftirqd/021:51:250
194626899408,11cyclictest1977003-21cstates20:35:122
1946265994023,13cyclictest937-21polkitd22:32:321
1946265994022,14cyclictest1977378-21memory20:35:201
1946265994021,14cyclictest1982352-21df_inode20:50:161
194626399408,7cyclictest141rcu_preempt22:20:210
1946263994023,12cyclictest438-20systemd-journal00:24:590
1946263994022,6cyclictest13-21ksoftirqd/019:30:300
194626899399,7cyclictest2061904-21sort00:35:012
194626899391,11cyclictest1966075-21cron20:05:002
1946265993923,12cyclictest923-21dbus-daemon23:45:041
1946265993922,12cyclictest438-20systemd-journal19:29:551
194626399398,6cyclictest19654042pluginstate20:00:230
1946263993923,12cyclictest2080-21gdbus00:37:100
1946263993921,6cyclictest13-21ksoftirqd/000:14:190
1946263993921,6cyclictest13-21ksoftirqd/000:09:300
1946263993913,5cyclictest141rcu_preempt19:25:020
194626399391,10cyclictest0-21swapper/023:43:590
1946265993823,12cyclictest2009146-21/usr/sbin/munin22:05:261
1946265993822,12cyclictest923-21dbus-daemon23:15:001
1946265993822,12cyclictest1-21systemd20:50:011
1946265993818,16cyclictest2051690-21cstates00:05:131
1946263993826,8cyclictest1969581-21systemd20:15:020
1946263993822,6cyclictest13-21ksoftirqd/022:28:510
1946263993821,11cyclictest438-20systemd-journal23:25:200
194626399381,11cyclictest1985233-21turbostat21:00:000
194626899379,11cyclictest976-21rs:main2
194626599373,9cyclictest26-21ksoftirqd/120:25:001
1946265993716,17cyclictest2023465-21cat22:45:311
1946263993730,4cyclictest4497-21PK-Backend23:47:400
1946263993727,6cyclictest2022207-21dump-pmu-power22:45:010
1946263993727,6cyclictest2018759-21munin-run22:35:000
1946263993720,5cyclictest13-21ksoftirqd/022:36:040
1946263993720,5cyclictest13-21ksoftirqd/021:42:430
1946265993623,9cyclictest2047627-21rm23:55:031
1946265993623,9cyclictest2047627-21rm23:55:031
1946265993620,12cyclictest2014211-21ntpq22:20:221
1946265993620,11cyclictest438-20systemd-journal22:04:551
1946265993620,11cyclictest2000583-21sshd21:42:191
1946265993620,11cyclictest1957049-21systemd-udevd19:39:341
1946265993617,14cyclictest1997784-21cpuspeed_turbos21:35:131
194626599361,15cyclictest1971487-21latency_hist20:20:021
1946263993619,12cyclictest438-20systemd-journal20:00:000
1946263993618,6cyclictest13-21ksoftirqd/023:19:430
20481332350,7sleep1251rcuc/123:55:141
194626599354,11cyclictest26-21ksoftirqd/119:15:421
1946265993523,7cyclictest2143-21upowerd20:59:471
1946265993519,12cyclictest923-21dbus-daemon22:40:011
1946265993519,11cyclictest2049977-21expr00:00:131
1946265993518,13cyclictest2059776-40lsb_release00:26:271
1946265993518,13cyclictest2053714-21memory00:10:211
1946265993518,12cyclictest438-20systemd-journal21:59:591
1946263993527,5cyclictest4497-21PK-Backend21:18:190
1946263993519,5cyclictest13-21ksoftirqd/022:02:470
1946263993519,11cyclictest1987393-21cut21:05:110
1946263993518,12cyclictest438-20systemd-journal20:51:190
194626399351,10cyclictest4497-21PK-Backend21:38:190
1946265993423,7cyclictest2030115-21if_enp2s023:05:201
1946265993423,7cyclictest1988679-21latency_hist21:10:011
1946265993420,10cyclictest2060809-21if_err_enp2s000:30:181
1946265993420,10cyclictest1954804-21sed19:30:281
1946265993419,11cyclictest1995642-21pluginstate21:30:021
1946265993417,13cyclictest2012623-21python322:15:271
1946265993417,12cyclictest2033611-21if_enp2s023:15:171
194626599341,5cyclictest4497-21PK-Backend21:11:241
194626599341,27cyclictest1032-21gdbus22:25:031
194626399349,6cyclictest141rcu_preempt20:17:360
1946263993425,6cyclictest2011497-21turbostat22:15:010
194626399341,9cyclictest0-21swapper/021:47:440
1946263993419,11cyclictest1979693-21seq20:41:260
1946263993417,6cyclictest13-21ksoftirqd/021:32:540
1946263993417,13cyclictest2039126-21awk23:30:240
1946263993417,13cyclictest2039126-21awk23:30:230
20316492330,8chrt0-21swapper/023:10:130
1946265993323,5cyclictest26-21ksoftirqd/121:47:031
194626599332,26cyclictest438-20systemd-journal19:46:211
1946265993321,8cyclictest1958086-21cut19:40:221
1946265993318,11cyclictest2058068-21apt-get00:25:021
1946265993318,11cyclictest2039430-21sed23:30:281
1946265993318,11cyclictest2039430-21sed23:30:281
1946265993315,13cyclictest1978524-21apt20:40:121
194626599331,29cyclictest4497-21PK-Backend19:11:201
194626599331,28cyclictest1986943-21cron21:05:011
1946263993326,4cyclictest4497-21PK-Backend23:58:140
1946263993324,6cyclictest1960607-21cut19:50:010
1946263993318,11cyclictest2028061-21awk23:00:150
1946263993317,12cyclictest1956936-21seq19:38:120
1946263993315,14cyclictest2345-21gdbus22:50:210
1946265993220,7cyclictest2028749-21uptime23:00:281
1946265993217,11cyclictest1966552-21cat20:05:131
1946263993221,7cyclictest1993036-21chrt21:20:210
194626399322,14cyclictest2330-21gsd-disk-utilit00:00:220
1946263993219,9cyclictest1990405-21cron21:15:000
1946263993218,10cyclictest2024077-21sed22:50:020
1946263993216,12cyclictest2013146-21seq22:19:180
1946263993216,11cyclictest2025926-21find22:55:100
1946263993215,13cyclictest1972421-21python320:20:260
194626399321,27cyclictest4497-21PK-Backend19:15:320
1946265993124,4cyclictest4497-21PK-Backend21:18:541
1946263993117,10cyclictest1975181-21grep20:30:130
1946263993115,12cyclictest2040891-21wc23:35:260
1946263993114,12cyclictest976-21rs:main0
194626399311,11cyclictest0-21swapper/019:12:180
194626399306,5cyclictest141rcu_preempt23:05:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional