You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 13:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue May 19, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2035980998578,4cyclictest2099151-21sh23:07:583
2035966998480,2cyclictest0-21swapper/020:30:020
2035966998379,2cyclictest0-21swapper/023:50:020
2035971997922,17cyclictest2052691-21ssh21:31:361
2035971997916,25cyclictest2077026-21ssh22:21:591
2035966997973,2cyclictest0-21swapper/021:50:020
2035971997825,13cyclictest2058589-21ssh21:44:021
2035971997824,20cyclictest2091481-21ssh22:51:551
2035971997822,19cyclictest2114685-21sed23:40:021
2035971997819,19cyclictest2039413-21latency_hist20:15:021
2035971997818,22cyclictest2093177-21ssh22:55:161
2035966997873,3cyclictest0-21swapper/019:25:010
203596699781,66cyclictest0-21swapper/022:05:020
2035971997724,18cyclictest2082317-21ssh22:33:061
2035971997624,15cyclictest2043312-21ssh21:12:181
2035971997624,14cyclictest2066271-21ssh22:00:011
2035966997667,6cyclictest12-21ksoftirqd/020:35:020
2035971997516,16cyclictest2056447-21ssh21:39:481
2035971997418,15cyclictest2037881-21latency_hist19:45:021
2035971997415,22cyclictest2130665-21ssh00:13:211
203596699741,70cyclictest0-21swapper/020:35:020
2035971997321,18cyclictest2069229-21ssh22:05:481
2035971997318,18cyclictest2072359-21ssh22:12:191
2035971997316,18cyclictest2126535-21ssh00:04:511
2035971997225,14cyclictest2123710-21ssh23:59:021
2035971997220,15cyclictest2134599-21ssh00:21:051
2035971997220,15cyclictest2061622-21ssh21:50:041
2035971997220,15cyclictest2050813-21ssh21:27:501
2035971997219,14cyclictest2137478-21ssh00:27:161
2035971997218,18cyclictest2096192-21ssh23:01:391
2035971997218,18cyclictest2096192-21ssh23:01:381
2035971997216,15cyclictest2049321-21systemctl21:25:011
2035971997117,16cyclictest2085581-21munin-run22:40:021
2035971997116,19cyclictest2109426-21ssh23:29:211
203596699711,3cyclictest150-21pr/ttyS219:15:020
2035971997021,16cyclictest2099587-21ssh23:08:531
2035971997021,13cyclictest2120270-21ssh23:51:451
2035971997020,15cyclictest2141695-21ssh00:35:461
2035971997017,15cyclictest2038259-21systemd-run19:55:011
203596699709,13cyclictest12-21ksoftirqd/021:25:020
203596699704,40cyclictest19350irq/34-mmc100:30:020
2035966997038,29cyclictest2130967-21ssh00:13:550
2035971996924,14cyclictest2073958-21ssh22:15:271
2035971996922,16cyclictest2059759-21sh21:46:201
2035971996917,15cyclictest2131708-21ssh00:15:051
2035971996915,21cyclictest2101566-21ssh23:13:021
2035971996915,17cyclictest2088606-21ssh22:45:531
2035966996965,2cyclictest0-21swapper/021:50:020
2035971996818,15cyclictest2129006-21munin-run00:10:011
2035971996818,14cyclictest2139142-21ssh00:30:281
2035971996818,14cyclictest2067664-21ssh22:02:441
2035971996817,14cyclictest2087814-21ssh22:44:321
2035971996816,15cyclictest2102781-21ssh23:15:171
2035971996719,15cyclictest2046366-21ssh21:18:441
2035971996718,15cyclictest2111380-21ssh23:33:191
2035966996711,39cyclictest19350irq/34-mmc123:05:020
2035966996711,39cyclictest19350irq/34-mmc123:05:020
2035971996614,15cyclictest2117762-21ssh23:46:281
2035971996614,15cyclictest2117762-21ssh23:46:281
2035971996518,15cyclictest2042320-21cat21:10:021
2035971996517,16cyclictest2080070-21ssh22:28:201
2035971996516,14cyclictest2105530-21ssh23:20:551
2035971996513,15cyclictest2116390-21ssh23:43:481
203596699646,13cyclictest12-21ksoftirqd/023:00:000
2035966996430,26cyclictest12-21ksoftirqd/019:30:020
2035971996315,16cyclictest2038791-21munin-run20:05:011
203596699622,10cyclictest12-21ksoftirqd/000:35:010
2035966996034,10cyclictest12-21ksoftirqd/021:45:000
2035966996032,22cyclictest19350irq/34-mmc122:25:020
203596699599,30cyclictest131rcu_preempt23:46:370
203596699599,30cyclictest131rcu_preempt23:46:370
203596699598,36cyclictest16950irq/224-800000023:10:010
2035966995928,15cyclictest2106061-21ssh23:22:150
2035966995847,9cyclictest12-21ksoftirqd/022:33:450
2035966995838,17cyclictest16850irq/134-800000023:41:490
2035966995830,8cyclictest12-21ksoftirqd/000:39:430
203596699581,41cyclictest2113499-21ssh23:37:490
203597699577,25cyclictest2070700-21kworker/u8:022:35:032
2035971995718,8cyclictest2036960-21munin-run19:30:001
2035966995746,2cyclictest0-21swapper/020:50:020
2035966995744,9cyclictest12-21ksoftirqd/022:00:010
2035966995729,12cyclictest2088806-21ssh22:46:200
2035966995710,7cyclictest131rcu_preempt19:20:020
2035966995710,7cyclictest131rcu_preempt19:20:010
203597699567,6cyclictest131rcu_preempt00:22:472
203596699562,38cyclictest2084129-21ssh22:36:430
2035966995613,39cyclictest0-21swapper/023:10:280
2035976995548,5cyclictest2039578-21kworker/2:221:40:022
203596699559,6cyclictest131rcu_preempt21:17:330
2035966995545,7cyclictest12-21ksoftirqd/019:45:010
203596699552,17cyclictest2133870-21true00:20:010
2035976995446,5cyclictest2046938-21kworker/2:323:16:282
2035976995446,5cyclictest2039578-21kworker/2:221:56:172
203596699545,6cyclictest121ksoftirqd/000:21:330
203596699542,9cyclictest12-21ksoftirqd/020:05:020
2035966995427,12cyclictest2070748-21ssh22:09:150
2035966995424,15cyclictest2124779-21ssh00:00:530
2035966995419,17cyclictest16950irq/224-800000022:20:010
2035966995412,17cyclictest0-21swapper/019:10:020
2035976995343,7cyclictest2046938-21kworker/2:323:40:252
203596699538,31cyclictest2042134-21kworker/u8:221:30:020
2035966995326,12cyclictest2042542-21ssh21:10:340
2035966995313,9cyclictest12-21ksoftirqd/020:10:000
2035976995244,5cyclictest2046938-21kworker/2:323:23:012
2035976995244,5cyclictest2046938-21kworker/2:322:25:352
2035976995243,5cyclictest2035564-21kworker/2:319:30:012
2035971995234,14cyclictest364-21snmpd20:51:211
203596699526,26cyclictest131rcu_preempt23:26:570
2035966995244,5cyclictest16850irq/134-800000000:00:010
203596699522,43cyclictest2039831-21unin-run20:25:010
2035966995220,16cyclictest2072570-21ssh22:12:530
2035966995211,15cyclictest12-21ksoftirqd/022:21:580
2035980995137,11cyclictest1-21systemd22:05:003
2035976995143,5cyclictest2039578-21kworker/2:200:32:022
2035976995142,6cyclictest2039578-21kworker/2:223:48:552
2035976995142,6cyclictest2039578-21kworker/2:223:48:542
2035976995142,6cyclictest2039578-21kworker/2:200:35:592
2035976995142,5cyclictest2046938-21kworker/2:322:55:012
2035976995140,7cyclictest2046938-21kworker/2:322:12:242
2035966995126,10cyclictest2110800-21ssh23:31:560
2035966995125,10cyclictest2102670-21ssh23:15:040
2035976995043,4cyclictest2039578-21kworker/2:200:00:282
2035976995042,5cyclictest2046938-21kworker/2:323:37:552
2035976995042,5cyclictest2039578-21kworker/2:221:45:022
2035976995041,5cyclictest2031406-21kworker/2:119:45:002
2035976995031,15cyclictest275-21dbus-daemon20:40:012
2035971995037,9cyclictest2041841-21cat21:00:021
2035971995037,10cyclictest364-21snmpd19:21:031
2035971995037,10cyclictest2039044-21munin-run20:10:011
2035971995036,11cyclictest3467602-21systemd-journal20:43:011
2035966995023,11cyclictest2050633-21ssh21:27:270
2035966995021,14cyclictest2091788-21ssh22:52:400
203598099494,3cyclictest131rcu_preempt20:00:023
2035976994941,6cyclictest2039578-21kworker/2:221:28:412
2035976994941,5cyclictest2046938-21kworker/2:323:30:022
2035976994941,5cyclictest2046938-21kworker/2:322:44:222
2035976994941,5cyclictest2046938-21kworker/2:322:40:032
2035976994941,5cyclictest2039578-21kworker/2:221:32:022
2035976994941,5cyclictest2039578-21kworker/2:221:22:072
2035976994941,5cyclictest2039578-21kworker/2:221:17:532
2035976994941,5cyclictest2039578-21kworker/2:220:45:022
2035976994941,5cyclictest2035564-21kworker/2:319:50:002
2035976994940,6cyclictest2046938-21kworker/2:323:12:002
2035971994937,9cyclictest3467602-21systemd-journal20:35:191
2035971994937,9cyclictest3467602-21systemd-journal20:26:001
2035971994936,10cyclictest364-21snmpd20:31:171
2035971994933,11cyclictest2036155-21cron19:15:001
2035971994928,16cyclictest2036459-21modprobe19:20:011
2035971994928,16cyclictest2036459-21modprobe19:20:011
2035966994931,13cyclictest275-21dbus-daemon19:40:000
2035966994916,29cyclictest16850irq/134-800000021:39:400
2035980994833,11cyclictest275-21dbus-daemon21:20:013
2035980994833,10cyclictest309-21in:imuxsock19:20:013
2035980994833,10cyclictest309-21in:imuxsock19:20:013
2035980994830,13cyclictest275-21dbus-daemon21:40:013
203597699489,6cyclictest131rcu_preempt21:55:002
2035976994841,4cyclictest2039578-21kworker/2:223:56:252
2035976994841,3cyclictest2046938-21kworker/2:322:15:072
2035976994840,5cyclictest2046938-21kworker/2:323:00:262
2035976994840,5cyclictest2046938-21kworker/2:323:00:262
2035976994840,5cyclictest2046938-21kworker/2:322:55:162
2035976994840,5cyclictest2046938-21kworker/2:322:25:002
2035976994840,5cyclictest2039578-21kworker/2:200:05:332
2035971994833,12cyclictest2039512-21sshd20:17:031
2035971994829,16cyclictest364-21snmpd21:04:031
2035966994839,6cyclictest16950irq/224-800000000:10:010
2035980994736,8cyclictest37-21ksoftirqd/323:34:593
2035980994735,8cyclictest2115553-21sshd23:41:533
2035980994732,11cyclictest37-21ksoftirqd/320:40:003
2035980994729,12cyclictest275-21dbus-daemon22:45:003
2035976994740,5cyclictest2039578-21kworker/2:221:49:142
2035976994739,6cyclictest2039578-21kworker/2:223:50:402
2035976994739,5cyclictest2039578-21kworker/2:222:04:072
2035976994739,5cyclictest2039578-21kworker/2:221:05:362
2035976994739,5cyclictest2039578-21kworker/2:200:11:132
2035976994739,5cyclictest2031406-21kworker/2:121:05:002
2035976994739,5cyclictest2031406-21kworker/2:120:20:022
2035971994735,8cyclictest48-21kcompactd020:46:111
2035971994734,10cyclictest364-21snmpd19:36:061
2035971994733,10cyclictest364-21snmpd19:46:091
203598099467,3cyclictest150-21pr/ttyS222:18:553
203598099465,4cyclictest131rcu_preempt21:50:013
2035980994631,11cyclictest3467602-21systemd-journal22:23:543
2035980994630,11cyclictest275-21dbus-daemon20:25:013
2035976994638,5cyclictest2039578-21kworker/2:200:16:052
203597699461,6cyclictest2089607-21ssh22:48:102
2035976994613,5cyclictest131rcu_preempt20:33:082
2035971994633,9cyclictest364-21snmpd19:55:131
203596699463,7cyclictest2041601-21modprobe21:00:000
2035966994628,14cyclictest2040833-21sshd20:43:550
2035976994540,3cyclictest2039578-21kworker/2:200:27:422
2035976994537,5cyclictest2039578-21kworker/2:220:55:022
2035976994536,6cyclictest2039578-21kworker/2:222:08:382
2035971994534,8cyclictest48-21kcompactd020:23:261
203596699453,6cyclictest309-21in:imuxsock21:10:000
203596699452,9cyclictest2038505-21munin-run20:00:000
2035966994517,12cyclictest2087457-21ssh22:43:480
2035966994510,5cyclictest131rcu_preempt20:15:000
203598099444,3cyclictest150-21pr/ttyS223:58:143
2035980994431,8cyclictest309-21in:imuxsock21:15:013
2035980994426,13cyclictest275-21dbus-daemon20:15:013
203598099442,11cyclictest2127597-21ssh00:06:463
2035976994436,5cyclictest2039578-21kworker/2:220:59:272
2035966994435,7cyclictest19350irq/34-mmc119:35:030
203598099434,4cyclictest131rcu_preempt23:00:073
203598099434,4cyclictest131rcu_preempt23:00:063
2035980994333,5cyclictest2042145-21diskmemload23:35:293
203598099432,4cyclictest131rcu_preempt00:02:423
203598099431,3cyclictest131rcu_preempt22:48:153
203598099431,10cyclictest2048052-21rm21:22:083
2035976994335,5cyclictest2046938-21kworker/2:323:05:172
2035976994335,5cyclictest2039578-21kworker/2:221:12:382
2035976994326,11cyclictest275-21dbus-daemon20:05:002
2035980994230,9cyclictest1-21systemd20:50:013
203598099422,9cyclictest2037974-21sshd19:48:553
2035980994229,10cyclictest275-21dbus-daemon21:00:023
2035980994226,12cyclictest275-21dbus-daemon23:15:023
2035980994226,11cyclictest0-21swapper/323:50:023
2035980994226,11cyclictest0-21swapper/323:50:013
203598099422,4cyclictest131rcu_preempt22:30:543
203598099421,3cyclictest131rcu_preempt00:13:253
203597699426,9cyclictest30-21ksoftirqd/220:45:562
2035976994234,5cyclictest2031406-21kworker/2:120:15:022
2035966994224,14cyclictest2041871-21sshd21:03:550
2035966994223,13cyclictest275-21dbus-daemon20:20:000
203598099414,3cyclictest131rcu_preempt21:33:193
2035980994129,9cyclictest2051775-21munin-run21:30:003
2035980994128,10cyclictest37-21ksoftirqd/319:15:003
2035980994127,11cyclictest1-21systemd20:05:003
2035980994127,10cyclictest2039014-21sshd20:09:553
2035980994116,7cyclictest361rcuc/322:00:013
203598099411,4cyclictest131rcu_preempt23:23:513
203598099411,4cyclictest131rcu_preempt23:18:113
203598099411,3cyclictest150-21pr/ttyS223:51:383
203598099411,3cyclictest150-21pr/ttyS222:56:533
203598099411,3cyclictest131rcu_preempt22:10:373
203598099411,3cyclictest131rcu_preempt00:17:443
203597699416,6cyclictest131rcu_preempt19:31:552
203597699413,5cyclictest131rcu_preempt23:30:082
2035971994130,8cyclictest364-21snmpd19:31:051
2035966994130,8cyclictest12-21ksoftirqd/020:50:020
203598099406,4cyclictest131rcu_preempt00:22:503
203598099405,3cyclictest131rcu_preempt22:50:223
203598099404,3cyclictest131rcu_preempt21:40:223
2035980994027,10cyclictest37-21ksoftirqd/300:28:123
203598099402,4cyclictest131rcu_preempt22:29:153
203597699403,4cyclictest131rcu_preempt20:10:002
2035966994034,4cyclictest0-21swapper/019:55:010
203598099395,3cyclictest131rcu_preempt22:38:353
203598099394,3cyclictest131rcu_preempt00:39:033
203598099393,4cyclictest131rcu_preempt00:34:223
203598099392,3cyclictest131rcu_preempt19:40:003
203598099391,4cyclictest131rcu_preempt22:05:473
2035976993931,5cyclictest2031406-21kworker/2:120:25:532
203598099384,3cyclictest131rcu_preempt21:51:143
203598099383,3cyclictest131rcu_preempt23:26:063
2035976993828,7cyclictest2036154-21cron19:15:002
2035976993824,11cyclictest30-21ksoftirqd/220:22:382
203596699389,6cyclictest131rcu_preempt19:50:000
203598099375,3cyclictest150-21pr/ttyS220:40:553
2035980993728,6cyclictest2036960-21/usr/sbin/munin19:30:013
2035980993728,5cyclictest2039635-21latency_hist20:20:023
2035980993727,7cyclictest2036722-21pluginstate19:25:013
203598099371,3cyclictest131rcu_preempt19:45:013
203598099363,3cyclictest131rcu_preempt20:29:553
203598099363,3cyclictest131rcu_preempt19:55:003
203598099361,4cyclictest131rcu_preempt19:35:023
2035976993627,6cyclictest274-21cron19:20:002
2035976993627,6cyclictest274-21cron19:20:002
203597699362,4cyclictest131rcu_preempt20:00:022
2035976993620,12cyclictest3467602-21systemd-journal19:25:002
203598099353,3cyclictest131rcu_preempt19:10:013
203598099351,4cyclictest131rcu_preempt20:35:013
203598099351,3cyclictest131rcu_preempt20:55:013
203598099342,3cyclictest131rcu_preempt21:05:003
2035980993420,10cyclictest275-21dbus-daemon21:10:003
203597699345,3cyclictest131rcu_preempt19:40:002
203597699342,5cyclictest131rcu_preempt19:54:552
2035971993319,10cyclictest2035974-21grep19:10:011
2035976993120,7cyclictest30-21ksoftirqd/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional