You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 23:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sat Jun 27, 2026 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2036922995226,23cyclictest2144501-21latency11:05:290
2036924995026,3cyclictest0-21swapper/212:00:352
2036924994923,3cyclictest0-21swapper/210:45:402
2036924994921,7cyclictest2080241-21sh09:20:462
2036923994935,10cyclictest1003-21mta-sts-daemon12:09:141
2036923994838,6cyclictest420-21avahi-daemon09:59:521
2036924994724,3cyclictest0-21swapper/210:50:392
2036924994722,4cyclictest2143780-21latency_hist11:05:022
2036923994735,9cyclictest2159786-21kernelversion11:30:291
203692299462,16cyclictest2193731-21ntpq12:25:320
203692299462,15cyclictest421-21dbus-daemon09:40:370
2036922994621,22cyclictest2129867-21timerwakeupswit10:40:440
203692299461,17cyclictest2101886-21ntpq09:55:380
2036922994538,4cyclictest2106866-21cron10:05:010
2036922994438,4cyclictest70-21kswapd009:30:350
2036922994425,16cyclictest2201900-21sendmail12:40:010
2036922994425,16cyclictest2201900-21sendmail12:40:010
2036924994334,6cyclictest2085374-21latency_hist09:30:022
2036922994336,4cyclictest2162070-21systemd-run11:35:010
2036922994330,4cyclictest2076004-21cron09:15:000
2036922994330,4cyclictest2076004-21cron09:15:000
203692299432,5cyclictest2089403-21ntpq09:35:330
203692299431,11cyclictest2052876-21ntpq08:00:350
203692499422,31cyclictest2187652-21ssh12:15:342
2036923994234,5cyclictest2128294-21ssh10:40:001
203692299422,6cyclictest2040891-21ntpq07:20:360
2036924994133,5cyclictest2051912-21idleruntime-cro08:00:012
2036923994132,5cyclictest1001-21mta-sts-daemon10:43:091
2036923994118,13cyclictest420-21avahi-daemon09:35:401
2036922994134,4cyclictest151ktimers/012:20:310
2036922994129,6cyclictest151ktimers/007:35:350
203692299411,14cyclictest2072410-21ntpq09:05:380
203692299411,12cyclictest2135548-21ntpq10:50:350
203692299411,11cyclictest2083093-21missed_timers09:25:310
2036923994032,5cyclictest2186658-21latency_hist12:15:021
2036923994032,5cyclictest2140575-21munin-plugin-st11:00:011
2036923994032,4cyclictest2195781-21cron12:30:011
203692299406,5cyclictest2054489-21/usr/sbin/munin08:05:380
203692299403,8cyclictest2039332-21ntpq07:15:350
203692299402,7cyclictest2055806-21ntpq08:10:340
203692299402,5cyclictest2069387-21ntpq08:55:360
203692299402,5cyclictest2043819-21ntpq07:30:330
203692299402,5cyclictest2043819-21ntpq07:30:330
203692299401,6cyclictest2114103-21ntpq10:15:340
203692299401,6cyclictest2114103-21ntpq10:15:340
203692299401,5cyclictest2057310-21ntpq08:15:330
2036922994012,25cyclictest2137846-21/usr/sbin/munin10:55:280
203692299401,11cyclictest2037542-21irqstats07:10:290
203692499397,27cyclictest750-21ntpd07:15:312
2036924993932,4cyclictest2086927-21head09:30:432
2036924993931,5cyclictest2128309-21munin-run10:40:012
2036923993931,5cyclictest2057932-21idleruntime-cro08:20:011
203692299395,4cyclictest2120154-21ntpq10:25:320
203692299395,4cyclictest2120154-21ntpq10:25:320
2036922993931,4cyclictest151ktimers/007:25:300
2036922993931,4cyclictest151ktimers/007:25:300
2036922993929,6cyclictest151ktimers/008:35:350
203692299392,5cyclictest2061849-21ntpq08:30:390
2036922993924,8cyclictest2178410-21sh12:00:320
203692299391,9cyclictest2186967-21aten_r2power_en12:15:220
203692299391,9cyclictest2067808-21ntpq08:50:350
2036922993918,19cyclictest0-21swapper/010:10:230
203692299391,7cyclictest2048326-21ntpq07:45:320
203692299391,16cyclictest2141804-21ps11:00:400
203692299391,12cyclictest2181455-21ntpq12:05:320
203692299391,11cyclictest2049411-21diskstats07:50:240
203692299391,10cyclictest2050896-21df07:55:240
203692299390,11cyclictest2167420-21ssh11:42:400
2036924993828,7cyclictest1236063-21cups-browsed12:33:292
2036923993831,4cyclictest1000-21mta-sts-daemon11:50:331
2036923993828,4cyclictest0-21swapper/111:40:361
2036923993810,24cyclictest2123284-21ssh10:30:331
2036923993810,24cyclictest2123284-21ssh10:30:331
203692299383,8cyclictest2064800-21ntpq08:40:320
2036922993829,6cyclictest1000-21mta-sts-daemon08:45:380
203692299382,6cyclictest2175332-21ntpq11:55:320
203692299382,6cyclictest2077039-21ntpq09:15:310
2036922993822,11cyclictest12650irq/32-eth010:45:340
2036922993817,18cyclictest2059420-21cron08:25:010
203692299381,5cyclictest2098712-21ntpq09:50:340
203692299381,4cyclictest2107934-21ntpq10:05:330
203692299381,12cyclictest2060350-21ntpq08:25:350
203692299381,12cyclictest2046833-21ntpq07:40:340
203692299381,11cyclictest2169317-21ssh11:45:370
203692299381,11cyclictest2127277-21ssh10:36:490
203692299380,15cyclictest2095632-21ssh09:45:330
2036924993726,4cyclictest0-21swapper/209:45:342
2036923993730,4cyclictest2122250-21latency_hist10:30:011
2036923993730,4cyclictest2122250-21latency_hist10:30:011
2036923993729,5cyclictest2198826-21idleruntime-cro12:35:011
2036923993729,3cyclictest0-21swapper/110:05:021
2036923993728,6cyclictest421-21dbus-daemon09:50:011
2036923993727,7cyclictest265-21systemd-journal07:20:011
203692299378,3cyclictest2164053-21ssh11:36:460
203692299372,7cyclictest2157068-21ntpq11:25:380
2036922993719,14cyclictest0-21swapper/011:15:210
2036922993716,18cyclictest0-21swapper/011:50:200
203692299371,11cyclictest2123299-21ntpq10:30:340
203692299371,11cyclictest2123299-21ntpq10:30:340
2036924993630,3cyclictest0-21swapper/210:15:012
2036924993628,5cyclictest1-21systemd12:10:012
2036924993622,11cyclictest2054319-21ntp_kernel_pll_08:05:342
203692499361,24cyclictest2101727-21ssh09:55:312
2036923993627,6cyclictest0-21swapper/110:25:001
2036923993627,6cyclictest0-21swapper/110:25:001
2036923993625,8cyclictest2163489-21timerandwakeup11:35:441
2036923993625,6cyclictest0-21swapper/111:00:331
203692299363,6cyclictest2147343-21diskstats11:10:260
203692299362,11cyclictest2184604-21ssh12:10:340
203692299361,9cyclictest2080325-21ntpq09:20:490
203692299361,8cyclictest2070262-21aten_r2power_po09:00:220
203692299361,11cyclictest2196773-21/usr/sbin/munin12:30:340
203692299360,12cyclictest2117917-21ssh10:21:110
203692299360,12cyclictest2117917-21ssh10:21:110
203692499353,25cyclictest2169178-21ssh11:45:312
2036924993528,4cyclictest2158980-21grep11:30:012
2036924993527,4cyclictest0-21swapper/210:05:342
2036923993530,2cyclictest0-21swapper/107:40:001
2036923993527,5cyclictest2201882-21idleruntime-cro12:40:011
2036923993527,5cyclictest2201882-21idleruntime-cro12:40:011
2036923993526,6cyclictest2131831-21aten_r2power_fr10:45:221
2036923993525,6cyclictest0-21swapper/109:35:021
203692399352,24cyclictest2107936-21ssh10:05:331
203692299351,5cyclictest2154007-21ntpq11:20:380
2036924993427,4cyclictest2195975-21latency_hist12:30:032
2036924993426,5cyclictest2137886-21aten_r2power_cu10:55:222
2036924993425,6cyclictest0-21swapper/209:10:242
2036924993425,6cyclictest0-21swapper/209:10:242
2036924993423,8cyclictest2153826-21needreboot11:20:312
2036924993423,8cyclictest2057220-21needreboot08:15:302
203692399343,26cyclictest2080263-21ssh09:20:461
203692399343,25cyclictest2092634-21ssh09:40:351
2036923993428,4cyclictest0-21swapper/108:10:051
2036923993427,4cyclictest1002-21mta-sts-daemon10:50:351
2036923993427,4cyclictest0-21swapper/111:15:361
2036923993426,4cyclictest0-21swapper/111:10:441
2036923993423,8cyclictest2144494-21latency11:05:291
2036923993422,9cyclictest2052816-21ntp_kernel_pll_08:00:321
2036923993421,8cyclictest2171440-21ls11:50:031
2036924993327,3cyclictest0-21swapper/211:40:362
2036924993327,3cyclictest0-21swapper/211:10:342
2036924993327,3cyclictest0-21swapper/209:40:012
2036924993326,4cyclictest2105433-21grep10:00:442
2036924993325,5cyclictest2151370-21unixbench_multi11:15:432
203692499332,25cyclictest2117063-21ssh10:20:292
203692499332,25cyclictest2117063-21ssh10:20:292
203692399335,22cyclictest2178401-21ssh12:00:311
2036923993324,6cyclictest420-21avahi-daemon09:11:591
2036923993324,6cyclictest420-21avahi-daemon09:11:591
2036924993226,3cyclictest0-21swapper/211:55:352
2036924993225,4cyclictest1000-21mta-sts-daemon09:00:372
2036924993225,3cyclictest0-21swapper/209:40:372
2036924993223,7cyclictest694-21cron09:20:012
2036924993223,6cyclictest539-21wpa_supplicant11:50:232
2036924993211,13cyclictest1236058-21cupsd07:10:382
203692399327,18cyclictest1236063-21cups-browsed11:20:361
2036923993225,4cyclictest2177406-21ssh12:00:011
2036923993225,4cyclictest1003-21mta-sts-daemon11:28:371
2036923993224,5cyclictest0-21swapper/109:55:011
2036923993224,4cyclictest0-21swapper/109:25:371
203692299325,4cyclictest2036930-21systemd-run07:10:010
2036924993123,5cyclictest2190061-21aten_r2power_en12:20:212
2036924993123,5cyclictest2066938-21systemd-run08:50:012
2036924993123,5cyclictest2060843-21unixbench_singl08:25:452
2036924993123,5cyclictest1001-21mta-sts-daemon08:55:332
203692499312,22cyclictest2129417-21ssh10:40:352
2036923993123,5cyclictest2063913-21grep08:40:011
203692399312,20cyclictest566-21gmain08:05:351
203692499303,21cyclictest2199838-21ssh12:35:332
203692499303,21cyclictest2199838-21ssh12:35:332
2036924993022,5cyclictest2045905-21munin-run07:40:002
2036924993021,6cyclictest1335524-21munin-node10:32:222
2036924993021,6cyclictest1335524-21munin-node10:32:222
2036924993021,5cyclictest2163178-21diskmemload11:35:372
203692399303,18cyclictest2111064-21ssh10:10:361
2036923993023,4cyclictest2064457-21fschecks_time08:40:261
2036923993022,3cyclictest0-21swapper/109:10:001
2036923993021,6cyclictest2066950-21latency_hist08:50:021
2036923993020,6cyclictest2191307-21unixbench_singl12:20:431
203692399301,22cyclictest2077173-21ssh09:15:371
2036924992924,2cyclictest0-21swapper/209:10:002
2036924992922,4cyclictest2100703-21cron09:55:012
2036924992922,4cyclictest2047314-21tail07:40:452
2036924992922,3cyclictest0-21swapper/210:15:322
2036924992922,3cyclictest0-21swapper/210:15:322
2036924992921,5cyclictest2050422-21systemd-run07:55:012
2036924992920,7cyclictest2058289-21aten_r2power_po08:20:222
2036923992923,2cyclictest0-21swapper/108:55:011
2036923992922,4cyclictest1000-21mta-sts-daemon07:45:391
2036923992922,3cyclictest0-21swapper/110:20:031
2036923992922,3cyclictest0-21swapper/110:20:031
2036923992920,6cyclictest2187240-21fschecks_time12:15:261
203692399291,22cyclictest2069375-21ntpq08:55:351
2036924992822,3cyclictest0-21swapper/208:35:022
2036924992821,3cyclictest0-21swapper/211:05:232
2036924992820,5cyclictest1-21systemd10:25:412
2036924992820,5cyclictest1-21systemd10:25:412
2036924992721,3cyclictest2047437-21kworker/u8:5+xprtiod08:40:262
2036924992720,4cyclictest2056038-21mailstats08:10:412
2036924992720,3cyclictest0-21swapper/212:10:432
2036923992721,3cyclictest0-21swapper/108:00:011
2036923992720,4cyclictest2043370-21cpuspeed07:30:231
2036923992720,4cyclictest2043370-21cpuspeed07:30:231
2036923992714,9cyclictest2046823-21sshd07:40:341
2036924992619,4cyclictest2041412-21cron07:25:012
2036924992618,5cyclictest2063906-21cron08:40:012
203692499261,18cyclictest2160016-21ntpq11:30:342
2036923992621,2cyclictest0-21swapper/107:10:461
2036924992518,4cyclictest1002-21mta-sts-daemon08:50:202
203692399253,12cyclictest1-21systemd09:00:371
2036923992517,4cyclictest0-21swapper/108:20:351
2036923992516,6cyclictest1003-21mta-sts-daemon07:50:371
203692499244,14cyclictest428-21mta-sts-daemon07:45:352
2036923992417,4cyclictest2062304-21unixbench_multi08:30:481
203692499232,16cyclictest2052973-21ps08:00:382
2036923992315,5cyclictest1003-21mta-sts-daemon08:25:371
203692399228,10cyclictest2042354-21sshd07:25:341
203692399228,10cyclictest2042354-21sshd07:25:341
2036924992111,7cyclictest265-21systemd-journal07:30:012
2036924992111,7cyclictest265-21systemd-journal07:30:012
203692399212,14cyclictest2040841-21ntpq07:20:341
2036924992013,4cyclictest2044408-21latency_hist07:35:002
2036924992013,4cyclictest2044408-21latency_hist07:35:002
203692599192,10cyclictest0-21swapper/311:30:343
203692599192,10cyclictest0-21swapper/310:00:343
203692599191,11cyclictest0-21swapper/309:10:293
203692599191,11cyclictest0-21swapper/309:10:293
203692599183,10cyclictest0-21swapper/309:40:333
203692599182,9cyclictest0-21swapper/309:45:363
203692599182,10cyclictest0-21swapper/312:00:353
203692599181,8cyclictest0-21swapper/309:30:273
203692599181,10cyclictest0-21swapper/310:30:313
203692599181,10cyclictest0-21swapper/310:30:313
2036924991812,3cyclictest0-21swapper/207:10:012
203692599173,9cyclictest0-21swapper/308:45:353
203692599173,8cyclictest0-21swapper/312:25:223
203692599172,9cyclictest0-21swapper/311:20:323
203692599172,9cyclictest0-21swapper/309:55:333
203692599172,9cyclictest0-21swapper/308:10:043
203692599172,8cyclictest0-21swapper/311:55:333
203692599172,8cyclictest0-21swapper/309:15:363
203692599172,6cyclictest0-21swapper/309:00:333
203692599172,10cyclictest0-21swapper/311:35:343
203692599172,10cyclictest0-21swapper/310:45:373
203692599171,10cyclictest0-21swapper/310:20:243
203692599171,10cyclictest0-21swapper/310:20:243
2036923991712,3cyclictest0-21swapper/107:10:021
203692599162,9cyclictest0-21swapper/311:45:323
203692599162,9cyclictest0-21swapper/311:10:233
203692599162,9cyclictest0-21swapper/310:05:343
203692599162,8cyclictest0-21swapper/312:35:343
203692599162,8cyclictest0-21swapper/312:35:343
203692599162,8cyclictest0-21swapper/312:05:313
203692599162,8cyclictest0-21swapper/311:25:323
203692599162,8cyclictest0-21swapper/308:55:353
203692599161,9cyclictest0-21swapper/311:05:363
203692599161,9cyclictest0-21swapper/310:15:383
203692599161,9cyclictest0-21swapper/310:15:383
203692599161,8cyclictest0-21swapper/311:40:313
203692599161,8cyclictest0-21swapper/311:00:233
203692599161,7cyclictest0-21swapper/307:30:313
203692599161,7cyclictest0-21swapper/307:30:313
203692599153,7cyclictest0-21swapper/310:25:333
203692599153,7cyclictest0-21swapper/310:25:333
203692599153,6cyclictest0-21swapper/312:30:243
203692599152,7cyclictest0-21swapper/309:20:343
203692599151,9cyclictest0-21swapper/312:20:353
203692599151,9cyclictest0-21swapper/311:53:333
203692599151,9cyclictest0-21swapper/310:50:273
203692599151,9cyclictest0-21swapper/309:35:353
203692599151,5cyclictest0-21swapper/308:05:303
203692599143,6cyclictest0-21swapper/308:15:313
203692599143,5cyclictest0-21swapper/307:20:313
203692599142,8cyclictest0-21swapper/312:15:313
203692599142,8cyclictest0-21swapper/309:28:403
203692599142,8cyclictest0-21swapper/308:45:023
203692599142,7cyclictest0-21swapper/310:55:353
203692599141,8cyclictest0-21swapper/311:15:213
203692599141,8cyclictest0-21swapper/310:10:293
203692599141,7cyclictest0-21swapper/310:35:353
203692599141,6cyclictest0-21swapper/307:40:343
203692599141,10cyclictest0-21swapper/309:50:233
203692599133,6cyclictest0-21swapper/309:10:003
203692599133,6cyclictest0-21swapper/307:15:343
203692599133,5cyclictest0-21swapper/307:55:343
203692599132,7cyclictest0-21swapper/310:40:223
203692599132,5cyclictest0-21swapper/308:25:313
203692599132,5cyclictest0-21swapper/307:25:313
203692599132,5cyclictest0-21swapper/307:25:313
203692599131,8cyclictest0-21swapper/312:10:223
203692599131,6cyclictest0-21swapper/308:20:333
203692599131,6cyclictest0-21swapper/307:50:333
203692599131,6cyclictest0-21swapper/307:45:363
203692599131,6cyclictest0-21swapper/307:35:323
203692599130,7cyclictest0-21swapper/308:00:353
203692599112,6cyclictest0-21swapper/308:50:363
203692599111,6cyclictest0-21swapper/307:10:333
203692599111,5cyclictest0-21swapper/308:35:303
203692599104,4cyclictest0-21swapper/308:30:433
20369259992,5cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional