You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-19 - 00:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sat Apr 18, 2026 12:43:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1926704996962,5cyclictest2105108-21awk12:10:330
1926706995949,7cyclictest2041268-21head10:45:502
192670699563,45cyclictest2004595-21ssh10:00:422
1926706995545,5cyclictest1962738-21diskmemload11:40:012
1926706995545,5cyclictest1962738-21diskmemload11:40:012
1926706995542,8cyclictest0-21swapper/211:20:382
192670699476,32cyclictest2097669-21ssh12:00:422
1926704994741,4cyclictest2007372-21ssh10:05:010
1926704994638,5cyclictest2096388-21basename12:00:010
1926706994436,5cyclictest2096355-21sh12:00:012
1926706994435,4cyclictest0-21swapper/210:05:372
1926705994434,5cyclictest0-21swapper/110:30:411
1926704994437,4cyclictest2100296-21ls12:05:010
1926704994433,8cyclictest1972694-21unixbench-2d09:20:550
1926706994334,6cyclictest2059163-21latency_hist11:10:022
1926706994334,6cyclictest1-21systemd11:05:022
1926705994331,8cyclictest261-21systemd-journal11:30:111
1926705994330,6cyclictest0-21swapper/109:45:491
192670599432,31cyclictest2109402-21ssh12:15:391
192670599432,31cyclictest2109402-21ssh12:15:391
1926704994336,4cyclictest2028073-21aten_r2power_vo10:30:290
1926704994335,5cyclictest2124206-21latency_hist12:35:020
1926704994335,5cyclictest1128531-21cups-browsed11:12:240
1926706994234,5cyclictest2128113-21munin-plugin-st12:40:012
1926706994234,5cyclictest1-21systemd11:50:012
1926706994232,4cyclictest0-21swapper/210:30:452
1926705994234,4cyclictest1958243-21kworker/u8:12-nfsiod12:15:011
1926705994233,5cyclictest1929743-21latency_hist07:20:021
1926705994228,10cyclictest1128531-21cups-browsed12:25:021
192670499423,34cyclictest2079156-21ssh11:35:380
192670499423,34cyclictest2079156-21ssh11:35:380
1926706994134,3cyclictest0-21swapper/210:15:402
1926706994133,4cyclictest1958243-21kworker/u8:12-nfsiod10:00:002
1926706994132,6cyclictest2066807-21ssh11:20:012
1926705994130,7cyclictest421-21dbus-daemon09:15:011
1926704994128,10cyclictest2112241-21wc12:20:020
1926704994128,10cyclictest2112241-21wc12:20:020
1926704994127,10cyclictest12350irq/32-eth011:20:380
1926704994118,20cyclictest2032719-21sh10:35:420
1926704994118,20cyclictest2032719-21sh10:35:420
1926706994033,5cyclictest1926703-21cyclictest09:20:552
1926706994033,4cyclictest2039448-21munin-plugin-st10:45:002
1926706994033,3cyclictest0-21swapper/210:15:012
1926706994031,7cyclictest1926709-21systemd-run07:10:022
1926706994031,6cyclictest1987056-21idleruntime-cro09:40:012
1926706994031,6cyclictest1951260-21fschecks_count08:30:332
1926705994032,3cyclictest0-21swapper/112:05:011
1926705994030,6cyclictest1950757-21systemd-run08:30:011
1926704994035,3cyclictest2111753-21kworker/u8:0+rpciod12:25:010
1926706993930,5cyclictest1970909-21cron09:20:012
192670699392,31cyclictest2020754-21rm10:20:432
192670599399,5cyclictest28-21ksoftirqd/111:45:401
1926705993930,6cyclictest421-21dbus-daemon11:40:021
1926705993930,6cyclictest421-21dbus-daemon11:40:021
1926705993930,6cyclictest2070611-21cron11:25:011
1926705993930,5cyclictest0-21swapper/111:10:451
1926704993930,6cyclictest1128531-21cups-browsed12:05:340
1926705993831,4cyclictest1987855-21fschecks_count09:40:311
1926705993831,3cyclictest0-21swapper/110:15:401
1926705993829,6cyclictest516-21in:imuxsock10:55:011
1926705993828,6cyclictest2036225-21fschecks_time10:40:301
1926705993828,6cyclictest0-21swapper/109:55:321
1926705993827,6cyclictest2040630-21ntp_kernel_pll_10:45:371
1926705993825,9cyclictest2063987-21needreboot11:15:361
1926704993812,21cyclictest2016767-21ssh10:15:450
192670699374,26cyclictest2060253-21ssh11:10:402
1926706993731,3cyclictest0-21swapper/212:15:292
1926706993731,3cyclictest0-21swapper/212:15:292
1926706993728,6cyclictest421-21dbus-daemon12:35:012
192670599373,28cyclictest1984396-21ssh09:35:461
1926705993729,5cyclictest2021299-21unixbench_singl10:20:521
1926705993729,5cyclictest1932697-21munin-run07:30:011
1926705993728,4cyclictest0-21swapper/112:30:381
1926705993726,7cyclictest1128531-21cups-browsed09:26:221
1926705993724,9cyclictest2007806-21aten_r2power_cu10:05:261
192670699362,28cyclictest2032624-21ssh10:35:392
192670699362,28cyclictest2032624-21ssh10:35:392
1926705993622,10cyclictest1955911-21latency08:45:361
1926705993613,5cyclictest28-21ksoftirqd/109:20:471
1926704993632,2cyclictest0-21swapper/009:20:010
1926704993629,3cyclictest0-21swapper/009:55:430
1926704993628,5cyclictest1128531-21cups-browsed10:12:240
1926706993527,5cyclictest1963391-21aten_r2power_vo09:10:302
1926706993526,6cyclictest1946251-21true08:15:012
1926706993526,5cyclictest1051-21mta-sts-daemon10:55:312
1926705993527,5cyclictest1968050-21ssh09:15:401
1926705993527,5cyclictest0-21swapper/111:45:011
192670599352,25cyclictest1980137-21ssh09:30:381
1926704993529,4cyclictest1996070-21/usr/sbin/munin09:50:350
1926704993529,4cyclictest1996070-21/usr/sbin/munin09:50:350
1926704993528,5cyclictest1992979-21unixbench-2d09:46:000
1926704993527,5cyclictest2045184-21unixbench_multi10:50:500
1926704993526,7cyclictest2082630-21needreboot11:40:350
1926704993525,8cyclictest2040924-21mailstats10:45:460
192670699347,20cyclictest2075562-21ssh11:30:382
1926706993427,3cyclictest0-21swapper/212:10:442
1926706993426,5cyclictest1947737-21munin-run08:20:012
192670599344,22cyclictest2090081-21ssh11:50:421
1926705993426,5cyclictest1940227-21cron07:55:001
1926705993426,5cyclictest1914062-21kworker/u8:14+xprtiod07:25:011
1926705993426,5cyclictest0-21swapper/109:10:001
1926705993425,6cyclictest2096357-21cron12:00:011
1926705993425,5cyclictest1958228-21munin-plugin-st08:55:011
1926705993425,4cyclictest0-21swapper/110:35:511
1926705993425,4cyclictest0-21swapper/110:35:511
1926704993428,4cyclictest2056162-21kernelversion11:05:380
1926704993427,4cyclictest1943267-21systemd-run08:05:010
1926704993427,4cyclictest1056-21mta-sts-daemon11:53:520
1926704993423,8cyclictest1-21systemd10:20:270
1926706993327,3cyclictest0-21swapper/209:40:302
1926706993327,3cyclictest0-21swapper/209:25:372
1926706993326,4cyclictest1944852-21latency_hist08:10:022
1926706993326,4cyclictest1944852-21latency_hist08:10:022
1926706993326,3cyclictest0-21swapper/211:25:422
1926706993322,8cyclictest1933507-21needreboot07:30:372
192670599338,12cyclictest28-21ksoftirqd/111:25:421
1926705993325,4cyclictest0-21swapper/110:00:401
1926705993324,5cyclictest2015444-21ssh10:15:011
1926705993322,8cyclictest2055267-21/usr/sbin/munin11:05:021
1926705993322,8cyclictest1927537-21ntp_kernel_err07:10:371
1926705993317,12cyclictest271ktimers/108:15:441
1926704993328,3cyclictest0-21swapper/008:10:010
1926704993328,3cyclictest0-21swapper/008:10:010
1926704993326,5cyclictest2064093-21ntp_kernel_pll_11:15:390
1926704993325,5cyclictest421-21dbus-daemon10:57:440
1926706993227,2cyclictest0-21swapper/212:10:022
1926706993224,6cyclictest1943245-21latency_hist08:05:022
1926706993224,5cyclictest1935621-21unixbench_singl07:35:492
1926706993224,5cyclictest1928708-21fschecks_count07:15:322
1926706993223,6cyclictest261-21systemd-journal12:30:002
1926706993223,6cyclictest0-21swapper/211:40:492
192670699322,23cyclictest1996574-21ssh09:50:462
192670699322,23cyclictest1996574-21ssh09:50:462
1926705993224,5cyclictest2125153-21missed_timers12:35:351
1926705993224,4cyclictest0-21swapper/108:35:431
1926705993210,5cyclictest28-21ksoftirqd/112:05:411
1926704993227,3cyclictest2120143-21cron12:30:000
1926704993226,3cyclictest0-21swapper/011:30:450
1926704993224,5cyclictest624-21lldpd09:26:010
1926706993125,3cyclictest0-21swapper/209:00:012
1926706993124,4cyclictest421-21dbus-daemon09:10:012
1926706993124,3cyclictest0-21swapper/209:45:512
1926706993123,5cyclictest2047375-21cron10:55:002
192670599317,4cyclictest28-21ksoftirqd/112:29:061
192670599315,20cyclictest1996608-21ps09:50:471
192670599315,20cyclictest1996608-21ps09:50:471
1926705993125,2cyclictest0-21swapper/110:25:451
1926705993119,8cyclictest2056177-21cut11:05:381
1926705993113,6cyclictest1926716-21munin-run07:10:021
1926705993111,4cyclictest28-21ksoftirqd/107:35:371
1926705993110,5cyclictest28-21ksoftirqd/108:40:341
1926705993110,5cyclictest28-21ksoftirqd/107:30:421
192670499314,21cyclictest1980122-21ssh09:30:380
1926704993126,3cyclictest1926703-21cyclictest12:35:150
1926704993125,4cyclictest2008100-21ssh10:05:300
1926704993124,5cyclictest2023886-21/usr/sbin/munin10:25:470
1926704993124,4cyclictest1987718-21aten2.4-expect09:40:290
1926704993122,6cyclictest1128531-21cups-browsed09:10:120
1926704993113,15cyclictest421-21dbus-daemon11:00:290
1926706993024,3cyclictest0-21swapper/207:40:412
1926706993022,5cyclictest1958121-21unixbench_multi08:50:522
1926706993022,5cyclictest1941285-21processes07:55:442
192670699302,22cyclictest2089934-21ssh11:50:372
1926705993022,5cyclictest2051313-21munin-run11:00:011
1926705993019,8cyclictest1951637-21ntp_kernel_pll_08:30:411
1926704993024,4cyclictest2037193-21grep10:40:490
1926704993022,6cyclictest1949474-21kworker/u8:8-xprtiod08:25:490
1926706992922,4cyclictest1940256-21systemd-run07:55:012
1926706992921,4cyclictest1938767-21systemd07:50:022
192670599294,17cyclictest428-21mta-sts-daemon08:55:401
1926705992921,5cyclictest0-21swapper/108:00:401
192670499293,18cyclictest1984190-21ssh09:35:390
1926704992923,4cyclictest1947727-21cron08:20:010
1926704992921,4cyclictest0-21swapper/008:20:360
1926704992920,6cyclictest1128531-21cups-browsed07:15:320
1926706992822,3cyclictest0-21swapper/212:20:372
1926706992822,3cyclictest0-21swapper/210:25:492
1926706992821,5cyclictest0-21swapper/208:45:512
1926706992820,5cyclictest1931225-21systemd-run07:25:022
192670699281,21cyclictest1980175-21ssh09:30:402
192670599287,6cyclictest28-21ksoftirqd/107:48:121
192670599286,5cyclictest28-21ksoftirqd/108:20:341
192670599286,4cyclictest28-21ksoftirqd/108:05:441
192670599286,4cyclictest28-21ksoftirqd/108:05:441
192670499283,11cyclictest1962738-21diskmemload11:28:190
192670499282,21cyclictest659-21snmpd07:30:390
1926704992818,7cyclictest1960637-21ntp_kernel_pll_09:00:400
1926704992815,9cyclictest151ktimers/007:45:400
192670599274,8cyclictest28-21ksoftirqd/109:05:011
1926704992721,4cyclictest2086897-21grep11:45:500
1926704992718,6cyclictest151ktimers/007:35:420
1926706992618,5cyclictest1932709-21systemd-run07:30:012
1926705992618,5cyclictest1941731-21cron08:00:001
1926705992614,3cyclictest171rcu_preempt07:40:371
192670499262,19cyclictest659-21snmpd08:10:370
1926704992621,3cyclictest1055-21mta-sts-daemon08:57:210
1926704992621,2cyclictest0-21swapper/008:48:120
1926704992620,4cyclictest1962649-21unixbench_singl09:05:510
1926704992620,3cyclictest0-21swapper/008:40:350
1926706992520,2cyclictest0-21swapper/208:35:492
1926706992519,3cyclictest0-21swapper/209:05:002
1926706992518,3cyclictest0-21swapper/208:20:422
192670599256,4cyclictest28-21ksoftirqd/108:10:421
1926704992518,4cyclictest1936521-21needreboot07:40:360
1926704992517,5cyclictest1940256-21systemd-run07:55:010
1926704992414,8cyclictest261-21systemd-journal08:54:110
1926704992414,7cyclictest0-21swapper/008:40:010
1926704992410,10cyclictest1932694-21munin-plugin-st07:30:010
1926706992316,4cyclictest1950622-21unixbench_multi08:25:492
1926706992315,5cyclictest1928178-21cron07:15:002
1926704992316,4cyclictest1927221-21fschecks_time07:10:310
1926706992217,2cyclictest0-21swapper/208:40:372
192670499222,14cyclictest1941114-21ntpq07:55:370
192670499212,14cyclictest1951755-21ps08:30:460
192670499211,9cyclictest1930477-21missed_timers07:20:370
1926704992110,8cyclictest421-21dbus-daemon07:10:020
192670799204,9cyclictest0-21swapper/311:15:423
192670799203,9cyclictest0-21swapper/311:25:423
192670799203,9cyclictest0-21swapper/310:45:373
192670799203,10cyclictest0-21swapper/310:40:393
192670799193,9cyclictest0-21swapper/310:00:433
192670799192,8cyclictest0-21swapper/309:40:363
192670799192,10cyclictest0-21swapper/311:10:393
192670799184,9cyclictest0-21swapper/312:00:363
192670799183,9cyclictest0-21swapper/310:10:383
192670799183,8cyclictest0-21swapper/312:05:443
192670799183,8cyclictest0-21swapper/311:45:403
192670799183,11cyclictest0-21swapper/309:10:413
192670799183,10cyclictest0-21swapper/309:15:413
192670799182,9cyclictest0-21swapper/312:30:383
192670799182,10cyclictest0-21swapper/311:40:393
192670799182,10cyclictest0-21swapper/309:50:403
192670799182,10cyclictest0-21swapper/309:50:403
192670799182,10cyclictest0-21swapper/309:45:493
192670799182,10cyclictest0-21swapper/309:25:433
192670799173,9cyclictest0-21swapper/310:15:443
192670799173,8cyclictest0-21swapper/310:35:403
192670799173,8cyclictest0-21swapper/310:35:403
192670799172,9cyclictest0-21swapper/312:15:443
192670799172,9cyclictest0-21swapper/312:15:443
192670799172,9cyclictest0-21swapper/311:50:393
192670799172,9cyclictest0-21swapper/311:20:383
192670799172,9cyclictest0-21swapper/310:05:353
192670799172,7cyclictest0-21swapper/308:40:393
192670799171,9cyclictest0-21swapper/310:20:383
192670799165,7cyclictest0-21swapper/310:50:393
192670799165,7cyclictest0-21swapper/309:55:403
192670799164,7cyclictest0-21swapper/309:20:373
192670799164,7cyclictest0-21swapper/308:50:453
192670799163,9cyclictest0-21swapper/312:25:383
192670799163,6cyclictest0-21swapper/307:48:133
192670799162,9cyclictest0-21swapper/312:35:373
192670799162,9cyclictest0-21swapper/311:05:483
192670799162,9cyclictest0-21swapper/310:30:373
192670799162,7cyclictest0-21swapper/308:45:383
192670799161,9cyclictest0-21swapper/311:00:343
192670799161,8cyclictest0-21swapper/308:05:423
192670799161,8cyclictest0-21swapper/308:05:423
192670799161,10cyclictest0-21swapper/312:20:383
192670799155,8cyclictest0-21swapper/311:35:413
192670799155,8cyclictest0-21swapper/311:35:413
192670799154,8cyclictest0-21swapper/310:25:363
192670799154,7cyclictest0-21swapper/311:30:413
192670799152,9cyclictest0-21swapper/309:35:293
192670799152,9cyclictest0-21swapper/309:30:273
192670799152,8cyclictest0-21swapper/312:10:343
192670799152,7cyclictest0-21swapper/308:35:413
192670799152,7cyclictest0-21swapper/307:35:373
192670799151,7cyclictest0-21swapper/308:15:383
192670799143,7cyclictest0-21swapper/311:55:273
192670799143,6cyclictest0-21swapper/307:50:423
192670799142,8cyclictest0-21swapper/309:10:013
192670799142,6cyclictest0-21swapper/308:10:373
192670799141,7cyclictest0-21swapper/307:40:373
192670799141,7cyclictest0-21swapper/307:25:393
192670799141,7cyclictest0-21swapper/307:20:023
192670799132,5cyclictest0-21swapper/310:55:113
192670799132,5cyclictest0-21swapper/308:30:383
192670799131,6cyclictest0-21swapper/308:20:353
192670799131,6cyclictest0-21swapper/308:05:003
192670799121,7cyclictest0-21swapper/308:55:443
192670799121,7cyclictest0-21swapper/307:20:293
192670799121,7cyclictest0-21swapper/307:10:283
192670799121,6cyclictest0-21swapper/307:30:383
192670799112,5cyclictest0-21swapper/309:00:373
192670799111,6cyclictest0-21swapper/308:25:283
192670799111,6cyclictest0-21swapper/307:55:323
192670799102,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional