You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-23 - 12:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Tue Jun 23, 2026 00:43:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1919995995445,6cyclictest2060848-21munin-run00:00:011
1919995995241,7cyclictest1-21systemd23:55:021
1919995995134,12cyclictest1003-21mta-sts-daemon22:30:341
1919995995134,12cyclictest1003-21mta-sts-daemon22:30:341
1919994995143,5cyclictest1999265-21systemd-run22:20:020
1919995994839,5cyclictest1-21systemd22:55:011
1919994994838,7cyclictest2021169-21aten_r2power_cu22:55:200
1919995994634,5cyclictest0-21swapper/123:00:331
1919995994615,13cyclictest1991159-21ntpq22:05:381
1919996994538,3cyclictest0-21swapper/221:45:352
191999599453,29cyclictest2034140-21ssh23:15:301
1919995994526,8cyclictest1978752-21ssh21:45:291
1919994994432,8cyclictest1991463-21timerandwakeup22:05:430
1919996994335,5cyclictest1945469-21cron20:35:012
1919996994335,5cyclictest1937997-21systemd-run20:10:012
1919996994334,6cyclictest2048580-21unin-run23:40:022
191999699433,32cyclictest1984833-21ssh21:55:282
1919996994233,6cyclictest420-21avahi-daemon00:20:192
1919995994231,4cyclictest0-21swapper/100:10:361
1919995994227,9cyclictest1929910-21ntpq19:40:301
1919995994227,9cyclictest1929910-21ntpq19:40:301
1919995994214,9cyclictest1975637-21ntpq21:40:301
1919995994212,9cyclictest1920851-21ntpq19:10:291
1919995994211,10cyclictest2082479-21munin-run00:35:011
1919994994233,7cyclictest1921491-21sh19:15:010
1919996994133,4cyclictest0-21swapper/221:20:432
1919995994135,3cyclictest0-21swapper/120:20:011
1919995994134,3cyclictest0-21swapper/121:30:231
1919995994131,6cyclictest0-21swapper/120:00:011
1919995994130,7cyclictest2023867-21idleruntime-cro23:00:011
1919995994117,9cyclictest1939480-21idleruntime-cro20:15:011
1919995994115,8cyclictest1950996-21ps20:50:351
1919995994115,6cyclictest2015743-21ntpq22:45:331
191999499412,33cyclictest2046566-21ssh23:35:330
1919996994032,5cyclictest1-21systemd23:20:012
191999599403,29cyclictest1955985-21diskmemload21:35:301
1919995994030,5cyclictest0-21swapper/123:15:011
1919995994024,10cyclictest1941939-21ntpq20:20:351
1919995994015,9cyclictest1966153-21missed_timers21:25:261
1919995994015,7cyclictest1953868-21ntpq21:00:301
1919995994013,9cyclictest2083689-21processes00:35:371
1919995994013,9cyclictest1984323-21aten_r2power_vo21:55:191
1919995994013,12cyclictest1962622-21aten_r2power_fr21:20:201
191999499400,35cyclictest1984962-21ssh21:55:320
1919996993931,5cyclictest2085670-21latency_hist00:40:022
1919996993930,6cyclictest420-21avahi-daemon23:56:082
191999699392,5cyclictest171rcu_preempt21:25:292
1919995993930,4cyclictest0-21swapper/121:15:241
1919995993929,6cyclictest439-21systemd-logind19:25:021
1919995993928,8cyclictest1935795-21needreboot20:00:291
1919995993913,8cyclictest1958487-21sh21:13:121
1919995993913,8cyclictest1946356-21ntpq20:35:281
191999499394,25cyclictest151ktimers/019:35:320
1919994993931,5cyclictest2080208-21latency00:30:260
1919994993931,5cyclictest1-21systemd21:25:010
1919994993916,20cyclictest12750irq/33-eth021:10:290
191999499391,15cyclictest1959121-21kworker/u8:7+rpciod21:45:330
1919996993831,4cyclictest1959097-21munin-plugin-st21:15:012
1919996993831,3cyclictest0-21swapper/222:30:002
1919996993830,5cyclictest163119-21cups-browsed21:32:402
191999599389,11cyclictest1938096-21latency_hist20:10:021
1919995993813,9cyclictest2079575-21grep00:30:021
1919995993813,9cyclictest1952178-21sed20:55:271
1919995993813,14cyclictest2040410-21ssh23:25:331
1919994993832,4cyclictest2060868-21awk00:00:010
1919996993730,3cyclictest0-21swapper/219:40:332
1919996993730,3cyclictest0-21swapper/219:40:332
1919996993729,5cyclictest1959741-21cut21:15:232
1919996993729,5cyclictest0-21swapper/220:50:012
1919996993729,3cyclictest0-21swapper/223:25:202
1919996993728,6cyclictest1990634-21fschecks_count22:05:252
1919996993725,8cyclictest2052537-21missed_timers23:45:262
1919995993729,4cyclictest0-21swapper/122:15:031
1919995993727,6cyclictest163119-21cups-browsed23:42:121
1919995993725,7cyclictest2027369-21aten_r2power_fr23:05:201
1919995993723,9cyclictest1944956-21ntpq20:30:351
1919995993721,12cyclictest2071211-21ntpq00:15:331
1919995993720,12cyclictest2012574-21ntpq22:40:291
1919995993713,8cyclictest1955371-21ntpq21:05:311
1919995993710,11cyclictest2003014-21awk22:25:211
1919996993630,3cyclictest0-21swapper/223:35:012
1919996993628,5cyclictest2023870-21idleruntime-cro23:00:012
1919996993626,8cyclictest1940283-21needreboot20:15:272
1919995993630,3cyclictest0-21swapper/119:55:021
1919995993625,7cyclictest265-21systemd-journal22:00:031
1919995993622,8cyclictest1931405-21ntpq19:45:311
1919995993621,12cyclictest1925376-21ntpq19:25:281
1919995993615,8cyclictest2065553-21sed00:05:421
1919995993613,7cyclictest2051880-21ssh23:45:061
1919995993611,8cyclictest2062774-21ssh00:01:201
1919995993611,13cyclictest2039289-21latency_hist23:25:031
1919994993631,3cyclictest2008468-21ssh22:35:000
1919994993631,3cyclictest2008468-21ssh22:35:000
1919994993630,3cyclictest0-21swapper/000:15:320
1919994993629,4cyclictest1943021-21fschecks_time20:25:240
1919994993628,5cyclictest163119-21cups-browsed22:35:340
1919996993529,3cyclictest0-21swapper/221:00:012
1919996993526,5cyclictest0-21swapper/200:00:302
1919995993524,5cyclictest0-21swapper/121:50:211
1919995993522,8cyclictest1922347-21ntpq19:15:321
1919995993514,7cyclictest2011022-21sh22:38:231
1919994993510,19cyclictest163116-21cupsd21:05:300
191999699343,23cyclictest2015732-21ssh22:45:322
1919996993428,3cyclictest0-21swapper/200:15:012
1919996993427,4cyclictest1931978-21sh19:50:002
1919996993427,3cyclictest0-21swapper/221:40:362
1919996993426,5cyclictest265-21systemd-journal23:05:012
1919996993426,5cyclictest2082491-21latency_hist00:35:022
1919996993426,5cyclictest2014687-21latency_hist22:45:012
1919996993426,5cyclictest2008470-21cron22:35:002
1919996993426,5cyclictest2008470-21cron22:35:002
1919996993426,5cyclictest1933873-21aten_r2power_vo19:55:222
1919996993421,3cyclictest2037797-21tail23:20:422
191999599343,23cyclictest1997132-21ssh22:15:291
1919995993427,3cyclictest2074271-21ntpq00:20:311
1919995993426,3cyclictest1943415-21ntpq20:25:331
1919995993423,7cyclictest1928323-21ntpq19:35:281
1919995993410,8cyclictest1926894-21ntpq19:30:301
191999499342,25cyclictest1981784-21ssh21:50:300
1919994993421,5cyclictest163119-21cups-browsed23:55:020
1919996993326,4cyclictest1942481-21cron20:25:012
1919996993324,7cyclictest2077319-21ntp_kernel_pll_00:25:312
191999599337,9cyclictest2042608-21ssh23:30:051
1919995993318,11cyclictest1947872-21sshd20:40:291
191999499332,8cyclictest171rcu_preempt19:45:340
1919994993327,4cyclictest1922972-21cron19:20:010
1919994993326,4cyclictest2036203-21awk23:20:010
1919996993226,4cyclictest1919993-21cyclictest23:50:412
1919996993226,4cyclictest1002-21mta-sts-daemon23:11:502
1919996993226,4cyclictest0-21swapper/222:35:202
1919996993226,3cyclictest0-21swapper/223:05:302
1919996993223,6cyclictest421-21dbus-daemon19:40:012
1919995993224,3cyclictest2000319-21ntpq22:20:321
1919995993221,8cyclictest1949459-21ntpq20:45:351
1919995993220,8cyclictest2046594-21ntp_states23:35:351
1919994993225,5cyclictest1934958-21cron20:00:010
1919994993225,4cyclictest2027121-21latency_hist23:05:020
1919994993224,6cyclictest1960291-21proc_pri21:15:360
1919996993123,5cyclictest2064468-21aten_r2power_vo00:05:202
1919996993123,5cyclictest1943869-21unixbench_multi20:25:422
1919996993122,7cyclictest694-21cron21:55:012
191999699311,4cyclictest171rcu_preempt21:00:302
191999499318,16cyclictest12650irq/32-eth000:20:310
1919994993125,4cyclictest421-21dbus-daemon23:10:010
1919994993125,4cyclictest2048915-21aten_r2power_cu23:40:200
1919994993125,4cyclictest1972133-21fschecks_time21:35:240
1919994993125,4cyclictest1935500-21fschecks_count20:00:240
1919994993125,4cyclictest1000-21mta-sts-daemon22:13:320
1919994993125,3cyclictest0-21swapper/000:35:210
1919994993124,4cyclictest2002934-21fschecks_count22:25:200
1919994993124,4cyclictest1939401-21unixbench_singl20:10:420
1919994993124,4cyclictest1002-21mta-sts-daemon00:11:260
1919994993122,5cyclictest1957926-21kworker/u8:1-rpciod22:40:300
1919994993113,15cyclictest0-21swapper/019:30:300
191999499311,25cyclictest2043486-21ssh23:30:340
191999699307,5cyclictest171rcu_preempt19:30:302
191999699304,4cyclictest171rcu_preempt22:10:332
1919996993025,2cyclictest0-21swapper/200:15:442
1919996993024,3cyclictest1959123-21kworker/u8:9+nfsiod22:05:012
1919996993022,5cyclictest2000836-21sed22:20:432
1919996993021,6cyclictest601-21gmain20:45:032
1919994993024,4cyclictest1951375-21unixbench_multi20:50:400
1919994993023,4cyclictest1942507-21systemd-run20:25:010
1919994993022,6cyclictest0-21swapper/000:05:310
1919994993022,5cyclictest439-21systemd-logind19:30:020
1919996992924,2cyclictest0-21swapper/222:50:402
1919996992924,2cyclictest0-21swapper/222:15:312
1919996992922,4cyclictest1974648-21latency_hist21:40:022
1919996992922,3cyclictest0-21swapper/223:40:352
1919995992919,7cyclictest1920006-21munin-run19:10:021
191999499299,16cyclictest1969438-21sshd21:30:300
1919994992923,3cyclictest0-21swapper/021:40:330
1919994992922,5cyclictest1945494-21systemd-run20:35:020
1919994992922,4cyclictest2031073-21ntp_kernel_pll_23:10:300
1919994992922,4cyclictest1990193-21latency_hist22:05:030
1919994992920,4cyclictest0-21swapper/020:45:320
1919994992914,11cyclictest12650irq/32-eth023:25:300
1919996992820,5cyclictest163119-21cups-browsed19:21:382
1919996992818,7cyclictest1002-21mta-sts-daemon20:50:142
1919994992823,3cyclictest1923518-21fschecks_count19:20:220
1919996992719,5cyclictest1946860-21tail20:35:402
1919996992719,5cyclictest1002-21mta-sts-daemon20:00:322
1919996992717,7cyclictest1922241-21missed_timers19:15:292
191999499275,5cyclictest171rcu_preempt21:25:300
1919994992722,3cyclictest2037161-21needreboot23:20:280
191999499272,20cyclictest665-21snmpd20:55:330
1919994992717,7cyclictest0-21swapper/020:40:010
1919994992713,10cyclictest151ktimers/000:25:320
191999499271,22cyclictest2052788-21ntpq23:45:350
191999499270,4cyclictest171rcu_preempt22:50:410
1919996992618,5cyclictest1001-21mta-sts-daemon21:05:352
191999499262,18cyclictest2000374-21ssh22:20:340
1919994992621,3cyclictest1933413-21head19:50:410
1919994992620,4cyclictest1937887-21unixbench_multi20:05:410
1919996992520,2cyclictest0-21swapper/219:55:012
191999499258,12cyclictest151ktimers/021:00:290
1919994992518,5cyclictest2015451-21latency22:45:260
1919994992518,5cyclictest1943954-21kworker/u8:3-rpciod20:45:020
1919996992416,6cyclictest1921309-21unixbench-2d19:10:402
1919994992419,2cyclictest0-21swapper/019:10:020
1919994992416,5cyclictest420-21avahi-daemon00:00:110
1919996992319,2cyclictest0-21swapper/219:25:332
1919996992316,3cyclictest0-21swapper/220:15:012
191999499236,10cyclictest428-21mta-sts-daemon19:40:330
191999499236,10cyclictest428-21mta-sts-daemon19:40:330
1919994992318,2cyclictest0-21swapper/020:20:010
191999799194,8cyclictest0-21swapper/300:05:363
191999799194,7cyclictest0-21swapper/300:15:343
191999799194,10cyclictest0-21swapper/323:50:303
191999799193,9cyclictest0-21swapper/300:30:353
191999799192,8cyclictest0-21swapper/323:00:323
191999799185,9cyclictest0-21swapper/300:35:323
191999799182,9cyclictest0-21swapper/322:15:303
191999799182,9cyclictest0-21swapper/300:10:333
191999799182,7cyclictest0-21swapper/320:40:323
191999799182,10cyclictest0-21swapper/321:10:313
191999799174,7cyclictest0-21swapper/300:20:303
191999799173,9cyclictest0-21swapper/300:25:313
191999799173,8cyclictest0-21swapper/321:40:303
191999799172,9cyclictest0-21swapper/323:45:313
191999799172,9cyclictest0-21swapper/321:50:303
191999799171,8cyclictest0-21swapper/323:05:303
191999799164,7cyclictest0-21swapper/321:20:323
191999799164,6cyclictest0-21swapper/322:50:293
191999799163,8cyclictest0-21swapper/323:30:293
191999799163,8cyclictest0-21swapper/322:35:343
191999799163,7cyclictest0-21swapper/323:15:323
191999799163,7cyclictest0-21swapper/322:30:373
191999799163,7cyclictest0-21swapper/322:30:373
191999799162,9cyclictest0-21swapper/323:10:303
191999799162,8cyclictest0-21swapper/323:40:303
191999799162,8cyclictest0-21swapper/322:40:283
191999799162,7cyclictest0-21swapper/320:25:333
191999799161,8cyclictest0-21swapper/319:20:353
191999799161,10cyclictest0-21swapper/300:00:363
191999799155,6cyclictest0-21swapper/323:55:283
191999799154,7cyclictest0-21swapper/323:35:303
191999799154,6cyclictest0-21swapper/321:15:283
191999799153,8cyclictest0-21swapper/321:25:283
191999799153,7cyclictest0-21swapper/323:25:363
191999799153,7cyclictest0-21swapper/322:25:283
191999799153,7cyclictest0-21swapper/322:00:363
191999799152,9cyclictest0-21swapper/323:20:423
191999799152,9cyclictest0-21swapper/322:55:393
191999799152,8cyclictest0-21swapper/321:55:273
191999799152,8cyclictest0-21swapper/321:45:283
191999799152,7cyclictest0-21swapper/322:20:193
191999799152,7cyclictest0-21swapper/321:05:313
191999799152,6cyclictest0-21swapper/319:35:283
191999799151,8cyclictest0-21swapper/322:45:313
191999799151,7cyclictest0-21swapper/319:10:313
191999699150,12cyclictest1919950-21/usr/sbin/munin19:10:012
191999799145,6cyclictest0-21swapper/319:15:383
191999799144,4cyclictest0-21swapper/319:50:283
191999799142,8cyclictest0-21swapper/322:10:303
191999799142,7cyclictest0-21swapper/321:35:243
191999799141,7cyclictest0-21swapper/320:10:293
191999799141,7cyclictest0-21swapper/319:25:263
191999799141,6cyclictest0-21swapper/320:20:323
191999799132,6cyclictest0-21swapper/320:20:013
191999799132,5cyclictest0-21swapper/321:00:293
191999799131,8cyclictest0-21swapper/322:05:393
191999799131,8cyclictest0-21swapper/321:30:303
191999799131,7cyclictest0-21swapper/319:45:303
191999799131,6cyclictest0-21swapper/320:30:283
191999799131,6cyclictest0-21swapper/320:05:293
191999799124,6cyclictest0-21swapper/320:55:293
191999799123,7cyclictest0-21swapper/320:50:293
191999799122,6cyclictest0-21swapper/320:00:313
191999799121,8cyclictest0-21swapper/319:40:293
191999799121,8cyclictest0-21swapper/319:40:293
191999799121,6cyclictest0-21swapper/319:55:313
191999799112,5cyclictest0-21swapper/319:30:293
191999799111,5cyclictest0-21swapper/320:35:223
191999799101,5cyclictest0-21swapper/320:45:283
19199979981,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional