You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-02 - 04:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Thu Jul 02, 2026 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
210390099533,39cyclictest2164927-21/usr/sbin/munin21:50:240
2103902995242,4cyclictest0-21swapper/220:20:382
210390299482,40cyclictest2164934-21ssh21:50:242
2103901994823,18cyclictest265-21systemd-journal19:52:221
2103902994637,4cyclictest0-21swapper/222:20:342
2103901994635,8cyclictest2214773-21proc_pri23:10:451
210390199461,34cyclictest2220728-21ssh23:20:361
2103900994532,10cyclictest2136884-21munin-plugin-st21:00:010
2103902994437,3cyclictest0-21swapper/220:40:452
2103902994333,7cyclictest2256323-21idleruntime-cro00:20:002
2103902994333,6cyclictest2149840-21/usr/sbin/munin21:25:282
2103902994234,5cyclictest2114422-21systemd-run19:45:022
2103901994235,4cyclictest2146061-21awk21:20:011
2103901994235,4cyclictest2105394-21munin-plugin-st19:15:001
2103901994232,7cyclictest2149183-21systemd-run21:25:021
2103901994232,7cyclictest2149183-21systemd-run21:25:021
2103902994132,6cyclictest2208203-21if_wlan023:00:312
2103902994132,6cyclictest2120515-21latency_hist20:05:022
2103902994125,12cyclictest2162076-21cpu21:45:252
2103901994133,5cyclictest2158850-21aten_r2power_en21:40:241
2103901994133,5cyclictest2134037-21latency_hist20:50:031
210390199412,30cyclictest2217651-21ssh23:15:341
210390199411,37cyclictest2152026-21kworker/u8:3+xprtiod21:35:001
210390199411,35cyclictest2168696-21ssh21:55:331
2103901994025,3cyclictest171rcu_preempt22:40:371
210390099404,29cyclictest567-21gdbus20:10:370
2103902993930,4cyclictest0-21swapper/200:25:372
2103901993933,3cyclictest0-21swapper/123:55:011
2103901993933,3cyclictest0-21swapper/122:15:371
2103901993932,4cyclictest2233287-21grep23:40:431
2103901993931,4cyclictest0-21swapper/100:10:451
2103901993931,4cyclictest0-21swapper/100:10:451
2103900993932,4cyclictest2105410-21systemd-run19:15:010
2103902993828,7cyclictest1002-21mta-sts-daemon23:10:202
2103902993828,6cyclictest371393-21cups-browsed23:42:192
2103901993830,5cyclictest2126425-21systemd-run20:25:011
2103901993829,6cyclictest443-21gpm22:50:011
2103901993829,6cyclictest2244119-21ssh00:00:011
2103900993823,12cyclictest2195328-21latency_hist22:40:020
2103900993823,12cyclictest2195328-21latency_hist22:40:020
2103902993730,4cyclictest2161500-21cron21:45:012
2103902993730,4cyclictest1000-21mta-sts-daemon22:25:402
2103902993729,5cyclictest2182969-21ssh22:20:012
2103902993724,7cyclictest0-21swapper/222:45:432
2103901993728,6cyclictest2190180-21ssh22:30:351
2103901993728,6cyclictest2190180-21ssh22:30:351
2103901993723,3cyclictest171rcu_preempt19:30:371
2103900993732,3cyclictest2213663-21latency_hist23:10:020
2103900993732,3cyclictest2213663-21latency_hist23:10:020
2103900993731,3cyclictest0-21swapper/022:05:330
2103900993729,5cyclictest2160020-21unixbench_multi21:40:450
210390299366,23cyclictest2229866-21ssh23:35:352
2103902993629,3cyclictest0-21swapper/222:00:352
210390199363,26cyclictest2263632-21ntpq00:30:391
2103901993629,4cyclictest2138303-21unixbench_singl21:00:451
2103901993627,6cyclictest2112977-21latency_hist19:40:021
2103901993627,5cyclictest2236087-21/usr/sbin/munin23:45:371
2103901993627,4cyclictest0-21swapper/121:30:001
2103901993622,3cyclictest171rcu_preempt21:35:391
210390199361,26cyclictest2193289-21ssh22:35:381
210390199361,26cyclictest2193289-21ssh22:35:381
2103900993626,7cyclictest2173287-21sshd22:03:220
2103900993622,11cyclictest2208520-21ntp_kernel_pll_23:00:380
2103900993620,9cyclictest12650irq/32-eth023:10:350
2103900993610,4cyclictest171rcu_preempt21:10:340
2103902993529,3cyclictest0-21swapper/221:35:002
2103902993527,5cyclictest2257121-21kernelversion00:20:302
2103902993527,5cyclictest2244126-40logrotate00:00:022
210390299350,4cyclictest171rcu_preempt19:55:372
2103901993527,5cyclictest2167659-21cron21:55:011
2103901993524,6cyclictest420-21avahi-daemon23:27:261
2103901993524,6cyclictest265-21systemd-journal19:20:011
2103900993528,4cyclictest2149821-21ssh21:25:280
2103900993527,6cyclictest2257971-21unixbench_multi00:20:480
2103900993526,6cyclictest1001-21mta-sts-daemon23:57:340
2103902993426,5cyclictest2106942-21latency_hist19:20:022
2103902993426,4cyclictest2192809-21sh22:35:272
2103902993426,4cyclictest2192809-21sh22:35:272
2103902993426,4cyclictest1003-21mta-sts-daemon22:41:212
2103902993426,4cyclictest0-21swapper/222:50:402
2103902993423,8cyclictest2175200-21switchtime22:05:462
210390199348,14cyclictest428-21mta-sts-daemon20:35:331
2103901993429,2cyclictest0-21swapper/122:15:011
2103900993428,4cyclictest2226418-21fschecks_count23:30:270
2103900993428,2cyclictest0-21swapper/021:55:330
210390099341,26cyclictest2229966-21rm23:35:380
2103902993327,3cyclictest0-21swapper/221:35:322
2103902993325,5cyclictest1001-21mta-sts-daemon00:11:312
2103902993325,5cyclictest1001-21mta-sts-daemon00:11:312
2103902993325,4cyclictest0-21swapper/221:15:352
210390299332,24cyclictest2248334-21ssh00:05:442
210390299332,24cyclictest2248334-21ssh00:05:442
210390199338,8cyclictest2115778-21unixbench_multi19:45:491
2103901993325,5cyclictest2256307-21cron00:20:001
2103901993320,3cyclictest171rcu_preempt00:20:361
2103901993312,5cyclictest28-21ksoftirqd/120:50:361
210390199330,6cyclictest28-21ksoftirqd/120:15:371
2103900993327,4cyclictest2256341-21ssh00:20:010
2103900993326,4cyclictest1003-21mta-sts-daemon22:14:250
2103900993326,4cyclictest0-21swapper/022:45:360
2103900993312,17cyclictest2219680-21kworker/u8:7+rpciod00:05:420
2103900993312,17cyclictest2219680-21kworker/u8:7+rpciod00:05:420
2103902993227,2cyclictest0-21swapper/221:00:022
2103902993226,3cyclictest0-21swapper/223:20:392
2103902993225,4cyclictest2266633-21ntp_offset00:35:372
2103902993225,3cyclictest0-21swapper/223:15:322
2103902993221,7cyclictest420-21avahi-daemon21:24:162
2103902993221,7cyclictest420-21avahi-daemon21:24:162
210390199325,5cyclictest28-21ksoftirqd/119:40:341
2103901993227,3cyclictest0-21swapper/123:00:311
2103901993226,2cyclictest0-21swapper/100:40:011
2103901993225,3cyclictest0-21swapper/123:37:111
2103901993224,3cyclictest0-21swapper/100:25:401
210390199322,22cyclictest2187189-21ssh22:25:391
2103901993220,2cyclictest171rcu_preempt20:10:261
2103901993218,3cyclictest171rcu_preempt20:40:371
210390199321,22cyclictest2162569-21ssh21:45:351
2103900993227,3cyclictest2095932-21kworker/u8:10-rpciod19:52:220
2103900993226,4cyclictest2268622-21latency_hist00:40:020
2103900993226,4cyclictest2250923-21gltestperf00:10:320
2103900993226,4cyclictest2250923-21gltestperf00:10:320
2103900993226,4cyclictest2238035-21latency_hist23:50:010
2103900993224,5cyclictest2112882-21cron19:40:010
2103900993224,4cyclictest12650irq/32-eth022:50:360
2103900993223,7cyclictest0-21swapper/021:35:000
2103900993222,8cyclictest2127211-21needreboot20:25:320
2103900993222,8cyclictest2127211-21needreboot20:25:320
2103900993221,3cyclictest2132771-21aten_r2power_vo20:45:260
2103900993220,4cyclictest265-21systemd-journal23:00:010
2103900993214,3cyclictest0-21swapper/022:30:390
2103900993214,3cyclictest0-21swapper/022:30:390
2103902993126,2cyclictest0-21swapper/220:40:002
2103902993124,3cyclictest0-21swapper/220:10:402
2103902993123,5cyclictest2138941-21gltestperf21:05:272
2103902993123,4cyclictest0-21swapper/223:30:342
210390199314,8cyclictest28-21ksoftirqd/123:30:371
2103901993125,3cyclictest0-21swapper/100:05:311
2103901993125,3cyclictest0-21swapper/100:05:311
2103901993123,5cyclictest694-21cron21:10:011
2103901993122,6cyclictest2143094-21latency_hist21:15:021
2103901993120,8cyclictest271ktimers/120:55:381
2103901993120,2cyclictest171rcu_preempt22:01:201
2103901993110,4cyclictest261rcuc/100:02:531
210390099316,6cyclictest2158450-21kworker/u8:12+events_unbound22:40:500
210390099312,7cyclictest2184093-21ntpq22:20:390
2103900993124,4cyclictest2164592-21latency_hist21:50:020
2103900993122,6cyclictest12650irq/32-eth021:15:390
2103900993122,4cyclictest0-21swapper/000:30:380
210390099311,24cyclictest2217661-21sh23:15:350
2103902993024,3cyclictest0-21swapper/223:50:442
2103902993023,5cyclictest0-21swapper/219:15:012
2103902993023,4cyclictest2107962-21processes19:20:412
2103902993023,4cyclictest2107962-21processes19:20:412
2103902993022,3cyclictest0-21swapper/223:05:372
2103902993022,3cyclictest0-21swapper/223:05:372
2103902993020,7cyclictest2205032-21grep22:55:262
2103902993019,6cyclictest0-21swapper/221:10:382
210390199303,23cyclictest2118306-21sshd19:55:351
210390199303,15cyclictest0-21swapper/120:25:331
210390199303,15cyclictest0-21swapper/120:25:331
2103901993023,4cyclictest2170784-21kworker/u8:9-xprtiod22:25:011
2103901993022,6cyclictest2213552-21idleruntime-cro23:10:011
2103901993022,6cyclictest2213552-21idleruntime-cro23:10:011
210390199302,15cyclictest2121424-21ps20:05:391
2103901993020,7cyclictest2174720-21missed_timers22:05:301
2103900993024,3cyclictest0-21swapper/000:25:490
2103900993023,4cyclictest2221011-21ssh23:20:460
2103900993023,3cyclictest0-21swapper/022:15:390
2103900993022,6cyclictest2137702-21needreboot21:00:310
210390099302,24cyclictest2146821-21awk21:20:260
210390099302,24cyclictest2146821-21awk21:20:260
210390099302,19cyclictest2187054-21ssh22:25:330
2103900993021,6cyclictest12650irq/32-eth023:25:360
210390299295,15cyclictest428-21mta-sts-daemon20:30:352
2103902992923,3cyclictest0-21swapper/219:45:492
2103902992922,4cyclictest1003-21mta-sts-daemon23:25:132
2103902992922,3cyclictest0-21swapper/219:35:012
2103902992920,7cyclictest2127910-21sh20:30:012
2103902992920,7cyclictest2127910-21sh20:30:012
210390299291,22cyclictest2168686-21ssh21:55:332
210390199293,21cyclictest371393-21cups-browsed22:50:381
2103901992923,3cyclictest261rcuc/120:00:391
2103901992910,6cyclictest171rcu_preempt20:30:201
210390099299,2cyclictest171rcu_preempt20:30:160
2103900992924,2cyclictest0-21swapper/019:50:020
2103902992821,5cyclictest1000-21mta-sts-daemon20:05:382
210390299280,11cyclictest66550snmpd20:15:292
210390199284,17cyclictest2205393-21sh22:55:331
210390099284,14cyclictest2118867-21kworker/0:1+rcu_gp20:00:020
2103900992816,10cyclictest2139773-21unixbench_multi21:05:440
210390299276,15cyclictest1001-21mta-sts-daemon19:35:392
210390299274,12cyclictest421-21dbus-daemon23:45:372
2103902992722,2cyclictest0-21swapper/222:10:422
2103902992721,3cyclictest0-21swapper/222:30:332
2103902992721,3cyclictest0-21swapper/222:30:332
2103902992719,4cyclictest0-21swapper/221:05:012
2103902992715,9cyclictest2109218-21needreboot19:25:322
210390199277,6cyclictest28-21ksoftirqd/119:10:011
2103900992713,11cyclictest12650irq/32-eth019:40:320
210390099271,20cyclictest2245155-21sh00:00:330
210390399262,18cyclictest0-21swapper/321:50:243
210390399261,16cyclictest0-21swapper/321:10:343
2103902992620,3cyclictest0-21swapper/200:00:452
2103902992619,3cyclictest0-21swapper/219:50:342
210390299261,20cyclictest2263547-21ssh00:30:352
210390199267,4cyclictest28-21ksoftirqd/119:25:451
210390199264,4cyclictest28-21ksoftirqd/119:20:371
210390199264,4cyclictest28-21ksoftirqd/119:20:371
2103900992619,4cyclictest2155825-21sh21:35:260
2103900992614,9cyclictest0-21swapper/023:45:010
2103900992613,10cyclictest0-21swapper/019:20:280
2103900992613,10cyclictest0-21swapper/019:20:280
2103900992520,3cyclictest2126196-21timerwakeupswit20:20:440
2103900992519,4cyclictest371393-21cups-browsed20:19:040
210390099251,5cyclictest2106262-21ntpq19:15:340
210390099251,18cyclictest12650irq/32-eth023:50:320
210390099249,3cyclictest171rcu_preempt20:40:330
210390099249,3cyclictest171rcu_preempt20:05:320
210390099242,5cyclictest171rcu_preempt19:30:420
210390399232,15cyclictest0-21swapper/300:10:463
210390399232,15cyclictest0-21swapper/300:10:463
210390399231,17cyclictest0-21swapper/320:40:433
210390399231,17cyclictest0-21swapper/319:25:353
210390299231,7cyclictest171rcu_preempt20:50:332
210390099238,3cyclictest2130357-21ntp_states20:35:380
210390399222,15cyclictest0-21swapper/323:05:363
210390399222,15cyclictest0-21swapper/323:05:363
210390399222,14cyclictest0-21swapper/321:35:353
210390399221,4cyclictest0-21swapper/322:15:373
2103902992213,6cyclictest265-21systemd-journal20:50:012
210390099227,3cyclictest0-21swapper/020:00:030
210390099224,3cyclictest171rcu_preempt19:10:010
210390099222,15cyclictest2134773-21ntpq20:50:340
2103900992214,6cyclictest2108628-21sshd19:25:220
210390399213,11cyclictest0-21swapper/321:25:343
210390399212,14cyclictest0-21swapper/323:20:383
210390399212,14cyclictest0-21swapper/322:55:363
210390399212,14cyclictest0-21swapper/322:20:343
210390399212,14cyclictest0-21swapper/321:15:383
210390399212,14cyclictest0-21swapper/300:00:333
210390399212,13cyclictest0-21swapper/322:35:353
210390399212,13cyclictest0-21swapper/322:35:353
210390399211,4cyclictest0-21swapper/322:05:363
210390399211,3cyclictest0-21swapper/322:45:423
210390399211,3cyclictest0-21swapper/321:55:333
210390399211,3cyclictest0-21swapper/321:30:363
210390399211,3cyclictest0-21swapper/321:00:343
210390399211,3cyclictest0-21swapper/320:35:263
210390399211,3cyclictest0-21swapper/320:15:353
210390399211,3cyclictest0-21swapper/320:05:383
210390399211,3cyclictest0-21swapper/319:30:383
210390399211,3cyclictest0-21swapper/300:30:383
210390399211,15cyclictest0-21swapper/322:30:353
210390399211,15cyclictest0-21swapper/322:30:353
210390399210,3cyclictest0-21swapper/323:45:363
210390399210,3cyclictest0-21swapper/323:25:323
210390399210,3cyclictest0-21swapper/319:35:343
210390399203,13cyclictest0-21swapper/322:50:353
210390399202,14cyclictest0-21swapper/323:35:383
210390399202,14cyclictest0-21swapper/322:25:333
210390399202,14cyclictest0-21swapper/320:00:333
210390399202,13cyclictest0-21swapper/321:45:343
210390399202,13cyclictest0-21swapper/320:20:343
210390399202,13cyclictest0-21swapper/319:50:343
210390399201,6cyclictest0-21swapper/319:45:233
210390399201,4cyclictest0-21swapper/323:40:333
210390399201,4cyclictest0-21swapper/323:00:433
210390399201,4cyclictest0-21swapper/319:55:383
210390399201,4cyclictest0-21swapper/319:10:353
210390399201,3cyclictest0-21swapper/323:55:363
210390399201,3cyclictest0-21swapper/323:15:353
210390399201,3cyclictest0-21swapper/322:10:353
210390399201,3cyclictest0-21swapper/321:05:323
210390399201,3cyclictest0-21swapper/320:55:273
210390399201,3cyclictest0-21swapper/320:45:373
210390399201,3cyclictest0-21swapper/320:25:263
210390399201,3cyclictest0-21swapper/320:25:263
210390399201,3cyclictest0-21swapper/319:20:353
210390399201,3cyclictest0-21swapper/319:20:353
210390399201,3cyclictest0-21swapper/319:15:343
210390399201,3cyclictest0-21swapper/300:20:333
210390399201,3cyclictest0-21swapper/300:15:363
210390399201,3cyclictest0-21swapper/300:05:463
210390399201,3cyclictest0-21swapper/300:05:463
210390399201,14cyclictest0-21swapper/322:00:313
210390399201,14cyclictest0-21swapper/321:40:343
210390399201,14cyclictest0-21swapper/300:25:333
210390399201,13cyclictest0-21swapper/323:50:293
210390399201,13cyclictest0-21swapper/323:10:333
210390399201,13cyclictest0-21swapper/321:20:303
210390399201,13cyclictest0-21swapper/321:20:303
210390399201,13cyclictest0-21swapper/320:10:333
210390399191,3cyclictest0-21swapper/323:30:143
210390399191,3cyclictest0-21swapper/322:40:033
210390399191,3cyclictest0-21swapper/320:50:253
210390399191,3cyclictest0-21swapper/319:40:333
210390399191,3cyclictest0-21swapper/300:35:293
210390399191,13cyclictest0-21swapper/320:30:323
210390399171,4cyclictest0-21swapper/319:10:013
210390299174,2cyclictest171rcu_preempt19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional