You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-17 - 01:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Fri Apr 17, 2026 00:43:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3173563998068,9cyclictest3313198-21awk23:20:001
3173563998068,9cyclictest3313198-21awk23:20:001
3173562996454,7cyclictest420-21avahi-daemon22:30:440
3173563996047,10cyclictest421-21dbus-daemon21:05:011
3173564995946,8cyclictest3217494-21run-parts21:20:002
3173564995946,8cyclictest3217494-21run-parts21:20:002
3173562995929,20cyclictest12350irq/32-eth021:25:340
3173562995135,10cyclictest12350irq/32-eth022:05:390
3173563995039,6cyclictest3205050-21systemd-run20:55:011
3173564994937,9cyclictest421-21dbus-daemon22:00:492
3173563994940,5cyclictest3357393-21systemd-run00:15:021
3173563994938,6cyclictest0-21swapper/122:45:371
317356399483,34cyclictest3322572-21ssh23:30:421
3173563994812,6cyclictest0-21swapper/120:30:391
3173563994736,7cyclictest3258613-21sendmail_mailqu22:10:501
3173563994636,5cyclictest3173567-21systemd-run19:10:011
3173563994634,8cyclictest420-21avahi-daemon21:45:011
317356399463,33cyclictest3338632-21ssh23:50:461
317356399462,37cyclictest3302401-21ssh23:05:401
3173564994538,3cyclictest0-21swapper/222:35:392
3173563994533,6cyclictest0-21swapper/122:00:381
3173563994528,7cyclictest3261637-21aten_r2power_cu22:15:281
3173563994528,7cyclictest3261637-21aten_r2power_cu22:15:281
3173562994540,3cyclictest0-21swapper/021:10:310
3173564994435,6cyclictest421-21dbus-daemon23:15:012
3173563994435,6cyclictest1-21systemd23:30:021
3173562994437,4cyclictest3305321-21latency_hist23:10:020
3173564994336,4cyclictest3301163-21grep23:05:012
3173563994335,5cyclictest3184168-21latency_hist19:45:021
3173562994326,14cyclictest0-21swapper/021:20:000
3173562994326,14cyclictest0-21swapper/021:20:000
3173564994232,5cyclictest0-21swapper/222:55:432
3173564994229,10cyclictest3358927-21switchtime00:15:492
3173564994229,10cyclictest3358927-21switchtime00:15:492
3173563994134,3cyclictest0-21swapper/122:35:381
3173563994132,6cyclictest3345300-21cupsd00:00:001
3173562994135,4cyclictest3290538-21sendmail_mailqu22:50:450
3173564994031,6cyclictest428-21mta-sts-daemon00:10:452
3173564994031,5cyclictest0-21swapper/200:30:402
3173564994030,7cyclictest3270224-21needreboot22:25:412
3173563994028,9cyclictest3233411-21grep21:40:021
3173562994026,12cyclictest3345299-40systemctl00:00:000
317356499394,26cyclictest3246417-21ssh21:55:412
3173564993930,6cyclictest3255020-21unixbench_singl22:05:492
3173564993928,6cyclictest0-21swapper/223:25:402
317356499391,31cyclictest3258436-21ssh22:10:452
3173563993930,6cyclictest3365377-21ssh00:25:001
3173563993930,5cyclictest3190057-21munin-plugin-st20:05:001
3173563993929,7cyclictest3184723-21latency19:45:341
3173563993928,7cyclictest3374197-21fschecks_time00:35:301
3173563993928,6cyclictest3337319-21sh23:50:001
3173564993830,5cyclictest3362399-21missed_timers00:20:352
3173564993830,5cyclictest3313205-21munin-run23:20:002
3173564993830,5cyclictest3313205-21munin-run23:20:002
3173564993829,6cyclictest3326744-21mailstats23:35:462
3173564993829,6cyclictest3326744-21mailstats23:35:462
3173564993829,5cyclictest3322862-21timerandwakeup23:30:472
3173564993828,6cyclictest3199850-21needreboot20:35:392
3173564993828,5cyclictest0-21swapper/221:25:402
3173563993831,3cyclictest0-21swapper/123:35:411
3173563993831,3cyclictest0-21swapper/123:35:411
3173563993829,5cyclictest3179572-21systemd-run19:30:011
3173563993828,4cyclictest0-21swapper/121:50:371
3173563993827,6cyclictest1051-21mta-sts-daemon22:50:391
3173563993823,11cyclictest3298997-21unixbench_singl23:00:531
3173563993814,15cyclictest3214733-21sshd21:15:381
3173563993814,15cyclictest3214733-21sshd21:15:381
3173562993830,5cyclictest1368112-21cups-browsed22:20:380
3173562993830,5cyclictest1368112-21cups-browsed22:20:380
3173564993728,6cyclictest3265144-21munin-run22:20:012
3173564993728,6cyclictest3265144-21munin-run22:20:012
3173564993727,6cyclictest3290953-21unixbench_singl22:50:502
3173564993725,9cyclictest3238549-21ntp_kernel_pll_21:45:412
317356499371,29cyclictest3226688-21ssh21:30:392
3173563993729,3cyclictest0-21swapper/122:55:431
3173563993727,6cyclictest3373508-21latency_hist00:35:021
317356299373,23cyclictest3218866-21ssh21:20:450
3173562993729,5cyclictest420-21avahi-daemon23:17:310
3173562993729,5cyclictest420-21avahi-daemon23:17:310
3173564993628,5cyclictest3281841-21fschecks_count22:40:302
3173564993628,5cyclictest3277047-21latency_hist22:35:022
3173564993628,5cyclictest3196068-21systemd-run20:25:012
3173564993626,6cyclictest3374276-21if_eth000:35:312
3173564993626,5cyclictest3233961-21aten_r2power_fr21:40:272
3173564993625,8cyclictest3346320-21latency00:00:322
317356499362,26cyclictest3218777-21ssh21:20:412
317356499361,25cyclictest3338622-21ssh23:50:452
3173563993628,5cyclictest3282496-21sendmail_mailqu22:40:441
3173563993628,5cyclictest3273159-21latency_hist22:30:011
3173563993627,6cyclictest3277172-21latency_hist22:35:021
3173563993624,8cyclictest3222518-21ls21:25:311
317356399362,26cyclictest3330645-21ssh23:40:451
317356399361,28cyclictest3246476-21ssh21:55:431
3173562993627,7cyclictest3250093-21kernelversion22:00:330
3173562993627,6cyclictest1368112-21cups-browsed21:58:260
317356499355,25cyclictest3366552-21ssh00:25:382
3173564993529,4cyclictest3194574-21kthreadd20:20:012
3173564993529,3cyclictest0-21swapper/222:45:402
3173564993521,11cyclictest421-21dbus-daemon20:05:422
317356399355,25cyclictest3218745-21ssh21:20:401
317356399353,24cyclictest659-21snmpd22:05:431
3173563993527,5cyclictest3306069-21kernelversion23:10:331
3173563993525,4cyclictest0-21swapper/100:17:551
3173563993525,4cyclictest0-21swapper/100:17:551
317356399351,5cyclictest171rcu_preempt20:40:431
3173563993514,13cyclictest261rcuc/123:20:421
3173562993529,4cyclictest3326032-21fschecks_time23:35:310
3173562993529,4cyclictest3326032-21fschecks_time23:35:310
3173562993529,3cyclictest0-21swapper/000:20:510
3173562993528,4cyclictest3258932-21unixbench-2d22:10:540
3173562993527,5cyclictest421-21dbus-daemon22:41:550
3173562993521,11cyclictest3265135-21cron22:20:010
3173562993521,11cyclictest3265135-21cron22:20:010
3173562993519,11cyclictest3374625-21ssh00:35:370
317356299351,28cyclictest3306409-21ssh23:10:390
3173564993428,3cyclictest0-21swapper/221:50:372
3173564993426,4cyclictest0-21swapper/223:05:422
3173563993428,3cyclictest0-21swapper/120:40:011
3173563993428,3cyclictest0-21swapper/120:00:001
3173563993425,6cyclictest3180071-21fschecks_count19:30:301
3173563993425,5cyclictest3345375-21kworker/u8:11-events_unbound00:00:271
3173563993423,6cyclictest0-21swapper/121:30:431
317356399341,14cyclictest0-21swapper/119:35:371
317356299343,26cyclictest3294291-21ssh22:55:380
3173562993428,3cyclictest3278145-21needreboot22:35:350
3173562993427,5cyclictest3243068-21unixbench_singl21:50:510
3173562993421,11cyclictest0-21swapper/023:35:020
3173564993325,5cyclictest3186070-21fschecks_time19:50:342
3173564993325,5cyclictest3186070-21fschecks_time19:50:342
3173564993321,9cyclictest1368112-21cups-browsed19:35:352
3173564993318,11cyclictest3266241-21needreboot22:20:352
3173564993318,11cyclictest3266241-21needreboot22:20:352
3173563993323,7cyclictest0-21swapper/100:30:021
3173562993327,4cyclictest1368112-21cups-browsed23:20:380
3173562993325,5cyclictest1368104-21cupsd21:36:130
3173562993320,10cyclictest0-21swapper/000:20:010
3173562993320,10cyclictest0-21swapper/000:20:010
3173564993222,7cyclictest3205043-21sh20:55:012
317356399322,23cyclictest3210740-21ssh21:10:371
3173563993220,9cyclictest3208913-21ntp_kernel_pll_21:05:381
3173563993215,12cyclictest1051-21mta-sts-daemon19:15:361
3173562993225,5cyclictest1475125-21munin-node19:50:550
3173562993225,5cyclictest1475125-21munin-node19:50:550
317356299322,26cyclictest3270458-21ssh22:25:500
317356499313,21cyclictest1475125-21munin-node19:45:392
3173564993124,5cyclictest0-21swapper/221:40:012
3173564993123,5cyclictest3317237-21sh23:25:002
3173563993123,4cyclictest0-21swapper/122:20:291
3173563993123,4cyclictest0-21swapper/122:20:291
3173563993122,5cyclictest3220216-21kworker/u8:0-xprtiod21:50:011
317356399312,22cyclictest3350552-21sshd00:05:381
3173563993120,8cyclictest3193050-21sh20:15:011
3173562993126,2cyclictest0-21swapper/023:50:000
3173562993125,4cyclictest261-21systemd-journal23:45:000
3173562993122,5cyclictest0-21swapper/022:45:440
3173564993025,2cyclictest0-21swapper/200:00:012
3173564993023,5cyclictest0-21swapper/219:20:012
3173564993020,7cyclictest3178053-21cron19:25:012
3173564993019,8cyclictest3211167-21timerandwakeup21:10:482
3173563993020,7cyclictest3175064-21sh19:15:011
3173562993025,3cyclictest0-21swapper/000:10:460
3173562993024,4cyclictest3181572-21fschecks_time19:35:300
3173562993024,3cyclictest3194528-21cron20:20:010
3173562993024,3cyclictest0-21swapper/020:00:380
3173562993023,4cyclictest3175115-21latency_hist19:15:020
3173562993021,7cyclictest440-21thd23:50:510
3173562993018,10cyclictest3366703-21ssh00:25:440
317356299301,24cyclictest3298302-21ssh23:00:390
3173564992923,2cyclictest0-21swapper/200:05:382
3173564992922,4cyclictest0-21swapper/219:10:022
3173564992922,3cyclictest0-21swapper/221:00:362
3173564992922,3cyclictest0-21swapper/220:30:442
3173564992922,3cyclictest0-21swapper/220:00:382
3173564992921,5cyclictest3193027-21cron20:15:012
3173563992923,3cyclictest0-21swapper/120:20:011
3173562992925,2cyclictest0-21swapper/020:25:010
3173562992923,4cyclictest3180077-21fschecks_time19:30:300
3173562992923,3cyclictest0-21swapper/000:00:470
3173562992921,6cyclictest3226762-21ntp_offset21:30:410
317356299291,23cyclictest3318499-21ntpq23:25:410
3173564992821,3cyclictest0-21swapper/223:45:372
317356499281,20cyclictest420-21avahi-daemon19:25:412
317356399282,19cyclictest421-21dbus-daemon20:05:421
3173563992820,5cyclictest3206526-21sendmail21:00:011
3173563992816,9cyclictest421-21dbus-daemon19:50:401
3173563992816,9cyclictest421-21dbus-daemon19:50:401
3173562992821,4cyclictest3353355-21munin-plugin-st00:10:010
3173562992819,6cyclictest3185428-21unixbench_multi19:45:490
3173562992814,12cyclictest3200276-21timerandwakeup20:35:550
3173564992721,3cyclictest0-21swapper/220:00:002
3173564992720,4cyclictest3176517-21kworker/u8:4-xprtiod20:30:022
3173564992720,3cyclictest0-21swapper/223:40:342
3173564992719,5cyclictest3206456-21head20:55:482
3173563992720,3cyclictest0-21swapper/120:25:011
3173562992720,4cyclictest3209529-21cron21:10:010
317356399269,12cyclictest421-21dbus-daemon19:25:021
3173563992618,5cyclictest1368112-21cups-browsed20:29:121
3173562992619,5cyclictest420-21avahi-daemon20:10:420
3173562992618,5cyclictest12350irq/32-eth021:40:370
3173562992616,7cyclictest0-21swapper/021:45:520
3173562992614,9cyclictest0-21swapper/020:55:000
3173564992516,6cyclictest1368112-21cups-browsed20:48:512
3173564992516,6cyclictest1368112-21cups-browsed20:48:512
3173563992518,4cyclictest1056-21mta-sts-daemon20:45:251
3173563992518,4cyclictest1056-21mta-sts-daemon20:45:251
3173562992519,4cyclictest3184052-21cron19:45:010
3173562992519,3cyclictest0-21swapper/000:30:350
3173562992513,9cyclictest3175899-21ntpq19:15:350
317356299251,15cyclictest12350irq/32-eth020:55:420
3173564992416,5cyclictest3202115-21latency_hist20:45:022
3173562992418,4cyclictest3179484-21unixbench_singl19:25:470
3173562992418,4cyclictest1056-21mta-sts-daemon20:25:390
3173562992413,8cyclictest1368112-21cups-browsed19:56:540
3173564992318,3cyclictest2685479-21kworker/u8:7-xprtiod19:30:432
3173564992317,3cyclictest0-21swapper/219:10:392
317356499231,17cyclictest3209095-21ps21:05:452
317356299232,16cyclictest3202919-21ntpq20:45:350
317356299232,16cyclictest3202919-21ntpq20:45:350
317356299232,15cyclictest3177517-21ntp_states19:20:410
3173562992312,8cyclictest0-21swapper/020:45:010
3173562992216,4cyclictest1054-21mta-sts-daemon20:05:270
317356299221,10cyclictest3198421-21ntpq20:30:380
317356499212,14cyclictest3183461-21ntpq19:40:392
3173562992016,2cyclictest0-21swapper/021:00:270
317356599183,10cyclictest0-21swapper/322:50:403
317356599182,9cyclictest0-21swapper/323:00:413
317356599182,9cyclictest0-21swapper/322:45:443
317356599182,9cyclictest0-21swapper/300:20:403
317356599182,8cyclictest0-21swapper/321:25:333
317356599182,11cyclictest0-21swapper/322:20:373
317356599182,11cyclictest0-21swapper/322:20:373
317356599174,7cyclictest0-21swapper/300:30:443
317356599173,9cyclictest0-21swapper/322:25:303
317356599172,9cyclictest0-21swapper/323:40:393
317356599172,9cyclictest0-21swapper/322:40:373
317356599172,9cyclictest0-21swapper/322:30:443
317356599172,9cyclictest0-21swapper/322:15:443
317356599172,9cyclictest0-21swapper/322:15:443
317356599172,9cyclictest0-21swapper/321:15:383
317356599172,9cyclictest0-21swapper/321:15:383
317356599172,8cyclictest0-21swapper/323:50:483
317356599172,7cyclictest0-21swapper/319:25:383
317356599172,10cyclictest0-21swapper/321:00:363
317356599172,10cyclictest0-21swapper/300:15:393
317356599172,10cyclictest0-21swapper/300:15:393
317356599171,10cyclictest0-21swapper/323:55:443
317356599164,8cyclictest0-21swapper/323:25:373
317356599164,7cyclictest0-21swapper/323:05:393
317356599163,9cyclictest0-21swapper/319:20:373
317356599163,7cyclictest0-21swapper/321:40:363
317356599162,9cyclictest0-21swapper/322:00:373
317356599162,9cyclictest0-21swapper/321:35:423
317356599162,9cyclictest0-21swapper/321:10:363
317356599162,9cyclictest0-21swapper/300:25:463
317356599162,8cyclictest0-21swapper/322:55:373
317356599162,8cyclictest0-21swapper/321:55:373
317356599162,8cyclictest0-21swapper/321:30:323
317356599162,7cyclictest0-21swapper/323:35:373
317356599162,7cyclictest0-21swapper/323:35:373
317356599161,9cyclictest0-21swapper/322:35:363
317356599161,9cyclictest0-21swapper/321:50:433
317356299163,8cyclictest420-21avahi-daemon19:10:010
317356599154,7cyclictest0-21swapper/300:00:353
317356599153,7cyclictest0-21swapper/300:05:393
317356599153,5cyclictest0-21swapper/319:15:343
317356599152,9cyclictest0-21swapper/323:10:273
317356599152,9cyclictest0-21swapper/321:45:363
317356599152,8cyclictest0-21swapper/323:30:343
317356599152,8cyclictest0-21swapper/322:10:403
317356599152,7cyclictest0-21swapper/319:40:373
317356599151,9cyclictest0-21swapper/321:20:393
317356599151,9cyclictest0-21swapper/300:35:363
317356599151,8cyclictest0-21swapper/323:15:353
317356599151,8cyclictest0-21swapper/323:15:353
317356599151,8cyclictest0-21swapper/300:10:303
317356599151,10cyclictest0-21swapper/322:05:323
317356599143,7cyclictest0-21swapper/323:45:363
317356599142,8cyclictest0-21swapper/323:20:383
317356599141,7cyclictest0-21swapper/320:35:403
317356599141,7cyclictest0-21swapper/320:30:363
317356599141,7cyclictest0-21swapper/320:20:383
317356599141,7cyclictest0-21swapper/320:00:383
317356599133,5cyclictest0-21swapper/319:50:403
317356599133,5cyclictest0-21swapper/319:50:403
317356599133,4cyclictest0-21swapper/320:50:393
317356599132,6cyclictest0-21swapper/320:05:373
317356599131,6cyclictest0-21swapper/320:45:343
317356599131,6cyclictest0-21swapper/320:45:343
317356599130,7cyclictest0-21swapper/320:55:373
317356599123,5cyclictest0-21swapper/319:30:373
317356599122,6cyclictest0-21swapper/320:25:373
317356599122,6cyclictest0-21swapper/320:15:383
317356599122,6cyclictest0-21swapper/320:10:333
317356599121,7cyclictest0-21swapper/321:05:413
317356599121,7cyclictest0-21swapper/320:00:003
317356599121,7cyclictest0-21swapper/319:45:393
317356599121,7cyclictest0-21swapper/319:35:353
317356599111,5cyclictest0-21swapper/320:40:293
317356599111,5cyclictest0-21swapper/319:10:343
31735659981,4cyclictest0-21swapper/319:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional