You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-20 - 13:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Mon Jul 20, 2026 12:43:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3394685996037,17cyclictest3481963-40mandb10:34:320
3394687995846,9cyclictest2602816-21cups-browsed11:30:532
3394686995546,6cyclictest1003-21mta-sts-daemon10:34:231
3394687995242,6cyclictest2602816-21cups-browsed10:50:532
3394686995139,8cyclictest0-21swapper/110:35:421
3394687994940,6cyclictest3429200-21systemd-run09:05:022
3394685994921,17cyclictest151ktimers/008:20:440
3394687994842,2cyclictest0-21swapper/210:34:322
339468699484,36cyclictest3549695-21ssh12:25:441
3394686994838,5cyclictest3436741-21munin-run09:20:011
3394685994825,19cyclictest3461002-21munin-run10:00:010
3394686994737,6cyclictest3442859-21latency_hist09:30:021
3394686994737,6cyclictest3442859-21latency_hist09:30:021
3394687994637,5cyclictest421-21dbus-daemon12:30:022
3394687994537,5cyclictest3547204-21unixbench-2d12:20:552
3394687994536,5cyclictest3533478-21latency_hist12:00:012
3394685994529,13cyclictest3423198-21systemd-run08:45:010
3394685994529,13cyclictest3423198-21systemd-run08:45:010
3394687994438,3cyclictest0-21swapper/211:05:482
3394687994434,6cyclictest2602816-21cups-browsed12:33:192
3394685994435,5cyclictest0-21swapper/011:15:490
3394687994335,3cyclictest0-21swapper/209:05:482
3394687994334,5cyclictest3406741-21latency_hist07:50:022
3394686994335,5cyclictest3422213-21fschecks_time08:40:371
3394686994335,5cyclictest3422213-21fschecks_time08:40:371
3394686994334,5cyclictest3460997-21latency_hist10:00:011
3394686994334,5cyclictest3396691-21fschecks_count07:15:391
3394687994228,9cyclictest420-21avahi-daemon10:39:182
339468799413,34cyclictest3528600-21ssh11:50:462
3394687994127,10cyclictest3470075-21cron10:15:012
3394686994132,5cyclictest0-21swapper/109:20:541
339468699411,37cyclictest3453066-21ssh09:45:461
3394685994134,4cyclictest3397659-21cron07:20:010
3394685994133,6cyclictest421-21dbus-daemon11:50:020
3394685994130,8cyclictest3436738-21sh09:20:010
3394687994031,5cyclictest3486276-21ssh10:40:472
3394686994032,5cyclictest3548640-21systemd-run12:25:021
3394686994031,5cyclictest3552573-21sort12:30:411
3394685994033,4cyclictest3399207-21systemd-run07:25:010
3394687993932,3cyclictest0-21swapper/209:15:462
3394687993929,7cyclictest3451925-21sh09:45:012
3394687993929,6cyclictest1003-21mta-sts-daemon12:35:202
3394687993928,7cyclictest0-21swapper/209:25:382
3394687993928,7cyclictest0-21swapper/209:25:382
3394686993933,3cyclictest0-21swapper/110:40:571
3394686993931,5cyclictest3456214-21/usr/sbin/munin09:50:491
3394685993932,5cyclictest485-21in:imuxsock11:00:010
339468799384,25cyclictest3474194-21ssh10:20:432
3394687993831,4cyclictest3445866-21systemd09:35:012
3394687993831,4cyclictest3445866-21systemd09:35:012
3394687993830,5cyclictest3402193-21systemd-run07:35:022
3394687993830,4cyclictest0-21swapper/210:45:442
3394687993829,5cyclictest3507009-21fschecks_count11:15:392
3394687993829,4cyclictest0-21swapper/211:11:032
3394687993829,4cyclictest0-21swapper/211:11:032
3394687993828,5cyclictest0-21swapper/211:20:462
3394687993826,8cyclictest420-21avahi-daemon12:03:252
3394686993832,3cyclictest0-21swapper/112:40:011
3394686993830,5cyclictest421-21dbus-daemon10:05:561
3394686993828,4cyclictest0-21swapper/108:50:421
3394685993833,3cyclictest3448875-21munin-run09:40:000
3394685993829,7cyclictest3557660-21munin-run12:40:010
339468799375,20cyclictest428-21mta-sts-daemon07:10:422
3394687993728,6cyclictest3399189-21cut07:25:012
3394687993725,9cyclictest3419486-21needreboot08:30:412
3394686993729,5cyclictest3429200-21systemd-run09:05:021
3394685993730,5cyclictest3418697-21systemd-run08:30:010
3394685993729,5cyclictest485-21in:imuxsock11:05:010
3394685993729,5cyclictest3485680-21aten_r2power_en10:40:350
3394685993728,6cyclictest12650irq/32-eth010:00:480
3394685993727,7cyclictest485-21in:imuxsock10:10:010
3394685993727,7cyclictest3534961-21switchtime12:00:550
3394685993725,9cyclictest3445953-21grep09:35:020
3394685993725,9cyclictest3445953-21grep09:35:020
339468599371,30cyclictest3546678-21sh12:20:420
339468799364,22cyclictest3462074-21ssh10:00:422
3394687993630,3cyclictest0-21swapper/209:50:422
3394687993629,3cyclictest0-21swapper/209:15:012
3394687993628,5cyclictest3521365-21munin-run11:40:012
3394687993628,4cyclictest0-21swapper/209:46:292
3394687993623,5cyclictest1001-21mta-sts-daemon08:10:362
3394686993628,5cyclictest3524396-21cron11:45:011
3394685993631,2cyclictest0-21swapper/010:20:010
3394687993528,3cyclictest0-21swapper/212:10:562
3394687993524,7cyclictest2602816-21cups-browsed10:25:112
339468799352,27cyclictest3495302-21ssh10:55:442
339468799352,24cyclictest3498457-21ssh11:00:482
339468699353,24cyclictest3471220-21ssh10:15:431
3394686993529,3cyclictest0-21swapper/112:05:501
3394686993528,4cyclictest3417184-21grep08:25:011
3394686993527,5cyclictest3517031-21unixbench_multi11:31:051
3394686993526,5cyclictest3501389-21ntp_kernel_pll_11:05:451
3394685993530,3cyclictest3394684-21cyclictest08:45:560
3394685993525,8cyclictest3504008-21fschecks_time11:10:410
3394685993525,8cyclictest3504008-21fschecks_time11:10:410
3394685993525,7cyclictest2602816-21cups-browsed09:40:480
339468799345,22cyclictest3543704-21ssh12:15:462
3394687993424,8cyclictest3404958-21timerwakeupswit07:40:542
3394687993422,8cyclictest420-21avahi-daemon08:35:382
3394686993425,6cyclictest1003-21mta-sts-daemon10:29:571
3394686993411,18cyclictest3497916-21df11:00:351
3394685993427,5cyclictest2602812-21cupsd08:55:500
3394685993427,4cyclictest3457440-21diskmemload09:53:200
339468799334,14cyclictest421-21dbus-daemon08:05:442
3394687993327,3cyclictest0-21swapper/210:00:012
3394687993325,5cyclictest3527624-21latency_hist11:50:032
3394687993325,5cyclictest3426178-21systemd-run08:55:022
3394687993324,6cyclictest3447332-21sh09:35:562
3394686993327,3cyclictest0-21swapper/110:15:021
3394686993327,3cyclictest0-21swapper/107:35:011
3394686993325,5cyclictest3533481-21munin-plugin-st12:00:011
3394686993325,5cyclictest3394707-21latency_hist07:10:011
3394686993325,5cyclictest3394707-21latency_hist07:10:011
3394686993325,4cyclictest0-21swapper/111:35:461
3394686993324,6cyclictest1-21systemd12:20:021
3394686993323,5cyclictest0-21swapper/109:10:431
3394686993321,9cyclictest3489660-21switchtime10:45:551
3394685993327,4cyclictest3440470-21fschecks_time09:25:380
3394685993327,4cyclictest3440470-21fschecks_time09:25:380
3394685993326,4cyclictest3425530-21sshd08:50:420
3394687993226,3cyclictest0-21swapper/211:45:022
3394687993225,3cyclictest0-21swapper/208:20:012
3394687993225,3cyclictest0-21swapper/208:00:012
3394687993223,4cyclictest0-21swapper/211:25:542
3394687993223,4cyclictest0-21swapper/210:05:422
3394687993221,8cyclictest3427064-21ntp_kernel_pll_08:55:452
339468799321,18cyclictest3437856-21ssh09:20:492
339468699324,14cyclictest3513512-21ssh11:25:461
3394686993225,4cyclictest3447116-21proc_pri09:35:511
3394686993224,5cyclictest421-21dbus-daemon07:40:011
3394686993224,5cyclictest3396240-21latency_hist07:15:021
3394686993224,5cyclictest2602816-21cups-browsed11:20:571
3394686993222,7cyclictest3450473-21timerwakeupswit09:40:561
3394685993227,3cyclictest3453201-21kworker/u8:0-nfsiod09:45:510
3394685993226,4cyclictest3467080-21kworker/u8:9-xprtiod11:40:500
3394685993225,4cyclictest3476137-21cron10:25:010
3394685993225,4cyclictest1001-21mta-sts-daemon11:50:140
3394685993224,5cyclictest3419199-21fschecks_time08:30:360
3394685993224,5cyclictest1001-21mta-sts-daemon10:45:390
3394687993124,3cyclictest0-21swapper/210:15:472
3394687993122,5cyclictest3397714-21systemd-run07:20:012
3394687993120,8cyclictest3424325-21switchtime08:45:552
3394686993124,4cyclictest3497264-21tr11:00:011
3394686993123,5cyclictest427-21polkitd10:50:191
339468599313,24cyclictest3438080-21ssh09:21:000
3394685993125,4cyclictest3500865-21aten_r2power_vo11:05:350
3394685993125,4cyclictest3420629-21cpuspeed08:35:350
3394685993125,4cyclictest3403706-21systemd-run07:40:010
3394685993124,4cyclictest3430556-21unixbench_multi09:05:550
3394685993122,6cyclictest3531331-21latency11:55:400
3394685993121,7cyclictest1-21systemd12:28:210
339468799303,22cyclictest3403665-21munin-run07:40:012
3394686993024,3cyclictest0-21swapper/110:25:011
3394686993022,6cyclictest0-21swapper/109:10:021
339468699302,22cyclictest3462081-21ssh10:00:421
3394686993021,6cyclictest3412692-21systemd-run08:10:011
3394686993021,6cyclictest1001-21mta-sts-daemon07:45:381
339468699301,22cyclictest3525630-21ssh11:45:501
3394685993024,4cyclictest3494305-21grep10:55:020
3394685993024,3cyclictest0-21swapper/010:25:590
3394685993023,4cyclictest3432328-21unixbench_multi09:10:550
3394685993023,4cyclictest0-21swapper/012:15:460
3394685993022,5cyclictest421-21dbus-daemon07:55:010
3394687992922,4cyclictest1001-21mta-sts-daemon12:05:142
3394687992921,5cyclictest2602816-21cups-browsed07:27:072
3394687992919,6cyclictest1-21systemd07:55:012
3394686992923,3cyclictest0-21swapper/108:30:011
3394686992922,4cyclictest3404146-21cpuspeed07:40:351
3394686992922,3cyclictest0-21swapper/112:10:461
339468699292,21cyclictest3504406-21sh11:10:491
339468699292,21cyclictest3504406-21sh11:10:491
3394686992921,4cyclictest0-21swapper/111:50:451
3394686992910,13cyclictest3507485-21diskmemload11:15:491
339468599294,4cyclictest3430670-21diskmemload10:10:420
3394685992924,3cyclictest0-21swapper/007:35:010
3394685992923,3cyclictest0-21swapper/011:31:040
3394685992922,4cyclictest3512441-21latency_hist11:25:020
3394685992920,7cyclictest3519400-21ntp_kernel_err11:35:440
3394686992821,4cyclictest3413974-21timerwakeupswit08:10:571
3394686992821,3cyclictest0-21swapper/112:00:491
3394686992815,10cyclictest1002-21mta-sts-daemon07:50:341
3394685992822,3cyclictest0-21swapper/012:30:340
3394685992819,6cyclictest3540498-21missed_timers12:10:400
3394685992818,7cyclictest0-21swapper/011:25:330
3394687992719,5cyclictest3417704-21fschecks_count08:25:382
339468699273,15cyclictest420-21avahi-daemon09:30:431
339468699273,15cyclictest420-21avahi-daemon09:30:431
3394686992720,3cyclictest0-21swapper/108:00:511
3394686992719,5cyclictest1-21systemd07:59:441
3394685992721,4cyclictest3537092-21telnet12:05:350
3394685992721,4cyclictest3413443-21ls08:10:410
3394685992721,4cyclictest3410202-21fschecks_count08:00:360
3394687992621,2cyclictest0-21swapper/208:25:012
3394686992620,3cyclictest0-21swapper/109:00:011
3394686992618,5cyclictest3421663-21munin-plugin-st08:40:001
3394686992618,5cyclictest2602816-21cups-browsed08:34:321
339468599262,20cyclictest2602816-21cups-browsed07:55:490
3394685992620,4cyclictest3430670-21diskmemload10:37:500
3394687992518,5cyclictest3394684-21cyclictest08:44:052
3394687992518,5cyclictest3394684-21cyclictest08:44:052
3394686992519,3cyclictest0-21swapper/108:50:001
339468599258,13cyclictest3411940-21sed08:05:400
3394685992519,4cyclictest3400616-21sed07:25:530
3394685992519,4cyclictest3395209-21fschecks_time07:10:370
3394687992416,5cyclictest3411117-21head08:00:572
3394686992418,3cyclictest0-21swapper/107:20:461
3394686992417,3cyclictest0-21swapper/108:15:461
3394685992417,4cyclictest1-21systemd08:20:010
339468699223,13cyclictest428-21mta-sts-daemon07:25:431
3394685992217,3cyclictest0-21swapper/007:45:370
3394685992217,2cyclictest0-21swapper/009:00:570
339468599221,18cyclictest3404567-21ntpq07:40:440
339468799209,7cyclictest3394696-21systemd-run07:10:012
339468799209,7cyclictest3394696-21systemd-run07:10:012
339468899182,7cyclictest0-21swapper/311:25:453
339468599181,14cyclictest3394641-21munin-run07:10:000
339468599181,14cyclictest3394641-21munin-run07:10:000
339468899172,4cyclictest0-21swapper/309:30:433
339468899172,4cyclictest0-21swapper/309:30:433
339468899163,7cyclictest0-21swapper/311:00:503
339468899163,7cyclictest0-21swapper/310:34:243
339468899161,8cyclictest0-21swapper/309:20:533
339468899152,7cyclictest0-21swapper/311:35:453
339468899151,6cyclictest0-21swapper/307:15:463
339468899142,7cyclictest0-21swapper/312:35:383
339468899142,6cyclictest0-21swapper/310:55:483
339468899142,5cyclictest0-21swapper/309:05:453
339468899141,7cyclictest0-21swapper/311:05:473
339468899141,6cyclictest0-21swapper/308:50:433
339468899132,7cyclictest0-21swapper/310:20:373
339468899132,6cyclictest0-21swapper/311:40:383
339468899132,6cyclictest0-21swapper/310:00:403
339468899132,6cyclictest0-21swapper/309:45:333
339468899132,6cyclictest0-21swapper/308:45:003
339468899132,6cyclictest0-21swapper/308:45:003
339468899132,6cyclictest0-21swapper/308:40:003
339468899132,6cyclictest0-21swapper/307:50:013
339468899132,5cyclictest0-21swapper/312:20:393
339468899132,5cyclictest0-21swapper/309:25:383
339468899132,5cyclictest0-21swapper/309:25:383
339468899132,5cyclictest0-21swapper/307:55:433
339468899132,5cyclictest0-21swapper/307:20:463
339468899131,8cyclictest0-21swapper/312:00:423
339468899131,7cyclictest0-21swapper/312:30:533
339468899131,7cyclictest0-21swapper/311:50:443
339468899131,7cyclictest0-21swapper/311:45:523
339468899131,7cyclictest0-21swapper/310:45:443
339468899131,6cyclictest0-21swapper/311:15:453
339468899131,6cyclictest0-21swapper/310:15:433
339468899131,6cyclictest0-21swapper/308:25:463
339468899131,5cyclictest0-21swapper/308:20:443
339468899130,6cyclictest0-21swapper/307:40:453
339468899124,6cyclictest0-21swapper/311:20:463
339468899122,6cyclictest0-21swapper/312:10:413
339468899122,6cyclictest0-21swapper/311:10:523
339468899122,6cyclictest0-21swapper/311:10:523
339468899122,6cyclictest0-21swapper/309:50:353
339468899122,6cyclictest0-21swapper/309:15:483
339468899122,5cyclictest0-21swapper/312:05:373
339468899122,5cyclictest0-21swapper/310:50:383
339468899121,8cyclictest0-21swapper/309:10:333
339468899121,7cyclictest0-21swapper/310:25:413
339468899121,7cyclictest0-21swapper/309:55:333
339468899121,6cyclictest0-21swapper/311:55:443
339468899121,6cyclictest0-21swapper/311:30:403
339468899121,6cyclictest0-21swapper/310:35:393
339468899121,6cyclictest0-21swapper/307:35:413
339468899121,5cyclictest0-21swapper/308:30:433
339468899121,5cyclictest0-21swapper/308:15:423
339468899121,5cyclictest0-21swapper/308:05:433
339468899111,6cyclictest0-21swapper/310:10:373
339468899111,6cyclictest0-21swapper/310:05:353
339468899111,6cyclictest0-21swapper/309:40:343
339468899111,6cyclictest0-21swapper/309:35:383
339468899111,6cyclictest0-21swapper/309:05:023
339468899111,5cyclictest0-21swapper/312:20:033
339468899102,4cyclictest0-21swapper/308:10:473
339468899101,6cyclictest0-21swapper/310:40:433
339468899101,5cyclictest0-21swapper/312:25:213
339468899101,5cyclictest0-21swapper/308:55:423
339468899101,5cyclictest0-21swapper/307:30:423
339468899101,5cyclictest0-21swapper/307:10:453
33946889991,5cyclictest0-21swapper/308:45:343
33946889991,5cyclictest0-21swapper/308:00:333
33946889991,5cyclictest0-21swapper/307:50:343
33946889991,5cyclictest0-21swapper/307:25:393
33946889971,5cyclictest0-21swapper/307:10:013
33946889971,5cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional