You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 13:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Thu Jul 23, 2026 00:43:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3929494996349,11cyclictest4033135-21grep23:05:010
3929496995839,16cyclictest3996118-21tr22:00:012
3929496995731,22cyclictest4068165-21unixbench-2d00:00:552
3929494995226,23cyclictest0-21swapper/022:25:510
3929494995224,17cyclictest12650irq/32-eth000:05:470
3929495995125,22cyclictest0-21swapper/100:25:521
3929494995143,5cyclictest4066519-21plymouth00:00:010
3929496995041,5cyclictest485-21in:imuxsock00:00:022
3929495995041,6cyclictest4008235-21ssh22:20:011
392949599503,7cyclictest4046349-21ssh23:25:441
3929494995042,5cyclictest0-21swapper/022:00:460
3929495994839,6cyclictest4066570-21cups-browsed00:00:021
3929495994836,8cyclictest421-21dbus-daemon22:00:011
3929494994830,11cyclictest0-21swapper/023:20:440
392949499473,35cyclictest4055570-21ssh23:40:500
392949499472,39cyclictest3969775-21ssh21:15:490
3929496994625,14cyclictest265-21systemd-journal19:55:472
3929494994637,6cyclictest4053105-21/usr/sbin/munin23:36:030
3929494994636,7cyclictest3986970-21ssh21:45:000
3929494994636,7cyclictest3986970-21ssh21:45:000
3929494994635,8cyclictest3965927-21/usr/sbin/munin21:10:560
3929495994537,5cyclictest4008840-21aten_r2power_vo22:20:361
3929495994535,6cyclictest0-21swapper/119:25:011
3929495994535,6cyclictest0-21swapper/119:25:011
3929495994524,18cyclictest0-21swapper/100:35:411
3929495994430,11cyclictest4011895-21cpu22:25:351
392949599442,7cyclictest3950010-21ntpq20:15:491
392949599442,14cyclictest3965088-21ps21:05:571
3929494994423,11cyclictest151ktimers/020:25:460
3929494994422,11cyclictest151ktimers/023:05:430
392949599433,8cyclictest3948437-21ntpq20:10:461
3929495994334,6cyclictest3987021-21latency_hist21:45:011
3929495994334,6cyclictest3987021-21latency_hist21:45:011
392949599432,6cyclictest4037376-21ssh23:10:491
392949599432,14cyclictest3978476-21diskstats21:30:391
3929494994331,9cyclictest427-21polkitd20:03:470
392949599421,10cyclictest4022193-21ntpq22:45:511
392949599414,6cyclictest4000248-21ssh22:05:461
392949599412,6cyclictest4084571-21ntpq00:30:421
392949599412,6cyclictest3945368-21ntpq20:00:451
392949599411,14cyclictest4031167-21ssh23:00:451
392949599411,11cyclictest3958880-21ntpq20:45:431
392949599411,10cyclictest3964010-21sshd21:01:471
392949599411,10cyclictest3955890-21ntpq20:35:451
392949599411,10cyclictest3940973-21ntpq19:45:531
3929494994133,5cyclictest4000425-21proc_pri22:05:530
3929494994132,7cyclictest4089618-21cat00:40:010
3929494994126,12cyclictest0-21swapper/022:25:010
3929496994032,5cyclictest4002195-21systemd-run22:10:012
3929496994031,6cyclictest4012318-21needreboot22:25:422
3929495994020,17cyclictest0-21swapper/121:40:011
3929495994020,17cyclictest0-21swapper/121:40:011
392949599401,9cyclictest4002628-21aten_r2power_en22:10:361
392949599401,9cyclictest3952931-21ntpq20:25:461
392949599401,9cyclictest3951493-21ntpq20:20:481
392949599401,9cyclictest3951493-21ntpq20:20:481
3929495994014,23cyclictest3930975-21ls19:15:011
392949599401,10cyclictest4028097-21ntpq22:55:441
392949599401,10cyclictest3966675-21ntpq21:10:461
392949599400,9cyclictest3954395-21ntpq20:30:461
3929494994032,4cyclictest0-21swapper/021:50:460
3929496993933,3cyclictest0-21swapper/223:40:512
3929496993931,5cyclictest4087110-21aten_r2power_po00:35:362
392949599399,3cyclictest0-21swapper/119:55:451
392949599394,27cyclictest4055521-21ssh23:40:481
3929495993928,5cyclictest0-21swapper/119:35:441
3929495993919,17cyclictest0-21swapper/123:50:431
392949599391,5cyclictest4075543-21ssh00:15:451
3929495993915,21cyclictest0-21swapper/123:35:481
392949499393,30cyclictest4003288-21sshd22:10:460
3929496993831,4cyclictest4070319-21fschecks_time00:05:402
3929496993831,4cyclictest4028415-21mailstats22:55:552
3929496993831,4cyclictest3964560-21fschecks_time21:05:442
3929496993830,5cyclictest2170916-21cups-browsed21:29:142
3929496993829,5cyclictest1001-21mta-sts-daemon23:03:562
392949599387,4cyclictest4042239-21munin-run23:20:011
392949599383,9cyclictest3997241-21ntpq22:00:461
392949599383,6cyclictest4078585-21ssh00:20:461
392949599382,7cyclictest4043414-21ssh23:20:481
392949599382,6cyclictest3961949-21ntpq20:55:461
392949599382,6cyclictest3931844-21ntpq19:15:461
3929495993822,13cyclictest0-21swapper/121:25:471
392949599381,6cyclictest4070808-21ntpq00:05:511
3929495993813,21cyclictest3990135-21latency_hist21:50:021
392949599381,14cyclictest4033809-21diskstats23:05:371
392949599381,12cyclictest3990850-21irqstats21:50:391
392949499389,20cyclictest665-21snmpd19:30:470
392949499388,18cyclictest428-21mta-sts-daemon19:40:460
392949499381,28cyclictest665-21snmpd19:35:480
392949599378,5cyclictest0-21swapper/121:15:301
3929495993721,12cyclictest0-21swapper/100:15:011
392949599371,9cyclictest3973982-21sh21:22:441
392949599371,8cyclictest4067759-21/usr/sbin/munin00:00:481
392949599371,30cyclictest4019060-21ssh22:40:441
392949599371,30cyclictest4019060-21ssh22:40:441
392949599371,13cyclictest4024672-21df22:50:381
392949599371,13cyclictest3960381-21ntpq20:50:471
392949599371,11cyclictest3934967-21ntpq19:25:481
392949599371,10cyclictest3946672-21irqstats20:05:401
3929494993727,7cyclictest1001-21mta-sts-daemon23:50:480
3929494993718,13cyclictest151ktimers/023:30:460
3929494993718,13cyclictest151ktimers/023:30:460
3929496993626,8cyclictest3985614-21unixbench_multi21:40:572
3929496993626,8cyclictest3985614-21unixbench_multi21:40:572
3929496993626,6cyclictest1001-21mta-sts-daemon21:45:142
392949599362,5cyclictest3941703-21/usr/sbin/munin19:50:581
392949599361,9cyclictest3939223-21memory19:40:411
3929495993613,19cyclictest0-21swapper/123:45:501
392949599361,15cyclictest4049467-21ssh23:30:471
392949599361,15cyclictest4049467-21ssh23:30:471
392949599361,12cyclictest4018127-21latency_hist22:40:031
392949599361,12cyclictest4018127-21latency_hist22:40:031
392949599361,10cyclictest3958043-21/usr/sbin/munin20:45:021
3929494993627,4cyclictest0-21swapper/000:00:440
3929494993623,10cyclictest3974715-21cron21:25:010
3929496993526,6cyclictest2170916-21cups-browsed19:30:362
392949699352,27cyclictest4025257-21sshd22:50:572
392949599351,9cyclictest4014953-21ssh22:31:191
392949599351,10cyclictest3936212-21memory19:30:401
3929494993530,3cyclictest4017908-21ssh22:40:010
3929494993530,3cyclictest4017908-21ssh22:40:010
3929494993525,7cyclictest4037794-21sed23:10:560
392949499351,28cyclictest3934968-21sshd19:25:480
392949499351,18cyclictest665-21snmpd19:45:510
3929496993427,4cyclictest3950623-21latency_hist20:20:022
3929496993425,6cyclictest421-21dbus-daemon22:15:012
3929496993424,7cyclictest4036164-21/usr/sbin/munin23:10:012
3929496993424,6cyclictest2170916-21cups-browsed23:19:302
392949699342,26cyclictest3966682-21sshd21:10:472
392949599343,8cyclictest1-21systemd19:10:011
3929494993427,4cyclictest4042277-21systemd-run23:20:010
3929494993427,4cyclictest3944507-21systemd-run20:00:010
3929494993427,4cyclictest1001-21mta-sts-daemon21:32:370
3929494993425,6cyclictest4066570-21cups-browsed00:33:260
3929494993422,8cyclictest4006766-21timerwakeupswit22:15:540
3929494993418,12cyclictest0-21swapper/022:45:570
3929496993326,4cyclictest3961026-21munin-plugin-st20:55:012
3929496993326,3cyclictest0-21swapper/223:40:012
3929496993325,5cyclictest3955023-21cron20:35:012
3929496993322,8cyclictest3980853-21munin-plugin-st21:35:012
3929496993321,9cyclictest4005772-21aten2.4-expect22:15:342
3929494993328,3cyclictest0-21swapper/023:45:560
3929494993326,4cyclictest1000-21mta-sts-daemon23:28:320
3929496993226,3cyclictest0-21swapper/223:25:362
3929496993226,3cyclictest0-21swapper/221:20:012
3929496993226,3cyclictest0-21swapper/200:15:002
3929496993225,3cyclictest0-21swapper/223:30:572
3929496993225,3cyclictest0-21swapper/223:30:572
3929496993224,5cyclictest3983911-21munin-plugin-st21:40:012
3929496993224,5cyclictest3983911-21munin-plugin-st21:40:012
3929496993224,5cyclictest3956519-21cron20:40:012
3929496993223,6cyclictest439-21systemd-logind20:15:012
3929494993223,6cyclictest3996099-21systemctl22:00:010
3929494993223,5cyclictest4017969-21kworker/u8:12-rpciod22:59:420
3929494993222,7cyclictest3933363-21ntp_kernel_pll_19:20:440
3929494993222,7cyclictest3933363-21ntp_kernel_pll_19:20:440
3929494993219,9cyclictest420-21avahi-daemon20:20:510
3929494993219,9cyclictest420-21avahi-daemon20:20:510
3929494993215,13cyclictest0-21swapper/020:15:470
3929496993124,4cyclictest1001-21mta-sts-daemon00:24:052
3929496993124,3cyclictest0-21swapper/223:25:022
3929496993123,5cyclictest4037892-21head23:10:572
392949699312,21cyclictest2170916-21cups-browsed19:50:462
392949499319,17cyclictest3987787-21awk21:45:390
3929494993126,2cyclictest0-21swapper/000:20:010
3929494993121,5cyclictest0-21swapper/000:12:530
3929494993120,8cyclictest0-21swapper/020:15:020
392949499311,23cyclictest3975885-21sshd21:25:470
392949699304,20cyclictest2170916-21cups-browsed20:05:482
3929496993027,2cyclictest0-21swapper/220:00:472
3929496993024,3cyclictest0-21swapper/200:33:372
3929496993021,5cyclictest3972194-21aten_r2power_po21:20:362
392949699301,23cyclictest4058518-21ssh23:45:442
3929494993024,4cyclictest3965460-21head21:06:020
3929494993023,3cyclictest0-21swapper/020:50:420
3929494993020,4cyclictest12750irq/33-eth019:10:440
392949499301,26cyclictest4081687-21ntpq00:25:480
3929496992923,4cyclictest3929493-21cyclictest22:32:552
3929496992921,5cyclictest3993046-21ssh21:55:002
3929496992921,4cyclictest1001-21mta-sts-daemon00:19:162
3929496992919,7cyclictest4063668-21latency_hist23:55:022
3929496992919,7cyclictest1001-21mta-sts-daemon22:36:172
3929496992919,7cyclictest1001-21mta-sts-daemon22:36:172
3929496992917,9cyclictest2170916-21cups-browsed21:05:022
3929494992922,5cyclictest2170916-21cups-browsed20:55:410
392949499292,21cyclictest3955910-21modprobe20:35:460
392949699287,14cyclictest428-21mta-sts-daemon22:00:492
392949699283,22cyclictest4022053-21ssh22:45:442
3929496992821,4cyclictest3931432-21cpuspeed19:15:372
3929496992821,4cyclictest1001-21mta-sts-daemon20:28:212
3929496992820,5cyclictest2170916-21cups-browsed19:45:022
3929496992819,6cyclictest567-21gdbus20:45:142
3929496992818,8cyclictest4083519-21latency_hist00:30:022
3929496992817,8cyclictest4010201-21sh22:21:332
3929494992821,5cyclictest4025336-21wc22:50:580
3929494992821,4cyclictest0-21swapper/020:30:470
3929494992820,5cyclictest4066570-21cups-browsed00:20:470
3929494992820,3cyclictest0-21swapper/021:37:180
3929494992820,3cyclictest0-21swapper/021:37:180
392949699278,15cyclictest421-21dbus-daemon20:20:512
392949699278,15cyclictest421-21dbus-daemon20:20:512
3929496992720,3cyclictest0-21swapper/219:10:462
3929496992719,6cyclictest0-21swapper/222:40:492
3929496992719,6cyclictest0-21swapper/222:40:492
3929494992720,3cyclictest0-21swapper/022:44:450
3929494992720,3cyclictest0-21swapper/022:44:450
3929496992620,3cyclictest0-21swapper/219:45:512
3929496992616,5cyclictest265-21systemd-journal20:45:012
3929496992615,8cyclictest3938489-21sendmail19:40:012
392949499265,15cyclictest665-21snmpd20:05:480
392949499262,18cyclictest2170916-21cups-browsed20:45:470
3929494992618,6cyclictest3942991-21sh19:55:010
3929494992613,10cyclictest3953548-21kworker/u8:12-xprtiod20:45:010
3929496992518,4cyclictest3962422-21unixbench_multi20:55:552
392949499258,14cyclictest3931504-21fschecks_time19:15:400
3929494992517,5cyclictest3982001-21kworker/u8:8-nfsiod22:34:110
3929496992418,3cyclictest0-21swapper/219:20:422
3929496992418,3cyclictest0-21swapper/219:20:422
3929496992215,4cyclictest3935371-21unixbench_multi19:25:552
3929496992214,5cyclictest265-21systemd-journal19:10:012
392949499221,16cyclictest3963413-21ntpq21:00:440
392949499221,16cyclictest3929632-21latency_hist19:10:020
392949899193,8cyclictest0-21swapper/323:35:513
392949899192,9cyclictest0-21swapper/300:25:453
392949899192,10cyclictest0-21swapper/323:50:573
392949899192,10cyclictest0-21swapper/300:30:383
392949899183,7cyclictest0-21swapper/320:15:433
392949899182,9cyclictest0-21swapper/322:25:453
392949899182,6cyclictest0-21swapper/321:05:483
392949899174,7cyclictest0-21swapper/322:15:473
392949899173,9cyclictest0-21swapper/321:20:463
392949899173,8cyclictest0-21swapper/323:20:483
392949899172,8cyclictest0-21swapper/321:10:473
392949899172,8cyclictest0-21swapper/320:03:483
392949899172,8cyclictest0-21swapper/300:15:503
392949899164,8cyclictest0-21swapper/300:20:453
392949899164,7cyclictest0-21swapper/322:40:433
392949899164,7cyclictest0-21swapper/322:40:433
392949899163,8cyclictest0-21swapper/300:05:453
392949899163,7cyclictest0-21swapper/323:55:503
392949899162,9cyclictest0-21swapper/321:35:513
392949899162,9cyclictest0-21swapper/321:35:513
392949899162,9cyclictest0-21swapper/320:30:473
392949899162,8cyclictest0-21swapper/323:30:483
392949899162,8cyclictest0-21swapper/323:30:483
392949899162,8cyclictest0-21swapper/323:25:363
392949899162,8cyclictest0-21swapper/323:05:393
392949899162,8cyclictest0-21swapper/322:50:323
392949899162,8cyclictest0-21swapper/322:20:433
392949899162,8cyclictest0-21swapper/321:30:363
392949899162,7cyclictest0-21swapper/320:10:443
392949899162,10cyclictest0-21swapper/322:45:393
392949899161,9cyclictest0-21swapper/323:10:493
392949899161,9cyclictest0-21swapper/322:55:463
392949899161,8cyclictest0-21swapper/323:00:533
392949899161,8cyclictest0-21swapper/322:05:523
392949899161,8cyclictest0-21swapper/320:40:453
392949899161,7cyclictest0-21swapper/320:55:463
392949899160,7cyclictest0-21swapper/319:40:463
392949899153,7cyclictest0-21swapper/323:45:383
392949899153,7cyclictest0-21swapper/323:15:463
392949899153,7cyclictest0-21swapper/321:45:453
392949899153,6cyclictest0-21swapper/319:55:013
392949899152,8cyclictest0-21swapper/323:40:473
392949899151,9cyclictest0-21swapper/321:55:453
392949899151,8cyclictest0-21swapper/322:10:403
392949899151,8cyclictest0-21swapper/322:00:463
392949899151,8cyclictest0-21swapper/321:52:473
392949899151,8cyclictest0-21swapper/319:45:513
392949899151,8cyclictest0-21swapper/300:35:453
392949899151,10cyclictest0-21swapper/321:25:363
392949899144,6cyclictest0-21swapper/300:03:143
392949899143,6cyclictest0-21swapper/321:00:473
392949899143,5cyclictest0-21swapper/320:45:463
392949899143,5cyclictest0-21swapper/320:25:463
392949899143,5cyclictest0-21swapper/319:25:463
392949899142,8cyclictest0-21swapper/322:31:043
392949899142,8cyclictest0-21swapper/319:20:013
392949899142,6cyclictest0-21swapper/319:20:433
392949899142,6cyclictest0-21swapper/319:20:433
392949899141,7cyclictest0-21swapper/320:55:023
392949899141,6cyclictest0-21swapper/319:55:453
392949899141,6cyclictest0-21swapper/319:35:443
392949899133,5cyclictest0-21swapper/321:40:343
392949899133,5cyclictest0-21swapper/321:40:343
392949899132,7cyclictest0-21swapper/321:15:413
392949899132,6cyclictest0-21swapper/300:12:533
392949899131,6cyclictest0-21swapper/319:30:443
392949899121,6cyclictest0-21swapper/320:35:463
392949899121,6cyclictest0-21swapper/319:10:443
392949899112,6cyclictest0-21swapper/320:20:473
392949899112,6cyclictest0-21swapper/320:20:473
392949899102,5cyclictest0-21swapper/322:38:253
392949899102,5cyclictest0-21swapper/322:38:253
392949899101,6cyclictest0-21swapper/320:05:393
39294989992,4cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional