You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-06 - 08:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Thu Aug 06, 2026 00:43:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
47083299565,43cyclictest551050-21ssh22:20:541
470833995444,6cyclictest583477-21ssh23:15:012
470831995240,9cyclictest265-21systemd-journal21:56:050
470831995137,11cyclictest604846-21cron23:50:010
470831995134,14cyclictest1-21systemd20:00:010
470833994936,9cyclictest588240-21unixbench_multi23:21:042
470833994936,9cyclictest588240-21unixbench_multi23:21:042
470832994941,5cyclictest494839-21systemd-run20:30:021
470832994838,6cyclictest0-21swapper/123:51:011
47083299483,39cyclictest609018-21ssh23:55:521
47083299483,38cyclictest630510-21ssh00:30:531
47083399461,40cyclictest476213-21ntpq19:25:512
470831994629,3cyclictest0-21swapper/022:35:560
470833994536,6cyclictest549834-21ssh22:20:012
470833994535,6cyclictest3057970-21cups-browsed22:51:002
470831994532,11cyclictest12650irq/32-eth023:55:530
470831994520,17cyclictest151ktimers/021:10:510
47083399442,36cyclictest538714-21ssh22:00:512
47083399442,34cyclictest535680-21ssh21:55:552
470833994418,22cyclictest0-21swapper/221:45:032
47083399441,36cyclictest497268-21ntpq20:35:552
470832994436,3cyclictest0-21swapper/121:55:011
470831994434,2cyclictest0-21swapper/020:40:010
470833994335,5cyclictest568129-21cron22:50:002
47083399432,34cyclictest504725-21ntpq21:00:522
47083399431,26cyclictest480921-21mailstats19:41:022
47083399431,24cyclictest506813-21diskmemload21:43:482
47083399431,24cyclictest506813-21diskmemload21:43:482
470832994335,5cyclictest421-21dbus-daemon22:05:011
470832994318,17cyclictest3057970-21cups-browsed22:40:551
470831994337,4cyclictest421-21dbus-daemon19:20:010
470833994232,6cyclictest554608-21unixbench_multi22:26:022
470833994232,6cyclictest554608-21unixbench_multi22:26:022
470833994219,20cyclictest0-21swapper/219:50:012
47083399421,36cyclictest499920-21/usr/sbin/munin20:45:452
470831994234,5cyclictest421-21dbus-daemon23:20:010
470831994233,6cyclictest3057970-21cups-browsed19:55:010
470831994233,6cyclictest3057970-21cups-browsed19:55:010
470831994231,8cyclictest265-21systemd-journal22:21:100
470833994125,12cyclictest620207-21latency_hist00:15:022
47083399412,33cyclictest578441-21ssh23:05:502
47083399411,37cyclictest556802-21/usr/sbin/munin22:30:462
47083399411,37cyclictest556802-21/usr/sbin/munin22:30:462
47083399411,37cyclictest501755-21ntpq20:50:532
47083399411,37cyclictest485168-21ntpq19:55:502
47083399411,35cyclictest572369-21ntpq22:55:572
47083399411,35cyclictest560274-21ntpq22:36:022
47083399411,35cyclictest494210-21ntpq20:25:522
470833994112,25cyclictest483437-21ethtool19:50:482
470833994112,25cyclictest483437-21ethtool19:50:482
470832994133,5cyclictest620187-21kworker/u8:1+xprtiod00:15:011
47083299413,29cyclictest421-21dbus-daemon00:35:581
470832994132,6cyclictest421-21dbus-daemon23:35:011
470832994112,5cyclictest171rcu_preempt22:25:511
470832994112,5cyclictest171rcu_preempt22:25:511
470831994133,4cyclictest572525-21processes22:56:030
470831994126,11cyclictest0-21swapper/019:25:560
470833994032,5cyclictest472317-21grep19:15:012
470833994032,5cyclictest1-21systemd23:40:012
470833994024,13cyclictest606333-21switchtime23:51:062
47083399402,34cyclictest532661-21ssh21:50:542
47083399402,33cyclictest627463-21ntpq00:25:522
47083399402,33cyclictest486709-21ntpq20:00:542
470833994021,16cyclictest479619-21timerwakeupswit19:36:012
470833994018,19cyclictest0-21swapper/200:25:022
47083399401,36cyclictest520409-21ntpq21:30:572
47083399401,36cyclictest520409-21ntpq21:30:572
47083399401,34cyclictest502796-21df_inode20:55:452
47083399401,34cyclictest498099-21/usr/sbin/munin20:40:412
47083399401,34cyclictest492704-21/usr/sbin/munin20:20:512
47083399401,34cyclictest477677-21ntpq19:30:492
47083399401,33cyclictest508009-21ntpq21:10:532
47083399401,31cyclictest511102-21ssh21:15:552
470833993930,7cyclictest475346-21sh19:25:012
47083399392,20cyclictest506813-21diskmemload23:19:012
47083399391,35cyclictest614567-21/usr/sbin/munin00:05:452
47083399391,35cyclictest590663-21ntpq23:25:502
47083399391,34cyclictest491316-21open_inodes20:15:582
47083399391,34cyclictest488253-21/usr/sbin/munin20:05:572
47083399391,34cyclictest473198-21ntpq19:15:532
47083399391,33cyclictest541761-21ntpq22:05:502
470832993932,3cyclictest0-21swapper/122:00:011
470832993929,6cyclictest485858-21latency_hist20:00:011
470832993928,6cyclictest525379-21latency_hist21:40:011
470832993928,6cyclictest525379-21latency_hist21:40:011
470832993927,9cyclictest504626-21needreboot21:00:481
470831993932,4cyclictest577437-21latency_hist23:05:020
470833993828,6cyclictest420-21avahi-daemon00:15:042
47083399382,32cyclictest506239-21ntpq21:05:552
47083399381,34cyclictest563216-21ntpq22:40:562
47083399381,33cyclictest629800-21/usr/sbin/munin00:31:072
47083399381,33cyclictest495735-21ntpq20:30:562
47083399381,31cyclictest599964-21ntpq23:40:552
47083299385,19cyclictest428-21mta-sts-daemon22:15:511
470832993830,5cyclictest624265-21grep00:20:491
470832993827,4cyclictest0-21swapper/121:10:501
470831993830,5cyclictest596202-21aten_r2power_fr23:35:420
470831993829,7cyclictest0-21swapper/021:25:480
470831993826,8cyclictest12650irq/32-eth022:00:580
47083199382,30cyclictest587780-21ssh23:20:570
47083199382,30cyclictest587780-21ssh23:20:570
470833993727,6cyclictest420-21avahi-daemon21:21:242
470833993724,10cyclictest0-21swapper/221:25:462
47083399371,31cyclictest633197-21/usr/sbin/munin00:35:462
47083399371,29cyclictest489304-21df20:10:432
47083399370,34cyclictest544935-21ntpq22:10:542
470832993730,4cyclictest0-21swapper/123:25:491
470832993730,3cyclictest0-21swapper/122:51:031
470832993729,5cyclictest612708-21unixbench_multi00:01:051
470832993727,7cyclictest3057970-21cups-browsed19:25:561
470832993727,7cyclictest0-21swapper/123:00:441
470832993726,7cyclictest420-21avahi-daemon23:45:021
470831993722,9cyclictest12650irq/32-eth021:15:590
47083399363,27cyclictest575430-21ntpq23:00:542
470833993630,3cyclictest0-21swapper/222:25:012
47083399362,30cyclictest602891-21ntpq23:45:492
47083399362,30cyclictest523432-21ntpq21:35:542
47083399362,30cyclictest523432-21ntpq21:35:542
47083399361,31cyclictest593774-21ntpq23:30:542
47083399361,30cyclictest612782-21users00:01:052
470832993628,4cyclictest0-21swapper/122:35:011
470832993628,4cyclictest0-21swapper/122:35:011
470832993628,4cyclictest0-21swapper/120:40:501
470832993628,4cyclictest0-21swapper/119:35:481
470832993627,5cyclictest0-21swapper/121:26:061
470832993627,4cyclictest0-21swapper/122:36:031
470832993626,5cyclictest0-21swapper/121:15:511
470831993631,2cyclictest0-21swapper/020:55:010
470831993630,4cyclictest590891-21wc23:25:590
470831993630,4cyclictest549850-21munin-run22:20:010
470831993629,4cyclictest0-21swapper/000:19:170
470831993629,3cyclictest0-21swapper/021:30:450
470831993629,3cyclictest0-21swapper/021:30:450
47083399351,29cyclictest506813-21diskmemload23:58:022
470832993528,4cyclictest531465-21munin-run21:50:001
470832993527,5cyclictest485-21in:imuxsock20:20:011
470832993527,5cyclictest478214-21unixbench_multi19:31:011
470832993525,7cyclictest1002-21mta-sts-daemon00:25:521
470832993525,6cyclictest28-21ksoftirqd/119:41:011
470831993528,5cyclictest1000-21mta-sts-daemon20:10:450
470831993528,4cyclictest581983-21timerwakeupswit23:11:030
470831993528,4cyclictest541831-21ntp_kernel_pll_22:05:520
470831993527,5cyclictest580382-21ssh23:10:010
470832993426,5cyclictest545414-21unixbench_multi22:11:031
470832993426,4cyclictest0-21swapper/122:10:021
470832993425,6cyclictest623232-21ls00:20:011
470832993421,10cyclictest420-21avahi-daemon22:46:031
47083199344,24cyclictest554079-21ssh22:25:510
47083199344,24cyclictest554079-21ssh22:25:510
470831993427,4cyclictest630131-21fschecks_time00:30:460
470831993425,6cyclictest479743-21head19:36:010
47083299333,20cyclictest587642-21ssh23:20:511
47083299333,20cyclictest587642-21ssh23:20:511
470832993326,5cyclictest528410-21munin-run21:45:011
470832993326,5cyclictest528410-21munin-run21:45:011
470832993326,4cyclictest497913-21latency_hist20:40:021
470832993325,5cyclictest572866-21unixbench-2d22:56:081
470832993323,5cyclictest0-21swapper/120:45:561
47083299332,22cyclictest596808-21ssh23:35:531
470831993327,4cyclictest505347-21systemd-run21:05:010
47083199332,24cyclictest599927-21ssh23:40:530
470831993321,7cyclictest151ktimers/019:41:010
470831993319,11cyclictest0-21swapper/020:30:010
470832993224,5cyclictest493245-21head20:21:031
470832993224,4cyclictest0-21swapper/123:45:511
470832993224,4cyclictest0-21swapper/100:05:571
47083299322,22cyclictest581628-21ssh23:10:561
47083199323,21cyclictest506813-21diskmemload21:45:570
470831993227,2cyclictest0-21swapper/019:50:010
470831993226,4cyclictest523180-21grep21:35:470
470831993226,4cyclictest523180-21grep21:35:470
470831993225,5cyclictest1-21systemd21:00:020
47083299315,20cyclictest584553-21ssh23:15:491
470832993123,5cyclictest472504-21latency_hist19:15:031
470831993126,3cyclictest635517-21kworker/u8:300:40:010
470831993125,4cyclictest626979-21sort00:25:410
470831993125,4cyclictest472213-21tail19:11:030
470831993124,3cyclictest0-21swapper/022:51:000
470831993122,7cyclictest557658-21grep22:31:030
470831993122,7cyclictest557658-21grep22:31:030
47083199312,27cyclictest0-21swapper/000:10:490
470831993121,8cyclictest495915-21mailstats20:31:030
470831993119,8cyclictest12650irq/32-eth023:30:510
47083199311,25cyclictest614787-21df00:05:470
47083399301,26cyclictest470790-21munin-run19:10:012
470832993022,5cyclictest522357-21latency_hist21:35:031
470832993022,5cyclictest522357-21latency_hist21:35:031
470832993022,5cyclictest503148-21ntp_kernel_err20:55:511
470831993018,9cyclictest614046-21cron00:05:010
470831993018,9cyclictest0-21swapper/022:15:010
470831993016,11cyclictest525892-21head21:40:430
470831993016,11cyclictest525892-21head21:40:430
47083199293,10cyclictest171rcu_preempt21:20:540
470831992924,3cyclictest605628-21gltestperf23:50:480
470831992922,5cyclictest532973-21ssh21:51:030
470831992912,12cyclictest12650irq/32-eth020:15:520
470831992911,14cyclictest12650irq/32-eth022:41:040
47083299286,17cyclictest514173-21ssh21:20:551
470832992822,2cyclictest0-21swapper/120:05:001
470832992821,4cyclictest506808-21cron21:10:011
470832992820,5cyclictest577907-21aten_r2power_po23:05:411
470831992812,9cyclictest151ktimers/019:20:540
470832992722,2cyclictest0-21swapper/120:35:011
470831992721,3cyclictest0-21swapper/021:10:010
470832992620,3cyclictest0-21swapper/119:50:531
470832992620,3cyclictest0-21swapper/119:50:531
470831992618,5cyclictest265-21systemd-journal22:50:010
47083299253,13cyclictest428-21mta-sts-daemon20:10:511
470832992517,5cyclictest482842-21systemd-run19:50:011
470832992516,3cyclictest0-21swapper/119:20:541
470831992519,4cyclictest487648-21aten_r2power_fr20:05:450
470831992517,5cyclictest3057970-21cups-browsed19:34:000
470831992515,7cyclictest487318-21cron20:05:000
470832992419,2cyclictest0-21swapper/119:10:011
470832992418,3cyclictest0-21swapper/120:10:001
470831992418,3cyclictest0-21swapper/000:21:050
470831992416,5cyclictest0-21swapper/020:20:530
470831992412,9cyclictest0-21swapper/020:50:020
470832992316,4cyclictest501178-21aten2.4-expect20:50:411
47083299231,17cyclictest473179-21awk19:15:521
470831992217,3cyclictest1000-21mta-sts-daemon19:10:030
470831992215,5cyclictest1002-21mta-sts-daemon20:40:480
47083499204,7cyclictest0-21swapper/322:30:563
47083499204,7cyclictest0-21swapper/322:30:563
47083499195,9cyclictest0-21swapper/322:15:523
47083499195,6cyclictest0-21swapper/300:06:003
47083499184,7cyclictest0-21swapper/323:30:523
47083499184,7cyclictest0-21swapper/321:15:533
47083499183,8cyclictest0-21swapper/321:30:563
47083499183,8cyclictest0-21swapper/321:30:563
47083499182,10cyclictest0-21swapper/300:00:013
47083499181,10cyclictest0-21swapper/321:20:553
47083499172,8cyclictest0-21swapper/323:40:573
47083499172,8cyclictest0-21swapper/322:10:503
47083499172,7cyclictest0-21swapper/322:40:563
47083499171,6cyclictest0-21swapper/319:50:543
47083499171,6cyclictest0-21swapper/319:50:543
47083499163,8cyclictest0-21swapper/323:51:003
47083499163,8cyclictest0-21swapper/322:55:573
47083499163,7cyclictest0-21swapper/300:35:573
47083499162,8cyclictest0-21swapper/323:15:513
47083499162,8cyclictest0-21swapper/322:35:563
47083499162,7cyclictest0-21swapper/321:50:523
47083499162,7cyclictest0-21swapper/321:40:593
47083499162,7cyclictest0-21swapper/321:40:593
47083499162,7cyclictest0-21swapper/319:15:563
47083499162,5cyclictest0-21swapper/319:20:523
47083499161,8cyclictest0-21swapper/322:45:563
47083499161,7cyclictest0-21swapper/319:45:503
47083499153,7cyclictest0-21swapper/321:25:523
47083499153,6cyclictest0-21swapper/321:00:503
47083499153,5cyclictest0-21swapper/320:00:563
47083499152,8cyclictest0-21swapper/323:35:513
47083499152,8cyclictest0-21swapper/323:20:583
47083499152,8cyclictest0-21swapper/323:20:583
47083499152,8cyclictest0-21swapper/321:35:513
47083499152,8cyclictest0-21swapper/321:35:513
47083499152,8cyclictest0-21swapper/300:10:483
47083499152,7cyclictest0-21swapper/323:05:513
47083499152,7cyclictest0-21swapper/300:30:533
47083499152,6cyclictest0-21swapper/323:45:493
47083499152,6cyclictest0-21swapper/320:25:503
47083499152,6cyclictest0-21swapper/320:20:503
47083499151,9cyclictest0-21swapper/321:55:593
47083499151,7cyclictest0-21swapper/300:20:583
47083499151,6cyclictest0-21swapper/320:15:533
47083499151,6cyclictest0-21swapper/320:10:513
47083499144,5cyclictest0-21swapper/322:05:563
47083499143,6cyclictest0-21swapper/321:10:553
47083499143,5cyclictest0-21swapper/320:35:563
47083499143,5cyclictest0-21swapper/300:00:533
47083499142,8cyclictest0-21swapper/300:25:533
47083499142,7cyclictest0-21swapper/323:10:503
47083499142,7cyclictest0-21swapper/323:02:123
47083499142,6cyclictest0-21swapper/320:40:513
47083499142,6cyclictest0-21swapper/319:40:543
47083499141,7cyclictest0-21swapper/322:25:473
47083499141,7cyclictest0-21swapper/322:25:473
47083499141,6cyclictest0-21swapper/320:50:513
47083499141,6cyclictest0-21swapper/319:30:503
47083499134,5cyclictest0-21swapper/322:50:503
47083499133,6cyclictest0-21swapper/321:45:513
47083499133,6cyclictest0-21swapper/300:15:463
47083499133,5cyclictest0-21swapper/320:05:573
47083499132,6cyclictest0-21swapper/322:20:513
47083499132,6cyclictest0-21swapper/320:00:013
47083499131,7cyclictest0-21swapper/323:25:503
47083499131,7cyclictest0-21swapper/322:00:513
47083499131,6cyclictest0-21swapper/321:05:553
47083499131,6cyclictest0-21swapper/320:30:563
47083499122,5cyclictest0-21swapper/320:55:523
47083499122,5cyclictest0-21swapper/319:25:463
47083499121,6cyclictest0-21swapper/319:35:513
47083499121,6cyclictest0-21swapper/319:10:563
47083499101,6cyclictest0-21swapper/320:50:023
4708349992,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional