You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-05 - 06:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Wed Aug 05, 2026 00:43:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2917796995645,7cyclictest3016565-21threads22:51:031
291779699512,39cyclictest2988635-21ssh22:05:521
2917797994940,6cyclictest3042636-21sh23:35:012
2917797994924,3cyclictest171rcu_preempt19:30:522
2917795994825,16cyclictest151ktimers/020:40:530
2917797994735,8cyclictest3053540-21head23:51:042
291779799472,36cyclictest958-21sendmail-mta20:50:532
291779799471,4cyclictest2953771-21diskmemload22:45:512
2917795994732,12cyclictest3068830-21ssh00:16:040
2917797994638,5cyclictest3036511-21ssh23:25:002
2917797994638,5cyclictest3027335-21latency_hist23:10:022
2917796994617,12cyclictest2935128-21ntpq20:05:501
2917795994639,4cyclictest151ktimers/020:55:530
2917796994521,18cyclictest3043761-21ntpq23:35:551
2917796994518,9cyclictest3049903-21ntpq23:45:541
2917795994537,5cyclictest0-21swapper/023:20:540
2917795994525,13cyclictest151ktimers/019:20:570
2917795994525,13cyclictest151ktimers/019:20:570
2917796994426,13cyclictest271ktimers/122:40:511
2917796994420,9cyclictest2952422-21date21:05:031
2917796994418,8cyclictest3057883-21ssh23:59:591
2917795994418,18cyclictest151ktimers/019:25:560
2917796994323,15cyclictest421-21dbus-daemon00:00:531
2917796994317,16cyclictest0-21swapper/119:30:521
2917797994233,5cyclictest2996692-21munin-run22:20:002
2917797994232,6cyclictest439-21systemd-logind20:00:022
291779699423,30cyclictest2985630-21ssh22:00:551
2917796994229,10cyclictest3068672-21timerandwakeup00:16:031
2917796994228,8cyclictest2930699-21ntpq19:50:541
2917796994217,7cyclictest3026583-21sh23:07:481
2917796994217,7cyclictest2944183-21ntpq20:35:581
2917796994215,12cyclictest3005111-21ssh22:32:401
2917795994230,8cyclictest151ktimers/023:25:550
2917797994133,5cyclictest3048746-21systemd-run23:45:012
2917797994132,6cyclictest2952303-21systemd-run21:05:022
2917797994130,7cyclictest2969683-21aten_r2power_en21:35:432
2917797994129,6cyclictest0-21swapper/200:25:542
2917796994134,4cyclictest3082420-21cron00:40:001
2917796994134,4cyclictest3082420-21cron00:40:001
2917796994128,8cyclictest2939656-21ntpq20:20:481
2917796994127,8cyclictest2921676-21ntpq19:20:531
2917796994127,8cyclictest2921676-21ntpq19:20:531
2917796994126,11cyclictest2947750-21sshd20:46:331
2917796994117,9cyclictest3062791-21users00:06:091
2917796994117,8cyclictest1002-21mta-sts-daemon20:10:521
2917795994128,10cyclictest2944638-21unixbench_multi20:36:080
2917795994125,12cyclictest3071401-21ssh00:20:570
2917797994032,5cyclictest3070229-21systemd00:20:022
2917797994031,6cyclictest2944813-21systemd-run20:40:012
291779799401,6cyclictest171rcu_preempt22:40:552
2917796994031,5cyclictest2999758-21cron22:25:001
2917796994028,9cyclictest2983033-21timerwakeupswit21:56:031
291779699402,29cyclictest3077451-21rm00:30:521
2917795994033,4cyclictest2999757-21cron22:25:000
2917795994033,4cyclictest2988412-21kernelversion22:05:490
2917795994033,4cyclictest2949341-21latency_hist20:55:020
2917795994031,6cyclictest3046858-21ssh23:40:540
2917797993930,6cyclictest2960055-21latency_hist21:20:032
2917797993930,5cyclictest0-21swapper/222:55:552
2917797993927,9cyclictest3071684-21switchtime00:21:052
2917796993927,7cyclictest2941270-21ntpq20:26:001
2917796993926,8cyclictest3028452-21ntpq23:10:531
2917796993925,11cyclictest2934265-21sshd20:03:341
2917796993925,10cyclictest2919504-21/usr/sbin/munin19:16:061
291779699392,31cyclictest3040693-21ssh23:30:511
2917796993915,9cyclictest2962554-21ssh21:23:371
2917795993932,4cyclictest2972330-21latency_hist21:40:020
2917795993931,5cyclictest2987555-21systemd-run22:05:020
2917795993928,9cyclictest3032060-21unixbench_multi23:16:040
2917795993920,13cyclictest12650irq/32-eth021:50:490
2917797993828,5cyclictest0-21swapper/221:05:482
291779799381,3cyclictest70-21kswapd000:30:562
291779699384,28cyclictest3034542-21ssh23:20:501
2917796993830,5cyclictest3071826-21timerwakeupswit00:21:061
2917796993828,7cyclictest421-21dbus-daemon21:00:021
2917796993825,9cyclictest2938114-21ntpq20:15:491
2917796993825,9cyclictest2926118-21ntpq19:35:501
2917796993824,11cyclictest2932123-21ntpq19:55:501
2917796993824,10cyclictest2945633-21ntpq20:40:521
2917796993824,10cyclictest1-21systemd21:35:441
2917796993821,12cyclictest3008098-21ssh22:37:301
2917796993820,12cyclictest3074354-21ssh00:25:511
2917796993817,7cyclictest2996751-21cat22:20:011
2917796993815,7cyclictest3024170-21sh23:04:481
2917795993832,3cyclictest1002-21mta-sts-daemon20:20:480
2917795993827,8cyclictest151ktimers/019:35:540
2917795993826,9cyclictest3056192-21wc23:55:580
2917795993818,14cyclictest421-21dbus-daemon20:25:550
2917797993729,5cyclictest2946307-21systemd-run20:45:022
2917797993729,5cyclictest1-21systemd22:10:022
291779799370,5cyclictest171rcu_preempt23:55:492
2917796993729,5cyclictest2973992-21unixbench_singl21:41:061
2917796993728,3cyclictest0-21swapper/100:10:451
2917796993727,7cyclictest2955223-21/usr/sbin/munin21:11:031
2917796993727,6cyclictest1215473-21cups-browsed23:50:151
2917796993726,7cyclictest3037754-21ntpq23:25:561
2917796993724,9cyclictest421-21dbus-daemon22:56:171
2917796993724,10cyclictest2922824-21/usr/sbin/munin19:25:461
2917796993722,11cyclictest2927768-21ntpq19:40:591
2917796993718,7cyclictest2948229-21awk20:50:441
2917796993715,12cyclictest171rcu_preempt21:52:591
2917795993730,5cyclictest421-21dbus-daemon22:30:010
2917795993729,5cyclictest2992344-21unixbench_singl22:11:040
2917795993728,6cyclictest420-21avahi-daemon22:35:340
2917795993726,4cyclictest1002-21mta-sts-daemon21:12:130
2917795993725,9cyclictest12650irq/32-eth022:45:580
291779799364,27cyclictest2997912-21ssh22:20:572
2917796993626,7cyclictest3000905-21ntpq22:25:521
2917796993625,7cyclictest2976530-21ntp_states21:45:541
2917796993625,7cyclictest2929254-21ntpq19:45:571
2917796993624,9cyclictest2966799-21/usr/sbin/munin21:30:431
2917796993624,9cyclictest2918708-21/usr/sbin/munin19:10:531
2917796993624,9cyclictest2918708-21/usr/sbin/munin19:10:531
2917796993623,10cyclictest3032003-21/usr/sbin/munin23:16:031
2917796993621,12cyclictest2958739-21ssh21:16:251
2917796993618,14cyclictest2993340-21ssh22:14:061
2917796993612,8cyclictest3047335-21timerwakeupswit23:41:061
291779599369,22cyclictest2958006-21ssh21:15:530
2917795993628,5cyclictest3061004-21cut00:05:010
2917795993627,7cyclictest2977021-21grep21:46:020
2917795993620,11cyclictest151ktimers/020:45:520
2917797993529,3cyclictest0-21swapper/222:40:012
2917797993527,5cyclictest3050424-21grep23:46:042
2917797993527,5cyclictest3018180-21latency_hist22:55:022
2917797993527,5cyclictest3002838-21systemd22:30:012
2917797993526,5cyclictest3080739-21sendmail_mailqu00:36:032
2917797993526,5cyclictest3080739-21sendmail_mailqu00:36:032
291779799352,26cyclictest3059044-21ssh00:00:502
2917796993523,8cyclictest2942619-21/usr/sbin/munin20:30:501
2917796993523,8cyclictest2942619-21/usr/sbin/munin20:30:501
2917796993522,9cyclictest2953118-21/usr/sbin/munin21:05:511
2917796993521,11cyclictest3013881-21ssh22:46:241
2917795993527,6cyclictest0-21swapper/023:05:520
2917795993525,7cyclictest2963950-21latency21:25:450
2917795993524,8cyclictest1215473-21cups-browsed21:05:470
2917795993523,4cyclictest2919166-21unixbench_multi19:11:020
2917795993523,4cyclictest2919166-21unixbench_multi19:11:020
2917795993514,15cyclictest3043750-21ssh23:35:540
291779799344,25cyclictest3062181-21ssh00:05:572
2917797993427,4cyclictest1003-21mta-sts-daemon21:56:572
2917797993427,3cyclictest0-21swapper/221:26:012
2917797993426,4cyclictest0-21swapper/221:30:452
2917797993425,5cyclictest3043226-21aten_r2power_vo23:35:452
2917797993423,7cyclictest0-21swapper/219:10:502
2917797993423,7cyclictest0-21swapper/219:10:502
2917796993420,10cyclictest2963255-21ssh21:25:041
291779599343,27cyclictest3010025-21ssh22:40:500
2917795993428,3cyclictest0-21swapper/023:00:530
2917795993426,5cyclictest3057970-21cups-browsed00:07:440
2917797993327,4cyclictest2993651-21awk22:15:012
2917797993326,4cyclictest2944805-21kworker/u8:11+xprtiod21:25:012
2917797993322,8cyclictest2950135-21ntp_kernel_pll_20:55:542
291779599333,23cyclictest421-21dbus-daemon21:30:550
2917795993327,4cyclictest1002-21mta-sts-daemon23:34:270
2917795993322,7cyclictest3052745-21/usr/sbin/munin23:50:460
2917795993319,10cyclictest0-21swapper/022:00:020
291779799325,9cyclictest171rcu_preempt19:35:542
2917797993224,5cyclictest439-21systemd-logind20:10:012
2917797993224,5cyclictest1-21systemd19:45:022
2917797993224,5cyclictest1003-21mta-sts-daemon22:03:472
2917797993223,6cyclictest2943290-21systemd-run20:35:012
2917797993223,6cyclictest2943290-21systemd-run20:35:012
2917797993223,6cyclictest1215473-21cups-browsed21:48:102
2917797993222,6cyclictest487-21rs:main2
2917797993221,8cyclictest3038125-21timerwakeupswit23:26:042
2917797993219,9cyclictest1003-21mta-sts-daemon19:15:512
291779799321,24cyclictest2955053-21ssh21:10:592
2917795993225,4cyclictest2961320-21plymouth21:21:010
2917795993214,13cyclictest3064885-21if_wlan000:10:460
291779799313,4cyclictest171rcu_preempt20:10:522
291779799312,5cyclictest171rcu_preempt23:00:552
2917797993124,4cyclictest2943500-21kworker/u8:5-xprtiod22:35:022
2917797993123,4cyclictest0-21swapper/221:55:022
291779599318,15cyclictest12650irq/32-eth000:35:570
291779599318,15cyclictest12650irq/32-eth000:35:570
2917795993122,6cyclictest3077713-21mailstats00:31:000
2917795993122,6cyclictest2931267-21cron19:55:010
291779799303,17cyclictest2973369-21ssh21:40:532
2917797993023,4cyclictest2931162-21unixbench_multi19:51:032
2917797993020,7cyclictest2941783-21/usr/sbin/munin20:30:012
2917795993024,4cyclictest1001-21mta-sts-daemon22:32:560
2917795993020,8cyclictest2937266-21cron20:15:010
291779799294,17cyclictest3065288-21ssh00:10:572
2917797992922,3cyclictest0-21swapper/223:15:452
2917795992923,3cyclictest0-21swapper/019:45:550
2917795992922,5cyclictest12650irq/32-eth023:10:550
291779599292,22cyclictest3074487-21ssh00:25:560
2917795992921,5cyclictest1002-21mta-sts-daemon23:45:250
2917795992919,7cyclictest1001-21mta-sts-daemon22:53:120
2917795992916,8cyclictest265-21systemd-journal21:41:320
2917797992822,3cyclictest0-21swapper/220:20:512
2917797992822,3cyclictest0-21swapper/220:05:012
2917796992818,7cyclictest2917754-21munin-run19:10:011
2917795992823,3cyclictest2938812-21kworker/u8:12+xprtiod20:20:010
2917795992821,5cyclictest1215473-21cups-browsed19:57:590
2917795992815,9cyclictest0-21swapper/022:55:520
2917795992811,12cyclictest151ktimers/019:15:560
2917797992719,3cyclictest0-21swapper/223:10:552
2917797992717,7cyclictest2946825-21gltestperf20:45:432
2917795992721,3cyclictest0-21swapper/021:00:560
2917795992716,7cyclictest12650irq/32-eth019:40:530
2917795992620,3cyclictest0-21swapper/022:15:400
2917797992518,5cyclictest1215473-21cups-browsed19:23:382
2917797992518,5cyclictest1215473-21cups-browsed19:23:382
2917795992516,7cyclictest2933653-21ntp_kernel_pll_20:00:510
2917795992419,2cyclictest0-21swapper/020:06:030
2917795992419,2cyclictest0-21swapper/019:30:420
2917795992418,3cyclictest0-21swapper/020:34:330
2917795992418,3cyclictest0-21swapper/020:34:330
2917797992316,5cyclictest2917794-21cyclictest19:25:452
291779899213,7cyclictest0-21swapper/319:55:553
2917797992114,3cyclictest0-21swapper/219:10:012
2917797992111,7cyclictest2935758-21kworker/u8:3+events_unbound20:15:502
291779899203,8cyclictest0-21swapper/323:45:553
291779899203,7cyclictest0-21swapper/322:40:523
291779899202,8cyclictest0-21swapper/319:50:523
291779899192,8cyclictest0-21swapper/322:15:533
291779899185,6cyclictest0-21swapper/300:15:563
291779899182,9cyclictest0-21swapper/323:30:563
291779899182,9cyclictest0-21swapper/323:20:513
291779899182,9cyclictest0-21swapper/323:00:513
291779899182,9cyclictest0-21swapper/300:30:543
291779899182,10cyclictest0-21swapper/321:50:533
291779599185,8cyclictest0-21swapper/019:10:010
291779899174,7cyclictest0-21swapper/323:25:583
291779899173,8cyclictest0-21swapper/322:00:563
291779899173,8cyclictest0-21swapper/300:20:573
291779899172,9cyclictest0-21swapper/321:10:573
291779899172,8cyclictest0-21swapper/322:30:513
291779899172,8cyclictest0-21swapper/321:55:503
291779899172,7cyclictest0-21swapper/323:05:533
291779899163,8cyclictest0-21swapper/323:15:513
291779899163,7cyclictest0-21swapper/323:40:543
291779899163,7cyclictest0-21swapper/322:50:503
291779899163,7cyclictest0-21swapper/322:45:523
291779899163,7cyclictest0-21swapper/322:05:543
291779899163,7cyclictest0-21swapper/321:35:553
291779899163,7cyclictest0-21swapper/300:00:423
291779899162,7cyclictest0-21swapper/322:25:553
291779899161,9cyclictest0-21swapper/321:41:323
291779899161,8cyclictest0-21swapper/320:45:513
291779899154,6cyclictest0-21swapper/322:35:543
291779899154,6cyclictest0-21swapper/300:40:013
291779899154,6cyclictest0-21swapper/300:40:013
291779899153,7cyclictest0-21swapper/322:20:473
291779899153,7cyclictest0-21swapper/300:10:523
291779899153,4cyclictest0-21swapper/319:45:513
291779899152,9cyclictest0-21swapper/319:15:013
291779899152,9cyclictest0-21swapper/319:15:013
291779899152,8cyclictest0-21swapper/323:10:513
291779899152,7cyclictest0-21swapper/323:50:493
291779899152,7cyclictest0-21swapper/323:35:463
291779899152,7cyclictest0-21swapper/322:55:553
291779899152,7cyclictest0-21swapper/321:30:543
291779899152,7cyclictest0-21swapper/300:25:533
291779899152,6cyclictest0-21swapper/321:00:543
291779899151,9cyclictest0-21swapper/321:49:443
291779899151,8cyclictest0-21swapper/323:55:443
291779899151,7cyclictest0-21swapper/321:20:553
291779899151,6cyclictest0-21swapper/320:30:513
291779899151,6cyclictest0-21swapper/320:30:513
291779899144,7cyclictest0-21swapper/322:10:523
291779899142,7cyclictest0-21swapper/320:10:523
291779899142,5cyclictest0-21swapper/320:25:533
291779899142,5cyclictest0-21swapper/319:15:513
291779899141,8cyclictest0-21swapper/321:25:383
291779899141,7cyclictest0-21swapper/319:35:503
291779899141,5cyclictest0-21swapper/320:50:523
291779899133,5cyclictest0-21swapper/300:05:463
291779899132,7cyclictest0-21swapper/321:10:013
291779899132,6cyclictest0-21swapper/320:00:553
291779899132,5cyclictest0-21swapper/320:55:543
291779899131,6cyclictest0-21swapper/319:25:523
291779899130,6cyclictest0-21swapper/319:20:523
291779899130,6cyclictest0-21swapper/319:20:523
291779899124,6cyclictest0-21swapper/321:15:503
291779899122,5cyclictest0-21swapper/320:35:533
291779899121,6cyclictest0-21swapper/320:40:593
291779899121,6cyclictest0-21swapper/320:15:503
291779899111,6cyclictest0-21swapper/320:20:483
291779899111,6cyclictest0-21swapper/320:05:433
291779899111,6cyclictest0-21swapper/319:40:473
291779899111,6cyclictest0-21swapper/319:30:503
29177989981,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional