You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-01 - 01:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Mon Jun 01, 2026 00:42:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3691187997257,11cyclictest1017-21sendmail-mta22:05:561
3691188996049,6cyclictest3853380-21ssh00:30:012
3691188996049,6cyclictest3853380-21ssh00:30:012
3691186995446,4cyclictest3768567-21munin-plugin-st22:15:010
3691188995328,21cyclictest1-21systemd00:00:012
3691187995245,4cyclictest1051-21mta-sts-daemon21:31:071
3691188995027,19cyclictest0-21swapper/223:40:002
3691188994926,19cyclictest3704738-21latency_hist19:55:012
3691186994938,9cyclictest3821359-21unixbench_multi23:36:130
3691186994933,12cyclictest12350irq/32-eth000:21:040
3691186994933,12cyclictest12350irq/32-eth000:21:040
369118899479,15cyclictest1952297-21cupsd20:16:062
3691188994737,7cyclictest3733537-21idleruntime-cro21:20:002
3691188994720,24cyclictest3727175-21diskmemload22:55:542
369118899462,19cyclictest3744325-21ssh21:36:032
3691186994635,5cyclictest151ktimers/022:40:590
3691186994633,9cyclictest151ktimers/021:26:050
3691188994532,9cyclictest0-21swapper/220:51:032
369118899452,19cyclictest3811318-21ssh23:21:042
369118899451,20cyclictest3769883-21ssh22:16:052
3691187994536,6cyclictest3724198-21systemd-run21:00:011
3691186994518,15cyclictest3833515-21ssh23:56:060
369118899443,12cyclictest3808035-21ssh23:16:002
3691188994417,5cyclictest1952300-21cups-browsed22:48:442
3691187994436,4cyclictest0-21swapper/122:15:021
3691187994435,6cyclictest3739905-21systemd-run21:30:011
3691188994319,3cyclictest0-21swapper/222:10:532
3691188994315,3cyclictest0-21swapper/222:21:032
3691187994328,10cyclictest0-21swapper/122:36:021
369118699433,16cyclictest3854574-21ssh00:31:000
369118699433,16cyclictest3854574-21ssh00:31:000
3691186994319,16cyclictest3818683-21ssh23:32:300
3691186994319,16cyclictest3818683-21ssh23:32:300
3691186994317,11cyclictest3788795-21missed_timers22:45:570
369118899424,12cyclictest428-21mta-sts-daemon22:31:012
369118899424,12cyclictest428-21mta-sts-daemon22:31:012
3691188994234,5cyclictest3750689-21ssh21:46:022
3691188994220,4cyclictest1051-21mta-sts-daemon20:46:042
3691187994233,6cyclictest421-21dbus-daemon20:35:021
3691187994230,9cyclictest421-21dbus-daemon22:00:021
3691186994227,12cyclictest3695081-21ntpq19:21:030
3691186994219,10cyclictest3805401-21tail23:11:120
369118899413,32cyclictest3848360-21ssh00:21:092
369118899413,32cyclictest3848360-21ssh00:21:092
3691188994113,7cyclictest3844573-21cpuspeed00:15:542
3691188994113,7cyclictest3844573-21cpuspeed00:15:542
369118799415,28cyclictest3848369-21ssh00:21:091
369118799415,28cyclictest3848369-21ssh00:21:091
369118799415,22cyclictest3835474-21cups-browsed00:11:001
369118799412,28cyclictest3804911-21ssh23:11:031
3691186994130,7cyclictest0-21swapper/023:41:020
3691186994127,8cyclictest3696568-21ntpq19:26:030
3691186994127,8cyclictest3696568-21ntpq19:26:030
3691186994126,11cyclictest3858639-21ssh00:36:520
3691188994033,4cyclictest0-21swapper/223:01:032
3691188994033,3cyclictest0-21swapper/222:01:042
3691188994031,4cyclictest0-21swapper/221:56:072
3691188994030,7cyclictest3703174-21/usr/sbin/munin19:50:012
3691188994030,7cyclictest3703174-21/usr/sbin/munin19:50:012
3691188994016,5cyclictest1952300-21cups-browsed19:10:572
369118799408,26cyclictest3795337-21ssh22:56:071
3691186994031,7cyclictest3698687-21sh19:35:010
3691186994031,7cyclictest3698687-21sh19:35:010
3691186994029,7cyclictest3757046-21ntpq21:56:050
3691186994027,8cyclictest3714540-21ntpq20:26:010
3691186994027,8cyclictest3699646-21ntpq19:36:050
369118899392,31cyclictest3841828-21sh00:10:572
369118899392,16cyclictest3804983-21ssh23:11:062
3691188993918,3cyclictest3813429-21kworker/u8:4-xprtiod23:26:112
3691188993918,3cyclictest3813429-21kworker/u8:4-xprtiod23:26:112
369118899391,16cyclictest3817677-21ssh23:31:042
369118899391,16cyclictest3817677-21ssh23:31:042
3691186993928,7cyclictest3726594-21ntpq21:06:050
3691186993928,7cyclictest3726594-21ntpq21:06:050
3691186993928,7cyclictest3707071-21ntpq20:01:070
3691186993928,7cyclictest3693644-21ntpq19:16:080
3691186993926,8cyclictest3725034-21ntpq21:00:590
3691186993926,8cyclictest3725034-21ntpq21:00:590
3691186993924,12cyclictest3845158-21ntpq00:16:080
3691186993924,12cyclictest3845158-21ntpq00:16:080
3691186993924,12cyclictest3719099-21ntpq20:41:040
3691186993918,16cyclictest0-21swapper/019:41:070
3691186993916,9cyclictest3764255-21sh22:06:470
369118899381,19cyclictest3792067-21sh22:51:012
369118799383,25cyclictest3851479-21ssh00:26:041
369118799383,25cyclictest3851479-21ssh00:26:041
3691187993830,5cyclictest3817854-21mailstats23:31:091
3691187993830,5cyclictest3817854-21mailstats23:31:091
3691187993830,5cyclictest3743743-21aten_r2power_vo21:35:521
3691186993829,4cyclictest3747624-21ntpq21:41:070
3691186993827,7cyclictest3752229-21ssh21:48:500
3691186993826,8cyclictest3719909-21/usr/sbin/munin20:45:510
3691186993823,11cyclictest3761924-21ssh22:04:130
3691186993821,13cyclictest3812017-21ssh23:21:360
3691186993819,15cyclictest3770807-21sh22:17:170
3691186993819,13cyclictest3753813-21sh21:50:590
3691186993818,9cyclictest3783332-21ssh22:36:370
3691186993818,7cyclictest3729459-21ssh21:12:280
3691186993818,7cyclictest3729459-21ssh21:12:280
3691186993816,16cyclictest0-21swapper/019:56:050
3691186993815,7cyclictest3732664-21sh21:17:320
3691186993813,12cyclictest3802688-21ssh23:07:170
3691188993732,3cyclictest0-21swapper/200:30:572
3691188993732,3cyclictest0-21swapper/200:30:572
3691187993729,5cyclictest3790919-21/usr/sbin/munin22:50:011
3691186993727,7cyclictest3710086-21ntpq20:11:070
3691186993727,7cyclictest3708567-21ntpq20:06:050
3691186993727,7cyclictest3708567-21ntpq20:06:050
3691186993725,8cyclictest3836733-21ntpq00:01:060
3691186993725,8cyclictest3795332-21ntpq22:56:060
3691186993725,8cyclictest3692130-21ntpq19:11:070
3691186993724,10cyclictest3830428-21ntpq23:51:050
3691186993724,10cyclictest3703669-21diskstats19:50:560
3691186993721,13cyclictest3711580-21ntpq20:16:030
3691186993721,12cyclictest3843223-21ssh00:13:190
3691186993720,13cyclictest3772569-21ssh22:20:540
3691186993720,13cyclictest3745396-21ssh21:37:340
3691186993717,14cyclictest0-21swapper/020:36:010
3691188993628,5cyclictest3695616-21sed19:21:152
3691188993627,5cyclictest3803797-21latency_hist23:10:022
3691188993627,5cyclictest3737229-21ssh21:25:502
3691187993610,9cyclictest171rcu_preempt19:31:021
3691187993610,9cyclictest171rcu_preempt19:31:021
3691186993630,4cyclictest3794113-21kworker/u8:6-nfsiod23:01:030
3691186993627,6cyclictest3827294-21ntpq23:46:060
3691186993626,7cyclictest3722052-21ntpq20:51:020
3691186993626,6cyclictest3713105-21ntpq20:21:100
3691186993625,8cyclictest3816022-21ssh23:29:010
3691186993625,8cyclictest3816022-21ssh23:29:010
3691186993625,8cyclictest3734886-21ntpq21:21:070
3691186993624,9cyclictest3716033-21ntpq20:31:000
3691186993624,9cyclictest3703135-21users19:46:180
3691186993624,9cyclictest3703135-21users19:46:180
3691186993623,8cyclictest3840456-21ssh00:09:310
3691186993623,10cyclictest3723637-21ntp_states20:56:060
3691186993620,12cyclictest3742940-21ssh21:34:400
3691186993618,6cyclictest3851357-21ntpq00:26:000
3691186993618,6cyclictest3851357-21ntpq00:26:000
369118899353,11cyclictest3836826-21ssh00:01:102
3691188993529,3cyclictest0-21swapper/220:40:012
3691188993526,5cyclictest3840839-21latency_hist00:10:032
3691187993527,5cyclictest421-21dbus-daemon23:05:011
3691187993527,5cyclictest3824590-21unixbench_singl23:41:141
3691187993527,5cyclictest3706700-21fschecks_count20:00:571
369118799352,26cyclictest3747473-21ssh21:41:011
3691186993526,6cyclictest3779485-21ntp_states22:31:070
3691186993526,6cyclictest3779485-21ntp_states22:31:070
3691186993522,9cyclictest3807129-21ssh23:15:160
3691186993521,10cyclictest3777156-21ssh22:27:110
3691186993521,10cyclictest3777156-21ssh22:27:110
369118899344,24cyclictest3763386-21ssh22:06:012
3691188993428,3cyclictest0-21swapper/222:40:002
3691188993426,5cyclictest3827791-21unixbench_singl23:46:132
3691188993425,5cyclictest0-21swapper/222:26:122
3691188993425,5cyclictest0-21swapper/222:26:122
3691187993426,5cyclictest3778992-21gltestperf22:30:551
3691187993426,5cyclictest3778992-21gltestperf22:30:551
3691187993425,7cyclictest3786195-21timerandwakeup22:41:121
3691187993425,6cyclictest3838770-21wc00:05:021
3691187993425,5cyclictest0-21swapper/123:36:051
3691186993421,9cyclictest3791529-21aten_r2power_fr22:50:520
3691188993328,2cyclictest0-21swapper/221:00:002
3691188993326,4cyclictest3729019-21unixbench_singl21:11:182
3691188993326,4cyclictest3729019-21unixbench_singl21:11:182
3691188993324,7cyclictest3785747-21ntp_kernel_pll_22:41:012
3691188993323,5cyclictest0-21swapper/221:31:042
369118899332,24cyclictest3830302-21ssh23:51:002
369118799333,23cyclictest3776150-21ssh22:26:001
369118799333,23cyclictest3776150-21ssh22:26:001
3691187993326,4cyclictest3814649-21sendmail_mailqu23:26:091
3691187993326,4cyclictest3814649-21sendmail_mailqu23:26:091
3691187993325,5cyclictest3859728-21cron00:40:001
3691187993325,5cyclictest3713680-21cron20:25:001
3691187993325,4cyclictest0-21swapper/119:20:011
369118799332,26cyclictest3760297-21sh22:01:061
3691187993321,8cyclictest3732096-21unixbench-2d21:16:181
369118799331,26cyclictest3769936-21ssh22:16:071
3691188993225,3cyclictest0-21swapper/220:11:132
3691188993222,7cyclictest3753836-21sh21:50:592
369118799328,5cyclictest171rcu_preempt23:16:031
3691187993224,5cyclictest3801961-21mailstats23:06:101
3691187993224,5cyclictest3692666-21idleruntime-cro19:15:011
3691187993222,7cyclictest3734314-21cpuspeed21:20:531
3691188993124,3cyclictest0-21swapper/219:20:012
369118799315,18cyclictest428-21mta-sts-daemon19:56:031
369118799313,23cyclictest3773134-21ssh22:21:081
3691187993122,6cyclictest3827402-21mailstats23:46:081
369118799312,24cyclictest3753814-21ssh21:50:581
3691187993119,9cyclictest3725023-21ntp_kernel_err21:00:591
3691187993119,9cyclictest3725023-21ntp_kernel_err21:00:591
3691188993024,3cyclictest0-21swapper/223:41:022
3691188993021,6cyclictest3749444-21cron21:45:002
3691188993021,5cyclictest1952300-21cups-browsed20:21:032
369118799305,15cyclictest3854625-21ssh00:31:031
369118799305,15cyclictest3854625-21ssh00:31:031
3691187993022,5cyclictest548-21gdbus00:15:571
3691187993022,5cyclictest548-21gdbus00:15:571
3691187993021,5cyclictest3813219-21systemd-run23:25:011
3691187993017,9cyclictest3703172-21latency_hist19:50:001
3691187993017,9cyclictest3703172-21latency_hist19:50:001
3691188992924,2cyclictest0-21swapper/200:40:012
3691188992914,11cyclictest351ktimers/219:31:042
3691188992914,11cyclictest351ktimers/219:31:042
3691187992923,4cyclictest1055-21mta-sts-daemon22:50:191
3691187992922,4cyclictest1051-21mta-sts-daemon19:23:381
369118799292,22cyclictest3833540-21ssh23:56:061
3691186992915,9cyclictest0-21swapper/019:10:000
3691188992821,3cyclictest0-21swapper/220:06:032
3691188992821,3cyclictest0-21swapper/220:06:032
3691188992820,6cyclictest3714927-21timerandwakeup20:26:132
3691188992819,4cyclictest0-21swapper/221:21:022
3691187992822,3cyclictest0-21swapper/120:20:001
369118799282,18cyclictest3750736-21ssh21:46:041
3691187992819,4cyclictest0-21swapper/123:50:591
3691187992818,7cyclictest3710574-21grep20:11:161
3691187992816,9cyclictest3717573-21ntp_kernel_pll_20:36:011
3691188992719,5cyclictest421-21dbus-daemon19:40:012
3691187992721,3cyclictest0-21swapper/121:10:581
3691187992721,3cyclictest0-21swapper/121:10:581
3691187992719,6cyclictest420-21avahi-daemon20:41:131
3691187992719,5cyclictest3701211-21proc_pri19:41:081
3691187992719,5cyclictest3691324-21latency_hist19:10:021
3691187992719,5cyclictest1952300-21cups-browsed20:25:261
369118799270,7cyclictest171rcu_preempt20:06:081
369118799270,7cyclictest171rcu_preempt20:06:081
369118899264,14cyclictest1051-21mta-sts-daemon21:06:022
369118899264,14cyclictest1051-21mta-sts-daemon21:06:022
369118899264,11cyclictest1056-21mta-sts-daemon19:56:032
369118899263,14cyclictest428-21mta-sts-daemon20:41:002
3691188992620,3cyclictest0-21swapper/220:31:062
3691188992619,4cyclictest1051-21mta-sts-daemon19:41:122
369118799264,14cyclictest428-21mta-sts-daemon20:46:021
3691187992620,4cyclictest3699333-21kernelversion19:35:561
3691188992517,5cyclictest3725210-21proc_pri21:01:072
3691188992517,5cyclictest3725210-21proc_pri21:01:072
3691188992418,3cyclictest0-21swapper/219:30:012
3691188992418,3cyclictest0-21swapper/219:30:012
3691188992415,6cyclictest3707672-21cron20:05:002
3691187992417,4cyclictest1056-21mta-sts-daemon20:52:031
3691187992317,3cyclictest0-21swapper/121:10:021
3691187992317,3cyclictest0-21swapper/121:10:021
3691187992216,3cyclictest1051-21mta-sts-daemon00:07:211
3691187992216,3cyclictest0-21swapper/119:52:581
3691187992215,5cyclictest0-21swapper/119:30:011
3691187992215,5cyclictest0-21swapper/119:30:011
369118899209,8cyclictest421-21dbus-daemon19:10:012
369118999182,8cyclictest0-21swapper/322:16:063
369118999173,8cyclictest0-21swapper/322:01:103
369118999161,7cyclictest0-21swapper/321:21:033
369118999158,5cyclictest0-21swapper/322:10:533
369118999154,5cyclictest0-21swapper/322:31:063
369118999154,5cyclictest0-21swapper/322:31:063
369118999153,6cyclictest0-21swapper/319:21:043
369118999152,6cyclictest0-21swapper/322:21:053
369118999151,6cyclictest0-21swapper/321:41:043
369118999151,6cyclictest0-21swapper/320:41:023
369118999151,5cyclictest0-21swapper/320:26:023
369118999143,6cyclictest0-21swapper/321:59:033
369118999143,6cyclictest0-21swapper/321:51:033
369118999142,6cyclictest0-21swapper/323:40:003
369118999142,5cyclictest0-21swapper/319:51:033
369118999141,8cyclictest0-21swapper/322:51:083
369118999141,6cyclictest0-21swapper/321:26:033
369118999141,6cyclictest0-21swapper/320:46:053
369118999141,6cyclictest0-21swapper/300:01:103
369118999141,5cyclictest0-21swapper/321:06:023
369118999141,5cyclictest0-21swapper/321:06:023
369118999133,7cyclictest0-21swapper/321:31:013
369118999133,4cyclictest0-21swapper/321:36:013
369118999132,6cyclictest0-21swapper/323:16:033
369118999132,6cyclictest0-21swapper/321:11:123
369118999132,6cyclictest0-21swapper/321:11:123
369118999132,6cyclictest0-21swapper/300:31:003
369118999132,6cyclictest0-21swapper/300:31:003
369118999132,6cyclictest0-21swapper/300:10:583
369118999132,5cyclictest0-21swapper/322:56:043
369118999132,5cyclictest0-21swapper/319:10:583
369118999131,7cyclictest0-21swapper/323:31:013
369118999131,7cyclictest0-21swapper/323:31:013
369118999131,7cyclictest0-21swapper/323:26:033
369118999131,7cyclictest0-21swapper/323:26:033
369118999131,7cyclictest0-21swapper/322:46:003
369118999131,7cyclictest0-21swapper/322:06:083
369118999131,6cyclictest0-21swapper/323:55:023
369118999131,6cyclictest0-21swapper/322:40:593
369118999131,6cyclictest0-21swapper/320:31:033
369118999131,6cyclictest0-21swapper/300:26:013
369118999131,6cyclictest0-21swapper/300:26:013
369118999131,5cyclictest0-21swapper/320:51:003
369118999131,5cyclictest0-21swapper/320:06:063
369118999131,5cyclictest0-21swapper/320:06:063
369118999125,5cyclictest0-21swapper/323:06:053
369118999123,5cyclictest0-21swapper/320:25:003
369118999122,6cyclictest0-21swapper/321:16:063
369118999122,6cyclictest0-21swapper/319:26:083
369118999122,6cyclictest0-21swapper/319:26:083
369118999122,5cyclictest0-21swapper/323:20:553
369118999121,7cyclictest0-21swapper/323:41:013
369118999121,7cyclictest0-21swapper/300:21:083
369118999121,7cyclictest0-21swapper/300:21:083
369118999121,7cyclictest0-21swapper/300:16:013
369118999121,7cyclictest0-21swapper/300:16:013
369118999121,6cyclictest0-21swapper/323:45:593
369118999121,6cyclictest0-21swapper/323:00:523
369118999121,6cyclictest0-21swapper/319:31:023
369118999121,6cyclictest0-21swapper/319:31:023
369118999121,5cyclictest0-21swapper/320:35:593
369118999121,5cyclictest0-21swapper/319:46:043
369118999121,5cyclictest0-21swapper/319:46:043
369118999121,5cyclictest0-21swapper/319:36:013
369118999114,5cyclictest0-21swapper/323:10:563
369118999114,5cyclictest0-21swapper/321:46:083
369118999113,4cyclictest0-21swapper/322:36:013
369118999112,5cyclictest0-21swapper/300:35:523
369118999112,4cyclictest0-21swapper/321:05:023
369118999112,4cyclictest0-21swapper/321:05:023
369118999111,6cyclictest0-21swapper/322:26:013
369118999111,6cyclictest0-21swapper/322:26:013
369118999111,6cyclictest0-21swapper/320:01:193
369118999111,5cyclictest0-21swapper/300:00:033
369118999102,4cyclictest0-21swapper/300:10:013
369118999101,5cyclictest0-21swapper/320:56:013
369118999101,5cyclictest0-21swapper/319:10:023
36911899994,4cyclictest0-21swapper/320:10:573
36911899991,5cyclictest0-21swapper/320:15:523
36911899991,5cyclictest0-21swapper/319:15:563
36911899982,4cyclictest0-21swapper/320:00:023
36911899981,5cyclictest0-21swapper/319:40:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional