You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-11 - 22:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Fri Jul 11, 2025 12:41:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2829810995440,9cyclictest422-21dbus-daemon11:20:012
2829808995241,6cyclictest151ktimers/008:18:030
2829808995241,6cyclictest151ktimers/008:18:030
2829808995228,19cyclictest12450irq/32-eth012:28:060
2829809995143,4cyclictest0-21swapper/111:52:571
2829810995039,7cyclictest2971035-21latency_hist11:55:022
2829809994940,5cyclictest421-21avahi-daemon11:23:071
2829810994839,5cyclictest2938311-21ssh11:05:012
2829810994839,5cyclictest2938311-21ssh11:05:012
2829809994823,21cyclictest2016377-21cups-browsed09:13:421
282980999468,35cyclictest2922990-21sh10:42:231
2829809994638,5cyclictest2935081-21idleruntime-cro11:00:011
2829809994637,6cyclictest422-21dbus-daemon12:30:011
2829809994621,22cyclictest2839872-21sendmail_mailqu07:43:051
2829808994636,7cyclictest2948031-21munin-run11:20:010
2829810994536,5cyclictest0-21swapper/208:14:502
2829809994538,3cyclictest0-21swapper/109:33:001
282981099445,32cyclictest2940161-21sshd11:07:572
282981099445,32cyclictest2940161-21sshd11:07:572
282981099442,34cyclictest2923982-21ssh10:43:062
2829809994437,4cyclictest2979704-21unixbench-2d12:08:141
2829809994437,4cyclictest2979704-21unixbench-2d12:08:141
2829809994426,15cyclictest2987652-21munin-plugin-st12:20:011
282980999442,36cyclictest2832271-21ntpq07:18:021
2829809994423,18cyclictest421-21avahi-daemon09:56:151
2829809994419,21cyclictest0-21swapper/110:00:351
2829809994419,21cyclictest0-21swapper/110:00:351
2829810994324,17cyclictest351ktimers/208:37:592
2829809994334,4cyclictest0-21swapper/111:02:581
2829809994334,4cyclictest0-21swapper/111:02:581
282980999433,33cyclictest2842681-21ntpq07:52:591
282980999432,36cyclictest2972764-21ntpq11:57:581
282980999432,34cyclictest2847273-21ntpq08:08:021
282980999432,34cyclictest2847273-21ntpq08:08:021
2829809994315,25cyclictest2930463-21ntp_kernel_pll_10:52:591
282980999431,37cyclictest2946710-21ntpq11:18:001
282980999431,25cyclictest2865280-21ntp_states09:08:071
2829810994234,5cyclictest2920536-21if_eth010:37:522
2829810994215,11cyclictest171rcu_preempt08:42:592
2829809994235,3cyclictest0-21swapper/110:38:011
2829809994234,4cyclictest2838768-21kworker/u8:3-nfsiod08:14:521
2829809994233,6cyclictest2906742-21latency_hist10:15:021
2829809994233,6cyclictest2906742-21latency_hist10:15:021
282980999422,35cyclictest2838264-21ntpq07:38:031
282980999421,37cyclictest2999404-21ssh12:37:131
282980999421,37cyclictest2974852-21ssh12:01:351
282980999421,36cyclictest2938691-21ssh11:05:321
282980999421,36cyclictest2938691-21ssh11:05:321
282980999421,36cyclictest2856152-21ntpq08:37:581
2829808994230,7cyclictest12450irq/32-eth010:12:590
2829808994230,7cyclictest12450irq/32-eth010:12:590
2829810994124,10cyclictest2865793-21diskmemload09:13:072
282980999418,31cyclictest2878443-21ssh09:29:371
2829809994133,5cyclictest2984025-21ssh12:15:011
282980999413,32cyclictest2863682-21ntpq09:02:571
2829809994125,13cyclictest0-21swapper/111:12:581
2829809994125,13cyclictest0-21swapper/111:12:581
282980999411,36cyclictest2910770-21ssh10:22:021
282980999411,36cyclictest2884482-21ssh09:38:451
282980999411,36cyclictest2854706-21ntpq08:33:021
282980999411,35cyclictest2966158-21ntpq11:48:021
282980999411,35cyclictest2916369-21ssh10:31:201
282980999411,34cyclictest2860741-21ntpq08:53:001
282980999411,25cyclictest2865793-21diskmemload09:53:331
2829809994110,29cyclictest2990900-21ssh12:24:241
282980999410,35cyclictest2861503-21sshd08:55:571
2829808994132,7cyclictest2918946-21idleruntime-cro10:35:010
2829808994129,8cyclictest12450irq/32-eth011:42:590
2829808994128,8cyclictest12450irq/32-eth009:38:000
2829808994119,14cyclictest12450irq/32-eth010:37:570
2829810994029,8cyclictest2888546-21sshd09:45:572
2829810994026,10cyclictest2877155-21missed_timers09:27:562
282980999402,34cyclictest2901217-21sh10:06:161
282980999402,34cyclictest2901217-21sh10:06:161
282980999401,37cyclictest2958100-21ssh11:36:211
282980999401,35cyclictest2997800-21ssh12:33:481
282980999401,35cyclictest2927412-21ssh10:48:071
282980999401,35cyclictest2906968-21ssh10:15:251
282980999401,35cyclictest2888573-21ssh09:46:011
282980999401,35cyclictest2873201-21ssh09:22:361
282980999401,35cyclictest2840661-21aten_r2power_po07:47:481
282980999401,35cyclictest2840661-21aten_r2power_po07:47:481
282980999401,34cyclictest2956246-21ntpq11:33:041
2829808994024,13cyclictest2833341-21fschecks_count07:22:510
2829810993931,5cyclictest2837292-21grep07:35:012
2829810993929,8cyclictest698-21cron09:45:012
282980999393,31cyclictest2844201-21ntpq07:58:111
282980999393,31cyclictest2844201-21ntpq07:58:111
2829809993924,11cyclictest0-21swapper/109:18:121
2829809993924,11cyclictest0-21swapper/109:18:121
282980999392,32cyclictest2857731-21ntpq08:43:011
282980999392,32cyclictest2845703-21ntpq08:03:011
282980999392,20cyclictest2865793-21diskmemload11:41:181
282980999391,36cyclictest2954184-21ssh11:29:591
282980999391,35cyclictest2830770-21ntpq07:13:031
282980999391,34cyclictest2835207-21ntpq07:28:001
282980999391,33cyclictest2913378-21ssh10:25:491
282980999391,33cyclictest2859261-21ntpq08:48:011
282980999391,33cyclictest2851673-21ntpq08:23:051
282980999391,33cyclictest2851673-21ntpq08:23:051
282980999391,33cyclictest2850249-21ntpq08:18:021
282980999391,33cyclictest2850249-21ntpq08:18:021
282980999391,33cyclictest2833704-21ntpq07:22:581
2829808993931,5cyclictest2898624-21basename10:02:470
2829808993931,5cyclictest2898624-21basename10:02:470
2829808993931,5cyclictest0-21swapper/009:52:570
2829810993830,5cyclictest2984037-21latency_hist12:15:022
282981099381,29cyclictest2943595-21ssh11:13:042
282981099381,29cyclictest2943595-21ssh11:13:042
282980999382,32cyclictest2852809-21diskstats08:27:531
282980999382,32cyclictest2836725-21ntpq07:32:591
282980999381,32cyclictest2885091-21ssh09:40:101
282980899383,11cyclictest14-21ksoftirqd/012:23:040
2829808993830,5cyclictest2943984-21unixbench-2d11:13:100
2829808993830,5cyclictest2943984-21unixbench-2d11:13:100
2829808993819,15cyclictest12450irq/32-eth011:38:030
282981099374,24cyclictest2996823-21ssh12:32:582
2829810993730,3cyclictest0-21swapper/211:45:012
2829810993728,6cyclictest421-21avahi-daemon11:25:022
2829810993728,6cyclictest2957496-21idleruntime-cro11:35:012
2829810993727,6cyclictest421-21avahi-daemon09:35:222
282981099371,6cyclictest171rcu_preempt11:46:002
2829808993729,6cyclictest2897322-21sendmail10:00:010
2829808993724,10cyclictest0-21swapper/008:55:010
2829808993712,20cyclictest2927250-21ssh10:48:020
2829810993625,7cyclictest2016377-21cups-browsed10:47:152
282980899366,23cyclictest12450irq/32-eth009:42:570
2829808993630,4cyclictest2912985-21munin-plugin-st10:25:010
2829808993630,4cyclictest2871160-21ntp_kernel_pll_09:18:050
2829808993630,4cyclictest2871160-21ntp_kernel_pll_09:18:050
2829808993630,4cyclictest2859521-21grep08:48:080
2829808993626,7cyclictest2852307-21idleruntime-cro08:25:010
2829808993626,7cyclictest2852307-21idleruntime-cro08:25:010
2829808993614,3cyclictest0-21swapper/010:18:010
2829810993525,6cyclictest2875292-21ssh09:25:012
2829810993525,6cyclictest2016377-21cups-browsed10:05:342
2829810993525,6cyclictest2016377-21cups-browsed10:05:342
2829810993521,10cyclictest2832826-21sh07:20:012
282980999350,32cyclictest2829821-21/usr/sbin/munin07:10:011
282980999350,32cyclictest2829821-21/usr/sbin/munin07:10:011
2829808993528,4cyclictest2838826-21systemd-run07:40:010
282981099347,21cyclictest2893074-21ssh09:53:072
282981099343,26cyclictest2865793-21diskmemload12:18:052
2829810993428,3cyclictest0-21swapper/210:00:012
2829810993425,6cyclictest2899914-21unixbench_singl10:03:172
2829810993425,6cyclictest2899914-21unixbench_singl10:03:172
2829810993425,5cyclictest2979028-21missed_timers12:07:592
2829810993425,5cyclictest2979028-21missed_timers12:07:592
2829810993423,9cyclictest2933530-21kernelversion10:57:572
282981099341,27cyclictest2871145-21ssh09:18:042
282981099341,27cyclictest2871145-21ssh09:18:042
282980899347,9cyclictest14-21ksoftirqd/008:03:040
282980899344,23cyclictest2966084-21ssh11:47:590
2829808993427,5cyclictest3001865-21sh12:40:010
2829808993427,4cyclictest2829807-21cyclictest11:21:580
2829808993426,6cyclictest2831838-21fschecks_time07:17:510
2829808993423,7cyclictest12450irq/32-eth012:00:010
2829808993422,8cyclictest12450irq/32-eth012:08:070
2829808993422,8cyclictest12450irq/32-eth012:08:070
2829810993326,4cyclictest0-21swapper/208:52:582
2829810993325,3cyclictest0-21swapper/212:38:082
2829808993328,2cyclictest0-21swapper/012:20:010
2829808993327,3cyclictest0-21swapper/010:28:040
2829808993320,6cyclictest14-21ksoftirqd/011:33:000
2829808993317,9cyclictest956-21mta-sts-daemon11:53:210
2829808993317,10cyclictest171rcu_preempt09:12:590
2829808993314,3cyclictest171rcu_preempt09:08:020
282981099325,17cyclictest2993226-21ssh12:27:582
2829810993226,3cyclictest0-21swapper/208:17:582
2829810993226,3cyclictest0-21swapper/208:17:582
2829810993225,3cyclictest0-21swapper/210:51:572
2829810993224,5cyclictest2954200-21latency_hist11:30:022
2829810993222,4cyclictest0-21swapper/208:28:012
282981099322,23cyclictest2908584-21ntpq10:18:022
282980899326,17cyclictest2902606-21ssh10:08:070
282980899326,17cyclictest2902606-21ssh10:08:070
2829808993224,5cyclictest421-21avahi-daemon12:00:360
2829808993224,5cyclictest2837190-21unixbench_multi07:33:090
2829808993224,5cyclictest0-21swapper/012:13:010
2829808993214,6cyclictest171rcu_preempt07:43:000
2829810993125,3cyclictest0-21swapper/207:30:012
2829810993123,5cyclictest2905194-21gltestperf10:12:522
2829810993123,5cyclictest2905194-21gltestperf10:12:522
2829810993122,6cyclictest2016377-21cups-browsed09:34:052
282981099312,23cyclictest2989726-21ssh12:23:002
2829810993120,8cyclictest2839482-21latency07:42:542
282980899316,8cyclictest14-21ksoftirqd/011:08:010
282980899316,8cyclictest14-21ksoftirqd/011:08:010
282980899314,20cyclictest2873972-21ssh09:23:010
282980899313,15cyclictest14-21ksoftirqd/011:02:570
282980899313,15cyclictest14-21ksoftirqd/011:02:570
2829808993126,4cyclictest191rcuc/012:31:000
2829808993125,2cyclictest0-21swapper/009:30:020
2829808993123,2cyclictest171rcu_preempt10:41:570
2829808993122,7cyclictest2857624-21needreboot08:42:570
2829808993121,7cyclictest12450irq/32-eth009:48:040
2829808993118,6cyclictest171rcu_preempt07:46:390
2829808993118,6cyclictest171rcu_preempt07:46:390
2829808993116,2cyclictest171rcu_preempt11:28:110
2829808993114,6cyclictest171rcu_preempt10:55:010
2829810993022,5cyclictest2844817-21systemd-run08:00:012
2829810993022,5cyclictest2844817-21systemd-run08:00:012
2829810993021,6cyclictest2959965-21unixbench_multi11:38:092
2829810993021,5cyclictest2862772-21idleruntime-cro09:00:012
2829810993020,7cyclictest2864290-21cron09:05:012
282981099301,23cyclictest2911811-21ssh10:23:052
282980899306,7cyclictest14-21ksoftirqd/009:32:570
2829808993016,2cyclictest171rcu_preempt07:12:570
282981099293,3cyclictest171rcu_preempt07:48:012
282981099293,3cyclictest171rcu_preempt07:48:012
2829810992923,3cyclictest0-21swapper/212:00:012
2829810992923,3cyclictest0-21swapper/207:23:092
282981099292,19cyclictest2917622-21ssh10:32:562
282980899295,9cyclictest14-21ksoftirqd/010:58:080
2829808992921,5cyclictest0-21swapper/009:02:580
2829808992920,5cyclictest151ktimers/008:33:030
2829808992913,3cyclictest0-21swapper/007:58:100
2829808992913,3cyclictest0-21swapper/007:58:100
2829808992912,5cyclictest2016377-21cups-browsed08:56:530
2829808992911,3cyclictest0-21swapper/008:13:000
2829810992820,5cyclictest2915864-21idleruntime-cro10:30:012
2829810992819,4cyclictest0-21swapper/212:02:572
2829808992818,3cyclictest171rcu_preempt07:10:010
2829808992818,3cyclictest171rcu_preempt07:10:010
2829808992818,2cyclictest171rcu_preempt07:53:040
2829808992816,3cyclictest171rcu_preempt08:28:060
2829808992813,3cyclictest171rcu_preempt08:37:510
2829810992718,6cyclictest2845212-21aten_r2power_vo08:02:502
282981099271,11cyclictest171rcu_preempt08:23:042
282981099271,11cyclictest171rcu_preempt08:23:042
282981099270,22cyclictest957-21mta-sts-daemon08:33:032
2829808992720,5cyclictest151ktimers/007:27:570
2829810992621,3cyclictest0-21swapper/207:15:022
282980899266,7cyclictest0-21swapper/008:07:580
282980899266,7cyclictest0-21swapper/008:07:580
2829810992519,3cyclictest0-21swapper/207:52:572
2829810992418,3cyclictest0-21swapper/209:10:012
2829810992415,6cyclictest2838783-21munin-plugin-st07:40:012
2829810992114,5cyclictest2847086-21missed_timers08:07:552
2829810992114,5cyclictest2847086-21missed_timers08:07:552
2829810992110,7cyclictest2858790-21/usr/sbin/munin08:47:492
282981199181,7cyclictest0-21swapper/308:42:593
282981199172,9cyclictest0-21swapper/309:33:003
282981199172,10cyclictest0-21swapper/308:14:533
2829810991711,4cyclictest2829765-21/usr/sbin/munin07:10:012
2829810991711,4cyclictest2829765-21/usr/sbin/munin07:10:012
282981199164,7cyclictest0-21swapper/311:33:033
282981199164,6cyclictest0-21swapper/312:28:033
282981199162,9cyclictest0-21swapper/311:38:003
282981199162,9cyclictest0-21swapper/310:52:573
282981199162,8cyclictest0-21swapper/312:03:033
282981199162,7cyclictest0-21swapper/311:43:013
282981199162,7cyclictest0-21swapper/309:13:033
282981199162,4cyclictest0-21swapper/308:47:563
282981199161,8cyclictest0-21swapper/311:52:573
282981199161,8cyclictest0-21swapper/310:08:023
282981199161,8cyclictest0-21swapper/310:08:023
282981199161,8cyclictest0-21swapper/309:53:023
282981199152,8cyclictest0-21swapper/312:38:023
282981199152,7cyclictest0-21swapper/312:12:543
282981199152,7cyclictest0-21swapper/310:32:563
282981199152,7cyclictest0-21swapper/310:27:573
282981199152,6cyclictest0-21swapper/310:12:573
282981199152,6cyclictest0-21swapper/310:12:573
282981199152,6cyclictest0-21swapper/308:33:023
282981199151,8cyclictest0-21swapper/311:23:103
282981199151,8cyclictest0-21swapper/311:08:043
282981199151,8cyclictest0-21swapper/311:08:043
282981199151,8cyclictest0-21swapper/309:42:583
282981199151,7cyclictest0-21swapper/311:45:593
282981199151,6cyclictest0-21swapper/309:08:013
282981199151,6cyclictest0-21swapper/308:57:593
282981199151,6cyclictest0-21swapper/308:23:113
282981199151,6cyclictest0-21swapper/308:23:113
282981199151,6cyclictest0-21swapper/307:58:113
282981199151,6cyclictest0-21swapper/307:58:113
282981199143,7cyclictest0-21swapper/310:47:583
282981199142,7cyclictest0-21swapper/312:22:553
282981199142,7cyclictest0-21swapper/310:18:013
282981199142,5cyclictest0-21swapper/307:52:573
282981199141,8cyclictest0-21swapper/311:13:003
282981199141,8cyclictest0-21swapper/311:13:003
282981199141,8cyclictest0-21swapper/310:58:103
282981199141,7cyclictest0-21swapper/310:22:553
282981199141,7cyclictest0-21swapper/309:58:013
282981199141,6cyclictest0-21swapper/309:02:563
282981199141,6cyclictest0-21swapper/307:48:003
282981199141,6cyclictest0-21swapper/307:48:003
282981199135,6cyclictest0-21swapper/312:33:033
282981199134,7cyclictest0-21swapper/309:27:503
282981199132,7cyclictest0-21swapper/311:58:003
282981199132,7cyclictest0-21swapper/311:18:033
282981199132,7cyclictest0-21swapper/310:02:573
282981199132,7cyclictest0-21swapper/310:02:573
282981199132,7cyclictest0-21swapper/309:47:573
282981199132,6cyclictest0-21swapper/307:42:573
282981199131,8cyclictest0-21swapper/310:43:003
282981199131,8cyclictest0-21swapper/310:37:483
282981199131,7cyclictest0-21swapper/309:37:553
282981199131,7cyclictest0-21swapper/309:22:493
282981199131,6cyclictest0-21swapper/311:27:523
282981199131,6cyclictest0-21swapper/308:28:033
282981199131,6cyclictest0-21swapper/307:22:563
282981199130,6cyclictest0-21swapper/307:38:003
282981199123,4cyclictest0-21swapper/307:13:053
282981199122,7cyclictest0-21swapper/311:05:013
282981199122,7cyclictest0-21swapper/311:05:013
282981199122,6cyclictest0-21swapper/312:17:533
282981199122,6cyclictest0-21swapper/309:17:473
282981199122,6cyclictest0-21swapper/309:17:473
282981199122,6cyclictest0-21swapper/308:17:583
282981199122,6cyclictest0-21swapper/308:17:583
282981199121,7cyclictest0-21swapper/312:07:473
282981199121,7cyclictest0-21swapper/312:07:473
282981199121,7cyclictest0-21swapper/307:32:583
282981199121,6cyclictest0-21swapper/308:38:023
282981199121,6cyclictest0-21swapper/308:03:013
282981199111,6cyclictest0-21swapper/307:27:583
282981199111,6cyclictest0-21swapper/307:17:523
282981199111,5cyclictest0-21swapper/308:07:573
282981199111,5cyclictest0-21swapper/308:07:573
282981199103,5cyclictest0-21swapper/308:52:563
282981199101,4cyclictest0-21swapper/307:10:023
282981199101,4cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional