You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 07:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Fri Nov 14, 2025 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2823690995129,17cyclictest0-21swapper/200:25:002
2823690995129,17cyclictest0-21swapper/200:25:002
2823681995041,6cyclictest121ktimers/019:10:130
2823694994735,9cyclictest391ktimers/321:35:123
2823681994733,10cyclictest121ktimers/000:20:150
2823681994733,10cyclictest121ktimers/000:20:140
2823690994636,6cyclictest2837842-21grep19:45:272
2823694994530,11cyclictest391ktimers/322:10:143
2823694994530,11cyclictest391ktimers/322:10:143
2823690994434,7cyclictest2890544-21sendmail22:20:002
2823690994422,17cyclictest2874744-21cron21:35:002
2823685994437,5cyclictest2836267-21kworker/u8:0+events_unbound19:47:411
2823685994437,4cyclictest2834890-21sendmail19:40:001
282369099432,10cyclictest2937234-21apt-get00:35:142
282369099431,8cyclictest2894291-21/usr/sbin/munin22:30:162
2823685994332,8cyclictest2893944-21idleruntime-cro22:30:011
2823690994227,10cyclictest311ktimers/221:55:122
2823694994133,5cyclictest2873058-21munin-plugin-st21:30:013
282369099412,9cyclictest2915862-21/usr/sbin/munin23:30:282
282369099412,11cyclictest2870332-21memory21:20:212
282369099411,9cyclictest2934675-21/usr/sbin/munin00:25:292
282369099411,9cyclictest2895983-21/usr/sbin/munin22:35:162
2823690994119,18cyclictest0-21swapper/221:10:002
2823690994119,18cyclictest0-21swapper/221:10:002
282369099411,8cyclictest2901599-21/usr/sbin/munin22:50:262
282369099411,6cyclictest2879221-21/usr/sbin/munin21:45:232
282369099411,10cyclictest2907018-21/usr/sbin/munin23:05:252
282369099409,27cyclictest215-21dbus-daemon22:08:022
282369099409,27cyclictest215-21dbus-daemon22:08:012
282369099408,7cyclictest2849703-21which20:20:242
282369099402,8cyclictest2871693-21df_inode21:25:142
282369099402,5cyclictest2881611-21cron21:53:012
282369099402,5cyclictest2826490-21irqstats19:15:202
282369099401,9cyclictest2918805-21/usr/sbin/munin23:40:202
282369099401,9cyclictest2890777-21/usr/sbin/munin22:20:142
282369099401,9cyclictest2859963-21/usr/sbin/munin20:50:242
282369099401,8cyclictest2855943-21/usr/sbin/munin20:40:182
282369099401,8cyclictest2840335-21/usr/sbin/munin19:55:152
282369099401,7cyclictest2931426-21/usr/sbin/munin00:15:252
282369099401,6cyclictest2877995-21users21:40:292
282369099401,12cyclictest2912017-21ntp_states23:20:212
282369099401,10cyclictest2900317-21grep22:45:292
2823685994032,4cyclictest0-21swapper/122:13:021
2823685994032,4cyclictest0-21swapper/122:13:021
2823681994027,9cyclictest2870892-21grep21:20:260
282369099393,4cyclictest2892577-21df22:25:142
282369099392,5cyclictest2823952-21df19:10:152
282369099392,4cyclictest2844172-21/usr/sbin/munin20:05:252
282369099391,8cyclictest2936138-21/usr/sbin/munin00:30:252
282369099391,8cyclictest2846116-21/usr/sbin/munin20:10:282
282369099391,7cyclictest2858841-21cron20:46:002
282369099391,6cyclictest2928826-21diskstats00:10:172
282369099391,6cyclictest2828407-21/usr/sbin/munin19:20:232
282369099391,5cyclictest2835212-21/usr/sbin/munin19:40:152
282369099391,4cyclictest2854787-21/usr/sbin/munin20:35:302
282369099391,12cyclictest2927129-21diskstats00:05:192
282369099391,11cyclictest2908141-21diskstats23:10:162
282369099391,11cyclictest2885424-21munin-run22:05:002
2823685993929,6cyclictest2916127-21grep23:30:291
2823681993931,4cyclictest0-21swapper/000:00:000
2823681993929,7cyclictest121ktimers/019:40:130
2823681993927,9cyclictest2890514-21cron22:20:000
2823694993826,9cyclictest2869640-21munin-run21:20:013
282369099382,5cyclictest2852497-21/usr/sbin/munin20:30:162
282369099382,4cyclictest2920876-21/usr/sbin/munin23:45:252
282369099382,3cyclictest2847711-21/usr/sbin/munin20:15:232
282369099382,3cyclictest2847711-21/usr/sbin/munin20:15:232
282369099381,8cyclictest2831585-21/usr/sbin/munin19:30:152
282369099381,7cyclictest2867487-21/usr/sbin/munin21:10:272
282369099381,7cyclictest2861287-21fschecks_count20:55:162
282369099381,6cyclictest2917208-21meminfo23:35:212
282369099381,3cyclictest2829783-21/usr/sbin/munin19:25:162
282369099381,10cyclictest2904855-21meminfo23:00:192
282369099377,5cyclictest2823772-21cat19:10:012
282369099373,5cyclictest2833971-21ntp_states19:35:232
282369099373,3cyclictest2838521-21/usr/sbin/munin19:50:232
282369099372,9cyclictest2925469-21grep00:00:162
282369099372,13cyclictest2863006-21fschecks_count21:00:192
282369099371,9cyclictest2843018-21users20:00:252
282369099371,8cyclictest2923793-21fschecks_time23:55:182
282369099371,6cyclictest2898580-21sed22:40:272
282369099371,5cyclictest2850682-21/usr/sbin/munin20:25:172
282369099371,4cyclictest2887486-21sort22:10:182
282369099371,4cyclictest2887486-21sort22:10:182
282369099371,10cyclictest2876240-21sed21:35:252
2823685993729,4cyclictest0-21swapper/119:25:011
2823681993729,4cyclictest0-21swapper/022:00:000
282369099362,5cyclictest2921997-21df_inode23:50:172
282369099362,4cyclictest2910026-21/usr/sbin/munin23:15:182
282369099362,4cyclictest2868345-21fschecks_time21:15:162
282369099361,4cyclictest2902691-21/usr/sbin/munin22:55:242
2823685993630,4cyclictest2843194-21kworker/u8:3+events_unbound20:02:451
2823685993629,4cyclictest2907318-21unixbench_multi23:05:271
2823685993628,4cyclictest0-21swapper/120:55:001
2823681993628,5cyclictest0-21swapper/021:15:130
2823694993527,4cyclictest0-21swapper/322:00:003
282369099351,4cyclictest2913499-21grep23:25:192
2823685993529,3cyclictest0-21swapper/100:25:201
2823685993524,7cyclictest0-21swapper/120:00:011
2823681993523,8cyclictest0-21swapper/023:25:010
2823694993426,5cyclictest2836581-21cron19:45:013
2823685993427,4cyclictest0-21swapper/119:15:011
2823685993426,5cyclictest2909196-21kworker/u8:1+events_unbound23:15:231
2823694993327,4cyclictest2875001-21apt-key21:35:013
2823694993327,3cyclictest0-21swapper/323:00:003
2823694993324,5cyclictest2900763-21audit_send_reply22:50:013
2823694993322,7cyclictest0-21swapper/300:18:143
2823694993322,6cyclictest0-21swapper/320:25:003
2823694993311,12cyclictest2889761-21grep22:15:243
2823685993325,4cyclictest0-21swapper/121:00:291
2823685993323,6cyclictest0-21swapper/119:15:001
2823681993325,4cyclictest0-21swapper/000:40:010
2823681993321,8cyclictest2842972-21grep20:00:250
2823694993224,4cyclictest0-21swapper/319:40:003
2823694993224,4cyclictest0-21swapper/300:30:293
2823694993223,4cyclictest0-21swapper/323:00:153
2823694993214,5cyclictest2823675-21cyclictest00:00:013
2823685993223,6cyclictest0-21swapper/121:40:291
2823685993222,5cyclictest0-21swapper/123:10:261
2823685993222,5cyclictest0-21swapper/121:35:011
2823681993222,6cyclictest0-21swapper/020:30:290
2823681993210,3cyclictest28662992cut21:10:010
282368199320,30cyclictest0-21swapper/022:10:140
282368199320,30cyclictest0-21swapper/022:10:140
282369499319,12cyclictest40-21ksoftirqd/300:35:193
282369499315,8cyclictest40-21ksoftirqd/320:05:203
2823694993123,4cyclictest0-21swapper/320:55:283
2823685993123,4cyclictest0-21swapper/122:30:281
2823685993120,6cyclictest0-21swapper/119:55:011
282368599311,4cyclictest131rcu_preempt00:15:121
2823681993126,3cyclictest0-21swapper/022:00:120
2823681993123,4cyclictest0-21swapper/021:10:260
2823681993122,6cyclictest2916523-21sh23:35:010
282369499307,6cyclictest40-21ksoftirqd/321:05:253
282369499307,6cyclictest40-21ksoftirqd/321:05:253
282369499307,6cyclictest0-21swapper/322:30:253
282369499303,9cyclictest40-21ksoftirqd/322:05:013
2823694993022,5cyclictest2897351-21idleruntime-cro22:40:013
2823694993020,7cyclictest381rcuc/321:47:503
2823685993023,5cyclictest2829027-21kworker/u8:1+events_unbound19:27:391
2823685993022,4cyclictest0-21swapper/121:55:151
2823685993022,4cyclictest0-21swapper/121:15:271
2823681993022,4cyclictest0-21swapper/019:25:010
2823681993021,6cyclictest0-21swapper/000:20:000
2823681993018,8cyclictest0-21swapper/020:00:010
282369499295,7cyclictest0-21swapper/321:40:253
2823694992921,4cyclictest0-21swapper/320:30:293
2823694992921,4cyclictest0-21swapper/300:25:203
2823694992920,5cyclictest0-21swapper/322:05:273
2823694992920,5cyclictest0-21swapper/322:05:273
2823694992919,6cyclictest0-21swapper/319:45:273
2823694992918,8cyclictest2932580-21meminfo00:20:233
2823694992918,8cyclictest2932580-21meminfo00:20:233
2823694992918,7cyclictest0-21swapper/320:10:223
2823694992917,8cyclictest1-21systemd23:08:063
2823694992912,4cyclictest131rcu_preempt19:10:123
2823685992922,5cyclictest2935204-21kworker/u8:2+events_unbound00:38:171
2823685992922,4cyclictest2866005-21cron21:10:001
2823685992922,4cyclictest2866005-21cron21:09:591
2823685992922,4cyclictest0-21swapper/100:25:001
2823685992922,4cyclictest0-21swapper/100:25:001
2823685992921,5cyclictest2910914-21kworker/u8:2+events_unbound23:28:211
2823685992920,6cyclictest231ktimers/123:55:141
2823681992924,3cyclictest13650irq/42-30be0000.ethernet20:25:120
2823681992922,4cyclictest0-21swapper/023:25:300
2823681992921,4cyclictest0-21swapper/022:40:270
2823681992920,6cyclictest2904156-21sh23:00:000
2823681992920,6cyclictest2877938-21grep21:40:290
282368199291,25cyclictest2872631-21grep21:25:250
282369499285,6cyclictest40-21ksoftirqd/323:35:183
282369499284,6cyclictest40-21ksoftirqd/322:41:423
282369499283,7cyclictest40-21ksoftirqd/321:50:183
2823694992820,4cyclictest0-21swapper/300:00:263
2823694992819,5cyclictest0-21swapper/319:15:183
2823694992819,4cyclictest0-21swapper/322:20:253
2823685992822,4cyclictest2878220-21kworker/u8:2+events_unbound22:04:411
2823685992822,3cyclictest0-21swapper/120:10:011
2823685992821,4cyclictest2857473-21latency_hist20:45:021
282368599280,4cyclictest131rcu_preempt23:00:121
2823681992821,4cyclictest0-21swapper/023:05:000
282369499277,5cyclictest40-21ksoftirqd/323:20:253
282369499274,7cyclictest40-21ksoftirqd/321:04:143
282369499273,7cyclictest40-21ksoftirqd/320:45:013
282369499273,6cyclictest40-21ksoftirqd/320:35:253
2823694992720,4cyclictest2829524-21latency_hist19:25:023
2823694992718,6cyclictest2921241-21unixbench_singl23:45:273
2823694992713,4cyclictest131rcu_preempt23:40:233
282368599279,4cyclictest131rcu_preempt22:55:001
2823685992721,4cyclictest2902182-21kworker/u8:1+events_unbound22:55:411
2823685992720,5cyclictest2918474-21kworker/u8:2+events_unbound23:49:301
2823685992715,10cyclictest24-21ksoftirqd/122:38:461
2823681992720,4cyclictest0-21swapper/021:05:260
2823681992720,3cyclictest2880088-21kthreadd21:50:010
2823681992720,3cyclictest0-21swapper/021:35:000
2823681992718,6cyclictest2845112-21sh20:10:010
2823681992717,6cyclictest0-21swapper/020:20:160
282368199271,23cyclictest2825624-21date19:15:010
282368199270,2cyclictest0-21swapper/022:20:120
282369499269,5cyclictest131rcu_preempt23:25:203
282369499269,3cyclictest131rcu_preempt19:26:393
282369499267,6cyclictest40-21ksoftirqd/320:50:163
282369499266,6cyclictest40-21ksoftirqd/321:15:013
282369499263,9cyclictest40-21ksoftirqd/320:25:163
2823694992617,6cyclictest2922069-21fschecks_time23:50:193
282369499261,18cyclictest0-21swapper/323:10:133
2823685992620,4cyclictest2878220-21kworker/u8:2+events_unbound21:46:311
2823685992620,4cyclictest2869555-21kworker/u8:1+events_unbound21:22:581
2823685992619,5cyclictest2935204-21kworker/u8:2+events_unbound00:34:471
2823685992619,5cyclictest2879948-21kworker/u8:1+events_unbound21:50:011
2823685992618,4cyclictest0-21swapper/123:20:261
2823685992615,8cyclictest2927581-21ntp_states00:05:261
2823681992621,2cyclictest13650irq/42-30be0000.ethernet20:55:490
2823681992618,4cyclictest2919500-21grep23:40:280
2823681992618,4cyclictest0-21swapper/019:40:010
2823681992617,5cyclictest0-21swapper/021:35:150
282369499259,4cyclictest131rcu_preempt19:50:263
282369499259,10cyclictest0-21swapper/300:13:143
282369499256,6cyclictest40-21ksoftirqd/320:15:203
282369499256,6cyclictest40-21ksoftirqd/320:15:203
282369499256,11cyclictest0-21swapper/319:55:113
282369499255,5cyclictest40-21ksoftirqd/322:25:163
282369499255,4cyclictest40-21ksoftirqd/300:05:253
2823694992514,7cyclictest2910209-21wc23:15:203
2823694992514,4cyclictest131rcu_preempt21:20:123
2823694992514,4cyclictest131rcu_preempt20:00:113
2823685992518,3cyclictest0-21swapper/119:40:281
2823685992516,7cyclictest2918232-21cron23:40:011
2823685992514,8cyclictest2922393-21cat23:50:221
282368199252,18cyclictest2917801-21grep23:35:260
2823681992519,3cyclictest0-21swapper/020:45:010
2823681992518,4cyclictest2897363-21sendmail22:40:010
2823681992515,6cyclictest2890137-1kworker/0:2H+mmc_complete00:35:000
282368199251,3cyclictest2859420-21apt-get20:50:160
282369499247,5cyclictest40-21ksoftirqd/320:40:293
282369499246,9cyclictest0-21swapper/322:50:283
282369499240,8cyclictest40-21ksoftirqd/319:30:153
2823685992418,4cyclictest2926866-21latency_hist00:05:011
2823685992418,4cyclictest2881478-21kworker/u8:3+events_unbound22:07:391
2823685992418,4cyclictest2881478-21kworker/u8:3+events_unbound22:07:391
2823685992418,4cyclictest2869555-21kworker/u8:1+events_unbound21:29:161
2823685992415,6cyclictest0-21swapper/121:00:001
282368599241,12cyclictest0-21swapper/120:37:411
282368599241,12cyclictest0-21swapper/120:33:421
2823681992418,3cyclictest0-21swapper/020:45:200
2823681992418,3cyclictest0-21swapper/019:50:020
2823681992417,3cyclictest0-21swapper/022:50:290
2823681992417,3cyclictest0-21swapper/022:30:170
2823681992416,4cyclictest0-21swapper/020:20:000
2823681992416,4cyclictest0-21swapper/020:19:590
2823681992413,7cyclictest2923355-21ls23:55:000
2823681992413,7cyclictest2847033-21awk20:15:000
282368199241,19cyclictest0-21swapper/022:48:060
282368199241,19cyclictest0-21swapper/021:04:270
282368199241,19cyclictest0-21swapper/019:30:320
282368199241,19cyclictest0-21swapper/000:00:190
282368199240,20cyclictest0-21swapper/023:17:120
282368599237,4cyclictest131rcu_preempt22:15:251
2823685992317,3cyclictest2868029-21latency_hist21:15:011
2823685992316,4cyclictest231ktimers/119:35:011
2823685992315,5cyclictest0-21swapper/120:15:241
2823685992315,5cyclictest0-21swapper/120:15:241
282368599231,12cyclictest0-21swapper/120:47:421
282368599230,12cyclictest0-21swapper/120:25:181
282368199231,20cyclictest0-21swapper/022:25:180
282368599221,7cyclictest299-21systemd-logind20:22:471
2823685992216,3cyclictest0-21swapper/100:10:231
2823685992214,6cyclictest2892073-21kworker/u8:2+events_unbound22:23:091
2823685992214,5cyclictest0-21swapper/123:40:181
2823681992217,2cyclictest0-21swapper/021:50:140
2823681992215,5cyclictest0-21swapper/022:10:000
2823681992215,5cyclictest0-21swapper/022:10:000
2823681992215,3cyclictest0-21swapper/000:28:330
2823681992214,4cyclictest0-21swapper/019:25:260
282368199221,3cyclictest0-21swapper/023:05:060
2823681992213,6cyclictest215-21dbus-daemon23:13:050
282368199220,18cyclictest0-21swapper/000:14:350
282368199211,17cyclictest2840001-21idleruntime-cro19:55:010
2823681992110,8cyclictest2920186-21seq23:45:150
2823685992013,4cyclictest2845788-21fschecks_time20:10:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional