You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 12:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Tue Nov 25, 2025 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2888777995140,7cyclictest2935569-21grep21:20:262
2888777995140,7cyclictest2935569-21grep21:20:262
2888782995040,5cyclictest2911490-21munin-plugin-st20:15:003
2888774995043,5cyclictest2986873-21kworker/u8:0+events_unbound00:23:371
2888771994942,5cyclictest2956708-21kworker/u8:1+events_unbound22:28:160
2888771994942,5cyclictest2956708-21kworker/u8:1+events_unbound22:28:160
2888782994819,10cyclictest2967433-21sed22:55:203
2888782994818,11cyclictest2946788-21if_eth121:55:173
2888782994721,9cyclictest2934396-21cut21:20:003
2888774994739,5cyclictest2961577-21cron22:40:001
2888782994613,15cyclictest2896304-21cut19:30:023
2888782994520,9cyclictest2917160-21/usr/sbin/munin20:30:153
2888782994519,9cyclictest4048818-1kworker/3:1H+kblockd22:40:013
2888782994518,11cyclictest2951784-21grep22:10:173
2888774994535,7cyclictest2912717-21unixbench_multi20:15:281
2888782994430,10cyclictest2909220-21/usr/sbin/munin20:05:243
2888782994419,9cyclictest2955582-21ntpq22:20:203
2888782994416,13cyclictest2900210-21latency19:40:203
2888771994426,5cyclictest131rcu_preempt23:50:160
2888771994425,5cyclictest2897622-21kworker/u8:2-events_freezable_power_19:57:230
2888782994334,5cyclictest2989309-21munin-run00:00:013
2888782994334,5cyclictest2989309-21munin-run00:00:003
2888782994325,12cyclictest2998080-21apt-get00:25:133
2888771994334,5cyclictest2908521-21fschecks_time20:05:160
2888782994232,7cyclictest2897117-21/usr/sbin/munin19:30:263
2888782994229,10cyclictest2976039-21/usr/sbin/munin23:20:193
2888782994228,11cyclictest2990165-21/usr/sbin/munin00:00:233
2888782994226,11cyclictest2948638-21/usr/sbin/munin22:00:183
2888782994216,12cyclictest0-21swapper/322:42:273
2888777994230,7cyclictest0-21swapper/223:55:002
2888771994221,10cyclictest131rcu_preempt19:50:070
2888782994130,7cyclictest2916020-21/usr/sbin/munin20:25:253
2888782994129,8cyclictest2980800-21/usr/sbin/munin23:35:143
2888782994128,7cyclictest2983141-21/usr/sbin/munin23:40:223
2888782994128,10cyclictest2903636-21/usr/sbin/munin19:50:213
2888782994127,9cyclictest2921708-21sshd20:42:213
2888782994125,11cyclictest2959021-21/usr/sbin/munin22:30:213
2888782994030,7cyclictest2994658-21apt-get00:15:133
2888782994030,6cyclictest2934850-21/usr/sbin/munin21:20:183
2888782994030,6cyclictest2934850-21/usr/sbin/munin21:20:173
2888782994029,7cyclictest2893081-21/usr/sbin/munin19:20:153
2888782994028,9cyclictest2978425-21/usr/sbin/munin23:25:263
2888782994028,9cyclictest2969504-21/usr/sbin/munin23:00:283
2888782994028,8cyclictest2954019-21sendmail_mailtr22:15:243
2888782994028,8cyclictest2931314-21/usr/sbin/munin21:10:193
2888782994028,7cyclictest2965758-21/usr/sbin/munin22:50:193
2888782994026,10cyclictest2972820-21/usr/sbin/munin23:10:233
2888771994024,4cyclictest131rcu_preempt20:56:360
2888771994021,9cyclictest131rcu_preempt21:22:170
2888771994021,9cyclictest131rcu_preempt21:22:170
2888782993932,4cyclictest2922177-21apt-get20:45:143
2888782993930,6cyclictest2927751-21idleruntime21:00:153
2888782993929,7cyclictest2938082-21/usr/sbin/munin21:30:173
2888782993929,6cyclictest2997360-21/usr/sbin/munin00:20:283
2888782993929,6cyclictest2924438-21ntp_states20:50:213
2888782993929,6cyclictest2920075-21sshd20:38:213
2888782993928,7cyclictest2889545-21/usr/sbin/munin19:10:233
2888782993927,9cyclictest2984879-21/usr/sbin/munin23:45:183
2888782993927,7cyclictest2939777-21/usr/sbin/munin21:35:173
2888782993921,13cyclictest2905021-21fschecks_count19:55:153
2888782993921,10cyclictest0-21swapper/320:20:133
2888777993930,5cyclictest0-21swapper/223:13:232
2888771993923,5cyclictest131rcu_preempt21:55:160
2888771993920,10cyclictest131rcu_preempt23:03:220
2888782993828,7cyclictest2936882-21/usr/sbin/munin21:25:233
2888782993828,7cyclictest2926864-21cron20:58:013
2888782993828,6cyclictest2929548-21/usr/sbin/munin21:05:183
2888782993827,8cyclictest2971517-21sshd23:06:213
2888782993826,9cyclictest2980305-21cron23:32:003
2888782993826,7cyclictest2906714-21df20:00:163
2888782993826,10cyclictest3000346-21ntp_states00:30:233
2888782993825,9cyclictest2890788-21munin-node19:15:023
2888782993825,9cyclictest2890788-21munin-node19:15:023
2888782993824,10cyclictest2943308-21/usr/sbin/munin21:45:183
2888782993813,9cyclictest637-21runrttasks00:35:093
2888777993830,5cyclictest2953921-21proc_pri22:15:232
2888774993830,3cyclictest0-21swapper/119:35:171
2888771993825,4cyclictest131rcu_preempt20:01:020
2888771993820,4cyclictest0-21swapper/023:40:000
2888771993820,3cyclictest0-21swapper/022:55:290
2888782993731,3cyclictest2898068-21apt-get19:35:153
2888782993727,7cyclictest2913040-21sshd20:18:213
2888782993727,6cyclictest2975506-21latency_hist23:20:013
2888782993726,8cyclictest2992544-21cron00:07:593
2888782993726,8cyclictest2957657-21/usr/sbin/munin22:25:253
2888782993726,8cyclictest2957657-21/usr/sbin/munin22:25:253
2888782993725,8cyclictest2944881-21/usr/sbin/munin21:50:133
2888782993716,8cyclictest0-21swapper/323:52:153
2888777993729,4cyclictest0-21swapper/223:10:002
2888774993730,4cyclictest2932458-21latency_hist21:15:011
2888774993725,9cyclictest231ktimers/122:55:131
288877199377,8cyclictest11-21ksoftirqd/022:15:010
2888771993728,4cyclictest0-21swapper/021:45:020
2888771993720,4cyclictest131rcu_preempt19:27:080
2888771993716,4cyclictest131rcu_preempt00:16:240
2888782993626,7cyclictest2963540-21apt-get22:45:133
2888782993626,7cyclictest2941617-21/usr/sbin/munin21:40:183
2888782993625,7cyclictest2993530-21ntp_states00:10:223
2888782993620,13cyclictest2950032-21fschecks_count22:05:153
2888782993617,13cyclictest2903135-21cat19:50:013
2888777993625,8cyclictest311ktimers/222:50:122
2888777993624,8cyclictest0-21swapper/223:00:002
2888774993629,4cyclictest2894424-21cron19:25:001
2888774993628,4cyclictest2930681-21munin-plugin-st21:10:001
2888774993626,6cyclictest0-21swapper/119:40:271
288877199368,8cyclictest11-21ksoftirqd/019:45:010
2888771993625,4cyclictest131rcu_preempt21:18:000
2888771993617,9cyclictest131rcu_preempt23:45:210
2888777993526,4cyclictest0-21swapper/219:35:012
2888774993521,5cyclictest2946088-21kworker/u8:1+events_unbound21:58:031
2888774993516,5cyclictest2944173-21kworker/u8:0+events_unbound21:47:021
288877199356,9cyclictest11-21ksoftirqd/000:40:000
2888771993527,4cyclictest0-21swapper/020:20:000
2888771993526,4cyclictest0-21swapper/021:35:000
2888771993524,7cyclictest2954353-21grep22:15:260
2888771993522,4cyclictest131rcu_preempt00:03:200
2888771993518,4cyclictest131rcu_preempt00:11:180
2888771993518,3cyclictest0-21swapper/020:47:510
2888771993517,4cyclictest131rcu_preempt22:37:420
2888771993517,10cyclictest131rcu_preempt22:41:430
2888771993516,5cyclictest131rcu_preempt19:32:060
288877199351,3cyclictest0-21swapper/021:45:130
2888771993510,17cyclictest11-21ksoftirqd/020:45:010
2888777993426,4cyclictest0-21swapper/221:55:252
2888777993422,9cyclictest0-21swapper/223:00:132
2888777993421,10cyclictest2944971-21fschecks_time21:50:142
2888774993416,6cyclictest2998063-21kworker/u8:2+events_unbound00:31:401
288877499341,10cyclictest24-21ksoftirqd/119:45:131
2888771993428,4cyclictest2986873-21kworker/u8:0+events_unbound00:28:360
2888771993427,4cyclictest151rcuc/019:10:230
2888771993423,4cyclictest131rcu_preempt22:50:150
2888771993422,5cyclictest131rcu_preempt21:00:230
2888771993422,4cyclictest131rcu_preempt23:56:280
2888771993422,4cyclictest131rcu_preempt23:56:270
2888771993420,4cyclictest131rcu_preempt20:22:320
2888771993418,4cyclictest131rcu_preempt00:06:120
2888771993416,4cyclictest131rcu_preempt23:24:090
2888771993415,4cyclictest131rcu_preempt00:31:300
2888774993326,3cyclictest0-21swapper/120:05:001
2888774993325,4cyclictest0-21swapper/121:25:001
2888774993325,4cyclictest0-21swapper/121:25:001
2888774993324,5cyclictest0-21swapper/119:50:171
2888774993323,7cyclictest2958185-21idleruntime-cro22:30:011
2888774993323,7cyclictest2958185-21idleruntime-cro22:30:011
2888774993312,4cyclictest131rcu_preempt20:25:131
288877199338,5cyclictest131rcu_preempt21:30:000
288877199337,13cyclictest121ktimers/023:15:170
2888771993324,4cyclictest0-21swapper/019:20:000
2888771993324,4cyclictest0-21swapper/019:20:000
2888771993322,4cyclictest131rcu_preempt23:31:030
2888771993317,5cyclictest131rcu_preempt22:20:380
2888771993317,4cyclictest131rcu_preempt20:50:230
2888771993316,4cyclictest131rcu_preempt22:45:530
2888771993316,4cyclictest131rcu_preempt21:08:550
2888771993316,4cyclictest131rcu_preempt20:11:410
2888771993315,4cyclictest131rcu_preempt22:32:260
2888771993315,4cyclictest131rcu_preempt22:02:260
2888771993314,10cyclictest131rcu_preempt19:23:010
2888777993224,4cyclictest0-21swapper/220:05:262
2888777993223,4cyclictest0-21swapper/200:05:182
2888777993220,9cyclictest2923646-21idleruntime-cro20:50:012
2888774993226,4cyclictest2980416-21kworker/u8:1+events_unbound23:33:281
2888774993226,4cyclictest2937506-21kworker/u8:0+events_unbound21:38:071
2888774993224,5cyclictest2920009-21kworker/u8:1+events_unbound20:50:241
2888774993221,6cyclictest0-21swapper/120:48:001
2888774993219,10cyclictest2915116-21auditctl20:25:001
2888774993216,5cyclictest2897622-21kworker/u8:2+events_unbound20:13:251
2888774993211,12cyclictest24-21ksoftirqd/119:55:151
2888771993224,2cyclictest0-21swapper/020:33:530
2888771993222,6cyclictest0-21swapper/023:25:260
2888771993222,6cyclictest0-21swapper/023:05:240
2888771993220,8cyclictest121ktimers/020:35:190
2888771993220,4cyclictest131rcu_preempt19:45:360
2888771993217,4cyclictest131rcu_preempt21:12:490
2888771993215,4cyclictest131rcu_preempt19:35:310
2888771993214,10cyclictest131rcu_preempt23:40:160
2888771993213,4cyclictest131rcu_preempt22:09:330
2888771993212,5cyclictest131rcu_preempt21:40:010
288877799313,5cyclictest131rcu_preempt21:30:132
2888777993125,4cyclictest2895897-21kworker/u8:0+events_unbound19:37:512
2888777993124,3cyclictest0-21swapper/223:45:242
2888777993123,4cyclictest0-21swapper/220:30:252
2888777993122,4cyclictest0-21swapper/222:05:252
2888774993125,4cyclictest2990767-21kworker/u8:2+events_unbound00:13:431
2888774993123,4cyclictest0-21swapper/123:05:251
2888774993123,4cyclictest0-21swapper/121:30:261
2888774993123,4cyclictest0-21swapper/119:35:011
288877499310,5cyclictest131rcu_preempt22:20:111
2888771993114,4cyclictest131rcu_preempt23:13:250
2888771993113,11cyclictest131rcu_preempt20:25:240
2888771993111,10cyclictest131rcu_preempt21:50:200
288877499309,12cyclictest24-21ksoftirqd/120:40:001
2888774993023,5cyclictest2954713-21kworker/u8:2+events_unbound22:33:421
2888774993023,5cyclictest2888351-21kworker/u8:1+events_unbound19:11:091
2888774993020,6cyclictest0-21swapper/123:25:011
2888774993020,6cyclictest0-21swapper/121:25:261
2888774993020,6cyclictest0-21swapper/121:00:251
2888774993020,4cyclictest2990767-21kworker/u8:2+events_unbound00:03:421
2888774993019,5cyclictest2915470-21kworker/u8:1+events_unbound20:30:291
2888774993015,7cyclictest131rcu_preempt19:15:031
2888774993015,7cyclictest131rcu_preempt19:15:021
2888771993012,10cyclictest131rcu_preempt00:20:180
2888777992922,3cyclictest0-21swapper/221:20:002
2888777992921,4cyclictest0-21swapper/221:10:012
2888777992920,5cyclictest2906360-21sh20:00:012
2888777992919,7cyclictest2958593-21grep22:30:152
288877499297,3cyclictest131rcu_preempt00:15:121
288877499294,3cyclictest131rcu_preempt20:55:131
2888774992921,4cyclictest0-21swapper/100:25:261
2888774992920,3cyclictest131rcu_preempt23:25:231
2888774992919,5cyclictest2971502-21kworker/u8:2+events_unbound23:11:161
2888774992918,7cyclictest0-21swapper/100:05:261
2888774992915,5cyclictest2980416-21kworker/u8:1+events_unbound23:35:021
2888774992915,4cyclictest131rcu_preempt21:15:271
2888777992821,4cyclictest3002966-21sendmail00:40:002
2888777992816,8cyclictest1-21systemd23:43:282
2888774992821,5cyclictest2986873-21kworker/u8:0+events_unbound23:56:581
2888774992821,5cyclictest2986873-21kworker/u8:0+events_unbound23:56:581
2888777992721,4cyclictest2894097-21kworker/u8:1+events_unbound19:22:482
2888777992718,5cyclictest0-21swapper/219:50:012
2888777992717,6cyclictest0-21swapper/223:15:272
2888777992716,7cyclictest0-21swapper/219:10:272
2888777992715,9cyclictest215-21dbus-daemon22:38:172
2888774992721,4cyclictest3002610-21kworker/u8:1+events_unbound00:38:341
2888774992721,4cyclictest2949887-21kworker/u8:0+events_unbound22:18:161
2888774992720,5cyclictest2963238-21kworker/u8:1+events_unbound23:18:191
2888774992719,5cyclictest2964588-21unixbench_singl22:45:261
2888774992718,6cyclictest2970125-21cron23:05:001
2888774992716,8cyclictest1-21systemd22:08:111
2888774992716,7cyclictest0-21swapper/122:55:021
2888774992715,6cyclictest2946088-21kworker/u8:1+events_unbound22:02:191
2888774992715,5cyclictest2983877-21kworker/u8:2+events_unbound23:51:341
2888774992713,4cyclictest131rcu_preempt23:45:011
2888777992620,4cyclictest2897622-21kworker/u8:2+events_unbound19:52:532
2888777992618,4cyclictest0-21swapper/220:40:172
2888777992618,4cyclictest0-21swapper/220:10:262
2888777992617,6cyclictest2952659-21unixbench_singl22:10:252
2888777992617,4cyclictest0-21swapper/221:10:262
2888777992617,4cyclictest0-21swapper/219:40:172
2888777992615,8cyclictest2995226-21ntp_states00:15:232
2888774992616,6cyclictest0-21swapper/119:25:261
2888774992614,9cyclictest221rcuc/122:13:351
2888774992611,4cyclictest131rcu_preempt21:40:261
2888777992519,4cyclictest2911181-21kworker/u8:0+events_unbound20:22:592
2888777992519,3cyclictest0-21swapper/222:25:192
2888777992519,3cyclictest0-21swapper/222:25:192
2888777992518,3cyclictest0-21swapper/221:28:052
2888777992517,4cyclictest0-21swapper/219:25:262
2888777992514,7cyclictest0-21swapper/200:35:012
2888774992518,4cyclictest0-21swapper/123:45:251
2888774992517,6cyclictest2909403-21kworker/u8:1+events_unbound20:07:251
2888774992512,11cyclictest24-21ksoftirqd/122:40:131
2888774992511,3cyclictest131rcu_preempt20:40:161
288877499251,12cyclictest0-21swapper/121:50:131
2888777992417,4cyclictest0-21swapper/223:55:252
2888777992417,4cyclictest0-21swapper/223:55:252
2888777992416,4cyclictest0-21swapper/220:15:282
2888777992416,3cyclictest0-21swapper/200:00:262
288877799241,4cyclictest2998525-21meminfo00:25:192
2888777992413,8cyclictest2996828-21meminfo00:20:212
2888777992318,3cyclictest2939944-21if_eth021:35:182
2888777992317,3cyclictest0-21swapper/220:25:222
2888777992315,5cyclictest0-21swapper/223:35:272
2888777992313,7cyclictest2892738-21basename19:20:002
2888777992313,7cyclictest2892738-21basename19:20:002
2888777992313,5cyclictest0-21swapper/222:45:272
288877799221,7cyclictest2976711-21grep23:20:262
2888777992214,5cyclictest2919301-21ntpq20:35:222
2888777992214,5cyclictest0-21swapper/221:40:172
288877799219,9cyclictest215-21dbus-daemon20:53:002
2888777992116,2cyclictest0-21swapper/222:40:122
2888777992115,3cyclictest637-21runrttasks00:10:212
2888777992115,3cyclictest0-21swapper/222:05:012
2888777992115,3cyclictest0-21swapper/221:45:212
2888777992113,4cyclictest0-21swapper/220:55:262
288877799202,3cyclictest131rcu_preempt20:00:132
2888777992014,3cyclictest2927906-21latency21:00:172
2888777992013,4cyclictest2977589-21fschecks_time23:25:162
2888777991912,5cyclictest311ktimers/223:30:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional