You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 21:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Wed Nov 26, 2025 12:39:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3010495995951,5cyclictest3067893-21aten_r2power_fr09:44:142
3010495995951,5cyclictest3067893-21aten_r2power_fr09:44:142
3010495995850,5cyclictest3091787-21munin-run10:20:002
3010495995850,5cyclictest3091787-21munin-run10:20:002
3010494994940,6cyclictest3128849-21ntpq11:14:121
3010494994838,6cyclictest421-21avahi-daemon09:09:121
3010495994739,5cyclictest3160799-21sendmail12:00:012
3010495994634,8cyclictest421-21avahi-daemon11:39:142
3010495994536,6cyclictest3183049-21latency_hist12:30:022
3010495994536,5cyclictest3059734-21unixbench_multi09:29:312
3010494994537,5cyclictest422-21dbus-daemon10:50:011
3010494994537,5cyclictest422-21dbus-daemon10:50:011
3010494994536,6cyclictest3056816-21systemd-run09:25:011
3010494994525,4cyclictest171rcu_preempt08:59:161
3010494994435,5cyclictest3143454-21systemd-run11:35:011
3010493994429,13cyclictest3136672-21latency_hist11:25:020
301049599432,34cyclictest3023385-21sshd07:54:402
3010494994335,5cyclictest3179256-21cron12:25:011
3010495994234,5cyclictest3078874-21cron10:00:012
3010495994234,5cyclictest3078874-21cron10:00:012
3010495994232,6cyclictest1-21systemd11:00:012
3010493994226,5cyclictest422-21dbus-daemon10:19:050
3010493994226,5cyclictest422-21dbus-daemon10:19:050
3010494994133,5cyclictest422-21dbus-daemon07:20:021
3010493994132,6cyclictest3094370-21sshd10:24:400
3010493994132,6cyclictest3094370-21sshd10:24:400
301049599405,29cyclictest3153163-21ssh11:49:152
3010495994033,3cyclictest0-21swapper/211:20:012
3010494994030,6cyclictest3078925-21systemd-run10:00:011
3010494994030,6cyclictest3078925-21systemd-run10:00:011
3010495993930,5cyclictest3190183-21grep12:40:012
3010495993929,7cyclictest956-21mta-sts-daemon12:04:072
301049499392,28cyclictest3049482-21ssh09:14:171
301049599385,25cyclictest3174451-21ssh12:19:152
3010495993831,3cyclictest0-21swapper/210:29:192
3010495993831,3cyclictest0-21swapper/210:29:192
3010495993829,6cyclictest3156018-21ssh11:54:092
3010495993829,5cyclictest3101712-21unixbench_multi10:34:302
3010495993829,5cyclictest3101712-21unixbench_multi10:34:302
3010495993820,13cyclictest351ktimers/211:04:172
301049499383,31cyclictest3184563-21ssh12:33:031
3010494993830,5cyclictest3074769-21ntp_kernel_err09:54:141
3010494993830,5cyclictest3037520-21latency_hist08:40:021
3010494993830,4cyclictest0-21swapper/111:55:011
3010494993830,4cyclictest0-21swapper/111:24:261
3010493993831,5cyclictest3119484-21systemd-run11:00:010
3010493993815,18cyclictest3174490-21ssh12:19:160
3010495993731,3cyclictest0-21swapper/207:40:012
3010495993730,4cyclictest3085476-21latency_hist10:10:022
3010495993729,5cyclictest3108740-21unixbench_singl10:44:292
3010495993729,5cyclictest3108740-21unixbench_singl10:44:292
301049599372,25cyclictest3167091-21ssh12:09:162
3010493993731,4cyclictest3133234-21cron11:20:010
3010493993731,3cyclictest2999489-21kworker/u8:2-xprtiod07:20:010
3010493993730,5cyclictest3088451-21cron10:15:010
3010493993729,5cyclictest3098516-21systemd-run10:30:010
3010493993729,5cyclictest3098516-21systemd-run10:30:010
3010495993628,5cyclictest3056809-21ssh09:25:012
3010495993625,8cyclictest3115013-21needreboot10:54:132
3010495993625,8cyclictest3115013-21needreboot10:54:132
3010494993630,3cyclictest0-21swapper/112:39:151
3010494993628,5cyclictest491-21gdbus07:44:221
3010494993628,5cyclictest3034495-21systemd-run08:30:011
3010494993625,7cyclictest421-21avahi-daemon08:49:201
3010494993625,7cyclictest3081055-21missed_timers10:04:091
3010493993630,3cyclictest0-21swapper/010:34:090
3010493993630,3cyclictest0-21swapper/010:34:090
3010495993527,5cyclictest3111236-21fschecks_time10:49:082
3010495993527,5cyclictest3111236-21fschecks_time10:49:082
3010495993527,5cyclictest3033020-21latency_hist08:25:022
3010495993526,6cyclictest3017015-21fschecks_count07:34:072
3010495993524,8cyclictest3013481-21sendmail07:20:012
3010495993524,7cyclictest422-21dbus-daemon11:43:392
3010494993527,5cyclictest3167681-21unixbench_singl12:09:271
3010494993527,4cyclictest955-21mta-sts-daemon11:56:271
3010494993523,8cyclictest955-21mta-sts-daemon07:24:171
3010493993528,4cyclictest3031515-21systemd-run08:20:020
3010493993527,5cyclictest421-21avahi-daemon11:46:570
3010493993527,5cyclictest3023994-21systemd-run07:55:010
3010493993510,20cyclictest3071443-21ssh09:49:110
3010495993427,4cyclictest3031475-21munin-plugin-st08:20:012
3010495993427,4cyclictest1-21systemd09:35:022
3010495993426,5cyclictest3105549-21basename10:40:012
3010495993426,5cyclictest3105549-21basename10:40:012
3010495993425,5cyclictest2199103-21cups-browsed08:49:192
3010495993422,9cyclictest3071932-21open_inodes09:49:232
3010494993426,5cyclictest3174962-21timerwakeupswit12:19:261
3010494993426,5cyclictest3028386-21tail08:09:261
3010494993426,3cyclictest0-21swapper/112:04:151
3010494993424,7cyclictest266-21systemd-journal10:15:001
3010494993423,8cyclictest3062216-21needreboot09:34:221
3010493993428,4cyclictest3022481-21cut07:50:010
3010493993428,4cyclictest3022481-21cut07:50:010
3010493993428,3cyclictest0-21swapper/010:00:020
3010493993428,3cyclictest0-21swapper/010:00:020
3010493993427,4cyclictest3168184-21latency_hist12:10:030
3010493993427,4cyclictest3140015-21cron11:30:010
3010493993414,16cyclictest3033538-21fschecks_time08:29:090
3010495993326,5cyclictest0-21swapper/208:05:022
3010495993326,3cyclictest0-21swapper/211:15:012
3010495993325,5cyclictest3037466-21latency_hist08:40:012
3010495993324,6cyclictest0-21swapper/211:35:012
3010495993323,6cyclictest421-21avahi-daemon10:21:492
3010495993323,6cyclictest421-21avahi-daemon10:21:492
3010495993320,3cyclictest3049606-21ssh09:14:222
3010494993326,5cyclictest3105598-21latency_hist10:40:011
3010494993326,5cyclictest3105598-21latency_hist10:40:011
3010494993325,5cyclictest955-21mta-sts-daemon07:29:161
3010494993325,5cyclictest3023568-21systemctl07:54:481
301049499332,26cyclictest3084496-21ssh10:09:121
301049399339,17cyclictest3159932-21sh11:59:150
3010493993326,5cyclictest0-21swapper/011:44:050
3010493993320,5cyclictest3033003-21true08:25:010
3010495993225,4cyclictest959-21mta-sts-daemon07:14:252
3010495993224,5cyclictest3030003-21latency_hist08:15:022
3010495993222,7cyclictest3028477-21munin-plugin-st08:10:022
301049499326,18cyclictest3097647-21ssh10:29:211
301049499326,18cyclictest3097647-21ssh10:29:211
301049499323,23cyclictest3065416-21ssh09:39:131
3010494993225,5cyclictest3119174-21unixbench_singl10:59:301
3010494993225,4cyclictest3025416-21unixbench_singl07:59:251
3010494993224,5cyclictest3146709-21head11:39:271
3010494993224,5cyclictest3041955-21cron08:55:011
3010494993224,5cyclictest3032988-21munin-plugin-st08:25:011
3010494993224,5cyclictest0-21swapper/112:14:131
301049499322,22cyclictest3068511-21ssh09:44:271
301049499322,22cyclictest3068511-21ssh09:44:271
3010493993226,3cyclictest0-21swapper/010:40:010
3010493993226,3cyclictest0-21swapper/010:40:010
3010493993225,4cyclictest0-21swapper/012:35:010
3010493993221,5cyclictest3171040-21ssh12:14:210
3010493993220,9cyclictest3043451-21cron09:00:010
3010493993219,10cyclictest0-21swapper/008:35:010
3010493993218,4cyclictest171rcu_preempt11:04:170
301049399321,23cyclictest3017382-21ntpq07:34:150
3010493993212,17cyclictest151ktimers/008:00:010
301049599314,12cyclictest3046477-21diskmemload12:22:222
3010495993125,3cyclictest0-21swapper/207:25:012
3010495993124,4cyclictest3139787-21unixbench_singl11:29:282
3010495993124,4cyclictest3136419-21unixbench_singl11:24:352
3010495993123,5cyclictest453-21systemd-logind09:00:022
3010495993123,5cyclictest421-21avahi-daemon11:09:062
3010495993122,6cyclictest2199080-21cupsd09:16:452
301049599312,23cyclictest3087594-21ssh10:14:152
3010494993125,3cyclictest0-21swapper/111:04:201
3010494993122,5cyclictest3131721-21aten2.4-expect11:19:041
3010494993121,7cyclictest3115713-21unixbench_singl10:54:271
3010494993121,7cyclictest3115713-21unixbench_singl10:54:271
3010494993120,8cyclictest3153167-21ntp_kernel_pll_11:49:161
3010494993113,12cyclictest421-21avahi-daemon08:44:211
3010494993113,12cyclictest421-21avahi-daemon08:44:211
301049499311,22cyclictest3101191-21ssh10:34:211
301049499311,22cyclictest3101191-21ssh10:34:211
3010493993125,4cyclictest422-21dbus-daemon08:12:140
3010493993124,5cyclictest421-21avahi-daemon09:09:340
3010493993124,5cyclictest3189266-21ssh12:39:160
3010493993122,6cyclictest422-21dbus-daemon12:02:570
3010493993118,10cyclictest3145397-21aten_r2power_cu11:39:040
3010493993110,12cyclictest12450irq/32-eth010:44:230
3010493993110,12cyclictest12450irq/32-eth010:44:230
3010495993024,3cyclictest0-21swapper/209:10:022
3010495993023,3cyclictest0-21swapper/212:14:192
3010495993021,6cyclictest3020427-21awk07:44:152
3010495993018,9cyclictest266-21systemd-journal08:45:012
3010495993018,9cyclictest266-21systemd-journal08:45:012
301049599301,24cyclictest3184554-21ssh12:33:022
3010494993023,4cyclictest956-21mta-sts-daemon08:34:191
301049499302,23cyclictest3149721-21ssh11:44:171
301049399303,21cyclictest3128914-21sh11:14:150
3010493993024,4cyclictest3179464-21latency_hist12:25:030
3010493993024,3cyclictest0-21swapper/008:55:010
3010493993023,4cyclictest956-21mta-sts-daemon09:35:260
3010493993022,4cyclictest3081747-21unixbench-2d10:04:250
3010493993020,7cyclictest3055625-21fschecks_count09:24:070
3010493993017,9cyclictest12450irq/32-eth009:54:140
3010493993017,10cyclictest0-21swapper/007:15:020
301049399301,24cyclictest3156338-21rm11:54:140
301049399301,22cyclictest3052973-21ssh09:19:190
301049499293,20cyclictest3090531-21ssh10:19:071
301049499293,20cyclictest3090531-21ssh10:19:071
3010494992922,5cyclictest3126393-21latency_hist11:10:021
3010494992922,4cyclictest3022707-21kworker/u8:0-rpciod09:30:011
3010494992921,6cyclictest266-21systemd-journal09:05:011
3010494992919,8cyclictest3072359-21unixbench_multi09:49:281
3010493992923,3cyclictest3019369-21unixbench_multi07:39:250
3010493992917,9cyclictest0-21swapper/012:29:070
3010493992916,9cyclictest12450irq/32-eth010:09:170
3010493992915,3cyclictest0-21swapper/009:14:080
3010493992914,5cyclictest3046477-21diskmemload11:09:180
3010495992823,2cyclictest0-21swapper/210:04:332
3010495992822,3cyclictest0-21swapper/208:55:022
3010495992821,3cyclictest0-21swapper/209:54:172
301049499284,17cyclictest3094351-21ssh10:24:391
301049499284,17cyclictest3094351-21ssh10:24:391
3010494992822,4cyclictest959-21mta-sts-daemon10:44:171
3010494992822,4cyclictest959-21mta-sts-daemon10:44:171
3010494992821,4cyclictest957-21mta-sts-daemon07:34:131
3010494992821,4cyclictest3052223-21aten_r2power_en09:19:041
3010494992821,3cyclictest0-21swapper/108:20:011
3010494992819,7cyclictest3027015-21latency_hist08:05:021
3010494992819,6cyclictest2199103-21cups-browsed08:11:301
3010493992819,7cyclictest422-21dbus-daemon09:45:020
3010493992814,11cyclictest3059716-21unixbench_multi09:29:310
3010493992814,10cyclictest12450irq/32-eth010:54:150
3010493992814,10cyclictest12450irq/32-eth010:54:150
3010495992721,3cyclictest0-21swapper/207:30:012
3010495992720,4cyclictest3021547-21gltestperf07:49:082
3010495992720,4cyclictest3021547-21gltestperf07:49:082
3010494992719,5cyclictest1-21systemd12:29:051
3010493992721,4cyclictest3093468-21kworker/u8:7-xprtiod11:34:170
3010493992720,5cyclictest2199103-21cups-browsed09:30:140
3010493992716,8cyclictest3028481-21munin-plugin-st08:10:010
3010495992620,3cyclictest0-21swapper/209:39:062
3010494992619,4cyclictest3011361-21ntp_kernel_err07:14:191
3010494992619,4cyclictest0-21swapper/111:29:171
301049399268,11cyclictest422-21dbus-daemon09:44:290
3010493992621,2cyclictest0-21swapper/010:48:530
3010493992621,2cyclictest0-21swapper/010:48:530
3010493992620,4cyclictest3044440-21ntp_offset09:04:150
3010493992617,7cyclictest491-21gdbus08:37:160
3010495992521,2cyclictest0-21swapper/208:00:032
301049599250,4cyclictest171rcu_preempt09:04:152
3010494992519,3cyclictest959-21mta-sts-daemon07:39:181
3010493992520,3cyclictest3016724-21kworker/u8:2-xprtiod08:44:140
3010493992520,3cyclictest3016724-21kworker/u8:2-xprtiod08:44:140
3010493992519,4cyclictest3016505-21systemd-run07:30:010
3010495992416,6cyclictest351ktimers/208:34:202
3010493992419,3cyclictest955-21mta-sts-daemon08:04:090
3010493992417,5cyclictest0-21swapper/008:50:000
301049599234,15cyclictest3034027-21ps08:29:222
3010495992315,5cyclictest3010505-21systemd-run07:10:022
3010494992316,4cyclictest3015015-21kworker/u8:8+xprtiod07:50:011
3010494992316,4cyclictest3015015-21kworker/u8:8+xprtiod07:50:011
301049399239,10cyclictest3020001-21df_inode07:44:060
301049399231,5cyclictest171rcu_preempt07:24:160
301049399208,9cyclictest422-21dbus-daemon07:10:010
3010494991914,2cyclictest0-21swapper/107:10:021
301049699182,9cyclictest0-21swapper/311:54:143
301049699173,8cyclictest0-21swapper/312:09:193
301049699173,8cyclictest0-21swapper/311:20:013
301049699171,8cyclictest0-21swapper/310:19:133
301049699171,8cyclictest0-21swapper/310:19:133
301049699171,10cyclictest0-21swapper/309:29:193
301049699163,8cyclictest0-21swapper/312:19:193
301049699162,7cyclictest0-21swapper/310:29:223
301049699162,7cyclictest0-21swapper/310:29:223
301049699161,9cyclictest0-21swapper/312:14:133
301049699154,7cyclictest0-21swapper/309:49:213
301049699153,6cyclictest0-21swapper/312:04:153
301049699152,9cyclictest0-21swapper/311:37:403
301049699152,8cyclictest0-21swapper/311:34:133
301049699152,8cyclictest0-21swapper/310:59:243
301049699152,8cyclictest0-21swapper/310:44:173
301049699152,8cyclictest0-21swapper/310:44:173
301049699152,7cyclictest0-21swapper/310:49:033
301049699152,7cyclictest0-21swapper/310:49:033
301049699152,7cyclictest0-21swapper/309:24:143
301049699151,7cyclictest0-21swapper/311:49:143
301049699151,7cyclictest0-21swapper/308:34:183
301049699143,6cyclictest0-21swapper/311:24:223
301049699142,7cyclictest0-21swapper/307:49:173
301049699142,7cyclictest0-21swapper/307:49:173
301049699142,6cyclictest0-21swapper/310:24:393
301049699142,6cyclictest0-21swapper/310:24:393
301049699142,6cyclictest0-21swapper/308:19:153
301049699142,6cyclictest0-21swapper/307:54:393
301049699142,5cyclictest0-21swapper/308:04:133
301049699141,9cyclictest0-21swapper/309:39:403
301049699141,8cyclictest0-21swapper/311:29:143
301049699141,7cyclictest0-21swapper/311:04:193
301049699141,7cyclictest0-21swapper/309:59:143
301049699141,7cyclictest0-21swapper/309:59:143
301049699141,7cyclictest0-21swapper/309:14:043
301049699133,7cyclictest0-21swapper/312:29:053
301049699132,7cyclictest0-21swapper/311:44:143
301049699132,7cyclictest0-21swapper/310:34:183
301049699132,7cyclictest0-21swapper/310:34:183
301049699132,7cyclictest0-21swapper/310:14:093
301049699132,7cyclictest0-21swapper/309:44:253
301049699132,7cyclictest0-21swapper/309:44:253
301049699132,6cyclictest0-21swapper/311:13:503
301049699132,6cyclictest0-21swapper/308:09:193
301049699131,8cyclictest0-21swapper/310:01:393
301049699131,7cyclictest0-21swapper/312:24:123
301049699131,7cyclictest0-21swapper/309:34:123
301049699131,6cyclictest0-21swapper/309:09:123
301049699131,6cyclictest0-21swapper/309:04:123
301049699131,6cyclictest0-21swapper/308:54:133
301049699131,6cyclictest0-21swapper/308:49:193
301049699131,6cyclictest0-21swapper/308:14:163
301049699131,5cyclictest0-21swapper/311:10:023
301049699131,5cyclictest0-21swapper/307:44:123
301049699122,6cyclictest0-21swapper/308:24:133
301049699122,6cyclictest0-21swapper/307:34:193
301049699122,6cyclictest0-21swapper/307:15:013
301049699122,5cyclictest0-21swapper/309:15:383
301049699122,4cyclictest0-21swapper/308:39:123
301049699121,8cyclictest0-21swapper/312:32:403
301049699121,7cyclictest0-21swapper/312:39:023
301049699121,7cyclictest0-21swapper/311:59:143
301049699121,7cyclictest0-21swapper/309:54:103
301049699121,6cyclictest0-21swapper/310:36:203
301049699121,6cyclictest0-21swapper/310:36:203
301049699121,6cyclictest0-21swapper/307:39:173
301049699121,6cyclictest0-21swapper/307:29:173
301049699113,4cyclictest0-21swapper/308:44:213
301049699113,4cyclictest0-21swapper/308:44:213
301049699112,6cyclictest0-21swapper/310:54:053
301049699112,6cyclictest0-21swapper/310:54:053
301049699112,5cyclictest0-21swapper/308:59:133
301049699111,7cyclictest0-21swapper/310:09:123
301049699111,6cyclictest0-21swapper/308:29:143
301049699111,6cyclictest0-21swapper/307:59:123
301049699111,6cyclictest0-21swapper/307:19:183
30104969991,5cyclictest0-21swapper/307:24:133
30104969991,5cyclictest0-21swapper/307:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional