You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 09:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Tue Mar 24, 2026 00:43:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3043311996536,23cyclictest0-21swapper/022:40:260
304331199625,47cyclictest3127721-21ssh22:05:230
3043313995646,7cyclictest421-21dbus-daemon00:25:292
3043313995449,2cyclictest0-21swapper/221:00:002
3043311995413,32cyclictest3097786-21ssh21:30:290
3043313995039,8cyclictest3127776-21sshd22:05:252
3043312995040,7cyclictest3131485-21aten_r2power_vo22:10:161
3043311995027,18cyclictest151ktimers/021:35:280
3043312994941,5cyclictest3238757-21unixbench_multi00:15:381
3043311994835,10cyclictest3211585-21cron23:45:010
3043311994821,18cyclictest151ktimers/019:40:220
3043313994742,2cyclictest0-21swapper/221:50:322
3043312994739,5cyclictest3237032-21cron00:15:011
3043312994637,6cyclictest3246862-21sendmail_mailqu00:25:331
3043312994634,8cyclictest1090850-21cups-browsed21:10:351
3043312994634,8cyclictest1090850-21cups-browsed21:10:351
3043311994634,9cyclictest0-21swapper/000:20:150
3043313994538,4cyclictest0-21swapper/221:30:342
3043313994524,14cyclictest3217043-21ssh23:50:292
3043313994524,14cyclictest3217043-21ssh23:50:292
304331299453,36cyclictest1090850-21cups-browsed21:05:271
304331299453,36cyclictest1090850-21cups-browsed21:05:271
3043313994418,8cyclictest3068192-21ntpq20:30:272
3043313994418,8cyclictest3068192-21ntpq20:30:272
304331299445,32cyclictest3242421-21ssh00:20:261
304331299443,36cyclictest3093504-21ssh21:25:301
3043311994430,9cyclictest420-21avahi-daemon20:00:300
3043313994335,5cyclictest3224319-21ssh00:00:002
3043313994332,7cyclictest3242376-21ssh00:20:242
3043313994330,10cyclictest420-21avahi-daemon19:20:012
3043313994330,10cyclictest420-21avahi-daemon19:20:012
3043313994315,9cyclictest3126247-21ssh22:04:192
3043313994311,15cyclictest3164521-21ssh22:49:132
3043312994337,3cyclictest0-21swapper/120:10:011
3043312994335,5cyclictest3207336-21munin-run23:40:011
304331299430,6cyclictest171rcu_preempt20:15:241
304331199432,6cyclictest171rcu_preempt22:10:260
3043311994326,14cyclictest3114172-21aten_r2power_en21:50:140
3043311994324,14cyclictest3237052-21ssh00:15:010
3043313994234,5cyclictest3111184-21unixbench_multi21:45:372
3043313994233,6cyclictest3258248-21systemd-run00:40:022
3043313994233,6cyclictest3258248-21systemd-run00:40:022
3043313994233,6cyclictest1054-21mta-sts-daemon00:00:282
3043312994236,3cyclictest0-21swapper/123:35:011
3043312994234,5cyclictest3122358-21systemd-run22:00:011
3043312994233,6cyclictest420-21avahi-daemon23:10:021
3043312994229,9cyclictest3111065-21timerandwakeup21:45:361
3043311994230,9cyclictest3083649-21latency_hist21:15:010
3043311994230,9cyclictest3083649-21latency_hist21:15:010
3043311994223,14cyclictest151ktimers/020:35:320
3043311994223,14cyclictest151ktimers/020:35:320
3043313994134,3cyclictest0-21swapper/222:35:182
3043313994121,12cyclictest3089169-21ntpq21:20:302
304331299411,23cyclictest3136321-21ssh22:15:251
3043313994029,7cyclictest3105148-21cron21:40:012
3043313994027,8cyclictest3048646-21ntpq19:25:222
3043313994027,10cyclictest3145618-21unixbench_singl22:25:382
3043313994027,10cyclictest3145618-21unixbench_singl22:25:382
3043313994016,8cyclictest3069282-21diskstats20:35:182
3043313994016,8cyclictest3069282-21diskstats20:35:182
3043313994012,10cyclictest3234548-21unixbench_singl00:10:382
304331299403,7cyclictest28-21ksoftirqd/122:20:271
304331299403,7cyclictest28-21ksoftirqd/122:20:271
3043312994035,2cyclictest0-21swapper/100:40:011
3043312994035,2cyclictest0-21swapper/100:40:011
3043312994032,5cyclictest3123220-21latency22:00:201
304331399394,29cyclictest3238301-21ssh00:15:322
3043313993932,4cyclictest3132657-21grep22:10:372
3043313993928,5cyclictest0-21swapper/221:55:262
3043313993916,15cyclictest3170325-21ssh22:55:232
3043313993915,8cyclictest3059026-21memory20:00:212
3043313993914,7cyclictest3078659-21ntpq21:05:242
3043313993914,7cyclictest3078659-21ntpq21:05:242
3043313993913,9cyclictest3096573-21munin-run21:30:012
3043313993912,10cyclictest3184133-21ssh23:11:232
3043312993931,5cyclictest1056-21mta-sts-daemon22:31:341
3043312993929,6cyclictest3072043-21aten_r2power_cu20:45:161
3043311993930,6cyclictest1090850-21cups-browsed21:20:400
3043311993927,8cyclictest12450irq/33-eth022:00:250
304331199391,29cyclictest3136548-21ssh22:15:320
3043313993830,5cyclictest3192124-21unixbench_multi23:20:392
3043313993830,5cyclictest3079296-21diskmemload21:15:172
3043313993829,4cyclictest3178905-21ntpq23:05:292
3043313993825,7cyclictest3050174-21ntpq19:30:232
3043313993814,7cyclictest3056239-21ntp_states19:50:262
3043313993812,9cyclictest3108475-21ssh21:43:152
3043313993810,9cyclictest3082952-21ssh21:13:522
3043313993810,9cyclictest3082952-21ssh21:13:522
3043312993827,8cyclictest3224363-21sendmail00:00:011
3043312993819,8cyclictest428-21mta-sts-daemon19:15:181
3043312993819,8cyclictest428-21mta-sts-daemon19:15:181
3043311993833,3cyclictest3198893-21systemd23:30:020
3043311993824,10cyclictest3052274-21cron19:40:010
3043313993727,7cyclictest3139135-21ssh22:19:202
3043313993724,8cyclictest3072634-21ntpq20:45:242
3043313993724,8cyclictest3071245-21ntpq20:40:242
3043313993723,9cyclictest3140818-21ntpq22:20:312
3043313993723,9cyclictest3140818-21ntpq22:20:312
3043313993723,8cyclictest3074208-21ntpq20:50:282
3043313993723,8cyclictest3074208-21ntpq20:50:282
3043313993723,8cyclictest3066648-21ntpq20:25:232
3043313993723,8cyclictest3066648-21ntpq20:25:232
3043313993716,14cyclictest0-21swapper/219:40:232
3043313993713,8cyclictest3061326-21cat20:10:022
3043313993713,6cyclictest3079296-21diskmemload22:40:472
3043313993713,11cyclictest3190347-21ssh23:19:562
304331399371,30cyclictest3229700-21ssh00:05:262
3043313993712,9cyclictest3077270-21ntp_states21:00:312
304331299373,26cyclictest3225476-21ssh00:00:251
3043312993731,3cyclictest0-21swapper/123:45:241
3043311993731,4cyclictest421-21dbus-daemon20:30:010
3043311993731,4cyclictest421-21dbus-daemon20:30:010
3043311993731,4cyclictest3084311-21ssh21:15:190
3043311993729,4cyclictest0-21swapper/023:20:010
3043311993728,6cyclictest1090850-21cups-browsed23:09:470
3043311993725,10cyclictest151ktimers/000:15:280
3043311993722,10cyclictest191rcuc/021:00:320
3043311993721,10cyclictest151ktimers/023:30:280
3043311993718,13cyclictest12350irq/32-eth023:55:290
304331199371,29cyclictest3093336-21sshd21:25:240
3043311993711,22cyclictest3119127-21sshd21:55:240
3043313993629,4cyclictest3177758-21latency_hist23:05:022
3043313993629,3cyclictest0-21swapper/223:35:202
3043313993615,4cyclictest3199646-21irqstats23:30:202
3043313993614,16cyclictest0-21swapper/219:20:222
3043313993613,6cyclictest3062160-21ntpq20:10:252
3043313993611,7cyclictest3212787-21ntp_states23:45:282
3043313993610,8cyclictest3208585-21ntpq23:40:302
304331299364,26cyclictest3161833-21ssh22:45:261
3043312993629,4cyclictest3182581-21fschecks_count23:10:161
3043312993628,5cyclictest3190063-21kworker/u8:7-xprtiod23:40:301
3043312993628,5cyclictest3100948-21latency_hist21:35:021
3043312993628,5cyclictest3070813-21fschecks_count20:40:151
3043312993628,5cyclictest3052831-21fschecks_time19:40:161
3043312993628,5cyclictest0-21swapper/123:20:011
304331299362,29cyclictest3114837-21ssh21:50:251
304331399359,7cyclictest3151486-21ssh22:33:392
3043313993525,6cyclictest421-21dbus-daemon20:00:012
3043313993524,8cyclictest421-21dbus-daemon19:39:452
3043313993523,8cyclictest3250876-21ntpq00:30:262
3043313993514,15cyclictest0-21swapper/220:20:272
3043313993512,7cyclictest3167306-21ssh22:51:372
3043313993511,7cyclictest3195761-21ntpq23:25:252
3043312993528,4cyclictest3156492-21ssh22:40:001
3043312993527,5cyclictest439-21systemd-logind19:35:011
3043312993523,8cyclictest420-21avahi-daemon22:56:241
3043311993529,4cyclictest3064331-21systemd-run20:20:010
3043311993529,3cyclictest0-21swapper/019:35:010
3043311993529,3cyclictest0-21swapper/000:05:020
3043311993523,10cyclictest0-21swapper/000:05:270
3043311993522,9cyclictest12350irq/32-eth000:25:300
3043311993522,10cyclictest3068099-21needreboot20:30:230
3043311993522,10cyclictest3068099-21needreboot20:30:230
3043311993516,14cyclictest12350irq/32-eth022:45:270
304331399349,8cyclictest3044139-21ntpq19:10:192
304331399349,8cyclictest3044139-21ntpq19:10:192
3043313993423,7cyclictest3063622-21/usr/sbin/munin20:15:232
3043313993420,10cyclictest3054636-21ntpq19:45:242
304331299343,26cyclictest3229828-21ssh00:05:301
3043312993427,4cyclictest0-21swapper/100:30:291
3043312993427,3cyclictest0-21swapper/122:40:261
3043312993427,3cyclictest0-21swapper/121:20:301
304331299342,24cyclictest3191500-21ssh23:20:261
3043312993421,10cyclictest695-21cron21:45:011
3043312993418,3cyclictest171rcu_preempt21:35:241
3043311993423,7cyclictest151ktimers/020:50:290
3043311993423,7cyclictest151ktimers/020:50:290
3043312993323,7cyclictest3217419-21timerandwakeup23:50:361
3043312993323,7cyclictest3217419-21timerandwakeup23:50:361
304331299332,22cyclictest3145083-21ssh22:25:301
304331299332,22cyclictest3145083-21ssh22:25:301
3043312993321,5cyclictest3169265-21ssh22:55:021
304331199333,22cyclictest3149114-21ssh22:30:210
304331199332,26cyclictest3153296-21ssh22:35:230
304331199332,24cyclictest3212841-21ssh23:45:300
304331199331,27cyclictest3250858-21ssh00:30:250
304331199331,25cyclictest3145076-21ssh22:25:300
304331199331,25cyclictest3145076-21ssh22:25:300
3043312993223,6cyclictest1090850-21cups-browsed19:25:161
3043312993223,5cyclictest3052318-21lockfile-remove19:40:011
3043312993220,3cyclictest3087933-21sendmail21:20:001
3043312993215,3cyclictest171rcu_preempt19:15:011
3043312993215,3cyclictest171rcu_preempt19:15:011
3043312993213,12cyclictest171rcu_preempt23:00:271
304331299321,25cyclictest28-21ksoftirqd/122:05:271
304331199324,6cyclictest171rcu_preempt23:20:320
3043311993223,6cyclictest420-21avahi-daemon23:11:590
3043311993223,6cyclictest420-21avahi-daemon22:24:290
3043311993223,6cyclictest420-21avahi-daemon22:24:290
3043311993222,8cyclictest1-21systemd00:38:240
3043311993222,8cyclictest1-21systemd00:38:240
3043311993220,8cyclictest12350irq/32-eth023:35:220
3043311993219,11cyclictest3055796-21fschecks_count19:50:160
3043312993125,2cyclictest0-21swapper/120:10:241
3043312993123,5cyclictest3196323-21unixbench_multi23:25:361
3043312993123,5cyclictest3057311-21fschecks_time19:55:171
304331299311,9cyclictest0-21swapper/119:20:261
304331199313,11cyclictest3106229-21ssh21:40:230
304331199311,24cyclictest3166188-21ssh22:50:290
304331299302,25cyclictest271ktimers/121:00:311
3043311993022,5cyclictest3216181-21/usr/sbin/munin23:50:160
3043311993022,5cyclictest3216181-21/usr/sbin/munin23:50:160
3043311993021,6cyclictest151ktimers/020:45:250
304331199301,23cyclictest3170433-21ssh22:55:280
3043313992915,9cyclictest3043259-21latency_hist19:10:022
3043312992917,4cyclictest3054758-21ntpq19:45:291
304331199298,15cyclictest753-21ntpd19:45:270
3043311992923,3cyclictest3044815-21systemd-run19:15:010
3043311992923,3cyclictest3044815-21systemd-run19:15:010
304331199291,24cyclictest3047207-21modprobe19:20:240
304331199291,20cyclictest3110552-21ssh21:45:240
3043311992813,13cyclictest151ktimers/021:05:300
3043311992813,13cyclictest151ktimers/021:05:300
3043312992718,3cyclictest171rcu_preempt20:00:241
3043312992713,4cyclictest0-21swapper/119:50:241
3043311992717,7cyclictest3071787-21cron20:45:000
304331299268,8cyclictest3069302-21grep20:35:191
304331299268,8cyclictest3069302-21grep20:35:191
304331299261,6cyclictest0-21swapper/120:55:271
304331199262,17cyclictest3174701-21ssh23:00:300
3043311992620,4cyclictest3058229-21head19:55:350
3043311992620,4cyclictest3038000-21kworker/u8:8-xprtiod20:10:010
304331299253,16cyclictest428-21mta-sts-daemon20:30:261
304331299253,16cyclictest428-21mta-sts-daemon20:30:261
3043312992519,3cyclictest0-21swapper/120:55:011
3043312992519,3cyclictest0-21swapper/120:55:011
3043311992519,4cyclictest3048929-21sensors_fan19:25:330
3043311992518,5cyclictest3062164-21collect220:10:250
3043311992517,6cyclictest3075997-21timerandwakeup20:55:360
3043311992516,7cyclictest1051-21mta-sts-daemon19:20:000
3043311992516,7cyclictest1051-21mta-sts-daemon19:20:000
304331299243,5cyclictest3066715-21ntpq20:25:271
304331299243,5cyclictest3066715-21ntpq20:25:271
3043312992418,3cyclictest0-21swapper/119:10:021
304331199242,16cyclictest3065163-21ntpq20:20:250
304331299235,6cyclictest171rcu_preempt20:20:261
304331499182,8cyclictest0-21swapper/322:15:243
304331199181,5cyclictest421-21dbus-daemon19:10:010
304331499173,7cyclictest0-21swapper/321:35:293
304331499171,8cyclictest0-21swapper/321:30:263
304331499162,8cyclictest0-21swapper/323:55:263
304331499153,6cyclictest0-21swapper/323:05:263
304331499153,6cyclictest0-21swapper/319:10:213
304331499153,6cyclictest0-21swapper/319:10:213
304331499152,8cyclictest0-21swapper/322:30:203
304331499152,8cyclictest0-21swapper/321:25:243
304331499152,7cyclictest0-21swapper/320:40:243
304331499152,7cyclictest0-21swapper/300:05:283
304331499152,6cyclictest0-21swapper/320:05:243
304331499151,7cyclictest0-21swapper/323:25:243
304331499151,7cyclictest0-21swapper/300:25:263
304331499151,7cyclictest0-21swapper/300:15:263
304331499151,6cyclictest0-21swapper/319:50:223
304331499151,5cyclictest0-21swapper/320:45:243
304331499143,6cyclictest0-21swapper/323:15:243
304331499143,6cyclictest0-21swapper/321:45:203
304331499143,6cyclictest0-21swapper/321:20:233
304331499142,7cyclictest0-21swapper/323:50:253
304331499142,7cyclictest0-21swapper/323:50:253
304331499142,7cyclictest0-21swapper/322:40:263
304331499142,6cyclictest0-21swapper/323:10:123
304331499142,6cyclictest0-21swapper/322:35:183
304331499142,5cyclictest0-21swapper/319:55:243
304331499141,8cyclictest0-21swapper/323:20:323
304331499141,8cyclictest0-21swapper/321:40:343
304331499141,8cyclictest0-21swapper/321:10:263
304331499141,8cyclictest0-21swapper/321:10:263
304331499141,7cyclictest0-21swapper/300:00:173
304331499133,8cyclictest0-21swapper/300:20:203
304331499133,6cyclictest0-21swapper/319:30:243
304331499132,7cyclictest0-21swapper/321:50:253
304331499132,6cyclictest0-21swapper/322:50:163
304331499132,6cyclictest0-21swapper/322:45:293
304331499132,5cyclictest0-21swapper/323:30:153
304331499131,7cyclictest0-21swapper/323:35:223
304331499131,7cyclictest0-21swapper/322:25:293
304331499131,7cyclictest0-21swapper/322:25:293
304331499131,7cyclictest0-21swapper/322:05:233
304331499131,7cyclictest0-21swapper/322:00:253
304331499131,7cyclictest0-21swapper/320:00:293
304331499131,6cyclictest0-21swapper/320:50:253
304331499131,6cyclictest0-21swapper/320:50:253
304331499131,6cyclictest0-21swapper/319:45:243
304331499131,5cyclictest0-21swapper/320:55:253
304331499124,6cyclictest0-21swapper/321:15:293
304331499124,6cyclictest0-21swapper/320:15:243
304331499123,5cyclictest0-21swapper/322:10:193
304331499123,5cyclictest0-21swapper/319:15:243
304331499123,5cyclictest0-21swapper/319:15:243
304331499122,6cyclictest0-21swapper/323:00:193
304331499122,6cyclictest0-21swapper/320:20:233
304331499122,6cyclictest0-21swapper/319:40:013
304331499121,7cyclictest0-21swapper/322:55:213
304331499121,7cyclictest0-21swapper/322:20:183
304331499121,7cyclictest0-21swapper/322:20:183
304331499121,6cyclictest0-21swapper/323:45:183
304331499121,6cyclictest0-21swapper/320:30:243
304331499121,6cyclictest0-21swapper/320:30:243
304331499121,6cyclictest0-21swapper/300:10:213
304331499121,5cyclictest0-21swapper/320:10:193
304331499113,6cyclictest0-21swapper/321:55:133
304331499112,7cyclictest0-21swapper/300:30:243
304331499112,5cyclictest0-21swapper/319:25:173
304331499111,6cyclictest0-21swapper/320:25:323
304331499111,6cyclictest0-21swapper/320:25:323
304331499111,6cyclictest0-21swapper/319:40:243
304331499111,6cyclictest0-21swapper/300:35:243
304331499111,6cyclictest0-21swapper/300:35:243
304331499111,5cyclictest0-21swapper/321:00:293
304331499110,6cyclictest0-21swapper/323:40:153
304331499102,5cyclictest0-21swapper/319:20:243
30433149992,4cyclictest0-21swapper/320:35:163
30433149992,4cyclictest0-21swapper/320:35:163
30433149991,5cyclictest0-21swapper/321:05:153
30433149991,5cyclictest0-21swapper/321:05:153
30433149981,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional