You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 22:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Tue Mar 31, 2026 12:43:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2225166996447,11cyclictest421-21dbus-daemon12:25:301
2225166996048,8cyclictest2400472-21/usr/sbin/munin12:00:021
2225165995619,29cyclictest2352389-21ssh11:00:270
222516699494,36cyclictest2290938-21ssh09:45:271
2225166994939,6cyclictest2302156-21systemd-run10:00:021
2225166994939,6cyclictest2302156-21systemd-run10:00:021
2225166994936,8cyclictest439-21systemd-logind10:00:401
2225165994941,5cyclictest2397539-21ntpq11:55:320
2225166994839,5cyclictest2318550-21systemd-run10:20:011
2225166994638,4cyclictest0-21swapper/110:55:011
2225166994635,8cyclictest439-21systemd-logind12:15:011
2225165994637,7cyclictest2238656-21sh07:55:010
2225165994637,7cyclictest2238656-21sh07:55:010
2225165994534,6cyclictest2290931-21ssh09:45:270
2225167994435,5cyclictest2402225-21grep12:00:412
2225167994434,4cyclictest0-21swapper/210:10:352
2225167994433,8cyclictest2269389-21sendmail09:20:012
2225165994430,5cyclictest420-21avahi-daemon10:25:300
222516799437,3cyclictest2261139-21diskmemload09:55:352
222516799437,3cyclictest2261139-21diskmemload09:55:352
222516799435,28cyclictest2262344-21ssh09:10:302
2225167994334,6cyclictest421-21dbus-daemon11:25:012
2225167994334,6cyclictest421-21dbus-daemon11:25:012
2225166994331,9cyclictest2266911-21timerwakeupswit09:15:401
222516599434,30cyclictest2278716-21ssh09:30:280
222516799424,20cyclictest2303219-21ssh10:00:302
2225167994231,7cyclictest420-21avahi-daemon12:35:472
2225166994234,5cyclictest2359469-21ssh11:10:011
2225165994235,5cyclictest1360225-21cups-browsed11:20:430
2225165994235,5cyclictest1360225-21cups-browsed11:20:430
2225165994235,4cyclictest2388204-21latency_hist11:45:020
2225167994133,5cyclictest2240125-21cron08:00:002
2225167994128,10cyclictest2409659-21needreboot12:10:282
222516699414,30cyclictest2413985-21ssh12:15:351
2225166994134,3cyclictest0-21swapper/107:35:011
2225166994133,5cyclictest2339037-21systemd-run10:45:021
2225166994133,5cyclictest2339037-21systemd-run10:45:021
222516699412,33cyclictest2401555-21ssh12:00:291
2225165994137,2cyclictest0-21swapper/012:30:400
2225165994131,4cyclictest0-21swapper/011:35:310
2225167994030,6cyclictest1056-21mta-sts-daemon09:00:252
2225167994030,6cyclictest1056-21mta-sts-daemon09:00:252
2225166994033,4cyclictest0-21swapper/111:20:011
2225166994030,4cyclictest0-21swapper/111:50:281
2225166994027,9cyclictest2294888-21missed_timers09:50:251
222516599404,30cyclictest2327900-21ssh10:30:320
2225165994029,8cyclictest2322597-21latency_hist10:25:020
2225165994017,6cyclictest12350irq/32-eth010:10:280
2225166993931,5cyclictest2252134-21grep08:40:011
2225165993930,6cyclictest420-21avahi-daemon09:36:040
2225165993930,6cyclictest420-21avahi-daemon09:36:040
2225167993831,4cyclictest2241670-21true08:05:012
2225167993830,5cyclictest2343099-21cron10:50:002
2225167993830,3cyclictest0-21swapper/212:05:312
2225167993830,3cyclictest0-21swapper/212:05:312
2225167993828,6cyclictest2347184-21cron10:55:002
2225166993829,7cyclictest2383996-21sendmail11:40:011
2225166993829,6cyclictest2340733-21unixbench_multi10:45:421
2225166993829,5cyclictest2353018-21tail11:00:401
2225165993830,5cyclictest420-21avahi-daemon08:45:360
2225165993824,12cyclictest2318507-21run-parts10:20:010
2225165993822,8cyclictest12350irq/32-eth010:45:310
2225167993729,3cyclictest0-21swapper/212:20:302
2225167993720,13cyclictest351ktimers/208:55:332
2225167993720,13cyclictest351ktimers/208:55:332
222516699373,25cyclictest2262262-21ssh09:10:271
2225166993730,4cyclictest2351324-21systemd-run11:00:021
2225166993728,5cyclictest0-21swapper/107:50:011
2225166993728,5cyclictest0-21swapper/107:50:011
2225166993727,6cyclictest0-21swapper/111:45:011
2225165993730,5cyclictest548-21gdbus09:15:010
2225165993730,5cyclictest2392708-21sh11:50:180
2225165993725,10cyclictest2287546-21unixbench_singl09:40:420
222516799363,27cyclictest2356714-21sh11:05:362
2225167993630,3cyclictest0-21swapper/212:30:422
2225167993628,5cyclictest2416827-21sendmail12:20:012
2225167993627,6cyclictest2289857-21munin-plugin-st09:45:002
2225167993627,4cyclictest0-21swapper/211:00:312
2225166993628,5cyclictest2430100-21grep12:35:241
2225166993628,5cyclictest2287546-21unixbench_singl09:40:421
222516599365,5cyclictest171rcu_preempt09:15:350
2225165993629,4cyclictest2225170-21systemd-run07:10:010
2225165993627,7cyclictest421-21dbus-daemon10:00:400
2225165993627,6cyclictest1360225-21cups-browsed11:45:080
222516599361,29cyclictest2372891-21ssh11:25:300
222516599361,29cyclictest2372891-21ssh11:25:300
222516799357,17cyclictest2377076-21ssh11:30:342
2225167993527,5cyclictest2336613-21unixbench_multi10:40:412
2225167993527,5cyclictest2336613-21unixbench_multi10:40:412
2225167993527,5cyclictest2252126-21cron08:40:012
222516799352,27cyclictest2372843-21ssh11:25:292
222516799352,27cyclictest2372843-21ssh11:25:292
222516799352,26cyclictest2389296-21sh11:45:302
2225166993529,3cyclictest0-21swapper/112:10:001
2225166993529,3cyclictest0-21swapper/112:10:001
2225166993528,3cyclictest0-21swapper/109:40:011
2225166993528,3cyclictest0-21swapper/109:40:011
2225166993527,5cyclictest2334871-21ssh10:40:011
2225166993525,7cyclictest2257496-21ntp_kernel_pll_08:55:281
2225166993525,7cyclictest2257496-21ntp_kernel_pll_08:55:281
2225166993522,8cyclictest2270549-21sh09:20:301
2225165993529,3cyclictest0-21swapper/011:30:290
2225165993527,5cyclictest1472561-21munin-node08:25:440
2225165993526,6cyclictest2252135-21cut08:40:020
2225167993427,3cyclictest0-21swapper/211:50:302
2225167993424,4cyclictest0-21swapper/209:20:332
222516799342,23cyclictest2261139-21diskmemload09:30:282
222516699345,20cyclictest428-21mta-sts-daemon11:30:271
2225166993426,5cyclictest421-21dbus-daemon10:25:011
2225166993426,5cyclictest2392235-21systemd-run11:50:011
2225166993425,4cyclictest0-21swapper/109:25:391
222516699341,27cyclictest2360585-21ssh11:10:281
222516699340,8cyclictest171rcu_preempt08:20:341
2225165993428,3cyclictest1055-21mta-sts-daemon11:10:280
2225165993427,4cyclictest0-21swapper/010:45:010
2225165993427,4cyclictest0-21swapper/010:45:010
2225165993426,5cyclictest1360225-21cups-browsed12:22:450
2225165993426,5cyclictest1360225-21cups-browsed09:57:460
2225165993426,5cyclictest1360225-21cups-browsed09:57:460
2225165993423,8cyclictest2259741-21latency_hist09:05:020
2225165993423,8cyclictest2259741-21latency_hist09:05:020
222516799336,5cyclictest171rcu_preempt09:50:272
222516799335,23cyclictest2282822-21ssh09:35:312
222516799335,23cyclictest2282822-21ssh09:35:312
2225167993328,2cyclictest0-21swapper/211:00:012
2225167993326,4cyclictest0-21swapper/210:25:332
2225167993325,5cyclictest2328360-21unixbench-2d10:30:402
2225167993325,5cyclictest2318727-21latency_hist10:20:032
2225166993327,3cyclictest0-21swapper/107:40:011
2225166993325,5cyclictest2425375-21acpi12:30:161
2225166993325,5cyclictest2256525-21unixbench_multi08:50:371
2225166993324,6cyclictest1360225-21cups-browsed07:40:301
222516699331,25cyclictest2368889-21ssh11:20:341
222516699331,25cyclictest2368889-21ssh11:20:341
2225165993327,4cyclictest2400909-21aten_r2power_en12:00:190
2225165993326,4cyclictest2348954-21unixbench_singl10:55:410
2225165993326,4cyclictest2226719-21latency_hist07:15:020
2225165993325,6cyclictest2240961-21needreboot08:00:250
2225165993325,6cyclictest0-21swapper/012:25:300
2225167993226,4cyclictest2225164-21cyclictest12:29:172
2225167993225,3cyclictest0-21swapper/211:20:012
2225167993223,6cyclictest0-21swapper/211:55:242
2225167993222,3cyclictest0-21swapper/211:40:292
2225166993226,3cyclictest0-21swapper/108:00:001
2225166993224,4cyclictest0-21swapper/110:10:301
2225166993223,5cyclictest1056-21mta-sts-daemon09:30:201
2225166993221,8cyclictest2261147-21idleruntime-cro09:10:001
2225166993221,6cyclictest0-21swapper/110:25:321
2225166993219,8cyclictest421-21dbus-daemon08:15:011
2225165993227,3cyclictest2244642-21debian-sa108:15:010
2225165993226,4cyclictest2227178-21fschecks_time07:15:220
2225165993225,5cyclictest1360225-21cups-browsed08:55:330
2225165993225,5cyclictest1360225-21cups-browsed08:55:330
2225165993223,6cyclictest1055-21mta-sts-daemon09:55:000
2225165993214,11cyclictest2430311-21ssh12:35:310
2225165993210,17cyclictest2307379-21ssh10:05:310
2225167993123,4cyclictest0-21swapper/211:10:322
2225167993123,4cyclictest0-21swapper/210:05:342
2225167993122,4cyclictest0-21swapper/209:45:212
2225167993121,7cyclictest2274177-21sh09:25:222
222516799310,3cyclictest171rcu_preempt08:15:352
2225166993122,6cyclictest2328414-21unixbench_multi10:30:401
222516599312,5cyclictest171rcu_preempt08:30:360
222516599311,21cyclictest2414018-21ssh12:15:360
2225167993022,5cyclictest2249689-21gltestperf08:30:232
222516799301,23cyclictest2332038-21ssh10:35:332
2225166993022,5cyclictest2310361-21latency_hist10:10:011
2225166993022,5cyclictest2259717-21latency_hist09:05:021
2225166993022,5cyclictest2259717-21latency_hist09:05:021
2225166993021,5cyclictest2372145-21head11:25:181
2225166993021,5cyclictest2372145-21head11:25:181
2225166993021,5cyclictest0-21swapper/112:20:401
2225166993019,8cyclictest2231162-21cut07:30:021
2225165993024,4cyclictest2293954-21kworker/u8:13-xprtiod10:54:150
222516599302,23cyclictest2405694-21diskmemload12:05:320
222516599302,23cyclictest2405694-21diskmemload12:05:320
2225165993021,7cyclictest2230562-21ntp_kernel_pll_07:25:290
2225165993020,7cyclictest2351537-21kworker/u8:7+nfsiod11:05:280
222516599301,9cyclictest171rcu_preempt12:10:310
222516599300,16cyclictest171rcu_preempt09:28:380
222516799293,20cyclictest2380996-21ssh11:35:292
2225167992924,2cyclictest0-21swapper/207:15:012
2225167992922,4cyclictest1054-21mta-sts-daemon10:21:122
2225167992921,5cyclictest2234813-21kernelversion07:40:242
2225167992918,8cyclictest2256032-21ntp_kernel_pll_08:50:272
2225166992921,5cyclictest2228685-21fschecks_count07:20:201
2225166992921,5cyclictest2226637-21cron07:15:011
2225166992919,7cyclictest2242549-21ntp_kernel_pll_08:05:301
2225165992924,3cyclictest2253210-21sendmail_mailqu08:40:400
2225165992924,3cyclictest1054-21mta-sts-daemon08:20:330
2225165992924,2cyclictest0-21swapper/011:15:250
2225165992921,6cyclictest2246144-21munin-plugin-st08:20:010
222516599290,15cyclictest171rcu_preempt07:20:230
2225167992823,3cyclictest1055-21mta-sts-daemon08:45:282
2225167992823,2cyclictest0-21swapper/207:30:012
2225167992821,4cyclictest2226913-21aten_r2power_cu07:15:192
2225167992819,6cyclictest1051-21mta-sts-daemon08:25:252
2225166992820,3cyclictest0-21swapper/108:00:301
2225165992811,16cyclictest0-21swapper/007:35:330
2225167992722,2cyclictest0-21swapper/209:10:022
2225167992722,2cyclictest0-21swapper/207:50:012
2225167992722,2cyclictest0-21swapper/207:50:012
2225166992719,5cyclictest2249037-21unixbench_multi08:25:411
2225166992719,4cyclictest2229689-21kworker/u8:10-xprtiod08:32:031
2225166992718,6cyclictest420-21avahi-daemon08:44:321
222516599275,10cyclictest2332007-21ssh10:35:320
2225165992721,4cyclictest2240176-21systemd-run08:00:010
2225165992720,5cyclictest2256652-21systemd-run08:55:010
2225165992720,4cyclictest2260163-21fschecks_count09:05:220
2225167992617,7cyclictest695-21cron08:15:012
2225167992617,7cyclictest2229500-21unixbench-2d07:20:372
2225167992617,6cyclictest548-21gdbus07:50:352
2225167992617,6cyclictest548-21gdbus07:50:352
2225166992618,5cyclictest2253911-21tail08:45:191
2225166992618,4cyclictest2228195-21kthreadd07:20:011
222516599265,11cyclictest171rcu_preempt07:30:280
2225167992518,5cyclictest1055-21mta-sts-daemon08:05:242
2225167992518,4cyclictest2231673-21grep07:30:222
2225167992517,5cyclictest2246622-21cpuspeed08:20:202
2225167992516,6cyclictest2252814-21kernelversion08:40:282
2225165992519,4cyclictest261-21systemd-journal09:23:160
2225165992517,6cyclictest2236554-21ntp_kernel_pll_07:45:310
2225165992517,6cyclictest2236554-21ntp_kernel_pll_07:45:310
2225167992418,3cyclictest0-21swapper/207:40:012
222516699246,12cyclictest1472561-21munin-node07:50:271
222516699246,12cyclictest1472561-21munin-node07:50:271
2225165992420,2cyclictest0-21swapper/007:40:220
222516599232,16cyclictest2241878-21/usr/sbin/munin08:05:410
2225166992217,3cyclictest0-21swapper/107:10:021
2225166992217,2cyclictest0-21swapper/108:15:221
2225167992114,5cyclictest2225128-21munin-run07:10:012
222516899185,7cyclictest0-21swapper/309:30:293
222516899182,9cyclictest0-21swapper/312:05:313
222516899182,9cyclictest0-21swapper/312:05:313
222516899182,8cyclictest0-21swapper/312:00:293
222516899181,9cyclictest0-21swapper/311:45:293
222516899174,8cyclictest0-21swapper/311:30:293
222516899172,9cyclictest0-21swapper/311:55:353
222516899172,9cyclictest0-21swapper/310:05:343
222516899172,8cyclictest0-21swapper/311:20:273
222516899172,8cyclictest0-21swapper/311:20:273
222516899172,8cyclictest0-21swapper/310:20:353
222516899172,8cyclictest0-21swapper/309:35:283
222516899172,8cyclictest0-21swapper/309:35:283
222516899171,11cyclictest0-21swapper/309:25:193
222516899164,7cyclictest0-21swapper/309:20:293
222516899163,6cyclictest0-21swapper/311:40:263
222516899162,8cyclictest0-21swapper/312:35:273
222516899161,9cyclictest0-21swapper/311:35:333
222516899161,9cyclictest0-21swapper/310:45:253
222516899153,8cyclictest0-21swapper/309:50:273
222516899153,6cyclictest0-21swapper/311:25:283
222516899153,6cyclictest0-21swapper/311:25:283
222516899153,6cyclictest0-21swapper/310:00:193
222516899153,6cyclictest0-21swapper/308:05:283
222516899152,8cyclictest0-21swapper/312:25:263
222516899152,8cyclictest0-21swapper/311:10:353
222516899152,8cyclictest0-21swapper/310:25:293
222516899152,7cyclictest0-21swapper/308:30:283
222516899151,9cyclictest0-21swapper/312:30:303
222516899151,9cyclictest0-21swapper/312:10:293
222516899151,9cyclictest0-21swapper/310:40:213
222516899151,9cyclictest0-21swapper/310:40:213
222516899151,7cyclictest0-21swapper/309:00:253
222516899151,7cyclictest0-21swapper/309:00:253
222516899151,10cyclictest0-21swapper/312:15:233
222516899145,6cyclictest0-21swapper/310:35:263
222516899144,7cyclictest0-21swapper/308:15:013
222516899144,4cyclictest0-21swapper/307:55:283
222516899143,7cyclictest0-21swapper/307:35:253
222516899143,6cyclictest0-21swapper/311:00:273
222516899143,6cyclictest0-21swapper/310:10:293
222516899142,8cyclictest0-21swapper/310:30:283
222516899142,8cyclictest0-21swapper/309:10:243
222516899142,7cyclictest0-21swapper/312:20:303
222516899142,7cyclictest0-21swapper/307:30:273
222516899141,9cyclictest0-21swapper/309:45:313
222516899141,8cyclictest0-21swapper/311:15:283
222516899141,8cyclictest0-21swapper/310:15:213
222516899141,7cyclictest0-21swapper/308:55:333
222516899141,7cyclictest0-21swapper/308:55:333
222516899141,6cyclictest0-21swapper/307:15:303
222516899141,10cyclictest0-21swapper/309:15:233
222516899133,5cyclictest0-21swapper/307:20:203
222516899132,7cyclictest0-21swapper/308:40:263
222516899132,6cyclictest0-21swapper/309:55:193
222516899132,6cyclictest0-21swapper/309:55:193
222516899132,5cyclictest0-21swapper/310:55:183
222516899131,9cyclictest0-21swapper/310:50:223
222516899131,8cyclictest0-21swapper/309:40:293
222516899131,7cyclictest0-21swapper/311:05:293
222516899131,7cyclictest0-21swapper/307:30:023
222516899131,6cyclictest0-21swapper/309:05:323
222516899131,6cyclictest0-21swapper/308:20:283
222516899131,6cyclictest0-21swapper/307:40:293
222516899121,7cyclictest0-21swapper/311:50:213
222516899121,6cyclictest0-21swapper/308:35:303
222516899115,4cyclictest0-21swapper/308:25:413
222516899111,6cyclictest0-21swapper/308:45:303
222516899105,3cyclictest0-21swapper/307:10:013
222516899104,3cyclictest0-21swapper/307:50:343
222516899104,3cyclictest0-21swapper/307:50:343
222516899102,4cyclictest0-21swapper/307:10:293
222516899101,6cyclictest0-21swapper/308:15:323
222516899101,6cyclictest0-21swapper/307:45:193
222516899101,6cyclictest0-21swapper/307:45:193
222516899100,6cyclictest0-21swapper/308:00:273
22251689994,3cyclictest0-21swapper/308:50:273
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional