You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-15 - 22:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Mon Jun 15, 2026 12:43:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
35642299546,44cyclictest151ktimers/009:05:280
356423995141,6cyclictest3759903-21cups-browsed11:38:081
356422995028,15cyclictest151ktimers/007:40:260
356423994737,6cyclictest567-21gdbus11:25:391
356424994638,5cyclictest378935-21systemd-run08:25:012
356424994638,5cyclictest378935-21systemd-run08:25:012
35642499462,39cyclictest403635-21ssh09:30:302
35642299462,38cyclictest151ktimers/008:15:270
356423994534,7cyclictest421-21dbus-daemon07:35:021
35642499442,35cyclictest396528-21ssh09:15:242
356424994333,6cyclictest3759903-21cups-browsed11:50:252
356422994335,4cyclictest0-21swapper/009:30:260
356424994235,4cyclictest402550-21systemd-run09:30:012
35642499421,33cyclictest428588-21ssh10:20:252
356424994133,5cyclictest446320-21munin-plugin-st10:50:012
35642399414,28cyclictest428598-21ssh10:20:261
356424994033,4cyclictest407547-21latency_hist09:40:012
356424994032,5cyclictest433871-21systemd-run10:30:012
356423994032,5cyclictest455719-21systemd-run11:05:011
356423994032,5cyclictest366910-21munin-plugin-st07:45:011
35642399400,9cyclictest28-21ksoftirqd/108:30:331
356424993931,5cyclictest454106-21unixbench-2d11:00:392
356424993929,7cyclictest1-21systemd10:45:012
356424993929,5cyclictest377393-21idleruntime-cro08:20:002
356423993932,3cyclictest0-21swapper/110:35:361
356423993931,5cyclictest489947-21ssh12:00:011
35642399392,29cyclictest441103-21ssh10:40:281
35642299393,27cyclictest512745-21ssh12:35:250
356422993931,5cyclictest0-21swapper/011:20:320
356424993829,6cyclictest465658-21fschecks_count11:20:192
356423993831,4cyclictest1002-21mta-sts-daemon07:50:271
356423993829,6cyclictest409460-21cron09:45:011
356423993829,6cyclictest3759903-21cups-browsed12:20:311
356423993828,7cyclictest439-21systemd-logind07:25:011
356423993828,7cyclictest439-21systemd-logind07:25:011
356422993829,6cyclictest3759903-21cups-browsed10:06:100
356422993826,6cyclictest151ktimers/008:00:290
356422993825,10cyclictest694-21cron07:30:000
356424993729,5cyclictest422673-21latency_hist10:10:022
356424993729,5cyclictest392398-21diskmemload12:15:012
35642499371,31cyclictest478554-21ssh11:40:302
356423993731,3cyclictest0-21swapper/109:00:011
356423993729,5cyclictest392398-21diskmemload12:00:171
356423993729,5cyclictest365452-21latency_hist07:40:021
356422993727,7cyclictest443220-21kworker/u8:18+nfsiod11:09:230
356422993720,12cyclictest151ktimers/009:00:310
356422993717,14cyclictest151ktimers/010:20:300
356424993628,5cyclictest384435-21wc08:40:322
356424993628,5cyclictest0-21swapper/211:35:012
356424993623,10cyclictest379753-21ntp_kernel_err08:25:252
356423993631,2cyclictest0-21swapper/108:40:011
356423993629,4cyclictest410997-21unixbench-2d09:45:351
356423993628,5cyclictest427624-21ssh10:20:011
356423993628,5cyclictest405681-21systemd09:35:021
356423993627,6cyclictest514909-21sendmail12:40:011
356423993626,6cyclictest0-21swapper/112:10:021
356423993615,16cyclictest261rcuc/112:10:281
356422993626,4cyclictest0-21swapper/008:40:400
356422993625,3cyclictest0-21swapper/009:55:290
356422993623,11cyclictest477473-21readlink11:40:010
356422993610,17cyclictest151ktimers/008:50:260
356424993530,3cyclictest407976-21kworker/u8:5-nfsiod10:50:132
356424993529,3cyclictest0-21swapper/211:30:012
356424993529,3cyclictest0-21swapper/210:20:012
356424993527,5cyclictest0-21swapper/208:35:022
35642399355,11cyclictest28-21ksoftirqd/107:45:261
356423993528,4cyclictest1001-21mta-sts-daemon08:10:331
356423993527,5cyclictest460186-21switchtime11:10:401
356423993526,6cyclictest414805-21head09:55:151
356423993510,5cyclictest28-21ksoftirqd/110:50:291
35642299354,22cyclictest490946-21ssh12:00:270
356422993529,4cyclictest378928-21systemd-run08:25:010
356422993529,4cyclictest378928-21systemd-run08:25:010
356422993527,3cyclictest0-21swapper/012:20:310
356422993524,8cyclictest0-21swapper/009:15:320
356422993512,19cyclictest665-21snmpd11:33:360
356424993427,4cyclictest483696-21ssh11:50:012
356424993427,4cyclictest452558-21munin-run11:00:012
356424993426,5cyclictest383390-21grep08:40:012
356424993426,5cyclictest3759903-21cups-browsed08:50:012
356424993425,3cyclictest0-21swapper/209:55:272
356423993427,4cyclictest393679-21sendmail_mailtr09:10:341
356423993426,5cyclictest1-21systemd11:20:011
356423993425,6cyclictest374292-21grep08:05:381
35642399342,27cyclictest506607-21ssh12:25:281
356423993419,10cyclictest421-21dbus-daemon11:21:471
356422993428,3cyclictest0-21swapper/008:10:020
356422993423,7cyclictest12750irq/33-eth010:40:320
356422993423,3cyclictest0-21swapper/011:05:010
356422993411,15cyclictest12650irq/32-eth011:40:340
356424993327,3cyclictest458828-21kworker/u8:9-xprtiod12:00:412
356424993326,4cyclictest0-21swapper/211:10:142
356424993325,5cyclictest493775-21if_wlan012:05:212
356424993323,4cyclictest0-21swapper/211:55:302
356424993322,5cyclictest171rcu_preempt07:30:282
356424993320,4cyclictest412360-21ssh09:49:232
356424993316,10cyclictest409472-21munin-run09:45:012
356423993325,5cyclictest444462-21mailstats10:45:361
356423993325,5cyclictest0-21swapper/110:05:011
35642399332,26cyclictest434838-21ssh10:30:271
35642399332,25cyclictest431837-21ssh10:25:301
35642299334,25cyclictest431768-21ssh10:25:270
356422993328,3cyclictest694-21cron10:40:010
356424993223,5cyclictest1-21systemd07:50:012
356424993220,9cyclictest508993-21aten_r2power_cu12:30:162
356424993217,10cyclictest0-21swapper/212:25:282
35642399329,6cyclictest28-21ksoftirqd/108:00:271
356423993224,5cyclictest1-21systemd09:20:011
356423993224,5cyclictest0-21swapper/109:10:011
356423993223,5cyclictest458969-21latency_hist11:10:021
35642399322,25cyclictest361913-21awk07:25:321
356422993226,4cyclictest485277-21grep11:50:370
356422993225,4cyclictest434977-21sshd10:30:330
356422993225,3cyclictest0-21swapper/007:25:000
356422993225,3cyclictest0-21swapper/007:25:000
356424993124,4cyclictest503033-21grep12:20:202
356424993124,4cyclictest1002-21mta-sts-daemon07:30:012
356424993124,4cyclictest0-21swapper/211:20:012
356424993123,5cyclictest393977-21unixbench_multi09:10:372
356424993121,8cyclictest357553-21switchtime07:10:352
356424993120,2cyclictest512720-21netstat12:35:242
356424993119,5cyclictest475469-21ntpq11:35:332
356424993119,5cyclictest360409-21ld07:20:322
356424993119,5cyclictest360409-21ld07:20:322
356424993116,5cyclictest171rcu_preempt09:00:262
356424993112,8cyclictest365572-21wc07:40:032
356423993125,3cyclictest0-21swapper/110:55:151
356423993124,4cyclictest1000-21mta-sts-daemon09:29:451
356423993124,3cyclictest494306-21kworker/u8:5-xprtiod12:35:021
356423993122,6cyclictest3759903-21cups-browsed09:21:461
356423993116,3cyclictest171rcu_preempt12:15:431
356422993127,2cyclictest0-21swapper/007:20:010
356422993126,3cyclictest0-21swapper/012:10:270
356422993125,4cyclictest3759903-21cups-browsed12:28:580
356422993122,6cyclictest12650irq/32-eth010:15:210
356422993116,11cyclictest151ktimers/007:35:320
356422993113,12cyclictest421-21dbus-daemon11:25:390
356424993024,4cyclictest341rcuc/207:15:282
356424993022,3cyclictest171rcu_preempt07:55:332
356424993021,4cyclictest0-21swapper/210:35:352
356424993020,4cyclictest424696-21latency_hist10:15:032
356424993019,4cyclictest413720-21ssh09:52:482
356424993018,3cyclictest457774-21ssh11:06:532
35642399308,6cyclictest28-21ksoftirqd/108:30:011
35642399307,5cyclictest28-21ksoftirqd/109:54:011
356423993024,3cyclictest0-21swapper/111:50:251
356423993024,3cyclictest0-21swapper/110:05:371
356423993022,5cyclictest477872-21/usr/sbin/munin11:40:161
356423993017,9cyclictest376794-21ntp_kernel_pll_08:15:301
356423993012,8cyclictest357894-21kworker/u8:5-xprtiod07:20:011
356423993010,5cyclictest28-21ksoftirqd/108:50:241
35642299308,16cyclictest462888-21ssh11:15:260
35642299303,21cyclictest459775-21ssh11:10:270
356422993025,3cyclictest0-21swapper/010:55:240
356422993024,4cyclictest407551-21ssh09:40:010
356422993024,3cyclictest0-21swapper/009:25:010
356422993023,4cyclictest386386-21cron08:50:010
356422993022,5cyclictest151ktimers/012:05:280
356422993022,5cyclictest0-21swapper/012:15:380
356422993016,11cyclictest0-21swapper/009:45:180
35642499297,11cyclictest1003-21mta-sts-daemon08:05:312
356424992921,5cyclictest387890-21grep08:55:012
356424992917,4cyclictest171rcu_preempt10:30:232
356424992916,2cyclictest500510-21rm12:15:342
356424992914,3cyclictest171rcu_preempt07:50:362
35642399298,15cyclictest421-21dbus-daemon11:45:281
356423992925,2cyclictest0-21swapper/111:35:011
356423992922,3cyclictest0-21swapper/109:35:321
356423992922,3cyclictest0-21swapper/108:25:001
356423992922,3cyclictest0-21swapper/108:25:001
356423992919,6cyclictest386416-21systemd-run08:50:021
35642299297,18cyclictest447323-21ntpq10:50:260
356422992923,4cyclictest489941-21grep12:00:010
35642299292,24cyclictest3759903-21cups-browsed09:10:240
356422992921,5cyclictest1003-21mta-sts-daemon09:27:550
356422992921,3cyclictest0-21swapper/011:45:300
356422992918,8cyclictest0-21swapper/010:05:010
356422992917,9cyclictest444080-21missed_timers10:45:230
356424992818,7cyclictest392420-21sh09:10:012
356424992816,4cyclictest399613-21ssh09:22:032
356424992813,3cyclictest171rcu_preempt10:00:062
356423992820,5cyclictest392398-21diskmemload10:15:021
356423992815,5cyclictest171rcu_preempt09:00:251
356422992823,2cyclictest0-21swapper/008:55:210
356422992820,6cyclictest487-21rs:main0
356422992819,7cyclictest368419-21true07:50:010
356424992720,5cyclictest0-21swapper/208:15:012
356424992719,6cyclictest1001-21mta-sts-daemon08:55:272
356424992719,4cyclictest0-21swapper/207:40:282
35642499271,21cyclictest372387-21sshd08:00:332
356423992720,4cyclictest384430-21proc_pri08:40:321
356423992719,4cyclictest371385-21latency_hist08:00:011
35642299273,13cyclictest428-21mta-sts-daemon07:10:280
356422992722,3cyclictest424608-21latency_hist10:15:020
356422992722,3cyclictest0-21swapper/008:40:010
35642299272,19cyclictest370854-21sshd07:55:320
356423992621,3cyclictest271ktimers/107:10:241
356422992619,4cyclictest511788-21systemd-run12:35:010
356422992618,6cyclictest381139-21missed_timers08:30:240
356422992618,6cyclictest356379-21cut07:10:010
356422992517,6cyclictest694-21cron07:50:030
356424992418,3cyclictest0-21swapper/207:10:012
35642399249,5cyclictest28-21ksoftirqd/107:10:011
35642299244,14cyclictest151ktimers/008:10:290
356422992419,3cyclictest151ktimers/008:25:250
356422992417,4cyclictest392398-21diskmemload09:43:320
356422992113,6cyclictest363104-21latency07:30:190
35642599195,8cyclictest0-21swapper/310:05:323
35642599193,8cyclictest0-21swapper/310:40:263
35642599183,8cyclictest0-21swapper/312:05:313
35642599183,10cyclictest0-21swapper/310:15:213
35642599181,8cyclictest0-21swapper/311:05:273
35642599173,9cyclictest0-21swapper/311:25:323
35642599173,9cyclictest0-21swapper/310:45:323
35642599173,7cyclictest0-21swapper/310:25:313
35642599173,10cyclictest0-21swapper/312:35:293
35642599172,8cyclictest0-21swapper/311:30:293
35642599172,7cyclictest0-21swapper/312:25:323
35642599171,9cyclictest0-21swapper/311:20:323
35642599171,9cyclictest0-21swapper/309:15:283
35642599171,7cyclictest0-21swapper/308:45:273
35642599164,8cyclictest0-21swapper/309:30:263
35642599164,7cyclictest0-21swapper/311:00:323
35642599162,9cyclictest0-21swapper/311:55:253
35642599162,8cyclictest0-21swapper/312:20:013
35642599162,8cyclictest0-21swapper/311:40:273
35642599162,8cyclictest0-21swapper/310:35:343
35642599162,8cyclictest0-21swapper/310:30:303
35642599161,8cyclictest0-21swapper/312:30:263
35642599161,7cyclictest0-21swapper/308:25:303
35642599161,7cyclictest0-21swapper/307:15:263
35642599161,6cyclictest0-21swapper/307:25:263
35642599161,10cyclictest0-21swapper/309:45:223
35642599154,5cyclictest0-21swapper/310:20:273
35642599153,6cyclictest0-21swapper/309:10:253
35642599152,8cyclictest0-21swapper/309:55:293
35642599152,8cyclictest0-21swapper/308:20:013
35642599152,7cyclictest0-21swapper/312:10:253
35642599152,6cyclictest0-21swapper/308:05:263
35642599152,6cyclictest0-21swapper/307:10:243
35642599152,5cyclictest0-21swapper/307:35:273
35642599151,9cyclictest0-21swapper/311:15:283
35642599151,7cyclictest0-21swapper/308:00:333
35642599151,6cyclictest0-21swapper/312:20:193
35642599151,6cyclictest0-21swapper/308:55:253
35642599142,8cyclictest0-21swapper/311:45:273
35642599142,7cyclictest0-21swapper/310:55:253
35642599142,7cyclictest0-21swapper/309:05:263
35642599142,6cyclictest0-21swapper/308:40:273
35642599142,6cyclictest0-21swapper/307:20:273
35642599142,6cyclictest0-21swapper/307:20:273
35642599142,5cyclictest0-21swapper/307:55:273
35642599141,9cyclictest0-21swapper/311:35:323
35642599141,8cyclictest0-21swapper/308:50:213
35642599141,7cyclictest0-21swapper/311:10:213
35642599133,7cyclictest0-21swapper/311:50:173
35642599133,6cyclictest0-21swapper/308:40:013
35642599133,5cyclictest0-21swapper/309:00:323
35642599132,7cyclictest0-21swapper/308:20:293
35642599132,7cyclictest0-21swapper/308:20:293
35642599131,8cyclictest0-21swapper/312:00:303
35642599131,7cyclictest0-21swapper/308:10:313
35642599131,6cyclictest0-21swapper/310:50:153
35642599131,6cyclictest0-21swapper/308:30:283
35642599131,6cyclictest0-21swapper/307:45:283
35642599131,6cyclictest0-21swapper/307:30:253
35642599122,6cyclictest0-21swapper/310:05:013
35642599122,6cyclictest0-21swapper/307:50:323
35642599122,5cyclictest0-21swapper/309:25:013
35642599121,6cyclictest0-21swapper/309:40:323
35642599121,6cyclictest0-21swapper/309:38:303
35642599112,5cyclictest0-21swapper/309:54:473
35642599112,4cyclictest0-21swapper/309:30:013
35642599102,5cyclictest0-21swapper/307:40:223
3564259991,5cyclictest0-21swapper/310:15:023
3564259991,5cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional