You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-20 - 18:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sun Sep 20, 2026 12:42:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2176339996150,6cyclictest1000-21mta-sts-daemon12:21:150
2176339995448,3cyclictest0-21swapper/009:15:010
2176340995243,6cyclictest2239814-21cron09:55:011
2176341994832,2cyclictest171rcu_preempt08:51:202
2176340994839,6cyclictest2212359-21systemd-run09:10:021
2176339994729,9cyclictest12650irq/32-eth009:36:160
2176341994636,7cyclictest2312741-21ssh11:55:012
2176341994536,6cyclictest2335594-21mailstats12:31:252
2176341994536,6cyclictest2258103-21munin-plugin-st10:25:012
217634099442,35cyclictest2256303-21ssh10:21:141
2176339994428,9cyclictest623-21sshd07:36:180
2176341994335,5cyclictest0-21swapper/211:00:012
2176341994335,5cyclictest0-21swapper/211:00:012
2176340994335,5cyclictest2203363-21systemd-run08:40:011
2176340994334,6cyclictest2195846-21systemd-run08:15:011
2176339994336,3cyclictest0-21swapper/011:46:140
2176341994235,4cyclictest1000-21mta-sts-daemon07:21:162
2176341994231,8cyclictest2340143-21awk12:40:012
2176339994232,6cyclictest0-21swapper/012:10:010
2176339994224,5cyclictest2181358-21fschecks_count07:26:080
2176341994134,3cyclictest0-21swapper/210:40:012
2176341994130,7cyclictest2253363-21ntp_states10:16:202
2176341994124,5cyclictest171rcu_preempt07:11:152
2176340994133,5cyclictest2324944-21ssh12:15:001
2176340994133,5cyclictest2324944-21ssh12:15:001
2176339994134,4cyclictest2209350-21systemd-run09:00:020
2176339994127,10cyclictest2197350-21tr08:20:010
2176339994123,12cyclictest151ktimers/010:41:170
2176341994034,3cyclictest0-21swapper/208:20:012
2176341994033,4cyclictest2179328-21grep07:20:012
2176341994032,5cyclictest2306680-21munin-plugin-st11:45:012
2176341994032,5cyclictest1003-21mta-sts-daemon10:43:582
2176341994032,4cyclictest2255386-21kworker/u8:7-rpciod11:20:012
2176341994031,6cyclictest1002-21mta-sts-daemon10:33:302
2176340994032,5cyclictest2255045-21munin-plugin-st10:20:011
2176340994030,7cyclictest2303645-21sendmail11:40:011
2176339994033,5cyclictest2242919-21ssh10:00:020
2176339994032,5cyclictest2224617-21latency_hist09:30:010
2176339994031,7cyclictest2288457-21grep11:15:000
2176339994031,7cyclictest2288457-21grep11:15:000
2176341993931,5cyclictest2324975-21systemd-run12:15:012
2176341993931,5cyclictest2324975-21systemd-run12:15:012
217634199391,6cyclictest2332309-21ssh12:26:162
217634099394,29cyclictest2222780-21ssh09:26:161
2176340993934,2cyclictest0-21swapper/112:00:011
217634099393,30cyclictest2213627-21ssh09:11:161
2176340993930,5cyclictest1001-21mta-sts-daemon10:13:251
2176339993932,4cyclictest2265279-21missed_timers10:36:120
2176339993931,4cyclictest2244126-21ssh10:01:140
2176339993931,4cyclictest2244126-21ssh10:01:140
2176339993918,13cyclictest12650irq/32-eth009:31:150
2176341993831,4cyclictest2295023-21kworker/u8:8-xprtiod11:26:162
2176341993830,5cyclictest421-21dbus-daemon11:35:012
2176341993830,4cyclictest2210859-21kworker/u8:17-xprtiod09:10:012
2176340993830,5cyclictest2247763-21grep10:06:261
2176340993830,5cyclictest2247763-21grep10:06:261
2176339993831,5cyclictest2255035-21cron10:20:010
2176339993825,9cyclictest12650irq/32-eth008:36:200
2176339993824,11cyclictest12750irq/33-eth011:16:200
217634199379,7cyclictest171rcu_preempt12:16:152
2176341993730,4cyclictest2294538-21kworker/u8:0-xprtiod11:40:012
2176341993729,5cyclictest2227641-21munin-run09:35:012
2176341993729,3cyclictest0-21swapper/212:01:172
2176340993730,3cyclictest0-21swapper/109:35:001
2176340993727,7cyclictest1-21systemd12:30:021
2176339993731,4cyclictest2259748-21awk10:26:260
2176339993725,8cyclictest2220236-21timerwakeupswit09:21:260
2176339993725,8cyclictest2220236-21timerwakeupswit09:21:260
2176339993718,14cyclictest12650irq/32-eth012:01:140
2176341993629,4cyclictest1001-21mta-sts-daemon10:54:202
2176341993627,6cyclictest2285555-21latency_hist11:10:022
2176341993617,13cyclictest351ktimers/209:46:182
2176340993628,5cyclictest2308026-21ssh11:46:181
217634099362,27cyclictest2265517-21sshd10:36:191
217634099361,30cyclictest2244147-21ssh10:01:151
217634099361,30cyclictest2244147-21ssh10:01:151
2176339993630,4cyclictest1002-21mta-sts-daemon11:51:170
2176339993612,19cyclictest2274564-21ssh10:51:170
2176341993528,5cyclictest2239831-21/usr/sbin/munin09:55:012
2176341993526,5cyclictest351ktimers/208:21:172
2176341993524,8cyclictest2216513-21missed_timers09:16:112
2176341993523,9cyclictest2313769-21kernelversion11:56:112
2176341993523,10cyclictest2209346-21kworker/u8:10+rpciod09:00:192
2176340993528,4cyclictest2188439-21latency_hist07:50:021
2176340993527,5cyclictest2319816-21if_wlan012:06:091
2176340993527,5cyclictest2191326-21idleruntime-cro08:00:011
2176340993526,5cyclictest2242858-21cron10:00:011
2176340993525,4cyclictest0-21swapper/111:51:161
217634099352,28cyclictest2329248-21sh12:21:151
2176340993521,6cyclictest271ktimers/111:01:181
2176340993521,6cyclictest271ktimers/111:01:181
217634099351,31cyclictest2184748-21sshd07:36:181
2176339993529,3cyclictest151ktimers/011:41:170
2176339993528,4cyclictest2234892-21missed_timers09:46:120
2176339993527,5cyclictest2191217-21tail07:56:270
2176339993527,5cyclictest151ktimers/011:26:180
2176339993514,15cyclictest12650irq/32-eth010:56:150
2176339993514,15cyclictest12650irq/32-eth010:56:150
2176339993513,16cyclictest151ktimers/011:56:150
2176341993429,2cyclictest0-21swapper/210:11:192
2176341993426,5cyclictest2281204-21tail11:01:282
2176341993426,5cyclictest2281204-21tail11:01:282
2176341993421,3cyclictest171rcu_preempt10:06:202
2176341993421,3cyclictest171rcu_preempt10:06:202
217634099343,26cyclictest2289736-21rm11:16:181
217634099343,23cyclictest2277641-21ssh10:56:181
217634099343,23cyclictest2277641-21ssh10:56:181
2176340993428,3cyclictest0-21swapper/109:00:011
2176340993427,4cyclictest1001-21mta-sts-daemon10:51:191
2176340993427,4cyclictest0-21swapper/109:40:011
2176340993426,5cyclictest2339823-21sh12:39:061
2176340993426,5cyclictest2286074-21aten_r2power_en11:11:061
2176340993426,5cyclictest2286074-21aten_r2power_en11:11:061
2176340993426,5cyclictest2205172-21aten_r2power_fr08:46:041
2176340993424,5cyclictest2201963-21kworker/u8:8-xprtiod09:00:241
2176340993419,11cyclictest265-21systemd-journal11:41:191
2176339993423,8cyclictest2303752-21latency_hist11:40:020
2176339993423,7cyclictest2334897-21/usr/sbin/munin12:31:080
2176339993419,8cyclictest12650irq/32-eth007:16:180
217634199335,23cyclictest2312522-21kworker/u8:7+nfsiod12:21:192
2176341993324,6cyclictest341rcuc/207:51:132
2176341993322,7cyclictest420-21avahi-daemon12:05:312
217634199331,23cyclictest2208767-21sshd08:56:192
2176340993327,3cyclictest0-21swapper/111:06:061
2176340993327,3cyclictest0-21swapper/110:35:021
2176340993326,4cyclictest2334864-21cpuspeed12:31:071
2176340993326,3cyclictest0-21swapper/111:26:191
2176340993322,7cyclictest1003-21mta-sts-daemon09:16:161
217633999334,23cyclictest2292886-21sh11:21:200
2176339993328,3cyclictest2231515-21cpuspeed09:41:060
2176339993327,4cyclictest2256557-21mailstats10:21:220
2176339993325,5cyclictest2272118-21head10:46:280
217634199323,19cyclictest2219846-21modprobe09:21:192
217634199323,19cyclictest2219846-21modprobe09:21:192
2176341993226,3cyclictest0-21swapper/211:21:212
2176341993225,3cyclictest0-21swapper/208:01:192
2176341993224,5cyclictest694-21cron08:15:012
2176341993224,3cyclictest0-21swapper/209:15:022
2176341993222,7cyclictest2259137-21sort10:26:102
2176341993222,3cyclictest2203371-21cat08:40:012
2176340993224,4cyclictest271ktimers/107:31:201
217634099322,23cyclictest2189301-21sshd07:51:191
2176340993218,9cyclictest2235082-21ssh09:46:181
2176339993223,6cyclictest0-21swapper/011:31:220
2176339993217,10cyclictest12650irq/32-eth012:11:160
2176339993217,10cyclictest12650irq/32-eth012:11:160
2176341993125,2cyclictest0-21swapper/210:01:282
2176341993125,2cyclictest0-21swapper/210:01:282
2176341993121,4cyclictest0-21swapper/208:46:192
2176340993125,3cyclictest0-21swapper/110:45:011
2176340993125,3cyclictest0-21swapper/108:35:011
2176340993123,5cyclictest2271135-21fschecks_time10:46:081
2176340993123,4cyclictest0-21swapper/107:30:011
2176340993121,7cyclictest421-21dbus-daemon11:23:301
217633999319,15cyclictest12650irq/32-eth012:26:200
217633999319,13cyclictest12650irq/32-eth011:06:170
2176339993124,4cyclictest2206340-21systemd-run08:50:010
2176339993122,5cyclictest0-21swapper/009:00:250
2176339993118,8cyclictest0-21swapper/007:51:140
2176339993113,14cyclictest1001-21mta-sts-daemon10:31:170
2176339993113,11cyclictest12650irq/32-eth012:36:220
217634199308,17cyclictest2232107-21ssh09:41:212
2176341993023,5cyclictest2270987-21ssh10:46:062
2176341993021,5cyclictest2183829-21munin-plugin-st07:35:002
217634099303,17cyclictest1393066-21cups-browsed07:16:171
2176340993025,2cyclictest0-21swapper/111:31:091
2176340993023,4cyclictest2259445-21ntp_offset10:26:181
2176339993023,5cyclictest421-21dbus-daemon07:10:020
2176339993019,8cyclictest0-21swapper/011:00:120
2176339993019,8cyclictest0-21swapper/011:00:120
2176341992923,3cyclictest0-21swapper/207:40:002
2176341992922,4cyclictest2188279-21unixbench_singl07:46:262
2176341992921,3cyclictest0-21swapper/211:46:152
2176341992917,8cyclictest265-21systemd-journal09:40:002
2176340992923,4cyclictest1001-21mta-sts-daemon08:26:171
2176340992922,3cyclictest0-21swapper/112:16:191
217634099292,20cyclictest2219847-21ssh09:21:191
217634099292,20cyclictest2219847-21ssh09:21:191
217634099292,20cyclictest2196788-21sshd08:16:191
2176340992921,5cyclictest1003-21mta-sts-daemon07:21:171
217633999293,19cyclictest1393066-21cups-browsed08:26:140
2176339992923,4cyclictest2201849-21systemd-run08:35:010
2176339992921,6cyclictest2195141-21needreboot08:11:130
217634199284,17cyclictest428-21mta-sts-daemon11:11:242
217634199284,17cyclictest428-21mta-sts-daemon11:11:242
2176341992822,3cyclictest0-21swapper/208:35:012
2176341992820,5cyclictest1393066-21cups-browsed09:29:202
2176340992822,3cyclictest0-21swapper/107:11:191
2176339992819,6cyclictest2183144-21needreboot07:31:120
2176341992719,3cyclictest0-21swapper/209:56:202
2176340992720,4cyclictest2233759-21munin-run09:45:011
2176340992720,4cyclictest1002-21mta-sts-daemon12:02:071
2176339992720,4cyclictest2238172-21kworker/u8:1-rpciod10:06:070
2176339992720,4cyclictest2238172-21kworker/u8:1-rpciod10:06:070
2176339992716,7cyclictest12650irq/32-eth009:06:230
2176339992715,10cyclictest0-21swapper/009:16:150
217633999271,13cyclictest2326387-21ps12:16:210
2176341992619,4cyclictest2193383-21gltestperf08:06:092
2176341992617,7cyclictest2191173-21unixbench-2d07:56:262
2176341992615,9cyclictest2204153-21needreboot08:41:142
2176341992615,9cyclictest2204153-21needreboot08:41:142
2176340992618,5cyclictest0-21swapper/108:41:071
2176340992618,5cyclictest0-21swapper/108:41:071
2176340992618,4cyclictest0-21swapper/108:21:171
2176340992612,11cyclictest271ktimers/107:41:171
2176339992619,4cyclictest2194244-21tail08:06:270
2176339992619,4cyclictest2191880-21fschecks_time08:01:110
2176339992617,6cyclictest2180702-21unixbench-2d07:21:270
2176339992616,6cyclictest2249523-21/usr/sbin/munin10:11:100
2176339992615,8cyclictest2238649-21unixbench_singl09:51:260
217634199253,5cyclictest0-21swapper/207:41:152
217633999252,16cyclictest12650irq/32-eth007:11:190
2176341992411,10cyclictest750-21ntpd08:26:162
2176340992418,3cyclictest0-21swapper/108:10:021
2176340992417,4cyclictest2207742-21sed08:51:261
2176339992416,5cyclictest1393066-21cups-browsed08:52:490
2176341992316,5cyclictest265-21systemd-journal07:30:012
2176340992317,3cyclictest1003-21mta-sts-daemon08:01:101
217633999232,16cyclictest2204317-21ntp_states08:41:220
217633999232,16cyclictest2204317-21ntp_states08:41:220
2176339992314,7cyclictest2198047-21latency08:21:100
217633999222,15cyclictest2187729-21ntpq07:46:150
2176339992210,7cyclictest2186215-21ntpq07:41:130
217634299213,9cyclictest0-21swapper/312:26:203
2176341992114,4cyclictest1-21systemd07:10:012
217634099218,9cyclictest421-21dbus-daemon07:10:011
217634299203,9cyclictest0-21swapper/311:16:213
217634299202,9cyclictest0-21swapper/307:16:173
217634299193,9cyclictest0-21swapper/310:46:193
217634299193,9cyclictest0-21swapper/309:51:133
217634299193,10cyclictest0-21swapper/312:31:233
217634299192,9cyclictest0-21swapper/312:21:133
217634299192,9cyclictest0-21swapper/310:26:163
217634299192,9cyclictest0-21swapper/309:46:133
217634299192,8cyclictest0-21swapper/311:21:143
217634299192,8cyclictest0-21swapper/311:06:153
217634299191,10cyclictest0-21swapper/307:56:193
217634299184,7cyclictest0-21swapper/307:41:193
217634299183,9cyclictest0-21swapper/309:36:203
217634299183,8cyclictest0-21swapper/312:36:223
217634299182,9cyclictest0-21swapper/310:21:183
217634299182,8cyclictest0-21swapper/310:06:133
217634299182,8cyclictest0-21swapper/310:06:133
217634299182,8cyclictest0-21swapper/309:21:133
217634299182,8cyclictest0-21swapper/309:21:133
217634299182,7cyclictest0-21swapper/308:06:183
217634299182,10cyclictest0-21swapper/309:01:173
217634299181,9cyclictest0-21swapper/311:36:163
217634299181,9cyclictest0-21swapper/308:36:193
217634299174,8cyclictest0-21swapper/311:46:193
217634299174,7cyclictest0-21swapper/312:11:163
217634299174,7cyclictest0-21swapper/312:11:163
217634299173,8cyclictest0-21swapper/311:56:173
217634299173,8cyclictest0-21swapper/311:01:143
217634299173,8cyclictest0-21swapper/311:01:143
217634299173,7cyclictest0-21swapper/308:56:183
217634299173,7cyclictest0-21swapper/307:46:183
217634299172,9cyclictest0-21swapper/311:51:163
217634299172,9cyclictest0-21swapper/311:26:153
217634299172,9cyclictest0-21swapper/309:16:143
217634299172,7cyclictest0-21swapper/308:21:153
217634299172,10cyclictest0-21swapper/310:41:153
217634299172,10cyclictest0-21swapper/309:41:213
217634299165,6cyclictest0-21swapper/312:01:133
217634299164,7cyclictest0-21swapper/312:16:193
217634299164,6cyclictest0-21swapper/308:01:193
217634299164,5cyclictest0-21swapper/309:06:163
217634299163,8cyclictest0-21swapper/308:16:213
217634299163,7cyclictest0-21swapper/308:46:143
217634299162,9cyclictest0-21swapper/310:01:103
217634299162,9cyclictest0-21swapper/310:01:103
217634299162,8cyclictest0-21swapper/310:56:143
217634299162,8cyclictest0-21swapper/310:56:143
217634299162,8cyclictest0-21swapper/310:36:203
217634299162,8cyclictest0-21swapper/307:31:203
217634299162,7cyclictest0-21swapper/311:30:293
217634299162,7cyclictest0-21swapper/309:56:063
217634299162,6cyclictest0-21swapper/308:51:163
217634299162,10cyclictest0-21swapper/310:51:183
217634299161,9cyclictest0-21swapper/310:11:143
217634299154,7cyclictest0-21swapper/309:31:153
217634299153,7cyclictest0-21swapper/310:15:403
217634299152,9cyclictest0-21swapper/311:11:203
217634299152,9cyclictest0-21swapper/311:11:203
217634299152,9cyclictest0-21swapper/309:26:203
217634299152,8cyclictest0-21swapper/310:31:183
217634299152,7cyclictest0-21swapper/308:11:183
217634299152,7cyclictest0-21swapper/307:26:183
217634299152,6cyclictest0-21swapper/308:31:183
217634299152,5cyclictest0-21swapper/307:51:133
217634299151,8cyclictest0-21swapper/307:36:183
217634299142,7cyclictest0-21swapper/312:10:023
217634299142,7cyclictest0-21swapper/311:41:133
217634299142,7cyclictest0-21swapper/307:21:173
217634299141,7cyclictest0-21swapper/307:11:153
217634299132,7cyclictest0-21swapper/309:11:123
217634299132,7cyclictest0-21swapper/308:41:173
217634299132,7cyclictest0-21swapper/308:41:173
217634299131,6cyclictest0-21swapper/308:25:193
217634299112,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional