You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-10-06 - 23:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue Oct 06, 2026 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1848699562,47cyclictest9558-21ssh12:10:160
1848699531,41cyclictest366-21ssh12:05:120
1848699511,39cyclictest29590-21ssh11:10:120
18521995024,17cyclictest33366irq/58-eth012:15:133
18486995010,29cyclictest12-21ksoftirqd/010:10:120
18499994819,21cyclictest21-21ksoftirqd/107:25:131
1848699473,34cyclictest26456-21ssh10:15:120
1848699445,33cyclictest12-21ksoftirqd/009:25:140
1852199422,27cyclictest33366irq/58-eth010:35:133
18521994121,13cyclictest33366irq/58-eth010:25:133
18499994132,3cyclictest21439-21sh10:30:131
1852199405,22cyclictest33366irq/58-eth010:15:143
18486994012,25cyclictest0-21swapper/012:25:130
1852199395,23cyclictest33366irq/58-eth012:25:133
1852199393,21cyclictest33366irq/58-eth009:30:163
1851099391,23cyclictest22650-21ssh12:35:132
1849999391,30cyclictest31499-21ssh10:00:131
1851099383,25cyclictest131rcu_preempt10:40:132
1851099383,25cyclictest131rcu_preempt10:40:122
1851099382,29cyclictest12237-21ssh11:16:412
1851099382,19cyclictest28207-21ssh11:44:582
1851099381,27cyclictest15479-21ssh11:20:132
1849999381,29cyclictest12935-21ssh09:50:111
1848699381,11cyclictest4608-21ssh09:10:150
1848699381,11cyclictest4608-21ssh09:10:150
1851099372,28cyclictest14521-21ssh12:10:412
1851099372,18cyclictest789-21ssh11:29:562
1849999376,23cyclictest21-21ksoftirqd/109:30:171
1849999373,8cyclictest131rcu_preempt08:35:121
1849999373,27cyclictest24599-21ssh11:25:141
1852199364,20cyclictest33366irq/58-eth010:55:113
1848699362,26cyclictest19537-21ssh11:40:120
1848699360,9cyclictest24190-21sed07:15:190
1852199353,17cyclictest33366irq/58-eth011:50:133
1848699351,9cyclictest17261-21kthreadcore08:15:170
1848699351,9cyclictest15771-21tr09:15:200
1848699351,9cyclictest12679-21latency_hist08:45:010
1848699351,9cyclictest12502-21fschecks_time10:25:150
1848699351,11cyclictest31648-21timerandwakeup12:20:260
1848699351,10cyclictest8980-21fschecks_time09:30:190
1848699351,10cyclictest6348-21/usr/sbin/munin11:15:230
1848699345,5cyclictest21992-21kthreadcore11:05:190
1848699342,24cyclictest13415-21ssh12:30:050
1848699341,9cyclictest30537-21missed_timers09:00:230
1848699341,9cyclictest23691-21kthreadcore08:55:200
1848699341,9cyclictest1102-21latency_hist11:30:020
1848699341,8cyclictest19483-21tr09:35:210
1848699341,11cyclictest4511-21sh11:30:210
1848699341,11cyclictest19012-21tr08:50:220
1848699341,10cyclictest26130-21fschecks_count08:25:150
1848699340,9cyclictest31400-21cut07:55:190
1848699340,11cyclictest618-21latency_hist07:25:020
18521993313,12cyclictest0-21swapper/309:55:133
1848699336,5cyclictest27359-21cat12:35:300
1848699332,22cyclictest7276-21rm10:40:140
1848699332,22cyclictest7276-21rm10:40:140
1848699331,9cyclictest32167-21/usr/sbin/munin11:45:220
1848699331,9cyclictest27548-21sendmail_mailqu12:00:270
1848699331,9cyclictest23017-21tr08:20:220
1848699331,9cyclictest18929-21if_err_eth112:15:180
1848699331,9cyclictest12976-21tune2fs08:45:150
1848699331,9cyclictest10320-21sendmail_mailqu08:05:240
1848699331,8cyclictest25425-21latency_hist10:50:020
1848699331,10cyclictest31136-21grep08:30:200
1848699330,10cyclictest4193-21awk09:10:010
1852199326,15cyclictest33366irq/58-eth010:50:133
1852199322,20cyclictest33366irq/58-eth008:25:123
1851099322,21cyclictest23577-21kworker/u8:212:00:142
1851099321,16cyclictest19889-21ssh11:04:492
1848699322,5cyclictest0-21swapper/011:56:410
1848699321,9cyclictest4784-21kthreadcore10:20:210
1848699321,9cyclictest25822-21fschecks_time10:50:150
1848699321,9cyclictest2289-21/usr/sbin/munin10:55:120
1848699321,9cyclictest22813-21sed09:55:180
1848699321,8cyclictest3936-21df09:45:140
1848699321,8cyclictest16782-21ssh10:09:490
1848699321,8cyclictest15248-21diskstats07:40:150
1848699321,8cyclictest13738-21ntp_192.168.11407:35:240
1848699321,7cyclictest15436-21unixbench_singl08:10:320
1852199313,16cyclictest33366irq/58-eth009:05:133
18521993124,3cyclictest33366irq/58-eth012:35:133
1851099312,17cyclictest22346irq/40-dwc2_hso09:50:132
1849999310,24cyclictest0-21swapper/111:05:141
1848699314,21cyclictest30588-21ssh10:35:150
1848699312,21cyclictest15494-21ssh11:20:140
1848699311,9cyclictest28166-21ls07:50:240
1848699311,9cyclictest27497-21irqcore09:40:190
1848699311,9cyclictest20612-21kthreadcore07:45:200
1848699311,8cyclictest6580-21idleruntime-cro08:05:020
1848699311,8cyclictest23344-21sed09:20:200
1848699311,8cyclictest23307-21apt-config07:15:010
1848699311,7cyclictest10885-21ps11:35:190
1848699311,10cyclictest4583-21grep07:25:300
1852199301,19cyclictest33366irq/58-eth011:10:133
1851099305,16cyclictest22346irq/40-dwc2_hso07:35:132
18499993017,9cyclictest18474-21cyclictest08:40:131
1848699306,3cyclictest0-21swapper/011:00:110
1848699304,5cyclictest19022-21sh09:51:540
1848699301,8cyclictest31676-21fschecks_count10:00:160
1848699301,7cyclictest6124-21sed07:30:170
1852199293,18cyclictest33366irq/58-eth011:40:123
1852199292,14cyclictest33366irq/58-eth009:10:163
1852199292,14cyclictest33366irq/58-eth009:10:153
18521992915,11cyclictest33366irq/58-eth009:50:133
1851099294,19cyclictest22346irq/40-dwc2_hso11:45:132
18510992920,2cyclictest0-21swapper/210:25:132
1848699290,9cyclictest7797-21basename08:40:000
1852199286,12cyclictest0-21swapper/312:00:143
1852199283,16cyclictest33366irq/58-eth009:25:133
1851099287,6cyclictest0-21swapper/211:53:412
1851099285,13cyclictest22346irq/40-dwc2_hso07:45:142
1851099284,19cyclictest10281-21ssh12:26:412
1851099283,11cyclictest542-21usb-storage11:30:112
1851099283,10cyclictest0-21swapper/211:06:412
1851099282,19cyclictest0-21swapper/207:15:152
1851099282,10cyclictest6252-21ssh12:06:412
1849999281,23cyclictest27729-21ssh12:20:111
1848699288,9cyclictest12-21ksoftirqd/011:50:140
1848699281,8cyclictest23312-21kthreadcore10:30:210
1848699281,8cyclictest18595-21latency_hist07:10:020
1852199273,16cyclictest0-21swapper/312:11:413
18521992712,10cyclictest33366irq/58-eth010:05:143
1851099273,16cyclictest753-21sendmail-mta09:20:152
1851099272,17cyclictest261rcuc/210:20:142
1852199263,15cyclictest0-21swapper/310:44:443
1852199263,15cyclictest0-21swapper/310:44:443
1851099264,14cyclictest21440-21sh10:30:142
1851099263,8cyclictest261rcuc/209:30:172
1851099262,10cyclictest4638-21ssh09:10:162
1851099262,10cyclictest4638-21ssh09:10:162
18510992614,5cyclictest591-21ssh10:53:302
1849999262,18cyclictest4717-21ssh11:49:411
1851099252,18cyclictest17162-21ssh11:37:422
1851099251,9cyclictest0-21swapper/208:00:152
1849999256,11cyclictest0-21swapper/107:45:141
1849999255,10cyclictest11403-21ssh11:00:111
1849999252,19cyclictest32018-21ssh09:25:141
1849999252,18cyclictest4186-21diskmemload11:19:531
1849999251,21cyclictest30569-21sh10:35:141
1852199243,10cyclictest33566irq/59-eth009:20:153
18521992413,6cyclictest0-21swapper/312:20:253
1851099242,15cyclictest1055-21tr07:25:182
1851099241,16cyclictest0-21swapper/210:45:132
1851099240,9cyclictest11802-21ls08:10:202
1849999248,7cyclictest21-21ksoftirqd/109:14:251
1849999248,7cyclictest21-21ksoftirqd/109:14:251
18499992412,8cyclictest201rcuc/109:15:121
1852199232,13cyclictest33366irq/58-eth012:05:143
1852199232,11cyclictest33366irq/58-eth009:15:123
18521992312,9cyclictest0-21swapper/311:45:123
1851099232,6cyclictest0-21swapper/212:31:412
1851099232,15cyclictest0-21swapper/210:55:122
18510992314,6cyclictest5981-21cron11:15:002
1851099230,7cyclictest20879-21ls10:10:222
1849999235,12cyclictest19547-21ssh11:40:131
18499992315,5cyclictest0-21swapper/107:50:131
1852199222,15cyclictest33-21ksoftirqd/311:15:123
1852199222,13cyclictest33366irq/58-eth007:35:123
1852199222,11cyclictest33366irq/58-eth008:50:153
1851099227,6cyclictest6253-21kthreadcore09:45:222
1851099223,16cyclictest21426-21ls09:35:242
1851099222,15cyclictest0-21swapper/212:22:412
18510992211,9cyclictest1049-21ssh10:18:042
1851099221,17cyclictest23464-21ssh12:00:022
1849999225,14cyclictest21-21ksoftirqd/108:05:101
1849999222,13cyclictest4989-21ssh11:30:241
18499992212,7cyclictest11137-21sshd12:27:411
1849999221,18cyclictest23733-21apt-get12:00:121
1849999221,17cyclictest22862-21apt-get09:20:151
1849999221,16cyclictest18622-21ssh12:15:141
18521992110,8cyclictest0-21swapper/311:39:313
18521992110,8cyclictest0-21swapper/310:47:443
1851099217,10cyclictest2903-21kthreadcore09:25:232
1851099212,11cyclictest22346irq/40-dwc2_hso08:55:132
1851099211,9cyclictest31397-21kthreadcore10:35:202
1851099211,9cyclictest1267-21tr10:00:222
1851099211,8cyclictest2579766ssh09:55:232
1851099211,11cyclictest28103-21kthreadcore09:00:192
1851099211,10cyclictest20943-21kthreadcore12:15:202
18510992110,9cyclictest1716-21ssh09:43:012
1851099210,19cyclictest6872-21timerandwakeup08:35:272
1849999215,12cyclictest21-21ksoftirqd/112:06:411
1849999212,14cyclictest19081-21ssh11:39:571
1849999212,10cyclictest28890-21kthreadcore09:40:211
18499992115,4cyclictest0-21swapper/110:05:311
18499992112,6cyclictest0-21swapper/111:55:261
18499992110,8cyclictest201rcuc/109:35:141
1852199202,12cyclictest0-21swapper/310:10:123
1852199201,13cyclictest0-21swapper/311:00:283
1851099203,12cyclictest22346irq/40-dwc2_hso09:15:122
1851099202,6cyclictest0-21swapper/207:30:092
1851099202,16cyclictest0-21swapper/207:10:152
1851099201,9cyclictest30985-21grep08:30:182
1851099201,9cyclictest18808-21switchtime07:40:272
1851099201,16cyclictest3045-21ntp_kernel_pll_09:05:242
1851099201,13cyclictest0-21swapper/208:25:112
1851099201,11cyclictest18889-21kthreadcore08:15:182
1851099201,10cyclictest22844-21kthreadcore08:20:212
1851099201,10cyclictest12598-21cron08:45:012
1851099201,10cyclictest11542-21kthreadcore10:05:222
18510992010,8cyclictest0-21swapper/207:50:322
18510992010,6cyclictest261rcuc/207:20:112
1849999208,9cyclictest11013-21cron11:00:021
1849999203,10cyclictest0-21swapper/108:00:131
1849999202,2cyclictest131rcu_preempt10:10:331
18499992014,4cyclictest21-21ksoftirqd/112:35:121
1849999201,13cyclictest14820-21latency08:10:251
1849999201,11cyclictest16814-21tr11:20:201
18499992010,7cyclictest18474-21cyclictest08:20:151
18499992010,7cyclictest17912-21sh12:30:341
1852199193,11cyclictest0-21swapper/311:27:403
1852199193,11cyclictest0-21swapper/311:05:403
1852199192,11cyclictest0-21swapper/308:45:143
1852199191,8cyclictest33366irq/58-eth012:30:163
18521991911,6cyclictest0-21swapper/311:20:183
1851099197,10cyclictest0-21swapper/208:50:272
1851099191,9cyclictest29624-21cpu07:55:132
1851099191,8cyclictest14684-21kthreadcore08:45:212
1851099191,10cyclictest8909-21kthreadcore08:05:192
1849999199,4cyclictest21-21ksoftirqd/110:54:291
1849999192,14cyclictest22977-21kthreadcore09:55:191
18499991911,6cyclictest24864-21sshd10:49:401
18499991911,6cyclictest22191-21cron08:55:021
18499991911,6cyclictest0-21swapper/107:10:291
1849999191,15cyclictest10486-21kthreadcore12:10:201
1849999191,11cyclictest27384-21cut09:00:151
1849999190,16cyclictest16319-21ls10:25:291
1849999190,10cyclictest3120-21wc09:05:271
1852199182,11cyclictest0-21swapper/311:55:423
1852199182,10cyclictest0-21swapper/311:30:053
1852199182,10cyclictest0-21swapper/308:00:153
1852199182,10cyclictest0-21swapper/307:15:043
1852199181,9cyclictest33366irq/58-eth010:20:143
1852199181,4cyclictest0-21swapper/307:30:103
1852199181,13cyclictest0-21swapper/307:40:233
1852199181,12cyclictest0-21swapper/307:45:293
1852199181,11cyclictest0-21swapper/308:55:163
1851099181,8cyclictest18474-21cyclictest07:10:022
1849999188,7cyclictest10310-21tr10:40:221
1849999188,7cyclictest10310-21tr10:40:211
1849999185,4cyclictest0-21swapper/108:15:201
1849999182,6cyclictest27280-21cat07:15:281
18499991812,4cyclictest0-21swapper/111:50:271
18499991812,4cyclictest0-21swapper/111:10:141
18499991812,4cyclictest0-21swapper/110:15:141
18499991810,6cyclictest21-21ksoftirqd/110:20:131
18499991810,5cyclictest21-21ksoftirqd/109:45:131
1852199179,6cyclictest0-21swapper/309:46:033
1852199179,3cyclictest0-21swapper/310:00:093
1852199179,3cyclictest0-21swapper/309:38:193
1852199178,6cyclictest33366irq/58-eth010:30:143
1852199172,7cyclictest0-21swapper/307:10:133
1852199171,11cyclictest0-21swapper/309:40:153
1852199171,11cyclictest0-21swapper/307:25:023
1849999179,5cyclictest1932-21sendmail_mailqu08:30:281
18499991713,2cyclictest21-21ksoftirqd/108:25:151
1849999171,14cyclictest28086-21kworker/1:207:20:281
1849999171,13cyclictest25385-21kthreadcore08:55:221
1849999171,13cyclictest15910-21kthreadcore07:40:191
18499991710,4cyclictest21-21ksoftirqd/107:55:151
1852199169,4cyclictest33366irq/58-eth009:00:143
1852199161,10cyclictest0-21swapper/308:35:133
1852199161,10cyclictest0-21swapper/308:15:183
1849999169,5cyclictest0-21swapper/107:30:161
1849999163,10cyclictest14253-21kthreadcore08:45:201
18499991610,4cyclictest0-21swapper/107:35:161
1852199157,5cyclictest33366irq/58-eth008:40:143
1849999153,10cyclictest18562-21latency_hist07:10:011
1852199149,3cyclictest0-21swapper/308:30:203
1852199141,8cyclictest0-21swapper/307:50:173
1852199141,8cyclictest0-21swapper/307:25:173
1852199131,7cyclictest0-21swapper/308:10:243
1852199131,10cyclictest0-21swapper/308:05:173
1852199121,9cyclictest0-21swapper/308:20:293
1852199121,9cyclictest0-21swapper/307:55:133
1852199111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional