You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-03 - 16:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Sat Jan 03, 2026 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21734991026,86cyclictest0-21swapper/008:00:260
21734991026,86cyclictest0-21swapper/008:00:260
227142950,2sleep30-21swapper/310:10:313
2173499887,64cyclictest0-21swapper/010:05:270
190562870,5sleep12173599cyclictest08:30:341
190562870,5sleep12173599cyclictest08:30:341
36982730,2sleep10-21swapper/107:50:131
21734997310,49cyclictest810-21munin-plugin-st12:10:120
2173499727,21cyclictest0-21swapper/007:30:240
21734997126,11cyclictest0-21swapper/010:30:250
2173499706,50cyclictest18137-21basename08:30:130
2173499704,53cyclictest25898-21grep08:50:320
48332690,2sleep10-21swapper/109:20:341
2173499696,52cyclictest8187-21cut11:00:300
2173499692,49cyclictest23903-21aten_rbpower_en08:45:270
2173499689,49cyclictest20222-21cron10:05:120
2173499689,49cyclictest20222-21cron10:05:120
312152670,1sleep20-21swapper/210:35:132
29102660,2sleep00-21swapper/012:15:280
2173499668,41cyclictest24217-21latency_hist11:45:130
21734996629,21cyclictest0-21swapper/010:50:260
2173499655,43cyclictest15052-21cron11:20:130
2173499647,42cyclictest23367-21grep10:10:400
2173499647,41cyclictest9959-21cron12:35:120
21734996435,8cyclictest128-21systemd-journal12:00:120
21734996435,8cyclictest128-21systemd-journal12:00:120
21734996421,13cyclictest0-21swapper/009:55:250
2173699625,6cyclictest0-21swapper/207:15:252
2173499625,44cyclictest31590-21tune2fs10:35:270
207342620,5sleep20-21swapper/207:05:402
138342620,5sleep32173799cyclictest11:15:293
254512610,5sleep125454-21aten_rbpower_en07:20:271
2173499619,49cyclictest101ktimersoftd/008:35:130
2173499619,49cyclictest101ktimersoftd/008:35:130
2173499616,52cyclictest101ktimersoftd/011:25:400
2173499616,42cyclictest28589-21tail08:55:410
2173499616,35cyclictest0-21swapper/011:20:320
2173499596,40cyclictest29355-21latency_hist10:30:130
2173499592,54cyclictest14290-21sshd08:17:320
2173499591,24cyclictest15049-21cut08:20:300
21734995828,8cyclictest32172-21grep09:05:380
2173499572,42cyclictest27142-21latency_hist07:25:130
229902560,4sleep02173499cyclictest08:40:400
2173499566,46cyclictest101ktimersoftd/009:35:120
2173499562,24cyclictest0-21swapper/009:45:130
21734995618,12cyclictest0-21swapper/007:40:240
2173499561,40cyclictest0-21swapper/011:50:270
21736995517,22cyclictest24101-21fschecks_count08:45:302
2173599552,6cyclictest0-21swapper/108:40:261
21735995512,13cyclictest0-21swapper/112:10:241
2173499555,33cyclictest24930-21uname11:45:320
21734995528,17cyclictest0-21swapper/008:35:360
21734995511,35cyclictest0-21swapper/009:30:130
2173799547,24cyclictest0-21swapper/308:45:263
21736995418,12cyclictest22043-21/usr/sbin/munin08:40:242
2173599542,18cyclictest0-21swapper/112:35:271
21735995411,20cyclictest0-21swapper/108:10:251
21734995434,12cyclictest6884-21cut12:25:310
21734995434,12cyclictest6884-21cut12:25:310
2173499542,40cyclictest0-21swapper/009:00:580
2173499541,27cyclictest0-21swapper/009:10:190
2173499541,27cyclictest0-21swapper/009:10:190
21736995320,17cyclictest1579-21latency12:10:332
21736995317,12cyclictest3782-21expr12:15:402
21734995333,8cyclictest0-21swapper/010:45:330
21734995333,8cyclictest0-21swapper/010:45:330
21734995330,5cyclictest15155-21aten_rbpower_vo09:50:300
21734995312,10cyclictest61-21kswapd007:10:240
21734995312,10cyclictest61-21kswapd007:10:240
2173499531,12cyclictest28190-21timerwakeupswit07:25:410
2173499526,4cyclictest31029-21grep12:00:420
2173499524,34cyclictest1373-21awk10:40:340
2173499524,33cyclictest10820-21cut08:10:120
21734995217,20cyclictest9905-21fschecks_time09:35:310
213412523,40sleep00-21swapper/007:08:380
21736995113,14cyclictest16435-21cat08:25:132
21735995111,19cyclictest0-21swapper/110:20:241
21734995128,11cyclictest0-21swapper/011:39:400
21734995124,6cyclictest24401-21fschecks_time10:15:290
21734995122,5cyclictest0-21swapper/007:50:310
2173499511,28cyclictest3032-21grep09:15:340
21736995020,18cyclictest10407-21/usr/sbin/munin09:35:382
21736995015,10cyclictest31555-21/usr/sbin/munin10:35:262
21736995014,7cyclictest25028-21memory11:45:332
21735995010,20cyclictest0-21swapper/107:10:251
21735995010,20cyclictest0-21swapper/107:10:251
21734995033,9cyclictest10362-21cat11:05:350
2173499502,7cyclictest0-21swapper/007:15:250
21734995022,6cyclictest0-21swapper/012:10:330
2173499501,45cyclictest20139-21unixbench_multi11:30:410
2173499501,26cyclictest0-21swapper/011:10:240
2173699498,27cyclictest0-21swapper/209:25:262
2173499494,32cyclictest5920-21chrt12:21:110
21734994925,11cyclictest0-21swapper/012:35:300
21734994923,16cyclictest0-21swapper/008:12:560
2173699489,10cyclictest7750irq/56-dwc_otg10:05:382
2173699483,9cyclictest7750irq/56-dwc_otg12:35:252
21735994821,5cyclictest0-21swapper/111:45:241
2173699478,25cyclictest0-21swapper/207:20:262
2173699477,25cyclictest0-21swapper/212:30:262
2173699473,14cyclictest15115-21latency_hist11:20:132
21736994720,15cyclictest15840-21if_wlan011:20:312
21734994734,3cyclictest0-21swapper/007:59:420
2173499472,33cyclictest0-21swapper/010:20:400
2173699467,24cyclictest0-21swapper/209:00:262
2173699467,24cyclictest0-21swapper/207:40:262
2173699463,5cyclictest0-21swapper/209:20:232
21736994621,12cyclictest21100-21/usr/sbin/munin08:35:382
21736994616,8cyclictest25867-21df08:50:312
21736994610,11cyclictest8535-21missed_timers11:00:342
21734994621,6cyclictest2086-21aten_rbpower_po07:45:270
2173699458,25cyclictest0-21swapper/210:50:282
2173699457,24cyclictest0-21swapper/209:30:262
2173699456,25cyclictest0-21swapper/211:10:252
21736994514,12cyclictest29358-21cat10:30:132
21736994511,11cyclictest7950irq/56-dwc_otg_08:15:302
21734994511,25cyclictest0-21swapper/010:55:270
188812450,6sleep218886-21fschecks_count10:00:322
188812450,6sleep218886-21fschecks_count10:00:322
21736994416,8cyclictest16791-21diskstats08:25:302
21736994412,13cyclictest28647-21/usr/sbin/munin08:55:412
2173499441,27cyclictest0-21swapper/009:46:310
2173799436,7cyclictest7750irq/56-dwc_otg08:40:263
21736994314,14cyclictest19032-21/usr/sbin/munin08:30:342
21736994314,14cyclictest19032-21/usr/sbin/munin08:30:342
21736994310,12cyclictest12827-21chrt09:43:292
21735994323,5cyclictest1324750wget09:45:261
21734994320,14cyclictest0-21swapper/009:22:120
2173499431,28cyclictest0-21swapper/007:35:370
2173699427,21cyclictest0-21swapper/212:00:272
21736994212,18cyclictest5111-21chrt07:52:072
2173699421,12cyclictest0-21swapper/212:20:372
2173799411,22cyclictest0-21swapper/311:05:253
2173699419,11cyclictest22374-21seq11:40:022
2173699416,10cyclictest7950irq/56-dwc_otg_09:50:372
2173699412,7cyclictest27860-21latency_hist11:55:132
21735994111,25cyclictest0-21swapper/108:45:261
21737994024,4cyclictest7950irq/56-dwc_otg_08:00:283
21737994024,4cyclictest7950irq/56-dwc_otg_08:00:283
21737994017,5cyclictest288712latency_hist07:30:133
2173699409,11cyclictest0-21swapper/210:49:232
2173699409,11cyclictest0-21swapper/210:49:232
21736994031,7cyclictest6576-21dpkg12:25:192
21736994031,7cyclictest6576-21dpkg12:25:192
21736994018,7cyclictest23455-21users10:10:402
21736994014,11cyclictest7141-21taskset07:59:582
2173599403,4cyclictest7750irq/56-dwc_otg11:50:261
21735994012,9cyclictest0-21swapper/109:05:381
2173599401,26cyclictest0-21swapper/111:40:131
21737993919,8cyclictest0-21swapper/307:40:403
21737993913,8cyclictest6884-21wc10:55:383
2173699398,12cyclictest3961-21idleruntime-cro09:20:122
2173699397,24cyclictest0-21swapper/207:30:262
2173699397,20cyclictest0-21swapper/208:00:262
2173699397,20cyclictest0-21swapper/208:00:262
2173699393,6cyclictest0-21swapper/210:55:272
2173699393,6cyclictest0-21swapper/209:10:282
2173699393,6cyclictest0-21swapper/209:10:282
2173699392,4cyclictest7850irq/56-dwc_otg_07:35:252
21737993815,11cyclictest9583-21/usr/sbin/munin12:30:413
2173699387,17cyclictest23279-21taskset07:14:442
2173699387,17cyclictest23279-21taskset07:14:442
2173699387,12cyclictest20292-21chrt11:31:422
2173699387,11cyclictest0-21swapper/211:45:142
21736993813,11cyclictest16824-21/usr/sbin/munin09:55:342
21736993810,14cyclictest1125-21runrttasks12:05:422
21735993820,5cyclictest4065-21fschecks_count07:50:301
21735993813,12cyclictest30429-21awk09:00:401
2173799373,15cyclictest22503-1kworker/3:1H11:20:253
21737993714,5cyclictest20868-21aten_rbpower_en11:35:273
2173799371,22cyclictest31590-21latency09:05:303
2173699378,18cyclictest7950irq/56-dwc_otg_10:41:072
21736993710,13cyclictest1125-21runrttasks10:24:422
2173599373,24cyclictest0-21swapper/111:05:251
2173599372,4cyclictest7850irq/56-dwc_otg_07:25:261
2173799362,6cyclictest11564-21apt-get11:10:243
2173799361,24cyclictest32574-21ls07:40:133
21737993611,5cyclictest31736-21aten_rbpower_en12:05:273
2173699366,7cyclictest14340-21grep09:45:402
2173699362,6cyclictest7750irq/56-dwc_otg10:15:222
21736993614,12cyclictest12366-21chrt08:12:282
2173599362,25cyclictest0-21swapper/107:55:381
21735993618,9cyclictest20489-21cut10:05:271
2173599361,5cyclictest30573-21modprobe07:35:121
2173599361,3cyclictest0-21swapper/109:40:241
2173799355,11cyclictest7950irq/56-dwc_otg_08:10:413
2173799351,9cyclictest27531-21grep08:55:273
21737993512,4cyclictest381ktimersoftd/307:30:303
21737993511,4cyclictest5116-21processes10:50:393
21737993510,5cyclictest9774-21cpuspeed09:35:293
2173699357,16cyclictest0-21swapper/211:06:342
2173699356,25cyclictest0-21swapper/209:05:242
2173699354,21cyclictest7950irq/56-dwc_otg_07:47:052
2173599352,5cyclictest7234-21grep10:55:411
2173599352,4cyclictest7750irq/56-dwc_otg09:40:131
136742350,5sleep32173799cyclictest09:45:313
2173799346,15cyclictest0-21swapper/309:10:193
2173799346,15cyclictest0-21swapper/309:10:193
2173799341,7cyclictest8551-21ps09:30:373
2173799341,5cyclictest30148-21uniq12:00:313
2173699345,9cyclictest7950irq/56-dwc_otg_08:07:262
21736993423,8cyclictest17209-21apt-get11:25:242
21736993410,10cyclictest0-21swapper/207:27:102
2173599347,4cyclictest0-21swapper/108:05:251
2173599346,16cyclictest7750irq/56-dwc_otg11:55:271
2173599346,16cyclictest7750irq/56-dwc_otg11:55:271
2173599346,16cyclictest0-21swapper/109:10:191
2173599346,16cyclictest0-21swapper/109:10:191
2173599346,11cyclictest7750irq/56-dwc_otg07:40:291
2173599341,6cyclictest0-21swapper/108:15:181
21735993415,5cyclictest23561-21/usr/sbin/munin07:15:301
2173599341,4cyclictest0-21swapper/111:10:231
2173799339,5cyclictest16206-21unixbench_singl09:50:443
2173799335,11cyclictest7950irq/56-dwc_otg_07:10:393
2173799335,11cyclictest7950irq/56-dwc_otg_07:10:393
2173799334,5cyclictest7750irq/56-dwc_otg12:10:273
2173799333,10cyclictest7850irq/56-dwc_otg_09:55:423
21737993315,5cyclictest6433-21/usr/sbin/munin07:55:383
21737993311,4cyclictest7950irq/56-dwc_otg_07:15:353
2173699333,15cyclictest0-21swapper/211:59:102
2173699333,15cyclictest0-21swapper/211:59:102
2173599337,5cyclictest0-21swapper/111:00:311
21735993322,8cyclictest0-21swapper/110:40:401
2173599332,15cyclictest0-21swapper/111:20:261
2173599331,5cyclictest8753-21ls12:30:301
2173599331,4cyclictest2633-21wget10:45:271
2173599331,4cyclictest2633-21wget10:45:271
21735993313,5cyclictest5148-21/usr/sbin/munin10:50:391
2173599331,21cyclictest7613-21ls08:00:281
2173599331,21cyclictest7613-21ls08:00:281
44932320,5sleep32173799cyclictest07:50:353
2173799322,12cyclictest7950irq/56-dwc_otg_08:30:293
2173799322,12cyclictest7950irq/56-dwc_otg_08:30:293
21737993215,4cyclictest25695-21fschecks_time07:20:303
2173799321,16cyclictest29477-21ls10:30:133
2173599326,5cyclictest17786-21/usr/sbin/munin09:55:411
2173599326,14cyclictest0-21swapper/112:25:321
2173599326,14cyclictest0-21swapper/112:25:321
2173599322,24cyclictest0-21swapper/110:32:421
2173599321,8cyclictest13528-21grep11:15:251
2173599321,7cyclictest18127-21cron08:30:131
21735993215,8cyclictest8524-21open_inodes09:30:371
2173799316,6cyclictest7850irq/56-dwc_otg_11:50:273
2173799314,13cyclictest7950irq/56-dwc_otg_10:42:573
21737993113,4cyclictest24034-21taskset11:43:103
2173799311,24cyclictest26608-21timerandwakeup08:50:423
2173799311,20cyclictest31600-21uniq10:35:273
2173799311,14cyclictest4177-21/usr/sbin/munin09:20:353
2173599313,9cyclictest0-21swapper/112:15:311
2173599312,3cyclictest0-21swapper/110:25:421
2173599311,9cyclictest23194-21timerandwakeup10:10:381
21735993115,8cyclictest7950irq/56-dwc_otg_08:35:311
2173799307,5cyclictest7950irq/56-dwc_otg_12:20:123
2173799307,13cyclictest0-21swapper/312:28:333
2173799307,13cyclictest0-21swapper/312:28:333
2173799306,5cyclictest7750irq/56-dwc_otg11:30:273
2173799306,4cyclictest7850irq/56-dwc_otg_10:20:323
2173799302,4cyclictest17307-21aten_rbpower_vo11:25:263
21737993015,7cyclictest39-21ksoftirqd/308:25:133
2173599305,12cyclictest7850irq/56-dwc_otg_11:40:261
2173599302,8cyclictest0-21swapper/108:51:321
2173599302,6cyclictest0-21swapper/111:30:261
2173599302,20cyclictest0-21swapper/110:35:181
2173599302,11cyclictest0-21swapper/108:20:361
2173599301,9cyclictest20308-21latency_hist10:05:121
2173599301,9cyclictest20308-21latency_hist10:05:121
21735993013,3cyclictest0-21swapper/112:05:131
180862300,3sleep118087-21switchtime11:25:381
2173799299,6cyclictest7950irq/56-dwc_otg_08:05:333
2173799299,4cyclictest7850irq/56-dwc_otg_09:15:403
2173799298,5cyclictest7950irq/56-dwc_otg_09:40:253
2173799295,7cyclictest10966-21proc_pri12:35:383
2173799294,4cyclictest7950irq/56-dwc_otg_10:00:323
2173799294,4cyclictest7950irq/56-dwc_otg_10:00:323
2173799294,13cyclictest7950irq/56-dwc_otg_11:55:283
2173799294,13cyclictest7950irq/56-dwc_otg_11:55:283
21737992918,7cyclictest4915-21fschecks_count12:20:273
21737992913,5cyclictest7950irq/56-dwc_otg_10:50:133
21737992913,5cyclictest7950irq/56-dwc_otg_10:50:133
21737992911,3cyclictest30170-21sed09:00:383
2173599293,5cyclictest128-21systemd-journal10:18:341
2173599291,7cyclictest6326-21latency_hist12:25:131
21735992911,10cyclictest31862-21cpuspeed12:05:281
2173799283,5cyclictest7850irq/56-dwc_otg_11:45:273
2173799281,5cyclictest20639-21irqstats08:35:313
21737992813,4cyclictest13412-21uname08:15:333
21737992812,3cyclictest7750irq/56-dwc_otg10:05:313
2173599285,4cyclictest7750irq/56-dwc_otg09:50:331
2173599282,9cyclictest0-21swapper/107:35:151
2173599282,10cyclictest7950irq/56-dwc_otg_09:15:401
2173599281,7cyclictest6461-21kernelversion09:25:331
2173599281,3cyclictest7750irq/56-dwc_otg08:55:261
214412285,15sleep10-21swapper/107:09:571
2173799274,4cyclictest7750irq/56-dwc_otg07:45:393
2173799273,5cyclictest7750irq/56-dwc_otg09:25:233
21737992723,2cyclictest0-21swapper/311:05:133
2173799266,6cyclictest0-21swapper/310:16:003
2173799262,7cyclictest471-21munin-node08:25:243
2173799262,11cyclictest0-21swapper/310:30:353
212732264,15sleep30-21swapper/307:07:523
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional