You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 22:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Thu Nov 27, 2025 12:43:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1567321050,2sleep20-21swapper/210:10:292
123112830,2sleep20-21swapper/211:38:492
123112830,2sleep20-21swapper/211:38:492
213582820,3sleep337-21rcuc/309:31:053
2740599787,42cyclictest0-21swapper/011:15:230
2740899775,19cyclictest7950irq/56-dwc_otg_07:25:213
2740699769,21cyclictest6465-21runrttasks12:15:211
2740699769,21cyclictest6465-21runrttasks12:15:211
238482750,5sleep123-21ksoftirqd/111:09:541
238482750,5sleep123-21ksoftirqd/111:09:541
226612740,1sleep30-21swapper/308:25:273
226612740,1sleep30-21swapper/308:25:273
108882730,2sleep010889-21sh10:03:020
263932710,1sleep10-21swapper/112:39:401
2740799694,15cyclictest7950irq/56-dwc_otg_08:35:292
2740899681,4cyclictest7950irq/56-dwc_otg_11:05:203
2740899681,4cyclictest7950irq/56-dwc_otg_11:05:203
2740799681,6cyclictest7850irq/56-dwc_otg_10:45:312
27405996831,22cyclictest14307-21grep09:20:390
160822670,1sleep30-21swapper/310:10:343
2740599653,18cyclictest7950irq/56-dwc_otg_07:10:120
2740899642,5cyclictest0-21swapper/309:55:093
278022630,2sleep10-21swapper/108:40:231
2740899631,5cyclictest26266-21ls10:25:293
2740899631,5cyclictest26266-21ls10:25:293
2740799636,5cyclictest16536-21aten_rbpower_en09:25:232
2740799636,5cyclictest16536-21aten_rbpower_en09:25:232
2740799631,7cyclictest21353-21unixbench_singl11:50:372
2740599636,39cyclictest0-21swapper/007:45:220
2740799622,4cyclictest27791-21wget08:40:232
27407996213,6cyclictest7850irq/56-dwc_otg_10:20:232
27405996228,10cyclictest7950irq/56-dwc_otg_11:40:270
2740599621,38cyclictest0-21swapper/012:25:190
93122610,2sleep20-21swapper/207:50:082
27408996120,8cyclictest22675-21latency_hist12:35:093
2740799618,7cyclictest9550irq/80-mmc011:00:092
2740799611,6cyclictest0-21swapper/208:45:302
2740799611,6cyclictest0-21swapper/208:45:302
27405996141,4cyclictest7750irq/56-dwc_otg11:05:370
27405996141,4cyclictest7750irq/56-dwc_otg11:05:370
63082600,2sleep20-21swapper/209:55:402
63082600,2sleep20-21swapper/209:55:402
2740899606,5cyclictest7950irq/56-dwc_otg_10:45:293
2740799606,4cyclictest7850irq/56-dwc_otg_10:25:312
2740799606,4cyclictest7850irq/56-dwc_otg_10:25:312
2740699605,6cyclictest0-21swapper/107:55:191
2740699601,10cyclictest0-21swapper/107:35:191
2740699601,10cyclictest0-21swapper/107:35:191
2740799596,4cyclictest25662-21modprobe08:35:092
2740799591,5cyclictest7850irq/56-dwc_otg_12:10:322
27407995910,16cyclictest7850irq/56-dwc_otg_09:20:252
2740599591,24cyclictest29828-21grep10:30:310
2740799584,4cyclictest7950irq/56-dwc_otg_12:24:412
2740799581,3cyclictest0-21swapper/211:29:012
2740699582,9cyclictest0-21swapper/109:05:191
207132581,3sleep00-21swapper/008:20:260
2740799573,33cyclictest0-21swapper/207:10:122
2740799571,5cyclictest4710-21munin-run09:55:092
2740799571,5cyclictest446-21cron12:05:092
2740799571,3cyclictest0-21swapper/212:35:222
2740899561,6cyclictest32282-21ssh09:46:303
2740799568,7cyclictest7850irq/56-dwc_otg_08:25:092
2740799566,10cyclictest7950irq/56-dwc_otg_09:14:422
2740799565,9cyclictest7750irq/56-dwc_otg10:00:392
2740799561,5cyclictest16846-21chrt12:27:132
2740599567,45cyclictest9-21ksoftirqd/011:10:240
128682560,2sleep30-21swapper/310:05:353
2740899552,10cyclictest7950irq/56-dwc_otg_11:00:213
2740799552,6cyclictest0-21swapper/209:44:202
2740799552,14cyclictest7950irq/56-dwc_otg_10:53:582
27407995513,5cyclictest7850irq/56-dwc_otg_09:09:442
2740899542,13cyclictest7950irq/56-dwc_otg_08:40:233
2740799541,5cyclictest7850irq/56-dwc_otg_09:17:442
2740599545,11cyclictest7850irq/56-dwc_otg_08:50:240
2740599542,29cyclictest1791-21grep08:55:370
27405995411,4cyclictest20139-21modprobe08:20:090
2740699535,13cyclictest23-21ksoftirqd/107:10:121
2740599532,29cyclictest20155-21unixbench_singl10:15:390
2740599532,26cyclictest0-21swapper/007:35:090
2740799522,15cyclictest7950irq/56-dwc_otg_12:30:212
2740699528,8cyclictest31187-21ssh12:05:211
27405995223,16cyclictest0-21swapper/011:01:580
2740599522,29cyclictest2720-21cut09:00:240
2740799512,7cyclictest0-21swapper/211:30:382
2740799511,3cyclictest7850irq/56-dwc_otg_09:40:092
27405995130,4cyclictest7950irq/56-dwc_otg_09:25:260
27405995130,4cyclictest7950irq/56-dwc_otg_09:25:260
27405995126,13cyclictest3794-21/usr/sbin/munin10:40:280
27405995125,13cyclictest7850irq/56-dwc_otg_09:10:320
2740599512,25cyclictest0-21swapper/010:10:080
2740599511,30cyclictest5250-21sed07:35:400
2740599511,30cyclictest5250-21sed07:35:400
27405995112,21cyclictest0-21swapper/007:15:200
27405995110,23cyclictest0-21swapper/009:50:380
2740799502,10cyclictest7950irq/56-dwc_otg_11:45:182
2740699502,9cyclictest0-21swapper/109:55:211
2740699502,9cyclictest0-21swapper/109:55:211
2740599502,5cyclictest13958-21latency_hist10:55:090
2740599502,36cyclictest5241-21kworker/0:111:30:090
2740599501,34cyclictest27297-21apt-get12:00:190
2740599501,24cyclictest23231-21grep10:20:380
27405995010,21cyclictest0-21swapper/009:40:340
27406994932,8cyclictest23-21ksoftirqd/107:50:231
27405994931,3cyclictest0-21swapper/012:05:270
27405994926,6cyclictest7950irq/56-dwc_otg_11:50:090
27405994925,4cyclictest7750irq/56-dwc_otg10:15:090
27405994923,14cyclictest0-21swapper/007:25:210
27405994919,15cyclictest21184-21sh09:30:420
2740899483,11cyclictest0-21swapper/307:10:123
2740699487,25cyclictest0-21swapper/107:45:221
2740699486,24cyclictest0-21swapper/109:10:221
2740599488,24cyclictest0-21swapper/012:30:240
2740599483,33cyclictest485-21kworker/0:009:49:190
2740599482,24cyclictest0-21swapper/012:10:330
2740599481,6cyclictest8310-21ssh10:00:090
2740599481,6cyclictest8310-21ssh10:00:090
2740599481,28cyclictest7714-21tail12:15:390
2740599481,28cyclictest7714-21tail12:15:390
2740599481,27cyclictest27765-21grep08:40:220
2740599481,26cyclictest17090-21tune2fs08:10:260
2740599481,26cyclictest0-21swapper/008:45:350
2740599481,26cyclictest0-21swapper/008:45:350
2740899475,6cyclictest0-21swapper/311:12:433
2740799477,27cyclictest0-21swapper/210:35:222
2740599478,19cyclictest0-21swapper/011:50:210
27405994724,12cyclictest1120-21runrttasks09:17:590
27405994715,15cyclictest798-21sh11:22:100
2740599471,30cyclictest0-21swapper/010:35:220
2740599471,26cyclictest0-21swapper/009:35:320
2740899466,25cyclictest0-21swapper/310:35:223
2740799467,25cyclictest0-21swapper/207:25:212
2740799465,14cyclictest0-21swapper/211:03:562
2740599462,23cyclictest0-21swapper/007:20:270
2740599461,28cyclictest26405-21ps10:25:340
2740599461,28cyclictest26405-21ps10:25:340
2740599454,25cyclictest11080-21cron07:55:080
27405994528,6cyclictest7950irq/56-dwc_otg_10:45:330
27405994524,12cyclictest1120-21runrttasks12:24:450
2740599451,26cyclictest22879-21/usr/sbin/munin12:35:290
2740899447,24cyclictest0-21swapper/309:05:203
27408994412,18cyclictest0-21swapper/310:00:263
2740899441,11cyclictest7578-21ls12:15:383
2740899441,11cyclictest7578-21ls12:15:383
2740699447,25cyclictest0-21swapper/111:30:221
27405994428,3cyclictest0-21swapper/008:30:340
27405994414,4cyclictest0-21swapper/011:35:220
27405994414,4cyclictest0-21swapper/011:35:220
27405994412,17cyclictest18545-1kworker/u9:211:55:120
27405994410,4cyclictest5432-21unixbench_singl09:05:400
270782443,35sleep00-21swapper/007:09:290
2740899437,24cyclictest0-21swapper/311:55:143
2740899437,24cyclictest0-21swapper/308:20:203
2740899437,22cyclictest0-21swapper/311:50:213
2740899437,22cyclictest0-21swapper/310:55:223
2740799437,21cyclictest0-21swapper/210:15:242
27406994321,11cyclictest23-21ksoftirqd/107:15:201
27406994314,4cyclictest7950irq/56-dwc_otg_12:30:211
2740599432,21cyclictest14851-21latency_hist08:05:090
27405994319,15cyclictest0-21swapper/008:09:250
2740599431,24cyclictest22260-21grep08:25:220
2740599431,24cyclictest22260-21grep08:25:220
27405994312,24cyclictest0-21swapper/011:33:360
178802430,7sleep12740699cyclictest08:10:361
2740799421,17cyclictest7950irq/56-dwc_otg_12:15:352
2740799421,17cyclictest7950irq/56-dwc_otg_12:15:352
27406994236,4cyclictest23-21ksoftirqd/108:25:191
27406994236,4cyclictest23-21ksoftirqd/108:25:191
2740599422,19cyclictest0-21swapper/007:40:360
27405994216,14cyclictest0-21swapper/010:55:110
2740599421,13cyclictest12754-21taskset07:58:270
27408994125,5cyclictest31678-21fschecks_count11:20:283
2740799414,12cyclictest0-21swapper/211:16:362
2740699416,6cyclictest2017-21fschecks_count09:50:281
2740699411,20cyclictest0-21swapper/110:40:321
2740699407,23cyclictest0-21swapper/110:50:221
2740699406,22cyclictest0-21swapper/111:20:241
2740699405,29cyclictest7950irq/56-dwc_otg_10:25:201
2740699405,29cyclictest7950irq/56-dwc_otg_10:25:201
27406994011,5cyclictest0-21swapper/108:50:251
27408993916,5cyclictest26635-21sh09:40:023
2740699397,13cyclictest7750irq/56-dwc_otg11:45:091
2740699392,16cyclictest24292-21ssh11:10:211
2740699391,35cyclictest27529-21ls12:00:221
2740599392,22cyclictest0-21swapper/008:36:140
2740799382,26cyclictest5723-21ls07:40:092
2740799382,26cyclictest5723-21ls07:40:092
2740699389,5cyclictest0-21swapper/111:55:401
2740899372,14cyclictest2491-21taskset09:00:193
2740899371,22cyclictest14153-21timerwakeupswit08:00:353
2740699377,5cyclictest0-21swapper/111:20:081
2740699371,3cyclictest0-21swapper/109:30:221
2740699371,32cyclictest14661-21ls10:55:291
2740699365,21cyclictest32761-21ssh10:35:151
2740699363,19cyclictest32211-21unixbench_singl07:20:371
27406993610,6cyclictest0-21swapper/109:25:201
27406993610,6cyclictest0-21swapper/109:25:201
2740899357,19cyclictest0-21swapper/310:15:243
2740899355,12cyclictest7950irq/56-dwc_otg_10:55:093
2740899352,15cyclictest7950irq/56-dwc_otg_10:30:293
2740899351,9cyclictest1429-21ps08:55:343
27408993516,5cyclictest3014-21sh12:10:283
2740899351,5cyclictest31272-21wget12:05:233
2740899351,26cyclictest0-21swapper/311:15:143
2740899351,18cyclictest5911-21ls11:30:103
2740799355,21cyclictest7850irq/56-dwc_otg_11:40:242
2740699358,6cyclictest7750irq/56-dwc_otg09:40:271
2740699353,26cyclictest7950irq/56-dwc_otg_09:15:221
27406993516,9cyclictest18696-21grep11:00:401
27406993514,5cyclictest14894-21awk12:25:221
27406993512,4cyclictest7850irq/56-dwc_otg_10:15:241
27406993512,11cyclictest0-21swapper/109:35:191
2740899346,13cyclictest7950irq/56-dwc_otg_10:40:273
2740899344,6cyclictest7950irq/56-dwc_otg_07:35:403
2740899344,6cyclictest7950irq/56-dwc_otg_07:35:403
2740899343,14cyclictest7950irq/56-dwc_otg_07:35:093
2740899341,7cyclictest12287-21grep07:55:373
2740899341,4cyclictest7950irq/56-dwc_otg_12:00:183
2740899341,4cyclictest7750irq/56-dwc_otg12:25:323
2740899341,24cyclictest0-21swapper/311:45:273
2740799347,16cyclictest0-21swapper/211:15:082
2740799345,16cyclictest7950irq/56-dwc_otg_09:30:422
2740799344,20cyclictest7850irq/56-dwc_otg_11:10:092
2740799344,20cyclictest7850irq/56-dwc_otg_11:10:092
27407993419,8cyclictest9974-21awk07:50:312
27407993419,8cyclictest5402chrt11:21:342
27407993416,7cyclictest1447-21latency_hist09:50:092
27407993414,9cyclictest1322450wget08:00:212
2740799341,3cyclictest0-21swapper/208:30:092
2740799341,3cyclictest0-21swapper/208:30:092
2740699346,7cyclictest7750irq/56-dwc_otg07:25:401
2740699341,23cyclictest15792-21switchtime08:05:341
2740899333,10cyclictest16993-21fschecks_count08:10:253
2740899331,8cyclictest20134-21munin-run08:20:093
2740899331,6cyclictest19639-21sh09:30:103
2740899331,6cyclictest19639-21sh09:30:103
2740899331,5cyclictest25657-21idleruntime-cro08:35:093
2740899331,22cyclictest14025-21ls09:20:373
2740799336,15cyclictest0-21swapper/207:30:222
2740799335,19cyclictest23669-21ssh11:55:142
2740799332,6cyclictest7850irq/56-dwc_otg_10:44:342
27407993320,7cyclictest301ktimersoftd/210:32:062
2740699338,5cyclictest7750irq/56-dwc_otg10:10:401
2740699338,13cyclictest7950irq/56-dwc_otg_08:00:231
2740699335,7cyclictest7750irq/56-dwc_otg11:35:271
2740699335,7cyclictest7750irq/56-dwc_otg11:35:271
2740699332,6cyclictest5727-21munin-run11:30:091
2740699332,21cyclictest0-21swapper/111:51:241
2740899328,4cyclictest0-21swapper/311:43:283
2740899325,4cyclictest7750irq/56-dwc_otg12:20:343
2740899321,6cyclictest0-21swapper/312:39:343
2740899321,5cyclictest0-21swapper/311:34:133
2740899321,4cyclictest6512-21ssh09:10:273
27408993213,5cyclictest16515-21latency_hist08:10:093
2740799328,5cyclictest7750irq/56-dwc_otg08:10:242
2740799327,11cyclictest7950irq/56-dwc_otg_07:55:382
2740799325,11cyclictest7950irq/56-dwc_otg_08:05:362
2740799324,23cyclictest1275-21ls08:55:302
2740799324,21cyclictest7850irq/56-dwc_otg_10:05:282
27407993218,6cyclictest1120-21runrttasks12:08:472
2740799321,6cyclictest7503-21idleruntime-cro07:45:082
2740799321,21cyclictest7850irq/56-dwc_otg_09:05:082
2740699328,3cyclictest0-21swapper/110:45:221
27406993222,4cyclictest7950irq/56-dwc_otg_12:10:271
2740699321,6cyclictest32478-21ssh10:35:091
2740899316,12cyclictest0-21swapper/311:35:283
2740899316,12cyclictest0-21swapper/311:35:283
2740899313,17cyclictest22126-21aten_rbpower_cu10:20:233
2740899312,18cyclictest0-21swapper/307:18:253
27408993116,7cyclictest0-21swapper/309:58:123
27408993116,7cyclictest0-21swapper/309:58:123
2740899311,20cyclictest27580-21latency09:40:303
27408993110,5cyclictest9757-21ssh09:15:233
2740799319,12cyclictest0-21swapper/207:15:312
2740799315,4cyclictest7850irq/56-dwc_otg_07:20:252
2740699318,14cyclictest0-21swapper/112:20:221
2740699318,11cyclictest7750irq/56-dwc_otg10:05:301
2740699317,15cyclictest0-21swapper/110:20:221
2740699316,5cyclictest7950irq/56-dwc_otg_09:00:341
2740699311,17cyclictest0-21swapper/110:00:191
2740899301,5cyclictest31651-21fschecks_time08:50:283
2740899301,11cyclictest7850irq/56-dwc_otg_08:45:353
2740899301,11cyclictest7850irq/56-dwc_otg_08:45:353
2740799301,4cyclictest0-21swapper/208:15:322
2740699307,12cyclictest12989-21ssh09:20:141
27406993022,5cyclictest5870-21diskmemload09:45:211
27406993013,4cyclictest2654-21if_wlan007:30:281
263392306,11sleep10-21swapper/107:05:381
2740899296,4cyclictest7950irq/56-dwc_otg_07:20:363
2740899291,8cyclictest0-21swapper/307:45:203
27408992915,7cyclictest7950irq/56-dwc_otg_08:35:313
2740899291,16cyclictest11198-21grep07:55:093
2740799294,8cyclictest7950irq/56-dwc_otg_08:50:242
2740699297,13cyclictest0-21swapper/108:31:321
2740699293,7cyclictest7485-21latency_hist07:45:091
2740699292,15cyclictest0-21swapper/108:15:311
2740699291,6cyclictest25948-21aten_rbpower_en08:35:211
2740899281,4cyclictest5939-21apt-get07:40:223
2740699284,16cyclictest0-21swapper/111:46:381
2740699282,6cyclictest7950irq/56-dwc_otg_08:20:221
268852273,15sleep30-21swapper/307:06:573
2740699267,4cyclictest23-21ksoftirqd/108:45:411
2740699267,4cyclictest23-21ksoftirqd/108:45:411
27406992517,6cyclictest1369-21ntp_states08:55:321
271232253,14sleep20-21swapper/207:10:042
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional