You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 00:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Mar 20, 2026 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2993922380,4sleep17750irq/56-dwc_otg07:05:321
1839221850,5sleep13100399cyclictest12:33:111
30994991011,86cyclictest1727-21idleruntime-cro11:35:060
30994991011,83cyclictest29408-21latency_hist08:35:070
3099499971,81cyclictest0-21swapper/012:20:100
3099499951,78cyclictest5035-21ssh11:00:060
3099499941,82cyclictest5477-21aten_rbpower_po11:00:190
3099499941,80cyclictest0-21swapper/010:30:190
3099499941,74cyclictest0-21swapper/009:45:060
3099499931,77cyclictest0-21swapper/008:40:220
3099499931,73cyclictest14411-21grep09:15:220
3099499931,40cyclictest29076-21latency_hist10:50:060
3099499921,77cyclictest0-21swapper/011:15:300
3099499921,77cyclictest0-21swapper/011:15:300
3099499921,70cyclictest23740-21ssh12:00:330
3099499911,72cyclictest6614-21ssh10:22:130
3099499901,74cyclictest0-21swapper/007:10:340
3099499901,74cyclictest0-21swapper/007:10:340
3099499901,70cyclictest6984-21cut12:20:070
3099499881,84cyclictest0-21swapper/011:13:490
3099499881,73cyclictest18505-21ssh10:36:010
3099499871,76cyclictest23856-21rm11:21:310
3099499871,72cyclictest0-21swapper/010:05:320
3099499871,68cyclictest17964-21ssh09:56:490
3099499871,68cyclictest17964-21ssh09:56:490
3099499861,70cyclictest29412-21ssh10:50:090
3099499851,81cyclictest0-21swapper/011:06:480
3099499851,72cyclictest9512-21diskmemload09:40:070
3099499851,70cyclictest27516-21seq08:30:060
3099499841,71cyclictest895-21rm12:12:050
3099499841,71cyclictest0-21swapper/011:35:290
3099499841,70cyclictest22650-21cat12:00:070
3099499841,66cyclictest10908-21ssh10:27:270
3099499831,70cyclictest0-21swapper/010:13:320
3099499831,68cyclictest22023-21ssh12:37:140
3099499831,65cyclictest14935-21runrttasks09:53:590
113052830,2sleep00-21swapper/009:10:480
3101099828,55cyclictest7750irq/56-dwc_otg11:35:192
3099499822,72cyclictest0-21swapper/008:06:450
3099499821,67cyclictest0-21swapper/011:49:160
3099499821,66cyclictest29662-21ssh11:29:320
3099499821,63cyclictest25985-21rm09:29:000
3099499821,63cyclictest25985-21rm09:29:000
35502810,2sleep10-21swapper/110:19:131
3099499811,66cyclictest0-21swapper/009:34:290
3099499811,65cyclictest14645-21ssh12:28:550
3099499811,63cyclictest25315-21chrt08:21:530
3099499811,62cyclictest28659-21ssh12:06:070
3099499811,61cyclictest0-21swapper/010:41:410
3099499803,67cyclictest16138-21taskset12:30:250
3099499801,70cyclictest0-21swapper/008:40:070
3099499801,66cyclictest0-21swapper/011:52:280
3099499801,64cyclictest1970-21ssh10:16:210
291352800,2sleep10-21swapper/110:10:281
3099499791,66cyclictest0-21swapper/011:43:370
3099499791,64cyclictest12573-21seq07:45:430
3099499782,64cyclictest11039-21chrt07:44:060
3099499781,68cyclictest21465-21ssh09:23:340
3099499781,64cyclictest0-21swapper/008:11:330
3099499771,73cyclictest2591-21/usr/sbin/munin07:20:220
3099499771,62cyclictest0-21swapper/009:50:070
3099499771,58cyclictest0-21swapper/010:02:270
3099499761,65cyclictest0-21swapper/007:58:370
3099499761,65cyclictest0-21swapper/007:58:370
3099499761,64cyclictest5580-21chrt08:56:190
3099499761,64cyclictest5580-21chrt08:56:190
3099499761,62cyclictest1737-21seq07:15:440
3099499751,61cyclictest0-21swapper/008:51:020
3099499751,61cyclictest0-21swapper/007:28:340
3099499751,60cyclictest2134-21runrttasks08:48:220
3099499751,55cyclictest23476-21sshd08:16:410
3099499741,61cyclictest0-21swapper/007:36:370
31010997213,40cyclictest28180-21sh11:26:502
3099499721,61cyclictest0-21swapper/007:31:270
3099499721,52cyclictest18032-21chrt08:01:120
49912710,2sleep10-21swapper/107:25:311
3099499711,61cyclictest0-21swapper/009:07:110
3099499701,51cyclictest14395-21chrt07:50:540
3101099679,14cyclictest0-21swapper/212:25:202
112512670,2sleep23101099cyclictest12:25:072
3099499661,52cyclictest0-21swapper/009:04:480
3101099652,45cyclictest7850irq/56-dwc_otg_09:21:182
3101099643,45cyclictest0-21swapper/210:05:362
3101099642,40cyclictest7850irq/56-dwc_otg_11:10:252
3101099641,6cyclictest18716-21rm10:36:242
3101099641,4cyclictest9213-21munin-run11:05:062
3101099631,39cyclictest7950irq/56-dwc_otg_10:56:192
3101099624,42cyclictest0-21swapper/210:00:352
3101099622,4cyclictest7850irq/56-dwc_otg_10:50:082
3102299614,16cyclictest27037-21ssh12:05:173
3101099615,17cyclictest7750irq/56-dwc_otg10:10:202
3101099612,5cyclictest8413-21latency_hist10:25:082
3101099611,45cyclictest25749-21taskset12:03:232
219422610,2sleep30-21swapper/309:24:243
134642610,3sleep213364-21latency_hist11:10:072
3101099601,49cyclictest15219-21ssh09:54:292
3101099596,35cyclictest7850irq/56-dwc_otg_12:18:312
3101099595,42cyclictest0-21swapper/211:53:042
3101099592,39cyclictest0-21swapper/209:25:342
3101099592,39cyclictest0-21swapper/209:25:342
3101099591,4cyclictest7950irq/56-dwc_otg_09:10:232
3101099591,41cyclictest22094-21ssh12:37:222
3100399592,5cyclictest7750irq/56-dwc_otg08:20:191
3101099583,36cyclictest7950irq/56-dwc_otg_10:30:192
3101099582,42cyclictest0-21swapper/211:41:122
3101099581,48cyclictest2461-21ssh12:14:552
3102299574,9cyclictest0-21swapper/309:00:173
3101099579,29cyclictest0-21swapper/209:40:272
3101099573,36cyclictest7950irq/56-dwc_otg_09:39:542
3101099572,42cyclictest20969-21ssh11:18:492
3101099572,42cyclictest20969-21ssh11:18:492
3101099572,42cyclictest0-21swapper/209:04:302
3101099571,4cyclictest7850irq/56-dwc_otg_08:40:332
3101099567,39cyclictest7850irq/56-dwc_otg_11:25:072
3101099563,42cyclictest0-21swapper/209:31:402
3101099561,4cyclictest18748-21apt-get11:55:192
3101099561,3cyclictest0-21swapper/211:32:202
3101099553,38cyclictest0-21swapper/210:29:312
3101099552,39cyclictest7750irq/56-dwc_otg08:30:182
3101099551,6cyclictest11084-21ssh09:49:432
3101099551,4cyclictest4166-21apt-get07:25:172
3101099546,36cyclictest7950irq/56-dwc_otg_12:31:282
3101099541,40cyclictest9512-21diskmemload10:00:052
3101099541,40cyclictest9512-21diskmemload10:00:052
3101099541,3cyclictest0-21swapper/208:00:072
3101099541,3cyclictest0-21swapper/208:00:072
3101099541,38cyclictest7950irq/56-dwc_otg_10:16:022
3102299532,10cyclictest26995-21ssh09:30:203
3101099534,31cyclictest0-21swapper/210:41:512
3101099531,39cyclictest7950irq/56-dwc_otg_08:16:522
3101099531,37cyclictest14944-21ssh09:15:292
3100399536,25cyclictest0-21swapper/111:55:201
31003995310,8cyclictest7850irq/56-dwc_otg_08:30:191
127392530,1sleep10-21swapper/110:30:121
3101099528,41cyclictest21307-21date08:10:322
3101099527,33cyclictest7850irq/56-dwc_otg_08:35:332
3101099526,27cyclictest0-21swapper/211:45:092
3101099522,3cyclictest0-21swapper/210:46:572
3101099521,4cyclictest2350-21/usr/sbin/munin07:20:192
3101099521,48cyclictest0-21swapper/208:20:172
3101099521,38cyclictest7950irq/56-dwc_otg_12:06:462
3102299516,6cyclictest7950irq/56-dwc_otg_08:25:183
3101099512,3cyclictest7850irq/56-dwc_otg_08:01:382
3101099511,47cyclictest2431-21latency_hist08:50:072
3101099511,41cyclictest0-21swapper/207:41:352
303742513,7sleep30-21swapper/307:06:163
3101099501,45cyclictest19266-21grep08:05:282
3101099501,40cyclictest0-21swapper/207:46:082
3101099501,37cyclictest7850irq/56-dwc_otg_07:50:222
3099499501,46cyclictest30985-21cyclictest07:10:070
3101099498,38cyclictest3098-21grep08:50:282
3101099494,41cyclictest8007-21fschecks_count07:35:232
3101099491,3cyclictest7950irq/56-dwc_otg_07:18:242
3100399496,5cyclictest7750irq/56-dwc_otg11:45:171
31003994917,7cyclictest23-21ksoftirqd/110:05:061
31022994814,17cyclictest7950irq/56-dwc_otg_12:10:173
3101099482,4cyclictest0-21swapper/208:25:172
3101099482,42cyclictest32172-21unixbench_multi07:10:342
3101099482,42cyclictest32172-21unixbench_multi07:10:342
3101099481,5cyclictest7087-21tail07:30:342
3101099481,5cyclictest0-21swapper/209:05:252
31003994822,5cyclictest23-21ksoftirqd/111:10:071
3100399481,3cyclictest0-21swapper/109:35:201
91532470,2sleep30-21swapper/311:43:333
3101099471,5cyclictest5405-21sed08:55:352
3101099471,5cyclictest5405-21sed08:55:352
3100399477,25cyclictest0-21swapper/108:25:131
31003994710,23cyclictest0-21swapper/111:10:201
3102299465,9cyclictest7950irq/56-dwc_otg_10:50:233
3100399467,25cyclictest0-21swapper/112:15:121
3100399467,24cyclictest0-21swapper/107:20:181
3100399464,9cyclictest0-21swapper/110:35:201
3102299447,6cyclictest7950irq/56-dwc_otg_07:30:193
3102299447,25cyclictest0-21swapper/308:15:203
3102299447,23cyclictest0-21swapper/307:40:203
3102299447,22cyclictest0-21swapper/307:25:193
3100399443,20cyclictest7850irq/56-dwc_otg_07:50:081
230482440,4sleep123043-21aten_rbpower_vo12:00:221
3102299435,24cyclictest908-21runrttasks12:00:213
3102299433,6cyclictest7950irq/56-dwc_otg_10:25:163
3102299432,5cyclictest7850irq/56-dwc_otg_07:45:213
31022994321,5cyclictest29157-21grep10:50:073
3101099431,5cyclictest31017-21apt-key07:10:072
3100399437,23cyclictest0-21swapper/108:55:201
3100399437,23cyclictest0-21swapper/108:55:201
31003994321,6cyclictest23-21ksoftirqd/111:15:241
31003994321,6cyclictest23-21ksoftirqd/111:15:241
3102299425,11cyclictest7850irq/56-dwc_otg_09:40:183
3102299424,3cyclictest7950irq/56-dwc_otg_10:00:253
3100399427,23cyclictest0-21swapper/107:10:111
3100399427,23cyclictest0-21swapper/107:10:111
3100399423,11cyclictest14140-21ssh09:15:181
31003994222,5cyclictest23617-21latency_hist12:40:071
31003994216,4cyclictest5014-21/usr/sbin/munin10:20:261
3100399417,25cyclictest0-21swapper/111:35:201
3102299407,22cyclictest0-21swapper/312:15:193
31022994020,7cyclictest7950irq/56-dwc_otg_11:50:063
3100399407,23cyclictest0-21swapper/107:15:131
3100399403,17cyclictest29333-21ssh10:50:071
31022993911,8cyclictest23100-21sort09:25:243
31022993911,8cyclictest23100-21sort09:25:243
31022993819,4cyclictest18290-1kworker/u9:011:30:273
31022993818,4cyclictest908-21runrttasks11:50:373
3102299381,24cyclictest27998-21cut08:30:243
3100399387,5cyclictest7750irq/56-dwc_otg11:00:231
3100399385,8cyclictest0-21swapper/108:40:191
3100399382,7cyclictest9613-21aten_rbpower_cu07:40:201
3100399382,26cyclictest6833-21ntp_states09:00:291
31003993820,4cyclictest23-21ksoftirqd/109:10:061
3102299376,7cyclictest7750irq/56-dwc_otg12:20:233
3102299372,4cyclictest0-21swapper/307:35:173
3102299372,21cyclictest7750irq/56-dwc_otg10:05:273
3100399379,5cyclictest7950irq/56-dwc_otg_12:05:321
3100399379,4cyclictest0-21swapper/111:30:191
3100399378,15cyclictest0-21swapper/110:00:071
3100399378,15cyclictest0-21swapper/110:00:071
3100399376,7cyclictest7950irq/56-dwc_otg_07:30:341
3100399374,25cyclictest23-21ksoftirqd/108:10:171
3100399374,22cyclictest7750irq/56-dwc_otg10:41:371
31003993719,5cyclictest23-21ksoftirqd/109:45:301
31003993715,6cyclictest23-21ksoftirqd/110:50:061
3102299364,3cyclictest7850irq/56-dwc_otg_10:55:173
3102299362,28cyclictest7950irq/56-dwc_otg_11:15:193
3102299362,28cyclictest7950irq/56-dwc_otg_11:15:193
3102299361,9cyclictest7950irq/56-dwc_otg_12:30:273
3102299361,25cyclictest577-21cut10:15:243
3102299361,22cyclictest7750irq/56-dwc_otg10:30:333
3102299361,21cyclictest25780-21dpkg08:25:073
31022993610,11cyclictest7950irq/56-dwc_otg_08:55:283
31022993610,11cyclictest7950irq/56-dwc_otg_08:55:283
3100399369,5cyclictest7850irq/56-dwc_otg_10:30:071
3100399366,20cyclictest11667-21ssh09:50:191
3100399366,12cyclictest7850irq/56-dwc_otg_09:40:321
3100399364,22cyclictest7750irq/56-dwc_otg11:29:301
31003993622,5cyclictest21-21rcuc/107:58:461
31003993622,5cyclictest21-21rcuc/107:58:461
31003993616,10cyclictest8663-21sh12:20:361
31003993614,6cyclictest2898-21cat08:50:251
3100399361,25cyclictest23-21ksoftirqd/111:40:171
3100399361,15cyclictest0-21swapper/108:05:311
3102299359,13cyclictest7950irq/56-dwc_otg_10:20:283
3102299355,7cyclictest7950irq/56-dwc_otg_07:20:333
3102299353,20cyclictest7850irq/56-dwc_otg_11:05:283
3102299352,6cyclictest7750irq/56-dwc_otg11:05:073
3102299351,4cyclictest0-21swapper/310:10:263
3100399351,9cyclictest22033-21ssh11:20:191
31003993515,6cyclictest7950irq/56-dwc_otg_10:05:381
3102299347,7cyclictest7850irq/56-dwc_otg_09:55:223
3102299347,7cyclictest7850irq/56-dwc_otg_09:55:223
3102299343,22cyclictest7950irq/56-dwc_otg_12:35:173
3102299341,23cyclictest9563-21ssh09:10:073
3100399349,13cyclictest0-21swapper/109:25:231
3100399349,13cyclictest0-21swapper/109:25:231
3100399343,19cyclictest8594-21sensors07:35:311
3100399343,16cyclictest7950irq/56-dwc_otg_09:20:341
3100399342,16cyclictest0-21swapper/109:30:171
3100399341,8cyclictest9912-21wget09:10:211
3102299338,12cyclictest0-21swapper/309:39:503
3102299335,14cyclictest7950irq/56-dwc_otg_08:40:073
3102299332,3cyclictest7950irq/56-dwc_otg_11:35:243
31022993314,5cyclictest500-21/usr/sbin/munin07:15:213
31022993311,4cyclictest37-21rcuc/311:10:323
3100399337,14cyclictest0-21swapper/112:10:201
3100399337,14cyclictest0-21swapper/108:00:201
3100399335,6cyclictest7850irq/56-dwc_otg_11:50:211
3100399332,5cyclictest15327-21munin-run12:30:071
3100399332,5cyclictest0-21swapper/110:58:141
3102299322,5cyclictest7750irq/56-dwc_otg11:25:223
3102299322,4cyclictest25926-21latency_hist11:25:073
3102299322,3cyclictest7950irq/56-dwc_otg_07:10:183
3102299322,3cyclictest7950irq/56-dwc_otg_07:10:183
3102299322,21cyclictest0-21swapper/309:51:263
3102299322,10cyclictest7750irq/56-dwc_otg10:35:403
3102299321,24cyclictest18944-21ls08:05:243
3102299321,15cyclictest185252cat08:05:073
3100399325,5cyclictest7850irq/56-dwc_otg_08:35:341
3100399321,25cyclictest13693-21latency07:50:241
3102299313,15cyclictest37-21rcuc/308:45:243
3102299311,5cyclictest7950irq/56-dwc_otg_11:55:193
31022993115,5cyclictest908-21runrttasks08:41:393
31022993111,4cyclictest7750irq/56-dwc_otg12:25:283
3100399311,15cyclictest7950irq/56-dwc_otg_08:15:291
3102299307,10cyclictest0-21swapper/309:15:133
3102299305,10cyclictest7750irq/56-dwc_otg09:10:303
3102299301,4cyclictest0-21swapper/309:45:253
3100399301,19cyclictest0-21swapper/108:45:191
3102299283,12cyclictest7950irq/56-dwc_otg_07:59:203
3102299283,12cyclictest7950irq/56-dwc_otg_07:59:203
3102299281,5cyclictest7850irq/56-dwc_otg_08:52:503
3102299281,5cyclictest22356-21unixbench_singl10:40:333
3102299274,5cyclictest37-21rcuc/307:50:303
3102299272,10cyclictest7950irq/56-dwc_otg_08:10:313
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional