You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 16:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Tue Aug 26, 2025 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
313282800,1sleep10-21swapper/107:10:231
287492800,2sleep30-21swapper/309:35:323
122322800,5sleep13084399cyclictest09:13:121
30842997625,5cyclictest0-21swapper/009:20:170
298542760,2sleep10-21swapper/108:35:241
30842997319,27cyclictest0-21swapper/011:00:200
30842997165,4cyclictest9-21ksoftirqd/011:50:190
30842996916,11cyclictest0-21swapper/008:55:180
182882670,4sleep123-21ksoftirqd/109:20:411
30842996648,5cyclictest101ktimersoftd/011:30:030
292552660,2sleep30-21swapper/307:05:243
49482640,2sleep24947-21processes08:55:302
30842996243,7cyclictest0-21swapper/011:40:170
30842996243,7cyclictest0-21swapper/011:40:170
3084299622,39cyclictest8957-21fschecks_time10:40:230
3084299614,6cyclictest0-21swapper/009:50:140
30842996017,23cyclictest9-21ksoftirqd/007:10:320
30842996010,29cyclictest0-21swapper/007:15:180
30842995919,27cyclictest13149-21grep11:30:360
3084299591,14cyclictest0-21swapper/010:55:180
30842995721,16cyclictest8050irq/56-dwc_otg_11:45:270
30842995630,6cyclictest8050irq/56-dwc_otg_10:25:250
3084299562,45cyclictest0-21swapper/012:32:320
3084299562,23cyclictest0-21swapper/011:15:100
30842995619,5cyclictest0-21swapper/010:15:180
3084299561,36cyclictest0-21swapper/010:25:040
30842995526,18cyclictest15820-21cpuspeed11:35:210
3084299551,35cyclictest12793-21idleruntime-cro07:50:030
3084299551,31cyclictest14149-21grep12:15:340
3084299551,30cyclictest10143-21ssh12:10:340
118352550,14sleep0101ktimersoftd/009:58:360
118352550,14sleep0101ktimersoftd/009:58:360
113602550,2sleep20-21swapper/212:12:272
3084299547,8cyclictest0-21swapper/007:25:170
3084299546,23cyclictest0-21swapper/012:25:150
30842995428,5cyclictest27546-21aten_rbpower_en08:30:190
3084299542,31cyclictest20313-21fschecks_count10:10:220
3084299541,45cyclictest2612-21ls09:45:230
30842995332,2cyclictest0-21swapper/009:15:160
30842995329,5cyclictest7950irq/56-dwc_otg08:05:200
30842995329,5cyclictest7950irq/56-dwc_otg08:05:200
30842995327,3cyclictest0-21swapper/009:10:250
3084299532,30cyclictest21392-21grep08:10:340
3084299531,5cyclictest12221-21cut10:45:200
3084299531,32cyclictest9-21ksoftirqd/009:45:040
3084299531,32cyclictest9-21ksoftirqd/009:45:040
3084299531,26cyclictest31831-21grep11:55:330
30842995224,3cyclictest0-21swapper/012:05:150
3084299522,31cyclictest0-21swapper/007:30:210
3084299519,31cyclictest0-21swapper/010:35:220
3084299513,29cyclictest9-21ksoftirqd/007:20:350
30842995131,6cyclictest26478-21rm11:05:170
3084299512,31cyclictest0-21swapper/008:00:040
3084299512,27cyclictest19008-21kworker/u8:012:20:340
30842995118,30cyclictest101ktimersoftd/011:22:540
3084299511,29cyclictest30942-21latency_hist09:40:040
3084599502,11cyclictest0-21swapper/307:26:313
3084299503,24cyclictest0-21swapper/008:25:150
30842995030,10cyclictest0-21swapper/008:45:040
3084299501,5cyclictest16733-21/usr/sbin/munin10:05:230
3084299501,34cyclictest1554-21aten_rbpower_en10:30:240
3084299501,28cyclictest0-21swapper/008:00:330
3084299501,12cyclictest0-21swapper/008:35:170
303672503,12sleep00-21swapper/007:07:240
3084299494,28cyclictest9-21ksoftirqd/008:15:210
3084299491,15cyclictest0-21swapper/012:04:210
3084399488,13cyclictest30388-21ssh10:25:161
30842994816,22cyclictest0-21swapper/008:21:420
3084599474,10cyclictest0-21swapper/307:35:163
3084499477,25cyclictest15812-21ssh10:50:172
30844994715,24cyclictest0-21swapper/210:05:172
30842994726,5cyclictest7950irq/56-dwc_otg10:55:030
3084299472,42cyclictest0-21swapper/012:35:170
30842994723,4cyclictest1322-21proc_pri08:45:280
30842994723,4cyclictest1322-21proc_pri08:45:280
3084299471,28cyclictest0-21swapper/011:10:310
3084499468,24cyclictest0-21swapper/211:50:102
3084499467,24cyclictest0-21swapper/208:35:092
3084399464,9cyclictest8050irq/56-dwc_otg_07:35:161
3084299462,25cyclictest0-21swapper/008:53:220
30842994622,11cyclictest912-21runrttasks09:30:100
3084299461,26cyclictest0-21swapper/010:00:290
3084299461,24cyclictest7612-21/usr/sbin/munin07:35:310
3084499456,25cyclictest0-21swapper/208:15:202
30843994523,6cyclictest12852-21grep10:00:191
30843994523,11cyclictest7950irq/56-dwc_otg12:30:191
3084299452,31cyclictest0-21swapper/009:05:490
30845994416,6cyclictest19622-21fschecks_count11:40:223
30845994416,6cyclictest19622-21fschecks_count11:40:223
3084499446,24cyclictest0-21swapper/208:25:172
3084399446,8cyclictest0-21swapper/109:35:151
3084399446,5cyclictest21815-21latency_hist08:15:031
3084299441,30cyclictest0-21swapper/009:27:570
287402430,3sleep330841-21cyclictest11:51:473
30845994214,20cyclictest24695-21ssh09:30:303
3084399425,9cyclictest0-21swapper/110:55:161
30843994215,12cyclictest0-21swapper/111:05:261
30842994216,4cyclictest136-21systemd-journal07:43:270
4152410,1sleep10-21swapper/109:41:341
4152410,1sleep10-21swapper/109:41:341
3084499417,23cyclictest0-21swapper/209:30:232
3084399419,19cyclictest8050irq/56-dwc_otg_09:55:191
3084399419,19cyclictest8050irq/56-dwc_otg_09:55:191
3084399417,24cyclictest8050irq/56-dwc_otg_10:35:221
25782410,3sleep32579-21latency07:20:283
30845994016,14cyclictest381ktimersoftd/311:25:183
30845994013,5cyclictest1192-21latency_hist10:30:043
3084499407,23cyclictest0-21swapper/208:45:172
3084499407,23cyclictest0-21swapper/208:45:172
3084499402,18cyclictest1420-21aten_rbpower_en11:15:192
3084499401,20cyclictest0-21swapper/207:30:302
30843994019,5cyclictest5482-21cat09:50:031
3084399401,4cyclictest9650irq/80-mmc009:50:451
3084299406,24cyclictest0-21swapper/007:50:170
30842994014,16cyclictest0-21swapper/009:00:070
3084599391,21cyclictest0-21swapper/310:54:263
30845993910,5cyclictest0-21swapper/310:15:363
3084499397,23cyclictest0-21swapper/207:55:182
30844993921,5cyclictest1285150wget10:00:192
3084399392,15cyclictest8050irq/56-dwc_otg_07:26:311
30843993914,6cyclictest912-21runrttasks12:25:051
3084599383,30cyclictest8150irq/56-dwc_otg_09:45:173
3084499386,2cyclictest0-21swapper/207:35:092
30844993816,9cyclictest7059-21ssh11:22:342
3084499381,12cyclictest9376-21diskmemload11:55:172
3084399383,3cyclictest8050irq/56-dwc_otg_10:15:191
3084399383,10cyclictest1809-21idleruntime-cro07:20:041
30843993821,5cyclictest953250wget07:40:191
3084399381,6cyclictest0-21swapper/111:30:181
255452380,5sleep23084499cyclictest10:17:462
3084599375,13cyclictest8150irq/56-dwc_otg_07:55:163
30845993716,5cyclictest381ktimersoftd/312:00:273
30845993715,12cyclictest912-21runrttasks10:06:113
30845993712,7cyclictest0-21swapper/309:20:243
3084499378,4cyclictest0-21swapper/209:18:312
3084499374,19cyclictest0-21swapper/211:05:042
30844993716,5cyclictest2728-21processes12:00:312
30844993711,6cyclictest19468-21sh10:55:192
30844993710,15cyclictest8690-21wget10:40:192
3084399379,12cyclictest8150irq/56-dwc_otg_10:50:191
3084399378,13cyclictest8150irq/56-dwc_otg_09:35:041
3084399375,15cyclictest8050irq/56-dwc_otg_11:50:181
3084399371,6cyclictest20697-21grep12:25:221
3084399371,22cyclictest9372-21apt-get09:10:041
30843993710,11cyclictest0-21swapper/110:20:221
3084599369,4cyclictest0-21swapper/311:37:433
3084599368,6cyclictest516-21in:imuxsock10:44:043
3084599364,6cyclictest2740-21head10:30:393
3084599362,3cyclictest8150irq/56-dwc_otg_11:05:143
30845993617,4cyclictest15507-21latency_hist10:50:043
30845993615,4cyclictest18435-21/usr/sbin/munin08:05:323
30845993615,4cyclictest18435-21/usr/sbin/munin08:05:323
3084599361,24cyclictest27689-21ls12:35:053
3084599361,23cyclictest0-21swapper/310:59:523
3084499367,15cyclictest8050irq/56-dwc_otg_12:30:412
3084499362,22cyclictest15851-21ls11:35:212
3084499361,24cyclictest2581-21timerwakeupswit10:30:382
30844993610,3cyclictest0-21swapper/211:40:262
30844993610,3cyclictest0-21swapper/211:40:262
3084399368,8cyclictest912-21runrttasks08:55:181
30843993614,13cyclictest0-21swapper/110:40:351
3084399361,21cyclictest8191-21ls11:25:041
314932350,1sleep10-21swapper/111:10:511
3084599359,5cyclictest0-21swapper/312:05:213
3084599358,6cyclictest0-21swapper/308:50:333
3084599357,24cyclictest0-21swapper/308:35:183
3084599355,4cyclictest0-21swapper/310:20:303
3084599352,25cyclictest8150irq/56-dwc_otg_12:35:163
3084599352,14cyclictest1750-21ssh11:15:243
30845993519,4cyclictest16905-21fschecks_time12:20:233
30845993514,6cyclictest912-21runrttasks09:59:483
30845993514,6cyclictest912-21runrttasks09:59:483
30845993512,5cyclictest8150irq/56-dwc_otg_11:25:043
3084499357,6cyclictest8050irq/56-dwc_otg_09:50:212
3084499353,3cyclictest8150irq/56-dwc_otg_07:10:212
3084499351,7cyclictest28305-21grep09:35:262
3084399358,18cyclictest5452-1kworker/u9:212:05:231
3084399357,14cyclictest8150irq/56-dwc_otg_11:20:031
3084399356,6cyclictest8150irq/56-dwc_otg_11:25:331
3084399354,20cyclictest2638-21ls07:20:291
3084399354,17cyclictest8150irq/56-dwc_otg_09:25:261
30843993520,5cyclictest230512chrt11:45:171
30843993519,4cyclictest15745-21sshd07:55:321
3084399351,4cyclictest0-21swapper/107:45:171
3084399351,23cyclictest0-21swapper/109:15:311
3084599348,4cyclictest0-21swapper/311:00:253
3084599347,14cyclictest0-21swapper/309:00:183
3084599346,6cyclictest12679-21rm11:30:303
3084599341,5cyclictest12357-21ssh09:13:303
30845993415,3cyclictest8050irq/56-dwc_otg_09:50:323
3084599341,3cyclictest8150irq/56-dwc_otg_08:30:163
30845993411,5cyclictest20413-21aten_rbpower_po12:25:183
3084499347,4cyclictest0-21swapper/210:45:422
3084499346,6cyclictest8050irq/56-dwc_otg_07:25:322
3084499346,3cyclictest0-21swapper/212:19:512
3084499345,19cyclictest16599-21ssh12:20:192
3084499341,6cyclictest7950irq/56-dwc_otg09:25:322
3084499341,26cyclictest0-21swapper/210:21:502
3084499341,10cyclictest7950irq/56-dwc_otg11:25:262
3084399346,13cyclictest8050irq/56-dwc_otg_10:05:201
3084399343,4cyclictest7950irq/56-dwc_otg11:00:351
3084399342,12cyclictest0-21swapper/110:30:381
30843993418,4cyclictest12881-21/usr/sbin/munin12:15:261
30843993414,4cyclictest17459-21/usr/sbin/munin08:00:311
30843993412,4cyclictest23639-21apt08:20:031
3084399341,23cyclictest13620-21missed_timers07:50:251
3084399340,3cyclictest8150irq/56-dwc_otg_08:50:171
3084599337,6cyclictest21044-21kernelversion09:25:243
3084599337,5cyclictest8050irq/56-dwc_otg_09:40:353
3084599337,5cyclictest8050irq/56-dwc_otg_09:40:353
3084599334,13cyclictest8150irq/56-dwc_otg_08:50:053
3084599334,13cyclictest8150irq/56-dwc_otg_08:50:053
30845993313,4cyclictest9439-21latency_hist09:10:043
30845993311,4cyclictest0-21swapper/310:12:503
3084599331,13cyclictest7950irq/56-dwc_otg08:40:193
3084499334,14cyclictest8050irq/56-dwc_otg_11:10:352
3084499331,7cyclictest16979-21fschecks_time08:00:232
30844993314,5cyclictest13061-21idleruntime-cro09:15:032
3084499331,14cyclictest8050irq/56-dwc_otg_08:40:232
3084499331,14cyclictest7950irq/56-dwc_otg08:30:272
30844993310,4cyclictest0-21swapper/210:13:062
3084399339,21cyclictest0-21swapper/108:30:181
3084399337,24cyclictest0-21swapper/110:45:191
3084399332,6cyclictest0-21swapper/110:10:171
3084399332,14cyclictest8150irq/56-dwc_otg_08:40:301
3084399331,5cyclictest7084-21unixbench_multi09:00:331
3084399331,21cyclictest0-21swapper/112:10:161
30843993310,6cyclictest15891-21ssh11:35:241
30843993310,5cyclictest31408-21grep12:40:041
305622333,23sleep10-21swapper/107:09:561
3084599327,4cyclictest0-21swapper/309:17:013
3084599325,10cyclictest8150irq/56-dwc_otg_07:50:313
3084599324,8cyclictest8150irq/56-dwc_otg_10:35:263
30845993213,6cyclictest126742chrt07:48:533
30845993213,4cyclictest23995-21cpu08:20:193
3084499327,5cyclictest8150irq/56-dwc_otg_09:20:342
3084499321,23cyclictest8336-21ls10:40:042
3084499321,23cyclictest0-21swapper/208:53:102
3084499321,14cyclictest8050irq/56-dwc_otg_12:05:322
3084399329,3cyclictest0-21swapper/111:55:301
3084399327,5cyclictest7950irq/56-dwc_otg08:20:321
3084399324,14cyclictest8150irq/56-dwc_otg_08:25:211
3084399324,11cyclictest8150irq/56-dwc_otg_08:05:341
3084399324,11cyclictest8150irq/56-dwc_otg_08:05:341
3084599314,3cyclictest0-21swapper/311:46:293
3084599313,11cyclictest8150irq/56-dwc_otg_12:10:373
3084599311,3cyclictest0-21swapper/312:15:153
30845993110,4cyclictest0-21swapper/311:57:153
3084499319,5cyclictest7950irq/56-dwc_otg09:59:012
3084499319,5cyclictest7950irq/56-dwc_otg09:59:012
3084499317,14cyclictest0-21swapper/207:40:172
3084499315,7cyclictest8050irq/56-dwc_otg_09:00:262
3084499315,12cyclictest8050irq/56-dwc_otg_11:30:232
30844993120,9cyclictest8150irq/56-dwc_otg_12:35:332
3084499311,21cyclictest30644-21uniq10:25:212
3084399316,6cyclictest7950irq/56-dwc_otg12:05:041
3084399312,4cyclictest0-21swapper/111:40:231
3084399312,4cyclictest0-21swapper/111:40:231
3084599305,6cyclictest27249-21munin-run08:30:033
3084599304,6cyclictest8150irq/56-dwc_otg_08:00:313
3084599304,6cyclictest1945-21latency_hist07:20:043
3084599304,13cyclictest0-21swapper/310:01:513
3084599301,4cyclictest8050irq/56-dwc_otg_11:14:553
3084599301,23cyclictest9655-21cut07:40:213
3084499306,15cyclictest0-21swapper/211:05:172
3084499304,6cyclictest23363-21grep11:45:252
3084499303,7cyclictest20593-21grep08:10:242
3084499303,13cyclictest8050irq/56-dwc_otg_07:20:242
3084499301,9cyclictest21609-21grep12:25:332
3084499301,6cyclictest0-21swapper/208:23:352
3084499301,4cyclictest7950irq/56-dwc_otg09:45:242
3084499301,10cyclictest31653-21/usr/sbin/munin09:40:242
3084499301,10cyclictest31653-21/usr/sbin/munin09:40:242
3084599293,11cyclictest8150irq/56-dwc_otg_07:10:293
3084599291,5cyclictest21347-21unixbench_multi08:10:343
3084499294,5cyclictest8050irq/56-dwc_otg_07:50:242
30844992910,4cyclictest0-21swapper/207:45:262
3084399291,5cyclictest7950irq/56-dwc_otg07:30:181
30843992911,5cyclictest221ktimersoftd/108:45:201
30843992911,5cyclictest221ktimersoftd/108:45:201
3084599283,7cyclictest0-21swapper/307:33:113
3084599281,14cyclictest0-21swapper/309:00:033
3084499281,16cyclictest0-21swapper/208:09:402
3084499281,16cyclictest0-21swapper/208:09:402
3048922813,7sleep20-21swapper/207:09:002
3084599271,6cyclictest23638-21munin-plugin-st08:20:033
3084499272,5cyclictest32683-21apt-get07:15:182
3084499261,5cyclictest0-21swapper/209:08:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional