You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-05 - 23:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Dec 05, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
88002850,1sleep10-21swapper/111:17:431
281422840,3sleep128144-21modprobe12:30:141
231432810,3sleep223146-21aten_rbpower_en09:20:262
231432810,3sleep223146-21aten_rbpower_en09:20:262
2383499802,24cyclictest26611-21uniq10:10:310
2383499791,27cyclictest10524-21ssh12:05:420
2383499791,27cyclictest10524-21ssh12:05:420
23834997624,4cyclictest24109-21munin-run12:25:130
23834997624,4cyclictest24109-21munin-run12:25:130
2383499751,59cyclictest11887-21sh10:35:280
2383499741,18cyclictest0-21swapper/011:40:230
2383499728,2cyclictest0-21swapper/009:15:140
2383499721,4cyclictest8352-21ssh09:45:140
23834997210,14cyclictest7424-21open_files11:15:380
12722710,2sleep31268-21ssh11:05:423
238532700,1sleep30-21swapper/310:50:563
2383499708,13cyclictest0-21swapper/009:55:280
2383499701,30cyclictest0-21swapper/011:57:140
23834997010,46cyclictest12594-21idleruntime-cro08:55:120
64792690,3sleep16336-21latency_hist11:15:141
23834996921,4cyclictest7750irq/56-dwc_otg09:20:280
23834996921,4cyclictest7750irq/56-dwc_otg09:20:280
2383499688,21cyclictest18470-21rm11:31:070
23834996710,13cyclictest0-21swapper/010:42:420
2383499661,46cyclictest23657-21grep10:50:420
2383499661,29cyclictest0-21swapper/010:27:310
2383499653,51cyclictest0-21swapper/009:30:350
2383499652,28cyclictest0-21swapper/011:29:100
234042640,2sleep20-21swapper/210:05:342
2383799637,4cyclictest0-21swapper/310:40:423
2383499624,12cyclictest0-21swapper/010:45:150
2383499622,46cyclictest27967-21latency_hist12:30:130
2383499622,42cyclictest6590-21latency_hist08:25:130
2383799612,6cyclictest24702-21fschecks_count12:25:323
2383499614,22cyclictest0-21swapper/007:10:250
2383499612,43cyclictest0-21swapper/009:50:130
2383499612,42cyclictest0-21swapper/012:40:130
2383499611,40cyclictest3749-21cut08:10:270
23837996044,8cyclictest15703-21sh09:55:263
2383799603,4cyclictest7750irq/56-dwc_otg12:25:143
2383799603,4cyclictest7750irq/56-dwc_otg12:25:143
2383499592,45cyclictest0-21swapper/012:10:440
2383499592,42cyclictest0-21swapper/009:35:280
23834995920,23cyclictest20427-21grep12:20:140
2383499591,48cyclictest29526-21grep12:30:440
2383799581,6cyclictest7950irq/56-dwc_otg_09:10:143
2383799581,49cyclictest0-21swapper/311:49:093
2383799581,14cyclictest0-21swapper/308:40:243
2383799581,12cyclictest0-21swapper/308:58:583
23837995720,24cyclictest28202-21ssh12:30:193
2383499572,45cyclictest0-21swapper/010:15:370
2383799562,5cyclictest0-21swapper/312:14:053
2383799561,6cyclictest26671-21/usr/sbin/munin10:10:313
2383799561,5cyclictest5622-21rm10:25:403
2383799561,17cyclictest0-21swapper/311:37:553
2383599564,6cyclictest7750irq/56-dwc_otg10:00:281
2383499561,39cyclictest0-21swapper/010:00:420
2383499561,13cyclictest0-21swapper/008:35:130
2383799555,7cyclictest0-21swapper/312:06:033
2383799555,7cyclictest0-21swapper/312:06:033
2383799555,4cyclictest7850irq/56-dwc_otg_09:30:443
2383799553,4cyclictest7850irq/56-dwc_otg_09:26:343
2383799552,7cyclictest0-21swapper/308:47:083
2383799552,7cyclictest0-21swapper/308:47:083
2383799551,6cyclictest460-21latency_hist07:55:143
2383799551,6cyclictest19601-21sh09:15:153
2383799551,4cyclictest0-21swapper/310:00:373
2383799551,11cyclictest9228-21wc09:45:353
2383699559,20cyclictest7950irq/56-dwc_otg_12:20:182
2383699559,20cyclictest7950irq/56-dwc_otg_12:20:182
23834995538,4cyclictest1-21systemd07:25:130
2383499551,43cyclictest0-21swapper/012:00:360
2383799544,10cyclictest0-21swapper/311:26:413
2383799541,6cyclictest6334-21wget12:00:253
2383799541,5cyclictest17924-21cat11:30:403
2383799541,3cyclictest0-21swapper/309:50:303
2383799541,16cyclictest0-21swapper/311:54:273
23837995410,12cyclictest1120-21runrttasks11:20:513
2383499541,39cyclictest27495-21proc_pri09:25:400
11222540,3sleep01105-21apt-get10:20:260
11222540,3sleep01105-21apt-get10:20:260
2383799536,4cyclictest7950irq/56-dwc_otg_10:07:033
2383799535,3cyclictest7950irq/56-dwc_otg_07:55:363
2383799531,6cyclictest22079-21sh10:50:073
23837995314,5cyclictest25699-21sh09:23:473
23837995314,5cyclictest25699-21sh09:23:473
2383799531,11cyclictest31714-21taskset10:16:543
2383799531,10cyclictest7541-21sensors11:15:393
23834995331,4cyclictest2691-21apt-config11:55:140
2383799529,8cyclictest6494-21munin-plugin-st08:25:133
2383799521,8cyclictest7950irq/56-dwc_otg_10:39:133
23837995214,4cyclictest32452-21latency_hist11:05:143
2383799521,3cyclictest7850irq/56-dwc_otg_11:40:313
2383799521,3cyclictest0-21swapper/312:35:233
2383799521,3cyclictest0-21swapper/310:24:403
2383799521,3cyclictest0-21swapper/310:24:403
2383799521,13cyclictest0-21swapper/312:19:593
2383699522,29cyclictest0-21swapper/209:30:262
2383499521,43cyclictest15588-21ssh09:55:140
23834995212,29cyclictest28452-21sh10:59:090
2383799516,4cyclictest7750irq/56-dwc_otg09:35:273
2383799511,3cyclictest0-21swapper/310:58:253
2383799511,12cyclictest8333-21cat09:45:133
2383799511,10cyclictest3250-21taskset08:08:193
2383799511,10cyclictest3250-21taskset08:08:193
23834995127,5cyclictest26059-21cron07:20:120
23834995127,5cyclictest26059-21cron07:20:120
2383499512,25cyclictest0-21swapper/009:10:390
2383799501,5cyclictest0-21swapper/308:13:563
2383699501,5cyclictest7750irq/56-dwc_otg09:00:202
2383499502,25cyclictest140-21systemd-journal10:32:050
2383799492,10cyclictest0-21swapper/309:12:533
2383799491,10cyclictest0-21swapper/307:11:233
2383499494,23cyclictest13-21rcuc/011:25:130
23834994922,11cyclictest7850irq/56-dwc_otg_11:45:280
2383499491,6cyclictest29913-21missed_timers11:00:370
234052493,10sleep00-21swapper/007:08:170
2383799481,9cyclictest1120-21runrttasks07:27:173
2383799481,5cyclictest9531-21users10:30:433
2383799481,12cyclictest15049-21runrttasks09:04:243
2383799481,10cyclictest0-21swapper/308:53:333
2383699488,25cyclictest0-21swapper/207:10:252
2383599483,3cyclictest0-21swapper/111:25:251
2383599482,4cyclictest30639-21ssh09:30:361
2383799471,3cyclictest0-21swapper/311:57:263
2383799471,12cyclictest0-21swapper/307:24:203
23836994732,8cyclictest4713-21sh10:25:252
2383499474,4cyclictest0-21swapper/008:55:380
2383499472,28cyclictest0-21swapper/008:40:370
2383499471,23cyclictest23212-21/usr/sbin/munin10:05:320
23834994711,3cyclictest0-21swapper/008:05:130
2383799467,25cyclictest0-21swapper/311:10:253
2383799461,12cyclictest0-21swapper/307:34:383
2383699463,5cyclictest7950irq/56-dwc_otg_10:00:342
2383599469,24cyclictest0-21swapper/107:55:251
2383599469,10cyclictest1120-21runrttasks11:55:271
23835994610,8cyclictest0-21swapper/109:25:271
2383499462,34cyclictest0-21swapper/008:28:420
23834994614,5cyclictest0-21swapper/011:10:230
2383499461,26cyclictest0-21swapper/007:25:240
2383699453,5cyclictest0-21swapper/212:15:262
2383599453,7cyclictest7750irq/56-dwc_otg10:15:251
23835994526,4cyclictest31990-21/usr/sbin/munin12:35:251
23835994512,7cyclictest7850irq/56-dwc_otg_07:25:411
2383499452,30cyclictest0-21swapper/008:18:100
2383499451,15cyclictest0-21swapper/011:05:370
2383699446,24cyclictest0-21swapper/212:35:242
2383599447,24cyclictest0-21swapper/111:20:271
2383499446,17cyclictest13-21rcuc/007:40:310
23834994424,14cyclictest0-21swapper/007:38:150
2383699434,3cyclictest7750irq/56-dwc_otg07:25:262
2383599435,4cyclictest7850irq/56-dwc_otg_09:55:261
2383499436,23cyclictest3629-21latency_hist08:10:130
2383499431,31cyclictest0-21swapper/011:35:180
2383499422,8cyclictest437-21latency_hist07:55:140
23834994218,12cyclictest0-21swapper/008:46:350
23834994218,12cyclictest0-21swapper/008:46:350
2383799417,24cyclictest0-21swapper/308:25:263
2383699417,23cyclictest0-21swapper/210:55:282
2383699417,21cyclictest0-21swapper/212:05:292
2383699417,21cyclictest0-21swapper/212:05:292
2383799401,8cyclictest0-21swapper/308:19:143
2383699408,22cyclictest0-21swapper/211:18:042
2383699405,21cyclictest2761-21ssh11:55:182
2383599402,9cyclictest1227-21apt-get09:35:221
2383499402,19cyclictest0-21swapper/007:48:320
23834994020,4cyclictest7850irq/56-dwc_otg_09:03:480
177292400,2sleep20-21swapper/211:30:372
93502390,2sleep20-21swapper/209:45:382
42252390,2sleep2301ktimersoftd/208:10:342
2383599397,22cyclictest0-21swapper/109:00:191
2383599396,17cyclictest7850irq/56-dwc_otg_10:05:431
23835993920,5cyclictest23-21ksoftirqd/110:30:251
23835993919,6cyclictest23-21ksoftirqd/108:10:401
2383499391,27cyclictest0-21swapper/008:00:140
2383499391,22cyclictest0-21swapper/007:31:330
2383799381,6cyclictest0-21swapper/307:48:463
2383699381,27cyclictest6518-21ls08:25:132
2383599385,4cyclictest0-21swapper/111:05:271
23835993819,5cyclictest23-21ksoftirqd/109:20:281
23835993819,5cyclictest23-21ksoftirqd/109:20:281
23835993816,13cyclictest23-21ksoftirqd/112:05:271
23835993816,13cyclictest23-21ksoftirqd/112:05:271
2383599381,4cyclictest16434-21perl09:10:281
2383499381,30cyclictest0-21swapper/008:37:230
2383499381,29cyclictest0-21swapper/009:07:080
160032380,2sleep10-21swapper/112:14:281
2383699378,18cyclictest0-21swapper/209:55:282
2383699372,22cyclictest31973-21ssh11:50:292
23836993710,5cyclictest0-21swapper/212:30:262
2383599372,5cyclictest0-21swapper/109:15:251
2383599372,11cyclictest15432-21aten_rbpower_vo10:40:261
23835993716,6cyclictest12156-21aten_rbpower_en09:50:271
84922360,5sleep22383699cyclictest08:30:142
2383699365,20cyclictest16070-21ssh09:10:242
23836993610,9cyclictest7850irq/56-dwc_otg_09:25:272
23835993616,5cyclictest433850http10:25:131
23835993616,5cyclictest433850http10:25:131
23835993612,5cyclictest23-21ksoftirqd/108:45:131
2383599361,22cyclictest25421-21latency_hist07:15:141
2383599361,22cyclictest21-21rcuc/110:25:251
2383499364,17cyclictest0-21swapper/008:05:240
2383799351,14cyclictest0-21swapper/307:44:573
23836993512,6cyclictest32264-21idleruntime-cro11:05:132
23836993511,5cyclictest1120-21runrttasks10:31:072
2383599356,5cyclictest7850irq/56-dwc_otg_10:10:241
2383599355,15cyclictest7850irq/56-dwc_otg_12:25:321
23835993522,4cyclictest221ktimersoftd/111:33:431
2383599352,23cyclictest24078-21latency_hist11:40:141
2383599352,16cyclictest2970350ssh11:00:341
2383799341,12cyclictest0-21swapper/307:39:483
2383699346,13cyclictest7850irq/56-dwc_otg_12:30:132
23836993421,11cyclictest7950irq/56-dwc_otg_11:22:232
2383699341,22cyclictest13464-21ls12:10:342
2383699341,11cyclictest7950irq/56-dwc_otg_10:35:422
2383599343,15cyclictest7850irq/56-dwc_otg_12:20:321
2383599343,15cyclictest7850irq/56-dwc_otg_12:20:321
2383599342,16cyclictest21-21rcuc/110:55:531
2383599341,23cyclictest25497-21timerwakeupswit11:40:411
23835993411,4cyclictest0-21swapper/111:50:411
23835993411,4cyclictest0-21swapper/110:51:581
23837993313,5cyclictest8897-21modprobe08:35:133
2383799331,13cyclictest0-21swapper/308:00:253
2383699339,14cyclictest0-21swapper/209:50:252
2383699332,4cyclictest25134-21ntp_kernel_pll_11:40:362
2383699331,24cyclictest19130-21ls10:45:302
2383599337,7cyclictest7850irq/56-dwc_otg_09:43:561
2383599333,4cyclictest7850irq/56-dwc_otg_07:50:131
2383599333,14cyclictest7850irq/56-dwc_otg_08:10:131
23835993311,7cyclictest23-21ksoftirqd/111:45:431
2383799321,15cyclictest0-21swapper/308:37:223
2383799321,11cyclictest0-21swapper/307:17:483
2383799321,11cyclictest0-21swapper/307:17:483
2383699328,12cyclictest0-21swapper/209:42:122
2383699324,13cyclictest7950irq/56-dwc_otg_11:45:392
2383699322,17cyclictest29973-21/usr/sbin/munin07:40:302
23836993213,10cyclictest15078-21sh11:26:342
2383699321,22cyclictest7361-21timerwakeupswit12:00:392
2383699321,21cyclictest23395-21ls10:50:402
2383599327,11cyclictest7850irq/56-dwc_otg_07:45:131
2383599322,20cyclictest0-21swapper/107:22:531
23835993213,4cyclictest16966-21ssh12:15:301
23835993212,5cyclictest21-21rcuc/110:50:131
2383599321,14cyclictest7850irq/56-dwc_otg_08:10:131
2383699317,4cyclictest7750irq/56-dwc_otg11:05:292
23836993114,5cyclictest15775-21if_wlan010:40:312
2383699311,3cyclictest0-21swapper/209:35:242
2383699311,20cyclictest10692-21timerandwakeup08:40:402
2383599318,14cyclictest0-21swapper/110:37:251
2383599318,11cyclictest0-21swapper/112:01:401
2383599315,15cyclictest0-21swapper/109:45:251
2383599311,7cyclictest7027-21fschecks_time08:25:301
2383599311,7cyclictest13152-21grep08:55:281
23835993117,8cyclictest12593-21latency_hist08:55:131
2383699305,6cyclictest0-21swapper/211:10:312
2383699305,4cyclictest7750irq/56-dwc_otg10:20:282
2383699305,4cyclictest7750irq/56-dwc_otg10:20:282
2383699304,5cyclictest7950irq/56-dwc_otg_07:55:292
2383699303,4cyclictest15769-21diskmemload10:18:432
2383699303,17cyclictest26564-21ssh10:10:312
23836993016,10cyclictest7950irq/56-dwc_otg_11:35:232
2383599304,13cyclictest7850irq/56-dwc_otg_08:37:081
2383599303,17cyclictest0-21swapper/108:46:181
2383599303,17cyclictest0-21swapper/108:46:181
2383599302,18cyclictest0-21swapper/108:32:221
23835993015,6cyclictest21-21rcuc/107:40:131
2383699299,6cyclictest8910-21idleruntime-cro08:35:132
2383699299,4cyclictest0-21swapper/209:19:332
2383699292,6cyclictest7950irq/56-dwc_otg_08:55:252
2383699292,18cyclictest0-21swapper/208:19:052
23836992912,4cyclictest1120-21runrttasks08:05:392
23836992912,4cyclictest1120-21runrttasks08:05:392
23836992912,10cyclictest28729-21taskset07:30:312
2383599294,13cyclictest7950irq/56-dwc_otg_08:19:431
2383599292,17cyclictest0-21swapper/109:09:031
2383699282,16cyclictest0-21swapper/207:52:532
2383699281,4cyclictest7750irq/56-dwc_otg08:53:462
23835992815,5cyclictest21-21rcuc/107:16:371
23835992815,5cyclictest21-21rcuc/107:16:371
2383699271,16cyclictest0-21swapper/209:05:412
2383699271,14cyclictest0-21swapper/208:40:142
2383599272,7cyclictest7950irq/56-dwc_otg_08:23:561
2383599271,15cyclictest0-21swapper/107:50:231
2383699267,5cyclictest7750irq/56-dwc_otg07:18:112
2383699267,5cyclictest7750irq/56-dwc_otg07:18:112
2383699264,9cyclictest7850irq/56-dwc_otg_07:24:202
2383599261,10cyclictest140-21systemd-journal08:03:291
231942263,15sleep20-21swapper/207:05:402
231842263,14sleep30-21swapper/307:05:323
231672263,13sleep10-21swapper/107:05:181
2383699256,3cyclictest0-21swapper/208:48:412
2383699256,3cyclictest0-21swapper/208:48:412
2383699251,6cyclictest22934-21kworker/u8:307:49:172
2383699251,5cyclictest0-21swapper/207:38:212
2383599259,6cyclictest21609-1kworker/1:0H07:35:131
2383699242,13cyclictest7950irq/56-dwc_otg_08:04:392
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional