You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 14:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Sun Nov 30, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2318999017,35cyclictest9-21ksoftirqd/022:20:230
231899765,59cyclictest0-21swapper/000:05:230
13092760,2sleep10-21swapper/120:35:301
13092760,2sleep10-21swapper/120:35:301
4572740,4sleep2452-21sensors21:35:362
231899747,27cyclictest0-21swapper/000:35:250
231899733,23cyclictest0-21swapper/021:20:230
110022730,6sleep1231999cyclictest23:19:261
231899725,14cyclictest0-21swapper/022:10:240
2318997211,18cyclictest17365-21ssh21:15:240
2318997211,18cyclictest17365-21ssh21:15:240
34982700,3sleep09-21ksoftirqd/021:40:300
270452700,1sleep10-21swapper/121:27:501
231899692,44cyclictest0-21swapper/000:25:260
231899692,42cyclictest2055-21sed20:35:390
231899692,42cyclictest2055-21sed20:35:390
2318996915,33cyclictest13-21rcuc/022:25:270
175282690,5sleep0231899cyclictest22:44:590
232099675,42cyclictest0-21swapper/221:05:302
232099673,41cyclictest28550-21/usr/sbin/munin22:15:242
232099662,5cyclictest4631-21sed22:25:392
231999664,14cyclictest17645-21ssh22:00:241
232099658,27cyclictest7850irq/56-dwc_otg_00:30:222
231899653,9cyclictest0-21swapper/022:30:240
2318996537,5cyclictest22498-21ntpq22:50:350
2318996531,13cyclictest11165-21latency_hist19:35:100
231899641,36cyclictest18817-21munin-plugin-st23:30:100
231899641,31cyclictest0-21swapper/023:55:220
2318996333,4cyclictest0-21swapper/019:45:250
231899631,37cyclictest3464-21unixbench_multi19:10:390
231899631,33cyclictest0-21swapper/000:00:240
2318996231,15cyclictest9193-21grep23:15:400
231899621,39cyclictest12928-21grep23:20:400
232099613,4cyclictest0-21swapper/221:55:292
232099612,47cyclictest14348-21uptime19:40:402
232099611,6cyclictest22909-21grep22:50:412
2318996145,5cyclictest101ktimersoftd/020:40:420
2318996130,13cyclictest7950irq/56-dwc_otg_21:30:110
2318996111,30cyclictest0-21swapper/022:45:240
232099606,3cyclictest0-21swapper/223:57:222
232099601,6cyclictest449-21in:imuxsock21:55:112
232099601,5cyclictest1790-21/usr/sbin/munin00:35:272
2319996022,30cyclictest7750irq/56-dwc_otg22:45:261
2318996038,14cyclictest0-21swapper/019:30:100
2318996031,20cyclictest0-21swapper/021:45:250
2318996018,20cyclictest0-21swapper/000:22:520
2318996018,20cyclictest0-21swapper/000:22:520
232199594,29cyclictest0-21swapper/320:45:233
232099599,35cyclictest0-21swapper/223:22:482
232099591,4cyclictest18535-21ssh22:00:422
231899597,24cyclictest0-21swapper/019:35:240
2318995927,5cyclictest7950irq/56-dwc_otg_21:10:250
231899591,33cyclictest11017-21fschecks_count22:35:290
232099587,37cyclictest7850irq/56-dwc_otg_00:20:342
232099587,37cyclictest7850irq/56-dwc_otg_00:20:342
232099586,38cyclictest7950irq/56-dwc_otg_22:45:102
232099585,40cyclictest1773-21switchtime20:35:372
232099585,40cyclictest1773-21switchtime20:35:372
232099584,4cyclictest7950irq/56-dwc_otg_20:30:112
232099581,5cyclictest24584-21ssh21:25:102
232099581,5cyclictest22995-21cut23:35:242
231899583,29cyclictest0-21swapper/021:35:400
231899582,32cyclictest8179-21grep20:55:260
231899581,46cyclictest31463-21wc20:30:110
231899581,32cyclictest1401-21wc00:35:110
232099574,4cyclictest7750irq/56-dwc_otg22:06:142
232099571,7cyclictest22920-21grep20:05:332
232099571,4cyclictest0-21swapper/223:02:382
232099571,42cyclictest0-21swapper/222:48:152
231899573,38cyclictest0-21swapper/021:00:240
2318995722,5cyclictest17107-21fschecks_count00:10:290
231899571,34cyclictest1120-21runrttasks22:00:310
231899571,28cyclictest0-21swapper/023:30:270
19382573,46sleep30-21swapper/319:08:453
232099563,42cyclictest7850irq/56-dwc_otg_23:07:462
232099561,3cyclictest0-21swapper/221:31:552
231899568,25cyclictest6571-21cut20:50:280
2318995617,23cyclictest25875-21fschecks_time22:55:300
2318995617,23cyclictest25875-21fschecks_time22:55:300
231899561,31cyclictest21042-21ssh00:15:310
231899561,30cyclictest26142-21aten_rbpower_cu20:15:250
232099555,41cyclictest7950irq/56-dwc_otg_21:44:062
232099552,8cyclictest7950irq/56-dwc_otg_23:18:382
232099552,6cyclictest7850irq/56-dwc_otg_22:59:032
232099552,6cyclictest7850irq/56-dwc_otg_22:59:032
232099552,40cyclictest7750irq/56-dwc_otg00:03:232
232099551,4cyclictest4498-21/usr/sbin/munin20:45:262
232099551,3cyclictest0-21swapper/223:14:362
2318995539,7cyclictest0-21swapper/021:33:190
2318995529,10cyclictest24874-21grep20:10:350
231899552,27cyclictest13435-21fschecks_count19:40:290
231899551,31cyclictest0-21swapper/019:20:240
231899551,31cyclictest0-21swapper/019:20:240
232099543,42cyclictest7950irq/56-dwc_otg_00:14:362
232099541,9cyclictest0-21swapper/220:10:422
232099541,45cyclictest17319-21ssh23:27:002
232099541,40cyclictest7850irq/56-dwc_otg_21:47:122
231899547,22cyclictest9-21ksoftirqd/023:10:220
2318995430,5cyclictest14101-21aten_rbpower_po21:55:260
2318995427,10cyclictest101ktimersoftd/023:35:390
231899542,48cyclictest101ktimersoftd/020:35:110
232099536,39cyclictest7950irq/56-dwc_otg_00:17:242
232099533,7cyclictest7950irq/56-dwc_otg_19:35:352
232099532,8cyclictest7850irq/56-dwc_otg_22:20:262
232099532,4cyclictest0-21swapper/221:10:462
232099531,9cyclictest29553-21ssh23:44:102
232099531,9cyclictest0-21swapper/221:00:032
232099531,5cyclictest14710-21ssh00:06:092
232099531,5cyclictest140-21systemd-journal19:46:072
232099531,3cyclictest7850irq/56-dwc_otg_21:03:102
2318995337,5cyclictest7950irq/56-dwc_otg_21:50:330
2318995322,5cyclictest0-21swapper/021:05:250
232099523,7cyclictest24958-21ssh22:10:222
2320995219,31cyclictest17353-21apt-get21:15:252
2320995219,31cyclictest17353-21apt-get21:15:252
232099521,8cyclictest0-21swapper/223:49:002
232099521,4cyclictest0-21swapper/223:31:372
232099521,43cyclictest0-21swapper/222:31:122
232099521,39cyclictest7850irq/56-dwc_otg_22:39:352
232099521,10cyclictest0-21swapper/221:27:562
2318995224,16cyclictest0-21swapper/023:52:410
231899521,30cyclictest28624-21ssh22:15:280
232099518,5cyclictest5935-21cat19:20:112
232099511,3cyclictest0-21swapper/223:53:432
2318995130,9cyclictest3936-21latency_hist23:10:100
2318995124,13cyclictest7950irq/56-dwc_otg_22:05:310
231899512,30cyclictest0-21swapper/020:00:390
2318995112,25cyclictest7950irq/56-dwc_otg_20:00:110
232099502,5cyclictest7950irq/56-dwc_otg_19:30:102
232099501,6cyclictest27986-21wget20:20:262
232099501,39cyclictest0-21swapper/220:19:142
231999503,9cyclictest0-21swapper/123:00:241
231999502,21cyclictest31535-21runrttasks20:30:241
231899507,7cyclictest0-21swapper/019:50:230
231899501,27cyclictest29617-21sort23:00:300
232099492,3cyclictest7850irq/56-dwc_otg_00:25:502
231999496,9cyclictest7850irq/56-dwc_otg_20:15:231
231899498,21cyclictest0-21swapper/020:45:320
231899492,34cyclictest5716-21modprobe19:20:100
232099481,8cyclictest21615-21grep20:00:412
232099481,6cyclictest0-21swapper/220:40:272
231999482,8cyclictest7750irq/56-dwc_otg20:50:241
231899481,30cyclictest0-21swapper/020:05:350
231899481,27cyclictest0-21swapper/023:40:240
232199477,26cyclictest0-21swapper/319:40:263
232099472,8cyclictest6005-21aten_rbpower_cu19:20:252
232099472,8cyclictest6005-21aten_rbpower_cu19:20:252
232099472,7cyclictest2434-21aten_rbpower_po19:10:252
232099472,5cyclictest31839-21fschecks_time20:30:302
232099472,5cyclictest0-21swapper/220:50:252
232099472,4cyclictest0-21swapper/219:30:272
231999473,8cyclictest0-21swapper/122:35:241
231899477,6cyclictest0-21swapper/023:45:230
232099462,4cyclictest0-21swapper/219:50:292
232099461,9cyclictest0-21swapper/219:55:392
2319994621,5cyclictest31522-21grep23:45:431
19662463,12sleep00-21swapper/019:09:070
231899451,27cyclictest29067-21grep20:20:410
231999447,23cyclictest0-21swapper/120:45:261
231999447,23cyclictest0-21swapper/120:20:261
87072430,2sleep10-21swapper/123:59:321
232199436,23cyclictest0-21swapper/319:50:253
2319994323,6cyclictest21171-21aten_rbpower_cu21:20:261
14082436,22sleep11531-21sed19:05:401
231999427,24cyclictest0-21swapper/100:15:241
133982420,2sleep30-21swapper/323:21:343
232199407,9cyclictest7850irq/56-dwc_otg_23:30:203
232199406,21cyclictest21147-21ssh21:20:263
231999402,12cyclictest0-21swapper/121:10:251
2319994015,7cyclictest26165-21unixbench_multi22:10:391
232199392,3cyclictest0-21swapper/323:10:233
231999397,24cyclictest0-21swapper/122:20:251
2319993928,7cyclictest221ktimersoftd/121:00:251
231999392,10cyclictest7850irq/56-dwc_otg_22:55:251
231999392,10cyclictest7850irq/56-dwc_otg_22:55:251
232199385,5cyclictest7950irq/56-dwc_otg_22:05:253
232199384,4cyclictest7950irq/56-dwc_otg_00:20:233
232199384,4cyclictest7950irq/56-dwc_otg_00:20:233
2321993819,4cyclictest28236-21idleruntime-cro22:15:103
2321993816,5cyclictest12888-21grep21:05:413
232199381,5cyclictest12950-21ls00:05:103
231999384,4cyclictest0-21swapper/121:35:221
231999381,26cyclictest22378-21timerwakeupswit22:05:421
232199378,5cyclictest0-21swapper/319:35:263
232199376,12cyclictest7750irq/56-dwc_otg22:35:213
232199373,4cyclictest0-21swapper/319:25:243
232199372,4cyclictest5794-21latency_hist19:20:113
231999377,21cyclictest0-21swapper/123:10:241
231999372,4cyclictest7850irq/56-dwc_otg_00:30:231
2319993717,5cyclictest27927-21basename00:25:281
219782370,4sleep1221ktimersoftd/123:34:041
232199367,13cyclictest7950irq/56-dwc_otg_23:50:403
232199362,4cyclictest0-21swapper/321:15:383
232199362,4cyclictest0-21swapper/321:15:383
232199361,17cyclictest0-21swapper/323:45:233
2321993611,5cyclictest0-21swapper/320:35:253
2321993611,5cyclictest0-21swapper/320:35:253
231999367,16cyclictest0-21swapper/119:55:161
231999366,15cyclictest7950irq/56-dwc_otg_19:40:111
231999363,7cyclictest24163-21ssh00:20:251
231999363,7cyclictest24163-21ssh00:20:251
231999363,25cyclictest1120-21runrttasks23:50:231
231999362,7cyclictest7750irq/56-dwc_otg21:40:301
231999361,8cyclictest0-21swapper/121:50:241
2319993615,5cyclictest21808-21/usr/sbin/munin22:50:261
231999361,26cyclictest7179-21cut21:45:301
131292360,16sleep17950irq/56-dwc_otg_23:20:591
232199358,6cyclictest7750irq/56-dwc_otg00:00:103
232199353,25cyclictest7950irq/56-dwc_otg_21:55:223
232199353,14cyclictest0-21swapper/323:10:113
231999355,4cyclictest7950irq/56-dwc_otg_20:55:401
231999354,8cyclictest0-21swapper/121:05:261
231999352,5cyclictest7750irq/56-dwc_otg22:40:251
2319993518,4cyclictest15323-21fschecks_time19:45:301
2319993510,14cyclictest0-21swapper/100:05:331
232199347,5cyclictest7950irq/56-dwc_otg_19:15:103
232199347,15cyclictest0-21swapper/320:15:173
232199344,5cyclictest0-21swapper/323:35:223
232199343,7cyclictest7750irq/56-dwc_otg20:00:283
231999349,9cyclictest0-21swapper/100:35:231
2319993428,4cyclictest23285-21sh23:35:281
2319993420,7cyclictest7850irq/56-dwc_otg_23:25:321
2319993416,5cyclictest1120-21runrttasks23:44:021
2319993415,5cyclictest29823-21aten_rbpower_vo20:25:261
232199339,5cyclictest39-21ksoftirqd/321:35:253
232199336,12cyclictest0-21swapper/321:50:333
232199334,9cyclictest7950irq/56-dwc_otg_00:35:323
232199334,19cyclictest25088-21ls20:10:393
231999339,8cyclictest17719-21awk00:10:371
231999332,5cyclictest28604-21df_inode21:30:261
231999332,12cyclictest0-21swapper/121:15:241
231999332,12cyclictest0-21swapper/121:15:241
2319993318,9cyclictest7950irq/56-dwc_otg_23:06:171
232199327,14cyclictest0-21swapper/319:30:263
232199327,13cyclictest0-21swapper/319:45:263
232199325,13cyclictest7950irq/56-dwc_otg_21:05:103
232199324,17cyclictest7750irq/56-dwc_otg23:15:233
232199324,14cyclictest7850irq/56-dwc_otg_00:05:323
2321993223,5cyclictest27015-21fschecks_time23:40:313
232199322,23cyclictest7950irq/56-dwc_otg_20:55:233
2321993221,8cyclictest381ktimersoftd/322:20:233
2321993212,11cyclictest37-21rcuc/322:30:213
232199321,19cyclictest381ktimersoftd/300:30:293
231999327,15cyclictest0-21swapper/119:10:241
231999324,14cyclictest7950irq/56-dwc_otg_19:25:291
231999323,9cyclictest0-21swapper/100:00:311
2319993217,8cyclictest17017-21forks19:50:291
2319993212,8cyclictest0-21swapper/122:15:361
231999321,22cyclictest24760-21missed_timers20:10:341
232199319,12cyclictest0-21swapper/323:28:053
232199317,11cyclictest0-21swapper/300:26:103
232199316,22cyclictest0-21swapper/322:15:273
232199314,4cyclictest7750irq/56-dwc_otg20:40:413
2321993124,4cyclictest6848-21wget21:45:253
2321993122,7cyclictest0-21swapper/319:55:163
232199312,19cyclictest0-21swapper/322:57:503
232199312,19cyclictest0-21swapper/322:57:503
232199311,4cyclictest0-21swapper/322:25:253
2321993114,5cyclictest1495250ssh21:10:443
232199311,19cyclictest6885-21ls20:50:323
2321993111,4cyclictest0-21swapper/321:33:253
231999313,4cyclictest0-21swapper/121:55:241
2319993121,7cyclictest11135-21idleruntime-cro19:35:101
2319993114,5cyclictest5578-21sh22:27:311
2319993114,12cyclictest1120-21runrttasks19:40:241
2319993112,4cyclictest5214-21grep19:15:411
231999311,15cyclictest0-21swapper/120:00:311
2319993110,6cyclictest0-21swapper/122:30:401
232199305,6cyclictest29014-21tail20:20:403
232199304,15cyclictest0-21swapper/319:20:223
232199304,15cyclictest0-21swapper/319:20:223
232199302,6cyclictest39-21ksoftirqd/322:00:263
232199301,16cyclictest7750irq/56-dwc_otg22:47:173
232199301,12cyclictest7750irq/56-dwc_otg20:25:373
231999306,4cyclictest7750irq/56-dwc_otg20:05:271
232199299,5cyclictest37-21rcuc/300:10:243
232199299,4cyclictest7950irq/56-dwc_otg_21:40:233
232199299,11cyclictest0-21swapper/322:52:093
232199298,4cyclictest0-21swapper/321:25:333
232199297,12cyclictest0-21swapper/323:00:413
232199292,22cyclictest32479-21sshd20:30:393
232199291,5cyclictest15000-21ssh22:40:343
232199291,15cyclictest0-21swapper/300:15:233
2319992911,5cyclictest2704-21apt-get20:40:241
232199287,14cyclictest0-21swapper/320:05:263
231999272,19cyclictest0-21swapper/119:25:081
231999272,19cyclictest0-21swapper/119:25:081
4542253,14sleep20-21swapper/219:05:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional