You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 16:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Sun Nov 30, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
131762880,4sleep113179-21fschecks_count07:30:281
5576998134,23cyclictest0-21swapper/012:20:220
5576998125,32cyclictest0-21swapper/010:50:240
5576998123,25cyclictest0-21swapper/010:30:260
101782790,2sleep00-21swapper/008:50:290
99752780,5sleep0557699cyclictest09:45:320
99752780,5sleep0557699cyclictest09:45:320
235462780,2sleep20-21swapper/210:50:392
5576997519,34cyclictest31270-21fschecks_count10:15:290
557699751,10cyclictest0-21swapper/009:30:210
5578996928,13cyclictest27744-21sh12:26:372
557699694,29cyclictest0-21swapper/009:10:230
557699694,29cyclictest0-21swapper/009:10:230
557699692,41cyclictest0-21swapper/011:30:220
33082690,4sleep09-21ksoftirqd/011:55:260
5578996830,23cyclictest31547-21/usr/sbin/munin08:20:322
5578996830,23cyclictest31547-21/usr/sbin/munin08:20:322
557699681,47cyclictest6822-21latency_hist11:15:110
5578996733,11cyclictest18093-21rm10:42:362
5576996715,28cyclictest18457-21aten_rbpower_vo07:45:250
557999665,12cyclictest7950irq/56-dwc_otg_09:55:103
5579996626,5cyclictest30718-21ssh10:15:113
5578996631,18cyclictest400-21timerwakeupswit11:50:372
5578996630,22cyclictest21511-21ssh10:48:122
5578996630,22cyclictest21511-21ssh10:48:122
557699663,31cyclictest15456-21fschecks_count09:05:280
5576996630,4cyclictest0-21swapper/012:25:240
5578996538,11cyclictest12940-21apt-get07:30:242
5578996529,13cyclictest7682-21ssh11:15:362
5578996528,22cyclictest907-21rm09:33:072
5578996526,20cyclictest17073-21sh12:13:022
557699654,49cyclictest0-21swapper/010:55:240
5576996540,14cyclictest0-21swapper/010:35:410
5576996527,5cyclictest0-21swapper/011:25:220
5576996524,10cyclictest0-21swapper/007:15:230
5578996431,19cyclictest16647-21/usr/sbin/munin09:55:282
5578996426,26cyclictest2318-21df10:20:292
5578996426,26cyclictest2318-21df10:20:292
5578996330,11cyclictest657-21diskstats08:25:282
5578996328,12cyclictest13653-21fschecks_count09:00:282
5578996325,10cyclictest7850irq/56-dwc_otg_07:10:122
5578996325,10cyclictest7850irq/56-dwc_otg_07:10:122
5578996229,22cyclictest30105-21ssh11:47:262
5578996229,13cyclictest15946-21ps09:05:362
5578996221,14cyclictest8978-21ls07:20:102
318722620,2sleep30-21swapper/308:20:373
318722620,2sleep30-21swapper/308:20:373
5578996132,18cyclictest11265-21ssh09:47:042
5578996132,18cyclictest11265-21ssh09:47:042
5578996129,13cyclictest14259-21rm12:10:102
5578996128,17cyclictest26765-21sh10:09:222
557699611,36cyclictest24184-21fschecks_count10:05:310
5578996030,18cyclictest11728-21ntp_states07:25:352
5578996026,14cyclictest10896-21taskset10:31:382
5578996026,12cyclictest18989-21/usr/sbin/munin12:15:312
5578996025,12cyclictest3339-21sed09:35:392
5578996024,21cyclictest8711-21ssh12:01:312
5578996024,13cyclictest24442-21tr11:40:272
5578996024,13cyclictest16202-21cron09:55:102
5578996023,14cyclictest6904-21grep10:25:402
5578996021,26cyclictest22825-21taskset10:03:292
5578996021,26cyclictest22825-21taskset10:03:292
557699606,40cyclictest0-21swapper/010:25:250
5578995939,8cyclictest15432-21ntp_states07:35:362
5578995939,8cyclictest15432-21ntp_states07:35:362
5578995931,7cyclictest5025-21users11:10:472
5578995931,11cyclictest1686-21ssh10:20:042
5578995930,13cyclictest29091-21ssh10:59:462
5578995928,13cyclictest5922-21ssh11:58:412
5578995928,11cyclictest16253-21ssh11:28:042
5578995925,20cyclictest12333-21ssh11:22:102
5578995924,14cyclictest32731-21aten_rbpower_en11:05:272
557699596,30cyclictest14975-21fschecks_count07:35:300
557699596,30cyclictest14975-21fschecks_count07:35:300
50542593,12sleep20-21swapper/207:07:012
5578995825,12cyclictest18119-21sed11:30:322
5576995822,6cyclictest28465-21unixbench_multi08:10:400
557699581,45cyclictest0-21swapper/008:15:390
5578995730,12cyclictest25952-21ssh09:21:542
5578995728,18cyclictest1370-21/usr/sbin/munin12:35:282
5578995726,12cyclictest31388-21ssh11:02:172
5578995724,13cyclictest19728-21ssh09:13:452
5578995724,13cyclictest19728-21ssh09:13:452
5578995723,20cyclictest30676-21runrttasks09:30:092
557799576,7cyclictest7950irq/56-dwc_otg_08:05:231
557699576,29cyclictest0-21swapper/011:35:250
5576995731,5cyclictest12190-21unixbench_singl07:25:400
5576995730,7cyclictest7850irq/56-dwc_otg_08:35:260
5576995730,13cyclictest0-21swapper/010:00:310
5576995730,13cyclictest0-21swapper/010:00:310
5576995725,18cyclictest24749-21fschecks_time11:40:320
5578995627,12cyclictest14617-21ssh10:37:212
5578995626,19cyclictest16764-21diskmemload11:36:592
5578995625,23cyclictest11312-21runrttasks08:54:502
5578995624,13cyclictest23955-21ssh12:21:232
5578995622,14cyclictest22300-21ssh09:16:512
557799562,11cyclictest7850irq/56-dwc_otg_09:35:221
5576995625,14cyclictest0-21swapper/010:25:120
557699561,19cyclictest0-21swapper/008:30:230
5578995523,12cyclictest13151-21apt09:00:102
557799554,5cyclictest7750irq/56-dwc_otg10:20:241
557799554,5cyclictest7750irq/56-dwc_otg10:20:241
5576995539,5cyclictest17235-21timerwakeupswit10:40:460
5576995530,4cyclictest7950irq/56-dwc_otg_09:55:310
557699552,30cyclictest8283-21sed11:15:430
557699552,19cyclictest11558-21ssh11:20:420
557699551,4cyclictest18136-21ssh12:15:110
557699551,37cyclictest0-21swapper/009:00:340
557699551,31cyclictest23520-21idleruntime-cro08:00:100
557699551,27cyclictest0-21swapper/009:20:100
5578995423,17cyclictest7923-21ssh09:42:382
557699543,27cyclictest8261-21unixbench_singl12:00:410
5576995426,5cyclictest9312-21aten_rbpower_po07:20:300
5576995423,14cyclictest7750irq/56-dwc_otg11:55:100
5576995422,18cyclictest0-21swapper/012:30:260
5576995422,13cyclictest101ktimersoftd/007:30:280
5578995327,12cyclictest705-21taskset12:35:002
5578995324,16cyclictest0-21swapper/207:23:012
5578995321,18cyclictest3592-21chrt08:31:072
5578995320,13cyclictest0-21swapper/208:00:382
557699533,34cyclictest7946-21runrttasks09:42:420
5576995328,5cyclictest13-21rcuc/009:35:290
557699531,33cyclictest13205-21tune2fs09:50:280
557699531,27cyclictest25310-21unixbench_multi09:20:380
327152530,3sleep10-21swapper/112:34:041
5578995229,11cyclictest7464-21sshd08:44:232
557699527,14cyclictest0-21swapper/008:00:250
5576995221,6cyclictest2389-21fschecks_count10:20:310
557699521,23cyclictest0-21swapper/008:20:330
557699521,23cyclictest0-21swapper/008:20:330
5578995122,9cyclictest7950irq/56-dwc_otg_07:41:172
557699512,30cyclictest20763-21grep07:50:420
557699511,26cyclictest3493-21latency_hist11:10:110
5578995025,11cyclictest23421-21chrt07:59:152
5578995021,12cyclictest19809-21chrt07:49:042
557699501,29cyclictest1159-21wget12:35:260
557699501,28cyclictest8552-21ntpq08:45:330
5578994921,12cyclictest30531-21ssh10:14:562
5578994918,19cyclictest7950irq/56-dwc_otg_08:38:552
557699491,6cyclictest25645-21aten_rbpower_po08:05:260
557699491,44cyclictest27440-21/usr/sbin/munin09:25:260
557699491,29cyclictest28750-21ntp_offset11:45:340
557999488,27cyclictest0-21swapper/307:55:253
557999488,24cyclictest0-21swapper/309:35:243
5578994821,14cyclictest0-21swapper/208:15:182
557699486,23cyclictest0-21swapper/008:55:390
5576994826,3cyclictest0-21swapper/012:15:340
557999477,24cyclictest0-21swapper/311:15:263
5578994723,12cyclictest0-21swapper/207:54:042
5578994722,11cyclictest1120-21runrttasks08:09:472
557999468,25cyclictest0-21swapper/308:45:173
557999461,9cyclictest7950irq/56-dwc_otg_07:45:223
5578994624,12cyclictest28721-21chrt08:12:432
5577994627,4cyclictest3330-21aten_rbpower_po11:55:271
5576994627,11cyclictest0-21swapper/008:43:260
557699462,27cyclictest27976-21latency10:10:310
557699462,26cyclictest0-21swapper/012:07:460
5576994616,21cyclictest0-21swapper/007:44:250
557699461,11cyclictest0-21swapper/011:00:240
557999457,25cyclictest0-21swapper/307:30:163
5576994528,5cyclictest13-21rcuc/008:25:390
557699452,26cyclictest0-21swapper/010:49:390
557699452,26cyclictest0-21swapper/010:49:390
51712453,35sleep00-21swapper/007:08:330
557999447,21cyclictest0-21swapper/309:05:253
557999446,9cyclictest7950irq/56-dwc_otg_08:55:293
557999446,25cyclictest0-21swapper/307:15:263
557799447,25cyclictest0-21swapper/108:50:241
557999437,24cyclictest0-21swapper/308:25:173
557999437,23cyclictest0-21swapper/312:15:253
557999437,22cyclictest0-21swapper/310:55:253
5578994313,12cyclictest7872-21/usr/sbin/munin08:45:252
557699437,25cyclictest0-21swapper/007:10:240
557699437,25cyclictest0-21swapper/007:10:240
268832431,36sleep10-21swapper/110:55:341
5579994220,4cyclictest29145-21sh11:45:393
557799428,26cyclictest7850irq/56-dwc_otg_08:10:251
557799424,6cyclictest7950irq/56-dwc_otg_09:30:231
5577994226,5cyclictest16764-21diskmemload10:35:341
5577994214,13cyclictest2682-21perl08:30:281
5577994116,4cyclictest7850irq/56-dwc_otg_10:00:281
5577994116,4cyclictest7850irq/56-dwc_otg_10:00:281
557799411,5cyclictest9256-21ssh10:30:241
557999405,25cyclictest0-21swapper/309:55:263
557999404,7cyclictest7950irq/56-dwc_otg_10:20:263
557999404,7cyclictest7950irq/56-dwc_otg_10:20:263
5579994016,9cyclictest7750irq/56-dwc_otg08:15:403
557999393,20cyclictest7850irq/56-dwc_otg_11:25:413
5577993930,4cyclictest0-21swapper/111:05:411
557799392,9cyclictest7750irq/56-dwc_otg10:05:261
5577993919,5cyclictest221ktimersoftd/109:25:251
557799391,8cyclictest7750irq/56-dwc_otg09:45:231
557799391,8cyclictest7750irq/56-dwc_otg09:45:231
119562390,2sleep10-21swapper/111:21:181
557999381,6cyclictest25866-21ssh10:55:103
557799388,15cyclictest7950irq/56-dwc_otg_12:15:111
557799384,24cyclictest0-21swapper/107:15:231
5577993818,5cyclictest22109-21sh10:49:341
5577993818,5cyclictest22109-21sh10:49:341
177402380,5sleep3557999cyclictest09:10:323
177402380,5sleep3557999cyclictest09:10:323
557799376,4cyclictest0-21swapper/107:25:231
557799374,21cyclictest0-21swapper/112:25:241
5577993718,6cyclictest7850irq/56-dwc_otg_10:50:241
557999366,21cyclictest0-21swapper/311:40:303
557999362,23cyclictest0-21swapper/308:00:253
5577993618,4cyclictest1120-21runrttasks11:49:211
5577993614,5cyclictest6942-21sed09:40:391
557799361,25cyclictest4112-21cut11:10:351
557799361,22cyclictest15459-21uniq09:05:281
5577993611,6cyclictest7950irq/56-dwc_otg_10:45:111
557999354,11cyclictest7950irq/56-dwc_otg_10:35:233
5579993518,6cyclictest39-21ksoftirqd/312:20:313
5579993513,6cyclictest10694-21fschecks_count11:20:303
557999351,25cyclictest5173-21timerwakeupswit08:35:383
557799357,3cyclictest7950irq/56-dwc_otg_10:25:231
557799355,14cyclictest7950irq/56-dwc_otg_09:20:231
557799351,8cyclictest17469-21ssh11:30:141
5577993512,6cyclictest7850irq/56-dwc_otg_09:15:421
44942354,14sleep30-21swapper/307:05:373
557999349,4cyclictest7950irq/56-dwc_otg_11:00:483
557999347,15cyclictest0-21swapper/309:20:233
5579993425,6cyclictest6998-21grep10:25:413
557999341,25cyclictest10236-21timerwakeupswit07:20:433
557999341,24cyclictest2072-21ssh12:35:383
557999341,19cyclictest0-21swapper/311:10:253
557799349,5cyclictest3602-1kworker/u9:109:50:341
557799349,10cyclictest7950irq/56-dwc_otg_12:05:401
557799346,12cyclictest7950irq/56-dwc_otg_11:40:111
557799344,5cyclictest2190-21gpgv08:30:111
557799342,4cyclictest0-21swapper/109:10:281
557799342,4cyclictest0-21swapper/109:10:281
557799342,12cyclictest3133150snmpd12:05:061
557799341,7cyclictest25764-21munin-run12:25:101
557799341,20cyclictest32352-21latency_hist11:05:111
557799341,19cyclictest18719-21df12:15:271
557999339,5cyclictest0-21swapper/310:00:283
557999339,5cyclictest0-21swapper/310:00:283
557999333,15cyclictest7950irq/56-dwc_otg_08:40:343
557999331,4cyclictest0-21swapper/310:45:363
557999331,4cyclictest0-21swapper/310:45:363
557999331,10cyclictest0-21swapper/309:41:153
5579993310,6cyclictest27323-21unixbench_multi12:25:403
557799338,13cyclictest7950irq/56-dwc_otg_07:35:251
557799338,13cyclictest7950irq/56-dwc_otg_07:35:251
557799336,5cyclictest7950irq/56-dwc_otg_12:40:111
557799333,15cyclictest7850irq/56-dwc_otg_11:40:241
5577993316,6cyclictest32167-21sh11:50:281
5577993315,4cyclictest1120-21runrttasks11:15:371
5577993314,10cyclictest16764-21diskmemload10:10:211
557999327,15cyclictest0-21swapper/309:45:243
557999327,15cyclictest0-21swapper/309:45:243
557999327,11cyclictest7950irq/56-dwc_otg_12:32:473
557999326,4cyclictest0-21swapper/312:15:113
557999325,21cyclictest20944-21ssh11:35:253
557999322,15cyclictest7850irq/56-dwc_otg_11:30:323
557999321,5cyclictest347250ssh11:55:283
557999321,4cyclictest0-21swapper/309:30:223
557999321,22cyclictest9444-21cpuspeed10:30:283
557999321,20cyclictest19022-21ls10:45:113
557799323,14cyclictest7950irq/56-dwc_otg_10:20:111
557799322,24cyclictest0-21swapper/109:55:281
5577993212,4cyclictest7950irq/56-dwc_otg_07:45:231
557799321,20cyclictest24688-21ls08:00:371
557999315,4cyclictest7750irq/56-dwc_otg09:20:103
557999315,12cyclictest7850irq/56-dwc_otg_10:06:483
557999313,15cyclictest7850irq/56-dwc_otg_09:00:213
557999312,17cyclictest0-21swapper/312:00:363
5579993115,9cyclictest1120-21runrttasks08:52:153
557999311,3cyclictest7850irq/56-dwc_otg_09:25:233
557999311,3cyclictest7750irq/56-dwc_otg11:50:263
557999311,11cyclictest0-21swapper/312:07:263
5579993111,10cyclictest1831-21latency_hist10:20:113
557799317,5cyclictest7950irq/56-dwc_otg_07:40:411
557799316,5cyclictest7750irq/56-dwc_otg07:50:361
557799316,13cyclictest7950irq/56-dwc_otg_08:25:111
557799316,13cyclictest7950irq/56-dwc_otg_08:25:111
5577993115,5cyclictest21944-21/usr/sbin/munin07:55:361
557799311,3cyclictest0-21swapper/107:10:241
557799311,3cyclictest0-21swapper/107:10:241
557999307,20cyclictest0-21swapper/308:30:153
557999305,5cyclictest7950irq/56-dwc_otg_07:10:303
557999305,5cyclictest7950irq/56-dwc_otg_07:10:303
557999302,20cyclictest0-21swapper/307:44:123
557999302,19cyclictest0-21swapper/311:08:303
557999301,6cyclictest7850irq/56-dwc_otg_08:10:293
557799302,15cyclictest7950irq/56-dwc_otg_08:35:391
5577993014,4cyclictest14398-21if_wlan011:25:301
5577993013,5cyclictest7805-21apt-config08:45:121
41452304,16sleep14218-21irqstats07:05:311
257732300,2sleep3381ktimersoftd/308:05:283
557999292,19cyclictest0-21swapper/307:29:353
557799294,7cyclictest7950irq/56-dwc_otg_08:40:401
557799291,15cyclictest0-21swapper/108:20:111
557799287,4cyclictest7750irq/56-dwc_otg07:20:431
557799285,13cyclictest7950irq/56-dwc_otg_08:55:341
557999271,11cyclictest7750irq/56-dwc_otg07:35:393
557999271,11cyclictest7750irq/56-dwc_otg07:35:393
557799271,11cyclictest7750irq/56-dwc_otg09:00:281
557999264,5cyclictest39-21ksoftirqd/307:50:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional