You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 07:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Fri Dec 05, 2025 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2709721440,4sleep30-21swapper/319:05:313
3048521070,2sleep00-21swapper/023:05:400
3048521070,2sleep00-21swapper/023:05:400
299062930,2sleep30-21swapper/323:05:303
299062930,2sleep30-21swapper/323:05:303
64302880,2sleep10-21swapper/122:34:321
64302880,2sleep10-21swapper/122:34:321
168322830,5sleep02844099cyclictest22:05:040
311542800,5sleep32844499cyclictest23:50:363
275892790,2sleep20-21swapper/219:05:392
144772750,6sleep37950irq/56-dwc_otg_00:11:413
107602750,1sleep00-21swapper/021:55:500
264412730,8sleep3381ktimersoftd/322:16:273
2844099709,49cyclictest21851-21ntpq22:10:380
2844099706,51cyclictest12423-21cron00:10:120
2844099704,43cyclictest0-21swapper/023:35:250
2844099694,60cyclictest20260-21ssh00:20:320
2844099687,48cyclictest13028-21/usr/sbin/munin00:10:330
2844099682,53cyclictest0-21swapper/022:30:280
2844099682,53cyclictest0-21swapper/022:30:280
2844099678,32cyclictest4337-21latency_hist23:15:120
28440996714,46cyclictest0-21swapper/019:10:150
2844099671,42cyclictest22547-21fschecks_count22:55:330
28442996625,25cyclictest13968-21ssh21:19:152
2844099668,43cyclictest5721-21wc21:05:360
2844099663,46cyclictest13943-21awk20:00:130
133462660,1sleep0101ktimersoftd/021:17:580
2844099659,52cyclictest26395-21awk23:45:120
2844199648,31cyclictest7750irq/56-dwc_otg19:10:151
2844099644,52cyclictest26880-21cut20:35:280
28442996324,15cyclictest27331-21memory23:45:362
28442996315,11cyclictest7950irq/56-dwc_otg_21:05:392
2844099636,54cyclictest0-21swapper/022:35:280
2844099628,45cyclictest14170-21sh22:44:340
2844099624,49cyclictest0-21swapper/023:17:380
28440996213,46cyclictest101ktimersoftd/021:35:410
2844099620,44cyclictest0-21swapper/021:27:280
28442996132,9cyclictest3426-21apt-get21:00:292
28442996119,16cyclictest12882-21basename00:10:312
2844099618,35cyclictest16673-21chrt20:05:370
2844099615,35cyclictest0-21swapper/021:24:410
28440996133,12cyclictest7850irq/56-dwc_otg_00:30:450
28440996133,12cyclictest7850irq/56-dwc_otg_00:30:450
2844099611,53cyclictest22613-21aten_rbpower_po21:30:330
135792610,1sleep00-21swapper/023:26:330
2844299604,22cyclictest7950irq/56-dwc_otg_21:10:242
28442996021,11cyclictest25080-21ssh21:33:192
28442996019,13cyclictest21207-21sh23:37:142
2844099607,46cyclictest0-21swapper/000:25:410
2844099606,41cyclictest0-21swapper/022:50:230
2844099604,44cyclictest32664-21awk22:25:340
2844099601,7cyclictest31870-21munin-run22:25:130
2844099601,43cyclictest7520-21ntp_kernel_err19:40:340
139982600,2sleep339-21ksoftirqd/321:19:183
28442995927,17cyclictest3908-21netstat19:30:342
28442995927,12cyclictest15573-21rm22:45:382
28442995921,20cyclictest7950irq/56-dwc_otg_21:57:292
28442995919,14cyclictest22470-21/usr/sbin/munin22:55:312
2844099593,47cyclictest0-21swapper/023:45:340
2844099592,12cyclictest3342-21chrt19:30:270
2844299585,13cyclictest7950irq/56-dwc_otg_23:55:272
28442995829,12cyclictest17548-21/usr/sbin/munin00:15:432
28442995824,13cyclictest31867-21chrt23:51:032
28442995821,23cyclictest16893-21awk23:30:482
28442995821,19cyclictest7950irq/56-dwc_otg_22:00:362
28442995821,12cyclictest16664-21/usr/sbin/munin20:05:362
28442995818,13cyclictest29845-21ssh00:35:322
2844199583,10cyclictest7850irq/56-dwc_otg_21:10:241
2844099588,40cyclictest0-21swapper/021:14:070
2844099585,43cyclictest0-21swapper/000:02:130
28440995815,40cyclictest101ktimersoftd/021:46:420
28440995815,40cyclictest101ktimersoftd/021:46:420
2844099581,52cyclictest0-21swapper/021:50:200
2844099581,44cyclictest1709-21grep19:25:290
2844099581,44cyclictest1709-21grep19:25:290
2844099581,24cyclictest7950irq/56-dwc_otg_23:20:260
28440995812,43cyclictest101ktimersoftd/020:30:300
28442995734,18cyclictest0-21swapper/219:10:152
28442995726,11cyclictest26830-21apt-get20:35:252
28442995718,12cyclictest6265-21taskset19:35:522
2844099573,43cyclictest18054-21rm22:49:580
28440995713,40cyclictest101ktimersoftd/000:15:410
28440995711,23cyclictest0-21swapper/022:15:270
2844299567,17cyclictest0-21swapper/222:15:252
28442995627,12cyclictest458-21ssh22:25:382
28442995620,13cyclictest14365-21cat20:00:302
28442995620,13cyclictest14365-21cat20:00:302
28442995618,14cyclictest10610-21chrt23:23:052
2844099568,4cyclictest31609-21tail23:50:410
2844099567,41cyclictest0-21swapper/020:00:310
2844099567,41cyclictest0-21swapper/020:00:310
2844099567,39cyclictest17195-21taskset23:31:270
2844099561,41cyclictest22918-21sshd20:24:380
2844099560,43cyclictest0-21swapper/023:59:080
28442995527,12cyclictest3383-21ssh21:46:422
28442995527,12cyclictest3383-21ssh21:46:422
28442995521,12cyclictest5437-21ntp_states23:15:372
28442995518,22cyclictest7850irq/56-dwc_otg_19:25:142
2844099557,33cyclictest30022-21grep00:35:350
2844099556,34cyclictest18287-21sed22:05:420
28442995428,15cyclictest1918-21/usr/sbin/munin19:25:322
28442995428,15cyclictest1918-21/usr/sbin/munin19:25:322
28442995418,21cyclictest29200-21ssh21:38:542
2844099545,38cyclictest10249-21taskset19:49:210
28440995432,11cyclictest7950irq/56-dwc_otg_23:00:290
28440995424,6cyclictest2781-21grep20:55:430
28440995423,16cyclictest30790-21wc19:15:320
28442995324,11cyclictest28464-21ssh00:32:492
28442995324,11cyclictest28464-21ssh00:32:492
28442995322,16cyclictest10465-21ssh22:39:552
28442995321,21cyclictest0-21swapper/200:00:302
28442995321,20cyclictest6797-21diskmemload00:30:052
28442995321,19cyclictest6797-21diskmemload21:42:142
28442995320,21cyclictest16892-21ssh21:22:212
28442995317,12cyclictest14296-21ssh22:44:512
28442995314,23cyclictest18634-21aten_rbpower_vo22:50:282
28440995324,13cyclictest7850irq/56-dwc_otg_20:50:130
28442995225,15cyclictest3138-21ssh23:12:362
28442995225,11cyclictest11902-21chrt19:52:222
28442995223,13cyclictest19037-21taskset20:11:212
28442995219,23cyclictest8136-21runrttasks19:41:422
28442995219,11cyclictest7850irq/56-dwc_otg_23:26:182
28442995216,18cyclictest7950irq/56-dwc_otg_22:10:362
28440995235,10cyclictest0-21swapper/021:41:130
2844299514,11cyclictest7950irq/56-dwc_otg_22:20:272
28442995124,17cyclictest28733-21ssh23:03:522
28442995120,11cyclictest9581-21if_wlan000:05:342
28442995120,11cyclictest7850irq/56-dwc_otg_19:20:142
247242510,1sleep20-21swapper/223:41:452
28442995025,11cyclictest5927-21ssh22:33:292
28442995025,11cyclictest5927-21ssh22:33:292
28442995022,14cyclictest30430-21/usr/sbin/munin20:45:242
28442995016,13cyclictest20931-21switchtime00:20:412
2844099509,30cyclictest29877-21taskset20:41:190
2844099507,28cyclictest21106-21taskset20:19:330
28440995036,6cyclictest7950irq/56-dwc_otg_20:10:290
28442994924,11cyclictest24560-21sshd20:27:392
28442994920,12cyclictest7759-21ssh21:52:332
2844199493,12cyclictest0-21swapper/121:45:291
2844199493,12cyclictest0-21swapper/121:45:291
28440994917,21cyclictest0-21swapper/020:26:120
140062490,1sleep30-21swapper/322:44:133
28442994819,12cyclictest10047-21taskset19:46:452
28440994821,5cyclictest10633-21aten_rbpower_en19:50:270
2844099481,29cyclictest936-21grep20:50:450
2844099481,29cyclictest936-21grep20:50:450
2844099480,36cyclictest3421-21tr21:00:290
280292482,14sleep00-21swapper/019:08:250
28442994716,10cyclictest0-21swapper/221:27:522
202252470,2sleep10-21swapper/122:09:081
2844499463,10cyclictest0-21swapper/321:45:293
2844499463,10cyclictest0-21swapper/321:45:293
2844299467,26cyclictest0-21swapper/220:50:302
2844299467,26cyclictest0-21swapper/220:50:302
28442994624,13cyclictest7850irq/56-dwc_otg_21:00:122
28442994622,11cyclictest6797-21diskmemload22:06:232
28442994617,10cyclictest7750irq/56-dwc_otg23:06:592
28442994617,10cyclictest7750irq/56-dwc_otg23:06:592
28441994622,5cyclictest689050wget19:40:261
28442994520,11cyclictest22728-21taskset20:22:132
28442994517,13cyclictest0-21swapper/220:33:052
28442994513,10cyclictest0-21swapper/220:16:302
28442994427,14cyclictest13103-21/usr/sbin/munin19:55:382
28441994422,4cyclictest14861-21aten_rbpower_po21:20:281
28441994417,6cyclictest4848-21/usr/sbin/munin23:15:291
2844099440,11cyclictest6263-21runrttasks19:35:500
28444994318,6cyclictest4365-21latency_hist23:15:133
28444994317,16cyclictest19014-21sh22:06:403
2844199439,4cyclictest7850irq/56-dwc_otg_22:55:261
28441994315,6cyclictest23-21ksoftirqd/119:30:391
2844499427,20cyclictest0-21swapper/320:25:253
28444994220,5cyclictest25878-21sh00:29:453
28442994230,9cyclictest28644-21apt-get20:40:262
2844199416,6cyclictest30093-21fschecks_time00:35:361
2844199414,9cyclictest29736-21grep20:40:441
28441994124,5cyclictest5175-21aten_rbpower_vo19:35:281
28441994124,11cyclictest23-21ksoftirqd/120:30:301
2844199411,8cyclictest14448-21grep20:00:321
2844199411,8cyclictest14448-21grep20:00:321
28441994113,13cyclictest7950irq/56-dwc_otg_22:40:331
2844099417,13cyclictest1208-21cron19:25:130
2844499403,8cyclictest0-21swapper/319:10:153
2844499402,31cyclictest0-21swapper/320:40:333
28444994021,5cyclictest8482-21sh23:20:293
28444994018,5cyclictest1785850wget20:10:293
2844499401,6cyclictest7950irq/56-dwc_otg_20:50:273
2844499401,6cyclictest7950irq/56-dwc_otg_20:50:273
2844199409,5cyclictest0-21swapper/123:40:361
2844199402,32cyclictest7950irq/56-dwc_otg_20:05:251
28441994010,5cyclictest7850irq/56-dwc_otg_23:50:361
28444993916,10cyclictest6797-21diskmemload22:30:533
28444993916,10cyclictest6797-21diskmemload22:30:533
2844199393,4cyclictest7750irq/56-dwc_otg23:10:141
2844199393,4cyclictest7750irq/56-dwc_otg23:10:141
2844199391,6cyclictest26748-21/usr/sbin/munin21:35:311
28441993916,5cyclictest4570-21grep21:00:461
28444993819,10cyclictest23335-21sh22:55:433
28444993818,6cyclictest7832-21sh00:03:073
28444993816,11cyclictest1539-21wget19:25:273
28444993816,11cyclictest1539-21wget19:25:273
28444993813,18cyclictest6797-21diskmemload23:28:313
28444993812,17cyclictest0-21swapper/300:20:343
2844199383,9cyclictest7400-21fschecks_count22:35:311
2844199382,20cyclictest0-21swapper/122:15:271
28441993814,4cyclictest19885-21fschecks_time20:15:311
28441993813,5cyclictest17029-21latency_hist22:05:141
28441993810,6cyclictest7850irq/56-dwc_otg_23:10:481
5422370,2sleep10-21swapper/121:43:471
2844499371,8cyclictest7950irq/56-dwc_otg_19:40:403
28444993712,4cyclictest0-21swapper/300:30:333
28444993712,4cyclictest0-21swapper/300:30:333
2844499371,22cyclictest3423-21cpuspeed19:30:273
2844199379,7cyclictest7750irq/56-dwc_otg19:55:421
2844199378,4cyclictest0-21swapper/100:20:421
2844199375,23cyclictest0-21swapper/100:08:101
2844199372,8cyclictest2098-21ssh23:55:341
2844199372,7cyclictest22365-21unixbench_singl22:10:441
2844199371,26cyclictest8965-21ls19:45:291
2844199371,20cyclictest8488-21taskset23:20:291
305852360,5sleep12844199cyclictest22:22:241
2844199363,6cyclictest19296-21memory21:25:371
2844199361,7cyclictest5837-21tune2fs00:00:341
28441993612,3cyclictest0-21swapper/121:05:421
2844199361,18cyclictest0-21swapper/121:31:591
28441993610,5cyclictest32413-21grep19:20:291
2844499359,16cyclictest0-21swapper/321:52:253
28444993517,5cyclictest4554-21unixbench_singl21:00:463
2844499351,23cyclictest10717-21ssh21:15:143
2844199357,6cyclictest7850irq/56-dwc_otg_22:45:291
2844199357,4cyclictest7750irq/56-dwc_otg23:25:431
2844199355,3cyclictest7750irq/56-dwc_otg22:25:271
2844199353,7cyclictest23434-21diskstats20:25:301
2844199353,13cyclictest7950irq/56-dwc_otg_00:10:261
2844199352,22cyclictest0-21swapper/121:56:271
2844199352,10cyclictest1797-21tune2fs19:25:301
2844199352,10cyclictest1797-21tune2fs19:25:301
28441993518,5cyclictest27682-21sh23:01:431
2844199351,6cyclictest20050-21/usr/sbin/munin23:35:391
28441993513,6cyclictest0-21swapper/120:10:391
2844199351,22cyclictest23489-21grep00:25:371
2844499347,14cyclictest0-21swapper/320:05:253
2844199346,4cyclictest0-21swapper/123:30:391
2844199345,6cyclictest30497-21cut20:45:271
2844199342,7cyclictest16972-21grep00:15:351
2844199341,4cyclictest5544-21apt-get21:50:251
28441993413,4cyclictest0-21swapper/123:45:441
2844499332,5cyclictest7750irq/56-dwc_otg22:50:283
2844499332,13cyclictest29135-21ssh21:38:473
28444993313,10cyclictest200272chrt23:35:383
2844199338,6cyclictest7850irq/56-dwc_otg_19:20:131
2844199337,22cyclictest0-21swapper/120:35:281
2844199337,15cyclictest0-21swapper/121:17:091
2844199333,3cyclictest7850irq/56-dwc_otg_00:30:261
2844199333,3cyclictest7850irq/56-dwc_otg_00:30:261
2844199332,5cyclictest1876-21fschecks_count20:55:321
2844499329,4cyclictest0-21swapper/323:02:563
2844499326,15cyclictest30864-21ls00:35:463
2844499322,6cyclictest0-21swapper/321:20:413
28444993220,9cyclictest7950irq/56-dwc_otg_22:10:273
28444993215,10cyclictest15595-21/usr/sbin/munin23:30:393
28444993214,9cyclictest30532-21basename20:45:283
28444993214,10cyclictest6797-21diskmemload23:41:303
2844499321,24cyclictest0-21swapper/300:15:253
2844199321,13cyclictest10604-21wget19:50:271
2844499319,15cyclictest0-21swapper/320:35:363
2844499314,12cyclictest7950irq/56-dwc_otg_23:20:133
2844499312,8cyclictest0-21swapper/321:07:333
28444993122,6cyclictest23838-21sh21:30:493
28444993121,3cyclictest7950irq/56-dwc_otg_22:20:263
2844499311,8cyclictest11867-21ssh21:58:063
2844499311,6cyclictest31331-21snmpd20:20:353
2844499311,5cyclictest12943-21apt-get22:00:203
28444993112,6cyclictest900-21grep22:25:453
28444993112,5cyclictest10377-21date19:50:133
28444993110,4cyclictest7750irq/56-dwc_otg22:40:123
2844199312,5cyclictest0-21swapper/122:50:241
2844199311,6cyclictest22094-21ntp_kernel_pll_20:20:371
299382300,2sleep30-21swapper/323:49:553
2844499308,13cyclictest0-21swapper/300:08:513
2844499301,6cyclictest7850irq/56-dwc_otg_19:55:423
2844499301,5cyclictest7950irq/56-dwc_otg_19:15:293
2844199308,4cyclictest7750irq/56-dwc_otg20:50:361
2844199308,4cyclictest7750irq/56-dwc_otg20:50:361
2844499297,6cyclictest7850irq/56-dwc_otg_20:00:343
2844499297,6cyclictest7850irq/56-dwc_otg_20:00:343
2844499295,5cyclictest7950irq/56-dwc_otg_20:15:303
2844499294,13cyclictest7950irq/56-dwc_otg_21:25:393
2844499292,7cyclictest0-21swapper/319:35:413
2844499291,5cyclictest7950irq/56-dwc_otg_22:45:293
28444992912,10cyclictest6797-21diskmemload21:41:343
2844499281,5cyclictest0-21swapper/319:20:233
265272283,11sleep10-21swapper/119:05:141
2844499276,4cyclictest0-21swapper/323:56:503
2844499275,14cyclictest7950irq/56-dwc_otg_20:55:313
2844499274,16cyclictest0-21swapper/319:50:383
2844499261,15cyclictest0-21swapper/320:30:363
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional