You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 09:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Thu Nov 27, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1458499967,46cyclictest493-21runrttasks00:10:200
14585999144,25cyclictest221ktimersoftd/123:40:181
14585999144,25cyclictest221ktimersoftd/123:40:181
179122820,6sleep11458599cyclictest21:45:101
14584998249,7cyclictest0-21swapper/022:20:220
1458499804,19cyclictest8784-21ssh22:15:210
1458499802,57cyclictest0-21swapper/022:00:230
64652750,2sleep20-21swapper/220:15:342
1458499751,36cyclictest0-21swapper/020:55:180
1458499733,45cyclictest0-21swapper/019:30:200
14584997230,27cyclictest0-21swapper/019:45:210
14584997228,11cyclictest0-21swapper/021:20:200
1458599702,17cyclictest7750irq/56-dwc_otg21:10:201
14584996941,4cyclictest13610-21latency_hist23:45:090
14584996941,4cyclictest13610-21latency_hist23:45:090
14584996811,14cyclictest0-21swapper/022:40:180
1458499680,50cyclictest0-21swapper/019:10:370
1458499672,41cyclictest0-21swapper/021:05:210
275732660,1sleep00-21swapper/023:20:300
44952650,2sleep10-21swapper/123:32:311
14584996545,5cyclictest30821-21apt-get23:25:220
322812630,1sleep10-21swapper/123:25:591
14587996321,6cyclictest19068-21grep00:35:343
1458799631,12cyclictest7950irq/56-dwc_otg_23:11:503
1458799631,12cyclictest7950irq/56-dwc_otg_23:11:503
1458499634,44cyclictest7609-21aten_rbpower_en20:20:210
1458499631,47cyclictest0-21swapper/020:50:250
14587996215,7cyclictest16481-21aten_rbpower_po19:15:223
1458499626,40cyclictest27928-21chrt22:40:080
1458499623,40cyclictest0-21swapper/021:35:190
1458499621,45cyclictest0-21swapper/021:25:190
1458799617,5cyclictest1629-21cat00:10:363
1458499615,48cyclictest0-21swapper/022:25:220
14584996124,5cyclictest0-21swapper/023:15:100
14584996124,5cyclictest0-21swapper/023:15:100
1458499611,44cyclictest26256-21awk21:55:260
1458799607,5cyclictest7850irq/56-dwc_otg_21:25:263
1458799601,5cyclictest24301-21rm23:58:283
1458799601,4cyclictest28224-21/usr/sbin/munin19:45:333
14587996011,4cyclictest0-21swapper/321:35:373
1458499606,4cyclictest18991-21tail23:50:400
14584996034,7cyclictest18428-21sort21:45:270
1458499601,47cyclictest18317-21/usr/sbin/munin00:35:290
307512590,5sleep31458799cyclictest22:00:383
1458499594,21cyclictest0-21swapper/023:05:190
1458499591,48cyclictest0-21swapper/020:30:290
1458499591,43cyclictest23092-21kworker/u8:322:59:540
1458799583,13cyclictest0-21swapper/319:40:323
1458799583,13cyclictest0-21swapper/319:40:323
14587995815,8cyclictest0-21swapper/323:45:253
14584995832,9cyclictest28913-21awk00:05:090
1458499581,42cyclictest15415-21unixbench_multi20:45:360
1458799579,3cyclictest0-21swapper/322:45:393
1458799571,5cyclictest1120-21runrttasks20:50:093
1458799571,5cyclictest0-21swapper/321:31:113
1458499574,41cyclictest0-21swapper/021:30:470
14584995737,9cyclictest101ktimersoftd/021:17:270
14584995732,4cyclictest1365-21fschecks_count22:05:260
14584995728,15cyclictest0-21swapper/021:02:190
1458499572,43cyclictest0-21swapper/020:15:240
1458499572,31cyclictest0-21swapper/000:30:240
1458499571,41cyclictest0-21swapper/023:30:220
1458499571,3cyclictest0-21swapper/023:20:090
1458499571,35cyclictest17412-21munin-plugin-st23:50:080
1458799568,4cyclictest0-21swapper/322:52:523
1458799568,4cyclictest0-21swapper/322:52:523
1458799562,6cyclictest7750irq/56-dwc_otg22:35:293
1458799561,9cyclictest7850irq/56-dwc_otg_00:30:273
14584995630,13cyclictest8108-21ssh00:20:240
1458499562,44cyclictest9-21ksoftirqd/022:10:390
1458499561,42cyclictest0-21swapper/000:19:260
1458499561,41cyclictest0-21swapper/021:10:260
1458499561,37cyclictest16503-21tr19:15:220
1458799559,4cyclictest0-21swapper/321:18:023
1458799556,10cyclictest7750irq/56-dwc_otg23:22:413
14584995531,9cyclictest11298-21chrt23:00:210
1458499551,25cyclictest0-21swapper/022:30:240
186092540,2sleep20-21swapper/223:10:012
1458799541,16cyclictest0-21swapper/322:12:423
14584995441,5cyclictest7850irq/56-dwc_otg_22:52:350
14584995441,5cyclictest7850irq/56-dwc_otg_22:52:350
14584995437,6cyclictest7750irq/56-dwc_otg21:50:250
1458499543,41cyclictest9354-21ssh23:39:310
14584995433,10cyclictest14011-21runrttasks20:44:360
1458799531,3cyclictest0-21swapper/321:22:543
14584995329,14cyclictest0-21swapper/020:10:340
1458499531,42cyclictest0-21swapper/021:44:530
1458499531,41cyclictest32521-21ls20:00:080
1458799526,4cyclictest0-21swapper/321:43:343
1458799521,6cyclictest7750irq/56-dwc_otg00:30:093
1458799521,6cyclictest7750irq/56-dwc_otg00:30:093
14584995233,4cyclictest7850irq/56-dwc_otg_00:05:190
14584995233,4cyclictest7850irq/56-dwc_otg_00:05:190
14584995233,4cyclictest11669-21aten_rbpower_en00:25:260
14584995233,4cyclictest11669-21aten_rbpower_en00:25:260
14584995228,15cyclictest0-21swapper/019:40:300
14584995228,15cyclictest0-21swapper/019:40:300
1458499522,29cyclictest0-21swapper/019:25:310
1458799512,3cyclictest7850irq/56-dwc_otg_23:05:303
1458699519,27cyclictest0-21swapper/219:10:222
14584995137,5cyclictest9363-21chrt20:25:430
14584995137,5cyclictest9363-21chrt20:25:430
1458499501,40cyclictest0-21swapper/023:55:270
1458499501,39cyclictest0-21swapper/019:22:190
1458799497,10cyclictest0-21swapper/323:25:143
1458699497,28cyclictest0-21swapper/220:45:222
1458499492,37cyclictest0-21swapper/020:35:530
1458499491,40cyclictest0-21swapper/019:35:270
1458499491,38cyclictest0-21swapper/022:45:100
14586994827,12cyclictest20416-21ssh21:47:422
1458699477,25cyclictest0-21swapper/221:00:222
14584994733,4cyclictest30565-21sshd19:53:270
1458799467,25cyclictest0-21swapper/320:30:183
1458799467,24cyclictest0-21swapper/320:20:213
14585994619,5cyclictest25493-21idleruntime-cro21:55:081
14584994632,3cyclictest0-21swapper/020:04:110
1458799456,36cyclictest0-21swapper/322:05:223
14587994525,5cyclictest1261050wget20:40:193
14586994517,14cyclictest1120-21runrttasks22:23:142
141012452,36sleep00-21swapper/019:07:210
141012452,36sleep00-21swapper/019:07:210
1458799441,25cyclictest31860-21ssh22:45:093
1458699447,25cyclictest0-21swapper/221:30:212
14586994414,17cyclictest7750irq/56-dwc_otg00:16:022
1458599442,7cyclictest11-21rcu_preempt23:55:191
1458599442,24cyclictest12662-21ssh22:20:211
1458699437,24cyclictest0-21swapper/219:30:222
1458699437,23cyclictest0-21swapper/220:30:142
14586994320,8cyclictest7850irq/56-dwc_otg_21:50:202
14586994318,10cyclictest7750irq/56-dwc_otg23:51:112
1458599436,8cyclictest7950irq/56-dwc_otg_22:45:221
14584994328,8cyclictest0-21swapper/020:09:560
166022420,2sleep20-21swapper/200:31:422
1458699427,25cyclictest0-21swapper/222:40:142
14586994219,8cyclictest7950irq/56-dwc_otg_21:55:292
1458799417,5cyclictest0-21swapper/321:05:093
1458699417,24cyclictest0-21swapper/222:10:232
1458699417,19cyclictest0-21swapper/219:55:242
14586994112,16cyclictest7850irq/56-dwc_otg_21:06:342
1458599417,11cyclictest7850irq/56-dwc_otg_22:10:211
14585994121,3cyclictest7950irq/56-dwc_otg_23:15:211
1458699407,23cyclictest0-21swapper/200:05:222
1458699407,23cyclictest0-21swapper/200:05:222
14586994017,8cyclictest7850irq/56-dwc_otg_23:00:242
14585994024,6cyclictest21308-21idleruntime-cro21:05:091
14585994016,4cyclictest4641-21idleruntime-cro22:10:091
1458699399,4cyclictest7750irq/56-dwc_otg20:20:202
1458699396,22cyclictest0-21swapper/220:55:212
1458699396,19cyclictest7950irq/56-dwc_otg_20:39:152
1458699395,20cyclictest7850irq/56-dwc_otg_21:25:392
14586993916,6cyclictest7750irq/56-dwc_otg19:40:242
14586993916,6cyclictest7750irq/56-dwc_otg19:40:242
14586993915,12cyclictest1120-21runrttasks22:55:122
1458699391,4cyclictest0-21swapper/200:00:212
14586993913,13cyclictest7750irq/56-dwc_otg20:50:192
1458599394,24cyclictest18902-21dpkg23:10:091
14585993912,15cyclictest0-21swapper/119:45:211
1458699388,14cyclictest7850irq/56-dwc_otg_21:36:122
1458699388,14cyclictest7750irq/56-dwc_otg21:17:292
1458699383,19cyclictest7950irq/56-dwc_otg_00:10:282
14586993823,13cyclictest8046-21ssh00:20:222
14586993819,7cyclictest7750irq/56-dwc_otg19:45:232
1458699381,29cyclictest31230-21ls23:25:302
14586993812,4cyclictest1120-21runrttasks00:00:082
14586993810,12cyclictest7750irq/56-dwc_otg22:18:082
1458599388,13cyclictest7850irq/56-dwc_otg_19:35:271
1458599384,4cyclictest0-21swapper/121:30:211
1458599381,26cyclictest15209-21cut00:30:271
14587993718,5cyclictest20916-21sh21:48:433
1458699379,12cyclictest7850irq/56-dwc_otg_21:20:132
1458699376,16cyclictest7850irq/56-dwc_otg_22:26:062
1458699373,10cyclictest7850irq/56-dwc_otg_22:30:292
14586993717,5cyclictest7750irq/56-dwc_otg23:45:112
14586993717,13cyclictest26250-21sh21:13:472
14586993714,8cyclictest7750irq/56-dwc_otg00:27:112
14586993714,8cyclictest7750irq/56-dwc_otg00:27:112
1458599378,15cyclictest0-21swapper/122:55:091
1458599378,15cyclictest0-21swapper/122:55:091
1458599377,6cyclictest0-21swapper/100:10:201
1458599374,8cyclictest0-21swapper/120:45:221
14585993723,5cyclictest2054550wget22:30:261
14585993714,5cyclictest30000-21sh23:24:251
14585993714,10cyclictest23105-21diskmemload00:01:291
14585993710,9cyclictest0-21swapper/120:20:201
95942360,5sleep31458799cyclictest00:21:163
1458799369,15cyclictest61-21kswapd000:05:193
1458799369,15cyclictest61-21kswapd000:05:193
1458799367,25cyclictest0-21swapper/320:15:243
1458799365,4cyclictest23148-21ssh21:10:093
1458799361,20cyclictest16172-21latency_hist19:15:093
14587993610,17cyclictest0-21swapper/322:25:273
1458699369,7cyclictest7950irq/56-dwc_otg_22:54:222
1458699369,7cyclictest7950irq/56-dwc_otg_22:54:222
1458699367,5cyclictest7850irq/56-dwc_otg_20:10:382
1458699365,21cyclictest19112-21ssh23:10:212
1458699365,21cyclictest19112-21ssh23:10:212
1458699363,12cyclictest7950irq/56-dwc_otg_22:37:232
1458699361,4cyclictest7950irq/56-dwc_otg_23:35:352
1458699361,25cyclictest0-21swapper/223:40:202
1458699361,25cyclictest0-21swapper/223:40:202
1458699361,25cyclictest0-21swapper/219:25:212
14585993618,10cyclictest0-21swapper/100:05:191
14585993618,10cyclictest0-21swapper/100:05:191
14585993611,5cyclictest19416-21cut23:10:251
14585993611,5cyclictest19416-21cut23:10:251
1458799354,6cyclictest7950irq/56-dwc_otg_22:25:093
1458799351,24cyclictest26110-21missed_timers00:00:313
14587993511,12cyclictest0-21swapper/322:55:193
1458699356,12cyclictest7950irq/56-dwc_otg_23:31:262
14586993517,8cyclictest7750irq/56-dwc_otg21:41:382
1458599355,6cyclictest0-21swapper/121:50:031
1458599352,15cyclictest16551-21ssh22:25:211
14585993516,5cyclictest14627-21apt00:30:091
14585993516,5cyclictest14627-21apt00:30:091
14585993513,5cyclictest221ktimersoftd/119:15:081
1458799346,14cyclictest7950irq/56-dwc_otg_23:35:093
14587993424,6cyclictest29336-21idleruntime-cro22:00:083
14587993415,5cyclictest6454-21fschecks_time23:35:263
14587993411,4cyclictest7750irq/56-dwc_otg23:45:093
14587993411,4cyclictest7750irq/56-dwc_otg23:45:093
1458699345,16cyclictest7950irq/56-dwc_otg_00:35:332
1458699343,3cyclictest7950irq/56-dwc_otg_20:40:172
1458599349,6cyclictest7850irq/56-dwc_otg_20:35:081
1458599349,6cyclictest21988-21irqstats21:05:271
1458599348,5cyclictest7750irq/56-dwc_otg21:25:251
1458599347,6cyclictest7750irq/56-dwc_otg23:50:291
1458599347,12cyclictest7950irq/56-dwc_otg_23:45:261
1458599344,7cyclictest7950irq/56-dwc_otg_23:05:081
1458599341,4cyclictest7609-21apt-get00:20:151
14585993412,14cyclictest0-21swapper/119:40:361
14585993412,14cyclictest0-21swapper/119:40:361
141382343,25sleep20-21swapper/219:07:522
141382343,25sleep20-21swapper/219:07:522
1458799336,15cyclictest0-21swapper/320:00:243
14587993314,9cyclictest23105-21diskmemload00:15:563
1458799331,22cyclictest3869-21ls20:10:093
1458799331,22cyclictest0-21swapper/322:31:363
1458699339,9cyclictest32504-21sh22:03:462
1458699337,10cyclictest7850irq/56-dwc_otg_23:20:092
1458699332,6cyclictest31-21ksoftirqd/222:09:262
1458599338,5cyclictest7750irq/56-dwc_otg00:20:091
1458599332,11cyclictest0-21swapper/120:10:211
14585993317,5cyclictest29454-21ssh21:17:231
14585993313,12cyclictest30595-1kworker/u9:122:45:041
1458599331,21cyclictest16735-21cut19:15:251
1458599331,20cyclictest0-21swapper/120:55:211
1458599331,16cyclictest31223-21ssh21:20:221
14585993310,6cyclictest0-21swapper/100:35:401
1458799326,11cyclictest7950irq/56-dwc_otg_23:50:253
1458799324,14cyclictest7950irq/56-dwc_otg_19:30:363
14587993210,4cyclictest0-21swapper/321:10:343
1458599325,7cyclictest7750irq/56-dwc_otg20:00:081
1458599324,7cyclictest7750irq/56-dwc_otg20:40:321
1458599323,5cyclictest24317-21apt-get22:35:151
1458599321,5cyclictest25779-21/usr/sbin/munin21:55:301
1458599321,2cyclictest0-21swapper/121:35:201
1458599321,21cyclictest8060-21ls22:55:301
1458599321,15cyclictest7143-21ssh23:35:351
1458799316,13cyclictest7950irq/56-dwc_otg_22:15:203
1458799315,12cyclictest7850irq/56-dwc_otg_23:15:233
14587993113,6cyclictest11752-21ssh23:00:273
1458799311,14cyclictest32662-21ls20:00:093
14587993111,4cyclictest22139-21cpuspeed21:50:233
1458699317,4cyclictest29087-21ssh23:22:342
1458699317,14cyclictest0-21swapper/222:45:232
1458699315,5cyclictest3153-21sed20:05:362
1458699314,4cyclictest7850irq/56-dwc_otg_19:15:262
1458699314,12cyclictest7950irq/56-dwc_otg_19:24:022
1458599317,13cyclictest0-21swapper/122:16:111
1458599311,21cyclictest20404-21uniq19:25:251
14585993111,5cyclictest21-21rcuc/122:00:221
1458799301,19cyclictest0-21swapper/320:36:063
1458799301,18cyclictest4215-21grep20:10:273
1458699301,7cyclictest1881-21sh20:05:082
1458599301,5cyclictest29610-21irqstats19:50:281
1458599301,18cyclictest2726-21ls20:05:301
1458799292,18cyclictest0-21swapper/319:38:343
1458799291,8cyclictest18627-21awk20:55:313
1458699291,4cyclictest7950irq/56-dwc_otg_19:35:342
1458599297,10cyclictest7750irq/56-dwc_otg19:20:291
1458599295,8cyclictest0-21swapper/119:34:091
14585992923,4cyclictest9895-21latency_hist20:30:101
14585992923,4cyclictest9895-21latency_hist20:30:101
1458599292,12cyclictest0-21swapper/120:00:251
1458599291,21cyclictest0-21swapper/120:54:531
1458599291,15cyclictest7750irq/56-dwc_otg20:15:321
304782281,4sleep20-21swapper/219:52:212
1458799288,9cyclictest7950irq/56-dwc_otg_19:25:123
1458799282,18cyclictest0-21swapper/319:55:003
1458699286,6cyclictest301ktimersoftd/220:25:452
1458699286,6cyclictest301ktimersoftd/220:25:452
1458599286,10cyclictest0-21swapper/120:39:471
140872283,19sleep10-21swapper/119:07:101
140872283,19sleep10-21swapper/119:07:101
1458799273,13cyclictest7950irq/56-dwc_otg_20:28:013
1458799273,13cyclictest7950irq/56-dwc_otg_20:28:013
1458799271,6cyclictest7950irq/56-dwc_otg_19:20:253
14587992712,4cyclictest17425-21seq20:52:033
127482263,14sleep331331-21snmpd19:05:143
127482263,14sleep331331-21snmpd19:05:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional