You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 00:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sat Nov 29, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1950323250,7sleep219506-21fschecks_time07:05:282
178062970,5sleep02100499cyclictest12:34:130
178062970,5sleep02100499cyclictest12:34:130
212292880,5sleep12100599cyclictest11:55:371
212292880,5sleep12100599cyclictest11:55:371
226892810,2sleep00-21swapper/009:41:500
306172770,2sleep330619-21rm11:24:593
164212740,4sleep20-21swapper/211:50:112
2100699721,41cyclictest21449-21/usr/sbin/munin10:25:322
21004997014,44cyclictest0-21swapper/012:05:420
2100699691,42cyclictest27561-21ls09:50:102
2100499679,46cyclictest0-21swapper/009:25:280
2100499679,46cyclictest0-21swapper/009:25:280
2100499674,46cyclictest0-21swapper/011:15:220
127872670,2sleep00-21swapper/012:27:280
274272650,2sleep00-21swapper/012:05:100
2100699653,43cyclictest7850irq/56-dwc_otg_09:20:262
2100699652,44cyclictest7950irq/56-dwc_otg_09:35:262
2100699651,5cyclictest20426-21chrt12:36:282
2100499648,43cyclictest0-21swapper/007:35:390
2100499645,47cyclictest9-21ksoftirqd/010:30:400
2100699635,42cyclictest7950irq/56-dwc_otg_12:25:102
2100699632,49cyclictest0-21swapper/210:00:542
2100699632,46cyclictest0-21swapper/210:08:282
2100499639,16cyclictest13-21rcuc/009:20:230
2100699627,36cyclictest7950irq/56-dwc_otg_12:30:452
2100699627,36cyclictest7950irq/56-dwc_otg_12:30:452
2100699626,43cyclictest7950irq/56-dwc_otg_08:40:272
2100699624,43cyclictest10311-21/usr/sbin/munin09:25:362
2100699624,43cyclictest10311-21/usr/sbin/munin09:25:362
2100699622,45cyclictest7950irq/56-dwc_otg_12:08:432
2100699621,5cyclictest15265-21ssh10:16:242
21004996228,3cyclictest0-21swapper/011:25:240
21004996228,3cyclictest0-21swapper/011:25:240
2100699613,33cyclictest7950irq/56-dwc_otg_10:20:262
2100699611,42cyclictest2181-21ssh10:44:442
2100599615,8cyclictest0-21swapper/109:30:301
2100499613,37cyclictest0-21swapper/007:20:230
2100499613,37cyclictest0-21swapper/007:20:230
2100499612,31cyclictest0-21swapper/008:30:220
2100699602,49cyclictest5658-21ssh12:18:212
2100699601,42cyclictest10978-21grep10:55:402
2100499601,41cyclictest3214-21wget07:50:250
21004996011,39cyclictest0-21swapper/011:50:230
2100699593,6cyclictest7750irq/56-dwc_otg08:55:232
2100699593,6cyclictest7750irq/56-dwc_otg08:55:232
2100699592,42cyclictest0-21swapper/211:53:152
2100699592,42cyclictest0-21swapper/211:19:442
2100699591,43cyclictest0-21swapper/212:11:312
2100499596,43cyclictest0-21swapper/010:39:560
2100499594,32cyclictest0-21swapper/010:05:290
21004995928,4cyclictest7750irq/56-dwc_otg11:20:230
2100499591,47cyclictest0-21swapper/008:40:370
2100699584,40cyclictest7950irq/56-dwc_otg_09:55:362
2100699584,38cyclictest9632-21fschecks_count11:40:312
2100699581,4cyclictest7950irq/56-dwc_otg_12:28:212
2100699581,47cyclictest22558-21seq11:57:292
2100699581,47cyclictest22558-21seq11:57:292
2100699581,3cyclictest7850irq/56-dwc_otg_12:00:182
2100699581,39cyclictest27804-21latency_hist10:35:102
2100599587,28cyclictest0-21swapper/107:20:261
2100599587,28cyclictest0-21swapper/107:20:261
21004995841,10cyclictest0-21swapper/009:50:230
2100699574,35cyclictest7850irq/56-dwc_otg_07:55:332
2100699571,45cyclictest23034-21ssh11:13:582
2100699571,40cyclictest7950irq/56-dwc_otg_10:50:372
2100599574,36cyclictest26568-21runrttasks07:25:241
21004995721,22cyclictest20557-21grep08:35:510
2100699565,33cyclictest31785-21diskmemload11:00:392
2100699562,36cyclictest0-21swapper/211:20:412
2100699562,32cyclictest0-21swapper/209:53:102
2100699561,41cyclictest0-21swapper/208:51:012
2100699561,39cyclictest31821-21ssh11:25:382
2100699561,39cyclictest31821-21ssh11:25:382
2100499567,25cyclictest0-21swapper/009:35:250
21004995611,33cyclictest13-21rcuc/007:55:280
204752563,12sleep30-21swapper/307:06:523
2100699552,36cyclictest7850irq/56-dwc_otg_07:10:382
2100699551,43cyclictest1120-21runrttasks09:44:472
2100699551,41cyclictest15488-21ssh09:31:282
2100499551,18cyclictest17177-21ssh09:35:110
2100699544,35cyclictest7950irq/56-dwc_otg_07:45:212
2100699542,4cyclictest0-21swapper/211:05:252
2100699541,5cyclictest3853-21chrt07:50:352
2100699541,3cyclictest7950irq/56-dwc_otg_09:17:492
2100499549,29cyclictest338-21unixbench_singl09:55:430
21004995437,9cyclictest7950irq/56-dwc_otg_11:10:100
2100499542,26cyclictest0-21swapper/008:50:100
21004995412,34cyclictest3101050kworker/u9:112:12:420
21004995411,34cyclictest0-21swapper/007:15:220
2100699533,42cyclictest9967-21ssh10:10:212
2100699533,38cyclictest7750irq/56-dwc_otg11:31:322
2100699532,41cyclictest0-21swapper/210:39:382
2100699532,36cyclictest0-21swapper/208:30:102
2100699531,40cyclictest0-21swapper/209:15:012
2100699531,38cyclictest0-21swapper/209:04:182
2100599532,13cyclictest0-21swapper/107:40:241
21004995313,5cyclictest0-21swapper/008:50:230
2100699521,39cyclictest0-21swapper/207:37:032
2100699521,38cyclictest0-21swapper/211:37:252
2100599524,11cyclictest32126-21ssh09:10:231
2100599523,10cyclictest0-21swapper/107:45:211
2100499526,40cyclictest13547-21kworker/0:108:25:240
2100499522,28cyclictest30108-21latency_hist07:35:110
2100499521,41cyclictest20618-21cat10:25:100
2100499521,29cyclictest1158-21latency_hist07:45:100
21007995112,9cyclictest7950irq/56-dwc_otg_11:15:253
2100699519,39cyclictest22941-21aten_rbpower_vo07:15:252
2100699512,3cyclictest19384-21apt-get08:35:232
2100699511,40cyclictest11632-21chrt08:11:442
2100699511,3cyclictest0-21swapper/208:45:222
2100599512,8cyclictest0-21swapper/112:10:231
2100499513,39cyclictest21699-21ssh10:25:370
21004995132,3cyclictest0-21swapper/011:45:320
21004995127,11cyclictest7950irq/56-dwc_otg_09:05:390
2100499512,29cyclictest18682-21sh12:35:220
21004995117,7cyclictest0-21swapper/010:40:240
2100499511,30cyclictest10549-21unixbench_multi11:40:420
2100499511,25cyclictest24754-21fschecks_count09:45:290
2100699501,5cyclictest9118-21ls08:05:322
21004995020,9cyclictest0-21swapper/011:55:380
21004995020,9cyclictest0-21swapper/011:55:380
21004995017,21cyclictest12737-21sh10:59:420
2100499501,33cyclictest140-21systemd-journal07:29:260
2100499501,23cyclictest32094-21/usr/sbin/munin09:10:370
2100699492,32cyclictest0-21swapper/208:35:032
2100699491,4cyclictest0-21swapper/207:25:232
2100699491,45cyclictest6792-21aten_rbpower_cu08:00:252
2100699491,45cyclictest6792-21aten_rbpower_cu08:00:252
21005994939,8cyclictest221ktimersoftd/111:30:221
21005994916,7cyclictest9714-21fschecks_time11:40:311
2100499499,31cyclictest0-21swapper/011:35:300
21004994939,4cyclictest7950irq/56-dwc_otg_11:10:420
21004994925,5cyclictest7850irq/56-dwc_otg_11:30:220
2100499491,9cyclictest10680-21fschecks_time08:10:300
2100499491,26cyclictest0-21swapper/008:00:270
2100499491,26cyclictest0-21swapper/008:00:270
2100699482,5cyclictest396-21switchtime07:40:372
2100699482,5cyclictest0-21swapper/209:05:252
2100699482,42cyclictest2802-21basename10:45:272
2100699481,6cyclictest25352-21/usr/sbin/munin07:20:352
2100699481,6cyclictest25352-21/usr/sbin/munin07:20:352
2100699481,44cyclictest13312-21taskset08:15:392
2100599485,18cyclictest31558-21ssh09:55:251
2100599482,5cyclictest7750irq/56-dwc_otg07:55:221
2100499487,29cyclictest0-21swapper/012:20:260
21004994823,9cyclictest0-21swapper/008:55:310
21004994823,9cyclictest0-21swapper/008:55:310
21004994822,13cyclictest0-21swapper/008:20:360
2100499481,15cyclictest31785-21diskmemload10:00:220
2100699471,43cyclictest14259-21fschecks_count08:20:272
2100599473,7cyclictest0-21swapper/109:35:241
21004994725,7cyclictest3794-21/usr/sbin/munin12:15:360
21004994722,5cyclictest6174-21/usr/sbin/munin10:50:340
2100499472,25cyclictest0-21swapper/010:45:280
2100499471,35cyclictest1262650ssh10:13:410
2100699461,42cyclictest28397-21aten_rbpower_cu07:30:232
21005994615,5cyclictest7950irq/56-dwc_otg_10:50:241
2100499461,23cyclictest17001-21idleruntime-cro10:20:100
2100499461,21cyclictest0-21swapper/009:15:340
2100799457,25cyclictest0-21swapper/308:25:253
2100799457,23cyclictest0-21swapper/308:20:233
2100599451,12cyclictest0-21swapper/108:05:231
21004994514,14cyclictest12975-21/usr/sbin/munin08:15:360
2100799446,4cyclictest0-21swapper/309:50:253
2100799442,12cyclictest7750irq/56-dwc_otg08:35:223
21004994416,10cyclictest0-21swapper/009:00:390
2100499441,24cyclictest140-21systemd-journal11:03:390
2100799436,24cyclictest0-21swapper/308:45:243
21007994319,13cyclictest374-21awk09:55:433
2100599435,15cyclictest7750irq/56-dwc_otg08:50:231
2100599432,17cyclictest7950irq/56-dwc_otg_12:15:241
21004994312,21cyclictest0-21swapper/007:10:310
207142432,35sleep00-21swapper/007:09:560
2100799427,22cyclictest0-21swapper/310:10:253
2100799421,5cyclictest7850irq/56-dwc_otg_12:30:233
2100799421,5cyclictest7850irq/56-dwc_otg_12:30:233
2100599427,25cyclictest0-21swapper/111:25:271
2100599427,25cyclictest0-21swapper/111:25:271
21004994211,16cyclictest0-21swapper/007:45:210
179672420,2sleep123-21ksoftirqd/111:50:581
303392410,2sleep30-21swapper/312:08:113
2100599417,24cyclictest0-21swapper/111:10:251
2100599411,5cyclictest10890-21fschecks_time12:25:301
118422410,2sleep10-21swapper/110:57:361
2100799400,4cyclictest7190-21diskstats09:20:273
2100599408,27cyclictest0-21swapper/110:20:261
21005994018,12cyclictest5974-21sh10:05:041
2100799398,25cyclictest7950irq/56-dwc_otg_11:35:223
2100599398,22cyclictest0-21swapper/108:55:241
2100599398,22cyclictest0-21swapper/108:55:241
2100599397,22cyclictest0-21swapper/108:35:251
2100599396,15cyclictest7950irq/56-dwc_otg_08:00:391
2100599396,15cyclictest7950irq/56-dwc_otg_08:00:391
21005993921,4cyclictest24448-21/usr/sbin/munin09:45:341
21005993914,5cyclictest23-21ksoftirqd/110:25:391
21007993816,10cyclictest0-21swapper/308:05:403
2100599386,22cyclictest18704-21ssh12:35:251
21005993813,12cyclictest13972-21fschecks_count10:15:301
2100799375,4cyclictest7850irq/56-dwc_otg_10:15:233
21007993726,5cyclictest31331-21snmpd11:03:393
2100599378,12cyclictest7850irq/56-dwc_otg_10:40:111
21004993711,11cyclictest0-21swapper/008:05:280
2100799364,27cyclictest7950irq/56-dwc_otg_09:40:233
2100799362,4cyclictest7950irq/56-dwc_otg_07:45:213
2100799361,9cyclictest0-21swapper/307:15:233
2100799361,24cyclictest22518-21ssh12:40:103
2100799361,20cyclictest19286-21ls08:35:113
2100599363,5cyclictest0-21swapper/109:03:161
2100599361,9cyclictest0-21swapper/107:50:241
2100599361,8cyclictest11526-21grep09:25:391
2100599361,8cyclictest11526-21grep09:25:391
21005993617,5cyclictest1120-21runrttasks12:00:351
2100599361,24cyclictest6822-21ls11:35:401
266652350,2sleep10-21swapper/110:32:441
2100799358,5cyclictest0-21swapper/310:05:423
2100799357,15cyclictest0-21swapper/308:00:253
2100799357,15cyclictest0-21swapper/308:00:253
21007993514,14cyclictest30428-21sh10:38:203
2100599351,7cyclictest0-21swapper/111:05:101
2100599351,24cyclictest30571-21ls09:05:311
21005993510,5cyclictest29456-21unixbench_multi07:30:381
2100799347,24cyclictest17843-21ls09:35:303
2100799347,17cyclictest0-21swapper/310:03:373
2100799342,5cyclictest31560-21ssh10:40:233
21007993416,10cyclictest21526-21grep11:10:413
21007993414,8cyclictest1254850http11:45:103
2100799341,24cyclictest0-21swapper/309:00:253
21007993410,13cyclictest31015-21rt-features09:05:383
2100599349,5cyclictest7950irq/56-dwc_otg_11:21:061
2100599346,13cyclictest7950irq/56-dwc_otg_10:45:261
2100599345,5cyclictest7950irq/56-dwc_otg_08:15:281
2100599342,5cyclictest0-21swapper/109:20:391
2100599342,4cyclictest0-21swapper/109:19:351
199282347,14sleep10-21swapper/107:05:361
2100799338,5cyclictest7750irq/56-dwc_otg07:45:103
2100799338,11cyclictest7850irq/56-dwc_otg_11:25:273
2100799338,11cyclictest7850irq/56-dwc_otg_11:25:273
21007993327,4cyclictest20738-21fschecks_time11:55:293
21007993327,4cyclictest20738-21fschecks_time11:55:293
21007993315,11cyclictest1120-21runrttasks10:49:043
2100599333,17cyclictest0-21swapper/110:10:551
21005993317,9cyclictest221ktimersoftd/107:19:451
21005993316,5cyclictest12159-1kworker/u9:011:45:361
2100599331,5cyclictest6346-21/usr/sbin/munin10:05:381
21005993313,5cyclictest17127-21fschecks_count11:05:281
2100599331,26cyclictest21294-21cut07:10:271
21005993312,4cyclictest0-21swapper/109:40:211
21007993221,5cyclictest8737-21sh10:53:403
21007993216,5cyclictest26255-21apt-key07:25:113
21007993216,5cyclictest26255-21apt-key07:25:113
2100599325,4cyclictest27764-21/usr/sbin/munin12:05:301
2100599321,20cyclictest16135-21ls08:25:301
21005993211,4cyclictest0-21swapper/112:20:291
2100799317,5cyclictest7750irq/56-dwc_otg09:30:223
2100799317,15cyclictest0-21swapper/311:05:163
2100799312,20cyclictest0-21swapper/309:48:423
2100799311,7cyclictest7950irq/56-dwc_otg_11:30:283
2100799311,6cyclictest7950irq/56-dwc_otg_10:55:263
2100799311,6cyclictest7950irq/56-dwc_otg_09:25:363
2100799311,6cyclictest7950irq/56-dwc_otg_09:25:363
2100799311,27cyclictest27454-21ls08:55:363
2100799311,27cyclictest27454-21ls08:55:363
2100799311,22cyclictest6816-21ls09:20:113
2100799311,21cyclictest0-21swapper/312:00:163
2100799311,20cyclictest21067-21cpuspeed10:25:273
2100599318,11cyclictest0-21swapper/108:40:341
2100599316,6cyclictest7750irq/56-dwc_otg11:15:301
2100599316,13cyclictest23-21ksoftirqd/112:30:341
2100599316,13cyclictest23-21ksoftirqd/112:30:341
21005993111,10cyclictest23-21ksoftirqd/108:45:321
2100799305,12cyclictest7850irq/56-dwc_otg_09:10:263
2100799302,5cyclictest0-21swapper/308:00:103
2100799302,3cyclictest0-21swapper/310:35:113
21007993017,4cyclictest1120-21runrttasks07:50:233
21007993016,5cyclictest7584-21open_files12:20:373
21007993014,4cyclictest1120-21runrttasks07:32:313
2100799301,24cyclictest31976-21cut12:10:303
21007993011,11cyclictest15041-21rm11:47:443
21007993010,6cyclictest18517-21unixbench_multi10:20:413
2100599308,17cyclictest21003-21cyclictest10:40:241
2100599306,6cyclictest11462-21unixbench_multi08:10:411
2100599306,15cyclictest0-21swapper/108:20:231
2100599304,4cyclictest7850irq/56-dwc_otg_09:50:201
2100799297,5cyclictest7850irq/56-dwc_otg_07:35:423
2100799292,4cyclictest7850irq/56-dwc_otg_08:50:313
2100799291,4cyclictest7950irq/56-dwc_otg_12:25:263
2100799291,14cyclictest0-21swapper/307:26:033
21007992911,11cyclictest22967-21latency_hist08:45:113
2100599294,6cyclictest7850irq/56-dwc_otg_08:30:321
2100599291,24cyclictest0-21swapper/107:35:281
2100799289,12cyclictest381ktimersoftd/308:10:323
2100799288,3cyclictest0-21swapper/312:16:163
2100799283,17cyclictest7950irq/56-dwc_otg_11:50:223
2100799282,15cyclictest7950irq/56-dwc_otg_08:20:093
2100799281,4cyclictest7950irq/56-dwc_otg_07:10:383
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional