You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 04:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Mon Nov 24, 2025 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2806624990,5sleep30-21swapper/319:05:323
2889722240,3sleep128899-21sed19:10:071
2684621140,2sleep00-21swapper/000:20:360
35602980,21sleep37950irq/56-dwc_otg_20:55:313
65222860,4sleep16490-21ssh23:55:251
245362840,4sleep024154-21/usr/sbin/munin20:25:250
2908899816,39cyclictest0-21swapper/021:55:130
299022750,5sleep229850-21ssh22:20:082
29088997412,14cyclictest0-21swapper/019:15:170
37312710,2sleep10-21swapper/100:34:301
2909099717,26cyclictest0-21swapper/223:05:202
29090997030,19cyclictest3672-21ssh22:27:292
29090996931,14cyclictest18599-21tune2fs22:05:232
29090996931,13cyclictest16703-21rm00:08:352
190102670,2sleep30-21swapper/320:10:273
29090996633,11cyclictest5482-21ls21:00:362
29090996631,18cyclictest16748-21ssh22:02:352
29090996631,18cyclictest16748-21ssh22:02:352
29090996630,25cyclictest25113-21wc20:25:342
2908899663,50cyclictest0-21swapper/023:20:420
2908899662,4cyclictest11302-21ssh23:20:080
2908899662,4cyclictest11302-21ssh23:20:080
29088996620,35cyclictest30018-21latency00:25:290
264142660,1sleep00-21swapper/021:33:400
29090996531,14cyclictest5415-21runrttasks22:30:212
29090996528,21cyclictest28565-21ssh22:17:512
2908899656,45cyclictest0-21swapper/023:30:190
29090996433,13cyclictest18317-21ssh23:28:512
29090996430,19cyclictest13601-21rm21:16:142
29090996430,19cyclictest13601-21rm21:16:142
29090996429,19cyclictest4580-21ssh21:46:362
29090996429,13cyclictest7850irq/56-dwc_otg_19:40:262
2908899642,32cyclictest0-21swapper/021:50:190
29088996414,36cyclictest140-21systemd-journal19:45:500
29090996331,13cyclictest22327-21gdbus20:20:082
29090996330,19cyclictest18666-21ssh21:23:342
29090996328,15cyclictest29017-21apt-get00:25:102
29090996326,12cyclictest5713-21latency_hist19:35:072
29090996326,12cyclictest5713-21latency_hist19:35:072
29090996233,11cyclictest32738-21ssh21:40:482
29090996231,12cyclictest28529-21ssh21:35:362
2908899622,5cyclictest20194-21awk20:15:070
2908899621,9cyclictest23257-21ssh23:35:200
29088996211,29cyclictest0-21swapper/020:30:350
122832620,2sleep20-21swapper/223:20:352
29090996133,17cyclictest12576-21ssh00:03:022
29090996131,9cyclictest24604-21apt-get22:55:192
29090996131,13cyclictest14865-21ssh22:41:382
29090996130,11cyclictest22632-21/usr/sbin/munin00:15:362
29090996128,12cyclictest7405-21ls23:15:082
29090996127,26cyclictest12016-21sh22:38:392
29090996126,17cyclictest10822-21ssh21:13:192
29090996125,21cyclictest0-21swapper/223:45:512
29090996125,21cyclictest0-21swapper/223:45:512
29090996125,12cyclictest26736-21apt23:40:072
29090996124,21cyclictest20399-21ssh22:50:042
29090996124,21cyclictest20399-21ssh22:50:042
2908899611,4cyclictest9755-21munin-run00:00:070
29090996029,12cyclictest11764-21ls21:55:362
29090996028,11cyclictest6011-21ssh00:36:252
29090996025,24cyclictest27793-21sh00:22:282
177362600,2sleep30-21swapper/300:10:083
29090995931,12cyclictest3500-21ssh00:33:592
29090995927,21cyclictest26972-21ssh23:40:112
29090995923,23cyclictest23939-21sh22:11:182
279272590,2sleep00-21swapper/019:05:290
29090995830,13cyclictest8193-21rm23:56:592
29090995828,20cyclictest19479-21ssh00:11:102
29090995825,11cyclictest0-21swapper/221:30:182
2908899581,4cyclictest5544-21grep00:35:360
2908899581,36cyclictest1045-21aten_rbpower_vo20:50:220
2908899581,34cyclictest0-21swapper/023:40:260
2909199576,16cyclictest0-21swapper/320:00:213
29090995731,16cyclictest5581-21ssh23:54:312
29090995728,14cyclictest31227-21ssh23:03:412
29090995719,19cyclictest9967-21ssh23:17:422
29090995719,19cyclictest9967-21ssh23:17:422
2908899574,30cyclictest0-21swapper/020:35:220
2908899571,36cyclictest25818-21grep22:55:370
2908899571,32cyclictest27327-21grep22:15:360
2908899571,32cyclictest27327-21grep22:15:360
2908899571,29cyclictest12250-21fschecks_count21:15:240
2908899571,29cyclictest12250-21fschecks_count21:15:240
29090995629,13cyclictest23006-21ssh22:52:322
29090995626,13cyclictest26714-21/usr/sbin/munin20:30:282
29090995626,12cyclictest5815-21aten_rbpower_cu19:35:202
29090995625,11cyclictest9846-21ssh21:54:282
29090995619,11cyclictest0-21swapper/219:22:062
29088995631,6cyclictest7950irq/56-dwc_otg_23:10:070
29088995630,4cyclictest7950irq/56-dwc_otg_21:10:180
29088995627,12cyclictest7950irq/56-dwc_otg_21:45:270
173742560,1sleep10-21swapper/122:03:531
173742560,1sleep10-21swapper/122:03:531
29090995530,11cyclictest22591-21ssh23:34:362
29090995527,12cyclictest32184-21ssh22:22:222
29090995524,21cyclictest21823-21ssh21:27:112
29088995529,5cyclictest0-21swapper/019:30:320
29088995529,5cyclictest0-21swapper/019:30:320
29088995527,8cyclictest7950irq/56-dwc_otg_20:45:340
29088995527,13cyclictest7950irq/56-dwc_otg_22:20:370
2908899551,4cyclictest32063-21ssh21:40:290
2908899551,30cyclictest13898-21proc_pri22:40:310
249942550,2sleep00-21swapper/022:13:260
29090995422,12cyclictest2293-21taskset20:52:342
2908999547,36cyclictest0-21swapper/100:00:211
2908899543,12cyclictest0-21swapper/021:35:210
29088995427,12cyclictest7850irq/56-dwc_otg_22:30:360
2908899541,10cyclictest5062-21grep23:10:380
159912540,1sleep30-21swapper/322:43:573
29090995327,14cyclictest17562-21/usr/sbin/munin20:05:322
29090995326,18cyclictest29115-21taskset20:36:142
29090995323,12cyclictest10474-21unixbench-2d19:45:362
29090995319,18cyclictest7950irq/56-dwc_otg_20:44:372
2908999534,24cyclictest3788-21sh23:10:201
29088995338,5cyclictest101ktimersoftd/000:30:340
29088995331,4cyclictest7950irq/56-dwc_otg_23:05:070
29090995241,8cyclictest11852-21ls19:50:292
29090995224,13cyclictest7750irq/56-dwc_otg19:19:142
29088995227,5cyclictest7850irq/56-dwc_otg_22:45:350
29088995227,5cyclictest7850irq/56-dwc_otg_22:45:350
2908899522,42cyclictest9-21ksoftirqd/021:00:210
29090995124,17cyclictest7585-21runrttasks21:06:012
29090995119,11cyclictest7750irq/56-dwc_otg20:55:212
29090995117,19cyclictest7950irq/56-dwc_otg_20:50:012
2908999511,30cyclictest5750-21ls19:35:071
2908999511,30cyclictest5750-21ls19:35:071
29088995129,5cyclictest7950irq/56-dwc_otg_23:50:350
29088995126,8cyclictest7850irq/56-dwc_otg_22:50:360
2908899511,4cyclictest21790-21ssh00:15:220
29090995022,10cyclictest7850irq/56-dwc_otg_19:11:192
2908999504,35cyclictest7750irq/56-dwc_otg22:40:301
29090994923,12cyclictest0-21swapper/219:27:042
29090994923,12cyclictest0-21swapper/219:27:042
29089994918,6cyclictest17841-21sh00:10:131
2908899499,19cyclictest0-21swapper/022:05:220
29088994929,5cyclictest7950irq/56-dwc_otg_22:35:340
2908899492,30cyclictest0-21swapper/019:20:280
29088994921,6cyclictest23276-21munin-run21:30:070
2908899491,6cyclictest15865-21/usr/sbin/munin21:20:290
2908899491,26cyclictest6640-21fschecks_time21:05:250
29090994832,12cyclictest15010-21wget20:00:222
29090994825,11cyclictest19971-21runrttasks20:12:292
2908999485,24cyclictest0-21swapper/123:40:191
2908899482,27cyclictest31664-21ntpq23:45:290
2908899482,27cyclictest31664-21ntpq23:45:290
29088994817,6cyclictest0-21swapper/000:05:240
29088994718,13cyclictest0-21swapper/020:00:350
29088994712,9cyclictest21941-21kworker/0:220:20:210
283792470,6sleep12908999cyclictest23:00:171
2908899468,16cyclictest9450irq/48-DMA09:28:480
2908899468,16cyclictest9450irq/48-DMA09:28:480
2908899463,8cyclictest21581-21grep20:15:370
2908899463,22cyclictest0-21swapper/019:35:170
2908899462,34cyclictest0-21swapper/023:28:580
267272460,4sleep126719-21cron23:40:071
2909199457,28cyclictest14505-21ssh22:00:223
2909199457,28cyclictest14505-21ssh22:00:223
29090994529,12cyclictest13330-21tune2fs19:55:252
29090994519,14cyclictest7950irq/56-dwc_otg_20:17:352
29089994514,19cyclictest7750irq/56-dwc_otg20:35:221
2908899452,19cyclictest17128-21fschecks_time20:05:250
2908899451,26cyclictest10179-21grep00:00:210
2909199445,25cyclictest0-21swapper/300:35:213
29088994423,9cyclictest7950irq/56-dwc_otg_22:25:360
29088994414,16cyclictest1120-21runrttasks19:42:100
2908899441,25cyclictest0-21swapper/000:10:330
2908899441,24cyclictest0-21swapper/022:00:380
2908899441,24cyclictest0-21swapper/022:00:380
2909199437,22cyclictest0-21swapper/320:35:223
29088994319,5cyclictest13596-21cat19:55:290
2908899431,6cyclictest29404-21fschecks_time19:10:260
2909199426,23cyclictest0-21swapper/300:10:133
2908999425,25cyclictest7750irq/56-dwc_otg23:05:181
2908999422,25cyclictest7750irq/56-dwc_otg21:20:271
2908899421,26cyclictest7850irq/56-dwc_otg_21:00:070
2909199417,24cyclictest0-21swapper/321:30:203
29091994125,9cyclictest381ktimersoftd/323:45:273
29091994125,9cyclictest381ktimersoftd/323:45:273
2908999418,23cyclictest0-21swapper/120:55:201
2908899411,21cyclictest0-21swapper/020:40:230
122182410,11sleep123-21ksoftirqd/123:20:341
2909199403,4cyclictest7750irq/56-dwc_otg19:45:203
29091994029,6cyclictest7950irq/56-dwc_otg_00:15:223
2909199402,5cyclictest7850irq/56-dwc_otg_20:50:203
29091994015,6cyclictest15676-21date21:20:083
29091994015,6cyclictest15676-21date21:20:083
2908999407,21cyclictest0-21swapper/100:35:211
2908999402,6cyclictest0-21swapper/120:50:191
29089994011,4cyclictest16883-21ssh22:45:171
29089994011,4cyclictest16883-21ssh22:45:171
2908899402,28cyclictest0-21swapper/019:50:290
2909199395,25cyclictest0-21swapper/323:55:213
2909199393,25cyclictest7850irq/56-dwc_otg_22:00:083
2909199392,3cyclictest0-21swapper/320:40:293
2908999398,6cyclictest23286-21sh21:30:071
2908999395,22cyclictest7750irq/56-dwc_otg21:55:371
2908999394,22cyclictest7750irq/56-dwc_otg22:10:221
2908999391,9cyclictest19539-21sed22:05:351
2909199382,16cyclictest0-21swapper/319:10:253
29091993817,6cyclictest5279-21sh21:48:013
2908999388,18cyclictest7750irq/56-dwc_otg22:15:221
2908999388,18cyclictest7750irq/56-dwc_otg22:15:221
2908999386,9cyclictest8873-21latency_hist22:35:071
2908999384,22cyclictest7750irq/56-dwc_otg23:29:431
2908999384,20cyclictest7750irq/56-dwc_otg22:55:361
2908999382,8cyclictest0-21swapper/123:45:181
2908999382,8cyclictest0-21swapper/123:45:181
2908999382,26cyclictest0-21swapper/120:45:221
29089993818,5cyclictest153150wget22:25:231
29089993810,14cyclictest0-21swapper/121:19:101
29089993810,14cyclictest0-21swapper/121:19:101
2909199373,5cyclictest7950irq/56-dwc_otg_19:20:253
29091993713,15cyclictest0-21swapper/321:50:333
2909199371,22cyclictest3609-21ls23:10:083
2908999375,4cyclictest0-21swapper/121:40:301
2908999375,20cyclictest7750irq/56-dwc_otg20:00:331
2908999372,24cyclictest7750irq/56-dwc_otg23:50:191
29089993717,4cyclictest22377-21sh23:34:101
2908999371,22cyclictest7750irq/56-dwc_otg00:05:331
278862370,1sleep30-21swapper/300:22:403
2909199367,12cyclictest7950irq/56-dwc_otg_23:00:353
29091993627,3cyclictest7850irq/56-dwc_otg_22:05:333
2909199361,21cyclictest381ktimersoftd/322:10:313
2908999363,7cyclictest0-21swapper/121:35:211
2908999363,5cyclictest9448-21wget19:45:221
29089993627,5cyclictest20796-21aten_rbpower_cu22:50:211
2908999362,15cyclictest7850irq/56-dwc_otg_21:05:251
29089993618,5cyclictest9174-21idleruntime-cro19:45:071
29089993617,5cyclictest29169-21aten_rbpower_en19:10:231
29089993613,5cyclictest20480-21aten_rbpower_en20:15:231
2908999361,11cyclictest0-21swapper/121:50:201
2909199354,15cyclictest7950irq/56-dwc_otg_00:25:263
2908999356,18cyclictest7750irq/56-dwc_otg21:50:081
29089993513,11cyclictest16819-21acpi20:05:181
2909199346,15cyclictest0-21swapper/322:15:213
2909199346,15cyclictest0-21swapper/322:15:213
2909199343,20cyclictest0-21swapper/321:20:263
29091993424,7cyclictest8113-21irqstats23:15:263
29091993424,7cyclictest8113-21irqstats23:15:263
2909199342,13cyclictest16794-21ssh23:25:503
2908999349,14cyclictest0-21swapper/100:20:071
2908999345,6cyclictest7950irq/56-dwc_otg_22:20:371
2908999344,21cyclictest7750irq/56-dwc_otg20:25:341
2908999343,21cyclictest7750irq/56-dwc_otg00:25:091
2908999341,26cyclictest6157-21ls19:35:241
2908999341,24cyclictest0-21swapper/120:30:241
29089993411,4cyclictest7850irq/56-dwc_otg_20:20:261
71922330,6sleep32909199cyclictest22:31:453
2909199338,6cyclictest7950irq/56-dwc_otg_19:55:243
2909199335,17cyclictest0-21swapper/319:25:133
2909199335,17cyclictest0-21swapper/319:25:133
2909199333,6cyclictest31829-21/usr/sbin/munin21:40:263
29091993319,5cyclictest381ktimersoftd/323:40:233
2909199331,8cyclictest8809-21grep21:10:273
29091993317,4cyclictest21489-21sh21:26:303
29091993310,4cyclictest37-21rcuc/323:35:243
29091993310,11cyclictest7950irq/56-dwc_otg_20:30:323
2908999336,17cyclictest7750irq/56-dwc_otg00:25:071
2908999332,8cyclictest7950irq/56-dwc_otg_22:35:221
2908999332,22cyclictest7750irq/56-dwc_otg21:10:351
2908999331,8cyclictest20193-21latency_hist20:15:071
29089993313,10cyclictest0-21swapper/121:00:371
2909199328,5cyclictest7950irq/56-dwc_otg_20:15:373
2909199325,6cyclictest7850irq/56-dwc_otg_22:25:313
2909199325,19cyclictest7750irq/56-dwc_otg20:20:263
29091993223,6cyclictest12552-21ssh00:03:003
2909199322,21cyclictest0-21swapper/319:17:373
29091993213,11cyclictest3973-21/usr/sbin/munin23:10:243
2909199321,22cyclictest0-21swapper/322:55:193
2908999329,14cyclictest7921-21diskmemload21:30:251
2908999323,21cyclictest7850irq/56-dwc_otg_20:00:071
2908999322,19cyclictest7750irq/56-dwc_otg20:40:301
29089993221,5cyclictest0-21swapper/119:15:171
2908999321,6cyclictest11325-21ls23:20:081
2908999321,6cyclictest11325-21ls23:20:081
2908999321,21cyclictest3792-21ls19:30:081
2908999321,21cyclictest3792-21ls19:30:081
2909199319,5cyclictest7750irq/56-dwc_otg23:20:413
2909199318,12cyclictest7950irq/56-dwc_otg_19:50:253
2909199316,16cyclictest0-21swapper/319:44:063
2909199314,18cyclictest30861-21grep22:20:333
29091993124,5cyclictest23172-21ssh22:52:523
2909199312,19cyclictest0-21swapper/320:25:303
2909199311,19cyclictest6383-21cut21:05:223
2909199309,10cyclictest0-21swapper/322:37:513
2909199306,15cyclictest0-21swapper/321:35:223
2909199306,15cyclictest0-21swapper/320:45:133
2909199303,13cyclictest7950irq/56-dwc_otg_00:30:303
2909199302,8cyclictest0-21swapper/321:01:013
2909199302,3cyclictest0-21swapper/323:55:073
2909199301,18cyclictest17509-21grep20:05:303
29089993024,4cyclictest12795-21idleruntime-cro19:55:071
2909199297,5cyclictest6899-21grep19:35:343
2909199297,11cyclictest7950irq/56-dwc_otg_22:45:303
2909199297,11cyclictest7950irq/56-dwc_otg_22:45:303
2909199296,14cyclictest0-21swapper/323:30:223
2909199291,19cyclictest0-21swapper/319:30:273
2909199291,19cyclictest0-21swapper/319:30:273
2908999291,20cyclictest0-21swapper/119:20:191
288142263,15sleep20-21swapper/219:10:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional