You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 21:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Thu Sep 18, 2025 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
242621120,5sleep342999cyclictest07:15:153
897221030,2sleep10-21swapper/110:31:091
310462890,1sleep00-21swapper/011:39:400
42699885,43cyclictest0-21swapper/010:30:280
42699871,40cyclictest28013-21fschecks_time10:55:240
426998624,4cyclictest16270-21grep07:50:310
42699862,32cyclictest0-21swapper/011:55:260
42699852,33cyclictest3519-21fschecks_count11:05:200
426998512,53cyclictest0-21swapper/009:25:160
251722850,5sleep042699cyclictest10:51:010
42699841,8cyclictest6378-21cut08:55:170
42699841,5cyclictest16205-21apt10:00:020
318972840,3sleep231-21ksoftirqd/207:05:302
42699827,37cyclictest21737-1kworker/u9:210:48:370
42699827,37cyclictest21737-1kworker/u9:210:48:370
42699821,42cyclictest29922-21ssh12:17:320
151642820,2sleep015128-21latency_hist12:40:030
42699802,23cyclictest0-21swapper/010:39:590
222142800,1sleep20-21swapper/211:28:192
42699782,7cyclictest1082-21ls09:40:020
42799721,5cyclictest24343-21proc_pri11:30:281
42699702,59cyclictest8154-21aten_rbpower_po09:00:160
42699702,54cyclictest29180-21unixbench_singl10:15:320
42699702,53cyclictest13-21rcuc/010:45:030
42699701,60cyclictest19170-21sh10:03:210
42699691,55cyclictest0-21swapper/008:50:200
42799681,7cyclictest6497-21fschecks_count08:55:201
42699688,43cyclictest388-21unixbench_singl11:40:320
42699682,56cyclictest0-21swapper/011:22:070
251342680,2sleep30-21swapper/309:28:113
42999661,4cyclictest0-21swapper/309:35:033
42799661,21cyclictest0-21swapper/112:31:171
42699662,6cyclictest12571-21unixbench_singl07:40:290
42699661,53cyclictest22189-21cron08:10:020
42699661,52cyclictest0-21swapper/009:10:250
42699661,41cyclictest0-21swapper/009:50:150
42699661,41cyclictest0-21swapper/009:50:150
31562660,2sleep30-21swapper/311:05:093
42799654,21cyclictest7950irq/56-dwc_otg_08:25:221
42699651,56cyclictest0-21swapper/009:15:360
42699651,52cyclictest24406-21/usr/sbin/munin11:30:290
42999644,19cyclictest0-21swapper/309:40:263
42999641,11cyclictest0-21swapper/311:55:263
426996411,42cyclictest0-21swapper/012:05:330
187972640,2sleep342999cyclictest10:02:333
42799632,20cyclictest7950irq/56-dwc_otg_11:25:151
42699638,44cyclictest11435-21diskmemload11:19:180
42699631,44cyclictest32328-21grep08:35:290
42699627,46cyclictest27695-21latency_hist08:25:030
42699627,46cyclictest27695-21latency_hist08:25:030
42699622,39cyclictest0-21swapper/010:25:150
42699621,5cyclictest2161-21latency_hist07:15:020
42999611,9cyclictest0-21swapper/312:17:313
429996118,4cyclictest14735-21sh09:14:293
42799617,14cyclictest7850irq/56-dwc_otg12:03:451
42699619,39cyclictest0-21swapper/011:55:030
42999601,7cyclictest24388-21/usr/sbin/munin11:30:293
42999601,6cyclictest6540-21fschecks_count08:55:203
42799602,6cyclictest3405-21/usr/sbin/munin11:05:181
427996017,7cyclictest7850irq/56-dwc_otg09:40:031
42699602,39cyclictest0-21swapper/012:10:240
42699601,46cyclictest8327-21timerandwakeup11:10:300
42799591,18cyclictest0-21swapper/111:54:481
42699591,36cyclictest4180-21timerandwakeup11:45:300
42799581,5cyclictest0-21swapper/112:15:131
426995836,7cyclictest0-21swapper/008:15:240
42699583,44cyclictest0-21swapper/011:30:020
64462570,2sleep10-21swapper/111:49:151
42799577,11cyclictest0-21swapper/109:23:281
42699572,41cyclictest0-21swapper/011:01:250
42699571,6cyclictest554-21ssh10:20:410
42699571,6cyclictest554-21ssh10:20:410
42699571,46cyclictest0-21swapper/007:25:220
42799568,13cyclictest0-21swapper/109:18:051
42799565,16cyclictest7850irq/56-dwc_otg10:40:181
42799561,17cyclictest0-21swapper/112:23:001
426995610,35cyclictest15914-21cat12:00:220
324072563,13sleep30-21swapper/307:07:223
42799551,5cyclictest7850irq/56-dwc_otg10:00:021
42799541,18cyclictest0-21swapper/110:48:071
42799541,18cyclictest0-21swapper/110:48:071
42799541,15cyclictest0-21swapper/110:51:361
42699544,35cyclictest0-21swapper/009:30:210
42699542,43cyclictest0-21swapper/008:30:300
42699542,40cyclictest0-21swapper/009:20:070
42799534,19cyclictest7850irq/56-dwc_otg12:35:201
42799532,4cyclictest7850irq/56-dwc_otg11:03:531
42799531,5cyclictest20278-21ps12:05:311
427995310,10cyclictest221ktimersoftd/110:37:061
42699532,42cyclictest0-21swapper/012:23:070
42699531,5cyclictest0-21swapper/008:45:550
426995311,30cyclictest11435-21diskmemload10:10:240
42999521,6cyclictest114050snmpd08:27:463
42999521,17cyclictest0-21swapper/311:42:353
42699522,26cyclictest0-21swapper/008:40:240
42699521,5cyclictest5300-21grep07:20:300
426995215,25cyclictest21508-21grep10:05:340
42699512,36cyclictest0-21swapper/009:45:250
42699511,46cyclictest1759-21fschecks_time09:40:220
42699511,39cyclictest0-21swapper/012:25:080
42699501,38cyclictest0-21swapper/007:35:150
42699501,38cyclictest0-21swapper/007:35:150
42999491,20cyclictest0-21swapper/309:46:033
42999491,20cyclictest0-21swapper/309:17:303
42699492,28cyclictest0-21swapper/008:00:300
71852480,4sleep2301ktimersoftd/208:55:292
42999487,25cyclictest0-21swapper/308:00:153
42699482,32cyclictest11636-21cat09:10:040
42699482,32cyclictest11636-21cat09:10:040
42699481,37cyclictest10172-21rm12:33:240
42699481,35cyclictest0-21swapper/008:10:150
42699481,22cyclictest0-21swapper/007:31:350
188582480,2sleep20-21swapper/208:00:232
428994714,5cyclictest12656-21sensors_temp09:10:282
427994722,9cyclictest17514-21sh11:22:061
42699471,5cyclictest1140-21snmpd07:57:570
42699471,28cyclictest0-21swapper/008:26:410
323992463,36sleep00-21swapper/007:07:150
42899453,32cyclictest1170850ssh12:35:252
42699451,25cyclictest13217-21/usr/sbin/munin07:45:130
42699441,27cyclictest0-21swapper/007:15:040
136442440,5sleep231-21ksoftirqd/210:37:222
42999436,29cyclictest7850irq/56-dwc_otg10:25:033
42999436,29cyclictest7850irq/56-dwc_otg10:25:033
42999431,27cyclictest0-21swapper/310:42:473
42799437,25cyclictest0-21swapper/112:25:171
42999427,20cyclictest0-21swapper/312:00:163
42899428,20cyclictest7850irq/56-dwc_otg09:15:262
42799427,26cyclictest0-21swapper/108:45:171
187172420,3sleep318716-21cut08:00:033
42999417,22cyclictest0-21swapper/308:45:173
42799417,24cyclictest18840-21apt-get08:00:161
42799416,25cyclictest7850irq/56-dwc_otg09:40:241
427994125,5cyclictest3153050ssh11:40:091
427994123,7cyclictest8050irq/56-dwc_otg_08:00:031
42799411,25cyclictest0-21swapper/110:16:391
305932410,2sleep20-21swapper/212:18:472
220002410,1sleep20-21swapper/209:24:232
429994014,24cyclictest2516150rm12:11:143
42999401,24cyclictest0-21swapper/307:25:213
42999401,10cyclictest27626-21wget11:35:163
428994022,5cyclictest7781-21sh11:10:232
42799403,24cyclictest7628-21ssh11:10:211
275392400,1sleep10-21swapper/110:55:081
42999398,5cyclictest409-21unixbench_multi12:20:353
42999396,24cyclictest0-21swapper/312:25:173
42999396,23cyclictest0-21swapper/311:20:173
429993921,9cyclictest5220-21rm11:46:573
429993920,5cyclictest8245-21threads11:50:303
429993918,5cyclictest381ktimersoftd/310:15:293
42999391,34cyclictest3441-21aten_rbpower_vo10:25:173
42899394,11cyclictest11399-21/usr/sbin/munin07:40:292
428993922,4cyclictest19891-21if_eth012:05:242
42799393,5cyclictest0-21swapper/107:35:211
42799393,5cyclictest0-21swapper/107:35:211
427993919,13cyclictest30742-21ssh11:39:061
42799391,8cyclictest0-21swapper/111:17:411
427993915,14cyclictest7850irq/56-dwc_otg12:10:231
42799391,3cyclictest0-21swapper/109:25:331
42799391,24cyclictest0-21swapper/107:20:161
42999383,6cyclictest0-21swapper/310:00:023
429993819,4cyclictest28904-21grep10:55:353
429993813,5cyclictest8507-21grep12:30:303
429993811,17cyclictest19911-21if_err_wlan011:25:253
428993814,5cyclictest1707-21sh12:22:482
42799384,5cyclictest12044-21ssh11:55:271
42799382,22cyclictest0-21swapper/107:50:071
42799381,6cyclictest22767-21sort08:10:211
42999379,17cyclictest0-21swapper/310:35:233
42999375,21cyclictest7850irq/56-dwc_otg09:20:283
429993713,5cyclictest8088-21fschecks_count07:30:173
42999371,21cyclictest0-21swapper/311:19:513
428993716,5cyclictest27681-21wget10:55:202
42799375,13cyclictest11786-21ssh09:10:161
427993712,6cyclictest221ktimersoftd/107:45:301
427993710,4cyclictest23254-21sh10:09:011
427993710,12cyclictest0-21swapper/107:15:191
42999369,5cyclictest22684-21fschecks_count08:10:203
42999361,9cyclictest0-21swapper/307:20:303
429993618,4cyclictest1111-21runrttasks12:05:233
42999361,20cyclictest0-21swapper/310:32:583
42999361,13cyclictest3176550ssh11:00:163
42899364,4cyclictest7950irq/56-dwc_otg_10:25:162
42899361,5cyclictest25018-21ssh10:50:422
42899361,5cyclictest24828-21ntp_states10:10:242
428993612,5cyclictest26705-21fschecks_time09:30:242
42799367,20cyclictest7850irq/56-dwc_otg07:15:031
42799365,18cyclictest0-21swapper/107:34:311
42799364,20cyclictest7850irq/56-dwc_otg08:30:191
427993621,8cyclictest29462-21ssh09:34:331
42799361,12cyclictest377-21timerandwakeup10:20:391
42799361,12cyclictest377-21timerandwakeup10:20:391
21062360,2sleep20-21swapper/207:15:022
42999358,7cyclictest7950irq/56-dwc_otg_08:10:023
42999357,5cyclictest7950irq/56-dwc_otg_09:50:163
42999357,5cyclictest7950irq/56-dwc_otg_09:50:163
42999352,4cyclictest7950irq/56-dwc_otg_10:55:023
42999351,8cyclictest24772-21ls08:15:223
42899357,13cyclictest7950irq/56-dwc_otg_09:50:182
42899357,13cyclictest7950irq/56-dwc_otg_09:50:182
42899352,15cyclictest7950irq/56-dwc_otg_09:25:492
42899351,5cyclictest172450ssh10:22:452
42899351,5cyclictest172450ssh10:22:452
42799354,7cyclictest11741-21cut07:40:191
42799354,22cyclictest7850irq/56-dwc_otg10:00:181
42799353,22cyclictest7850irq/56-dwc_otg10:10:111
42799352,2cyclictest0-21swapper/109:45:231
42799351,6cyclictest8149-21aten_rbpower_po09:00:161
427993512,4cyclictest8965-21/usr/sbin/munin09:50:131
427993512,4cyclictest8965-21/usr/sbin/munin09:50:131
42799351,18cyclictest0-21swapper/108:36:201
42799351,12cyclictest24236-21/usr/sbin/munin08:15:241
310802355,19sleep131081-21/usr/sbin/munin07:05:181
42999348,5cyclictest8168-21/usr/sbin/munin09:00:163
42999345,4cyclictest0-21swapper/312:37:103
429993419,9cyclictest9806-21sh11:12:553
42999341,5cyclictest2046-21munin-run07:15:023
42999341,23cyclictest0-21swapper/308:30:223
42999341,19cyclictest0-21swapper/310:10:103
42899347,4cyclictest0-21swapper/211:41:072
42899346,8cyclictest7950irq/56-dwc_otg_12:25:282
42899341,4cyclictest8050irq/56-dwc_otg_09:45:152
428993412,6cyclictest18443-1kworker/u9:010:01:562
42799349,17cyclictest7850irq/56-dwc_otg10:25:151
42799348,14cyclictest7639-21grep07:30:031
42799344,21cyclictest7850irq/56-dwc_otg08:20:311
42799344,21cyclictest7850irq/56-dwc_otg08:20:311
42799341,7cyclictest1954-21sed08:40:311
427993411,5cyclictest4591-21cpu08:50:181
48892330,3sleep24893-21if_wlan008:50:222
42999331,6cyclictest7950irq/56-dwc_otg_08:35:313
429993313,4cyclictest20349-21ssh10:45:273
429993313,4cyclictest20349-21ssh10:45:273
42999331,23cyclictest5383-21ls08:50:293
42999331,18cyclictest0-21swapper/309:35:313
42899338,10cyclictest7850irq/56-dwc_otg08:25:032
42899338,10cyclictest7850irq/56-dwc_otg08:25:032
42899337,5cyclictest7950irq/56-dwc_otg_11:15:272
42899337,13cyclictest8050irq/56-dwc_otg_12:30:122
42899336,5cyclictest8050irq/56-dwc_otg_08:30:022
42899336,5cyclictest7950irq/56-dwc_otg_11:45:322
42899333,15cyclictest836-21ssh09:39:322
428993319,4cyclictest23474-21/usr/sbin/munin11:30:302
428993317,5cyclictest9674-21aten_rbpower_en07:35:172
428993317,5cyclictest9674-21aten_rbpower_en07:35:172
42999326,19cyclictest7850irq/56-dwc_otg08:40:243
42999321,9cyclictest0-21swapper/307:44:413
42999321,6cyclictest8050irq/56-dwc_otg_08:20:173
42999321,6cyclictest8050irq/56-dwc_otg_08:20:173
42999321,5cyclictest15189-21aten_rbpower_en07:50:173
42999321,23cyclictest0-21swapper/307:35:363
42999321,23cyclictest0-21swapper/307:35:363
42999321,11cyclictest0-21swapper/310:05:343
42899328,7cyclictest0-21swapper/212:14:422
42899326,5cyclictest8050irq/56-dwc_otg_11:50:322
42899325,6cyclictest8050irq/56-dwc_otg_10:48:082
42899325,6cyclictest8050irq/56-dwc_otg_10:48:082
42899325,15cyclictest7850irq/56-dwc_otg10:15:212
42899322,6cyclictest3045-21ntpq07:15:242
428993210,5cyclictest8050irq/56-dwc_otg_09:55:272
42799322,3cyclictest0-21swapper/109:05:311
42799322,3cyclictest0-21swapper/109:05:311
42799321,6cyclictest22188-21cron08:10:021
42999312,25cyclictest11432-21sh09:10:033
42999312,25cyclictest11432-21sh09:10:033
429993112,6cyclictest37-21rcuc/307:50:023
42899318,10cyclictest8050irq/56-dwc_otg_11:05:232
42899316,13cyclictest8050irq/56-dwc_otg_11:20:282
428993117,5cyclictest1111-21runrttasks10:34:542
42899311,5cyclictest3090650ssh11:39:252
42899308,5cyclictest8050irq/56-dwc_otg_07:50:022
42899308,11cyclictest8050irq/56-dwc_otg_08:45:242
42899306,4cyclictest8050irq/56-dwc_otg_08:00:022
42899306,11cyclictest7950irq/56-dwc_otg_07:30:022
42899305,5cyclictest7850irq/56-dwc_otg11:55:282
42899303,6cyclictest7850irq/56-dwc_otg11:00:152
42899303,4cyclictest8050irq/56-dwc_otg_07:50:202
42899302,5cyclictest15484-21/usr/sbin/munin10:40:342
42899301,6cyclictest8050irq/56-dwc_otg_07:30:242
42899301,3cyclictest0-21swapper/208:35:172
42899301,20cyclictest0-21swapper/209:43:272
42899299,4cyclictest0-21swapper/210:09:542
42899294,13cyclictest8050irq/56-dwc_otg_12:00:292
42899292,5cyclictest24248-21apt-get08:15:142
42899292,11cyclictest8050irq/56-dwc_otg_09:00:192
428992915,4cyclictest301ktimersoftd/209:07:442
428992915,4cyclictest301ktimersoftd/209:07:442
42899284,5cyclictest29-21rcuc/207:20:242
42899283,4cyclictest7850irq/56-dwc_otg08:10:242
42899272,3cyclictest7950irq/56-dwc_otg_08:30:222
428992710,5cyclictest1795-21grep08:40:292
42899261,5cyclictest0-21swapper/208:07:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional