You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 09:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Thu Sep 18, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
44292940,2sleep30-21swapper/321:52:583
26322910,3sleep337-21rcuc/322:32:033
27026999027,6cyclictest7950irq/56-dwc_otg_00:40:020
2702699885,39cyclictest0-21swapper/019:15:170
2702699881,58cyclictest0-21swapper/021:10:110
27026998610,32cyclictest4616-21latency_hist21:10:030
292752850,2sleep10-21swapper/121:41:001
2702699846,15cyclictest0-21swapper/023:30:140
2702699842,30cyclictest0-21swapper/000:20:200
273132830,4sleep027315-21fschecks_count19:10:200
27026998021,6cyclictest30942-21fschecks_count19:20:200
2702699801,42cyclictest0-21swapper/022:38:480
291002780,6sleep337-21rcuc/323:05:323
291002780,6sleep337-21rcuc/323:05:323
229112750,2sleep20-21swapper/222:58:392
2702699747,51cyclictest0-21swapper/022:55:180
27027997113,12cyclictest0-21swapper/121:35:161
111042710,5sleep02702699cyclictest23:25:220
2702699705,48cyclictest0-21swapper/020:30:290
2702699701,59cyclictest20713-21missed_timers20:20:230
318972690,2sleep10-21swapper/122:28:261
27028996614,10cyclictest8050irq/56-dwc_otg_23:55:152
2702799662,17cyclictest7850irq/56-dwc_otg21:30:161
2702699661,48cyclictest31345-21grep23:50:320
2702699652,47cyclictest0-21swapper/022:08:220
27026996510,51cyclictest0-21swapper/023:00:270
2702699643,58cyclictest101ktimersoftd/022:30:170
2702699642,48cyclictest0-21swapper/000:30:270
2702699641,51cyclictest20115-21aten_rbpower_cu21:30:180
2702699632,48cyclictest0-21swapper/021:53:580
2702899623,25cyclictest0-21swapper/222:00:132
2702699611,45cyclictest19889-21sort20:20:030
27026996111,25cyclictest0-21swapper/021:25:170
2702799609,18cyclictest8050irq/56-dwc_otg_20:50:201
2702699601,47cyclictest500-21rs:main0
2702699601,43cyclictest12091-21grep19:55:290
27026995915,37cyclictest101ktimersoftd/022:50:140
2702699582,45cyclictest0-21swapper/000:00:400
2702699582,43cyclictest0-21swapper/021:45:260
2702699582,40cyclictest0-21swapper/019:50:030
2702699581,46cyclictest0-21swapper/019:26:320
27026995730,13cyclictest8050irq/56-dwc_otg_21:40:260
2702699572,43cyclictest0-21swapper/019:40:160
2702699572,33cyclictest31405-21fschecks_time20:50:240
2702699561,44cyclictest28120-21ntp_kernel_pll_20:40:240
2702699561,44cyclictest28120-21ntp_kernel_pll_20:40:240
2702699561,42cyclictest0-21swapper/020:55:210
2702699561,39cyclictest4449-21ssh23:15:460
242702560,1sleep30-21swapper/323:42:003
27029995514,26cyclictest0-21swapper/322:35:173
2702799557,16cyclictest0-21swapper/123:05:171
2702799557,16cyclictest0-21swapper/123:05:171
2702699559,25cyclictest1111-21runrttasks22:46:360
2702699552,40cyclictest26555-21unixbench_singl22:20:340
2702699552,39cyclictest4764-21latency_hist00:00:020
2702699552,39cyclictest0-21swapper/023:40:240
2702699551,38cyclictest0-21swapper/021:22:370
2702699551,28cyclictest0-21swapper/020:05:170
2702799544,7cyclictest7950irq/56-dwc_otg_22:00:151
2702699541,38cyclictest0-21swapper/020:35:270
224732540,1sleep00-21swapper/020:25:230
27028995318,9cyclictest0-21swapper/221:35:162
27026995325,3cyclictest0-21swapper/023:45:300
2702699532,12cyclictest0-21swapper/020:15:020
2702699531,26cyclictest0-21swapper/000:25:240
42592520,4sleep3381ktimersoftd/323:15:313
2702999526,27cyclictest0-21swapper/319:10:163
2702799526,6cyclictest0-21swapper/122:40:151
27027995213,25cyclictest0-21swapper/100:25:161
2702699527,25cyclictest0-21swapper/000:15:230
27026995232,11cyclictest7950irq/56-dwc_otg_22:05:020
2702699521,43cyclictest0-21swapper/023:20:140
2702699521,42cyclictest0-21swapper/021:19:380
2702699521,41cyclictest0-21swapper/022:40:220
27026995137,5cyclictest17495-21/usr/sbin/munin22:10:260
27026995127,16cyclictest0-21swapper/019:38:140
2702699512,39cyclictest2732-21latency_hist23:15:020
2702699512,38cyclictest0-21swapper/000:06:360
2702699511,47cyclictest0-21swapper/019:30:300
2702699511,38cyclictest29287-21aten_rbpower_vo22:25:170
27027995024,22cyclictest23973-21ssh23:00:181
27027995011,18cyclictest8050irq/56-dwc_otg_21:15:281
2702699504,33cyclictest0-21swapper/020:00:120
27026995034,5cyclictest7850irq/56-dwc_otg22:15:270
2702699501,39cyclictest11459-21tr00:10:190
2702699501,39cyclictest11459-21tr00:10:190
2702799498,26cyclictest0-21swapper/120:25:161
2702699491,4cyclictest0-21swapper/023:35:210
2702699491,4cyclictest0-21swapper/023:35:210
2702699491,40cyclictest0-21swapper/021:36:560
2702799489,21cyclictest14359-21/usr/sbin/munin22:05:261
27026994832,4cyclictest7950irq/56-dwc_otg_20:45:250
2702999477,23cyclictest0-21swapper/321:55:133
2702999476,24cyclictest14485-21sh23:30:143
2702899471,6cyclictest31451-21ls21:45:032
2702799477,26cyclictest0-21swapper/120:15:171
27027994727,4cyclictest8050irq/56-dwc_otg_00:20:341
2702799465,21cyclictest16263-21aten_rbpower_vo22:50:171
27027994628,4cyclictest7850irq/56-dwc_otg19:55:201
27027994610,12cyclictest0-21swapper/123:55:161
2702699463,33cyclictest0-21swapper/023:05:060
2702699463,33cyclictest0-21swapper/023:05:060
27029994533,8cyclictest381ktimersoftd/323:35:153
27029994533,8cyclictest381ktimersoftd/323:35:153
2702899453,7cyclictest7850irq/56-dwc_otg21:15:152
27028994525,8cyclictest20064-21ssh21:30:162
27028994510,5cyclictest5398-21fschecks_time00:00:182
2702799457,25cyclictest0-21swapper/123:50:171
27027994515,18cyclictest7950irq/56-dwc_otg_22:13:571
27027994513,13cyclictest8050irq/56-dwc_otg_00:06:081
27027994512,21cyclictest0-21swapper/119:35:161
2702699451,32cyclictest0-21swapper/021:00:200
2702799448,22cyclictest0-21swapper/120:10:161
2702799447,22cyclictest3522-21ssh22:33:531
27027994424,9cyclictest7850irq/56-dwc_otg00:00:261
27027994417,5cyclictest7850irq/56-dwc_otg21:25:191
27027994415,10cyclictest1012-21wc19:25:261
2702999436,23cyclictest0-21swapper/319:40:163
2702799437,24cyclictest0-21swapper/123:15:151
2702799437,24cyclictest0-21swapper/121:45:141
2702799436,18cyclictest0-21swapper/123:40:171
2702799435,17cyclictest0-21swapper/122:59:101
27027994324,9cyclictest7850irq/56-dwc_otg00:14:501
27027994324,9cyclictest7850irq/56-dwc_otg00:14:501
27027994317,11cyclictest7850irq/56-dwc_otg23:29:591
27027994313,11cyclictest8050irq/56-dwc_otg_23:35:171
27027994313,11cyclictest8050irq/56-dwc_otg_23:35:171
2702799428,20cyclictest29304-21ssh23:49:121
2702799427,21cyclictest6385-21ssh22:36:391
2702799427,19cyclictest16843-21ssh00:17:181
2702799421,9cyclictest0-21swapper/120:40:151
2702799421,9cyclictest0-21swapper/120:40:151
27027994217,12cyclictest8050irq/56-dwc_otg_22:17:071
27027994211,14cyclictest17566-21ssh23:34:101
2702699422,31cyclictest0-21swapper/019:50:240
241972420,3sleep2301ktimersoftd/223:41:502
285432410,2sleep30-21swapper/323:47:433
2702899417,25cyclictest0-21swapper/219:20:162
2702899413,4cyclictest7850irq/56-dwc_otg21:55:102
2702799418,11cyclictest31909-21ssh23:10:201
2702799411,7cyclictest0-21swapper/100:30:171
27027994115,8cyclictest7850irq/56-dwc_otg20:58:331
2702799411,26cyclictest14761-21ssh22:47:541
2702999407,25cyclictest8050irq/56-dwc_otg_23:20:173
2702999406,15cyclictest8050irq/56-dwc_otg_00:30:333
2702899405,8cyclictest7950irq/56-dwc_otg_22:40:022
2702899404,10cyclictest0-21swapper/220:25:162
2702899401,6cyclictest13645-21ssh22:05:172
2702799404,8cyclictest0-21swapper/121:10:111
27027994027,5cyclictest221ktimersoftd/119:43:231
2702799402,15cyclictest0-21swapper/120:45:141
27027994010,3cyclictest0-21swapper/120:30:171
2702999395,24cyclictest23947-21ssh21:35:183
27029993917,5cyclictest23041-21sh22:58:553
2702799396,4cyclictest7598-21ntpq23:20:271
27027993916,12cyclictest8050irq/56-dwc_otg_22:25:021
27029993819,5cyclictest11402-21acpi00:10:163
27029993819,5cyclictest11402-21acpi00:10:163
2702899389,6cyclictest7950irq/56-dwc_otg_23:10:222
2702899386,25cyclictest0-21swapper/219:55:082
27028993822,5cyclictest20385-21ssh23:37:002
27028993822,5cyclictest20385-21ssh23:37:002
27028993810,19cyclictest31-21ksoftirqd/222:20:162
2702799389,12cyclictest3860-21sh21:51:491
2702799388,15cyclictest20297-21cat20:20:171
2702799385,17cyclictest31662-21/usr/sbin/munin19:20:291
2702799384,20cyclictest0-21swapper/119:50:151
2702799381,22cyclictest29819-21ssh00:37:151
27027993811,5cyclictest0-21swapper/121:20:141
27029993719,10cyclictest32267-21seq20:50:433
27029993718,12cyclictest14989-21ssh21:23:083
27028993730,3cyclictest0-21swapper/200:10:172
27028993730,3cyclictest0-21swapper/200:10:172
2702899371,24cyclictest0-21swapper/220:30:152
2702899371,20cyclictest16296-21ls20:10:032
2702799378,15cyclictest30251-21seq19:17:491
2702799377,15cyclictest0-21swapper/121:57:051
2702799373,20cyclictest3315-21chrt19:34:001
2702799371,22cyclictest28403-21taskset19:12:281
146492371,2sleep30-21swapper/320:05:183
70522360,1sleep30-21swapper/300:02:003
2702999369,3cyclictest0-21swapper/300:21:053
2702999368,17cyclictest0-21swapper/320:45:133
2702999365,9cyclictest0-21swapper/321:30:173
27029993630,4cyclictest29716-21ls19:15:303
2702899367,23cyclictest0-21swapper/221:20:152
2702899363,16cyclictest31514-21grep23:10:032
2702899363,16cyclictest31514-21grep23:10:032
27028993616,6cyclictest3264-21sh23:15:182
27028993616,4cyclictest18645-21if_err_eth000:20:252
27028993614,4cyclictest17566-21aten_rbpower_en22:10:172
2702899361,24cyclictest26651-21timerwakeupswit20:35:302
27029993519,9cyclictest26150-21ssh23:02:203
27029993512,5cyclictest8826-21latency_hist19:50:033
2702999351,23cyclictest14643-21ssh22:05:313
27029993510,6cyclictest0-21swapper/321:25:313
2702899357,24cyclictest0-21swapper/219:45:162
2702899357,23cyclictest0-21swapper/220:20:162
2702899357,22cyclictest0-21swapper/223:45:172
2702899357,22cyclictest0-21swapper/200:30:182
2702899356,5cyclictest7850irq/56-dwc_otg00:40:022
2702899356,24cyclictest0-21swapper/219:25:132
2702899356,21cyclictest0-21swapper/221:10:132
2702899351,22cyclictest0-21swapper/223:25:172
2702999347,14cyclictest0-21swapper/320:30:083
2702999345,20cyclictest9731-21ssh22:00:163
2702999344,6cyclictest8050irq/56-dwc_otg_20:00:283
27029993413,4cyclictest381ktimersoftd/321:45:263
2702899347,14cyclictest0-21swapper/221:00:162
2702899347,13cyclictest0-21swapper/222:15:372
2702899344,12cyclictest7950irq/56-dwc_otg_19:35:222
2702899342,5cyclictest7850irq/56-dwc_otg22:30:322
2702899342,12cyclictest7950irq/56-dwc_otg_00:25:232
27028993411,5cyclictest17929-21idleruntime-cro00:20:022
2702799347,8cyclictest7850irq/56-dwc_otg20:35:201
2702799341,5cyclictest16155-21latency_hist20:10:021
2702999336,7cyclictest7850irq/56-dwc_otg22:45:313
2702999335,19cyclictest31710-21ssh23:10:183
2702999332,4cyclictest7950irq/56-dwc_otg_21:00:243
27029993311,12cyclictest0-21swapper/322:53:453
2702899335,4cyclictest7950irq/56-dwc_otg_22:25:152
2702899332,14cyclictest0-21swapper/200:05:172
2702899331,6cyclictest31588-21/usr/sbin/munin21:45:272
27028993316,4cyclictest14762-21uniq23:30:172
2702999328,10cyclictest7950irq/56-dwc_otg_22:20:163
2702999327,15cyclictest0-21swapper/322:15:163
2702999323,3cyclictest8050irq/56-dwc_otg_23:51:593
2702999321,20cyclictest7850irq/56-dwc_otg22:26:143
2702999321,15cyclictest0-21swapper/300:25:573
27029993210,5cyclictest8050irq/56-dwc_otg_21:10:023
2702899329,5cyclictest19992-21latency_hist20:20:032
2702899327,6cyclictest7950irq/56-dwc_otg_22:45:202
2702899327,22cyclictest8429-21ssh22:40:172
2702899325,14cyclictest8050irq/56-dwc_otg_21:25:322
2702899324,12cyclictest8050irq/56-dwc_otg_20:45:032
2702899324,12cyclictest8050irq/56-dwc_otg_20:45:032
2702899322,2cyclictest0-21swapper/223:20:252
2702799328,7cyclictest0-21swapper/121:04:031
2702799328,4cyclictest8050irq/56-dwc_otg_21:07:131
2702799321,6cyclictest12811-21aten_rbpower_cu20:00:141
2702799321,21cyclictest0-21swapper/119:45:161
259502324,16sleep00-21swapper/019:05:270
2702999318,7cyclictest25590-1kworker/u9:223:25:203
2702999318,4cyclictest0-21swapper/321:40:223
2702999317,5cyclictest0-21swapper/321:11:203
2702999316,15cyclictest0-21swapper/321:15:173
27029993114,4cyclictest20816-21sh22:14:323
2702899316,6cyclictest8050irq/56-dwc_otg_23:03:042
2702899313,18cyclictest0-21swapper/219:42:052
2702899312,7cyclictest8050irq/56-dwc_otg_19:20:022
2702899311,24cyclictest16471-21grep22:50:202
27028993111,12cyclictest301ktimersoftd/221:05:472
27028993110,6cyclictest1487-21grep20:55:302
2702999308,4cyclictest0-21swapper/300:35:273
2702999307,9cyclictest7950irq/56-dwc_otg_20:25:023
2702999306,6cyclictest7950irq/56-dwc_otg_19:25:133
2702999305,6cyclictest8050irq/56-dwc_otg_20:40:163
2702999305,6cyclictest8050irq/56-dwc_otg_20:40:163
2702999304,4cyclictest7950irq/56-dwc_otg_22:40:143
2702999302,6cyclictest7850irq/56-dwc_otg20:30:023
27029993010,5cyclictest26397-21processes20:35:273
2702899303,4cyclictest9071-21apt-get19:50:152
2702899302,5cyclictest0-21swapper/220:10:232
27028993024,4cyclictest1968-21aten_rbpower_vo19:30:182
2702899302,3cyclictest0-21swapper/223:50:252
2702899301,9cyclictest32130-21grep20:50:332
2702899301,20cyclictest0-21swapper/221:52:482
2702999296,14cyclictest0-21swapper/319:20:163
2702999295,5cyclictest10634-21munin-run19:55:023
2702999295,4cyclictest0-21swapper/300:18:583
2702899293,13cyclictest7950irq/56-dwc_otg_20:45:222
2702999284,6cyclictest0-21swapper/323:55:343
2702999282,13cyclictest0-21swapper/320:15:433
2702999281,5cyclictest16381-21/usr/sbin/munin20:10:163
2702999281,4cyclictest7950irq/56-dwc_otg_19:35:293
2702999281,18cyclictest11887-21timerandwakeup19:55:283
2702899283,3cyclictest7850irq/56-dwc_otg20:00:292
2702899282,6cyclictest7850irq/56-dwc_otg19:10:172
266302286,14sleep10-21swapper/119:08:241
27029992717,7cyclictest2098-21fschecks_count19:30:203
27029992711,4cyclictest11031-21seq00:09:383
266542273,15sleep20-21swapper/219:08:412
265962264,15sleep30-21swapper/319:07:573
2702999251,4cyclictest8050irq/56-dwc_otg_20:55:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional