You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 12:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Wed Mar 11, 2026 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6273991001,47cyclictest1158-21ssh22:45:120
101092940,3sleep10-21swapper/121:39:461
627399913,44cyclictest11404-21sh00:15:200
82412890,2sleep00-21swapper/022:53:280
6273998813,39cyclictest0-21swapper/021:40:080
6273998813,39cyclictest0-21swapper/021:40:080
29502880,5sleep0627399cyclictest22:09:080
271782840,1sleep10-21swapper/122:36:341
627399832,4cyclictest0-21swapper/023:45:130
163672820,2sleep216366-21ssh00:20:302
122872790,2sleep10-21swapper/119:25:221
627399782,41cyclictest24260-21latency_hist20:00:020
627399782,41cyclictest24260-21latency_hist20:00:020
627399772,37cyclictest61-21kswapd000:25:130
35192770,2sleep13520-21cat20:30:221
128502760,2sleep20-21swapper/221:41:562
128502760,2sleep20-21swapper/221:41:562
6273997465,3cyclictest7950irq/56-dwc_otg_00:10:140
6273997465,3cyclictest7950irq/56-dwc_otg_00:10:140
48472740,2sleep30-21swapper/322:10:283
31002720,5sleep1627799cyclictest22:46:371
140052700,1sleep30-21swapper/323:39:413
6273996640,14cyclictest7950irq/56-dwc_otg_21:45:140
6273996528,11cyclictest6264-21cyclictest00:30:120
205602640,2sleep2301ktimersoftd/219:50:022
6273996313,30cyclictest23244-21latency_hist21:55:020
6273996213,25cyclictest16897-21kworker/u8:020:05:010
6273996136,6cyclictest7750irq/56-dwc_otg20:10:010
113512600,5sleep3627999cyclictest22:18:533
627399592,31cyclictest7408-21sed23:30:290
6273995916,23cyclictest9072-21grep22:15:280
627399591,39cyclictest6564-21fschecks_count19:10:190
58342593,13sleep10-21swapper/119:07:471
58342593,13sleep10-21swapper/119:07:471
627799586,16cyclictest21-21rcuc/120:50:021
627399581,36cyclictest13-21rcuc/022:55:310
627399581,36cyclictest13-21rcuc/022:55:310
627399581,32cyclictest0-21swapper/019:30:130
6273995727,6cyclictest7850irq/56-dwc_otg_20:35:300
627399571,35cyclictest4074-21grep20:30:300
6273995713,24cyclictest29535-21sh23:57:470
259572570,3sleep025920-21ssh23:15:020
6273995629,7cyclictest29811-21latency_hist20:15:020
627399562,44cyclictest0-21swapper/000:08:460
627399562,32cyclictest31443-21ssh00:40:020
627399562,31cyclictest6712-21sh21:35:190
627399561,26cyclictest0-21swapper/023:40:180
627399552,29cyclictest21655-21latency_hist21:15:020
627399551,30cyclictest18288-21fschecks_time23:05:200
6273995512,28cyclictest0-21swapper/021:20:170
6273995427,6cyclictest31519-21grep21:25:280
6273995427,6cyclictest31519-21grep21:25:280
6273995426,5cyclictest16379-21ls22:25:020
6273995424,12cyclictest7750irq/56-dwc_otg20:20:010
627399541,37cyclictest0-21swapper/021:30:260
627399541,33cyclictest22210-21unixbench_multi22:30:290
627399541,33cyclictest22210-21unixbench_multi22:30:290
627399540,30cyclictest23084-21head21:15:310
6273995330,5cyclictest7850irq/56-dwc_otg_22:00:290
6273995322,14cyclictest7950irq/56-dwc_otg_20:45:290
627399527,5cyclictest0-21swapper/022:43:320
6273995223,18cyclictest4171-21uniq22:10:190
627399521,27cyclictest0-21swapper/023:00:260
6273995130,5cyclictest7850irq/56-dwc_otg_22:35:250
6273995123,3cyclictest0-21swapper/020:40:190
627399512,30cyclictest8858-21proc_pri19:15:250
6273995116,14cyclictest2422-21/usr/sbin/munin23:25:200
6273995116,14cyclictest2422-21/usr/sbin/munin23:25:200
627399511,41cyclictest0-21swapper/022:26:070
627399511,24cyclictest11377-21switchtime23:35:280
294452510,2sleep129446-21sh23:57:381
145382500,2sleep30-21swapper/300:19:223
6278994936,10cyclictest29-21rcuc/222:35:122
6277994926,5cyclictest520-21cron21:50:011
627399493,29cyclictest14080-21fschecks_time21:00:170
627399492,9cyclictest0-21swapper/020:55:130
627399492,9cyclictest0-21swapper/020:55:130
627399491,8cyclictest15332-21aten_rbpower_en00:20:160
627399491,32cyclictest0-21swapper/023:20:390
627399491,26cyclictest11845-21grep19:25:150
627399491,25cyclictest0-21swapper/021:55:260
59112492,14sleep00-21swapper/019:08:470
59112492,14sleep00-21swapper/019:08:470
627399481,25cyclictest0-21swapper/020:50:240
283882480,1sleep30-21swapper/322:38:423
627899475,19cyclictest13164-21ls22:20:232
627399472,17cyclictest0-21swapper/000:05:020
627999464,22cyclictest0-21swapper/321:20:133
627899468,24cyclictest0-21swapper/220:10:072
6278994610,5cyclictest18691-21ssh23:45:132
627399465,20cyclictest26976-21ssh23:15:260
6273994625,6cyclictest7750irq/56-dwc_otg21:05:130
627899456,25cyclictest0-21swapper/220:50:152
627899454,16cyclictest8545-21sh21:37:012
6278994517,6cyclictest13438-21latency_hist23:00:032
6278994517,6cyclictest13438-21latency_hist23:00:032
6278994511,12cyclictest5692-21ssh22:50:212
6278994510,18cyclictest6538-21ssh00:09:562
6277994521,5cyclictest2948-21sensors23:25:271
6277994521,5cyclictest2948-21sensors23:25:271
6277994510,11cyclictest21-21rcuc/100:15:171
6273994516,16cyclictest25433-21sh23:52:530
627899449,21cyclictest16146-21/usr/sbin/munin19:35:272
627899447,20cyclictest7950irq/56-dwc_otg_23:15:162
627899446,23cyclictest15406-21ssh23:01:152
6278994414,11cyclictest7950irq/56-dwc_otg_22:45:172
6278994410,21cyclictest19931-21netstat21:50:232
627399443,8cyclictest7950irq/56-dwc_otg_19:35:280
627399442,23cyclictest0-21swapper/019:20:200
627899436,22cyclictest15250-21fschecks_count21:45:172
627899435,18cyclictest6103-21chrt22:12:122
6278994323,8cyclictest7750irq/56-dwc_otg22:28:322
6278994315,8cyclictest11327-21diskstats00:15:192
6278994312,14cyclictest7950irq/56-dwc_otg_22:15:162
6278994310,13cyclictest10025-21sed20:50:022
627799437,13cyclictest23-21ksoftirqd/123:50:181
627799432,18cyclictest7850irq/56-dwc_otg_23:05:201
273822430,3sleep121-21rcuc/100:35:021
6279994224,9cyclictest381ktimersoftd/321:17:313
627899429,20cyclictest5193-21chrt23:28:532
627899429,20cyclictest5193-21chrt23:28:532
627899426,21cyclictest23971-21ssh23:50:282
627899422,25cyclictest7950irq/56-dwc_otg_22:35:022
627899422,25cyclictest7950irq/56-dwc_otg_22:35:022
6278994217,9cyclictest7750irq/56-dwc_otg19:30:172
6278994211,11cyclictest5384-21runrttasks21:33:572
627799428,5cyclictest23-21ksoftirqd/121:10:011
627799425,11cyclictest21-21rcuc/122:06:381
627399426,25cyclictest0-21swapper/020:25:130
627399425,23cyclictest505-21rs:main0
627399422,22cyclictest0-21swapper/019:51:120
6279994117,14cyclictest39-21ksoftirqd/320:50:183
627899415,20cyclictest17501-21/usr/sbin/munin21:10:312
6278994119,8cyclictest7950irq/56-dwc_otg_22:06:452
6277994117,4cyclictest5373-21aten_rbpower_en22:50:161
6277994114,4cyclictest14755-21latency_hist21:45:031
6277994114,4cyclictest14755-21latency_hist21:45:031
627899407,20cyclictest0-21swapper/221:05:152
627899406,20cyclictest13063-21taskset19:27:122
627899406,13cyclictest1364-21df20:25:152
6273994018,14cyclictest0-21swapper/019:42:210
627899397,12cyclictest8066-21ssh23:31:342
627899397,12cyclictest24891-21sed20:00:222
627899393,23cyclictest21192-21ssh00:26:242
627799393,4cyclictest7950irq/56-dwc_otg_21:20:111
6277993924,10cyclictest221ktimersoftd/120:35:151
627399391,26cyclictest0-21swapper/020:24:570
238982391,3sleep10-21swapper/123:11:301
6279993819,4cyclictest39-21ksoftirqd/323:15:023
627999381,26cyclictest27150-21timerwakeupswit20:05:293
627999381,24cyclictest0-21swapper/320:00:163
627899389,19cyclictest1647950kworker/u9:223:42:132
627899388,11cyclictest0-21swapper/219:55:122
627899388,11cyclictest0-21swapper/219:55:122
627899387,12cyclictest24404-21ssh21:17:462
6278993818,10cyclictest7950irq/56-dwc_otg_23:12:302
6278993817,7cyclictest7950irq/56-dwc_otg_23:36:452
627799386,5cyclictest23-21ksoftirqd/123:49:521
627799384,19cyclictest23-21ksoftirqd/100:36:541
627799383,14cyclictest221ktimersoftd/123:05:021
627799382,5cyclictest7950irq/56-dwc_otg_20:15:141
6277993819,8cyclictest7850irq/56-dwc_otg_21:55:451
6277993816,8cyclictest221ktimersoftd/122:28:501
6277993815,5cyclictest7850irq/56-dwc_otg_00:15:021
6277993815,5cyclictest7850irq/56-dwc_otg_00:15:021
6277993813,12cyclictest0-21swapper/122:31:501
6277993813,12cyclictest0-21swapper/122:31:501
627799381,23cyclictest0-21swapper/119:15:141
6277993812,15cyclictest1055150wget23:35:171
165112380,1sleep30-21swapper/323:41:423
68012370,1sleep10-21swapper/122:13:241
627999375,18cyclictest7950irq/56-dwc_otg_22:35:023
627999375,18cyclictest7950irq/56-dwc_otg_22:35:023
627999371,23cyclictest10839-21ssh00:15:033
627999371,23cyclictest10839-21ssh00:15:033
627899377,4cyclictest0-21swapper/223:05:312
627899374,13cyclictest28873-21awk00:35:312
627899373,16cyclictest747-21ssh22:44:552
627899372,13cyclictest30961-21cut23:20:222
6278993715,10cyclictest301ktimersoftd/222:01:372
6278993710,16cyclictest1698-21ssh00:03:402
627899370,19cyclictest28530-21ssh23:55:592
627799377,6cyclictest21-21rcuc/122:20:201
627799377,23cyclictest0-21swapper/120:10:071
627799375,10cyclictest23-21ksoftirqd/119:30:161
627799372,12cyclictest23-21ksoftirqd/122:56:081
627799372,12cyclictest23-21ksoftirqd/122:56:081
6277993720,8cyclictest23247-21sh21:15:461
6277993717,4cyclictest16910-21latency_hist19:40:021
6277993715,4cyclictest7950irq/56-dwc_otg_20:00:171
627799371,20cyclictest31654-21munin-run22:05:011
627999369,17cyclictest39-21ksoftirqd/322:45:143
627999367,25cyclictest0-21swapper/321:45:143
627999366,6cyclictest39-21ksoftirqd/319:35:173
627999364,23cyclictest7950irq/56-dwc_otg_23:35:013
627999364,17cyclictest7950irq/56-dwc_otg_23:00:023
627999364,17cyclictest7950irq/56-dwc_otg_23:00:023
627999362,3cyclictest7950irq/56-dwc_otg_21:30:203
6279993611,12cyclictest7750irq/56-dwc_otg23:20:183
627899367,18cyclictest0-21swapper/221:55:152
627899366,12cyclictest26011-21apt-get21:20:112
627899363,17cyclictest25687-21kworker/u8:021:25:092
627899363,17cyclictest25687-21kworker/u8:021:25:092
627899361,15cyclictest7950irq/56-dwc_otg_00:12:442
627899361,15cyclictest7950irq/56-dwc_otg_00:12:442
627799367,24cyclictest0-21swapper/120:50:151
6277993624,4cyclictest221ktimersoftd/120:40:271
6277993617,8cyclictest7850irq/56-dwc_otg_19:46:241
6277993616,12cyclictest221ktimersoftd/123:21:071
6277993612,4cyclictest0-21swapper/121:30:161
6277993610,14cyclictest0-21swapper/100:01:331
627999352,16cyclictest0-21swapper/319:51:533
627999351,26cyclictest12885-21ls20:55:273
627999351,26cyclictest12885-21ls20:55:273
627999351,21cyclictest0-21swapper/300:35:143
627899355,17cyclictest7950irq/56-dwc_otg_20:05:552
627899352,8cyclictest0-21swapper/200:32:372
627899351,20cyclictest0-21swapper/220:33:512
627799359,5cyclictest19504-21fschecks_count21:50:191
627799353,16cyclictest17462-21fschecks_time19:40:191
627799352,15cyclictest0-21swapper/123:35:021
627799351,4cyclictest7950irq/56-dwc_otg_20:20:141
6277993514,4cyclictest6363-21aten_rbpower_en19:10:161
6277993513,10cyclictest221ktimersoftd/122:15:161
6277993511,10cyclictest7950irq/56-dwc_otg_23:15:221
6277993510,16cyclictest221ktimersoftd/100:20:561
175682350,4sleep117573-21aten_rbpower_cu21:10:151
627999348,7cyclictest37-21rcuc/322:50:143
627999346,13cyclictest23066-21diskstats23:50:173
627999345,3cyclictest0-21swapper/321:57:293
627999344,7cyclictest7950irq/56-dwc_otg_23:15:193
627999344,20cyclictest39-21ksoftirqd/323:56:143
627999344,18cyclictest39-21ksoftirqd/321:15:023
627999342,8cyclictest7950irq/56-dwc_otg_20:35:143
627899345,6cyclictest0-21swapper/220:20:032
627799342,18cyclictest221ktimersoftd/123:42:081
6277993417,11cyclictest221ktimersoftd/100:07:121
627799341,13cyclictest31226-21switchtime21:25:261
627799341,13cyclictest31226-21switchtime21:25:261
627999338,5cyclictest7950irq/56-dwc_otg_22:20:153
627999338,13cyclictest39-21ksoftirqd/320:25:273
627999338,13cyclictest0-21swapper/320:50:023
627999332,4cyclictest0-21swapper/300:30:213
627999332,18cyclictest0-21swapper/322:25:223
6279993320,5cyclictest381ktimersoftd/321:44:533
6279993320,5cyclictest381ktimersoftd/321:44:533
6279993319,7cyclictest870150ssh21:37:183
6279993310,14cyclictest7750irq/56-dwc_otg00:09:363
6279993310,14cyclictest381ktimersoftd/321:50:233
627999330,23cyclictest0-21swapper/319:55:143
627999330,23cyclictest0-21swapper/319:55:143
627899336,15cyclictest0-21swapper/220:42:122
627899334,13cyclictest6166-21taskset20:38:002
627899332,18cyclictest9355-21chrt19:16:132
627899331,5cyclictest0-21swapper/220:15:162
627799335,12cyclictest21-21rcuc/120:05:461
627799333,7cyclictest0-21swapper/122:40:281
627799331,13cyclictest0-21swapper/100:29:041
627999328,16cyclictest14332-21missed_timers23:00:223
627999327,8cyclictest7850irq/56-dwc_otg_00:20:293
627999325,13cyclictest7950irq/56-dwc_otg_21:05:223
627999324,19cyclictest18704-21ssh23:45:153
627999323,17cyclictest0-21swapper/319:45:023
6279993226,4cyclictest39-21ksoftirqd/323:05:173
6279993226,4cyclictest381ktimersoftd/321:00:113
627999322,13cyclictest7950irq/56-dwc_otg_19:20:023
627999321,22cyclictest1066-21ls20:25:023
627999321,15cyclictest7950irq/56-dwc_otg_00:04:033
627999321,15cyclictest3240-21grep23:25:303
627999321,15cyclictest3240-21grep23:25:303
627899321,6cyclictest7850irq/56-dwc_otg_19:40:122
627799327,12cyclictest21-21rcuc/120:25:281
627799325,3cyclictest7950irq/56-dwc_otg_19:55:121
627799325,3cyclictest7950irq/56-dwc_otg_19:55:121
627799323,11cyclictest23-21ksoftirqd/119:55:031
627999317,15cyclictest0-21swapper/320:15:163
627999316,5cyclictest7950irq/56-dwc_otg_19:15:013
627999316,16cyclictest0-21swapper/319:20:163
627999315,22cyclictest7950irq/56-dwc_otg_20:40:143
627999314,17cyclictest39-21ksoftirqd/321:28:273
627999314,17cyclictest39-21ksoftirqd/321:28:273
6279993114,4cyclictest29516-21aten_rbpower_cu22:40:143
627999311,19cyclictest20662-21cat19:50:023
6279993110,4cyclictest0-21swapper/322:05:123
627899316,4cyclictest7950irq/56-dwc_otg_19:13:292
6277993117,5cyclictest10122-21cpuspeed19:20:171
627999306,14cyclictest37-21rcuc/319:30:173
627999304,14cyclictest37-21rcuc/322:04:193
627999301,4cyclictest20325-21smartpower00:25:283
627999297,4cyclictest11731-21/usr/sbin/munin19:25:123
6279992922,5cyclictest0-21swapper/320:30:163
627999292,18cyclictest0-21swapper/320:14:093
627899299,4cyclictest15256-21seq21:03:502
627899296,15cyclictest0-21swapper/219:50:152
627799294,12cyclictest13993-21cpuspeed21:00:151
59662294,16sleep20-21swapper/219:09:292
59662294,16sleep20-21swapper/219:09:292
627899287,18cyclictest0-21swapper/220:55:152
627899287,18cyclictest0-21swapper/220:55:152
627899281,17cyclictest10254-21uniq19:20:192
627799284,7cyclictest12257-21fschecks_count20:55:191
627799284,7cyclictest12257-21fschecks_count20:55:191
57882262,17sleep30-21swapper/319:07:123
57882262,17sleep30-21swapper/319:07:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional