You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-22 - 03:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Mon Jun 22, 2026 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2714325950,3sleep10-21swapper/119:05:211
2724224520,6sleep30-21swapper/319:05:233
2883199935,29cyclictest0-21swapper/023:40:180
257652850,5sleep22883399cyclictest21:30:442
28833998018,13cyclictest7950irq/56-dwc_otg_00:35:162
2883499774,14cyclictest2418-21ssh23:40:183
17152770,6sleep12883299cyclictest22:20:271
17152770,6sleep12883299cyclictest22:20:271
156702770,7sleep32883499cyclictest23:16:093
156702770,7sleep32883499cyclictest23:16:093
28833997423,10cyclictest8050irq/56-dwc_otg_22:40:172
28831997115,43cyclictest11610-21latency_hist21:15:060
323722690,1sleep30-21swapper/319:20:063
28831996813,35cyclictest0-21swapper/022:20:340
28831996813,35cyclictest0-21swapper/022:20:340
39042670,2sleep30-21swapper/321:44:543
39042670,2sleep30-21swapper/321:44:543
270442677,15sleep00-21swapper/019:05:210
2883199662,7cyclictest6914-21grep21:05:360
2883199657,51cyclictest9650irq/80-mmc023:15:070
28831996511,37cyclictest0-21swapper/021:50:250
308732640,2sleep30-21swapper/322:55:323
285262643,14sleep20-21swapper/219:09:402
321052630,2sleep10-21swapper/122:57:071
28832996311,7cyclictest7950irq/56-dwc_otg_00:30:201
28831996313,30cyclictest18671-21timerwakeupswit22:40:340
28831996217,25cyclictest4977-21grep21:00:330
28834996122,9cyclictest14405-21ssh21:17:403
2883299619,19cyclictest7850irq/56-dwc_otg22:50:201
2883199612,29cyclictest0-21swapper/019:40:170
2883199612,29cyclictest0-21swapper/019:40:170
2883299602,14cyclictest0-21swapper/123:05:171
28831996020,26cyclictest10711-21latency_hist19:50:060
28831995914,20cyclictest101ktimersoftd/019:25:300
2883199591,31cyclictest17562-21sed20:05:350
2883199591,31cyclictest17562-21sed20:05:350
2883199581,10cyclictest26567-21ssh23:30:170
2883199577,28cyclictest0-21swapper/020:00:340
2883199571,7cyclictest15773-21latency_hist21:20:070
28831995716,29cyclictest0-21swapper/021:25:310
28831995711,23cyclictest0-21swapper/023:50:280
28833995610,28cyclictest25823-21ssh22:50:202
28831995628,5cyclictest13874-21ssh21:55:340
2883199561,44cyclictest12679-21grep00:30:350
2883199561,41cyclictest31108-21cat00:15:070
2883199559,28cyclictest4329-21ntp_kernel_pll_19:30:270
28831995523,8cyclictest19310-21grep20:10:340
2883199551,5cyclictest2068-21cron23:40:060
2883199551,46cyclictest0-21swapper/022:33:420
2883299546,30cyclictest0-21swapper/119:15:201
2883299542,14cyclictest7850irq/56-dwc_otg22:00:181
2883199541,26cyclictest1385-21grep20:50:340
158652540,2sleep30-21swapper/321:59:033
2883299537,30cyclictest0-21swapper/120:35:171
28831995327,5cyclictest29979-21/usr/sbin/munin22:55:200
28831995327,15cyclictest7850irq/56-dwc_otg20:30:300
2883199532,39cyclictest4341-21unixbench_singl00:20:340
2883199532,30cyclictest12529-21cron19:55:060
28831995315,23cyclictest101ktimersoftd/023:15:190
28831995315,23cyclictest101ktimersoftd/023:15:190
2883199531,32cyclictest3172-21unixbench_singl20:55:340
2883199531,32cyclictest3172-21unixbench_singl20:55:340
28831995235,11cyclictest0-21swapper/023:45:190
28831995235,11cyclictest0-21swapper/023:45:190
28831995229,5cyclictest7950irq/56-dwc_otg_20:40:320
28831995229,4cyclictest8050irq/56-dwc_otg_23:20:330
2883199522,35cyclictest0-21swapper/021:37:000
28831995220,8cyclictest25472-21rm22:10:240
2883199521,33cyclictest31999-21grep20:45:320
28831995213,10cyclictest0-21swapper/023:00:200
2883199521,28cyclictest6995-21grep23:05:340
28831995212,22cyclictest29865-21/usr/sbin/munin22:15:270
2883499515,34cyclictest8050irq/56-dwc_otg_21:45:173
2883399513,9cyclictest10522-21ssh00:29:102
2883399511,6cyclictest0-21swapper/220:05:192
2883399511,6cyclictest0-21swapper/220:05:192
28833995112,13cyclictest8050irq/56-dwc_otg_23:25:182
28832995132,4cyclictest8050irq/56-dwc_otg_21:55:181
2883299512,10cyclictest0-21swapper/120:50:171
28831995131,4cyclictest7950irq/56-dwc_otg_20:00:070
28831995130,4cyclictest7850irq/56-dwc_otg22:35:260
28831995127,11cyclictest7850irq/56-dwc_otg00:05:230
2883199512,29cyclictest0-21swapper/019:20:060
2883199511,4cyclictest8946-21munin-run22:30:060
2883199511,22cyclictest26245-21idleruntime-cro23:30:060
28831995111,5cyclictest16004-21grep23:55:340
28831995110,20cyclictest22867-21grep22:45:350
28831995110,20cyclictest22867-21grep22:45:350
2883399502,13cyclictest0-21swapper/200:30:202
2883299502,11cyclictest0-21swapper/120:05:181
2883299502,11cyclictest0-21swapper/120:05:181
28831995014,5cyclictest22755-21latency_hist00:05:060
28831995014,22cyclictest25819-21sh22:50:200
28831995013,24cyclictest101ktimersoftd/022:05:350
28831995013,17cyclictest0-21swapper/021:30:350
2883199501,29cyclictest416-21ls21:40:210
2883199501,29cyclictest416-21ls21:40:210
2883399482,5cyclictest8050irq/56-dwc_otg_19:20:182
28833994811,8cyclictest7850irq/56-dwc_otg23:05:182
2883299487,6cyclictest7850irq/56-dwc_otg00:29:101
2883299487,22cyclictest7672-21ssh21:10:191
2883299486,25cyclictest0-21swapper/121:05:121
2883299481,10cyclictest0-21swapper/120:45:151
28831994826,7cyclictest22548-21/usr/sbin/munin20:20:300
28831994826,4cyclictest7950irq/56-dwc_otg_21:49:550
28831994816,20cyclictest0-21swapper/022:00:080
2883499477,25cyclictest0-21swapper/300:15:183
2883499476,10cyclictest11264-21ls23:50:263
2883399477,24cyclictest0-21swapper/221:05:112
2883399471,6cyclictest0-21swapper/220:50:182
2883299472,7cyclictest1628750ssh00:35:281
2883199471,4cyclictest7492-21diskstats00:25:240
2883399467,24cyclictest0-21swapper/220:35:172
2883299467,27cyclictest0-21swapper/123:10:191
28831994625,4cyclictest0-21swapper/020:35:320
2883199461,25cyclictest5548-21aten_rbpower_po19:35:210
2883499457,23cyclictest0-21swapper/320:40:183
28834994518,6cyclictest20933-21sh22:05:103
2883399457,25cyclictest0-21swapper/223:10:192
2883399457,24cyclictest0-21swapper/219:15:202
2883299457,26cyclictest0-21swapper/119:45:201
2883299453,6cyclictest7950irq/56-dwc_otg_20:10:181
28831994521,6cyclictest21097-21grep20:15:330
2883199451,27cyclictest19935-21rm21:25:060
2883199451,25cyclictest24120-21latency20:25:280
2883499447,24cyclictest0-21swapper/323:35:203
28834994429,4cyclictest7950irq/56-dwc_otg_21:35:373
2883399446,5cyclictest8050irq/56-dwc_otg_21:20:252
2883399446,24cyclictest0-21swapper/219:45:202
28833994418,16cyclictest31-21ksoftirqd/221:55:182
2883199441,26cyclictest19330-21ssh00:40:060
2883199441,24cyclictest1225-21runrttasks00:17:210
17032440,19sleep18050irq/56-dwc_otg_23:00:211
2883299437,25cyclictest0-21swapper/123:30:201
2883299437,23cyclictest0-21swapper/120:20:191
2883199439,5cyclictest7850irq/56-dwc_otg19:10:210
2883499427,21cyclictest0-21swapper/319:50:113
2883499427,20cyclictest0-21swapper/319:25:193
28833994217,16cyclictest0-21swapper/223:55:192
2883299426,22cyclictest27254-21ssh00:10:201
2883299426,22cyclictest0-21swapper/119:50:111
28831994212,14cyclictest75050snmpd19:20:400
52782411,3sleep225814-1kworker/u9:200:22:122
2883399417,22cyclictest0-21swapper/223:30:202
2883399412,5cyclictest22756-21apt00:05:062
2883499404,21cyclictest0-21swapper/322:40:203
2883399401,8cyclictest8403-21grep19:40:342
2883399401,8cyclictest8403-21grep19:40:342
2883399401,4cyclictest0-21swapper/220:55:192
2883399401,4cyclictest0-21swapper/220:55:192
2883299405,23cyclictest0-21swapper/120:25:221
2883299404,7cyclictest23-21ksoftirqd/122:15:211
2883299401,7cyclictest14922-21hostname23:55:201
49972390,2sleep20-21swapper/222:25:072
49972390,2sleep20-21swapper/222:25:072
2883499397,18cyclictest0-21swapper/319:35:203
2883399398,24cyclictest8050irq/56-dwc_otg_22:05:172
2883399398,22cyclictest0-21swapper/221:10:192
2883299393,6cyclictest0-21swapper/100:20:341
2883299392,6cyclictest23-21ksoftirqd/121:00:231
28832993923,4cyclictest21053-21aten_rbpower_cu22:05:191
2883299392,26cyclictest0-21swapper/123:35:311
28834993815,5cyclictest3929-21fschecks_count19:30:223
28833993820,8cyclictest28592-21aten_rbpower_en21:35:212
2883299387,21cyclictest0-21swapper/119:20:201
2883299386,20cyclictest6565-21ssh23:45:201
2883299386,20cyclictest6565-21ssh23:45:201
2883299386,20cyclictest22352-21ssh23:25:121
2883299383,11cyclictest0-21swapper/120:55:191
2883299383,11cyclictest0-21swapper/120:55:191
2883299381,29cyclictest28708-21ls21:35:221
249062380,1sleep30-21swapper/300:06:303
190882380,2sleep10-21swapper/100:00:201
168332380,2sleep30-21swapper/322:00:203
2883499376,23cyclictest0-21swapper/319:10:223
2883499375,23cyclictest29904-21ls22:15:273
28834993712,5cyclictest14689-21unixbench_singl22:35:353
2883399376,4cyclictest7950irq/56-dwc_otg_23:45:352
2883399376,4cyclictest7950irq/56-dwc_otg_23:45:352
2883399376,4cyclictest7950irq/56-dwc_otg_23:20:292
2883399373,6cyclictest8050irq/56-dwc_otg_22:15:512
28833993719,4cyclictest29-21rcuc/221:48:142
2883299379,12cyclictest9550irq/48-DMA15:26:471
2883299378,6cyclictest8050irq/56-dwc_otg_22:30:361
2883299376,21cyclictest0-21swapper/120:15:191
2883299372,3cyclictest8050irq/56-dwc_otg_23:40:181
84032360,1sleep30-21swapper/322:29:073
78752360,9sleep2301ktimersoftd/222:28:092
57752360,6sleep18050irq/56-dwc_otg_22:25:261
2883499361,25cyclictest1937-21ls20:55:073
2883399366,6cyclictest7950irq/56-dwc_otg_23:45:062
2883399362,5cyclictest0-21swapper/220:45:172
28833993620,13cyclictest8050irq/56-dwc_otg_22:30:172
28833993619,5cyclictest29-21rcuc/223:16:322
28833993619,5cyclictest29-21rcuc/223:16:322
2883399361,31cyclictest14814-21cut20:00:232
2883399361,24cyclictest25670-21cut20:30:242
2883299369,12cyclictest19828-21kworker/1:121:25:471
2883299365,6cyclictest7950irq/56-dwc_otg_21:20:251
2883299365,20cyclictest23701-21ssh22:46:551
2883299365,20cyclictest23701-21ssh22:46:551
2883299362,6cyclictest8359-21tail19:40:331
2883299362,6cyclictest8359-21tail19:40:331
2883299362,5cyclictest0-21swapper/123:50:341
2883299362,25cyclictest0-21swapper/100:19:101
2883299361,5cyclictest21-21rcuc/122:10:441
28832993613,6cyclictest5060-1kworker/u9:121:50:191
28832993610,6cyclictest0-21swapper/121:15:151
28832993610,10cyclictest21-21rcuc/100:05:241
28834993526,7cyclictest99142chrt23:10:173
28834993517,5cyclictest7310-21diskmemload00:10:173
28834993511,10cyclictest8050irq/56-dwc_otg_23:30:383
2883399359,5cyclictest7950irq/56-dwc_otg_19:35:062
2883399356,10cyclictest8050irq/56-dwc_otg_00:05:372
2883399355,3cyclictest7850irq/56-dwc_otg21:25:182
2883399353,6cyclictest0-21swapper/219:25:172
28833993517,11cyclictest1594-21ssh23:00:152
2883299352,4cyclictest5218-21modprobe19:35:061
28832993524,6cyclictest221ktimersoftd/122:35:191
28832993520,4cyclictest579-21sh21:40:241
28832993520,4cyclictest579-21sh21:40:241
28832993515,4cyclictest21-21rcuc/119:59:121
28832993512,17cyclictest23-21ksoftirqd/121:45:261
2883499343,8cyclictest0-21swapper/319:40:173
2883499343,8cyclictest0-21swapper/319:40:173
2883499341,28cyclictest16377-21cut21:20:253
28834993412,4cyclictest0-21swapper/300:25:243
2883399347,7cyclictest8050irq/56-dwc_otg_00:15:262
2883399344,6cyclictest8050irq/56-dwc_otg_21:00:192
2883399344,5cyclictest31-21ksoftirqd/221:42:462
2883399344,5cyclictest31-21ksoftirqd/221:42:462
2883399342,5cyclictest0-21swapper/222:45:362
2883399342,5cyclictest0-21swapper/222:45:362
2883399341,4cyclictest8050irq/56-dwc_otg_21:17:062
2883299343,4cyclictest0-21swapper/123:20:061
2883299343,4cyclictest0-21swapper/123:20:061
28832993416,9cyclictest727-21runrttasks19:35:231
135972340,5sleep22883399cyclictest19:55:322
2883499332,8cyclictest8050irq/56-dwc_otg_20:35:173
2883499331,7cyclictest18764-21latency_hist00:00:063
2883499331,23cyclictest26265-21ls23:30:063
2883499331,22cyclictest25192-21latency_hist20:30:063
2883499331,15cyclictest0-21swapper/300:03:383
28834993311,14cyclictest16489-21basename20:05:213
28834993311,14cyclictest16489-21basename20:05:213
28834993311,12cyclictest0-21swapper/322:50:243
2883399336,15cyclictest0-21swapper/220:20:192
2883399334,7cyclictest8050irq/56-dwc_otg_22:39:122
2883399332,14cyclictest31-21ksoftirqd/222:10:442
28833993312,4cyclictest29-21rcuc/200:10:472
2883399331,22cyclictest9593-21timerandwakeup21:50:352
28832993314,12cyclictest25165-21sh21:30:301
2883499329,5cyclictest21708-21latency_hist20:20:063
2883499327,15cyclictest0-21swapper/321:05:213
2883499326,5cyclictest7850irq/56-dwc_otg00:40:063
2883499325,18cyclictest7950irq/56-dwc_otg_22:23:493
2883499325,18cyclictest7950irq/56-dwc_otg_22:23:493
2883499324,6cyclictest10651-21basename19:50:053
2883499324,5cyclictest7950irq/56-dwc_otg_21:35:063
2883499324,18cyclictest7850irq/56-dwc_otg22:10:263
28834993214,5cyclictest727-21runrttasks20:11:353
28834993212,5cyclictest7958-21fschecks_count21:10:233
2883499321,22cyclictest0-21swapper/300:34:353
28834993211,5cyclictest0-21swapper/321:50:223
2883399326,5cyclictest7950irq/56-dwc_otg_19:10:322
2883399325,6cyclictest31-21ksoftirqd/222:00:362
2883399323,4cyclictest31-21ksoftirqd/223:35:252
2883299328,4cyclictest21-21rcuc/122:41:521
2883299325,7cyclictest0-21swapper/119:25:181
2883499318,15cyclictest28830-21cyclictest20:30:293
2883499317,12cyclictest0-21swapper/321:25:323
2883499316,17cyclictest7950irq/56-dwc_otg_23:21:273
2883499314,7cyclictest7850irq/56-dwc_otg23:00:253
2883499314,15cyclictest8050irq/56-dwc_otg_23:45:183
2883499314,15cyclictest8050irq/56-dwc_otg_23:45:183
28834993113,10cyclictest10617-21sh22:30:373
2883499311,21cyclictest0-21swapper/321:04:513
28834993111,11cyclictest24558-21taskset22:48:273
28834993111,11cyclictest24558-21taskset22:48:273
2883399314,5cyclictest0-21swapper/220:10:172
2883399313,12cyclictest8050irq/56-dwc_otg_20:25:332
28833993117,11cyclictest8050irq/56-dwc_otg_22:55:192
2883399311,6cyclictest12411-21taskset23:51:192
28833993114,5cyclictest29-21rcuc/219:50:552
2883299312,8cyclictest8050irq/56-dwc_otg_19:10:271
2883499308,11cyclictest37-21rcuc/320:55:273
2883499308,11cyclictest37-21rcuc/320:55:273
2883499302,7cyclictest8050irq/56-dwc_otg_00:20:203
28834993023,5cyclictest381ktimersoftd/320:45:153
2883399306,10cyclictest31-21ksoftirqd/219:40:062
2883399302,12cyclictest8050irq/56-dwc_otg_20:18:382
2883299303,7cyclictest0-21swapper/120:00:201
2883499295,5cyclictest7950irq/56-dwc_otg_23:05:343
2883499295,13cyclictest39-21ksoftirqd/320:20:353
2883299294,13cyclictest8050irq/56-dwc_otg_20:30:341
2883499281,16cyclictest0-21swapper/320:00:253
2883299282,7cyclictest30202-21sed20:40:331
2883499271,24cyclictest0-21swapper/319:20:203
2883499271,17cyclictest0-21swapper/319:55:273
28833992620,4cyclictest29092-21basename20:40:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional