You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-04 - 02:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Thu Jul 03, 2025 12:43:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2301621070,6sleep13004599cyclictest11:03:421
2977229952,11sleep0101ktimersoftd/007:10:110
3004599947,39cyclictest0-21swapper/107:50:251
197262910,4sleep219711-21ssh12:25:402
223412820,4sleep322328-21timerandwakeup10:15:463
30044997748,14cyclictest7850irq/56-dwc_otg10:50:260
30044997748,14cyclictest7850irq/56-dwc_otg10:50:260
188222740,1sleep30-21swapper/312:25:293
3004599723,7cyclictest0-21swapper/110:25:271
30044997152,5cyclictest7950irq/56-dwc_otg_08:55:120
3004499706,53cyclictest0-21swapper/010:20:270
30044997035,4cyclictest8050irq/56-dwc_otg_10:00:240
302042690,3sleep130194-21munin-run08:40:111
3004599693,24cyclictest0-21swapper/110:20:271
3004499693,29cyclictest0-21swapper/007:15:230
3004499692,37cyclictest0-21swapper/010:10:240
30044996839,9cyclictest0-21swapper/008:30:220
3004599676,22cyclictest7850irq/56-dwc_otg08:40:251
30044996740,7cyclictest7950irq/56-dwc_otg_11:15:110
3004499661,46cyclictest0-21swapper/011:35:300
3004699656,11cyclictest7850irq/56-dwc_otg09:20:242
3004499653,48cyclictest0-21swapper/008:15:220
164032650,5sleep13004599cyclictest09:20:321
3004499632,42cyclictest29478-21grep09:40:470
3004499631,35cyclictest10206-21apt12:15:110
3004799626,21cyclictest381ktimersoftd/309:55:243
3004699602,18cyclictest8050irq/56-dwc_otg_11:55:262
30044996033,4cyclictest8050irq/56-dwc_otg_12:25:220
132662600,31sleep013-21rcuc/007:50:390
3004499596,32cyclictest0-21swapper/007:45:120
3004499596,32cyclictest0-21swapper/007:45:120
3004499592,47cyclictest0-21swapper/010:40:410
30044995819,22cyclictest16994-21sh09:20:400
3004499573,14cyclictest26458-21ntp_kernel_err09:35:400
3004499572,46cyclictest12765-21fschecks_time09:15:300
3004499572,29cyclictest21982-21gpgv09:30:120
3004499571,42cyclictest0-21swapper/008:40:280
23872570,2sleep32388-21unixbench_singl07:20:393
3004499561,37cyclictest3821-21unixbench_singl12:05:450
3004499561,36cyclictest0-21swapper/007:25:250
179222560,2sleep2301ktimersoftd/211:41:112
3004599557,10cyclictest7950irq/56-dwc_otg_11:45:251
30044995533,4cyclictest7950irq/56-dwc_otg_10:05:250
30044995527,13cyclictest8050irq/56-dwc_otg_07:20:260
3004499552,45cyclictest0-21swapper/011:50:400
30044995518,25cyclictest2215-21idleruntime-cro12:05:120
3004499551,45cyclictest27057-21grep12:35:280
30045995414,25cyclictest0-21swapper/108:05:261
30044995421,15cyclictest10296-21wc11:30:400
3004699531,7cyclictest0-21swapper/210:40:242
30044995332,6cyclictest7850irq/56-dwc_otg10:00:010
3004499532,4cyclictest0-21swapper/008:35:410
3004499532,42cyclictest14900-21fschecks_time12:20:320
30044995323,7cyclictest2268-21sh11:20:260
3004499521,7cyclictest5367-21cut09:00:260
3004599515,32cyclictest7850irq/56-dwc_otg08:50:281
3004599511,33cyclictest0-21swapper/110:10:121
3004499515,5cyclictest0-21swapper/008:00:380
3004499512,29cyclictest0-21swapper/011:15:490
3004499512,29cyclictest0-21swapper/011:15:490
30044995117,5cyclictest2953-21awk10:35:110
3004499511,25cyclictest24132-21sh11:05:290
3004799502,11cyclictest8050irq/56-dwc_otg_08:30:243
3004599504,9cyclictest4750irq/17-3f00b88008:00:211
30044995028,13cyclictest20155-21ntp_kernel_pll_08:10:340
3004499502,24cyclictest0-21swapper/010:45:340
30044995021,5cyclictest0-21swapper/011:00:120
30044995019,10cyclictest0-21swapper/009:30:240
3004499501,39cyclictest4870-21chrt07:30:260
3004499501,33cyclictest955-21runrttasks10:39:050
3004499501,25cyclictest20111-21aten_rbpower_vo11:45:270
3004699493,4cyclictest8050irq/56-dwc_otg_08:35:252
30046994922,5cyclictest21623-21sh10:15:362
3004699491,29cyclictest27815-21ls08:30:382
30045994918,5cyclictest23967-21unixbench_multi12:30:431
30044994927,5cyclictest7950irq/56-dwc_otg_11:00:390
3004499492,24cyclictest0-21swapper/010:25:280
30044994921,5cyclictest17701-21unixbench_singl11:40:420
3004499491,6cyclictest23512-21ntpq12:30:380
3004499491,17cyclictest17903-21grep08:05:300
232262490,2sleep023228-21ssh10:17:130
3004699483,6cyclictest0-21swapper/209:05:222
3004599488,25cyclictest0-21swapper/109:25:171
30045994824,4cyclictest16421-21/usr/sbin/munin11:40:241
3004499482,37cyclictest0-21swapper/007:35:320
30044994822,14cyclictest0-21swapper/009:08:350
30044994822,12cyclictest14610-21ls07:55:340
3004599477,26cyclictest0-21swapper/109:10:271
30045994716,15cyclictest19490-21sshd08:10:231
3004599471,4cyclictest0-21swapper/111:20:431
30044994724,4cyclictest60-21kswapd007:45:180
3004499472,20cyclictest0-21swapper/009:50:320
3004499472,20cyclictest0-21swapper/009:50:320
3004799467,24cyclictest0-21swapper/307:35:273
3004599467,24cyclictest0-21swapper/112:35:241
3004499461,33cyclictest516-21munin-node07:10:230
3004499461,32cyclictest7531-21ssh11:26:200
3004799459,6cyclictest39-21ksoftirqd/308:10:123
3004599458,24cyclictest7850irq/56-dwc_otg11:05:251
3004599457,25cyclictest0-21swapper/108:55:181
3004599457,25cyclictest0-21swapper/107:10:261
30044994523,4cyclictest7950irq/56-dwc_otg_11:55:230
30044994522,3cyclictest0-21swapper/012:15:330
30044994517,5cyclictest25269-21fschecks_count08:25:350
30044994513,5cyclictest9310-21tune2fs09:10:310
3004499451,33cyclictest0-21swapper/008:48:070
3004499451,24cyclictest955-21runrttasks09:46:240
3004799446,25cyclictest0-21swapper/309:05:233
3004599447,26cyclictest7850irq/56-dwc_otg09:00:261
3004599444,26cyclictest7850irq/56-dwc_otg10:55:121
3004599444,26cyclictest7850irq/56-dwc_otg10:55:121
3004599442,24cyclictest0-21swapper/108:30:221
30045994411,23cyclictest0-21swapper/107:15:231
30044994425,5cyclictest7950irq/56-dwc_otg_08:55:240
3004499442,33cyclictest0-21swapper/008:21:200
3004799437,25cyclictest0-21swapper/308:00:233
3004799437,21cyclictest0-21swapper/307:25:253
3004699436,15cyclictest7950irq/56-dwc_otg_08:45:282
3004599435,25cyclictest7850irq/56-dwc_otg09:30:301
30045994326,4cyclictest12992-21basename11:35:271
42562420,2sleep10-21swapper/112:06:351
3004799428,27cyclictest0-21swapper/311:30:263
3004799421,8cyclictest0-21swapper/309:50:353
3004799421,8cyclictest0-21swapper/309:50:353
3004699427,23cyclictest0-21swapper/209:35:262
3004699427,21cyclictest0-21swapper/207:55:262
3004699424,25cyclictest0-21swapper/211:30:262
3004599427,23cyclictest0-21swapper/109:40:261
3004599425,15cyclictest969-21snmpd12:10:261
3004799417,21cyclictest0-21swapper/310:10:263
3004699417,24cyclictest0-21swapper/210:45:272
3004699417,24cyclictest0-21swapper/207:25:252
30046994114,9cyclictest301ktimersoftd/212:20:382
3004599413,27cyclictest7850irq/56-dwc_otg07:20:221
224682410,1sleep30-21swapper/311:48:093
3004799403,22cyclictest0-21swapper/307:55:263
3004799401,23cyclictest8050irq/56-dwc_otg_09:25:183
30046994016,4cyclictest26153-21unixbench_singl10:20:432
3004599409,23cyclictest7850irq/56-dwc_otg09:40:111
3004599402,8cyclictest17942-21switchtime10:55:381
3004599402,25cyclictest7850irq/56-dwc_otg11:19:501
3004599402,25cyclictest7850irq/56-dwc_otg11:19:501
3004599401,26cyclictest7850irq/56-dwc_otg12:00:311
3004599401,24cyclictest0-21swapper/111:55:241
30045994011,19cyclictest3092-21basename07:25:251
3004799397,17cyclictest39-21ksoftirqd/310:05:253
3004699397,23cyclictest0-21swapper/209:55:272
3004699394,23cyclictest0-21swapper/211:35:182
30046993916,6cyclictest19141-21latency_hist08:10:132
3004599397,20cyclictest7850irq/56-dwc_otg11:30:371
3004799381,3cyclictest0-21swapper/309:35:113
3004799381,26cyclictest457-21grep09:45:323
3004799381,22cyclictest28826-21ls08:35:283
30046993814,11cyclictest8670-21diskmemload10:08:442
30046993810,5cyclictest7950irq/56-dwc_otg_09:15:112
3004599386,7cyclictest7850irq/56-dwc_otg09:20:121
3004599385,5cyclictest9588-21grep07:40:401
3004599385,5cyclictest9588-21grep07:40:401
3004599385,19cyclictest7850irq/56-dwc_otg09:50:181
3004599385,19cyclictest7850irq/56-dwc_otg09:50:181
3004599384,5cyclictest8263-21grep09:05:371
3004599384,23cyclictest7850irq/56-dwc_otg08:00:041
3004599383,4cyclictest0-21swapper/111:25:251
3004599381,26cyclictest7213-21ssh10:40:311
30045993812,5cyclictest0-21swapper/111:50:281
59782370,22sleep28050irq/56-dwc_otg_10:39:002
3004799375,19cyclictest10132-21ssh10:45:273
3004799374,20cyclictest39-21ksoftirqd/311:16:543
3004799374,20cyclictest39-21ksoftirqd/311:16:543
3004799371,24cyclictest0-21swapper/311:10:273
3004699379,21cyclictest0-21swapper/212:30:272
3004699375,9cyclictest29-21rcuc/209:46:462
3004699372,17cyclictest29-21rcuc/212:15:162
3004699371,15cyclictest31-21ksoftirqd/210:25:432
3004599375,23cyclictest17833-21ssh10:10:261
3004599375,20cyclictest7850irq/56-dwc_otg12:15:431
3004599371,25cyclictest8050irq/56-dwc_otg_08:20:121
3004599371,24cyclictest7850irq/56-dwc_otg10:15:251
30045993711,4cyclictest0-21swapper/109:59:491
322792360,2sleep30-21swapper/312:00:373
3004799367,26cyclictest0-21swapper/311:35:183
3004699369,10cyclictest29-21rcuc/207:50:122
30046993618,6cyclictest7850irq/56-dwc_otg10:00:392
3004699361,3cyclictest7950irq/56-dwc_otg_12:37:522
3004699361,22cyclictest1489-21cut07:20:282
3004599362,23cyclictest7850irq/56-dwc_otg10:45:331
3004799358,6cyclictest39-21ksoftirqd/312:20:113
3004799353,26cyclictest8050irq/56-dwc_otg_07:40:263
3004799353,26cyclictest8050irq/56-dwc_otg_07:40:263
30047993520,7cyclictest7595-21rm12:10:403
3004799351,26cyclictest24165-21ls11:50:303
3004799351,23cyclictest11324-21ls07:45:413
3004799351,17cyclictest39-21ksoftirqd/309:25:123
3004699355,21cyclictest31056-21ssh11:15:262
3004699355,21cyclictest31056-21ssh11:15:262
30046993528,5cyclictest24548-21ssh11:50:352
3004699352,5cyclictest3526-21wget09:50:272
3004699352,5cyclictest3526-21wget09:50:272
3004699352,19cyclictest29-21rcuc/208:10:382
30046993516,11cyclictest29-21rcuc/209:29:362
3004699351,5cyclictest30434-21/usr/sbin/munin08:40:402
3004699351,26cyclictest13029-21chrt07:50:372
3004699351,20cyclictest1007-21latency_hist07:20:122
3004599354,23cyclictest7850irq/56-dwc_otg08:25:111
3004599354,21cyclictest7850irq/56-dwc_otg10:35:461
3004599352,21cyclictest7850irq/56-dwc_otg11:10:451
3004599352,16cyclictest0-21swapper/110:00:411
14592350,6sleep23004699cyclictest12:03:222
10802350,3sleep11081-21switchtime09:45:401
3004799347,21cyclictest0-21swapper/309:35:263
3004799346,13cyclictest8050irq/56-dwc_otg_08:25:113
3004799344,15cyclictest7950irq/56-dwc_otg_09:15:303
3004799342,19cyclictest0-21swapper/308:45:253
30047993418,4cyclictest16677-21idleruntime-cro10:55:123
30047993418,4cyclictest16677-21idleruntime-cro10:55:123
30047993414,10cyclictest30734-21ntp_kernel_err07:10:353
3004799341,24cyclictest0-21swapper/310:20:273
3004699347,7cyclictest7950irq/56-dwc_otg_11:05:422
3004699347,6cyclictest7950irq/56-dwc_otg_11:20:412
3004699345,6cyclictest29-21rcuc/210:58:102
30046993412,4cyclictest17875-21sh10:10:262
3004599345,22cyclictest7950irq/56-dwc_otg_12:25:291
3004599341,21cyclictest10607-21cut07:45:301
3004799335,19cyclictest8021-21grep10:40:413
3004799335,13cyclictest7950irq/56-dwc_otg_12:05:453
30047993326,5cyclictest39-21ksoftirqd/307:50:233
30047993321,5cyclictest22947-21ssh11:03:323
30047993313,5cyclictest8670-21diskmemload12:35:393
3004799331,17cyclictest37-21rcuc/308:10:133
3004799331,17cyclictest0-21swapper/309:10:313
3004799331,12cyclictest0-21swapper/310:57:433
3004699339,6cyclictest7850irq/56-dwc_otg10:30:522
3004699339,14cyclictest0-21swapper/209:30:182
3004699337,6cyclictest0-21swapper/211:27:092
3004699335,7cyclictest31-21ksoftirqd/207:10:382
30046993319,5cyclictest31020-21sh09:42:282
3004699331,7cyclictest7850irq/56-dwc_otg08:20:272
30046993317,7cyclictest4893-21aten_rbpower_cu07:30:272
3004699331,17cyclictest7950irq/56-dwc_otg_10:50:332
3004699331,17cyclictest7950irq/56-dwc_otg_10:50:332
3004599338,12cyclictest0-21swapper/112:20:251
3004599337,15cyclictest0-21swapper/107:30:271
30045993318,5cyclictest32104-21sh10:30:181
289322337,10sleep20-21swapper/207:05:392
3004799328,11cyclictest39-21ksoftirqd/309:41:293
3004799322,7cyclictest39-21ksoftirqd/310:36:333
3004799322,22cyclictest0-21swapper/310:26:143
30047993214,6cyclictest39-21ksoftirqd/308:40:323
30047993213,4cyclictest39-21ksoftirqd/310:00:373
3004799321,23cyclictest32442-21missed_timers07:15:343
3004799321,21cyclictest803-21timerwakeupswit10:30:423
3004799321,19cyclictest24985-21ls11:05:413
3004799321,17cyclictest0-21swapper/311:20:243
3004799321,14cyclictest0-21swapper/312:30:403
3004699327,15cyclictest0-21swapper/208:50:272
3004699327,14cyclictest0-21swapper/211:00:272
3004699325,12cyclictest7950irq/56-dwc_otg_11:46:172
30046993218,6cyclictest22851-21cron08:20:112
3004699321,24cyclictest5543-21cut09:00:292
3004599325,17cyclictest7950irq/56-dwc_otg_07:35:311
30045993212,6cyclictest0-21swapper/108:45:391
3004799316,4cyclictest7850irq/56-dwc_otg11:55:463
3004799311,21cyclictest0-21swapper/311:40:143
30046993114,7cyclictest4900-21ssh12:07:512
30046993112,4cyclictest8050irq/56-dwc_otg_08:25:442
3004699311,20cyclictest29-21rcuc/211:11:382
3004799307,4cyclictest8050irq/56-dwc_otg_12:20:123
3004799304,10cyclictest39-21ksoftirqd/308:25:333
3004799303,6cyclictest783750ssh11:27:023
3004799301,5cyclictest37-21rcuc/308:55:113
3004799301,20cyclictest0-21swapper/307:30:273
3004699302,11cyclictest7850irq/56-dwc_otg08:55:282
3004699301,17cyclictest8050irq/56-dwc_otg_12:12:212
3004699301,10cyclictest14734-21ssh09:18:402
3004799292,18cyclictest0-21swapper/309:00:403
30047992921,6cyclictest37-21rcuc/308:56:043
3004699297,12cyclictest29-21rcuc/207:35:482
295702297,15sleep10-21swapper/107:07:411
3004799283,13cyclictest8050irq/56-dwc_otg_08:15:353
3004699281,7cyclictest8513-21basename07:40:262
3004699281,7cyclictest8513-21basename07:40:262
3004599281,15cyclictest26582-21latency_hist08:30:111
295402273,14sleep30-21swapper/307:07:183
3004699262,6cyclictest7850irq/56-dwc_otg08:00:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional