You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 22:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Wed Sep 17, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3256121530,2sleep32009799cyclictest09:59:293
20094991006,61cyclictest0-21swapper/010:25:170
2009499873,32cyclictest0-21swapper/009:00:160
194972873,13sleep00-21swapper/007:05:550
20094998612,39cyclictest0-21swapper/008:50:160
2009499854,32cyclictest0-21swapper/012:30:140
20094998433,12cyclictest0-21swapper/012:35:150
203732800,2sleep00-21swapper/008:40:200
13652790,2sleep32009799cyclictest09:14:003
13652790,2sleep32009799cyclictest09:14:003
36812730,6sleep32009799cyclictest11:35:343
99462720,2sleep22009699cyclictest08:10:232
2009799721,45cyclictest2123-21/usr/sbin/munin09:15:133
20095997216,10cyclictest8050irq/56-dwc_otg_11:25:161
2009499711,38cyclictest0-21swapper/010:45:130
2009799702,50cyclictest15604-21taskset11:07:593
33662690,2sleep10-21swapper/110:03:381
20096996931,18cyclictest24423-21/usr/sbin/munin12:05:232
199402690,2sleep20-21swapper/210:25:452
2009799681,45cyclictest11303-21ssh12:30:403
2009799671,54cyclictest24018-21ssh10:31:593
2009799671,52cyclictest8906-21ssh10:59:133
2009799671,52cyclictest26713-21ls10:35:323
20095996715,14cyclictest7950irq/56-dwc_otg_10:45:131
2009799661,53cyclictest0-21swapper/308:45:133
2009799661,45cyclictest6107-21diskstats08:00:213
2009799661,43cyclictest12106-21ssh09:30:143
2009799661,41cyclictest19787-21ssh09:40:273
20096996524,13cyclictest30950-21modprobe09:10:022
20094996514,8cyclictest23673-21munin-run11:20:020
2009799642,46cyclictest4561-21sh12:22:003
2009799642,46cyclictest4561-21sh12:22:003
2009799641,52cyclictest10854-21ssh10:14:243
2009799641,44cyclictest6918-21sshd09:20:373
2009799641,42cyclictest7850irq/56-dwc_otg10:00:243
2009799641,42cyclictest24252-21/usr/sbin/munin12:05:213
2009799641,42cyclictest1301550rm10:16:153
20096996427,10cyclictest27783-21ssh09:51:262
2009499642,43cyclictest0-21swapper/011:35:220
2009499641,42cyclictest25776-21wget08:55:160
2009799631,53cyclictest19112-21kernelversion10:25:233
2009799631,38cyclictest1031-21/usr/sbin/munin10:45:253
20094996310,40cyclictest9068-21unixbench_multi10:10:310
278982620,1sleep10-21swapper/112:10:191
2009799625,33cyclictest8050irq/56-dwc_otg_12:35:223
2009799622,41cyclictest17111-21ssh10:22:273
2009799621,49cyclictest31777-21ssh10:43:393
2009799621,48cyclictest11833-21ssh11:02:333
2009799621,48cyclictest11833-21ssh11:02:333
2009799621,45cyclictest0-21swapper/311:33:553
2009799621,45cyclictest0-21swapper/311:33:553
20096996239,8cyclictest22841-21ntp_states08:45:272
20096996231,12cyclictest1011-21/usr/sbin/munin10:45:252
20096996227,23cyclictest25382-21cat10:35:022
2009499621,58cyclictest0-21swapper/011:00:290
2009499621,58cyclictest0-21swapper/011:00:290
2009499621,47cyclictest20404-21head09:40:340
2009799616,45cyclictest5561-21unixbench_singl10:05:323
2009799616,45cyclictest5561-21unixbench_singl10:05:323
2009799613,42cyclictest27892-21ssh11:25:243
2009799611,50cyclictest17459-21ssh09:36:413
2009799611,47cyclictest8477-21ssh12:27:353
2009799611,45cyclictest30668-21taskset09:06:583
2009799611,45cyclictest26246-21ssh11:22:513
2009799611,44cyclictest24985-21cat08:50:333
20096996134,18cyclictest28243-21ntpq09:00:272
20096996130,11cyclictest5926-21/usr/sbin/munin08:00:172
2009499616,44cyclictest0-21swapper/007:30:170
2009799602,44cyclictest462-21avahi-daemon09:25:233
20097996017,40cyclictest544-21aten_rbpower_vo07:45:173
2009799601,49cyclictest0-21swapper/311:51:433
2009799601,40cyclictest23946-21ssh09:45:463
2009799601,39cyclictest8781-21sed08:05:303
2009799601,39cyclictest11728-21sh11:47:583
20096996030,17cyclictest4501-21if_eth007:55:222
20096996021,19cyclictest10231-21ssh09:25:522
2009599607,26cyclictest0-21swapper/112:15:151
2009499604,31cyclictest9-21ksoftirqd/010:05:280
2009499604,31cyclictest9-21ksoftirqd/010:05:280
189392600,7sleep10-21swapper/107:05:261
172472600,2sleep20-21swapper/211:55:262
155492600,3sleep25168-21kworker/2:112:36:532
2009799591,54cyclictest16493-21apt-get11:55:153
2009799591,45cyclictest3650-21taskset07:52:453
2009799591,44cyclictest22410-21ssh11:17:093
2009799591,42cyclictest9704-21diskstats08:10:203
2009799591,36cyclictest8050irq/56-dwc_otg_12:02:173
20096995929,19cyclictest6963-21ssh10:08:472
20096995929,19cyclictest6963-21ssh10:08:472
20096995924,19cyclictest14491-21sshd10:19:382
2009499595,39cyclictest0-21swapper/007:55:260
20094995928,7cyclictest0-21swapper/010:20:160
2009799581,43cyclictest536-21ssh12:16:193
2009799581,42cyclictest18275-21ssh11:11:013
2009799581,40cyclictest8050irq/56-dwc_otg_07:55:273
2009799581,40cyclictest29163-21chrt12:11:013
2009799581,40cyclictest28810-21runrttasks09:01:263
2009799581,38cyclictest7850irq/56-dwc_otg11:42:253
2009799581,37cyclictest27540-21ssh09:50:523
2009799581,37cyclictest20975-21ls08:40:273
20096995830,15cyclictest23627-21ssh11:19:582
20096995825,11cyclictest24633-21ntp_states08:50:292
20096995823,13cyclictest8500-21ssh10:10:242
20096995820,12cyclictest27434-21ssh10:36:462
20096995819,24cyclictest8050irq/56-dwc_otg_11:45:162
2009699581,32cyclictest0-21swapper/209:45:152
2009499585,35cyclictest9-21ksoftirqd/012:10:310
20094995842,4cyclictest101ktimersoftd/012:05:150
2009799571,41cyclictest8050irq/56-dwc_otg_10:51:103
20096995728,9cyclictest20817-21ls12:00:232
20096995727,12cyclictest27893-21fschecks_count11:25:242
20096995722,18cyclictest31475-21ssh09:56:442
20094995743,6cyclictest13-21rcuc/009:40:020
2009799561,41cyclictest0-21swapper/307:37:023
2009799561,41cyclictest0-21swapper/307:37:023
2009799561,39cyclictest19724-21chrt08:35:363
2009799561,36cyclictest32407-21chrt07:41:543
20096995630,7cyclictest913-21df10:00:192
20096995628,13cyclictest5576-21ssh11:39:482
20096995626,12cyclictest13466-21df_inode11:05:212
20096995626,12cyclictest10447-21seq12:30:222
20096995618,15cyclictest8050irq/56-dwc_otg_09:34:352
2009499569,29cyclictest12885-21idleruntime-cro08:20:020
2009499562,37cyclictest0-21swapper/009:58:040
2009499562,31cyclictest10007-21grep09:25:310
2009499562,28cyclictest0-21swapper/008:25:200
2009499562,28cyclictest0-21swapper/008:25:200
20094995617,24cyclictest0-21swapper/011:50:330
2009499561,33cyclictest22494-21fschecks_time10:30:230
20094995610,11cyclictest0-21swapper/011:25:160
2009799551,38cyclictest27188-21chrt08:58:423
2009799551,38cyclictest23469-21sshd07:18:383
20096995525,19cyclictest23651-21cron11:20:032
20096995525,11cyclictest2108-21/usr/sbin/munin09:15:302
20096995524,16cyclictest7950irq/56-dwc_otg_09:40:332
20096995522,15cyclictest8615-21ssh12:27:532
2009499558,11cyclictest7950irq/56-dwc_otg_09:05:170
2009499552,34cyclictest20515-21fschecks_count12:00:200
2009499551,29cyclictest5293-21latency_hist09:20:020
2009499551,27cyclictest0-21swapper/007:40:130
2009799541,49cyclictest20224-21apt-get07:10:153
2009799541,36cyclictest14610-21taskset08:24:163
2009799541,35cyclictest16396-21taskset08:29:083
2009799541,35cyclictest16396-21taskset08:29:083
20096995423,12cyclictest6917-21ssh09:20:372
20096995423,11cyclictest8153-21ssh11:42:392
20096995422,12cyclictest31247-21ssh10:42:302
2009499546,24cyclictest13-21rcuc/010:59:290
2009499545,37cyclictest0-21swapper/007:10:170
2009499541,20cyclictest30314-21ls07:35:280
2009499541,20cyclictest30314-21ls07:35:280
240992530,1sleep10-21swapper/107:20:191
2009799534,41cyclictest0-21swapper/307:23:423
2009799531,46cyclictest16953-21/usr/sbin/munin08:30:173
20096995331,13cyclictest2080-21sed07:50:022
20096995325,15cyclictest17860-21sh09:37:422
20096995323,19cyclictest6230-21taskset10:54:162
20096995321,13cyclictest9078-21ssh10:59:362
20096995319,21cyclictest422-21ssh11:31:212
20096995319,21cyclictest422-21ssh11:31:212
20096995319,10cyclictest16420-21taskset08:29:272
20096995319,10cyclictest16420-21taskset08:29:272
2009499533,27cyclictest0-21swapper/009:30:230
2009499532,36cyclictest24959-21grep07:20:290
2009499532,30cyclictest0-21swapper/009:48:280
20094995321,17cyclictest0-21swapper/007:20:020
20094995313,26cyclictest0-21swapper/009:50:160
2009499531,29cyclictest31269-21aten_rbpower_cu11:30:170
2009499531,29cyclictest31269-21aten_rbpower_cu11:30:170
57612520,2sleep013-21rcuc/012:24:390
57612520,2sleep013-21rcuc/012:24:390
20096995234,15cyclictest31323-21apt-get12:15:132
20096995222,19cyclictest16787-21/usr/sbin/munin11:10:232
20096995222,17cyclictest0-21swapper/211:52:412
20096995221,17cyclictest4708-21ssh12:22:202
20096995221,17cyclictest4708-21ssh12:22:202
2009499521,35cyclictest7004-21threads11:40:300
2009499521,34cyclictest0-21swapper/009:13:590
2009499521,34cyclictest0-21swapper/009:13:590
2009799511,46cyclictest12124-21/usr/sbin/munin08:15:293
2009799510,38cyclictest8050irq/56-dwc_otg_07:28:573
20096995125,15cyclictest15807-21open_files10:20:282
20094995134,5cyclictest13-21rcuc/009:20:340
20094995124,5cyclictest3759-21/usr/sbin/munin10:50:240
20094995120,5cyclictest0-21swapper/011:05:170
20094995116,10cyclictest16531-21grep11:55:160
2009499511,36cyclictest0-21swapper/007:45:270
2009499511,30cyclictest25630-21/usr/sbin/munin10:35:320
2009799501,46cyclictest28928-21sshd07:33:383
20096995024,11cyclictest29059-21latency_hist07:35:032
20096995022,13cyclictest30272-21ssh12:13:262
20096995021,19cyclictest32673-21ssh09:11:182
20096995021,19cyclictest32673-21ssh09:11:182
2009599506,13cyclictest7950irq/56-dwc_otg_11:30:151
2009599506,13cyclictest7950irq/56-dwc_otg_11:30:151
2009499502,34cyclictest0-21swapper/008:35:210
20094995016,22cyclictest0-21swapper/008:05:150
2009499501,34cyclictest11003-21sshd08:14:370
2009499501,34cyclictest10704-21ssh11:45:310
20096994922,12cyclictest27095-21run-parts07:28:502
20096994917,11cyclictest27212-21taskset08:59:012
20094994922,4cyclictest32503-21unixbench-2d12:15:300
20096994832,13cyclictest21046-21/usr/sbin/munin08:40:292
20096994824,12cyclictest2256-21taskset07:50:042
20096994819,18cyclictest7850irq/56-dwc_otg11:02:552
20096994819,18cyclictest7850irq/56-dwc_otg11:02:552
20096994819,10cyclictest0-21swapper/208:05:502
2009499486,8cyclictest0-21swapper/012:25:150
2009499481,24cyclictest29100-21sshd10:40:070
20096994724,12cyclictest12546-21mv08:16:152
20096994721,13cyclictest32406-21chrt07:41:532
20096994720,10cyclictest23488-21runrttasks07:18:532
20096994717,11cyclictest19866-21chrt08:37:242
2009599476,12cyclictest0-21swapper/107:40:131
2009499472,43cyclictest101ktimersoftd/008:45:240
2009499472,23cyclictest0-21swapper/011:10:030
2009499472,23cyclictest0-21swapper/010:17:330
20094994720,11cyclictest5967-21aten_rbpower_po08:00:180
2009499471,23cyclictest116350ssh10:00:230
2009699462,9cyclictest8050irq/56-dwc_otg_07:35:122
2009699462,9cyclictest8050irq/56-dwc_otg_07:35:122
20096994621,13cyclictest0-21swapper/207:24:042
20096994618,11cyclictest14390-21runrttasks08:21:322
2009599467,25cyclictest0-21swapper/108:15:171
2009599461,26cyclictest7850irq/56-dwc_otg07:35:151
2009599461,26cyclictest7850irq/56-dwc_otg07:35:151
20096994530,12cyclictest20800-21wc07:10:322
2009599457,26cyclictest0-21swapper/108:45:151
2009599457,25cyclictest0-21swapper/107:10:151
2009499456,25cyclictest0-21swapper/007:25:180
2009499451,31cyclictest0-21swapper/011:21:130
2009599447,24cyclictest0-21swapper/112:00:161
2009599442,23cyclictest0-21swapper/109:00:161
2009499441,13cyclictest0-21swapper/007:50:280
20096994330,11cyclictest17592-21cat08:30:262
20094994329,3cyclictest0-21swapper/008:20:230
2009599423,2cyclictest0-21swapper/110:25:151
20095994212,14cyclictest7850irq/56-dwc_otg10:05:451
20095994212,14cyclictest7850irq/56-dwc_otg10:05:451
187942427,24sleep30-21swapper/307:05:253
2009599418,6cyclictest7850irq/56-dwc_otg07:31:381
2009599406,20cyclictest0-21swapper/112:30:141
2009599401,20cyclictest21511-21grep08:40:341
2009599397,23cyclictest0-21swapper/108:50:161
2009599394,7cyclictest7950irq/56-dwc_otg_11:50:021
2009599391,4cyclictest7950irq/56-dwc_otg_10:30:161
2009499392,18cyclictest0-21swapper/008:30:250
20095993827,7cyclictest221ktimersoftd/108:30:131
20095993827,7cyclictest204032chrt11:15:151
20095993824,4cyclictest25972-21fschecks_time10:35:231
2009599381,6cyclictest6376-21uniq11:40:211
20095993812,6cyclictest24048-21aten_rbpower_cu12:05:181
2009599377,24cyclictest0-21swapper/108:10:171
2009599373,8cyclictest3248-21processes11:35:301
2009599372,6cyclictest0-21swapper/108:35:171
2009599372,22cyclictest3544-21switchtime12:20:291
2009599372,22cyclictest3544-21switchtime12:20:291
2009599372,19cyclictest9650irq/80-mmc009:45:311
20095993716,6cyclictest564450wget09:20:191
20095993625,4cyclictest8050irq/56-dwc_otg_11:00:161
20095993625,4cyclictest8050irq/56-dwc_otg_11:00:161
20095993611,11cyclictest0-21swapper/111:55:181
2009599356,19cyclictest8050irq/56-dwc_otg_09:10:021
2009599354,18cyclictest8050irq/56-dwc_otg_08:05:201
20095993521,5cyclictest16035-21fschecks_time09:35:181
20095993510,13cyclictest0-21swapper/109:58:131
2009599347,5cyclictest25126-21grep11:20:301
2009599345,14cyclictest8050irq/56-dwc_otg_10:40:361
20095993419,3cyclictest7850irq/56-dwc_otg10:55:161
2009599341,19cyclictest0-21swapper/109:30:151
2009599341,18cyclictest32379-21ls09:10:301
2009599341,18cyclictest32379-21ls09:10:301
2009599336,9cyclictest7850irq/56-dwc_otg11:05:171
2009599332,6cyclictest2993-21/usr/sbin/munin09:15:331
20095993320,2cyclictest0-21swapper/112:35:231
2009599331,25cyclictest25933-21grep07:25:211
2009599328,4cyclictest0-21swapper/111:53:521
2009599327,15cyclictest0-21swapper/107:45:161
2009599327,11cyclictest8050irq/56-dwc_otg_07:55:211
2009599326,4cyclictest8050irq/56-dwc_otg_10:20:331
2009599325,6cyclictest26086-21/usr/sbin/munin09:50:301
2009599324,15cyclictest8050irq/56-dwc_otg_10:16:371
20095993213,6cyclictest19711-21cat09:40:251
2009599321,16cyclictest7850irq/56-dwc_otg08:20:171
2009599317,13cyclictest8050irq/56-dwc_otg_10:10:261
2009599316,5cyclictest7850irq/56-dwc_otg09:25:201
2009599311,7cyclictest7028-21grep08:00:331
20095993114,6cyclictest0-21swapper/112:27:361
20095993114,10cyclictest22029-21apt-get07:15:151
2009599311,19cyclictest3716-21ssh10:50:081
188642310,2sleep10-21swapper/111:12:221
2009599301,2cyclictest0-21swapper/108:25:141
2009599301,2cyclictest0-21swapper/108:25:141
2009599301,14cyclictest3949-21ls07:55:021
184832288,11sleep218494-21/usr/sbin/munin07:05:202
2009599278,4cyclictest8050irq/56-dwc_otg_08:55:311
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional