You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Fri May 02, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2636521660,1sleep30-21swapper/321:30:453
3157299941,83cyclictest13798-21sed23:20:460
3157299931,72cyclictest26144-21/usr/sbin/munin22:55:350
3157299931,72cyclictest26144-21/usr/sbin/munin22:55:350
3157299921,76cyclictest0-21swapper/022:05:310
3157299911,80cyclictest8503-21latency_hist23:15:160
3157299891,79cyclictest11793-21/usr/sbin/munin19:45:450
3157299891,77cyclictest28696-21ntpq20:30:390
3157299891,77cyclictest0-21swapper/021:29:030
3157299891,71cyclictest7394-21ntp_kernel_pll_22:30:400
3157299881,74cyclictest19352-21ssh23:29:010
3157299881,73cyclictest0-21swapper/023:06:390
3157299871,77cyclictest0-21swapper/022:39:010
3157299871,77cyclictest0-21swapper/021:40:030
3157299871,75cyclictest2382-21rm22:25:060
3157299871,74cyclictest18247-21rm22:02:430
3157299871,72cyclictest0-21swapper/020:00:430
3157299871,72cyclictest0-21swapper/020:00:430
25672870,3sleep2301ktimersoftd/200:33:022
84082860,2sleep20-21swapper/221:05:322
3157299867,20cyclictest0-21swapper/021:20:330
3157299861,68cyclictest0-21swapper/022:47:410
3157299861,68cyclictest0-21swapper/000:09:380
3157299861,68cyclictest0-21swapper/000:09:380
3157299852,75cyclictest0-21swapper/019:16:000
3157299851,71cyclictest4196-21tune2fs00:35:350
3157299851,67cyclictest0-21swapper/021:45:440
305952850,5sleep20-21swapper/219:05:422
3157299841,72cyclictest5408-21runrttasks20:55:440
3157299841,69cyclictest12536-21sh00:01:290
3157299831,72cyclictest0-21swapper/023:53:140
3157299831,70cyclictest25135-21sh22:11:070
3157299831,70cyclictest25135-21sh22:11:070
3157299831,70cyclictest0-21swapper/022:00:150
3157299831,69cyclictest25997-21rm23:36:520
3157299831,69cyclictest25997-21rm23:36:520
3157299831,68cyclictest31070-21sh23:01:080
3157299831,66cyclictest19926-21wc20:05:460
3157299831,48cyclictest0-21swapper/019:55:160
3157299822,61cyclictest0-21swapper/021:15:340
3157299821,73cyclictest0-21swapper/021:34:350
3157299821,70cyclictest19362-21ps00:10:420
3157299821,69cyclictest9726-21ssh23:58:440
3157299821,68cyclictest0-21swapper/000:20:320
3157299812,67cyclictest0-21swapper/023:31:240
3157299811,69cyclictest0-21swapper/020:50:330
3157299811,65cyclictest31354-21sh23:45:030
3157299802,26cyclictest13702-21munin-run21:15:160
3157299801,71cyclictest17452-21sh22:44:330
3157299801,62cyclictest0-21swapper/000:25:590
3157299791,63cyclictest1884-21ssh00:31:250
3157299791,60cyclictest8995-21ssh21:50:380
3157299791,60cyclictest0-21swapper/020:25:060
3157299791,39cyclictest24438-21rm22:53:090
3157299781,68cyclictest9865-21diskmemload00:17:120
3157299781,66cyclictest5831-21taskset19:26:170
3157299781,64cyclictest2937-21latency_hist23:50:160
3157299781,59cyclictest11381-21runrttasks19:43:160
3157299771,68cyclictest10992-21ssh23:17:380
3157299771,65cyclictest29411-21ssh22:17:030
3157299771,62cyclictest0-21swapper/019:36:410
3157299771,59cyclictest5140-21ssh22:27:510
3157299771,59cyclictest5140-21ssh22:27:510
3157299761,61cyclictest26206-21chrt20:25:250
3157299761,60cyclictest3097-21ssh21:42:530
3157399741,52cyclictest26294-21/usr/sbin/munin22:55:361
3157399741,52cyclictest26294-21/usr/sbin/munin22:55:361
3157299741,62cyclictest31265-21chrt20:39:010
3157299741,57cyclictest6382-21/usr/sbin/munin19:30:450
3157299741,57cyclictest0-21swapper/021:06:510
3157399731,60cyclictest2812-21apt-get22:25:291
3157399731,60cyclictest2812-21apt-get22:25:291
3157399723,52cyclictest19166-21sed21:20:521
3157399721,56cyclictest0-21swapper/120:50:161
3157299721,62cyclictest0-21swapper/020:11:110
3157299721,59cyclictest0-21swapper/019:13:230
3157399717,43cyclictest0-21swapper/100:20:461
3157399711,54cyclictest4250-21ssh23:09:471
3157399711,46cyclictest14278-21/usr/sbin/munin21:15:331
3157399701,54cyclictest22068-21/usr/sbin/munin22:50:321
3157399701,44cyclictest23393-21ssh23:34:261
3157399701,41cyclictest29118-21unixbench_multi20:30:451
3157299701,60cyclictest7729-21runrttasks21:01:190
3157399691,47cyclictest30623-21ssh22:19:311
3157399691,42cyclictest6104-21chrt20:58:301
3157399691,42cyclictest13844-21awk23:20:471
3157399691,41cyclictest27479-21ssh23:39:551
3157399691,41cyclictest27479-21ssh23:39:551
3157299691,65cyclictest0-21swapper/019:20:310
3157299691,58cyclictest23875-21seq20:16:430
3157399681,44cyclictest5729-21timerwakeupswit21:45:461
3157399681,44cyclictest20247-21ssh22:47:231
3157399681,43cyclictest25162-21taskset22:11:091
3157399681,43cyclictest25162-21taskset22:11:091
3157299681,54cyclictest0-21swapper/019:58:030
3157399671,54cyclictest17679-21ssh22:45:001
3157399671,52cyclictest7849-21taskset19:34:071
3157399671,48cyclictest32556-21ssh23:04:131
3157399671,46cyclictest9413-21ssh22:33:491
3157299671,30cyclictest137-21systemd-journal20:44:350
3157399662,43cyclictest318-21ssh21:40:061
3157399661,52cyclictest2366-21ssh22:25:041
3157399661,50cyclictest28708-21ssh21:34:371
3157399661,46cyclictest31392-21sh23:45:081
3157399661,46cyclictest12702-21ssh00:01:481
3157399661,45cyclictest22548-21ssh22:08:411
3157399661,45cyclictest1903-21ssh00:31:291
3157399661,44cyclictest4030-21chrt19:21:211
3157399661,42cyclictest15696-21awk22:00:151
3157399661,40cyclictest5715-21ssh23:53:171
3157399651,55cyclictest6925-21ssh23:12:121
3157399651,49cyclictest5613-21ssh00:36:541
3157399651,45cyclictest3174350ssh00:28:341
3157399651,42cyclictest11008-21ssh23:17:401
3157399651,39cyclictest19161-21sh23:28:381
3157399641,54cyclictest15928-21ssh22:00:191
3157399641,53cyclictest0-21swapper/120:06:141
3157399641,50cyclictest10210-21ssh21:51:491
3157399641,44cyclictest8577-21ssh23:56:201
3157399641,44cyclictest2904-21/usr/sbin/munin20:50:321
3157299641,60cyclictest0-21swapper/020:45:450
31573996318,41cyclictest14653-21/usr/sbin/munin00:05:371
31573996318,41cyclictest14653-21/usr/sbin/munin00:05:371
3157399631,54cyclictest26210-21munin-node20:25:291
3157399631,45cyclictest1808-21runrttasks23:48:071
3157399631,43cyclictest17183-21taskset20:00:251
3157399631,43cyclictest17183-21taskset20:00:251
3157399631,43cyclictest13666-21ssh21:15:141
3157399621,50cyclictest23904-21taskset20:17:061
3157399621,50cyclictest23449-21ssh21:26:391
3157399621,45cyclictest728-21ssh21:40:281
3157399611,42cyclictest2421-21chrt19:18:521
3157399601,46cyclictest22446-21aten_rbpower_vo00:15:311
3157399601,41cyclictest12021-21ssh22:36:161
31573996010,32cyclictest31043-21resolvconf20:36:131
3157399596,40cyclictest0-21swapper/100:12:231
3157399581,43cyclictest9691-21taskset19:39:381
3157399581,37cyclictest22028-21taskset20:11:121
144812580,4sleep214478-21latency_hist00:05:162
3157399561,44cyclictest0-21swapper/119:29:171
3157399561,42cyclictest666-21seq20:44:371
202132560,1sleep30-21swapper/322:47:193
3157399551,47cyclictest12606-21/usr/sbin/munin19:45:451
3157399551,44cyclictest0-21swapper/121:06:581
311782552,13sleep00-21swapper/019:08:360
3157399542,42cyclictest0-21swapper/119:41:591
3157399541,48cyclictest0-21swapper/119:10:171
3157399541,41cyclictest7763-21chrt21:01:451
3157399541,37cyclictest0-21swapper/119:55:201
3157599523,14cyclictest31423-21ssh22:20:283
3157399521,48cyclictest24433-21apt-get20:20:331
3157399521,47cyclictest14504-21ls19:50:431
3157499515,10cyclictest31805-21ssh23:45:302
3157599491,9cyclictest10474-21ssh22:35:283
3157599488,26cyclictest0-21swapper/300:20:323
3157599477,26cyclictest0-21swapper/320:45:343
3157599477,25cyclictest0-21swapper/320:05:333
3157599476,26cyclictest9865-21diskmemload00:35:303
3157599457,25cyclictest0-21swapper/323:10:213
31574994519,6cyclictest1615050wget22:00:322
3157599448,20cyclictest0-21swapper/323:55:383
3157599427,25cyclictest0-21swapper/320:50:213
3157599418,4cyclictest0-21swapper/323:51:343
3157599417,24cyclictest0-21swapper/322:50:323
3157599412,8cyclictest7850irq/56-dwc_otg00:15:283
3157499417,8cyclictest7950irq/56-dwc_otg_23:07:052
3157499417,24cyclictest0-21swapper/220:00:322
3157499417,24cyclictest0-21swapper/220:00:322
31574994119,5cyclictest20593-21grep22:05:412
31574994115,14cyclictest1108-21aten_rbpower_vo20:45:352
281172410,20sleep38050irq/56-dwc_otg_22:15:383
3157599407,23cyclictest0-21swapper/300:10:303
3157499408,23cyclictest0-21swapper/222:50:322
3157499407,23cyclictest0-21swapper/222:25:322
3157499407,23cyclictest0-21swapper/222:25:322
3157499401,3cyclictest0-21swapper/223:55:272
3157599391,28cyclictest10674-21ssh00:00:173
3157599391,24cyclictest23563-21ls20:15:463
3157499399,14cyclictest0-21swapper/223:05:152
3157499397,23cyclictest0-21swapper/221:55:302
3157499391,8cyclictest7850irq/56-dwc_otg21:40:302
3157599387,22cyclictest0-21swapper/322:25:323
3157599387,22cyclictest0-21swapper/322:25:323
3157599381,25cyclictest7043-21ls22:30:353
3157599381,25cyclictest12515-21cut21:55:353
3157499387,21cyclictest0-21swapper/223:20:312
3157499381,7cyclictest0-21swapper/222:10:262
3157499381,7cyclictest0-21swapper/222:10:262
3157499381,19cyclictest8050irq/56-dwc_otg_21:15:452
227162380,2sleep20-21swapper/200:15:352
164612380,2sleep30-21swapper/322:42:323
31575993724,5cyclictest31316-21ssh00:27:263
31575993714,5cyclictest24385-21ntp_kernel_pll_22:10:373
31575993714,5cyclictest24385-21ntp_kernel_pll_22:10:373
31575993714,4cyclictest878-21runrttasks00:05:383
31575993714,4cyclictest878-21runrttasks00:05:383
31575993713,4cyclictest1989550wget22:05:333
31574993720,4cyclictest24733-21sh23:35:372
31574993720,4cyclictest24733-21sh23:35:372
3157499371,29cyclictest0-21swapper/223:26:532
3157599363,27cyclictest7950irq/56-dwc_otg_22:59:383
3157599363,27cyclictest7950irq/56-dwc_otg_22:59:383
3157499362,6cyclictest8050irq/56-dwc_otg_00:10:282
31574993616,4cyclictest29-21rcuc/200:20:412
161732360,2sleep20-21swapper/200:06:132
161732360,2sleep20-21swapper/200:06:132
3157599357,24cyclictest0-21swapper/323:15:313
3157599351,18cyclictest17100-21ls20:00:163
3157499358,11cyclictest8050irq/56-dwc_otg_23:00:162
3157499358,11cyclictest8050irq/56-dwc_otg_23:00:162
3157499357,6cyclictest7950irq/56-dwc_otg_21:50:462
31574993520,5cyclictest18248-21fschecks_time21:20:392
31574993514,13cyclictest29-21rcuc/222:25:172
3157599347,9cyclictest0-21swapper/321:45:263
3157599347,15cyclictest0-21swapper/321:35:333
3157599347,15cyclictest0-21swapper/319:50:313
3157599346,18cyclictest12632-21ssh23:20:313
3157599345,18cyclictest8050irq/56-dwc_otg_23:45:473
3157599341,5cyclictest0-21swapper/319:10:313
3157599341,24cyclictest21799-21ls20:10:463
3157599341,20cyclictest29952-21fschecks_time23:00:353
3157599341,20cyclictest23813-21kworker/u8:321:21:073
3157499347,11cyclictest9844-21kworker/u8:121:40:162
31574993414,18cyclictest1906150wget20:05:332
31574993412,6cyclictest8418-21cron23:15:162
31574993412,12cyclictest9865-21diskmemload23:45:152
310552342,25sleep10-21swapper/119:07:081
108482340,3sleep3381ktimersoftd/321:10:403
3157599339,6cyclictest0-21swapper/323:25:403
3157599335,15cyclictest0-21swapper/319:45:453
3157599332,21cyclictest0-21swapper/320:57:473
31575993315,4cyclictest878-21runrttasks23:42:433
3157599331,17cyclictest8050irq/56-dwc_otg_00:30:313
31575993310,4cyclictest7950irq/56-dwc_otg_21:10:163
31574993316,5cyclictest31-21ksoftirqd/223:50:472
3157499331,5cyclictest1833250ssh22:45:312
31574993310,5cyclictest8050irq/56-dwc_otg_19:40:152
31574993310,5cyclictest0-21swapper/222:40:282
307042335,13sleep330809-21/usr/sbin/munin19:05:443
246182331,4sleep224627-21sh21:29:042
3157599328,12cyclictest0-21swapper/321:51:263
3157599327,5cyclictest7950irq/56-dwc_otg_21:25:343
3157599327,23cyclictest0-21swapper/321:40:323
3157599326,14cyclictest0-21swapper/320:00:323
3157599326,14cyclictest0-21swapper/320:00:323
3157599322,5cyclictest8248-21aten_rbpower_en19:35:293
3157599322,20cyclictest0-21swapper/319:29:443
31575993215,6cyclictest308-21sed20:40:503
3157599321,16cyclictest9865-21diskmemload23:35:363
3157599321,16cyclictest9865-21diskmemload23:35:363
3157499328,6cyclictest7850irq/56-dwc_otg20:40:342
3157499325,4cyclictest7950irq/56-dwc_otg_20:30:332
3157499325,12cyclictest7950irq/56-dwc_otg_22:30:322
3157499324,15cyclictest0-21swapper/223:15:412
3157499322,6cyclictest7850irq/56-dwc_otg19:40:492
3157499322,14cyclictest8050irq/56-dwc_otg_22:15:292
31574993217,12cyclictest8050irq/56-dwc_otg_22:40:162
31574993215,4cyclictest29-21rcuc/220:00:162
3157599317,15cyclictest0-21swapper/322:00:303
3157599314,6cyclictest6424-21wget19:30:303
3157499317,4cyclictest7950irq/56-dwc_otg_21:45:372
3157499317,10cyclictest7850irq/56-dwc_otg21:10:332
3157499316,5cyclictest0-21swapper/200:35:362
3157499315,7cyclictest6788-21sort21:00:372
3157499313,6cyclictest7850irq/56-dwc_otg19:25:382
3157499312,5cyclictest32444-21cron00:30:152
3157499311,7cyclictest14343-21ps19:50:422
3157499311,5cyclictest20440-21cut23:30:332
31574993112,4cyclictest878-21runrttasks19:17:212
3157499311,20cyclictest12679-21switchtime19:45:462
31574993111,11cyclictest0-21swapper/221:34:242
3157599308,11cyclictest7950irq/56-dwc_otg_21:00:353
3157599305,12cyclictest7950irq/56-dwc_otg_19:22:583
3157599304,4cyclictest8050irq/56-dwc_otg_20:25:303
3157599303,8cyclictest7950irq/56-dwc_otg_20:30:303
3157599303,12cyclictest8050irq/56-dwc_otg_20:25:163
3157599302,6cyclictest0-21swapper/323:05:313
31575993022,5cyclictest10266-21fschecks_count19:40:403
3157599302,19cyclictest0-21swapper/319:19:093
3157499302,15cyclictest0-21swapper/220:25:312
3157499301,9cyclictest32720-21grep19:10:452
3157599298,4cyclictest0-21swapper/321:19:073
3157499296,7cyclictest7950irq/56-dwc_otg_20:10:332
3157499296,15cyclictest0-21swapper/220:50:212
3157499292,12cyclictest0-21swapper/220:20:312
31574992916,4cyclictest29-21rcuc/220:55:372
3157599288,6cyclictest8050irq/56-dwc_otg_20:35:303
3157599284,4cyclictest7850irq/56-dwc_otg23:30:393
3157499284,21cyclictest22775-21cpuspeed20:15:342
3157499282,6cyclictest30074-21fschecks_count20:35:332
3157499273,10cyclictest7950irq/56-dwc_otg_19:22:312
3157499272,14cyclictest8050irq/56-dwc_otg_19:30:422
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional