You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-21 - 19:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

ARMv7 Processor rev 4 (v7l), Linux 4.18.16-rt9 (Profile)

Latency plot of shadow in rack #7, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack7slot3s.osadl.org (updated Thu Oct 21, 2021 12:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18565991071,5cyclictest8050irq/56-dwc_otg_08:55:341
18564991031,80cyclictest29130-21apt-get11:50:360
78792960,3sleep20-21swapper/208:10:202
1856499941,84cyclictest0-21swapper/011:40:550
182212900,3sleep118194-21ssh11:35:541
18564998933,43cyclictest29612-21fschecks_time09:50:510
1856499841,69cyclictest0-21swapper/011:45:470
1856499841,68cyclictest0-21swapper/007:30:500
1856499831,57cyclictest0-21swapper/009:30:300
1856499812,67cyclictest0-21swapper/012:20:370
1856499812,25cyclictest0-21swapper/008:05:360
1856499811,65cyclictest29456-21ssh12:30:370
1856499806,65cyclictest0-21swapper/011:15:250
18564997917,14cyclictest7655-21sh10:04:030
18564997917,14cyclictest7655-21sh10:04:030
1856499791,30cyclictest9113-21vmstat08:10:530
130492790,4sleep213053-21fschecks_count10:50:382
1856499783,65cyclictest362-21apt-config11:15:200
18564997826,4cyclictest25420-21fschecks_count09:45:410
18564997826,4cyclictest25420-21fschecks_count09:45:410
1856499762,26cyclictest0-21swapper/010:26:140
1856499732,61cyclictest29354-21diskmemload11:00:540
1856499732,60cyclictest0-21swapper/010:59:510
1856499731,63cyclictest18299-21rm10:16:170
1856499731,52cyclictest0-21swapper/009:10:350
18564997312,15cyclictest0-21swapper/011:55:440
1856499721,29cyclictest0-21swapper/011:31:100
66912710,2sleep20-21swapper/212:01:592
66912710,2sleep20-21swapper/212:01:592
41332710,2sleep30-21swapper/311:19:453
1856499712,59cyclictest0-21swapper/008:47:090
1856499712,59cyclictest0-21swapper/008:47:090
1856499712,15cyclictest0-21swapper/009:25:520
18564997117,10cyclictest0-21swapper/007:52:180
1856499711,59cyclictest0-21swapper/011:26:170
1856499711,57cyclictest0-21swapper/011:36:020
18564997029,31cyclictest134-21systemd-journal08:41:260
1856499702,28cyclictest0-21swapper/012:09:490
1856499701,27cyclictest0-21swapper/009:20:410
1856499699,16cyclictest0-21swapper/010:14:250
1856499692,52cyclictest0-21swapper/012:15:350
1856499692,52cyclictest0-21swapper/012:15:350
18564996917,15cyclictest0-21swapper/008:33:160
1856499691,59cyclictest0-21swapper/010:46:480
1856499691,59cyclictest0-21swapper/010:44:290
18564996913,10cyclictest8481-21latency_hist12:05:210
18564996913,10cyclictest8481-21latency_hist12:05:210
1856499691,29cyclictest5660-21ps11:20:480
1856499691,29cyclictest5660-21ps11:20:480
1856499691,28cyclictest0-21swapper/009:37:010
1856499691,27cyclictest32508-21chrt07:46:560
1856499691,27cyclictest134-21systemd-journal09:17:390
1856499691,27cyclictest134-21systemd-journal09:17:390
1856499690,30cyclictest16129-21rm12:14:220
1856499687,11cyclictest4080-21awk08:00:200
1856499685,24cyclictest10635-21timerandwakeup08:15:510
1856499682,63cyclictest1432-21latency12:35:430
18564996720,8cyclictest0-21swapper/010:34:450
18564996718,11cyclictest0-21swapper/012:28:590
18564996716,9cyclictest8050irq/56-dwc_otg_08:55:480
1856499671,54cyclictest0-21swapper/010:37:110
18564996619,9cyclictest8050irq/56-dwc_otg_11:05:440
18564996619,9cyclictest8050irq/56-dwc_otg_11:05:440
18564996617,27cyclictest27549-21idleruntime-cro09:05:200
1856499661,29cyclictest0-21swapper/010:53:530
1856499661,28cyclictest10268-21run-parts10:06:210
1856499661,21cyclictest0-21swapper/009:41:440
1856499661,21cyclictest0-21swapper/007:25:500
1856499651,49cyclictest9-21ksoftirqd/007:20:530
1856499651,49cyclictest9-21ksoftirqd/007:20:530
1856499651,27cyclictest0-21swapper/009:56:010
1856499641,26cyclictest22649-21ssh10:21:490
1856599636,10cyclictest0-21swapper/111:03:531
18565996213,6cyclictest26028-21wc12:25:501
18566996110,13cyclictest25084-21ssh11:45:332
1856599613,5cyclictest7950irq/56-dwc_otg_11:40:581
1856599613,4cyclictest8050irq/56-dwc_otg_08:05:401
1856599613,3cyclictest7950irq/56-dwc_otg_07:55:201
1856599612,5cyclictest4071-21munin-run08:00:201
18565996124,34cyclictest8718-21ssh09:25:061
18565996118,3cyclictest0-21swapper/111:15:261
1856499612,45cyclictest0-21swapper/009:05:520
18564996113,16cyclictest4386-21aten_rbpower_po08:00:370
18564996113,16cyclictest4386-21aten_rbpower_po08:00:370
1856599605,10cyclictest0-21swapper/110:59:431
1856499601,27cyclictest0-21swapper/008:38:490
1856499599,11cyclictest0-21swapper/007:12:370
1856499594,16cyclictest0-21swapper/007:44:170
1856599581,6cyclictest12828-21rm09:30:201
1856599581,10cyclictest0-21swapper/110:06:351
18564995815,15cyclictest0-21swapper/008:25:140
1856599571,5cyclictest8050irq/56-dwc_otg_09:45:451
1856599571,5cyclictest8050irq/56-dwc_otg_09:45:451
1856499571,17cyclictest0-21swapper/008:25:370
1856599556,9cyclictest29254-21cut11:50:371
1856499554,39cyclictest9-21ksoftirqd/008:50:570
1856499551,37cyclictest0-21swapper/007:35:370
278052540,2sleep20-21swapper/209:48:262
278052540,2sleep20-21swapper/209:48:262
1856699549,8cyclictest8050irq/56-dwc_otg_11:25:332
1856699544,9cyclictest7850irq/56-dwc_otg07:45:342
1856599547,4cyclictest6092-1kworker/u9:210:11:131
1856599546,4cyclictest0-21swapper/110:48:371
1856599541,5cyclictest1-21systemd11:45:491
1856599537,43cyclictest9319-21ls12:05:401
18565995312,5cyclictest28339-21sh10:29:551
1856599521,11cyclictest0-21swapper/110:32:321
1856499521,40cyclictest0-21swapper/007:16:020
177372504,31sleep00-21swapper/007:05:530
61782490,36sleep38050irq/56-dwc_otg_10:01:113
61782490,36sleep38050irq/56-dwc_otg_10:01:113
1856599492,14cyclictest22813-21ssh10:22:081
1856799488,25cyclictest0-21swapper/309:05:383
1856799483,31cyclictest37-21rcuc/308:55:343
18566994718,11cyclictest8050irq/56-dwc_otg_09:35:562
1856599462,5cyclictest8050irq/56-dwc_otg_09:50:441
18565994610,23cyclictest0-21swapper/108:10:361
18565994519,5cyclictest16739-21latency_hist11:35:201
35992430,3sleep231-21ksoftirqd/209:17:422
35992430,3sleep231-21ksoftirqd/209:17:422
195942430,18sleep11856599cyclictest08:40:501
1856699436,25cyclictest0-21swapper/211:30:252
1856699436,23cyclictest0-21swapper/210:00:292
1856699436,23cyclictest0-21swapper/210:00:292
1856599436,25cyclictest0-21swapper/107:20:371
1856599436,25cyclictest0-21swapper/107:20:371
18567994218,5cyclictest25647-21if_wlan009:45:443
18567994218,5cyclictest25647-21if_wlan009:45:443
1856699427,23cyclictest0-21swapper/212:15:382
1856699427,23cyclictest0-21swapper/212:15:382
1856699426,23cyclictest0-21swapper/208:15:362
1856599421,22cyclictest4937-21ssh09:20:201
1856599421,22cyclictest4937-21ssh09:20:201
1856699417,23cyclictest0-21swapper/207:15:422
1856699416,9cyclictest0-21swapper/211:05:052
18565994125,4cyclictest1126-21snmpd09:58:411
18567994010,13cyclictest7950irq/56-dwc_otg_12:35:203
1856699407,21cyclictest0-21swapper/208:45:342
1856699407,21cyclictest0-21swapper/208:45:342
1856699405,27cyclictest29152-21ssh11:50:362
289142390,5sleep21856699cyclictest07:37:042
18567993917,4cyclictest434350wget08:00:363
18567993917,4cyclictest434350wget08:00:363
1856699392,4cyclictest0-21swapper/207:30:322
18566993919,9cyclictest8050irq/56-dwc_otg_10:47:412
1856599396,23cyclictest0-21swapper/108:00:361
1856599396,23cyclictest0-21swapper/108:00:361
1856799387,22cyclictest0-21swapper/312:00:373
1856799387,22cyclictest0-21swapper/312:00:373
1856799385,20cyclictest16775-21ssh10:15:363
18567993818,5cyclictest1115-21runrttasks09:22:463
1856799381,27cyclictest381ktimersoftd/312:20:483
1856699385,5cyclictest0-21swapper/211:05:472
1856699385,5cyclictest0-21swapper/211:05:472
1856599383,5cyclictest8050irq/56-dwc_otg_09:35:481
18565993819,8cyclictest8050irq/56-dwc_otg_12:05:211
18565993819,8cyclictest8050irq/56-dwc_otg_12:05:211
18565993819,6cyclictest29029-21sh11:10:351
1856599381,23cyclictest7850irq/56-dwc_otg07:15:581
197112370,6sleep11856599cyclictest12:18:251
197112370,6sleep11856599cyclictest12:18:251
1856799371,9cyclictest0-21swapper/307:40:363
18567993717,5cyclictest11852-21sh11:29:063
18567993712,16cyclictest1011-21fschecks_time10:35:423
18566993730,5cyclictest20853-21rm10:20:362
1856699371,20cyclictest7850irq/56-dwc_otg09:05:522
1856699371,10cyclictest30526-21grep07:40:542
1856599375,19cyclictest12984-21ssh12:10:361
1856599373,6cyclictest8050irq/56-dwc_otg_08:30:211
18565993717,10cyclictest5658-21sh11:20:481
18565993717,10cyclictest5658-21sh11:20:481
18565993714,14cyclictest0-21swapper/107:35:361
311262361,2sleep10-21swapper/112:31:581
1856799363,10cyclictest8050irq/56-dwc_otg_12:11:423
18567993622,4cyclictest39-21ksoftirqd/310:58:533
1856799361,7cyclictest7672-21modprobe08:10:193
1856699365,4cyclictest7850irq/56-dwc_otg11:40:202
1856599368,6cyclictest7950irq/56-dwc_otg_09:15:211
1856599368,11cyclictest8050irq/56-dwc_otg_10:20:201
1856599367,25cyclictest0-21swapper/109:05:381
1856599362,22cyclictest0-21swapper/111:25:381
18565993615,5cyclictest29427-21/usr/sbin/munin07:40:491
1856799358,15cyclictest14938-21ssh09:31:433
1856799353,18cyclictest8940-21timerwakeupswit08:10:513
18567993528,4cyclictest2603-21cut07:55:373
1856799352,18cyclictest381ktimersoftd/312:35:493
1856799351,18cyclictest1307850ssh10:50:393
1856699355,5cyclictest24466-21fschecks_time08:55:412
18566993526,5cyclictest30270-21grep11:10:512
18566993518,4cyclictest1115-21runrttasks12:33:452
1856699351,4cyclictest0-21swapper/207:55:352
18566993513,5cyclictest78650wget11:55:402
1856699351,23cyclictest8271-21ls08:10:412
1856699351,22cyclictest24807-21ssh09:45:202
1856599355,20cyclictest8050irq/56-dwc_otg_10:00:381
1856599355,20cyclictest8050irq/56-dwc_otg_10:00:381
18565993519,9cyclictest16715-21ssh09:35:161
18565993513,8cyclictest99242chrt08:15:391
160592350,2sleep10-21swapper/108:30:521
1856799345,20cyclictest20886-21ssh11:00:363
1856799345,18cyclictest19014-21fschecks_count08:40:413
1856799342,13cyclictest0-21swapper/307:19:153
18567993418,6cyclictest2441850ssh11:44:423
18567993415,4cyclictest605-21aten_rbpower_cu07:50:343
18567993414,5cyclictest29354-21diskmemload11:39:223
18567993414,10cyclictest8050irq/56-dwc_otg_10:45:423
1856799341,21cyclictest0-21swapper/311:20:583
1856799341,21cyclictest0-21swapper/311:20:583
1856799341,15cyclictest0-21swapper/309:10:453
1856699349,5cyclictest0-21swapper/212:29:312
1856699349,15cyclictest0-21swapper/210:35:262
1856699345,5cyclictest8050irq/56-dwc_otg_09:51:042
1856699345,20cyclictest852-21ssh12:35:362
18566993420,4cyclictest8050irq/56-dwc_otg_10:00:202
18566993417,8cyclictest24796-21sh10:25:252
18566993412,5cyclictest134-21systemd-journal09:25:562
1856599347,4cyclictest0-21swapper/112:35:551
18565993421,4cyclictest8050irq/56-dwc_otg_10:44:251
1856599341,17cyclictest17646-21grep08:35:491
153142340,2sleep30-21swapper/310:13:153
1856799337,4cyclictest12257-21sh10:10:063
1856799331,23cyclictest17535-21ls08:35:473
1856799331,20cyclictest25453-21cut12:25:423
1856799331,18cyclictest4378-21latency_hist10:00:203
1856799331,16cyclictest22148-21grep10:20:553
1856799331,12cyclictest29547-21grep09:50:513
18567993310,12cyclictest0-21swapper/311:08:363
18567993310,12cyclictest0-21swapper/311:08:363
1856699337,4cyclictest0-21swapper/211:20:522
1856699337,4cyclictest0-21swapper/211:20:522
18566993317,9cyclictest18760-21/usr/sbin/munin08:40:402
18566993312,4cyclictest8050irq/56-dwc_otg_12:20:362
1856599337,3cyclictest0-21swapper/108:20:441
1856599335,19cyclictest21098-21ssh12:20:341
1856599335,15cyclictest21-21rcuc/109:41:031
1856799327,7cyclictest18384-21tail09:35:573
1856799324,5cyclictest1126-21snmpd12:09:053
1856799322,20cyclictest0-21swapper/312:20:213
1856799322,20cyclictest0-21swapper/312:20:213
1856799321,6cyclictest29595-21ssh10:30:473
1856799321,22cyclictest0-21swapper/310:28:423
18567993211,12cyclictest10036-21ssh09:25:553
18566993214,5cyclictest22303-21/usr/sbin/munin07:20:382
18566993214,5cyclictest22303-21/usr/sbin/munin07:20:382
1856699321,3cyclictest0-21swapper/209:30:262
1856599325,4cyclictest7850irq/56-dwc_otg11:55:561
18565993212,8cyclictest0-21swapper/110:35:441
1856599321,18cyclictest0-21swapper/107:45:521
182122323,22sleep10-21swapper/107:09:181
1856799317,14cyclictest8050irq/56-dwc_otg_09:40:373
1856799313,12cyclictest8050irq/56-dwc_otg_11:30:213
1856799312,9cyclictest0-21swapper/311:47:383
1856799312,6cyclictest7850irq/56-dwc_otg07:25:423
18567993115,6cyclictest8050irq/56-dwc_otg_10:40:363
1856799311,20cyclictest0-21swapper/311:11:433
1856799311,18cyclictest31840-21wc07:45:483
1856799311,12cyclictest29177-21wget11:50:363
1856699319,12cyclictest0-21swapper/209:22:372
1856699318,6cyclictest14498-21unixbench_singl08:25:562
1856699317,14cyclictest0-21swapper/212:05:272
1856699316,5cyclictest941-21idleruntime07:50:412
1856699312,8cyclictest13074-21fschecks_count10:10:422
1856699312,5cyclictest0-21swapper/211:15:442
1856699312,5cyclictest0-21swapper/208:25:202
18566993124,5cyclictest17141-21fschecks_time10:15:432
18566993113,4cyclictest0-21swapper/211:40:432
1856699311,21cyclictest11086-21seq10:07:522
1856599312,3cyclictest0-21swapper/111:09:041
1856599312,3cyclictest0-21swapper/111:09:041
18565993120,8cyclictest0-21swapper/110:50:351
70742300,5sleep21856699cyclictest10:42:492
1856799303,15cyclictest13439-21wget08:25:383
1856799302,19cyclictest0-21swapper/309:15:273
1856799302,19cyclictest0-21swapper/309:15:273
18567993013,4cyclictest1115-21runrttasks08:50:223
1856799301,18cyclictest11278-21modprobe08:20:203
1856699308,4cyclictest7950irq/56-dwc_otg_10:55:502
1856699306,7cyclictest8050irq/56-dwc_otg_07:15:202
1856699306,5cyclictest8050irq/56-dwc_otg_09:10:492
1856699306,5cyclictest8050irq/56-dwc_otg_08:50:422
1856699305,7cyclictest7950irq/56-dwc_otg_10:30:392
18566993020,7cyclictest16695-21cron08:35:202
1856699301,6cyclictest25986-21tr09:00:352
1856599302,4cyclictest7950irq/56-dwc_otg_07:10:571
18565993013,4cyclictest25704-21latency_hist07:30:201
1856599301,15cyclictest0-21swapper/109:00:391
1856799298,12cyclictest8050irq/56-dwc_otg_07:35:353
1856799297,14cyclictest8050irq/56-dwc_otg_09:00:533
1856799296,14cyclictest0-21swapper/308:45:343
1856799296,14cyclictest0-21swapper/308:45:343
1856799293,16cyclictest23090-21switchtime07:20:513
1856799293,16cyclictest23090-21switchtime07:20:513
1856799291,7cyclictest18609-21apt-get07:10:443
1856799291,4cyclictest15264-21aten_rbpower_vo08:30:383
1856699297,4cyclictest7950irq/56-dwc_otg_07:26:282
1856699295,5cyclictest7850irq/56-dwc_otg12:10:412
1856699291,18cyclictest18520-21latency_hist08:40:202
1856599295,6cyclictest0-21swapper/107:30:381
1856599291,11cyclictest0-21swapper/108:45:341
1856599291,11cyclictest0-21swapper/108:45:341
1856799282,4cyclictest711-21apt-get11:55:353
1856799282,4cyclictest11540-21apt-get08:20:333
18567992824,2cyclictest0-21swapper/307:30:483
1856699285,21cyclictest4636-21uniq08:00:422
1856699285,21cyclictest4636-21uniq08:00:422
174112280,2sleep20-21swapper/207:05:462
1856599261,10cyclictest0-21swapper/108:50:521
180452264,14sleep30-21swapper/307:07:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional