Updated to version v1.8.0 (#15)

Updated to version v1.8.0

Author: aijunpeng

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,30 +1,33 @@
 ---
 name: Bug report
 about: Create a report to help us improve
-title: ''
+title: ""
 labels: bug
-assignees: ''
-
+assignees: ""
 ---
 
 **Describe the bug**
+
 <!--- A clear and concise description of what the bug is -->
 
 **To Reproduce**
+
 <!--- Steps to reproduce the behavior -->
 
 **Expected behavior**
+
 <!--- A clear and concise description of what you expected to happen -->
 
 **Please complete the following information about the solution:**
+
 - [ ] Version: [e.g. v1.0.0]
 
-To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "_(SO0134) - The AWS CloudFormation template for deployment of the AWS DevOps Monitoring Dashboard. Version **v1.0.0**_". You can also find the version from [releases](https://github.com/aws-solutions/aws-devops-monitoring-dashboard/releases)
+To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "_(SO0143) - The AWS CloudFormation template for deployment of the DevOps Monitoring Dashboard on AWS. Version **v1.0.0**_". You can also find the version from [releases](https://github.com/aws-solutions/aws-devops-monitoring-dashboard/releases)
 
 - [ ] Region: [e.g. us-east-1]
 - [ ] Was the solution modified from the version published on this repository?
 - [ ] If the answer to the previous question was yes, are the changes available on GitHub?
-- [ ] Have you checked your [service quotas](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for the sevices this solution uses?
+- [ ] Have you checked your [service quotas](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for the services this solution uses?
 - [ ] Were there any errors in the CloudWatch Logs? [How to enable debug mode?](https://github.com/aws-solutions/aws-devops-monitoring-dashboard/#enable-debug-mode)
 
 **Screenshots**

CHANGELOG.md

@@ -4,10 +4,15 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.8.0] - 2022-10-31
+### Added
+- Added multi-account multi-region data ingestion
+- Added tag filter for AWS CodeCommit, CodeBuild and CodePipeline
+
 ## [1.5.0] - 2022-04-19
 ### Added
 - Added GitHub integration - GitHub activity metric for push events
-- Added Mean Time to Recovery (MTTR) metric for Code Pipeline 
+- Added Mean Time to Recovery (MTTR) metric for Code Pipeline
 
 ## [1.1.0] - 2021-06-16
 ### Added

CONTRIBUTING.md

@@ -23,7 +23,7 @@ reported the issue. Please try to include as much information as you can. Detail
 ## Contributing via Pull Requests
 Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
 
-1. You are working against the latest source on the *master* branch.
+1. You are working against the latest source on the *main* branch.
 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
 3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
 

README.md

@@ -69,7 +69,7 @@ do_replace() {
 # Ex:
 # #!/bin/bash
 # SOLUTION_ID='SO0134'
-# SOLUTION_NAME='AWS DevOps Monitoring Dashboard'
+# SOLUTION_NAME='DevOps Monitoring Dashboard on AWS'
 # SOLUTION_TRADEMARKEDNAME='aws-devops-monitoring-dashboard'
 if [[ -e './solution_env.sh' ]]; then
     chmod +x ./solution_env.sh
@@ -158,9 +158,19 @@ for folder in */ ; do
     done
 
     if [ -e "requirements.txt" ]; then
-        pip3 install -q -r requirements.txt --upgrade --target ./
+        echo "Installing python lambda using virtual environment"
+        python3 -m venv .venv-test
+        echo "Activating virtual environment"
+        source .venv-test/bin/activate
+        echo "Executing pip3 install -q -r requirements.txt --upgrade --target ./"
+        python3 -m pip install -q -r requirements.txt --upgrade --target ./
+        echo "Deactivating virtual environment"
+        deactivate
+        echo "Deleting python virtual environment"
+        rm -fr .venv-test
     elif [ -e "package.json" ]; then
-        # do_cmd npm ci --only=prod
+        echo "Installing node dependencies"
+        echo "Executing do_cmd npm install --production"
         do_cmd npm install --production
     fi
 

deployment/build-s3-dist.sh

@@ -15,85 +15,61 @@ used by the AWS Solutions publishing pipeline.
 These placeholders are then replaced with the appropriate values using the default find/replace operation run by the pipeline.
 
 Before:
+
 ```
 "examplefunction67F55935": {
-      "Type": "AWS::Lambda::Function",
-      "Properties": {
-        "Code": {
-          "S3Bucket": {
-            "Ref": "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7S3Bucket54E71A95"
-          },
-          "S3Key": {
-            "Fn::Join": [
-              "",
-              [
-                {
-                  "Fn::Select": [
-                    0,
-                    {
-                      "Fn::Split": [
-                        "||",
-                        {
-                          "Ref": "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7S3VersionKeyC789D8B1"
-                        }
-                      ]
-                    }
-                  ]
-                },
-                {
-                  "Fn::Select": [
-                    1,
-                    {
-                      "Fn::Split": [
-                        "||",
-                        {
-                          "Ref": "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7S3VersionKeyC789D8B1"
-                        }
-                      ]
-                    }
-                  ]
-                }
-              ]
-            ]
-          }
-        }, ...
-        Handler: "index.handler", ...
+  "Type": "AWS::Lambda::Function",
+  "Properties": {
+  "Code": {
+    "S3Bucket": {
+      "Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
+    },
+    "S3Key": "5efbeaf6fad70c6f155399d9bb43debfa8efd4a4c9b5e055a103fe7b12754e76.zip"
+  },
+  "Handler": "index.handler",
+  "Runtime": "nodejs16.x"
+  },
+  ...
+}
 ```
 
 After helper function run:
+
 ```
 "examplefunction67F55935": {
-      "Type": "AWS::Lambda::Function",
-      "Properties": {
-        "Code": {
-          "S3Bucket": "%%BUCKET_NAME%%",
-          "S3Key": "%%SOLUTION_NAME%%/%%VERSION%%/assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7.zip"
-        }, ...
-        "Handler": "assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7/index.handler"
+  "Type": "AWS::Lambda::Function",
+  "Properties": {
+    "Code": {
+      "S3Bucket": "%%BUCKET_NAME%%",
+      "S3Key": "%%SOLUTION_NAME%%/%%VERSION%%/assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7.zip"
+    }, ...
+    "Handler": "index.handler"
 ```
 
 After build script run:
+
 ```
 "examplefunction67F55935": {
-      "Type": "AWS::Lambda::Function",
-      "Properties": {
-        "Code": {
-          "S3Bucket": "solutions",
-          "S3Key": "trademarked-solution-name/v1.0.0/asset.d513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7.zip"
-        }, ...
-        "Handler": "assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7/index.handler"
+  "Type": "AWS::Lambda::Function",
+  "Properties": {
+    "Code": {
+      "S3Bucket": "solutions",
+      "S3Key": "trademarked-solution-name/v1.0.0/assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7.zip"
+    }, ...
+    "Handler": "index.handler"
 ```
 
 After CloudFormation deployment:
+
 ```
 "examplefunction67F55935": {
-      "Type": "AWS::Lambda::Function",
-      "Properties": {
-        "Code": {
-          "S3Bucket": "solutions-us-east-1",
-          "S3Key": "trademarked-solution-name/v1.0.0/asset.d513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7.zip"
-        }, ...
-        "Handler": "assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7/index.handler"
+  "Type": "AWS::Lambda::Function",
+  "Properties": {
+    "Code": {
+      "S3Bucket": "solutions-us-east-1",
+      "S3Key": "trademarked-solution-name/v1.0.0/assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7.zip"
+    }, ...
+    "Handler": "index.handler"
 ```
 
 #### Template cleanup
@@ -103,6 +79,7 @@ been used to specify Lambda source code properties. This allows solution-specifi
 removes unnecessary clutter.
 
 Before:
+
 ```
 "Parameters": {
     "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7S3Bucket54E71A95": {
@@ -129,9 +106,10 @@ Before:
         "Type" : "String"
     }
   }
-  ```
+```
 
 After:
+
 ```
 "Parameters": {
     "CorsEnabled" : {
@@ -146,7 +124,8 @@ After:
         "Type" : "String"
     }
   }
-  ```
+```
+
+---
 
-***
 © Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

deployment/cdk-solution-helper/README.md

@@ -7,6 +7,9 @@ const fs = require('fs');
 // Paths
 const global_s3_assets = '../global-s3-assets';
 
+//this regular express also takes into account lambda functions defined in nested stacks
+const _regex = /[\w]*AssetParameters/g;
+
 // For each template in global_s3_assets ...
 fs.readdirSync(global_s3_assets).forEach(file => {
     // Import and parse template file
@@ -16,28 +19,50 @@ fs.readdirSync(global_s3_assets).forEach(file => {
     // Clean-up Lambda function code dependencies
     const resources = (template.Resources) ? template.Resources : {};
     const lambdaFunctions = Object.keys(resources).filter(function (key) {
-        return resources[key].Type === 'AWS::Lambda::Function';
+        return (resources[key].Type === 'AWS::Lambda::Function');
     });
 
     lambdaFunctions.forEach(function (f) {
         const fn = template.Resources[f];
-        if (fn.Properties.Code.hasOwnProperty('S3Bucket')) {
+        let prop;
+        if (fn.Properties.hasOwnProperty('Code')) {
+            prop = fn.Properties.Code;
+        } else if (fn.Properties.hasOwnProperty('Content')) {
+            prop = fn.Properties.Content;
+        }
+
+        console.debug(`fn: ${JSON.stringify(fn)}`);
+        console.debug(`prop: ${JSON.stringify(prop)}`);
+
+        if (prop.hasOwnProperty("S3Bucket")) {
             // Set the S3 key reference
-            let artifactHash = Object.assign(fn.Properties.Code.S3Bucket.Ref);
-            // console.debug(`Old artificatHash is ${artifactHash}`);
-            artifactHash = artifactHash.replace(/[\w]*AssetParameters/g, '');
-            artifactHash = artifactHash.substring(0, artifactHash.indexOf('S3Bucket'));
-            // console.debug(`New artificatHash is ${artifactHash}`);
+            let artifactHash = Object.assign(prop.S3Key);
+            console.debug(`artifactHash is ${artifactHash}`);
             const assetPath = `asset${artifactHash}`;
-            fn.Properties.Code.S3Key = `%%SOLUTION_NAME%%/%%VERSION%%/${assetPath}.zip`;
-
+            prop.S3Key = `%%SOLUTION_NAME%%/%%VERSION%%/${assetPath}`;
+     
             // Set the S3 bucket reference
-            fn.Properties.Code.S3Bucket = {
-                'Fn::Sub': '%%BUCKET_NAME%%-${AWS::Region}'
-            };
-        } else {
-            // console.debug(`Here is the fn dump ${JSON.stringify(fn)}`);
+            prop.S3Bucket = {
+            "Fn::Sub": "%%BUCKET_NAME%%-${AWS::Region}",
+        };
+      } else {
+        console.warn(`No S3Bucket Property found for ${JSON.stringify(prop)}`);
+      }
+    });
+
+    // Clean-up Lambda Layer code dependencies
+    const lambdaLayers = Object.keys(resources).filter(function (key) {
+    return resources[key].Type === "AWS::Lambda::LayerVersion";
+    });
+    lambdaLayers.forEach(function (l) {
+    const layer = template.Resources[l];
+    if (layer.Properties.Content.hasOwnProperty('S3Bucket')) {
+        let s3Key = Object.assign(layer.Properties.Content.S3Key);
+        layer.Properties.Content.S3Key = `%%SOLUTION_NAME%%/%%VERSION%%/${s3Key}`;
+        layer.Properties.Content.S3Bucket = {
+        'Fn::Sub': '%%BUCKET_NAME%%-${AWS::Region}'
         }
+    }
     });
 
     // Clean-up nested template stack dependencies
@@ -51,7 +76,7 @@ fs.readdirSync(global_s3_assets).forEach(file => {
             'Fn::Join': [
                 '',
                 [
-                    `https://%%TEMPLATE_BUCKET_NAME%%.s3.`,
+                    'https://%%TEMPLATE_BUCKET_NAME%%.s3.',
                     {
                         'Ref' : 'AWS::URLSuffix'
                     },
@@ -60,23 +85,10 @@ fs.readdirSync(global_s3_assets).forEach(file => {
                 ]
             ]
         };
-        // fn.Properties.TemplateURL = {
-        //     'Fn::Join': [
-        //         '',
-        //         [
-        //             'https://s3.',
-        //             {
-        //                 'Ref' : 'AWS::URLSuffix'
-        //             },
-        //             '/',
-        //             `%%TEMPLATE_BUCKET_NAME%%/%%SOLUTION_NAME%%/%%VERSION%%/${fn.Metadata.nestedStackFileName}`
-        //         ]
-        //     ]
-        // };
 
         const params = fn.Properties.Parameters ? fn.Properties.Parameters : {};
         const nestedStackParameters = Object.keys(params).filter(function(key) {
-            if (key.search(/[\w]*AssetParameters/g) > -1) {
+            if (key.search(_regex) > -1) {
                 return true;
             }
             return false;
@@ -90,12 +102,9 @@ fs.readdirSync(global_s3_assets).forEach(file => {
     // Clean-up parameters section
     const parameters = (template.Parameters) ? template.Parameters : {};
     const assetParameters = Object.keys(parameters).filter(function (key) {
-        // console.debug(`key to analyze ${key}`);
-        if (key.search(/[\w]*AssetParameters/g) > -1) {
-            // console.debug('Pattern match');
+        if (key.search(_regex) > -1) {
             return true;
         }
-        // console.debug('Pattern did not match');
         return false;
     });
     assetParameters.forEach(function (a) {

deployment/cdk-solution-helper/index.js

@@ -1,6 +1,9 @@
 {
   "name": "cdk-solution-helper",
   "version": "0.1.0",
+  "devDependencies": {
+    "fs": "0.0.1-security"
+  },
   "dependencies": {
     "fs": "0.0.1-security"
   }