New serverless pattern - lambda-SQS-best-Practices-CDK

lambda-sqs-best-practices-cdk/README.md

@@ -0,0 +1,149 @@
+# Lambda SQS Best Practices with AWS CDK
+
+This pattern demonstrates a production-ready implementation of AWS Lambda processing messages from Amazon SQS using AWS CDK. It serves as a reference architecture for building robust, observable, and maintainable serverless applications, featuring AWS Lambda Powertools integration for enhanced observability through structured logging, custom metrics, and distributed tracing with X-Ray. The pattern implements comprehensive error handling with automatic retries and Dead Letter Queue (DLQ) configuration, along with a detailed CloudWatch Dashboard for operational monitoring. Security is enforced through least privilege IAM roles, while operational excellence is maintained through proper resource configurations and cost optimizations. This enterprise-grade solution includes batch message processing, configurable timeouts, message validation, and a complete monitoring strategy, making it ideal for teams building production serverless applications that require high reliability, observability, and maintainability.
+
+
+<img src="./resources/Lambda-SQS-Best-Practice.png" alt="Architecture" width="100%"/>
+
+Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
+
+## Requirements
+
+* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
+* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
+* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+* [Node.js 20 or greater](https://nodejs.org/en/download/) installed
+* [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) installed
+
+## Deployment Instructions
+
+1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
+    ``` 
+    git clone https://github.com/aws-samples/serverless-patterns
+    ```
+1. Change directory to the pattern directory:
+    ```
+    cd serverless-patterns/lambda-sqs-best-practices-cdk
+    ```
+
+1. Install cdk dependencies
+   ```
+   npm install
+   ```
+
+1. Install lambda dependencies
+   ```
+   cd lambda
+   npm install
+   ```
+
+1. Deploy cdk stack
+    ```
+    cd ..
+    cdk deploy
+
+    ```
+
+Note: If you are using CDK for the first time then bootstrap CDK in your account by using below command:
+
+```
+cdk bootstrap aws://ACCOUNT-NUMBER-1/REGION-1
+
+```
+
+## How it works
+
+This pattern sets up:
+
+1. An SQS queue with a Dead Letter Queue (DLQ) for failed message handling
+2. A Lambda function with:
+   - AWS Lambda Powertools integration
+   - Structured logging
+   - Custom metrics
+   - X-Ray tracing
+3. A CloudWatch Dashboard for operational monitoring
+4. Least priviledge permissions implemented on roles and policies
+
+The Lambda function:
+- Processes messages in batches
+- Validates message format
+- Simulates downstream API calls with random failures (5% failure rate)
+- Demonstrates handling of external service dependencies
+- Handles errors gracefully
+- Reports metrics and traces
+- Uses structured logging
+
+Failed messages are:
+- Logged with error details
+- Sent to DLQ after 3 retries
+- Monitored via CloudWatch metrics
+
+## Testing
+
+The pattern includes a load testing script to verify functionality:
+
+1. Set the Queue URL environment variable:
+```
+export QUEUE_URL=$(aws cloudformation describe-stacks --stack-name LambdaSqsBestPracticesCdkStack --query 'Stacks[0].Outputs[?OutputKey==`QueueUrl`].OutputValue' --output text)
+
+export AWS_REGION=us-east-1  # or your AWS region
+```
+
+2. Rum test script
+```
+npm run test           # 100 messages
+npm run test:small    # 50 messages
+npm run test:medium   # 200 messages
+npm run test:large    # 500 messages
+
+```
+
+
+## Monitoring Guide
+
+Locating Resources
+
+```
+1. Navigate to AWS CloudFormation Console
+2. Select the stack "LambdaSqsBestPracticesCdkStack"
+3. Go to the "Resources" tab
+4. Here you can find:
+   - All resources created by the stack
+   - Direct links to each resource's console
+   - Resource physical IDs and types
+   - Current status of each resource
+```
+
+CloudWatch Logs
+
+```
+1. Navigate to CloudWatch Console > Log Groups
+2. Find /aws/lambda/BatchProcessingLambdaFunction
+3. View structured logs with:
+    * Batch processing information
+    * Error details
+```
+
+Metrics Dashboard
+
+```
+1. Go to CloudWatch > Dashboards
+2. Find the dashboard “SQS-Processing-Dashboard”
+3. Monitor:
+    * Message processing success rate
+    * Error rates
+    * DLQ message count
+    * Lambda Processing duration 
+
+```
+
+<img src="./resources/SQS_operational_dashboard.png" alt="Architecture" width="100%"/>
+
+## Cleanup
+
+To remove all deployed resources:
+
+```
+cdk destroy
+```
+

lambda-sqs-best-practices-cdk/bin/lambda-sqs-best-practices-cdk.js

@@ -0,0 +1,6 @@
+#!/usr/bin/env node
+const cdk = require('aws-cdk-lib');
+const { LambdaSqsBestPracticesCdkStack } = require('../lib/lambda-sqs-best-practices-cdk-stack');
+
+const app = new cdk.App();
+new LambdaSqsBestPracticesCdkStack(app, 'LambdaSqsBestPracticesCdkStack', {});

lambda-sqs-best-practices-cdk/cdk.json

@@ -0,0 +1,88 @@
+{
+  "app": "node bin/lambda-sqs-best-practices-cdk.js",
+  "watch": {
+    "include": [
+      "**"
+    ],
+    "exclude": [
+      "README.md",
+      "cdk*.json",
+      "jest.config.js",
+      "package*.json",
+      "yarn.lock",
+      "node_modules",
+      "test"
+    ]
+  },
+  "context": {
+    "@aws-cdk/aws-lambda:recognizeLayerVersion": true,
+    "@aws-cdk/core:checkSecretUsage": true,
+    "@aws-cdk/core:target-partitions": [
+      "aws",
+      "aws-cn"
+    ],
+    "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
+    "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
+    "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
+    "@aws-cdk/aws-iam:minimizePolicies": true,
+    "@aws-cdk/core:validateSnapshotRemovalPolicy": true,
+    "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
+    "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
+    "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
+    "@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
+    "@aws-cdk/core:enablePartitionLiterals": true,
+    "@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
+    "@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
+    "@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
+    "@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
+    "@aws-cdk/aws-route53-patters:useCertificate": true,
+    "@aws-cdk/customresources:installLatestAwsSdkDefault": false,
+    "@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
+    "@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
+    "@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
+    "@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
+    "@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
+    "@aws-cdk/aws-redshift:columnId": true,
+    "@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
+    "@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
+    "@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
+    "@aws-cdk/aws-kms:aliasNameRef": true,
+    "@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
+    "@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
+    "@aws-cdk/aws-efs:denyAnonymousAccess": true,
+    "@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
+    "@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
+    "@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
+    "@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
+    "@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
+    "@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
+    "@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
+    "@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
+    "@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
+    "@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
+    "@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
+    "@aws-cdk/aws-eks:nodegroupNameAttribute": true,
+    "@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
+    "@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
+    "@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false,
+    "@aws-cdk/aws-s3:keepNotificationInImportedBucket": false,
+    "@aws-cdk/aws-ecs:enableImdsBlockingDeprecatedFeature": false,
+    "@aws-cdk/aws-ecs:disableEcsImdsBlocking": true,
+    "@aws-cdk/aws-ecs:reduceEc2FargateCloudWatchPermissions": true,
+    "@aws-cdk/aws-dynamodb:resourcePolicyPerReplica": true,
+    "@aws-cdk/aws-ec2:ec2SumTImeoutEnabled": true,
+    "@aws-cdk/aws-appsync:appSyncGraphQLAPIScopeLambdaPermission": true,
+    "@aws-cdk/aws-rds:setCorrectValueForDatabaseInstanceReadReplicaInstanceResourceId": true,
+    "@aws-cdk/core:cfnIncludeRejectComplexResourceUpdateCreatePolicyIntrinsics": true,
+    "@aws-cdk/aws-lambda-nodejs:sdkV3ExcludeSmithyPackages": true,
+    "@aws-cdk/aws-stepfunctions-tasks:fixRunEcsTaskPolicy": true,
+    "@aws-cdk/aws-ec2:bastionHostUseAmazonLinux2023ByDefault": true,
+    "@aws-cdk/aws-route53-targets:userPoolDomainNameMethodWithoutCustomResource": true,
+    "@aws-cdk/aws-elasticloadbalancingV2:albDualstackWithoutPublicIpv4SecurityGroupRulesDefault": true,
+    "@aws-cdk/aws-iam:oidcRejectUnauthorizedConnections": true,
+    "@aws-cdk/core:enableAdditionalMetadataCollection": true,
+    "@aws-cdk/aws-lambda:createNewPoliciesWithAddToRolePolicy": true,
+    "@aws-cdk/aws-s3:setUniqueReplicationRoleName": true,
+    "@aws-cdk/aws-events:requireEventBusPolicySid": true
+  }
+}

lambda-sqs-best-practices-cdk/example-pattern.json

@@ -0,0 +1,98 @@
+{
+  "title": "Lambda SQS best practices with Powertools",
+  "description": "Creates a Lambda function with SQS trigger implementing best practices using AWS Lambda Powertools, including structured logging, metrics, and tracing",
+  "language": "nodejs",
+  "level": "200",
+  "framework": "AWS CDK",
+  "introBox": {
+    "headline": "How it works",
+    "text": [
+      "This pattern demonstrates how to implement a Lambda function with SQS trigger using AWS best practices. It includes AWS Lambda Powertools for structured logging, metrics, and tracing, along with proper error handling and dead-letter queue configuration.",
+      "The pattern implements message validation, batch processing with partial failures, and comprehensive operational monitoring through CloudWatch dashboards.",
+      "The implementation includes load testing capabilities to verify the system's behavior under different scenarios including invalid messages and error conditions.",
+      "This pattern deploys an SQS Queue, Dead Letter Queue, Lambda function, CloudWatch Dashboard, and necessary IAM roles and permissions."
+    ]
+  },
+  "gitHub": {
+    "template": {
+      "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/lambda-sqs-best-practices-cdk",
+      "templateURL": "serverless-patterns/lambda-sqs-best-practices-cdk",
+      "projectFolder": "lambda-sqs-best-practices-cdk",
+      "templateFile": "lib/lambda-sqs-best-practices-cdk-stack.js"
+    }
+  },
+  "resources": {
+    "bullets": [
+      {
+        "text": "AWS Lambda Powertools for Node.js",
+        "link": "https://www.npmjs.com/org/aws-lambda-powertools"
+      },
+      {
+        "text": "AWS Lambda with SQS",
+        "link": "https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html"
+      },
+      {
+        "text": "SQS Dead Letter Queues",
+        "link": "https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html"
+      },
+      {
+        "text": "Lambda Function Scaling with SQS",
+        "link": "https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-scaling.html"
+      }
+    ]
+  },
+  "deploy": {
+    "text": [
+      "git clone https://github.com/aws-samples/serverless-patterns",
+      "cd serverless-patterns/lambda-sqs-best-practices-cdk",
+      "Install cdk dependencies",
+      "npm install",
+      "Install lambda dependencies",
+      "cd lambda",
+      "npm install",
+      "cd .."
+      "cdk deploy"
+    ]
+  },
+  "testing": {
+    "text": [
+      "After deployment, you can test the pattern using the included load test script:",
+      "export QUEUE_URL=$(aws cloudformation describe-stacks --stack-name LambdaSqsBestPracticesCdkStack --query 'Stacks[0].Outputs[?OutputKey==`QueueUrl`].OutputValue' --output text)",
+      "export AWS_REGION=us-east-1  # or your AWS region",
+      "npm run test          # Sends 100 messages",
+      "npm run test:small    # Sends 50 messages",
+      "npm run test:medium   # Sends 200 messages",
+      "npm run test:large    # Sends 500 messages",
+      "Monitor the results in the CloudWatch Dashboard created by the stack."
+    ]
+  },
+  "cleanup": {
+    "text": [
+      "Delete the stack: <code>cdk destroy</code>"
+    ]
+  },
+  "authors": [
+    {
+      "name": "Shubham More",
+      "image": "https://avatars.githubusercontent.com/u/150242047?s=400&u=aaa2db07529d3e1e82ec59daacb05b6abc7c3a5f&v=4",
+      "bio": "Cloud Support Engineer II - SVLS",
+      "linkedin": "shubham-more-1b6aa185"
+    },
+    {
+      "name": "Umang Aggarwal",
+      "image": "https://avatars.githubusercontent.com/u/75968399?v=4&size=64",
+      "bio": "Cloud Support Engineer II - SVLS",
+      "linkedin": "umangaggarwal"
+    },
+    {
+      "name": "Vaibhav Jain",
+      "bio": "AWS - Sr. Application Architect",
+      "linkedin": "https://www.linkedin.com/in/vaibhavjainv/"
+     },
+     {
+      "name": "Adam Wagner",
+      "bio": "AWS - Principal Serverless Solutions Architect",
+      "linkedin": "https://www.linkedin.com/in/adam-wagner-4bb412/"
+     }
+  ]
+}