A production-ready, fully event-driven async processing pipeline on AWS — defined end-to-end as Infrastructure-as-Code with Terraform and the SourceFuse ARC module library.
This blueprint is designed to be cloned, configured, and deployed by any team in minutes, then customised to fit their own application. It includes three built-in compliance profiles (general, hipaa, pci) so the same codebase can be used for development, healthcare (PHI), and payment-card workloads without changing Terraform.
- Highlights
- Architecture
- What is Event-Driven Architecture?
- Module Dependency Graph
- Repository Layout
- Prerequisites
- Quick Start — GitHub Actions (Recommended)
- Quick Start — Local Terraform
- Important: Editing
terraform.tfvarsfor Local Runs - Compliance Profiles
- KMS Key Policy — Important Note
- Documentation
- AI Assistant Integration (ARC IaC MCP)
- Contributing
- License
- Isolated state per module — Each Terraform module has its own S3-backed state file, limiting blast radius and enabling parallel teams to own their slices.
- Compliance profiles out of the box —
general,hipaa, andpcitfvars shipped per module; the CI/CD pipeline selects one at deploy time. - OIDC-based GitHub Actions — No long-lived AWS access keys; jobs assume an IAM role via
AWS_ROLE_ARN. - Fully async and resilient — SNS fan-out, SQS with built-in DLQ, Lambda
ReportBatchItemFailuresfor per-message retry, and DynamoDB idempotency viaConditionExpression. - End-to-end encryption — Single customer-managed KMS CMK encrypts SNS, SQS, DynamoDB, Lambda env vars, and S3.
- Working sample app — A Node.js Lambda handler (
sample-app/index.js) with SNS envelope parsing, DynamoDB write, 90-day TTL, and idempotent event storage. - Full CI/CD —
create.ymldeploys all modules in dependency order;destroy.ymltears down in reverse order.
Event-driven architecture (EDA) is an application design pattern where services communicate asynchronously by producing and consuming events, rather than making synchronous API calls to each other.
Key concepts:
- Event producers publish messages to a central bus (SNS topic) without knowing who will consume them.
- Event consumers subscribe to the bus, receive messages via a queue (SQS), and process them independently.
- Decoupling — producers and consumers evolve independently; adding a new consumer requires no change to the producer.
Benefits:
- Resilience — failed consumers don't affect producers; DLQ catches unprocessable messages.
- Scalability — Lambda and SQS scale independently with message volume.
- Replay — DynamoDB event store + DLQ allow replaying missed or failed events.
- Fan-out — one SNS publish can trigger multiple SQS queues and consumers simultaneously.
- Auditability — every event is persisted to DynamoDB with a TTL; PCI profile enables DynamoDB Streams for full audit trail.
💡 In this blueprint: A publisher sends an event to SNS. SNS fans it out to an SQS queue. The Lambda consumer is triggered via Event Source Mapping, processes the message, and persists it to DynamoDB. Failed messages go to a DLQ after
max_receive_countretries. All data at rest is encrypted with a single CMK.
| Service | Module | Role |
|---|---|---|
| KMS | arc-kms 1.0.12 |
Customer-managed CMK — root of the encryption trust chain |
| S3 | arc-s3 0.0.8 |
Lambda deployment package storage |
| SNS | arc-sns 0.0.3 |
Fan-out event bus — producers publish here |
| SQS | arc-sqs 0.0.2 |
Consumer queue + DLQ — durable async buffer |
| Lambda | arc-lambda-function 0.0.3 |
Event consumer — triggered by SQS ESM |
| DynamoDB | arc-dynamodb 0.0.1 |
Event store — persists processed events |
bootstrap
│
▼
01-kms
├──────────────┬──────────────┬──────────┐
▼ ▼ ▼ ▼
02-s3 03-sns 05-dynamodb (06-lambda deps)
│
▼
04-sqs
│
▼
06-lambda ◄── 02-s3, 04-sqs, 05-dynamodb, 01-kms
arc-iac-blueprint-event-driven/
├── bootstrap/ # Creates S3 + DynamoDB state backend
├── modules/
│ ├── 01-kms/ # Customer-managed KMS key (encrypts everything)
│ ├── 02-s3/ # Lambda artifacts bucket
│ ├── 03-sns/ # SNS event bus topic
│ ├── 04-sqs/ # SQS consumer queue + DLQ + SNS subscription
│ ├── 05-dynamodb/ # DynamoDB event store table
│ └── 06-lambda/ # Lambda consumer + SQS event source mapping
├── sample-app/
│ ├── index.js # Lambda handler (SNS envelope parsing, DynamoDB write, TTL)
│ └── package.json
├── scripts/
│ └── apply-module.sh # Local helper to apply a single module
├── .github/workflows/
│ ├── create.yml # Full create pipeline (all modules, dependency order)
│ └── destroy.yml # Full destroy pipeline (reverse order)
└── docs/
└── 01-architecture.md # Architecture deep dive
Each module contains:
modules/<nn>-<name>/
├── main.tf # Module block + supporting resources
├── variables.tf # Input variables
├── outputs.tf # Output values
├── config.hcl # Partial backend config (key path only)
└── tfvars/
├── general.tfvars
├── hipaa.tfvars
└── pci.tfvars
- Terraform
>= 1.3.0 - AWS account with permissions for KMS, S3, SNS, SQS, Lambda, DynamoDB, IAM, CloudWatch Logs
- AWS CLI configured locally (
aws configure) — only required for the local workflow - GitHub repository — fork this repo to run GitHub Actions against your own AWS account
- Node.js 20 — only needed locally to package
sample-app/before the first deploy
The bundled workflow provisions everything from scratch: it resolves the S3 state bucket and DynamoDB lock table from SSM, then deploys every module in dependency order.
- Fork this repository into your own GitHub org.
- Create an IAM role in AWS that trusts GitHub's OIDC provider.
- In your fork, go to Settings → Secrets and variables → Actions → Secrets and add:
| Secret | Value |
|---|---|
AWS_ROLE_ARN |
arn:aws:iam::<account-id>:role/<oidc-role-name> |
- In Settings → Variables add:
| Variable | Value |
|---|---|
NAMESPACE |
arc (or your org prefix) |
TERRAFORM_VERSION |
1.9.8 |
The bootstrap module creates the S3 state bucket and DynamoDB lock table and stores the config in SSM. Run it once from your laptop:
cd bootstrap
terraform init -backend=false
terraform apply \
-var="namespace=arc" \
-var="environment=dev" \
-var="region=us-east-1"Before deploying 06-lambda, package and upload the sample app:
cd sample-app
npm install
zip -r lambda.zip index.js package.json node_modules
aws s3 cp lambda.zip s3://<namespace>-<env>-events-lambda/lambda.zip- Go to Actions → Create Event-Driven Architecture Infrastructure → Run workflow.
- Fill in the inputs:
| Input | Example | Description |
|---|---|---|
environment |
dev |
dev, staging, prod |
region |
us-east-1 |
AWS region |
compliance_profile |
general |
general, hipaa, or pci |
namespace |
arc |
Resource name prefix |
- Watch the job graph —
resolve-backendfirst, thenkms, then modules fan out in dependency order, thenlambdalast. - The summary job prints the status of every module.
🔒 Tip: Re-running the workflow with the same inputs is idempotent.
Note: The included GitHub Actions workflow (
.github/workflows/create.yml) is provided as a reference template.Recommendation for production use: Split the bundled workflow into separate pipelines:
Pipeline Responsibility CI — Terraform Plan Run terraform planon PRsCD — Terraform Apply Run terraform applyafter merge, per module with approval gatesApp pipeline Package Lambda zip, upload to S3, update function code
cd bootstrap
terraform init -backend=false
terraform apply \
-var="namespace=arc" \
-var="environment=dev" \
-var="region=us-east-1" \
-var="enable_bucket_force_destroy=true"
BUCKET=$(terraform output -raw state_bucket_name)
TABLE=$(terraform output -raw lock_table_name)
cd ..cd sample-app
npm install
zip -r lambda.zip index.js package.json node_modules
aws s3 cp lambda.zip s3://${BUCKET%-terraform-state}-events-lambda/lambda.zip
cd ..Use the helper script — it handles backend config and compliance profile selection:
export NAMESPACE=arc
./scripts/apply-module.sh 01-kms dev us-east-1 general
./scripts/apply-module.sh 02-s3 dev us-east-1 general
./scripts/apply-module.sh 03-sns dev us-east-1 general
./scripts/apply-module.sh 04-sqs dev us-east-1 general
./scripts/apply-module.sh 05-dynamodb dev us-east-1 general
./scripts/apply-module.sh 06-lambda dev us-east-1 generalUsage: ./scripts/apply-module.sh <module-name> [env] [region] [compliance-profile]
# Publish a test event to SNS
SNS_TOPIC_ARN=$(aws sns list-topics \
--query "Topics[?contains(TopicArn,'<namespace>-<env>')].TopicArn | [0]" \
--output text --region us-east-1)
aws sns publish \
--topic-arn "$SNS_TOPIC_ARN" \
--message '{"eventType":"order.created","orderId":"test-001","amount":99.99}' \
--region us-east-1
# After ~10s, check DynamoDB
aws dynamodb scan \
--table-name <namespace>-<env>-event-store \
--region us-east-1 \
--query "Items[*].{eventId:eventId.S, source:source.S, timestamp:timestamp.S}"The GitHub Actions workflow passes namespace, environment, region, and other inputs as command-line -var flags. These overrides always win over terraform.tfvars.
When running locally, either use the helper script (recommended) or set variables manually:
scripts/apply-module.sh passes the right -var flags automatically. Set NAMESPACE as an environment variable:
export NAMESPACE=arc
./scripts/apply-module.sh 04-sqs dev us-east-1 hipaacd modules/06-lambda
cp tfvars/hipaa.tfvars terraform.tfvars
cat >> terraform.tfvars <<'EOF'
namespace = "arc"
environment = "dev"
region = "us-east-1"
state_bucket_name = "arc-dev-terraform-state"
EOF
terraform init \
-backend-config=config.hcl \
-backend-config="bucket=arc-dev-terraform-state" \
-backend-config="dynamodb_table=arc-dev-terraform-locks" \
-backend-config="region=us-east-1"
terraform apply -auto-approve| Variable | Purpose | Example |
|---|---|---|
namespace |
Resource name prefix | arc |
environment |
Environment tag | dev |
region |
AWS region | us-east-1 |
state_bucket_name |
S3 state bucket (from bootstrap) | arc-dev-terraform-state |
tags |
Tags applied to every resource | { ManagedBy = "Terraform" } |
⚠️ DisclaimerThe
hipaaandpciprofiles implement security controls that support HIPAA and PCI-DSS compliance. They are not a certified, complete, or guaranteed compliant configuration. Compliance requires your full environment, organisational processes, a signed BAA (HIPAA), and a QSA assessment (PCI-DSS). You are responsible for validating your own compliance.
| Profile | Use case | Highlights |
|---|---|---|
general |
Dev, internal tools, non-regulated | KMS rotation, SSE-KMS, long polling, 3-retry DLQ, PAY_PER_REQUEST DynamoDB |
hipaa |
Healthcare workloads with PHI | S3 access logging, Lambda X-Ray Active, Lambda Insights, batch size 5, 14-day SQS retention, DynamoDB PITR + deletion protection |
pci |
Payment card data | All HIPAA controls + DynamoDB Streams (NEW_AND_OLD_IMAGES), PROVISIONED billing with autoscaling, batch size 1 |
| Setting | general | hipaa | pci |
|---|---|---|---|
| KMS key rotation | ✅ | ✅ | ✅ |
| S3 access logging | — | ✅ | ✅ |
| SQS message retention | 4 days | 14 days | 14 days |
| SQS visibility timeout | 60s | 120s | 120s |
| DLQ max receive count | 3 | 5 | 5 |
| Lambda X-Ray tracing | PassThrough | Active | Active |
| Lambda Insights | — | ✅ | ✅ |
| Lambda batch size | 10 | 5 | 1 |
| DynamoDB billing | PAY_PER_REQUEST | PAY_PER_REQUEST | PROVISIONED |
| DynamoDB autoscaling | — | — | ✅ |
| DynamoDB PITR | — | ✅ | ✅ |
| DynamoDB deletion protection | — | ✅ | ✅ |
| DynamoDB Streams | — | — | ✅ (NEW_AND_OLD_IMAGES) |
This blueprint uses a single CMK to encrypt all services (SNS, SQS, DynamoDB, Lambda env vars, S3). The 01-kms module generates a key policy with the following principals — all four are required for the pipeline to work:
| Principal | Why needed |
|---|---|
Root account (arn:aws:iam::<account>:root) |
Enables IAM delegation |
| Lambda execution role | Decrypt SQS messages + DynamoDB SSE reads/writes |
sns.amazonaws.com |
SNS must encrypt messages before delivering to KMS-encrypted SQS |
lambda.amazonaws.com |
Lambda ESM poller decrypts SQS messages before invoking the function |
⚠️ Missing the SNS or Lambda service principal in the key policy will cause silent message loss — SNS will silently fail to deliver to SQS, and the SQS ESM will silently consume messages without triggering Lambda. No errors appear in CloudWatch.
| Topic | Location |
|---|---|
| Architecture deep dive | docs/01-architecture.md |
The ARC IaC MCP Server is a hosted Model Context Protocol service that lets AI assistants browse, search, scaffold, compare, and security-scan any of the SourceFuse ARC Terraform modules — directly from natural language.
What you can do with it:
- Discover — search and filter modules by keyword or AWS resource type.
- Understand — get inputs, outputs, and resources for any module without leaving your editor.
- Scaffold — generate production-ready, multi-file Terraform with cross-module wiring.
- Secure — scan generated or existing HCL for misconfigurations before it hits a PR.
- Compare — diff modules side-by-side to make informed architectural decisions.
The MCP endpoint is https://arc-iac-mcp.sourcef.us/mcp. Pick your client:
Kiro CLI:
# Add to your .kiro/settings/mcp.json
{
"mcpServers": {
"arc-iac": {
"url": "https://arc-iac-mcp.sourcef.us/mcp"
}
}
}Claude Code CLI:
claude mcp add arc-iac --transport http https://arc-iac-mcp.sourcef.us/mcpClaude Desktop — edit ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"arc-iac": {
"url": "https://arc-iac-mcp.sourcef.us/mcp"
}
}
}- "List all ARC modules sorted by downloads"
- "What inputs does
arc-sqsrequire?" - "Scaffold a production-ready
arc-lambda-functiontriggered by SQS" - "Compare
arc-snsandarc-sqs— what resources does each create?" - "Scan this Terraform before I raise a PR:
<paste HCL>"
Apache License 2.0 — see LICENSE for the full text.
We welcome contributions! Please read CONTRIBUTING.md for the development workflow, coding standards, and pull-request process.
Built with the SourceFuse ARC Terraform module library and the AWS event-driven services ecosystem.