Arc Event-Driven Architecture Blueprint

A production-ready, fully event-driven async processing pipeline on AWS — defined end-to-end as Infrastructure-as-Code with Terraform and the SourceFuse ARC module library.

This blueprint is designed to be cloned, configured, and deployed by any team in minutes, then customised to fit their own application. It includes three built-in compliance profiles (general, hipaa, pci) so the same codebase can be used for development, healthcare (PHI), and payment-card workloads without changing Terraform.

Highlights
Architecture
What is Event-Driven Architecture?
Module Dependency Graph
Repository Layout
Prerequisites
Quick Start — GitHub Actions (Recommended)
Quick Start — Local Terraform
Important: Editing terraform.tfvars for Local Runs
Compliance Profiles
KMS Key Policy — Important Note
Documentation
AI Assistant Integration (ARC IaC MCP)
Contributing
License

Highlights

Isolated state per module — Each Terraform module has its own S3-backed state file, limiting blast radius and enabling parallel teams to own their slices.
Compliance profiles out of the box — general, hipaa, and pci tfvars shipped per module; the CI/CD pipeline selects one at deploy time.
OIDC-based GitHub Actions — No long-lived AWS access keys; jobs assume an IAM role via AWS_ROLE_ARN.
Fully async and resilient — SNS fan-out, SQS with built-in DLQ, Lambda ReportBatchItemFailures for per-message retry, and DynamoDB idempotency via ConditionExpression.
End-to-end encryption — Single customer-managed KMS CMK encrypts SNS, SQS, DynamoDB, Lambda env vars, and S3.
Working sample app — A Node.js Lambda handler (sample-app/index.js) with SNS envelope parsing, DynamoDB write, 90-day TTL, and idempotent event storage.
Full CI/CD — create.yml deploys all modules in dependency order; destroy.yml tears down in reverse order.

Architecture

What is Event-Driven Architecture?

Event-driven architecture (EDA) is an application design pattern where services communicate asynchronously by producing and consuming events, rather than making synchronous API calls to each other.

Key concepts:

Event producers publish messages to a central bus (SNS topic) without knowing who will consume them.
Event consumers subscribe to the bus, receive messages via a queue (SQS), and process them independently.
Decoupling — producers and consumers evolve independently; adding a new consumer requires no change to the producer.

Benefits:

Resilience — failed consumers don't affect producers; DLQ catches unprocessable messages.
Scalability — Lambda and SQS scale independently with message volume.
Replay — DynamoDB event store + DLQ allow replaying missed or failed events.
Fan-out — one SNS publish can trigger multiple SQS queues and consumers simultaneously.
Auditability — every event is persisted to DynamoDB with a TTL; PCI profile enables DynamoDB Streams for full audit trail.

💡 In this blueprint: A publisher sends an event to SNS. SNS fans it out to an SQS queue. The Lambda consumer is triggered via Event Source Mapping, processes the message, and persists it to DynamoDB. Failed messages go to a DLQ after max_receive_count retries. All data at rest is encrypted with a single CMK.

Architecture Flow

Service Reference

Service	Module	Role
KMS	`arc-kms 1.0.12`	Customer-managed CMK — root of the encryption trust chain
S3	`arc-s3 0.0.8`	Lambda deployment package storage
SNS	`arc-sns 0.0.3`	Fan-out event bus — producers publish here
SQS	`arc-sqs 0.0.2`	Consumer queue + DLQ — durable async buffer
Lambda	`arc-lambda-function 0.0.3`	Event consumer — triggered by SQS ESM
DynamoDB	`arc-dynamodb 0.0.1`	Event store — persists processed events

Module Dependency Graph

bootstrap
    │
    ▼
01-kms
    ├──────────────┬──────────────┬──────────┐
    ▼              ▼              ▼          ▼
  02-s3          03-sns        05-dynamodb  (06-lambda deps)
                   │
                   ▼
                 04-sqs
                   │
                   ▼
               06-lambda ◄── 02-s3, 04-sqs, 05-dynamodb, 01-kms

Repository Layout

arc-iac-blueprint-event-driven/
├── bootstrap/                  # Creates S3 + DynamoDB state backend
├── modules/
│   ├── 01-kms/                 # Customer-managed KMS key (encrypts everything)
│   ├── 02-s3/                  # Lambda artifacts bucket
│   ├── 03-sns/                 # SNS event bus topic
│   ├── 04-sqs/                 # SQS consumer queue + DLQ + SNS subscription
│   ├── 05-dynamodb/            # DynamoDB event store table
│   └── 06-lambda/              # Lambda consumer + SQS event source mapping
├── sample-app/
│   ├── index.js                # Lambda handler (SNS envelope parsing, DynamoDB write, TTL)
│   └── package.json
├── scripts/
│   └── apply-module.sh         # Local helper to apply a single module
├── .github/workflows/
│   ├── create.yml              # Full create pipeline (all modules, dependency order)
│   └── destroy.yml             # Full destroy pipeline (reverse order)
└── docs/
    └── 01-architecture.md      # Architecture deep dive

Each module contains:

modules/<nn>-<name>/
├── main.tf          # Module block + supporting resources
├── variables.tf     # Input variables
├── outputs.tf       # Output values
├── config.hcl       # Partial backend config (key path only)
└── tfvars/
    ├── general.tfvars
    ├── hipaa.tfvars
    └── pci.tfvars

Prerequisites

Terraform >= 1.3.0
AWS account with permissions for KMS, S3, SNS, SQS, Lambda, DynamoDB, IAM, CloudWatch Logs
AWS CLI configured locally (aws configure) — only required for the local workflow
GitHub repository — fork this repo to run GitHub Actions against your own AWS account
Node.js 20 — only needed locally to package sample-app/ before the first deploy

Quick Start — GitHub Actions (Recommended)

The bundled workflow provisions everything from scratch: it resolves the S3 state bucket and DynamoDB lock table from SSM, then deploys every module in dependency order.

1. One-time setup

Fork this repository into your own GitHub org.
Create an IAM role in AWS that trusts GitHub's OIDC provider.
In your fork, go to Settings → Secrets and variables → Actions → Secrets and add:

Secret	Value
`AWS_ROLE_ARN`	`arn:aws:iam::<account-id>:role/<oidc-role-name>`

In Settings → Variables add:

Variable	Value
`NAMESPACE`	`arc` (or your org prefix)
`TERRAFORM_VERSION`	`1.9.8`

2. Bootstrap (first time only — run locally)

The bootstrap module creates the S3 state bucket and DynamoDB lock table and stores the config in SSM. Run it once from your laptop:

cd bootstrap
terraform init -backend=false
terraform apply \
  -var="namespace=arc" \
  -var="environment=dev" \
  -var="region=us-east-1"

3. Upload Lambda package

Before deploying 06-lambda, package and upload the sample app:

cd sample-app
npm install
zip -r lambda.zip index.js package.json node_modules
aws s3 cp lambda.zip s3://<namespace>-<env>-events-lambda/lambda.zip

4. Deploy

Go to Actions → Create Event-Driven Architecture Infrastructure → Run workflow.
Fill in the inputs:

Input	Example	Description
`environment`	`dev`	`dev`, `staging`, `prod`
`region`	`us-east-1`	AWS region
`compliance_profile`	`general`	`general`, `hipaa`, or `pci`
`namespace`	`arc`	Resource name prefix

Watch the job graph — resolve-backend first, then kms, then modules fan out in dependency order, then lambda last.
The summary job prints the status of every module.

🔒 Tip: Re-running the workflow with the same inputs is idempotent.

Note: The included GitHub Actions workflow (.github/workflows/create.yml) is provided as a reference template.

Recommendation for production use: Split the bundled workflow into separate pipelines:

Pipeline Responsibility

CI — Terraform Plan Run terraform plan on PRs

CD — Terraform Apply Run terraform apply after merge, per module with approval gates

App pipeline Package Lambda zip, upload to S3, update function code

Pipeline	Responsibility
CI — Terraform Plan	Run `terraform plan` on PRs
CD — Terraform Apply	Run `terraform apply` after merge, per module with approval gates
App pipeline	Package Lambda zip, upload to S3, update function code

Quick Start — Local Terraform

1. Bootstrap the state backend

cd bootstrap
terraform init -backend=false
terraform apply \
  -var="namespace=arc" \
  -var="environment=dev" \
  -var="region=us-east-1" \
  -var="enable_bucket_force_destroy=true"

BUCKET=$(terraform output -raw state_bucket_name)
TABLE=$(terraform output -raw lock_table_name)
cd ..

2. Upload Lambda package

cd sample-app
npm install
zip -r lambda.zip index.js package.json node_modules
aws s3 cp lambda.zip s3://${BUCKET%-terraform-state}-events-lambda/lambda.zip
cd ..

3. Deploy each module in order

Use the helper script — it handles backend config and compliance profile selection:

export NAMESPACE=arc

./scripts/apply-module.sh 01-kms      dev us-east-1 general
./scripts/apply-module.sh 02-s3       dev us-east-1 general
./scripts/apply-module.sh 03-sns      dev us-east-1 general
./scripts/apply-module.sh 04-sqs      dev us-east-1 general
./scripts/apply-module.sh 05-dynamodb dev us-east-1 general
./scripts/apply-module.sh 06-lambda   dev us-east-1 general

Usage: ./scripts/apply-module.sh <module-name> [env] [region] [compliance-profile]

4. Test end-to-end

# Publish a test event to SNS
SNS_TOPIC_ARN=$(aws sns list-topics \
  --query "Topics[?contains(TopicArn,'<namespace>-<env>')].TopicArn | [0]" \
  --output text --region us-east-1)

aws sns publish \
  --topic-arn "$SNS_TOPIC_ARN" \
  --message '{"eventType":"order.created","orderId":"test-001","amount":99.99}' \
  --region us-east-1

# After ~10s, check DynamoDB
aws dynamodb scan \
  --table-name <namespace>-<env>-event-store \
  --region us-east-1 \
  --query "Items[*].{eventId:eventId.S, source:source.S, timestamp:timestamp.S}"

⚠️ Important: Editing `terraform.tfvars` for Local Runs

The GitHub Actions workflow passes namespace, environment, region, and other inputs as command-line -var flags. These overrides always win over terraform.tfvars.

When running locally, either use the helper script (recommended) or set variables manually:

Option A — Use the helper script (recommended)

scripts/apply-module.sh passes the right -var flags automatically. Set NAMESPACE as an environment variable:

export NAMESPACE=arc
./scripts/apply-module.sh 04-sqs dev us-east-1 hipaa

Option B — Edit `terraform.tfvars` by hand

cd modules/06-lambda
cp tfvars/hipaa.tfvars terraform.tfvars
cat >> terraform.tfvars <<'EOF'
namespace         = "arc"
environment       = "dev"
region            = "us-east-1"
state_bucket_name = "arc-dev-terraform-state"
EOF

terraform init \
  -backend-config=config.hcl \
  -backend-config="bucket=arc-dev-terraform-state" \
  -backend-config="dynamodb_table=arc-dev-terraform-locks" \
  -backend-config="region=us-east-1"

terraform apply -auto-approve

Variables required across all modules

Variable	Purpose	Example
`namespace`	Resource name prefix	`arc`
`environment`	Environment tag	`dev`
`region`	AWS region	`us-east-1`
`state_bucket_name`	S3 state bucket (from bootstrap)	`arc-dev-terraform-state`
`tags`	Tags applied to every resource	`{ ManagedBy = "Terraform" }`

Compliance Profiles

⚠️ Disclaimer

The hipaa and pci profiles implement security controls that support HIPAA and PCI-DSS compliance. They are not a certified, complete, or guaranteed compliant configuration. Compliance requires your full environment, organisational processes, a signed BAA (HIPAA), and a QSA assessment (PCI-DSS). You are responsible for validating your own compliance.

Profile	Use case	Highlights
`general`	Dev, internal tools, non-regulated	KMS rotation, SSE-KMS, long polling, 3-retry DLQ, PAY_PER_REQUEST DynamoDB
`hipaa`	Healthcare workloads with PHI	S3 access logging, Lambda X-Ray Active, Lambda Insights, batch size 5, 14-day SQS retention, DynamoDB PITR + deletion protection
`pci`	Payment card data	All HIPAA controls + DynamoDB Streams (`NEW_AND_OLD_IMAGES`), PROVISIONED billing with autoscaling, batch size 1

Compliance comparison table

Setting	general	hipaa	pci
KMS key rotation	✅	✅	✅
S3 access logging	—	✅	✅
SQS message retention	4 days	14 days	14 days
SQS visibility timeout	60s	120s	120s
DLQ max receive count	3	5	5
Lambda X-Ray tracing	PassThrough	Active	Active
Lambda Insights	—	✅	✅
Lambda batch size	10	5	1
DynamoDB billing	PAY_PER_REQUEST	PAY_PER_REQUEST	PROVISIONED
DynamoDB autoscaling	—	—	✅
DynamoDB PITR	—	✅	✅
DynamoDB deletion protection	—	✅	✅
DynamoDB Streams	—	—	✅ (NEW_AND_OLD_IMAGES)

KMS Key Policy — Important Note

This blueprint uses a single CMK to encrypt all services (SNS, SQS, DynamoDB, Lambda env vars, S3). The 01-kms module generates a key policy with the following principals — all four are required for the pipeline to work:

Principal	Why needed
Root account (`arn:aws:iam::<account>:root`)	Enables IAM delegation
Lambda execution role	Decrypt SQS messages + DynamoDB SSE reads/writes
`sns.amazonaws.com`	SNS must encrypt messages before delivering to KMS-encrypted SQS
`lambda.amazonaws.com`	Lambda ESM poller decrypts SQS messages before invoking the function

⚠️ Missing the SNS or Lambda service principal in the key policy will cause silent message loss — SNS will silently fail to deliver to SQS, and the SQS ESM will silently consume messages without triggering Lambda. No errors appear in CloudWatch.

Documentation

Topic	Location
Architecture deep dive	`docs/01-architecture.md`

AI Assistant Integration (ARC IaC MCP)

The ARC IaC MCP Server is a hosted Model Context Protocol service that lets AI assistants browse, search, scaffold, compare, and security-scan any of the SourceFuse ARC Terraform modules — directly from natural language.

What you can do with it:

Discover — search and filter modules by keyword or AWS resource type.
Understand — get inputs, outputs, and resources for any module without leaving your editor.
Scaffold — generate production-ready, multi-file Terraform with cross-module wiring.
Secure — scan generated or existing HCL for misconfigurations before it hits a PR.
Compare — diff modules side-by-side to make informed architectural decisions.

Setup (one minute)

The MCP endpoint is https://arc-iac-mcp.sourcef.us/mcp. Pick your client:

Kiro CLI:

# Add to your .kiro/settings/mcp.json
{
  "mcpServers": {
    "arc-iac": {
      "url": "https://arc-iac-mcp.sourcef.us/mcp"
    }
  }
}

Claude Code CLI:

claude mcp add arc-iac --transport http https://arc-iac-mcp.sourcef.us/mcp

Claude Desktop — edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "arc-iac": {
      "url": "https://arc-iac-mcp.sourcef.us/mcp"
    }
  }
}

Example prompts to try

"List all ARC modules sorted by downloads"
"What inputs does arc-sqs require?"
"Scaffold a production-ready arc-lambda-function triggered by SQS"
"Compare arc-sns and arc-sqs — what resources does each create?"
"Scan this Terraform before I raise a PR: <paste HCL>"

License

Apache License 2.0 — see LICENSE for the full text.

Contributing

We welcome contributions! Please read CONTRIBUTING.md for the development workflow, coding standards, and pull-request process.

Acknowledgements

Built with the SourceFuse ARC Terraform module library and the AWS event-driven services ecosystem.

Navigation Menu

Uh oh!

Arc Event-Driven Architecture Blueprint

Table of Contents

Highlights

Architecture

What is Event-Driven Architecture?

Architecture Flow

Service Reference

Module Dependency Graph

Repository Layout

Prerequisites

Quick Start — GitHub Actions (Recommended)

1. One-time setup

2. Bootstrap (first time only — run locally)

3. Upload Lambda package

4. Deploy

Quick Start — Local Terraform

1. Bootstrap the state backend

2. Upload Lambda package

3. Deploy each module in order

4. Test end-to-end

⚠️ Important: Editing `terraform.tfvars` for Local Runs

Option A — Use the helper script (recommended)

Option B — Edit `terraform.tfvars` by hand

Variables required across all modules

Compliance Profiles

Compliance comparison table

KMS Key Policy — Important Note

Documentation

AI Assistant Integration (ARC IaC MCP)

Setup (one minute)

Example prompts to try

License

Contributing

Acknowledgements

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Arc Event-Driven Architecture Blueprint

Table of Contents

Highlights

Architecture

What is Event-Driven Architecture?

Architecture Flow

Service Reference

Module Dependency Graph

Repository Layout

Prerequisites

Quick Start — GitHub Actions (Recommended)

1. One-time setup

2. Bootstrap (first time only — run locally)

3. Upload Lambda package

4. Deploy

Quick Start — Local Terraform

1. Bootstrap the state backend

2. Upload Lambda package

3. Deploy each module in order

4. Test end-to-end

⚠️ Important: Editing terraform.tfvars for Local Runs

Option A — Use the helper script (recommended)

Option B — Edit terraform.tfvars by hand

Variables required across all modules

Compliance Profiles

Compliance comparison table

KMS Key Policy — Important Note

Documentation

AI Assistant Integration (ARC IaC MCP)

Setup (one minute)

Example prompts to try

License

Contributing

Acknowledgements

⚠️ Important: Editing `terraform.tfvars` for Local Runs

Option B — Edit `terraform.tfvars` by hand