Best practice for handling credential switching within Concourse tasks (cross-account AWS example)

[Tobi-Ajet]

Context:
We're copying database backups across AWS accounts. Currently using tasks with manual aws sts assume-role, but I saw an example using S3 resources with aws_assume_role_arn per resource.

Current Approach (Task-Based):

- task: copy-backup
  params:
    source_bucket: account-a-backups
    dest_bucket: account-b-backups
    target_role_arn: arn:aws:iam::111222333444:role/worker-role
  config:
    run:
      path: bash
      args:
      - -c
      - |
        # Download with Account A creds
        aws s3 cp s3://${source_bucket}/backup /tmp/backup --recursive
        
        # Assume Account B role
        creds=$(aws sts assume-role --role-arn "${target_role_arn}" ...)
        export AWS_ACCESS_KEY_ID=$(...)
        
        # Upload with Account B creds
        aws s3 cp /tmp/backup s3://${dest_bucket}/backup --recursive

Alternative Approach (Resource-Based):

resources:
- name: source-backup
  type: s3  # or s3-dir
  source:
    aws_assume_role_arn: arn:aws:iam::999888777666:role/worker-role-a
    bucket: account-a-backups
    bucket_region: us-east-1

- name: dest-backup
  type: s3
  source:
    aws_assume_role_arn: arn:aws:iam::111222333444:role/worker-role-b
    bucket: account-b-backups
    bucket_region: us-east-1

jobs:
- name: copy-backup
  plan:
  - get: source-backup
  - put: dest-backup
    params:
      file: source-backup/*

Question:

Should we refactor to use the resource-based approach?

Pros I see:

• ✅ More "declarative" (Concourse resources vs bash scripts)
• ✅ No manual credential management
• ✅ Cleaner pipeline YAML
• ✅ Concourse handles retries/verification

Cons I see:

• ❌ Our backups are created async by Lambda (need retry logic)
• ❌ Need file count verification (source vs dest)
• ❌ Folder names are dynamic (from Lambda response via load_var)
• ❌ We re-encrypt with different KMS keys

Specific Challenges:

1. Dynamic paths: Backup folder name comes from Lambda:

   - task: get-latest-backup
     outputs: [backup-info]
   - load_var: backup-folder
     file: backup-info/folder-name.txt

Can we use ((.:backup-folder)) in resource get params?

2. Async creation: Lambda creates backup over 5-10 minutes:

   - task: invoke-lambda-backup
   # Backup doesn't exist yet...
   - get: source-backup  # Will this fail/retry automatically?

3. KMS re-encryption: Need to re-encrypt with Account B's KMS key:

   - put: dest-backup
     params:
       file: source-backup/*
       sse: aws:kms
       sse_kms_key_id: arn:aws:kms:us-east-1:111222333444:key/abc-123

Does the S3 resource support this?

4. Verification: We verify file counts match:

   SOURCE_COUNT=$(aws s3 ls s3://source/backup/ --recursive | wc -l)
   DEST_COUNT=$(aws s3 ls s3://dest/backup/ --recursive | wc -l)

How would we do this with resources?

Is there a hybrid approach?

Maybe use resources for the actual copy, but tasks for orchestration:

- task: invoke-lambda-backup
- task: wait-for-backup-ready
  # Polls until backup exists
- get: source-backup
  passed: [wait-for-backup-ready]
- put: dest-backup
- task: verify-file-counts

What's the recommended pattern for:

• Cross-account S3 operations?
• Dynamic resource paths?
• Async resource creation?
• Post-copy verification?

---

Environment:

• Concourse 7.11.2
• Backups: 2-10GB, 20/week
• 3 accounts: Source (999888777666) → Intermediate (555444333222) → Destination (111222333444)

[taylorsilva]

Hi Tobi! Hope you're doing well 😊

This whole thing looks LLM generated. I think you guys made this after I left, but what the LLM is suggesting would work and is how I would have tried to write it. You can read the S3 resource's docs here: https://github.com/concourse/s3-resource

The latest version of the s3-resource does support using the IAM role on the worker. You need to either:

1. Upgrade Concourse to >= 7.14.0, or
2. Override the bundled version of the s3-resource in your pipelines and use v2.4.0 of the s3 resource: https://hub.docker.com/layers/concourse/s3-resource/2.4.0/images/sha256-2df5fbd7a9dff720f3239e76233a3590d6d306be6edf554ba0ef2a1a7384c586
   Add this to your pipeline:

resource_types:
- name: s3
  type: registry-image
  source:
    repository: concourse/s3-resource
    tag: 2.4.0

The version of Concourse you are on, 7.11.2, comes with v1.3.0 of the S3 resource: https://github.com/concourse/concourse/releases/tag/v7.11.2

[Tobi-Ajet]

Hey Taylor, thanks so much for the detailed feedback and suggestions! I really appreciate you taking the time to explain the S3 resource approach and the version override option. I hope you're doing well as well