feat(comet_eks): byo_s3_irsa_roles — customer bring-your-own-S3 IAM (DND-1423)

[jms200]

User description

What

Adds a reusable, opt-in byo_s3_irsa_roles map to the module. Each entry provisions the IAM that accompanies a customer-supplied S3 bucket:

• one IRSA role (assumable via web-identity by the listed Kubernetes ServiceAccounts),
• a customer-managed policy scoped to the supplied bucket ARN(s),
• the attachment.

It follows the existing Loki IRSA triplet exactly (data.aws_iam_policy_document → aws_iam_policy → module ... iam-role-for-service-accounts-eks ~> 5.39), so the OIDC :sub/:aud trust is built by the upstream module from namespace_service_accounts.

byo_s3_irsa_roles = {
  s3access = {
    bucket_arns                = ["arn:aws:s3:::my-bucket"]   # perms scoped to these (+ /*)
    namespace_service_accounts = ["ns:opik-clickhouse", ...]  # drives the IRSA :sub trust
    actions                    = null   # optional; defaults to the CH-backup S3 action set
    role_name_override         = null   # optional; adopt an out-of-band role in place
    policy_name_override       = null   # optional; adopt an out-of-band policy in place
  }
}

Why

Generalizes the BYO-S3 access role that was previously created out-of-band during onboarding (e.g. Zoox's ZooxS3Access). Because that role's trusted-SA list lived only in live AWS and not in code, a missing ServiceAccount went unreviewed and silently broke ClickHouse remote backups for ≥7 days (DND-1413). Codifying it makes the trust list reviewable in PRs, and any future BYO-S3 customer gets the same primitive from day one instead of a hand-rolled role.

Design notes

• Least-privilege by default — permissions are scoped to the supplied bucket ARN(s), not arn:aws:s3:::*.
• Adoptable in place — role_name_override / policy_name_override let an existing out-of-band role/policy be adopted via terraform import without recreation (stable ARN → IRSA annotations keep working).
• Zero blast radius — empty map default; no effect on any existing consumer.
• SA list is an input, not hardcoded — grounded in live AWS: Zoox trusts 4 SAs, circuit trusts a different 4; the set genuinely varies by install type.

Testing

• terraform init -backend=false + terraform validate → "the configuration is valid" (only pre-existing deprecation warnings, unrelated to this change).
• terraform fmt clean.

Release

New MINOR (new feature family) — tag v1.21.0 after merge.

Follow-up (separate PRs, not here)

• comet-devops Zoox wrapper: bump ?ref → v1.21.0 and adopt ZooxS3Access onto this feature via byo_s3_irsa_roles + import blocks (normalize-on-adopt: trims the out-of-band node-role / s3.amazonaws.com / ad-hoc-user trust statements and scopes perms to the bucket).

Ref: DND-1423 (follow-up to DND-1413).

🤖 Generated with Claude Code

---

Generated description

Below is a concise technical summary of the changes proposed in this PR:
Add a new byo_s3_irsa_roles input to the root comet_eks module and its modules/comet_eks implementation to provision per-bucket IRSA roles, customer-managed S3 policies, and policy attachments for customer-supplied buckets. Generalize the existing iam-role-for-service-accounts-eks flow so namespace_service_accounts drive OIDC trust while keeping ClickHouse backup access scoped and opt-in.

Topic	Details
BYO S3 access	Provision per-bucket IRSA roles and scoped S3 policies for customer-owned buckets via aws_iam_policy_document, aws_iam_policy, and iam-role-for-service-accounts-eks. Modified files (3) main.tf modules/comet_eks/main.tf variables.tf Latest Contributors(2) User Commit Date jms200 feat(comet_eks): add b... July 22, 2026 alexb@comet.com feat(providers): upgra... July 20, 2026
Input validation	Validate bucket ARNs, ServiceAccount subjects, and optional name overrides so BYO-S3 entries stay safe and adoptable in place. Modified files (1) modules/comet_eks/variables.tf Latest Contributors(2) User Commit Date jms200 fix(comet_eks): valida... July 22, 2026 alexb@comet.com feat(providers): upgra... July 20, 2026

_{Review this PR on Baz | Customize your next review}