feat(eks): land Chunks C+D on main — module is Kubernetes-API-free (v5.0.0) (#56)

* feat(eks): remove all kubernetes_* resources; module is now k8s-API-free in effect (v5.0.0)

Deletes the last in-cluster resources this module created, moving each to its
GitOps owner:
  - storage_class.gp3 / .comet_generic       -> comet-infra umbrella (ArgoCD)
  - namespace.monitoring                      -> comet-infra umbrella (ArgoCD)
  - secret.monitoring                         -> External Secrets Operator (owned)
  - annotations.app/admin_ns_node_selector    -> DROPPED (obsolete under Auto Mode;
                                                 NodePools/NodeClasses schedule now)
  - namespace.redis_insights                  -> agentro-role/rbac local module

Also removes time_sleep.wait_for_alb_webhook (only the storage classes / monitoring
ns depended on it) and every now-orphaned variable at both module and root level:
enable_monitoring_setup, manage_monitoring_secret, monitoring_namespace,
create_comet_generic_storage_class, storage_class_reclaim_policy (+ eks_* root
aliases), enable_namespace_nodegroup_pinning, app_namespace, admin_pinned_namespaces,
enable_redis_insights_ns, and the comet_eks pass-throughs for all of them.

Kept: grafana_admin_user / grafana_admin_password root vars — they feed
comet_secretsmanager (the AWS Secrets Manager secret ESO reads), not the EKS module.

No kubernetes_/helm_/kubectl_ resource remains in the module. The kubernetes
provider is now declared-but-unused; it is dropped in the next chunk (D) together
with time_sleep.wait_for_cluster_access. terraform init + validate pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(eks): drop kubernetes provider — module is Kubernetes-API-free (v5.0.0)

With every kubernetes_* resource gone (Chunk C), the kubernetes provider was
declared-but-unused. Remove it from required_providers (root + comet_eks) and
delete the exec-auth provider "kubernetes" config block in providers.tf (the
chicken-and-egg source: it wired the provider to the not-yet-reachable cluster
endpoint). Also delete time_sleep.wait_for_cluster_access — the last time_sleep,
now with zero dependents.

Result: terraform providers lists only aws / tls / time / random (+ cloudinit /
null, transitive node-bootstrap deps of terraform-aws-modules/eks). No
kubernetes/helm/kubectl anywhere. A step-1 apply on a private-endpoint EKS cluster
the runner cannot yet reach now makes ZERO Kubernetes API-server connections;
all in-cluster state is owned by ArgoCD (comet-infra) + ESO + the agentro RBAC
module. terraform init + validate pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* remove terraform.tfvars

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: obezpalko

README.md

@@ -14,6 +14,7 @@ Terraform module for deploying infrastructure components to run CometML.
   - Clone the repository to your local machine: `git clone https://github.com/comet-ml/terraform_aws_comet.git`
   - Move into the deployment directory: `cd terraform-aws-comet`
   - Initialize the directory: `terraform init`
+  - Copy the example variables file: `cp terraform.tfvars.example terraform.tfvars` (`terraform.tfvars` is gitignored — never commit it)
   - Within terraform.tfvars, set your module toggles to enable the desired infrastructure components and set any related inputs
   - Provision the resources: `terraform apply`
 

main.tf

@@ -324,10 +324,6 @@ module "comet_eks" {
   external_secrets_iam_role_name_override = var.external_secrets_iam_role_name_override
   secretsmanager_environment              = var.secretsmanager_environment
 
-  # Storage class configuration
-  storage_class_reclaim_policy       = var.eks_storage_class_reclaim_policy
-  create_comet_generic_storage_class = var.eks_create_comet_generic_storage_class
-
   # Loki IRSA for S3 access
   enable_loki                 = var.enable_loki_bucket
   loki_s3_bucket_arn          = var.enable_s3 && var.enable_loki_bucket ? module.comet_s3[0].comet_loki_bucket_arn : null
@@ -340,13 +336,6 @@ module "comet_eks" {
   enable_cloudwatch_exporter                 = var.enable_cloudwatch_exporter
   cloudwatch_exporter_iam_role_name_override = var.cloudwatch_exporter_iam_role_name_override
 
-  # Monitoring namespace and Grafana credentials
-  enable_monitoring_setup  = var.enable_monitoring_setup
-  manage_monitoring_secret = var.manage_monitoring_secret
-  monitoring_namespace     = var.monitoring_namespace
-  grafana_admin_user       = var.grafana_admin_user
-  grafana_admin_password   = var.grafana_admin_password
-
   # EKS Auto Mode (mutually exclusive with Karpenter)
   enable_auto_mode     = var.eks_enable_auto_mode
   auto_mode_node_pools = var.eks_auto_mode_node_pools
@@ -373,13 +362,6 @@ module "comet_eks" {
   enable_ci_runners_eks_api_access    = var.enable_ci_runners_eks_api_access
   ci_runners_cidr                     = var.ci_runners_cidr
 
-  # Namespace nodegroup pinning
-  enable_namespace_nodegroup_pinning = var.enable_namespace_nodegroup_pinning
-  app_namespace                      = var.app_namespace
-  admin_pinned_namespaces            = var.admin_pinned_namespaces
-
-  # Redis Insights namespace + agentro port-forward RBAC
-  enable_redis_insights_ns = var.enable_redis_insights_ns
 }
 
 module "comet_elasticache" {

modules/comet_eks/main.tf

@@ -583,13 +583,6 @@ module "irsa-ebs-csi" {
   }
 }
 
-resource "time_sleep" "wait_for_cluster_access" {
-  count = var.eks_enable_cluster_creator_admin_permissions ? 1 : 0
-
-  depends_on      = [module.eks]
-  create_duration = "60s"
-}
-
 # State migration: native addons moved out of the eks_blueprints_addons module.
 # coredns/kube-proxy/metrics-server now live on module.eks.addons (aws_eks_addon.this),
 # vpc-cni is provisioned before_compute (aws_eks_addon.before_compute), and
@@ -730,93 +723,11 @@ resource "aws_iam_role_policy" "external_dns" {
   policy = data.aws_iam_policy_document.external_dns[0].json
 }
 
-# Best-effort settle delay before this module creates Services/objects that a
-# webhook might mutate.
-#
-# The AWS Load Balancer Controller is now deployed out-of-band by ArgoCD, NOT by
-# this module, so its webhook readiness is OUTSIDE Terraform's dependency graph —
-# there is no in-graph resource to wait on. We therefore cannot truly gate on the
-# ALB webhook here; this is a fixed post-cluster-access delay only. Real ordering
-# for anything that depends on the ALB webhook must be enforced ArgoCD-side (sync
-# waves / health checks), not here. depends_on is on wait_for_cluster_access
-# (which this module DOES own), not on eks_blueprints_addons (which no longer
-# installs the controller).
-resource "time_sleep" "wait_for_alb_webhook" {
-  count = var.eks_aws_load_balancer_controller ? 1 : 0
-
-  depends_on      = [time_sleep.wait_for_cluster_access]
-  create_duration = "60s"
-}
-
-locals {
-  # Build tag specifications for EBS CSI driver
-  # Each tag needs to be a separate tagSpecification_N parameter with format "key=value"
-  # Note: common_tags passed from root module already includes Terraform=true and Environment tags
-  common_tags_list = [for k, v in var.common_tags : "${k}=${v}"]
-
-  # Base tags for gp3 storage class (only StorageClass identifier, other tags come from common_tags)
-  gp3_base_tags  = ["StorageClass=gp3"]
-  gp3_all_tags   = concat(local.gp3_base_tags, local.common_tags_list)
-  gp3_tag_params = { for idx, tag in local.gp3_all_tags : "tagSpecification_${idx + 1}" => tag }
-
-  # Base tags for comet-generic storage class (only StorageClass identifier, other tags come from common_tags)
-  comet_generic_base_tags  = ["StorageClass=comet-generic"]
-  comet_generic_all_tags   = concat(local.comet_generic_base_tags, local.common_tags_list)
-  comet_generic_tag_params = { for idx, tag in local.comet_generic_all_tags : "tagSpecification_${idx + 1}" => tag }
-}
-
-resource "kubernetes_storage_class" "gp3" {
-  depends_on = [time_sleep.wait_for_cluster_access]
-
-  metadata {
-    name   = "gp3"
-    labels = var.common_tags
-    annotations = {
-      "storageclass.kubernetes.io/is-default-class" = "true"
-    }
-  }
-
-  storage_provisioner = "ebs.csi.aws.com"
-
-  parameters = merge(
-    {
-      type = "gp3"
-      # Optionally, set iops and throughput:
-      # iops       = "3000"
-      # throughput = "125"
-    },
-    local.gp3_tag_params
-  )
-
-  reclaim_policy         = var.storage_class_reclaim_policy
-  volume_binding_mode    = "WaitForFirstConsumer"
-  allow_volume_expansion = true
-}
-
-resource "kubernetes_storage_class" "comet_generic" {
-  # Some deployments have comet-generic created by the comet-ml Helm chart
-  # (Helm/ArgoCD-owned). Set create_comet_generic_storage_class=false there so
-  # this module does not fight the chart over ownership of the same SC.
-  count = var.create_comet_generic_storage_class ? 1 : 0
-
-  depends_on = [time_sleep.wait_for_cluster_access]
-
-  metadata {
-    name   = "comet-generic"
-    labels = var.common_tags
-  }
-
-  storage_provisioner = "ebs.csi.aws.com"
-
-  parameters = merge(
-    { type = "gp3" },
-    local.comet_generic_tag_params
-  )
-
-  reclaim_policy         = var.storage_class_reclaim_policy
-  volume_binding_mode    = "WaitForFirstConsumer"
-  allow_volume_expansion = true
-}
+# StorageClasses (gp3 default + comet-generic) moved to the comet-infra umbrella
+# chart (ArgoCD-owned) — see comet-devops-helm/charts/comet-infra. The former
+# wait_for_alb_webhook settle delay went with them; ordering for ALB-webhook
+# dependents is enforced ArgoCD-side (sync waves), not in Terraform. This module
+# no longer touches the Kubernetes API for storage classes.
 
 #########################################
 #### Cluster Autoscaler IRSA Role ####
@@ -1146,41 +1057,10 @@ module "cloudwatch_exporter_irsa_role" {
 #########################################
 #### Monitoring Namespace and Secrets ####
 #########################################
-resource "kubernetes_namespace" "monitoring" {
-  count = var.enable_monitoring_setup ? 1 : 0
-
-  metadata {
-    name = var.monitoring_namespace
-  }
-
-  depends_on = [
-    module.eks,
-    time_sleep.wait_for_alb_webhook
-  ]
-}
-
-resource "kubernetes_secret" "monitoring" {
-  # Set manage_monitoring_secret = false where the monitoring Secret is owned by
-  # External Secrets Operator (ExternalSecret with creationPolicy: Owner). Letting
-  # Terraform also manage it causes a reconcile fight: TF strips ESO's labels and
-  # replaces the whole data map (dropping ESO-only keys) on every apply.
-  count = var.enable_monitoring_setup && var.manage_monitoring_secret ? 1 : 0
-
-  metadata {
-    name      = "monitoring"
-    namespace = kubernetes_namespace.monitoring[0].metadata[0].name
-  }
-
-  data = {
-    grafana-admin-user     = var.grafana_admin_user
-    grafana-admin-password = var.grafana_admin_password
-  }
-
-  type      = "Opaque"
-  immutable = false
-
-  depends_on = [kubernetes_namespace.monitoring]
-}
+# The monitoring namespace moved to the comet-infra umbrella chart (ArgoCD-owned).
+# The monitoring Secret is owned by External Secrets Operator (ExternalSecret with
+# creationPolicy: Owner) — Terraform no longer creates it. Both are out of this
+# module so it never touches the Kubernetes API for monitoring bootstrap.
 
 #########################################
 #### Karpenter Prerequisites ####
@@ -1626,58 +1506,13 @@ resource "aws_vpc_security_group_ingress_rule" "eks_api" {
 #########################################
 #### Namespace nodegroup pinning ####
 #########################################
-# Annotates namespaces with scheduler.alpha.kubernetes.io/node-selector to route
-# every Pod admitted into the namespace onto a specific node group. Skips
-# kube-system + monitoring (they host DaemonSets and must schedule everywhere).
-# Patches in place — does NOT create the namespace; create via Helm first.
-
-resource "kubernetes_annotations" "app_ns_node_selector" {
-  count = var.enable_namespace_nodegroup_pinning ? 1 : 0
-
-  depends_on = [time_sleep.wait_for_cluster_access]
-
-  api_version = "v1"
-  kind        = "Namespace"
-  metadata {
-    name = coalesce(var.app_namespace, var.environment)
-  }
-  annotations = {
-    "scheduler.alpha.kubernetes.io/node-selector" = "nodegroup_name=comet"
-  }
-  force = true
-}
-
-resource "kubernetes_annotations" "admin_ns_node_selector" {
-  for_each = var.enable_namespace_nodegroup_pinning ? toset(var.admin_pinned_namespaces) : []
-
-  depends_on = [time_sleep.wait_for_cluster_access]
-
-  api_version = "v1"
-  kind        = "Namespace"
-  metadata {
-    name = each.value
-  }
-  annotations = {
-    "scheduler.alpha.kubernetes.io/node-selector" = "nodegroup_name=admin"
-  }
-  force = true
-}
+# DROPPED. The scheduler.alpha.kubernetes.io/node-selector annotations pinned
+# app/admin namespaces onto legacy managed node groups (nodegroup_name=…). Under
+# EKS Auto Mode, scheduling is handled by NodePools/NodeClasses (comet-infra), so
+# this in-cluster patching is obsolete and has been removed.
 
 #########################################
 #### Redis Insights namespace ####
 #########################################
-# Operational debug surface — provides a namespace for the redis-insights helm
-# chart (installed by FRED-helm-apply) pinned to the admin NG.
-
-resource "kubernetes_namespace" "redis_insights" {
-  count = var.enable_redis_insights_ns ? 1 : 0
-
-  depends_on = [time_sleep.wait_for_cluster_access]
-
-  metadata {
-    name = "redis-insights"
-    annotations = {
-      "scheduler.alpha.kubernetes.io/node-selector" = "nodegroup_name=admin"
-    }
-  }
-}
+# Moved to the agentro-role/rbac local module (comet-devops), which owns agentro's
+# in-cluster objects in one place. Not created by this module anymore.