feat(kubernetes): add service account name to KubernetesRunner config…

[jbclaramonte]

…uration

Changes: AWS IRSA Support and Kubernetes Security Enhancements

Overview

This document describes two critical enhancements to Zoe's Kubernetes runner that enable secure access to AWS resources (like MSK) and compliance with modern Kubernetes security policies.

1. AWS IRSA (IAM Roles for Service Accounts) Support

What is IRSA?

IRSA is an AWS EKS feature that allows Kubernetes pods to assume IAM roles without requiring:

• AWS credentials stored in environment variables
• Instance profiles on EC2 nodes
• Shared credentials across multiple pods

Instead, EKS uses a Service Account with annotations to map it to an IAM role, and pods automatically receive temporary AWS credentials via the AWS STS (Security Token Service).

Changes Made

1.1. KubernetesRunner Configuration (zoe-service/src/runners/kubernetes.kt)

Added serviceAccountName field to the Config data class:

data class Config(
    val deletePodsAfterCompletion: Boolean,
    val zoeImage: String,
    val cpu: String,
    val memory: String,
    val timeoutMs: Long?,
    val annotations: Map<String, String>,
    val serviceAccountName: String?  // ← NEW: Optional service account name
)

Modified pod generation to set the service account:

private fun generatePodObject(image: String, annotations: Map<String, String>, args: List<String>): Pod {
    val pod = loadFileFromResources("pod.template.json")?.parseJson<Pod>() ?: userError("pod template not found !")
    return pod.apply {
        metadata.name = "zoe-${UUID.randomUUID()}"
        metadata.labels = labels
        metadata.annotations = annotations

        // Set service account name if provided
        configuration.serviceAccountName?.let {
            spec.serviceAccountName = it
        }

        // ... rest of configuration
    }
}

1.2. CLI Configuration (zoe-cli/src/config/config.kt)

Extended KubernetesRunnerConfig:

data class KubernetesRunnerConfig(
    val namespace: String,
    val context: String? = null,
    val cpu: String = "0.5",
    val memory: String = "512M",
    val deletePodsAfterCompletion: Boolean = true,
    val timeoutMs: Long = 300000,
    val image: DockerImageConfig = DockerImageConfig(),
    val annotations: Map<String, String> = emptyMap(),
    val serviceAccountName: String? = null  // ← NEW: Service account configuration
)

Usage Example

Example configuration file (examples/config/kubernetes/service-account-example.yml):

clusters:
  default:
    props:
      bootstrap.servers: "broker:9092"
      # ... other Kafka properties

runners:
  default: "kubernetes"
  config:
    kubernetes:
      namespace: "zoe-namespace"
      serviceAccountName: "zoe-service-account"  # Service Account for IRSA
      cpu: "1"
      memory: "512M"
      timeoutMs: 300000
      deletePodAfterCompletion: true

Required Kubernetes Service Account setup:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: zoe-service-account
  namespace: zoe-namespace
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ZoeKafkaAccess

IAM Role Trust Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.region.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.region.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:zoe-namespace:zoe-service-account"
        }
      }
    }
  ]
}

Benefits

✅ No credential management: No need to inject AWS credentials into pods
✅ Fine-grained access control: Each service account can have different IAM permissions
✅ Audit trail: AWS CloudTrail logs show which service account made each API call
✅ Automatic credential rotation: STS credentials are automatically refreshed
✅ Works with MSK IAM authentication: Enables secure access to AWS MSK clusters

---

2. Kubernetes Security Context Enhancements

The Problem

Modern Kubernetes clusters (especially EKS, GKE in production environments) enforce strict security policies via admission webhooks like Gatekeeper or Pod Security Admission. These policies reject pods that don't meet minimum security standards.

Original error encountered:

admission webhook "validation.gatekeeper.sh" denied the request:
[psp-allow-privilege-escalation-container] Privilege escalation container is not allowed: create-output-file
[psp-required-drop-capabilities] init container <create-output-file> is not dropping all required capabilities.
Container must drop all of ["AUDIT_WRITE", "DAC_OVERRIDE", "FOWNER", "FSETID", "KILL", "MKNOD", "NET_RAW", "SETFCAP", "SETPCAP", "SYS_CHROOT"] or "ALL".

Root Cause

The original pod.template.json had no securityContext definitions, which meant:

Default Behavior	Security Policy Requirement
❌ allowPrivilegeEscalation: true	✅ Must be false
❌ All Linux capabilities enabled	✅ Must drop all capabilities
❌ Can run as root (UID 0)	✅ Must run as non-root
❌ No seccomp profile	✅ Must use RuntimeDefault

Changes Made to pod.template.json

2.1. Pod-level Security Context

Added to spec.securityContext:

{
  "spec": {
    "securityContext": {
      "runAsNonRoot": true,       // Prevents running as root
      "runAsUser": 1000,          // Forces non-privileged UID
      "fsGroup": 1000,            // Ensures shared volume access
      "seccompProfile": {
        "type": "RuntimeDefault"  // Enables syscall filtering
      }
    },
    // ... rest of spec
  }
}

Why this matters:

• runAsNonRoot: If a container is compromised, the attacker doesn't have root privileges
• fsGroup: All containers in the pod can read/write the shared /output volume
• seccompProfile: Blocks dangerous system calls like ptrace, reboot, mount, etc.

2.2. Container-level Security Contexts

Added to each container (init container create-output-file, main container zoe, and sidecar tailer):

{
  "securityContext": {
    "allowPrivilegeEscalation": false,  // Blocks setuid/setgid exploits
    "capabilities": {
      "drop": ["ALL"]                   // Removes all Linux capabilities
    },
    "runAsNonRoot": true,
    "runAsUser": 1000,
    "seccompProfile": {
      "type": "RuntimeDefault"
    }
  }
}

Linux Capabilities Dropped:

By dropping ALL capabilities, we remove privileges like:

• CAP_NET_RAW: Creating raw sockets (network sniffing)
• CAP_SYS_ADMIN: Mounting filesystems, loading kernel modules
• CAP_DAC_OVERRIDE: Bypassing file permissions
• CAP_KILL: Sending signals to arbitrary processes
• CAP_SETUID/CAP_SETGID: Changing user/group IDs

Why Zoe doesn't need these capabilities:

• Zoe only needs to connect to Kafka over network sockets (standard TCP)
• Write to its own /output volume
• Execute Java code

None of these operations require special Linux capabilities.

Security Benefits

✅ Least privilege principle: Pods run with minimal permissions
✅ Defense in depth: Security at both pod and container levels
✅ Exploit mitigation: Even if a container is compromised, damage is limited
✅ Compliance: Meets PCI-DSS, SOC2, and other security standards
✅ Production-ready: Compatible with hardened Kubernetes clusters

Before and After Comparison

Before (INSECURE):

{
  "name": "create-output-file",
  "image": "alpine:3.9.5",
  "command": ["touch", "/output/response.txt"],
  "volumeMounts": [...]
  // ❌ No securityContext - runs with default privileges
}

After (SECURE):

{
  "name": "create-output-file",
  "image": "alpine:3.9.5",
  "command": ["touch", "/output/response.txt"],
  "volumeMounts": [...],
  "securityContext": {
    "allowPrivilegeEscalation": false,
    "capabilities": { "drop": ["ALL"] },
    "runAsNonRoot": true,
    "runAsUser": 1000,
    "seccompProfile": { "type": "RuntimeDefault" }
  }
}

---

Testing

Test Coverage

Added comprehensive tests in zoe-service/test/runners/KubernetesRunnerTest.kt:

• ✅ Verify service account name is correctly set when provided
• ✅ Verify service account is not set when not configured
• ✅ Verify security contexts are present in all containers
• ✅ Verify pod-level security context is configured

Manual Testing

# Build with changes
./gradlew clean zoe-cli:installDist

# Test with IRSA-enabled configuration
aws-vault exec <profile> -- zoe-dev -e <env-with-service-account> topics list

# Verify pod is created with correct service account
kubectl get pod <zoe-pod-name> -o jsonpath='{.spec.serviceAccountName}'

# Verify security contexts
kubectl get pod <zoe-pod-name> -o yaml | grep -A 10 securityContext

---

Deployment Considerations

For IRSA Support

1. Create IAM Role with necessary permissions (MSK access, S3, etc.)
2. Set up OIDC Provider for your EKS cluster (usually done during cluster creation)
3. Create Kubernetes Service Account with eks.amazonaws.com/role-arn annotation
4. Update Zoe configuration to reference the service account name

For Security Contexts

✅ No additional setup required - these changes make Zoe compatible with secured clusters by default

⚠️ Note: If your cluster uses a different UID range (e.g., enforces UID > 10000), you may need to adjust runAsUser in the template.

---

Backward Compatibility

✅ Fully backward compatible:

• serviceAccountName is optional - if not specified, pods use the default service account
• Security contexts don't break existing functionality - Zoe containers don't need special privileges
• Works on both legacy and modern Kubernetes clusters

---

References

• AWS IRSA Documentation
• Kubernetes Pod Security Standards
• Gatekeeper Policy Examples
• Linux Capabilities
• Seccomp Profiles