Allow running command containers as non-root user with EBS backed (and other) workspace volumes

[patrobinson]

When using a storage provisioner, such as aws-ebs-csi, the default permission mask for the volume mount is 755 and there's no way to change it.

That means when you specify a workspace from one of these provisioners you get a volume owned by root, which is not-writable by non-root users:

config.yaml snipped

config:
  agent-config:
    hooksVolume:
      name: buildkite-hooks
      configMap:
        defaultMode: 493
        name: buildkite-agent-hooks
  workspace-volume:
    ephemeral:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 1Gi
    name: workspace-volume

configMap pre-checkout hook:

#!/bin/sh

ls -l /
ls -l /workspace
ls -l /workspace/sockets || true

This means using a command container that has the USER set to something other than root isn't possible.
Firstly our agent can't start, because it needs access to write a job-api socket file. You get:

Error: Error setting up Job API: creating job API server: creating socket server: creating socket directory: mkdir /workspace/sockets: permission denied

This can be worked around by mounting an in-memory container to the /workspace/sockets directory with the following config snipped:

  pod-spec-patch:
    containers:
      - name: container-0
        volumeMounts:
          - name: socket-dir
            mountPath: /workspace/sockets
    volumes:
      - name: socket-dir
        emptyDir:
          medium: Memory

But the command container still has no ability to write to /workspace directory, which is a common use case.

We already handle this issue within the checkout container by ensuring all files are created as the same user as the SecurityContext provided https://github.com/buildkite/agent-stack-k8s/blob/main/internal/controller/scheduler/scheduler.go#L837-L838

But since the volume as mounted as 755, the checkout container has no permission to write to the directory.

I tried to patch this by telling the checkout container to change ownership of the directory to the relevant user before it changes users,

                checkoutContainer.Args = []string{fmt.Sprintf(`set -exufo pipefail
 addgroup -g %d buildkite-agent
 adduser -D -u %d -G buildkite-agent -h /workspace buildkite-agent
+chown -R %d:%d /workspace
 su buildkite-agent -c "%s && buildkite-agent-entrypoint bootstrap"`,
+                       podGroup,
+                       podUser,
                        podGroup,
                        podUser,
                        gitConfigCmd,

But it causes the agent to fail to start with:

2024-12-13 04:00:30 INFO   Build Path doesn't exist, creating it (/workspace/build)
fatal: failed to create builds path: mkdir /workspace/build: permission denied

So I landed on this solution, copying the existing implementation that checkout uses to ensure it runs as root then chown the directory in the init container.

Result:

[patrobinson]

This could be refactored, it's pretty similar to the createCheckoutContainer code

But gathering early feedback first.

[DrJosh9000]

This seems like a good solution.

[patrobinson]

@DrJosh9000 thanks, I've marked this as ready for review

[DrJosh9000]

LGTM with one name change. I also have some minor cleanup suggestions but these also apply to the existing createCheckoutContainer, so feel free to split off into a followup.