Helm hook Job cannot run under restricted Pod Security admission

[danish9039]

What happened

The Helm hook Job created by hook.upgradeCrd cannot be scheduled in a namespace that enforces the restricted Pod Security Standard, so the CustomResourceDefinition upgrade never runs there. There is no chart value that can fix it.

Details

charts/spark-operator-chart/templates/hook/job.yaml sets a container security context:

securityContext:
  readOnlyRootFilesystem: true
  privileged: false
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

The restricted profile additionally requires runAsNonRoot: true and a seccompProfile. Neither is set, there is no pod-level securityContext, and the hook: values block exposes only upgradeCrd, image, nodeSelector, affinity and tolerations — so an operator cannot supply them either.

The controller and webhook are unaffected: controller.podSecurityContext, controller.securityContext, webhook.podSecurityContext and webhook.securityContext all exist. The hook Job appears to be the one component that was missed.

Reproduction

kubectl create namespace spark-operator
kubectl label namespace spark-operator \
  pod-security.kubernetes.io/enforce=restricted

helm upgrade --install spark-operator spark-operator/spark-operator \
  --namespace spark-operator \
  --set hook.upgradeCrd=true

The hook pod is rejected by admission, so helm upgrade does not complete.

A complication worth flagging

docker/Dockerfile.kubectl builds from alpine and sets no USER, so the image runs as UID 0. Adding runAsNonRoot: true alone would therefore make the pod fail to start with "container has runAsNonRoot and image will run as root". A fix needs a runAsUser as well, or a USER line in the Dockerfile.

Setting runAsUser in the chart looks viable — kubectl does not need root, readOnlyRootFilesystem is already enabled, and the service account token is world-readable — but it should be verified on a cluster rather than assumed.

Proposed change

Add hook.podSecurityContext and hook.securityContext values, wired into templates/hook/job.yaml with the same with pattern used by the controller and webhook templates.

One decision worth your input before I send a pull request:

1. Restricted-compliant defaults — the hook works out of the box on hardened clusters, at the cost of changing current rendering.
2. Empty {} defaults — no behaviour change, and each operator opts in.

I lean towards the first, since the container already drops all capabilities and disallows privilege escalation, but it is your call.

Context

The Kubeflow Community Distribution is adding a Helm chart that wraps this one (kubeflow/community-distribution#3575). It ships with hook.upgradeCrd: false and documents that the definitions are not upgraded by helm upgrade, because the hook cannot run in the kubeflow namespace.

Happy to send the pull request, including helm-unittest coverage in tests/hook/job_test.yaml and a regenerated README.md, once you have indicated which default you prefer.

[github-actions]

🎉 Welcome to the Kubeflow Spark Operator! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon!
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.

Join the community:

• Slack: Join our #kubeflow-spark-on-kubernetes Slack channel.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏