Controller liveness/readiness probe timeouts are not configurable (hardcoded to Kubernetes default of 1s)

[aashishshrestha09]

What happened?

The controller-manager Deployment in the Helm chart hardcodes its liveness and readiness probes and never sets timeoutSeconds or failureThreshold:

https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/deploy/helm/pulumi-operator/templates/deployment.yaml#L98-L109

Because timeoutSeconds is unset, it falls back to the Kubernetes default of timeoutSeconds: 1 (with failureThreshold: 3).

When the manager is busy during long-running Pulumi operations, the /healthz endpoint can fail to respond within that 1-second window. After 3 consecutive misses (~3 × periodSeconds), the kubelet sends SIGTERM (the container exits with code 143) and the in-flight update is cancelled mid-run. On restart the controller can re-enter the same condition, and the cancellation can leave behind orphaned state locks / stuck workspaces that the controller does not self-recover from.

Both /healthz and /readyz are wired to healthz.Ping:

https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/operator/cmd/main.go#L307-L312

so the probe handler itself is trivial — the failures are caused by the process being momentarily starved, which a 1s timeout does not tolerate.

Context

PR #1059 added configurable leader-election timeouts for exactly this "don't interrupt long-running operations" reason, but the probe timeouts were left at the aggressive Kubernetes defaults. Raising the pod's CPU/memory limits does not help, since the kill is driven by probe latency (SIGTERM/143), not OOM.

Expected

Operators should be able to tune the liveness/readiness probes (timeoutSeconds, failureThreshold, periodSeconds, initialDelaySeconds) via values.yaml, the same way leader-election timeouts are already configurable — without forking the chart.

Proposed fix

Make the probes configurable in the chart, preserving the current defaults (timeoutSeconds: 1, failureThreshold: 3) so there is no behavior change for existing users. PR: see below.

Environment

• Chart: deploy/helm/pulumi-operator
• Operator: v2.7.0

[aashishshrestha09]

Proposed fix in #1255 — makes the probes configurable via values.yaml while preserving the current defaults.

[guineveresaenger]

Hi @aashishshrestha09, thank you for filing this feature request.

Can you tell us a bit more on why this is necessary? I understand that configurability might be in general desirable, but I would like to see an example of the hypothetical scenario described, as I'm not seeing other reports of in-flight updates being cancelled due to this.