Bound how long a wedged Update can hold a reconcile worker

[bpalermo]

Proposed changes

An update controller reconcile blocks for the entire Pulumi operation, so each one occupies a MaxConcurrentReconciles slot for as long as that operation runs. Nothing bounded it: the reconcile context has no deadline (the only context.WithTimeout covers Connect), all four operation RPCs use grpc.WaitForReady(true), and an agent that is alive but wedged inside a provider call keeps its connection perfectly healthy while never sending another message — so Recv() blocks indefinitely. Because the reconcile never returns, Progressing=True is never revised and the deferred activeReconciles cleanup never runs, so mapWorkspaceToUpdate also declines to re-enqueue it.

Idle timeout on the operation stream

New --update-idle-timeout (default 30m), overridable per update via Update.spec.idleTimeout. Because applyUpdateTemplate merge-patches the whole Update, this is reachable from a Stack through spec.updateTemplate.spec.idleTimeout with no Stack-side plumbing.

It measures silence rather than total duration and is reset on every message, so a slow but progressing update is never interrupted — which matters given #1293 reports legitimate runs reaching 2h22m and 13.8h. 0 disables it.

The operation runs on a context derived from the reconcile's rather than the reconcile context itself. That distinction is the point: setStatusBlockFromGRPCErr sets Complete=True in memory and the caller then persists it with updateStatus(ctx, obj). Cancelling the reconcile context to stop the stream would fail that write, leaving the Update Progressing forever — the very state being fixed. When the reconcile context is cancelled instead, as on manager shutdown, the write is skipped deliberately so the restarted operator's existing Progressing guard marks it Aborted.

Tightened TCP keepalive to workspace pods

Worth being precise about what this does and does not buy, because my first reading of the problem was wrong. net.Dialer already enables keepalive with a 15s idle, so these connections were never unprobed — I verified SO_KEEPALIVE=1 and a 15s idle on a plain (&net.Dialer{}).Dial. What the defaults leave to the system is the probe interval and count: 75s × 9 on Linux, so a black-holed peer takes roughly 11 minutes to notice. This brings that to about a minute.

So it is hardening, not the fix — a lost node or partition is bounded sooner. It cannot see a connection that is healthy while the operation is wedged; that is what the idle timeout is for. It also benefits the workspace controller, which has the same unbounded-RPC exposure over the same dial path.

Deliberately TCP-level rather than gRPC-level. gRPC pings are policed by the server's keepalive.EnforcementPolicy, whose default MinTime is 5 minutes; since the agent image is pinned per Stack, the operator routinely talks to older agents whose policy it cannot know, and exceeding it earns a GOAWAY / ENHANCE_YOUR_CALM that would kill healthy long-running updates. TCP probes are invisible to HTTP/2, so they are safe against every agent version ever shipped.

Agent-side server keepalive

keepalive.ServerParameters on the agent, so it notices a departed operator, closes the transport, and lets the orphaned operation unwind via the existing Cancel() path. Server-initiated pings are not subject to any enforcement policy, so this needs no client cooperation. No EnforcementPolicy is set, on purpose: tightening its MinTime would let a new agent reject pings from a client behaving correctly for the default policy it was written against.

A separate worker budget for the update controller

MaxConcurrentReconciles was set once, manager-wide. The update controller is the only one whose reconciles block for hours, so sharing a pool means long operations crowd out the stack and workspace controllers — and queue the update controller's own cheap terminal transitions (deleting → Canceled) behind hour-long ones. That priority inversion is why a deleted Update could sit in Terminating unable to reach Complete, which is what the finalizer needs.

New --update-max-concurrent-reconciles (0 = use the shared value).

Honour GRACEFUL_SHUTDOWN_TIMEOUT_DURATION

It is set to 5m in deploy/operator_template.yaml, and read nowhere — so shutdown used controller-runtime's 30s default and cut short exactly the in-flight updates the pod's 300s terminationGracePeriodSeconds was sized to allow. Now wired, with a matching flag.

Tuning-surface fixes called out in the issue

controllerOpts.MaxConcurrentReconciles, _ = strconv.Atoi(s)

The discarded error made a malformed MAX_CONCURRENT_RECONCILES silently 0, which controller-runtime treats as 1 — a silent drop from 25 workers to one with nothing in the logs. Adds --max-concurrent-reconciles, rejects non-positive values, and applies the same non-silent parsing to the adjacent LEADER_ELECTION_* / KUBE_API_TIMEOUT overrides. Also drops a misleading empty controller.Options{} in the stack controller's setup.

Docs

docs/metrics.md gains a section on reading the update controller's workqueue, since resource usage cannot distinguish idle from saturated here. Per the issue's own correction, it states plainly what workqueue_longest_running_processor_seconds cannot show — it tracks only the longest actively-running reconcile, so it is blind to Updates stranded by a killed operator, and a large value is not proof of a problem — and gives the detection method that actually worked: group Progressing=True Updates by lastTransitionTime, and gate any deletion on the workspace's pulumi container start time, not the pod's. Also corrects the stated MaxConcurrentReconciles default from 10 to 25.

Testing

• TestUpdateIdleTimeout — a silent stream is abandoned and the terminal status actually reaches the API server (the point of the derived context); asserts Complete=True, Failed=True / IdleTimeout.
• TestUpdateIdleTimeoutDisabled — a zero timeout arms nothing, and a long gap is not cut off.
• TestIdleTimeoutFor — precedence between the per-Update value and the operator default, including that an explicit zero disables rather than falls back.
• connect_test.go — the dialer applies the tuned probe interval and count, read back off the socket. Note it asserts the tuned values rather than SO_KEEPALIVE, because that is on by default and would pass regardless; I checked the test fails when the tuning is removed.
• main_test.go — table tests over valid / malformed / empty / zero / negative for the env overrides and the concurrency validation.

make test (operator + agent) and golangci-lint are clean, with no codegen drift. make test-e2e has not been run, and the failure mode has not been reproduced end-to-end against a live cluster.

Related issues

Fixes #1293. The Terminating-with-finalizer half of that issue is handled in #1295, which touches the same stack-controller function as its own fix.

🤖 Generated with Claude Code

[github-actions]

PR is now waiting for a maintainer to run the acceptance tests. This PR will only perform build and linting.
Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR