Fix Stack wedging when status.currentUpdate is never cleared

[bpalermo]

Proposed changes

A Stack could be left with status.currentUpdate naming an Update that would never resolve, after which it never started another update — Reconciling=True / Ready=False indefinitely, with the resources it manages silently unreconciled. #1292 reports 77 wedged Stacks across two clusters, the oldest at 174h, each managing live DNS records.

Two shapes, same effect and same manual recovery.

Shape A — the Update was garbage-collected

stack_controller.go logged update object not found; will retry and returned with no requeue and without clearing the reference, so nothing self-healed. The pulumi.com/reconciliation-request annotation could not rescue it either: the check runs before any sync logic, and observedReconcileRequest is persisted by that same branch — so the request was marked observed with nothing done.

• Absence is now confirmed against the API server via an uncached reader, since readCurrentUpdate goes through the cache and IsNotFound there only means "not in the informer". A cold or lagging cache is no longer mistaken for a deleted object.
• An Update that exists but is Terminating is left alone: its deletion bumps generation, which wakes the update controller, which drives it to a terminal state this controller can absorb normally.
• If the Stack's own update is still running it is adopted rather than abandoned, so we never start a second concurrent pulumi operation against one workspace.
• Otherwise the reference is cleared, a new StackUpdateLost event is emitted, and an error is returned so the retry goes through the workqueue's exponential backoff. That backoff is load-bearing: returning nil makes controller-runtime Forget() the item and reset the rate limiter, so an Update being deleted systemically (its Workspace is gone, and every replacement is cascade-deleted in turn) would become an unbounded create/delete loop.

status.lastUpdate is deliberately left untouched. The lost update's real outcome is unknown — most often it never ran — so recording a synthetic failure would both misreport history and spend the Stack's retry budget (maxUpdateFailures) on an operator-side fault. Visibility comes from the event and condition message instead, which is the "nothing distinguishes this from a healthy Stack" complaint in the issue.

Shape B — the Update was present and terminal, but never processed

Here lastUpdate still described the previous update — often a successful one — so the Stack looked healthy while being completely stuck.

The cause is a generation gate. isUpdateComplete required status.observedGeneration == metadata.generation, but:

• nothing writes an Update's status once it is Complete — the update controller short-circuits on that condition and returns without calling updateStatus; while
• the apiserver bumps generation when a deletion is blocked by a finalizer (registry/store.go, generation is incremented on the transition into deleting).

So an Update that completes and is then deleted — by its TTL, or by a workspace cascade — sits at generation N+1 / observedGeneration N permanently, looking forever unfinished. The issue hypothesised "a missed edge on the completion watch"; it is slightly worse than that, because updateCompletePredicate uses the same helper, so Create returned false for such an Update and operator restarts did not even enqueue the Stack. The edge was unreachable, not merely missed.

The gate is dropped. This is safe because an Update's spec is never modified after creation, so a generation bump cannot mean the recorded result is stale.

Two windows that needed no crash at all

The recovery paths after a failed Create and a failed addUpdateFinalizer both did _ = saveStatus(). A conflict on either left status.currentUpdate naming an Update that was deleted or never created — no crash required, which likely makes these the highest-probability cause in a busy fleet. Those errors are now returned.

Releasing Updates stuck in Terminating

Also sweeps Updates that are being deleted while still holding finalizer.stack.pulumi.com, which can otherwise never finish deleting. The sweep uses a read-modify-write rather than the happy path's server-side apply, because that apply can only drop a finalizer owned by its own field manager — so a finalizer written by an older operator version survives silently, leaving an object removable only by hand-patching a controller-owned field. (This is what the unused isFinalizerOwnedByLegacyManager helper was anticipating.) It skips the update currently being waited on, and skips non-terminal ones, which the update controller marks Canceled first.

This part addresses the deletion deadlock described in #1293 rather than #1292, but it lives in the same function as the changes above; splitting it out would mean two PRs editing saveStatus/Reconcile concurrently. Its changelog entry cites #1293 accordingly.

Also

Requeue on a slow timer while an update is in flight, so an Update that never reaches a terminal state cannot strand the Stack. Kept coarse (5m) because each pass re-resolves the source, which for a git source is a network round trip per in-flight Stack.

Testing

New envtest specs in stack_controller_test.go:

• the Update is gone → reference cleared, event emitted, error returned; a fresh Update on the following reconcile
• the Update is absent from the cache but present in the API → reference retained, requeued, no duplicate Update (guards the regression this fix could introduce)
• the reference is lost while the update is still running → adopted, not duplicated
• the Update is terminal but its generation was bumped by a blocked deletion → result absorbed instead of waited on forever
• a previous Update stuck Terminating with a legacy-owned finalizer → released; and the update still being waited on → untouched
• lastUpdate.Failures unchanged across a heal

I confirmed the shape-B and reaper specs fail against the pre-fix code rather than passing vacuously.

make test (operator + agent) and golangci-lint are clean, with no codegen drift. make test-e2e has not been run, and neither failure mode has been reproduced end-to-end against a live cluster.

Related issues

Fixes #1292. The finalizer sweep also addresses the deletion deadlock in #1293.

🤖 Generated with Claude Code

[github-actions]

PR is now waiting for a maintainer to run the acceptance tests. This PR will only perform build and linting.
Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR