Make the failed-update retry backoff configurable (24h cap is too slow for fast-moving resources)

[bpalermo]

Version: v2.9.0

Request

Expose the failed-update retry backoff as configuration — at minimum its cap, ideally the factor too.

Today it is hardcoded in stack_controller.go:

const maxUpdateFailures = 3

func cooldown(stack *pulumiv1.Stack) time.Duration {
    ...
    backoff := wait.Backoff{
        Duration: 10 * time.Second,
        Factor:   3,
        Cap:      24 * time.Hour,
        Steps:    math.MaxInt,
        Jitter:   0,
    }

which gives 10s × 3^failures, capped at 24 hours:

failures	next retry
1	30s
3	4.5m
5	40m
8	18.2h
9+	24h

Why it matters

For slow-moving infrastructure a 24-hour cap is sensible. For fast-moving resources it is far too slow: we use Stacks to reconcile DNS records, where a day of staleness is an outage rather than a delay.

Concretely, a fleet of ~280 Stacks hit a transient problem that left most of them failing. By the time the underlying cause was fixed, many had reached 8+ failures and so were 18–24 hours from their next attempt. The fix was live but unobservable — the fleet looked permanently frozen rather than recovering, and the only way to verify anything was to force reconciliation by hand.

Why the existing knobs don't cover it

The two polling paths that could otherwise retry sooner are both unavailable to a failed Stack:

• ContinueResyncOnCommitMatch schedules a resync only when the last update succeeded:

  if instance.Status.LastUpdate.State == shared.SucceededStackStateMessage && sess.stack.ContinueResyncOnCommitMatch {

• The source-tracking poll requires spec.branch. For a Stack pinned to spec.commit — mutually exclusive with branch — trackBranch is false and no poll is scheduled.

So for a commit-pinned Stack, cooldown() is the only retry path, and resyncFrequencySeconds has no effect on it. That asymmetry is surprising: an operator who sets resyncFrequencySeconds: 1800 reasonably expects retries at least that often, and gets 24h instead.

Workaround, and its cost

Bumping pulumi.com/reconciliation-request forces a reconcile and bypasses the backoff. But driving that on a schedule means one Update object per Stack per cycle, which interacts badly with ttlAfterCompleted: at a 15-minute cadence over ~280 Stacks with a 24h TTL, that is ~27,000 live Update objects. The knob that makes recovery observable therefore fights the knob that makes it survivable.

Suggested shape

Any of these would resolve it:

• spec.retryBackoffCap (duration), defaulting to today's 24h.
• Honour resyncFrequencySeconds as an upper bound on the retry interval, so min(cooldown, resyncFreq) governs. This needs no new field and matches the natural reading of the existing one.
• Operator-level flag or env var, as with MAX_CONCURRENT_RECONCILES, for deployments where the whole fleet has the same characteristics.

A shorter cap does risk hammering a persistently broken Stack, which is presumably why 24h was chosen — so a configurable default rather than a changed default seems right.