[Flow Control] Queue-wait TTL should split by unavailability regime, like the status code in #2046

[LukeAVanDrie]

What would you like to be added:

Split the flow control queue-wait budget by unavailability regime, the same way #2046 / #2047 split the TTL-eviction status code.

That issue established the discriminator and the table:

	Capacity rejection	TTL eviction
Empty pool	503	503
Non-empty saturated	429	429

The budget should follow the same seam:

	Queue-wait budget	Expiry means
Empty pool	long, sized to cold start	503, genuine outage
Non-empty saturated	short, sized to a TTFB SLO	429, backpressure

The signal already exists: pkg/epp/flowcontrol/controller/internal/processor.go carries poolEmpty and uses it to pick QueueOutcomeRejectedNoEndpoints over QueueOutcomeRejectedCapacity at the capacity boundary (#1751).

This was option [3] in the design discussion on #2123:

3. Pause or extend the TTL while the pool is empty, so it measures "the pool is up and you still
   couldn't dispatch."

[3] is probably the right end state, but it's new mechanism.

#2123 shipped option [1] (60s plus documentation) to unblock the default-on flip (#2104 / #2180). This issue is the follow-through.

Open questions, roughly in order of how much they shape the design:

1. What defines the empty-pool regime. poolEmpty is the obvious predicate, but the utilization detector scores stale-metric endpoints as 1.0 saturated (pkg/epp/saturationdetector/utilization/detector.go:124-125). A total scrape failure therefore presents as the saturated regime (short budget, 429) when the honest reading is unavailability. Decide whether the predicate is "no endpoints in the pool" or "no endpoint with fresh metrics".

2. Regime is not fixed at enqueue. The pool going empty to non-empty is the entire point of scale-from-zero, so the effective deadline has to be re-evaluated while queued, not frozen at admission. Likely shape: the item carries two budgets and the expiry scan charges elapsed time to whichever regime is current.

3. Transition accounting. A pod appears at t=90s, past the short saturation budget but inside the long cold-start budget. Charging against an already-exhausted saturation budget would shed the request the instant it becomes servable, which is pathological. Probably grant a fresh saturation budget on the transition, since the request only just became dispatchable.

4. Config surface. Keep defaultRequestTTL as the saturation budget and add a second field for the no-endpoint budget, or generalize per band. Explicit 0s must keep meaning "disabled". Related: the per-priority and per-request TTL scopes discussed on fix(flowcontrol): apply effective defaults for DefaultRequestTTL and per-band MaxRequests #2123 are still unbuilt.

5. Pairs with a gateway deadline. A long cold-start budget only helps if the caller waits. The well-lit guides configure no gateway request timeout, which is why the TTL currently acts as the primary deadline rather than a backstop. Guides shipping a timeout is arguably the more important fix; the two should be documented together.

6. Metrics. A distinct outcome for no-endpoint expiry keeps the regimes separable on dashboards. The queue-duration metric already labels by outcome.

Why is this needed:

defaultRequestTTL is a single budget (60s since #2123). One number is serving two regimes that want opposite values.

No viable endpoint (scale-from-zero, total outage). Waiting is the only path to success, so the budget wants to be long: a cold start is image pull plus weight load, minutes not seconds. 60s is too short, which is why #2123's own release note already tells scale-to-zero users to raise it.

Endpoints exist but are saturated. Waiting past the time-to-first-token budget yields a degraded response anyway; the useful answer is to shed so the caller can retry or load-shed elsewhere. Here 60s is far too long. Nothing interactive has a 60s TTFB budget, and the legacy admission path rejected sheddable traffic immediately.

A single default cannot serve both, so it protects neither. It also costs observability: any client or gateway deadline under 60s wins the race, and inferOutcome (pkg/epp/flowcontrol/controller/internal/item.go:178-184) then records EvictedContextCancelled rather than EvictedTTL. The shed-under-saturation signal disappears and the symptom reads as "clients went away".

The default already bites in practice. Flipping the gate on by default (#2180) broke two e2e disruption specs: with the gate on, an empty pool is a scale-from-zero waiting room, and QueueOutcomeRejectedNoEndpoints only fires at queue capacity (processor.go:294-306), which the 5000/band default never reaches for a probe request. So the request holds for the full 60s TTL while the test's 10s client timeout and 30s probe window both expire first. #2180 works around it with a per-test defaultRequestTTL: 5s. Under the split that override becomes an honest statement ("this deployment does not do scale-from-zero") rather than a fudged timeout.

One sizing note worth keeping regardless of the design chosen: under a dispatch halt, queue occupancy is arrival rate times TTL. #2123's own math puts the 5000/band cap binding at roughly 83 req/s with a 60s TTL, versus roughly 330-500 req/s at 10-15s. A short saturation budget makes the capacity limit bind before the time limit, which is the ordering you want, because a capacity rejection is immediate and attributable while a TTL eviction is neither.

Related: #2046, #2047, #2097, #2123, #2104, #1751, #1187.

[satyamg1620]

/assign