Skip to content

fix(flowcontrol): make stale-metrics saturation observable and correct pool_saturation semantics - #2165

Open
LukeAVanDrie wants to merge 6 commits into
llm-d:mainfrom
LukeAVanDrie:fix/fc-saturation-staleness
Open

fix(flowcontrol): make stale-metrics saturation observable and correct pool_saturation semantics#2165
LukeAVanDrie wants to merge 6 commits into
llm-d:mainfrom
LukeAVanDrie:fix/fc-saturation-staleness

Conversation

@LukeAVanDrie

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

The utilization detector scores endpoints with missing or stale metrics as fully saturated, and both detectors score an empty pool as 1.0. At saturation 1.0 the dispatch cycle halts, so a fleet-wide failure of model-server metrics collection (scrape path, port, TLS, auth) halts all dispatch with nothing attributing the stall to staleness. Separately, the pool_saturation help text claimed a 0.0-1.0 range, but both detectors legitimately return values above 1, so operators cannot interpret a 1.7 reading.

Changes:

  • New gauge flow_control_stale_endpoints{detector}: candidate endpoints scored as saturated because their metrics are missing or older than the staleness threshold, as of the most recent evaluation. An empty candidate list records 0, so the gauge cannot hold a stale value and misattribute an empty-pool stall to a scrape failure. Emitted under the llm_d_epp prefix only (see [Flow Control] Decide a cardinality policy for caller-controlled flow control labels #2106).
  • The utilization detector logs the stale condition at default verbosity, at most once per 30s. Saturation runs every dispatch cycle (about 1ms), so the log is bounded by time.
  • Corrected pool_saturation help text on both the llm_d_epp gauge and the deprecated twin: 1.0 is the gating set point, values above 1.0 indicate the magnitude of oversubscription, and empty-pool and stale-metrics conditions read as 1.0 (fail-closed). The staleness sentence is scoped to the utilization detector; the concurrency detector has no staleness concept. Values are not clamped, since the magnitude above 1.0 pairs with the queue utilization gauges ([Flow Control] [Graduation Blocker] Per-band and global queue utilization ratio gauges #2102) as an autoscaling signal.
  • docs/metrics.md updated to match. The equivalent Grafana fix (the Pool Saturation panel caps its axis at 1.2, clipping readings above it) applies to the llm-d repo's dashboard copy, since fix: Remove grafna dashboard from router #2062 removed the dashboard from this repo; it will land with the llm-d observability doc updates.

Decision recorded here: fail-closed stays for stale metrics on a non-empty pool. Admitting blind on missing data risks overloading model servers with no backpressure signal at all, so the halt is kept and made observable. The utilization detector README documents the decision and the operational dependency of dispatch on scrape health.

Tests: the gauge counts nil-metrics and stale endpoints, returns to zero on recovery and on an empty candidate list, and the log throttle is asserted. Detector, metrics, and flowcontrol packages pass under -race.

Which issue(s) this PR fixes:

Fixes #2100

Part of #1187.

Release note:

Flow control now distinguishes saturation caused by missing or stale model-server metrics from genuine overload: a new `flow_control_stale_endpoints` gauge and a rate-limited EPP log attribute dispatch stalls to metrics collection problems. The `flow_control_pool_saturation` help text now documents the actual semantics: 1.0 is the gating set point, values above 1.0 indicate oversubscription magnitude, and empty-pool or stale-metrics conditions read as 1.0 (fail-closed).

@github-actions github-actions Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. labels Jul 24, 2026
…ation help text

The utilization detector scores endpoints with missing or stale metrics
as fully saturated, and both detectors score an empty pool as 1.0.
Dispatch halts at saturation 1.0, so a fleet-wide scrape failure stops
all dispatch and nothing points at metrics collection as the cause. The
pool_saturation help text also claimed a 0.0-1.0 range, but both
detectors return values above 1 when the pool is oversubscribed.

- Add flow_control_stale_endpoints{detector}: candidate endpoints
  scored as saturated because their metrics are missing or older than
  the staleness threshold. Emitted under the llm_d_epp prefix only.
- Log the stale condition from the utilization detector at default
  verbosity, at most once per 30s. Saturation runs every dispatch cycle
  (~1ms), so the log is bounded by time, not call count.
- Fix the pool_saturation help text on both gauges: 1.0 is the gating
  set point, values above 1.0 measure oversubscription, and empty-pool
  and stale-metrics conditions read as 1.0 (fail-closed). Values stay
  unclamped.
- Keep fail-closed for stale metrics on a non-empty pool and document
  the decision in the detector README: admitting blind on missing data
  would overload model servers with no backpressure signal at all.

Fixes llm-d#2100
Part of llm-d#1187

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
@LukeAVanDrie
LukeAVanDrie force-pushed the fix/fc-saturation-staleness branch from 499e3ad to c769ae9 Compare July 24, 2026 07:13
@LukeAVanDrie
LukeAVanDrie marked this pull request as ready for review July 24, 2026 07:17
@LukeAVanDrie
LukeAVanDrie requested review from a team and shmuelk as code owners July 24, 2026 07:17
@LukeAVanDrie
LukeAVanDrie requested review from ahg-g and elevran July 24, 2026 07:17
…ion README

Staleness tends to correlate with overload, so failing open would hide
exactly the wrong endpoints. Also note a staleness policy can be added
later without changing the default.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Flow Control] [Graduation Blocker] Stale-metrics saturation silently halts all dispatch; pool_saturation help text contradicts its semantics

1 participant