fix(flowcontrol): make stale-metrics saturation observable and correct pool_saturation semantics - #2165
Open
LukeAVanDrie wants to merge 6 commits into
Open
fix(flowcontrol): make stale-metrics saturation observable and correct pool_saturation semantics#2165LukeAVanDrie wants to merge 6 commits into
LukeAVanDrie wants to merge 6 commits into
Conversation
…ation help text
The utilization detector scores endpoints with missing or stale metrics
as fully saturated, and both detectors score an empty pool as 1.0.
Dispatch halts at saturation 1.0, so a fleet-wide scrape failure stops
all dispatch and nothing points at metrics collection as the cause. The
pool_saturation help text also claimed a 0.0-1.0 range, but both
detectors return values above 1 when the pool is oversubscribed.
- Add flow_control_stale_endpoints{detector}: candidate endpoints
scored as saturated because their metrics are missing or older than
the staleness threshold. Emitted under the llm_d_epp prefix only.
- Log the stale condition from the utilization detector at default
verbosity, at most once per 30s. Saturation runs every dispatch cycle
(~1ms), so the log is bounded by time, not call count.
- Fix the pool_saturation help text on both gauges: 1.0 is the gating
set point, values above 1.0 measure oversubscription, and empty-pool
and stale-metrics conditions read as 1.0 (fail-closed). Values stay
unclamped.
- Keep fail-closed for stale metrics on a non-empty pool and document
the decision in the detector README: admitting blind on missing data
would overload model servers with no backpressure signal at all.
Fixes llm-d#2100
Part of llm-d#1187
Signed-off-by: Luke Van Drie <lukevandrie@google.com>
LukeAVanDrie
force-pushed
the
fix/fc-saturation-staleness
branch
from
July 24, 2026 07:13
499e3ad to
c769ae9
Compare
…ion README Staleness tends to correlate with overload, so failing open would hide exactly the wrong endpoints. Also note a staleness policy can be added later without changing the default. Signed-off-by: Luke Van Drie <lukevandrie@google.com>
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
The utilization detector scores endpoints with missing or stale metrics as fully saturated, and both detectors score an empty pool as 1.0. At saturation 1.0 the dispatch cycle halts, so a fleet-wide failure of model-server metrics collection (scrape path, port, TLS, auth) halts all dispatch with nothing attributing the stall to staleness. Separately, the
pool_saturationhelp text claimed a 0.0-1.0 range, but both detectors legitimately return values above 1, so operators cannot interpret a 1.7 reading.Changes:
flow_control_stale_endpoints{detector}: candidate endpoints scored as saturated because their metrics are missing or older than the staleness threshold, as of the most recent evaluation. An empty candidate list records 0, so the gauge cannot hold a stale value and misattribute an empty-pool stall to a scrape failure. Emitted under thellm_d_eppprefix only (see [Flow Control] Decide a cardinality policy for caller-controlled flow control labels #2106).Saturationruns every dispatch cycle (about 1ms), so the log is bounded by time.pool_saturationhelp text on both thellm_d_eppgauge and the deprecated twin: 1.0 is the gating set point, values above 1.0 indicate the magnitude of oversubscription, and empty-pool and stale-metrics conditions read as 1.0 (fail-closed). The staleness sentence is scoped to the utilization detector; the concurrency detector has no staleness concept. Values are not clamped, since the magnitude above 1.0 pairs with the queue utilization gauges ([Flow Control] [Graduation Blocker] Per-band and global queue utilization ratio gauges #2102) as an autoscaling signal.docs/metrics.mdupdated to match. The equivalent Grafana fix (the Pool Saturation panel caps its axis at 1.2, clipping readings above it) applies to the llm-d repo's dashboard copy, since fix: Remove grafna dashboard from router #2062 removed the dashboard from this repo; it will land with the llm-d observability doc updates.Decision recorded here: fail-closed stays for stale metrics on a non-empty pool. Admitting blind on missing data risks overloading model servers with no backpressure signal at all, so the halt is kept and made observable. The utilization detector README documents the decision and the operational dependency of dispatch on scrape health.
Tests: the gauge counts nil-metrics and stale endpoints, returns to zero on recovery and on an empty candidate list, and the log throttle is asserted. Detector, metrics, and flowcontrol packages pass under
-race.Which issue(s) this PR fixes:
Fixes #2100
Part of #1187.
Release note: