Follow-up from #982, motivated by decision-endpoint consumers with self-hosted multi-replica backends.
On the full-proxy path, brightstaff stamps x-plano-prefix-hash on upstream requests and prefix_affinity: true clusters consistent-hash on it (STRICT_DNS + RING_HASH), so the same prompt prefix lands on the vLLM replica holding its warm KV cache.
Consumers of the /routing/v1/* decision endpoints get none of that: brightstaff already computes the prefix hash on this path (resolve_session uses it for drift detection), but RoutingDecisionResponse does not return it. Callers who dispatch inference themselves have nothing to key replica stickiness on, so vLLM prefix-cache hit rates decay roughly as 1/N with replica count -- undermining the warmth premise the routing budget prices against.
Proposed change (small):
- Add
prefix_hash (16-hex string, matching the header format) to RoutingDecisionResponse, skip_serializing_if = "Option::is_none".
- Document that callers can feed it to a consistent-hash LB policy (Envoy ring-hash or equivalent) in front of their replicas, mirroring Plano's own
prefix_affinity clusters.
Backwards compatible: additive field, omitted when the request opted out (X-Plano-Cache: off) or has no anchorable prompt.
Follow-up from #982, motivated by decision-endpoint consumers with self-hosted multi-replica backends.
On the full-proxy path, brightstaff stamps
x-plano-prefix-hashon upstream requests andprefix_affinity: trueclusters consistent-hash on it (STRICT_DNS + RING_HASH), so the same prompt prefix lands on the vLLM replica holding its warm KV cache.Consumers of the
/routing/v1/*decision endpoints get none of that: brightstaff already computes the prefix hash on this path (resolve_sessionuses it for drift detection), butRoutingDecisionResponsedoes not return it. Callers who dispatch inference themselves have nothing to key replica stickiness on, so vLLM prefix-cache hit rates decay roughly as 1/N with replica count -- undermining the warmth premise the routing budget prices against.Proposed change (small):
prefix_hash(16-hex string, matching the header format) toRoutingDecisionResponse,skip_serializing_if = "Option::is_none".prefix_affinityclusters.Backwards compatible: additive field, omitted when the request opted out (
X-Plano-Cache: off) or has no anchorable prompt.