Skip to content

Improve switch-cost accuracy: cross-model tokenizer variance #991

Description

@adilhafeez

Follow-up from #982 review (raised by @knn-do).

The switch-cost gate prices a switch as context_tokens x (candidate_uncached_rate - anchor_cached_rate), using a single context-token count for both sides. But tokenizers differ materially across model families (OpenAI's is notably more efficient than Anthropic's and most open models), so the same context can be 10-30% more tokens on one side of the switch than the other.

Current state after #982:

  • actual_context_tokens() counts with the tiktoken-based counter for the request model, falling back to chars/4.
  • On the full-proxy path the binding is refined with the provider's real reported prompt-token count, which fixes the anchor side over time — but the candidate side is still priced with the anchor-derived count.
  • The A->B->A case is already handled (bounded route history + warm-token credit, feat(routing): automatic prompt caching + a per-session routing budget #982).

Possible directions (needs data before committing):

  • Count context per-side with each model's own tokenizer when known.
  • Apply a per-family correction factor derived from observed usage.
  • Measure how large the error actually is in practice from recorded usage before adding complexity.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions