You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The switch-cost gate prices a switch as context_tokens x (candidate_uncached_rate - anchor_cached_rate), using a single context-token count for both sides. But tokenizers differ materially across model families (OpenAI's is notably more efficient than Anthropic's and most open models), so the same context can be 10-30% more tokens on one side of the switch than the other.
actual_context_tokens() counts with the tiktoken-based counter for the request model, falling back to chars/4.
On the full-proxy path the binding is refined with the provider's real reported prompt-token count, which fixes the anchor side over time — but the candidate side is still priced with the anchor-derived count.
Follow-up from #982 review (raised by @knn-do).
The switch-cost gate prices a switch as
context_tokens x (candidate_uncached_rate - anchor_cached_rate), using a single context-token count for both sides. But tokenizers differ materially across model families (OpenAI's is notably more efficient than Anthropic's and most open models), so the same context can be 10-30% more tokens on one side of the switch than the other.Current state after #982:
actual_context_tokens()counts with the tiktoken-based counter for the request model, falling back to chars/4.Possible directions (needs data before committing):