Skip to content

feat(logging): record the OpenRouter provider that served each call (upstream_provider) - #151

Merged
cboettig merged 1 commit into
mainfrom
feat/log-upstream-provider
Oct 12, 2026
Merged

cboettig merged 1 commit into
mainfrom
feat/log-upstream-provider

Conversation

@cboettig

@cboettig cboettig commented Oct 9, 2026

Copy link
Copy Markdown
Member

Why

OpenRouter routes one model id across many hosts, and they differ a lot. GLM-5.2 has 34 endpoints with output prices from $1.68 to $10/M and quantizations from fp4 to fp8. MiMo-v2.6-pro has 4 endpoints, all priced the same. Our response log records only our route (provider: "openrouter"), so we can't answer questions like:

  • Why is our effective GLM-5.2 output price $3.68/M when cheaper hosts exist?
  • Is MiMo slow everywhere, or on one host?
  • Which hosts return 429s?

The OpenRouter response body already names the host it picked, in its top-level provider field. This PR logs it.

Providers change from day to day, so the goal is to collect data over time, not to label any one host as "fast" or "slow".

Change

  • log_response writes upstream_provider when the upstream body has a provider field. That means successful OpenRouter calls only; the field is absent for NRP/nimbus and on errors. Failed OpenRouter calls already name their providers inside error.
  • New test test_upstream_provider_logged_when_present. All 42 tests in test_logging.py pass.
  • Added to the LOGGING.md response-field table and a CHANGELOG entry under Unreleased.
  • Adds DISCLOSURE.md, which the repo lacked, disclosing LLM tool use in the repo's development.

It isn't a flat Parquet column yet; query it with json_extract_string(entry,'$.upstream_provider'). If it proves useful, it can be promoted to a column in the consolidation jobs later, alongside the team traffic dashboard work.

OpenRouter routes one model id across many hosts and names the one it
picked in the response body's `provider`; the log kept only our route
("openrouter"), so cost/latency couldn't be broken down by host.
Logged as `upstream_provider` on successful OpenRouter responses.

Also adds DISCLOSURE.md (tool-use disclosure), which the repo lacked.
@cboettig
cboettig merged commit b478737 into main Oct 12, 2026
1 check passed
@cboettig
cboettig deleted the feat/log-upstream-provider branch October 12, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant