Skip to content

feat(eval): add optional API Route reader - #167

Closed
DennyHo0917 wants to merge 2 commits into
StarTrail-org:mainfrom
DennyHo0917:feat/api-route-reader
Closed

DennyHo0917 wants to merge 2 commits into
StarTrail-org:mainfrom
DennyHo0917:feat/api-route-reader

Conversation

@DennyHo0917

@DennyHo0917 DennyHo0917 commented Oct 1, 2026 •

Copy link
Copy Markdown

Summary

Add an optional API Route reader to the evaluation CLI, following the existing Cheaper Inference (#163) and Atlas Cloud (#159) reader integrations.

  • Select explicitly with --api-route; the base URL is https://global.api-route.com/v1.
  • Resolve only --api-key or API_ROUTE_API_KEY, preserving exact account model IDs. Other providers' keys are never used.
  • Force the existing OpenAI-compatible client even for model IDs containing gemini.
  • Reject other provider flags, LiteLLM and endpoint overrides when this reader is selected.
  • Use the shared application-level retry policy for timeouts, connection errors and rate limits. SDK-level retries remain disabled; Atlas Cloud's existing opt-out is unchanged.
  • Add setup documentation and offline regression tests. Existing reader defaults and dependencies remain unchanged.

Validation

Latest retry follow-up: 9ff3427.

  • uv run --frozen --extra dev pytest tests/test_api_route_client.py tests/test_atlascloud_client.py tests/test_cheaperinference_client.py tests/test_eval_model_config.py tests/test_litellm_client.py -q: 62 passed.
  • uvx ruff check . and uvx ruff format --check .: passed across the repository (106 files already formatted); git diff --check: passed.
  • Actual run_bench.py --help succeeds and exposes --api-route. Actual CLI rejects all six competing provider flags and --api-base with exit code 2 before loading a dataset or generating an answer.
  • Supplemental offline check using the locked OpenAI SDK 2.38.0 with HTTPX MockTransport: 9 scenarios passed, covering transient recovery, exhausted retry budgets, non-retryable responses and the existing retry opt-out. No external inference requests were sent.
  • Full local suite: 149 passed, 7 skipped, 8 renderer tests failed. Chromium cannot start its local IPC socket in this cloud sandbox (socket() failed: Operation not permitted), so this is not a clean full-suite result.
  • No live inference, provider billing/idempotency, vision-model compatibility or end-to-end retrieval benchmark was tested. The docs require users to verify image support for screenshot readers. No dependency or lockfile change was needed.

Disclosure

I operate API Route. This contribution was prepared with AI assistance and checked locally. It adds no model context/pricing/capability metadata or default model recommendation.

Signed-off-by: DennyHo0917 <149746199+DennyHo0917@users.noreply.github.com>
@vercel

vercel Bot commented Oct 1, 2026

Copy link
Copy Markdown

@DennyHo0917 is attempting to deploy a commit to the andylizf's projects Team on Vercel.

A member of the Team first needs to authorize it.

@ASuresh0524

Copy link
Copy Markdown
Collaborator

@DennyHo0917 — this has sat 3.5 weeks with no human review. Reviewed now; the code is fine and the hold is a policy question that isn't about your patch.

On the code. It follows the existing provider pattern closely — get_api_route_config mirrors get_cheaperinference_config, the mutual-exclusion check covers all six existing provider flags, and --api-base is correctly rejected against a fixed endpoint. It merges cleanly onto today's main and your 4 checks are green. One substantive question:

retry_requests=not (args.atlascloud or args.api_route),

This opts API Route out of request retries. --atlascloud presumably had a reason (double-billing on retry? non-idempotent endpoint?), but there's no comment saying what it is, and you've now inherited it for a second provider. Could you say why retries are wrong for API Route specifically? If it's just copied from the line above, it should probably come out — silently disabling retries makes benchmark runs flakier in a way that's hard to debug later. Either way a one-line comment on that expression would save the next person.

On the policy question, which is the actual blocker. main already carries six third-party provider flags (--open-router, --commonstack, --orcarouter, --atlascloud, --cheaperinference, --litellm). This would be the seventh, and #190 is open right now adding an eighth (Runcrate) through the same four files in the same shape. Separately #53 has been held four months on exactly this question for the demos/ equivalent, with "we'll discuss internally" and no answer, while #175 merged last week in that same shape.

So there's a rule the project hasn't written down, and four contributors are waiting on it. That's not your problem to solve, but it's the honest reason this hasn't merged.

@andylizf @yichuan-w — worth deciding once rather than four times. The question is whether eval/ accepts an open-ended list of hosted inference providers, and if so whether each needs a maintainer-verifiable reason to exist alongside --litellm, which can already reach most of them. My read: --litellm subsumes most of these, so the bar for a dedicated flag should be a capability LiteLLM can't express — and if that's the rule, it should be in eval/README.md so contributors stop writing these blind.

@DennyHo0917 the retry_requests answer is the only thing I need from you; the rest is ours to unblock.

🤖 Generated with Claude Code

@DennyHo0917

DennyHo0917 commented Oct 27, 2026 •

Copy link
Copy Markdown
Author

@ASuresh0524 thanks for catching this. The earlier duplicate-generation rationale was a precaution, not a verified API Route-specific retry or billing contract. It was not sufficient grounds for inheriting Atlas Cloud's opt-out.

Removed API Route from that exception in 9ff3427: retry_requests=not args.atlascloud. Added the one-line comment and updated the README and tests. Atlas Cloud's behavior is unchanged.

The actual retry owner here is LLMClient.generate, not the SDK: the shared OpenAI-compatible client already sets max_retries=0. With the existing defaults, repeated failures in one category allow 3 total attempts for timeouts, 12 for connection errors, and 6 for rate limits, with the existing backoff. Timeout/rate-limit errors share a counter, so those are not a universal cap for mixed failures. Generic 400/500 responses still propagate without retry; that shared behavior is unchanged. This does not establish server-side idempotency or guarantee against duplicate billing.

Validation on the updated tree:

  • 62 focused tests passed, including API Route recovery/exhaustion, CLI retry wiring, Atlas Cloud, Cheaper Inference, model config and LiteLLM.
  • Full-repository Ruff lint/format and CLI help/provider-conflict checks passed.
  • 9 additional offline scenarios passed with the real locked OpenAI SDK 2.38.0 and HTTPX MockTransport, confirming request counts without calling a live provider.
  • Full local suite: 149 passed, 7 skipped, 8 renderer failures; Chromium's local IPC socket is blocked in this cloud sandbox. I am not claiming a full local test pass or live inference/billing validation.

The broader provider-admission policy remains separate from this retry fix.

CI update (05:57 UTC): run #461 for 9ff3427 has now completed successfully: Python lint, Tests, Frontend build and Plugin validate are all green. The CI test log reports 254 passed, 2 skipped. The separate Vercel status still requires deployment-team authorization.

@andylizf

Copy link
Copy Markdown
Contributor

Thanks, Aakash. I agree this needs a written rule rather than another case-by-case call.

For eval/, I’m setting the rule this way:

  • Any OpenAI-compatible hosted reader is configured through the existing --litellm path: --model openai/<catalog-model-id>, with --api-base and --api-key.
  • A provider-specific flag is accepted only when it exposes a capability LiteLLM cannot express and that gap affects evaluation, such as provider-specific authentication or protocol behavior. The PR must document and test the gap.
  • Existing provider flags remain for compatibility, but do not set a precedent for adding more.

The new commit puts API Route on the shared retry policy, so the code question is resolved. API Route still fits the generic path, and the PR does not establish a capability LiteLLM cannot express. #190 should follow the same rule.

Denny, thanks for the update and tests. I’ll put this rule in eval/README.md so contributors can apply it before opening another provider PR.

@ASuresh0524

Copy link
Copy Markdown
Collaborator

Closing this against the hosted reader integration policy, which landed in eval/README.md in #192 — after you opened this, so this is not a case of ignoring a rule that existed.

The policy says:

A dedicated provider flag is reserved for a capability the generic path cannot express, such as provider-specific authentication or protocol behavior that affects evaluation. A PR adding one must document and test that missing capability. [...] an OpenAI-compatible endpoint alone is not grounds for adding another flag.

API Route is reachable today with no code at all:

export API_ROUTE_API_KEY=your-api-key
uv run eval/run_bench.py --task simpleqa \
    --model openai/<provider-model-id> \
    --api-base https://global.api-route.com/v1 \
    --api-key "$API_ROUTE_API_KEY" \
    --litellm --num-examples 1 --max-concurrent 1 --max-tokens 256

The one thing this PR calls out that --api-base alone cannot do — stopping a model id containing gemini from being routed to the native Google SDK — is exactly what the openai/ prefix handles. LiteLLM's OpenAI-compatible transport is selected by that prefix, not inferred from the model name, so openai/google/gemini-3.1-pro goes over HTTP to the base URL you gave it.

So there is no capability gap left for a dedicated flag to fill, and adding one costs a boolean, a config function, a dispatch branch and a README row, all maintained against a URL we don't control.

This is not a judgement on the code. It is careful and it is tested, and it follows the shape of #159 and #163 exactly — which is the problem: that precedent is what #192 was written to stop. You were building to the standard the repo was setting at the time.

If the generic command above does not actually work for API Route, say so and I'll reopen — a demonstrated gap is precisely what the policy reserves a flag for. And if a documented row for API Route would help others, a docs-only PR adding one to the hosted-reader section is welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants