Repository navigation
feat(eval): add optional API Route reader - #167
DennyHo0917 wants to merge 2 commits into
Conversation
Signed-off-by: DennyHo0917 <149746199+DennyHo0917@users.noreply.github.com>
|
@DennyHo0917 is attempting to deploy a commit to the andylizf's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
@DennyHo0917 — this has sat 3.5 weeks with no human review. Reviewed now; the code is fine and the hold is a policy question that isn't about your patch. On the code. It follows the existing provider pattern closely — retry_requests=not (args.atlascloud or args.api_route),This opts API Route out of request retries. On the policy question, which is the actual blocker. So there's a rule the project hasn't written down, and four contributors are waiting on it. That's not your problem to solve, but it's the honest reason this hasn't merged. @andylizf @yichuan-w — worth deciding once rather than four times. The question is whether @DennyHo0917 the 🤖 Generated with Claude Code |
|
@ASuresh0524 thanks for catching this. The earlier duplicate-generation rationale was a precaution, not a verified API Route-specific retry or billing contract. It was not sufficient grounds for inheriting Atlas Cloud's opt-out. Removed API Route from that exception in 9ff3427: The actual retry owner here is Validation on the updated tree:
The broader provider-admission policy remains separate from this retry fix. CI update (05:57 UTC): run #461 for 9ff3427 has now completed successfully: Python lint, Tests, Frontend build and Plugin validate are all green. The CI test log reports 254 passed, 2 skipped. The separate Vercel status still requires deployment-team authorization. |
|
Thanks, Aakash. I agree this needs a written rule rather than another case-by-case call. For
The new commit puts API Route on the shared retry policy, so the code question is resolved. API Route still fits the generic path, and the PR does not establish a capability LiteLLM cannot express. #190 should follow the same rule. Denny, thanks for the update and tests. I’ll put this rule in |
|
Closing this against the hosted reader integration policy, which landed in The policy says:
API Route is reachable today with no code at all: export API_ROUTE_API_KEY=your-api-key
uv run eval/run_bench.py --task simpleqa \
--model openai/<provider-model-id> \
--api-base https://global.api-route.com/v1 \
--api-key "$API_ROUTE_API_KEY" \
--litellm --num-examples 1 --max-concurrent 1 --max-tokens 256The one thing this PR calls out that So there is no capability gap left for a dedicated flag to fill, and adding one costs a boolean, a config function, a dispatch branch and a README row, all maintained against a URL we don't control. This is not a judgement on the code. It is careful and it is tested, and it follows the shape of #159 and #163 exactly — which is the problem: that precedent is what #192 was written to stop. You were building to the standard the repo was setting at the time. If the generic command above does not actually work for API Route, say so and I'll reopen — a demonstrated gap is precisely what the policy reserves a flag for. And if a documented row for API Route would help others, a docs-only PR adding one to the hosted-reader section is welcome. |
Summary
Add an optional API Route reader to the evaluation CLI, following the existing Cheaper Inference (#163) and Atlas Cloud (#159) reader integrations.
--api-route; the base URL ishttps://global.api-route.com/v1.--api-keyorAPI_ROUTE_API_KEY, preserving exact account model IDs. Other providers' keys are never used.gemini.Validation
Latest retry follow-up: 9ff3427.
uv run --frozen --extra dev pytest tests/test_api_route_client.py tests/test_atlascloud_client.py tests/test_cheaperinference_client.py tests/test_eval_model_config.py tests/test_litellm_client.py -q: 62 passed.uvx ruff check .anduvx ruff format --check .: passed across the repository (106 files already formatted);git diff --check: passed.run_bench.py --helpsucceeds and exposes--api-route. Actual CLI rejects all six competing provider flags and--api-basewith exit code 2 before loading a dataset or generating an answer.socket() failed: Operation not permitted), so this is not a clean full-suite result.Disclosure
I operate API Route. This contribution was prepared with AI assistance and checked locally. It adds no model context/pricing/capability metadata or default model recommendation.