fix: validate provider prefixes in unverified model list

[enyst]

Problem

get_unverified_models() currently splits LiteLLM model identifiers on / or . and treats the first token as a provider.

LiteLLM’s registries include many non-provider prefixes (e.g. us.*, eu.*, low/…, 1024-x-1024/…). These prefixes are not LiteLLM providers, but they end up being surfaced as providers in UNVERIFIED_MODELS_EXCLUDING_BEDROCK, leaking noise into downstream apps (e.g. OpenHands-CLI provider dropdown).

Fix

• Fix Model list in SDK includes many non-provider "unverified" entries derived from litellm.model_list; need filtering/normalization #1665
• Validate extracted provider prefixes against litellm.provider_list.
• If the prefix is not a real LiteLLM provider, treat the entire identifier as a model id under the other bucket (do not emit it as a provider).

Tests

• Expanded tests/sdk/llm/test_model_list.py to assert invalid prefixes like custom-provider, us.*, and 1024-x-1024/* do not become providers.

This matches the direction discussed in OpenHands-CLI PR #322 (push provider validation upstream so consumers don’t need heuristics).

@enyst can click here to continue refining the PR

---

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	eclipse-temurin:17-jdk	Link
python	amd64, arm64	nikolaik/python-nodejs:python3.12-nodejs22	Link
golang	amd64, arm64	golang:1.21-bookworm	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:9432b02-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-9432b02-python \
  ghcr.io/openhands/agent-server:9432b02-python

All tags pushed for this build

ghcr.io/openhands/agent-server:9432b02-golang-amd64
ghcr.io/openhands/agent-server:9432b02-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:9432b02-golang-arm64
ghcr.io/openhands/agent-server:9432b02-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:9432b02-java-amd64
ghcr.io/openhands/agent-server:9432b02-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:9432b02-java-arm64
ghcr.io/openhands/agent-server:9432b02-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:9432b02-python-amd64
ghcr.io/openhands/agent-server:9432b02-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-amd64
ghcr.io/openhands/agent-server:9432b02-python-arm64
ghcr.io/openhands/agent-server:9432b02-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-arm64
ghcr.io/openhands/agent-server:9432b02-golang
ghcr.io/openhands/agent-server:9432b02-java
ghcr.io/openhands/agent-server:9432b02-python

About Multi-Architecture Support

• Each variant tag (e.g., 9432b02-python) is a multi-arch manifest supporting both amd64 and arm64
• Docker automatically pulls the correct architecture for your platform
• Individual architecture tags (e.g., 9432b02-python-amd64) are also available if needed

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
openhands-sdk/openhands/sdk/llm/utils
unverified_models.py	85	7	91%	45–46, 51, 73, 94–95, 106
TOTAL	14989	4413	70%

[enyst]

Hi! I’m OpenHands (automated agent) working on this PR.

I ran the repo-required checks and did live verification to ensure the provider list is correct and downstream apps (e.g. OpenHands-CLI) won’t need heuristics.

Local checks (required by SDK README)

• uv run make build ✅
• uv run pre-commit run --all-files ✅

Unit tests

• uv run pytest -q tests/sdk/llm/test_model_list.py ✅ (3 tests passed)

Live verification: provider list correctness

What was wrong before

get_unverified_models() used to split litellm.model_list/litellm.model_cost strings and treat the first token as a provider, which produced bogus providers like us, eu, low, 1024-x-1024, etc.

What the fix does

We now validate extracted provider prefixes against litellm.provider_list.
If the prefix is not a real LiteLLM provider, we bucket the full identifier under other (no bogus provider key).

Results (after fix)

• litellm.provider_list providers: 112
• get_unverified_models() provider keys: 58
• Invalid provider keys in get_unverified_models() (i.e. not in litellm.provider_list): 0
• Previously bogus keys (us, eu, apac, low, medium, high, 1024-x-1024, meta-llama, …): not present ✅

I also compared the SDK’s provider keys with the eval LiteLLM proxy’s provider catalog.

Eval proxy provider catalog

Using the env-provided LITELLM_API_KEY against the eval proxy base URL (found in openhands-sdk/openhands/sdk/agent/base.py and elsewhere):

• Base URL: https://llm-proxy.eval.all-hands.dev
• GET /v1/model/info returned 1050 entries ✅
• Providers observable from model_name prefixes: ['anthropic', 'bedrock', 'deepseek', 'fireworks_ai', 'gemini', 'hosted_vllm', 'mistral', 'moonshot', 'openai', 'openrouter', 'vertex_ai']

Why proxy providers != sdk providers:

• The eval proxy exposes its own curated routing surface (includes anthropic/*, bedrock/*, etc.).
• The SDK’s get_unverified_models() is based on LiteLLM’s local registry lists (litellm.model_list + litellm.model_cost) excluding bedrock by design, and those lists do not include anthropic/… and hosted_vllm/… as model keys (they appear as plain model ids like claude-… or as prefixed under other providers like openrouter/anthropic/...).
• Importantly, with this PR, everything we do emit as a provider is validated against litellm.provider_list, which is the key contract needed by downstream provider dropdowns.

Live example runs

Hello world (OpenAI)

Command:

LLM_MODEL=openai/gpt-4o-mini LLM_API_KEY=$OPENAI_API_KEY uv run python examples/01_standalone_sdk/01_hello_world.py

Result: ✅ Completed successfully and created FACTS.txt.

Responses reasoning example

This example defaults to an openhands/... model which rewrites to litellm_proxy/... and uses a default base URL of https://llm-proxy.app.all-hands.dev/.

The env-provided LITELLM_API_KEY is valid for the eval proxy, not the app proxy:

• https://llm-proxy.eval.all-hands.dev/v1/model/info → 200 ✅
• https://llm-proxy.app.all-hands.dev/v1/model/info → 401 (token not found) ❌

So I ran the reasoning example against the eval proxy explicitly:

LLM_API_KEY=$LITELLM_API_KEY LLM_BASE_URL=https://llm-proxy.eval.all-hands.dev   uv run python examples/01_standalone_sdk/23_responses_reasoning.py

Result: ✅ Completed successfully (conversation ran, created then deleted FACTS.txt).

Hello world (Claude)

Direct Anthropic run failed due to account credit balance too low (400 from Anthropic) in this environment:

LLM_MODEL=anthropic/claude-3-5-haiku-latest LLM_API_KEY=$ANTHROPIC_API_KEY ...

Result: ❌ Your credit balance is too low to access the Anthropic API

Workaround for verification in this environment: use the eval proxy (same as above) or use an OpenAI model as demonstrated.

---

If you’d like, I can also add an explicit unit test that asserts common bogus prefixes (e.g. us, 1024-x-1024) never appear as provider keys when using the real LiteLLM registry, but the current tests already cover the validation behavior via controlled inputs.

[all-hands-bot]

The PR correctly validates provider prefixes against LiteLLM's provider list, but has a critical logic issue where validation is silently skipped if the provider list fails to load (line 150). Also, the broad exception handling could mask important errors. Details in inline comments below.

[all-hands-bot]

Found several issues that need attention: missing error handling, inconsistent fallback behavior, and a test that does not work as intended. Details in inline comments.