Skip to content

fix(sglang): support msgspec-based ServerArgs (sglang >= 0.5.20) - #12155

Open
pos-ei-don wants to merge 1 commit into
mudler:masterfrom
pos-ei-don:fix/sglang-msgspec-serverargs
Open

pos-ei-don wants to merge 1 commit into
mudler:masterfrom
pos-ei-don:fix/sglang-msgspec-serverargs

Conversation

@pos-ei-don

Copy link
Copy Markdown
Contributor

Problem

backend/python/sglang/backend.py::_apply_engine_args validates engine_args keys against the fields of ServerArgs:

valid = {f.name for f in dataclasses.fields(ServerArgs)}

sglang 0.5.20 moved its config tier from dataclasses to msgspec.Struct (sgl-project/sglang#38753, follow-up #38958). dataclasses.fields() raises there:

TypeError: must be called with a dataclass type or instance

_apply_engine_args runs on every LoadModel, so no model loads at all on the sglang backend once sglang >= 0.5.20 is installed — including models that set no engine_args beyond the defaults. It surfaces as a generic Failed to load model <name>: Unexpected <class 'TypeError'> that does not name the cause.

The requirements carry no upper bound (sglang>=0.5.11, sglang[all]>=0.5.11), so a fresh backend build picks up 0.5.20 and runs into this.

There is some irony in the docstring right above the line: validation exists so that a typo "fails early with a close-match suggestion instead of producing a confusing TypeError deep inside engine startup". This is now that confusing TypeError.

Fix

Introspect both shapes. msgspec structs carry their field names in __struct_fields__, so validation and the close-match suggestion keep working unchanged, and the code stays compatible with older dataclass-based sglang — no version bound needed.

Also adds a test that pins the msgspec path with a stand-in, so it stays covered regardless of which sglang version happens to be installed in CI.

Verification

NVIDIA DGX Spark (GB10, aarch64), l4t13 profile, CUDA 13, LocalAI v4.10.0 core, backend built against sglang 0.5.20.

Measured inside the built backend venv:

sglang 0.5.20
dataclasses.is_dataclass(ServerArgs)                   -> False
len(ServerArgs.__struct_fields__)                      -> 499
difflib.get_close_matches("mem_fraction_statik", ...)  -> ['mem_fraction_static']

python -m unittest test in backend/python/sglang/, run against that venv:

result
before this change 5 errors — 4x TypeError: must be called with a dataclass type or instance (every engine_args test) + 1 unrelated, see below
after this change 1 error — only the unrelated one

End to end: before the change every LoadModel fails with the TypeError above; after it, an 80B NVFP4 model loads and serves, and the regression suite we run against this box is green (chat template, structured output, tool calls via API, streaming parity, agent end-to-end, prefix-cache TTFT).

The validation itself was added in #9686, which predates the sglang change.

Unrelated failure noticed while testing

test_build_prompt_forwards_enable_thinking fails on current master independently of this PR — the SimpleNamespace request stub has no Images / Videos attributes, which _build_prompt reads since the media-parts change:

AttributeError: 'types.SimpleNamespace' object has no attribute 'Images'

Left untouched here to keep this PR to one concern. Happy to send a separate one-liner for the stub if useful.

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

@mudler code review of a39cb43 looks good: the dataclass path remains intact, the struct path preserves unknown-key validation and typo suggestions, and the regression test restores the patched ServerArgs in a finally block. I checked SGLang v0.5.20's source, which constructs ServerArgs with msgspec.defstruct, and git diff --check passes against latest master (96ccb8e).

This is not merge-ready yet: DCO reports a missing Signed-off-by on a39cb43, which needs the contributor's certification. No backend CI result is currently reported. I could not independently run the Python tests in this environment because no runnable Python interpreter is available; the runtime results in the PR description are contributor-reported. No code changes requested from this review.

@pos-ei-don
pos-ei-don force-pushed the fix/sglang-msgspec-serverargs branch from a39cb43 to 9831ccb Compare September 19, 2026 22:12
sglang 0.5.20 moved its config tier from dataclasses to msgspec.Struct
(sgl-project/sglang#38753). _apply_engine_args validates engine_args keys
via dataclasses.fields(ServerArgs), which raises TypeError there. That
call runs on every LoadModel, so no model loads at all on the sglang
backend once sglang >= 0.5.20 is installed, and the error surfaces as a
generic "Unexpected <class 'TypeError'>" that does not name the cause.

Introspect both shapes: msgspec structs carry their field names in
__struct_fields__, so key validation and the close-match suggestion keep
working, and older dataclass-based sglang stays supported.

Adds a test that pins the msgspec path with a stand-in, so it is covered
regardless of which sglang version is installed.

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
@pos-ei-don
pos-ei-don force-pushed the fix/sglang-msgspec-serverargs branch from 9831ccb to 933050c Compare September 19, 2026 22:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants