fix(sglang): support msgspec-based ServerArgs (sglang >= 0.5.20) - #12155
pos-ei-don wants to merge 1 commit into
Conversation
|
@mudler code review of a39cb43 looks good: the dataclass path remains intact, the struct path preserves unknown-key validation and typo suggestions, and the regression test restores the patched ServerArgs in a finally block. I checked SGLang v0.5.20's source, which constructs ServerArgs with msgspec.defstruct, and This is not merge-ready yet: DCO reports a missing Signed-off-by on a39cb43, which needs the contributor's certification. No backend CI result is currently reported. I could not independently run the Python tests in this environment because no runnable Python interpreter is available; the runtime results in the PR description are contributor-reported. No code changes requested from this review. |
a39cb43 to
9831ccb
Compare
sglang 0.5.20 moved its config tier from dataclasses to msgspec.Struct (sgl-project/sglang#38753). _apply_engine_args validates engine_args keys via dataclasses.fields(ServerArgs), which raises TypeError there. That call runs on every LoadModel, so no model loads at all on the sglang backend once sglang >= 0.5.20 is installed, and the error surfaces as a generic "Unexpected <class 'TypeError'>" that does not name the cause. Introspect both shapes: msgspec structs carry their field names in __struct_fields__, so key validation and the close-match suggestion keep working, and older dataclass-based sglang stays supported. Adds a test that pins the msgspec path with a stand-in, so it is covered regardless of which sglang version is installed. Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
9831ccb to
933050c
Compare
Problem
backend/python/sglang/backend.py::_apply_engine_argsvalidatesengine_argskeys against the fields ofServerArgs:sglang 0.5.20 moved its config tier from dataclasses to
msgspec.Struct(sgl-project/sglang#38753, follow-up #38958).dataclasses.fields()raises there:_apply_engine_argsruns on everyLoadModel, so no model loads at all on the sglang backend once sglang >= 0.5.20 is installed — including models that set noengine_argsbeyond the defaults. It surfaces as a genericFailed to load model <name>: Unexpected <class 'TypeError'>that does not name the cause.The requirements carry no upper bound (
sglang>=0.5.11,sglang[all]>=0.5.11), so a fresh backend build picks up 0.5.20 and runs into this.There is some irony in the docstring right above the line: validation exists so that a typo "fails early with a close-match suggestion instead of producing a confusing
TypeErrordeep inside engine startup". This is now that confusing TypeError.Fix
Introspect both shapes. msgspec structs carry their field names in
__struct_fields__, so validation and the close-match suggestion keep working unchanged, and the code stays compatible with older dataclass-based sglang — no version bound needed.Also adds a test that pins the msgspec path with a stand-in, so it stays covered regardless of which sglang version happens to be installed in CI.
Verification
NVIDIA DGX Spark (GB10, aarch64), l4t13 profile, CUDA 13, LocalAI v4.10.0 core, backend built against sglang 0.5.20.
Measured inside the built backend venv:
python -m unittest testinbackend/python/sglang/, run against that venv:TypeError: must be called with a dataclass type or instance(everyengine_argstest) + 1 unrelated, see belowEnd to end: before the change every
LoadModelfails with the TypeError above; after it, an 80B NVFP4 model loads and serves, and the regression suite we run against this box is green (chat template, structured output, tool calls via API, streaming parity, agent end-to-end, prefix-cache TTFT).The validation itself was added in #9686, which predates the sglang change.
Unrelated failure noticed while testing
test_build_prompt_forwards_enable_thinkingfails on currentmasterindependently of this PR — theSimpleNamespacerequest stub has noImages/Videosattributes, which_build_promptreads since the media-parts change:Left untouched here to keep this PR to one concern. Happy to send a separate one-liner for the stub if useful.