feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify - #12140
Open
mudler-agent wants to merge 6 commits into
Open
mudler-agent wants to merge 6 commits into
mudler-agent wants to merge 6 commits into
Conversation
mudler
force-pushed
the
row/gliner25-ner-clean
branch
from
September 19, 2026 23:10
9ea04c4 to
24343cf
Compare
Wire the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner, ABI v27) so LocalAI can serve zero-shot named entity recognition through the existing TokenClassify gRPC method. backend.go: TokenClassify method on *VllmCpp calls vllm_gliner_ner with the text and labels, copies the C-owned entity array into protobuf TokenClassifyEntity messages, and frees the result. govllmcpp.go: cNerEntity and cNerResult Go POD mirrors matching the C structs; vllmGlinerNer and vllmNerResultFree purego bindings; abiVersion bumped 26 -> 27. options.go: ner_labels, ner_threshold, ner_max_width parsed from engine_args. pkg/grpc: ClassifyModel interface and TokenClassify server handler (follows the Embedding locking pattern). core/config: vllm-cpp backend declares MethodTokenClassify and UsecaseTokenClassify. docs/content/features/vllm-cpp.md: NER section documenting the engine_args keys and the host-forward contract. Assisted-by: MAKI:regolo/glm5.2 [maki]
Use the govet directive for the C-owned NER array, matching the other purego pointer conversions. The array remains valid until its deferred free; the misspelled directive caused CI to flag this conversion. Assisted-by: Codex:gpt-6 golangci-lint
Add POST /v1/systemone, /v1/systemone/permute, and /v1/systemone/separate to LocalAI, mirroring the kev project's structured-extraction API. Each endpoint runs zero-shot NER over the rendered state text and builds kev-compatible answers for three question types: noul (binary entity presence), choice (pick one option), and score (pick one level). The TokenClassifyRequest proto gains a `repeated string labels` field so each question can supply its own labels at inference time, and TokenClassifier gains TokenClassifyWithLabels for per-call label selection. The vllm-cpp backend uses request labels when non-empty, falling back to configured ner_labels then the built-in defaults. Helpers (renderState, softmax, choiceConfidence, scoreConfidence, r2) are ported from kev/api.py and mirrored in vllm.cpp's api_server.cpp so both servers produce the same answer shape. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki]
The SystemOne permute endpoint uses math/rand with a caller-supplied seed for reproducible option permutations, matching kev's random.seed. gosec flags this as G404 (weak RNG). Add #nosec with a comment naming the intent: this is reproducibility, not cryptography. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki]
Advance VLLM_CPP_VERSION from f3cd97e to 5058268d, the commit that landed GLiNER2.5 zero-shot NER support (PR #3224) in vllm.cpp. This brings the DeBERTa v2 encoder, GLiNER2 boundary head, C ABI NER functions, and server endpoints into the LocalAI vllm-cpp backend. The ABI version (27) and Go struct mirrors already match. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki]
The SystemOne handler was passing question IDs as NER labels for noul questions and bare key names for choice questions, so the model never matched any entities. Port the label mapping from vllm.cpp's ParseSystemOneBody: - noul: use the rendered instructions field (with instr alias) as the NER label, not the question ID - choice: use optionText(name, desc) — "name: description" or "name" when the description is null/empty — not the bare key - score: already correct (rendered criteria text) - permute: shuffle indices and build parallel key/label arrays so the NER call uses the optionText labels while the response is keyed by the original option names Also add the instructions field to the SystemOneQuestion schema struct (accepted alongside the instr backward-compat alias). Verified end-to-end against the real GLiNER2.5 model: noul questions now find "Apple Inc. is" (organization, 0.999) and "Tim Cook is" (person, 0.852) where they previously returned zero entities. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki]
mudler
force-pushed
the
row/gliner25-ner-clean
branch
from
September 20, 2026 10:37
c90a915 to
590c503
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(vllm-cpp): add GLiNER2.5 NER and kev-compatible SystemOne API
Wires the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner,
ABI v27) so LocalAI can serve zero-shot named entity recognition, then
adds the full kev-compatible SystemOne structured-extraction API on top.
Depends on mudler/vllm.cpp#3224, which adds the GLiNER2.5 model, the
DeBERTa v2 encoder, the boundary head, the C ABI vllm_gliner_ner
function, and the /v1/ner and /v1/systemone server endpoints.
NER backend
vllm_gliner_ner with text and labels, copies the C-owned entity array
into protobuf TokenClassifyEntity messages, and frees the result. Uses
in.Labels when non-empty, falling back to configured ner_labels then
defaultNerLabels.
mirrors matching the C structs; vllmGlinerNer and vllmNerResultFree
purego bindings; abiVersion bumped 26 -> 27.
parsed from engine_args.
26 -> 27; cNerEntity and cNerResult struct mirror tests added.
TokenClassifyRequest for per-request zero-shot label selection.
AIModel, follows the Embedding pattern).
MethodTokenClassify and UsecaseTokenClassify.
TokenClassifyWithLabels method on TokenClassifier, ModelTokenClassify
passes Labels in the gRPC request.
SystemOne API (kev-compatible)
Mirrors the API from https://github.com/jaredpalmer/kev. Three POST
endpoints run zero-shot NER over rendered state text and build
kev-compatible answers for three question types: noul (binary entity
presence), choice (pick one option), and score (pick one level).
SystemOneResponse, SystemOneAnswer, SystemOneEntity, SystemOneUsage,
SystemOnePermuteRequest, SystemOnePermuteRun, SystemOnePermuteResponse.
SystemOnePermuteEndpoint (re-runs one choice question under n_perm
option orders with seeded RNG), SystemOneSeparateEndpoint (N
independent NER passes, one per question). Helpers: renderState,
softmax, choiceConfidence, scoreConfidence, r2, parseSystemOneRequest,
buildSystemOneAnswer, resolveClassifier — ported from kev/api.py.
three POST routes.
documenting the endpoints, question types, and engine_args keys.
The helper functions are mirrored in vllm.cpp's api_server.cpp so both
servers produce the same answer shape.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]