Skip to content

feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify - #12140

Open
mudler-agent wants to merge 6 commits into
masterfrom
row/gliner25-ner-clean
Open

mudler-agent wants to merge 6 commits into
masterfrom
row/gliner25-ner-clean

Conversation

@mudler-agent

@mudler-agent mudler-agent commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

feat(vllm-cpp): add GLiNER2.5 NER and kev-compatible SystemOne API

Wires the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner,
ABI v27) so LocalAI can serve zero-shot named entity recognition, then
adds the full kev-compatible SystemOne structured-extraction API on top.

Depends on mudler/vllm.cpp#3224, which adds the GLiNER2.5 model, the
DeBERTa v2 encoder, the boundary head, the C ABI vllm_gliner_ner
function, and the /v1/ner and /v1/systemone server endpoints.

NER backend

  • backend/go/vllm-cpp/backend.go: TokenClassify method on *VllmCpp calls
    vllm_gliner_ner with text and labels, copies the C-owned entity array
    into protobuf TokenClassifyEntity messages, and frees the result. Uses
    in.Labels when non-empty, falling back to configured ner_labels then
    defaultNerLabels.
  • backend/go/vllm-cpp/govllmcpp.go: cNerEntity and cNerResult Go POD
    mirrors matching the C structs; vllmGlinerNer and vllmNerResultFree
    purego bindings; abiVersion bumped 26 -> 27.
  • backend/go/vllm-cpp/options.go: ner_labels, ner_threshold, ner_max_width
    parsed from engine_args.
  • backend/go/vllm-cpp/vllmcpp_test.go: ABI version assertion updated
    26 -> 27; cNerEntity and cNerResult struct mirror tests added.
  • backend/backend.proto: repeated string labels = 4 added to
    TokenClassifyRequest for per-request zero-shot label selection.
  • pkg/grpc/interface.go: ClassifyModel interface (opt-in extension to
    AIModel, follows the Embedding pattern).
  • pkg/grpc/server.go: TokenClassify gRPC handler.
  • core/config/backend_capabilities.go: vllm-cpp backend declares
    MethodTokenClassify and UsecaseTokenClassify.
  • core/backend/token_classify.go: Labels field in TokenClassifyOptions,
    TokenClassifyWithLabels method on TokenClassifier, ModelTokenClassify
    passes Labels in the gRPC request.

SystemOne API (kev-compatible)

Mirrors the API from https://github.com/jaredpalmer/kev. Three POST
endpoints run zero-shot NER over rendered state text and build
kev-compatible answers for three question types: noul (binary entity
presence), choice (pick one option), and score (pick one level).

  • core/schema/systemone.go: SystemOneRequest, SystemOneQuestion,
    SystemOneResponse, SystemOneAnswer, SystemOneEntity, SystemOneUsage,
    SystemOnePermuteRequest, SystemOnePermuteRun, SystemOnePermuteResponse.
  • core/http/endpoints/localai/systemone.go: SystemOneEndpoint,
    SystemOnePermuteEndpoint (re-runs one choice question under n_perm
    option orders with seeded RNG), SystemOneSeparateEndpoint (N
    independent NER passes, one per question). Helpers: renderState,
    softmax, choiceConfidence, scoreConfidence, r2, parseSystemOneRequest,
    buildSystemOneAnswer, resolveClassifier — ported from kev/api.py.
  • core/http/routes/systemone.go: RegisterSystemOneRoutes wiring the
    three POST routes.
  • core/http/app.go: route registration alongside RegisterPIIRoutes.
  • docs/content/features/vllm-cpp.md: NER section + SystemOne API section
    documenting the endpoints, question types, and engine_args keys.

The helper functions are mirrored in vllm.cpp's api_server.cpp so both
servers produce the same answer shape.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]

Comment thread core/http/endpoints/localai/systemone.go Fixed
@mudler
mudler force-pushed the row/gliner25-ner-clean branch from 9ea04c4 to 24343cf Compare September 19, 2026 23:10
mudler and others added 6 commits September 20, 2026 10:34
Wire the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner,
ABI v27) so LocalAI can serve zero-shot named entity recognition through
the existing TokenClassify gRPC method.

backend.go: TokenClassify method on *VllmCpp calls vllm_gliner_ner with
the text and labels, copies the C-owned entity array into protobuf
TokenClassifyEntity messages, and frees the result.

govllmcpp.go: cNerEntity and cNerResult Go POD mirrors matching the C
structs; vllmGlinerNer and vllmNerResultFree purego bindings; abiVersion
bumped 26 -> 27.

options.go: ner_labels, ner_threshold, ner_max_width parsed from
engine_args.

pkg/grpc: ClassifyModel interface and TokenClassify server handler
(follows the Embedding locking pattern).

core/config: vllm-cpp backend declares MethodTokenClassify and
UsecaseTokenClassify.

docs/content/features/vllm-cpp.md: NER section documenting the
engine_args keys and the host-forward contract.

Assisted-by: MAKI:regolo/glm5.2 [maki]
Use the govet directive for the C-owned NER array, matching the other
purego pointer conversions. The array remains valid until its deferred
free; the misspelled directive caused CI to flag this conversion.

Assisted-by: Codex:gpt-6 golangci-lint
Add POST /v1/systemone, /v1/systemone/permute, and
/v1/systemone/separate to LocalAI, mirroring the kev project's
structured-extraction API. Each endpoint runs zero-shot NER over the
rendered state text and builds kev-compatible answers for three question
types: noul (binary entity presence), choice (pick one option), and
score (pick one level).

The TokenClassifyRequest proto gains a `repeated string labels` field so
each question can supply its own labels at inference time, and
TokenClassifier gains TokenClassifyWithLabels for per-call label
selection. The vllm-cpp backend uses request labels when non-empty,
falling back to configured ner_labels then the built-in defaults.

Helpers (renderState, softmax, choiceConfidence, scoreConfidence, r2)
are ported from kev/api.py and mirrored in vllm.cpp's api_server.cpp so
both servers produce the same answer shape.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
The SystemOne permute endpoint uses math/rand with a caller-supplied
seed for reproducible option permutations, matching kev's random.seed.
gosec flags this as G404 (weak RNG). Add #nosec with a comment naming
the intent: this is reproducibility, not cryptography.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Advance VLLM_CPP_VERSION from f3cd97e to 5058268d, the commit that
landed GLiNER2.5 zero-shot NER support (PR #3224) in vllm.cpp. This
brings the DeBERTa v2 encoder, GLiNER2 boundary head, C ABI NER
functions, and server endpoints into the LocalAI vllm-cpp backend.
The ABI version (27) and Go struct mirrors already match.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
The SystemOne handler was passing question IDs as NER labels for noul
questions and bare key names for choice questions, so the model never
matched any entities. Port the label mapping from vllm.cpp's
ParseSystemOneBody:

- noul: use the rendered instructions field (with instr alias) as the
  NER label, not the question ID
- choice: use optionText(name, desc) — "name: description" or "name"
  when the description is null/empty — not the bare key
- score: already correct (rendered criteria text)
- permute: shuffle indices and build parallel key/label arrays so the
  NER call uses the optionText labels while the response is keyed by
  the original option names

Also add the instructions field to the SystemOneQuestion schema struct
(accepted alongside the instr backward-compat alias).

Verified end-to-end against the real GLiNER2.5 model: noul questions
now find "Apple Inc. is" (organization, 0.999) and "Tim Cook is"
(person, 0.852) where they previously returned zero entities.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
@mudler
mudler force-pushed the row/gliner25-ner-clean branch from c90a915 to 590c503 Compare September 20, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants