Skip to content

edge-runtime.md

Latest commit

 

History

History
158 lines (129 loc) · 7.74 KB

edge-runtime.md

File metadata and controls

158 lines (129 loc) · 7.74 KB
Raw

Model-edge runtime ABI

c/edge_runtime.h is Colibri's engine-neutral local C boundary for the two ends of a distributed inference route:

text -> tokenizer -> embedding -> one or more Segment engines -> final head

It is not a network protocol. Colibri owns tokenization, model boundary math, formats and kernels; a consumer such as Lumabri owns discovery, transport, placement, leases and session lifecycle. Together with segment_runtime.h, this lets a chatter process run a real model without loading every transformer layer locally.

The ABI is additive. Ordinary Colibri CLIs do not link or register Edge adapters and keep their existing initialization and inference paths.

External consumers can build the complete CPU-baseline runtime with make -C c segment-edge-library. The resulting c/build/segment/libcolibri_segment_edge.a contains both public runtimes and all seven built-in adapters. Consumers link it after their own objects and call the explicit registration functions below; ordinary Colibri executables do not link this archive.

Lifecycle and compatibility

  1. Explicitly register the required adapters during process initialization. There are no link-time constructors, including on Windows.
  2. Open one model boundary with coli_edge_engine_open.
  3. Read its capabilities and require an exact match with the routed Segment chain for state_schema, numeric_class, state dtype and state width.
  4. Tokenize and embed the prompt, run it through the complete Segment chain, then call coli_edge_select on the last row.
  5. For decode, embed the selected token and repeat at the next position.
  6. Close the Edge engine after its sessions have drained.

The caller initializes every public structure's struct_size. Capability queries zero the caller's complete allocation before copying fields known to the runtime, so an older runtime cannot leave future extension fields uninitialized.

Version 1 deliberately exposes deterministic greedy selection. Sampling is a future additive operation; it is not emulated outside Colibri. Registration must finish before concurrent lookup. A consumer should currently host one active model per chatter process because some existing engine configuration is process-global; Qwen's production tokenizer makes that restriction explicit and rejects a second live Qwen Edge engine.

Real adapters

c/edge_adapters.h exposes explicit registration functions for every current Colibri model family:

Adapter Boundary state Edge residency
GLM-5.2 hidden state tokenizer, embedding, final norm and head
Inkling hidden state tokenizer, optional embed norm, final norm and head
Kimi K3 hidden plus all AttnRes blocks tokenizer, final transforms and head; embedding rows stream on demand
OLMoE hidden state tokenizer, embedding, final norm and head
Qwen3.6 hidden state tokenizer, embedding, final norm and head
Qwen3.8-Flash-Next four-stream hyper state tokenizer, native BF16 embedding, final hyper mixer and native BF16 head
DeepSeek V4 expanded mHC state tokenizer and small final mHC tensors; BF16 embedding rows and head tiles stream on demand

Every adapter currently advertises CPU only. Accelerator flags will be added only when that adapter actually executes its Edge operations on the matching Colibri backend. resident_bytes reports the adapter-owned model tensors and can be bounded with memory_limit_bytes; model files and tokenizer metadata remain additional mapped/indexed resources.

API contract

Tokenize and detokenize support a sizing pass by passing a null output with zero capacity. coli_edge_embed accepts exactly one token ID per row and emits rows * state_width values in the advertised dtype. coli_edge_select applies the model's exact final transform and output head to every input row, returning one greedy token and an optional score per row. Edge ABI v2 also exposes coli_edge_logits, which returns the complete row-major vocabulary logits after that same model-specific transform. Colibri therefore remains the owner of model math, while a serving caller can apply temperature, top-p and its own reproducible RNG policy without duplicating any of the six heads. argmax(logits) is release-gated against coli_edge_select for every family.

The runtime validates structure sizes, activation geometry, batch limits, output capacities and cancellation before entering an adapter. Model adapters also validate token IDs and checkpoint boundary tensors. Operational and compatibility errors are returned to the caller; the network layer decides whether to retry, migrate or fall back locally. Checkpoint discovery still uses the engines' shared fail-closed safetensors/config loaders: a malformed or hostile model directory is a process-fatal deployment error, as it is for the standalone engines, rather than a recoverable request error.

All-family release gate

make -C c edge-adapters verifies that all seven real adapters coexist in one runtime. edge-adapters-real is the stronger gate: for every family it loads the real Edge adapter and a full real Segment range, round-trips text, embeds an independent oracle prompt, performs prefill plus decode and compares three greedy tokens with that oracle. It also requires generated tokens to be detokenizable and requires exact Edge/Segment capability identity. For each family, the gate additionally computes the full logits for the first decode row and requires their argmax to equal the already oracle-checked greedy token.

Generate the existing tiny checkpoints from c/ (they are test data and are not committed):

python3 tools/make_glm_oracle.py
python3 tools/make_edge_tiny_tokenizer.py glm_tiny --vocab-size 256

python3 tools/make_tiny_inkling.py tiny_inkling
python3 tools/make_edge_tiny_tokenizer.py tiny_inkling --vocab-size 256

python3 tools/make_kimi_k3_tiny.py --output kimi_k3_tiny --force
python3 tools/make_edge_tiny_tokenizer.py kimi_k3_tiny --vocab-size 320

python3 tools/make_olmoe_tiny.py --output olmoe_tiny_src --force
python3 tools/make_edge_tiny_tokenizer.py olmoe_tiny_src --vocab-size 128
python3 tools/convert_olmoe_merged.py --model olmoe_tiny_src \
  --out olmoe_tiny_merged --min-free-gb 0

python3 tools/make_qwen36_tiny.py --out qwen36_edge_src \
  --emit-ref qwen36_edge_src/ref_qwen36.json --ref-mode full --max-new 8
python3 tools/make_edge_tiny_tokenizer.py qwen36_edge_src --vocab-size 320
python3 tools/convert_qwen36.py --model qwen36_edge_src \
  --out qwen36_edge_i8 --ebits 8 --no-readme

python3 tools/make_qwen38_tiny.py --out qwen38_tiny
python3 tools/make_edge_tiny_tokenizer.py qwen38_tiny --vocab-size 64

python3 tools/make_deepseek_v4_tiny.py --output deepseek_v4_edge_tiny --force
python3 tools/make_edge_tiny_tokenizer.py deepseek_v4_edge_tiny --vocab-size 128

Inkling's generator requires c/tools/oracle-requirements.txt. Then run the single all-family gate:

make -C c edge-adapters-real \
  GLM_EDGE_MODEL=glm_tiny GLM_EDGE_REF=ref_glm.json \
  INKLING_EDGE_MODEL=tiny_inkling \
  INKLING_EDGE_REF=tiny_inkling/ref_inkling.json \
  KIMI_EDGE_MODEL=kimi_k3_tiny KIMI_EDGE_REF=kimi_k3_tiny/ref.json \
  OLMOE_EDGE_MODEL=olmoe_tiny_merged \
  OLMOE_EDGE_REF=olmoe_tiny_src/ref_olmoe.json \
  QWEN_EDGE_MODEL=qwen36_edge_i8 \
  QWEN_EDGE_REF=qwen36_edge_src/ref_qwen36.json \
  QWEN38_EDGE_MODEL=qwen38_tiny \
  QWEN38_EDGE_REF=qwen38_tiny/ref.json \
  DEEPSEEK_EDGE_MODEL=deepseek_v4_edge_tiny \
  DEEPSEEK_EDGE_REF=deepseek_v4_edge_tiny/ref.json

The fixture tokenizer helper only fills the tokenizer deliberately omitted by math-only generators. Shipping checkpoints always use their production tokenizer. A family is not considered supported from registration or a synthetic adapter alone: this oracle gate must pass for all current families.