Discussing Further Issues Related to Project Engineering

[ZacharyZcR]

We’ll explore more issues related to the project’s engineering, including the SDK, improved architecture, a Rust rewrite, the user interface, and testing enhancements. If we’re going to build a robust project, we may need to finalize these details. We might also expand the project to incorporate technologies like JIT, making flexible and efficient use of VRAM, RAM, and disk space, and achieving further optimization through hotspot analysis and loading of large models.

If the project maintainer is interested in this, I would be happy to take charge of the integration and development. This is a very promising and ambitious project.

[JustVugg]

Great energy — and after #16 there's no doubt you can drive serious pieces of this. Let's sort the list into what fits the project's soul and what doesn't.

The one hard no: a Rust rewrite. colibrì's identity is a single C file you can read in an afternoon — gcc and go, no dependency tree. Performance-wise a rewrite buys nothing here: the hot paths are bounded by disk and memory bandwidth, not by the language. A Rust port as a sibling project elsewhere is of course fine, but the reference engine stays C.

Everything else is welcome, and most of it already has a seed to grow from:

• Tiered VRAM/RAM/disk — you literally built the VRAM tier in feat: add tiered CUDA acceleration for routed experts #16. Deepening it (smarter placement, per-expert precision by heat, promotion/demotion at runtime) is a natural continuation.
• Hotspot analysis — the learning cache (.coli_usage + AUTOPIN with confidence scaling) is exactly this today. Real headroom: expert co-occurrence analysis, disk-layout reordering so co-firing experts read sequentially, and the just-landed router-lookahead (PILOT=1, 71.6% next-layer recall) which wants tuning on machines where the disk isn't saturated.
• Testing — your REPLAY fixture is now the base for reproducible kernel A/Bs. A regression suite around it would be valuable, with one constraint: the maintainer deliberately runs no CI — everything must stay a local one-command affair like setup.sh.
• SDK / API surface — [Feature] Web API? #14 asks for an OpenAI-compatible server, which is the most-requested integration and a thin wrapper over the existing SERVE protocol. That's the highest-leverage "SDK" piece by far.
• JIT — interesting, but concrete case first: the hot loops are already shape-specialized by hand (AVX2/VNNI/NEON). If profiling shows a dispatch cost worth burning, numbers welcome.

On "taking charge of integration": the way this project works well is scoped ownership — you clearly own the GPU/tiering + benchmark-harness territory already, and PRs there will keep getting fast reviews. Architecture and identity decisions (pure C core, GLM-only, simplicity over features) stay with @JustVugg.

[JustVugg]

---

📊 Community vote: Discord server?

@JustVugg is ready to create a project Discord so design discussions like this one can happen in real time (GitHub issues stay the home for bugs, benchmarks and decisions).

Vote with a reaction on THIS comment:

• 👍 — yes, I'd join a colibrì Discord
• 👎 — no, keep everything on GitHub

If the 👍 win, the invite link lands in the README.

[ZacharyZcR]

Great energy — and after #16 there's no doubt you can drive serious pieces of this. Let's sort the list into what fits the project's soul and what doesn't.

The one hard no: a Rust rewrite. colibrì's identity is a single C file you can read in an afternoon — gcc and go, no dependency tree. Performance-wise a rewrite buys nothing here: the hot paths are bounded by disk and memory bandwidth, not by the language. A Rust port as a sibling project elsewhere is of course fine, but the reference engine stays C.

Everything else is welcome, and most of it already has a seed to grow from:

• Tiered VRAM/RAM/disk — you literally built the VRAM tier in feat: add tiered CUDA acceleration for routed experts #16. Deepening it (smarter placement, per-expert precision by heat, promotion/demotion at runtime) is a natural continuation.
• Hotspot analysis — the learning cache (.coli_usage + AUTOPIN with confidence scaling) is exactly this today. Real headroom: expert co-occurrence analysis, disk-layout reordering so co-firing experts read sequentially, and the just-landed router-lookahead (PILOT=1, 71.6% next-layer recall) which wants tuning on machines where the disk isn't saturated.
• Testing — your REPLAY fixture is now the base for reproducible kernel A/Bs. A regression suite around it would be valuable, with one constraint: the maintainer deliberately runs no CI — everything must stay a local one-command affair like setup.sh.
• SDK / API surface — [Feature] Web API? #14 asks for an OpenAI-compatible server, which is the most-requested integration and a thin wrapper over the existing SERVE protocol. That's the highest-leverage "SDK" piece by far.
• JIT — interesting, but concrete case first: the hot loops are already shape-specialized by hand (AVX2/VNNI/NEON). If profiling shows a dispatch cost worth burning, numbers welcome.

On "taking charge of integration": the way this project works well is scoped ownership — you clearly own the GPU/tiering + benchmark-harness territory already, and PRs there will keep getting fast reviews. Architecture and identity decisions (pure C core, GLM-only, simplicity over features) stay with @JustVugg.

That's great, but perhaps we could communicate directly—even through a translator. Although the AI-generated responses are very polished, they always feel a bit stiff. I'll discuss deeper roadmap issues with you.

[JustVugg]

Of course, I'm here, I always read everything! That's why I'm promoting a Discord server. If there's interest, I'll create one and we can talk there. We can still talk and organize something, maybe a Zoom call or something.

[ZacharyZcR]

Of course, I'm here, I always read everything! That's why I'm promoting a Discord server. If there's interest, I'll create one and we can talk there. We can still talk and organize something, maybe a Zoom call or something.

Dude, I really like you—let's get this project started!

[kostadin-damyanov-prime]

This may be a stupid question but if this project can use CUDA, is it possible to add vulcan support so that non-nvidia PCs can enjoy hardware acceleration too?

[ZacharyZcR]

This may be a stupid question but if this project can use CUDA, is it possible to add vulcan support so that non-nvidia PCs can enjoy hardware acceleration too?

The answer is yes; we are actively developing it.

[maikelthedev]

This may be a stupid question but if this project can use CUDA, is it possible to add vulcan support so that non-nvidia PCs can enjoy hardware acceleration too?

Yes please.

Do we have in here, in the README.md the instructions to run the figures @ZacharyZcR got on his PR up to date?

[JustVugg]

Adopting the dev branch — thank you @ZacharyZcR for the suggestion, it's a good call.

Going forward:

• main stays the stable branch — what users clone, always known-good (engine passes the token-exact oracle).
• dev is the integration branch. Please open new PRs against dev. Reviewed PRs land there first; once a batch is tested and stable, it's fast-forwarded into main. This keeps main clean instead of absorbing every PR one at a time — exactly the orderly flow you described.

Every PR still gets the same review on either branch (clean build, oracle 32/32 TF + 20/20 greedy, targeted validation). In-flight PRs already targeting main (the Metal rebase #72, the CUDA line #68, the PILOT rebase #78) can finish where they are; new work goes to dev.

On maintainer merge access: for now the review throughput is being handled, so we'll keep merges flowing through the maintainer to keep main's provenance clean and simple. As volume grows we can revisit delegating dev merges with the right protections in place. The scoped-ownership model stands — the GPU/CUDA/Metal territory is clearly yours, and PRs there keep getting fast, thorough reviews. This is exactly the kind of process maturity a project moving this fast needs — appreciated.

[JustVugg]

Open-PR status board — please rebase onto dev

main moved a fair bit this week (English runtime output #85, OMP hot-team #77, PIPE #79, iobench/oracle-guard fixes, and more), so every open PR is currently conflicting and needs a rebase onto dev (the new integration branch). Here's exactly what each one needs — most are just a rebase; a couple need a small decision first. Once green on dev, I review and land.

PR	author	what it needs
#68 unify runtime	@ZacharyZcR	✅ topp fix confirmed — just rebase onto dev, then it lands
#72 Metal backend	@codearranger	rebase onto dev (a user is running it at 1.83 tok/s, #87 — strong reason to land)
#78 PILOT_REAL	@kryptt	rebase + reconcile with your own #79 (PIPE) — do PIPE and PILOT_REAL share one I/O pool? (details on the PR)
#80 full-resident placement	@ZacharyZcR	rebase onto dev
#83 NVFP4 dequant	@kryptt	rebase/retarget to dev (converter-only, should be a clean review)
#84 Vulkan iGPU	@kryptt	rebase onto dev (GPU path validated on your hardware)
#75 Windows GPU	@nalepy	rebase onto dev (Windows/CUDA path validated on your hardware)
#71 Windows + VRAM tiers	@nuspy	rebase + split — parts overlap the merged #40 (Windows) and #68 (VRAM); keep only your net-new pieces (dynamic-RAM MEMWATCH, domain profiles)

Workflow reminder: open/retarget PRs against dev, not main. Every PR still gets the full review (0-warning build, oracle 32/32 TF + 20/20 greedy, targeted validation) before it lands on dev; stable batches fast-forward into main. I tried hand-rebasing a couple myself to save you the work — and stopped where the conflict was a design decision (two concurrent features composing) or untestable GPU code, because a resolution I can't validate is worse than none. Those are the ones with notes above. Thank you all — this is a lot of good work; the rebases are the last mile.

[digivasserver-ai]

Engineering Workarounds- If you want to turn this into a usable or highly optimized "frugal AI" setup, there are a few infrastructure paths you can take:

"Quoted"
Not all experts are created equal. In MoE models, certain experts act as "shared knowledge" hubs and are called 10x more frequently than others.

By tracking expert activation frequency during the first few tokens, you can pin the top 20% most active experts into your physical RAM.

The remaining rare experts are left on the disk. This hybrid approach can jump-start generation speeds.

[ZacharyZcR]

Engineering Workarounds- If you want to turn this into a usable or highly optimized "frugal AI" setup, there are a few infrastructure paths you can take:

"Quoted" Not all experts are created equal. In MoE models, certain experts act as "shared knowledge" hubs and are called 10x more frequently than others.

By tracking expert activation frequency during the first few tokens, you can pin the top 20% most active experts into your physical RAM.

The remaining rare experts are left on the disk. This hybrid approach can jump-start generation speeds.

In fact, it may be different from what you think: every time the disk expert is called, it causes significant latency, which increases progressively from VRAM to RAM to the disk.

[maikelthedev]

Thank to this project now I do "override-tensor" in my llama.cpp presets.ini file and distribute the experts RAM and VRAM, keeping the other 3 types always in GPU, whenever I use a model bigger than my VRAM.It has produced good results.Yet I wish there was a way to know the exact value of each expert. So far I've only figured there are four major types in models like Qwen3.6-35B (I'm using it for experimentation with colibrí, it fits in my VRAM but I'm forcing it to distribute in 3 places one of them NVMe to see what I learn), my goal is eventually DeepSeek v4 Flash, once I understand all the pieces. I'm still far from anything usable anything that's come out of my experiments, it's been mostly exploratory with BigPickle (free model from opencode) trying from different angles. From: ZacharyZcR ***@***.***>Sent: Sunday, July 12, 2026 8:10:42 pmTo: JustVugg/colibri ***@***.***>Cc: Maikel Frias Mosquea ***@***.***>; Comment ***@***.***>Subject: Re: [JustVugg/colibri] Discussing Further Issues Related to Project Engineering (Issue #19)ZacharyZcR left a comment (JustVugg/colibri#19)Engineering Workarounds- If you want to turn this into a usable or highly optimized "frugal AI" setup, there are a few infrastructure paths you can take:"Quoted" Not all experts are created equal. In MoE models, certain experts act as "shared knowledge" hubs and are called 10x more frequently than others.By tracking expert activation frequency during the first few tokens, you can pin the top 20% most active experts into your physical RAM.The remaining rare experts are left on the disk. This hybrid approach can jump-start generation speeds.In fact, it may be different from what you think: every time the disk expert is called, it causes significant latency, which increases progressively from VRAM to RAM to the disk.—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.

[sharma-open-source]

Hi

Intersting project.

I've been building a project (NunSpark) that lands on the same core thesis as Colibrì from a different angle, and hit some of the same walls. Sharing in case any of it is useful; happy to go deeper on anything.

What we are (so you know the overlap boundary): MLX / Apple-Silicon, Python on stock mlx-lm. General-purpose — we stream whole transformer layers per token across ~24 dense + some MoE arches, rather than hand-optimizing one model. So we're not a drop-in for your C/x86/GLM-5.2 world, but several findings are hardware-independent.

Things that might save you time:

Prefetch is net-neutral once disk is saturated — we found this independently. Your PILOT=1 router-lookahead is off by default for exactly the reason we shelved our background page-cache warmer: when the drive is ~80%+ saturated by streaming, there's no spare bandwidth to overlap compute into. Our conclusion after a full investigation: I/O warming/prefetch is not the lever — speculative depth is. If you're weighing effort, prefetch only pays off on the balanced-hardware boxes you already identified (the Ryzen AI 9 case).

The disk-bound regime rewards much deeper speculation than GPU serving. This is our main finding and it's model-independent: because draft compute hides behind the dominant weight-read cost, you can push draft depth (K) far past what's sane for GPU-bound serving — each accepted draft token saves a whole weight-read pass instead of costing latency. We measured acceptance multipliers rising 2.5×→5× as K grew while naive tok/s fell (opposite slopes). Your MTP head drafts shallow (~2.2–2.8 tok/forward); it may be worth testing a deeper draft budget — with expert streaming, more accepted tokens = fewer 11 GB expert sweeps.

Lossless is achievable if the verification path doesn't shape-dependent-round. You noted your quantized integer kernels round differently for batched (S>1) vs single-token, so MTP speculation is "lossless in exact arithmetic" but not byte-identical. On our side, running verification through the same kernels as single-token decode gives genuinely bit-identical output (our --accept-top-k 1 mode). If byte-reproducibility ever matters to you, the fix is making the S>1 verify path round identically to S=1, not changing the speculation logic.

Draft-head precision has a cliff — worth documenting loudly. Your int4-MTP-head → 0% acceptance finding matches our experience that the drafter's output head is precision-sensitive in a way the backbone isn't. We keep our drafter head at bf16 for the same reason. Might be worth a prominent warning so contributors don't quantize it uniformly.

What we'd genuinely want from you (so this isn't one-directional): your expert-level streaming + L+1-router prefetch is exactly the MoE path we've deferred — we stream whole dense layers, which is wasteful for MoE where only ~40B/744B activate. Your resident-dense-tier / streamed-routed-expert split is the cleanest writeup of that pattern we've seen.

Everything's MIT on our side — repo/benchmarks available if any of the above is worth a closer look. Cheers.

https://github.com/sharma-open-source/Nunspark