ModelForge

A self-hosted LLM runtime platform for serving GGUF models through an OpenAI-compatible API -- with Hugging Face downloads, model lifecycle management, usage metering, billing primitives, and separate customer and administrator portals.

ModelForge is built for teams that need private, CPU-first inference and data residency. Model weights and prompts remain on infrastructure you control. The default backend uses prebuilt llama-server binaries, so local operators do not need a C++ compiler or CUDA toolchain.

Highlights

OpenAI-compatible API -- streaming and non-streaming /v1/chat/completions, plus tools/response_format metadata and model: "auto"
Hugging Face model browser -- search GGUF repositories, pick quantizations, download with resume/SHA-256 verification, and auto-register into the catalog
Chat playground -- subscriber ChatGPT-style chat page and floating bubble, with collapsible reasoning blocks for <think> models
Immutable execution ledger -- every request gets an InferenceRequest with attempts, timings, request-time pricing, and idempotent quota commits
Signed usage receipts -- Ed25519 receipts with public-key verification and export (/usage/receipts, /verify-receipt)
Budget-aware routing & policies -- versioned routing/budget/data/tool policies with PII redaction and atomic spend ceilings
Opt-in Core Inspector -- one-shot diagnostic capture of pipeline, routing, generation, and metering events without storing prompt or response text
Residency reservations -- warm-model leases that protect capacity from LRU eviction, plus local node heartbeats and deployments
SLO enforcement & credits -- latency/availability windows with automatic service-credit ledger entries
Evaluations & canaries -- revision-gated eval suites and traffic-split channels
Knowledge & memory -- tenant knowledge bases, chunk embeddings, retrieval cost attribution, and retention controls
Local federation simulation -- loopback node transport with production mTLS adapter boundaries
LM Studio-style local serving -- discover GGUF files, register them, and load them on demand with progress feedback
Process-isolated inference -- each loaded model runs in a loopback-only llama-server process
RAM-aware model pool -- configurable budget with reservation-aware eviction
Multi-tenant access -- plans, API keys, quotas, rate limits, and model entitlements
Operations console -- dashboards for requests, receipts, policies, nodes, SLOs, evaluations, and audit events
Usage and billing pipeline -- token metering, BullMQ workers, invoices, and pluggable payment adapters
CPU-first defaults -- mmap, physical-core thread sizing, and conservative per-model concurrency

System architecture

ModelForge keeps the web control plane, API gateway, and inference runtime separate. Only the web application and API gateway are intended to be exposed. Inference ports remain private on loopback or an internal network.

flowchart LR
    subgraph Clients
        Browser[Web browser]
        SDK[OpenAI SDK / API client]
        HF[Hugging Face Hub]
    end

    subgraph ControlPlane["Control plane"]
        Web["Next.js 16<br/>Auth.js + Tailwind CSS"]
    end

    subgraph GatewayPlane["API gateway"]
        Gateway["Express 5<br/>Auth · Quotas · Metering · Downloads"]
        UsageWorker["Usage worker"]
        InvoiceWorker["Invoice worker"]
    end

    subgraph DataPlane["Inference data plane -- private"]
        Pool["Model process pool<br/>RAM budget + LRU"]
        LlamaA["llama-server<br/>Model A"]
        LlamaB["llama-server<br/>Model B"]
        Rust["Optional Rust gRPC engine<br/>Continuous batching"]
        Weights[("GGUF weights")]
    end

    subgraph Storage
        Postgres[("PostgreSQL")]
        Redis[("Redis / BullMQ<br/>optional locally")]
    end

    Browser -->|HTTPS| Web
    Web -->|internal token| Gateway
    SDK -->|Bearer API key| Gateway
    Gateway -->|search / resolve| HF
    HF -->|GGUF download| Weights
    Gateway --> Pool
    Pool --> LlamaA
    Pool --> LlamaB
    Gateway -. optional backend .-> Rust
    Weights --> LlamaA
    Weights --> LlamaB
    Weights --> Rust
    Gateway --> Postgres
    Gateway -->|usage jobs| Redis
    Redis --> UsageWorker
    Redis --> InvoiceWorker
    UsageWorker --> Postgres
    InvoiceWorker --> Postgres

Component responsibilities

Component	Responsibility
`apps/web`	Authenticated customer and admin UI; browser traffic reaches internal services through server-side routes/actions
`apps/gateway`	Public OpenAI API, API-key authentication, plan enforcement, quotas, Hugging Face downloads, inference orchestration, and metering
`llama-server` pool	Default inference backend; one private OS process per loaded model
`apps/inference-engine`	Optional Rust gRPC backend with mmap model loading and continuous batching
PostgreSQL	Users, subscriptions, plans, model registry, API-key hashes, usage events, receipts, and invoices
Redis / BullMQ	Production-grade asynchronous usage and invoice jobs; optional for local development

Request and data flow

Chat completion

sequenceDiagram
    autonumber
    participant C as OpenAI client
    participant G as API gateway
    participant DB as PostgreSQL
    participant P as Model pool
    participant L as llama-server
    participant Q as BullMQ / usage store

    C->>G: POST /v1/chat/completions + Bearer mf_...
    G->>DB: Verify API-key hash, plan, model access, quota
    DB-->>G: Customer + entitlement
    G->>P: Resolve registered model

    alt Model is not resident and auto-load is enabled
        P->>DB: Read hosted-model configuration
        P->>L: Spawn with GGUF, context, threads, loopback port
        L-->>P: Health ready
    end

    G->>L: OpenAI-compatible completion request
    loop Streaming response
        L-->>G: Token delta
        G-->>C: Server-Sent Event
    end
    L-->>G: Final usage + finish reason
    G->>Q: Enqueue metering event
    G-->>C: [DONE]

When Redis is enabled, usage events are written asynchronously through BullMQ. For lightweight local setups with REDIS_ENABLED=false, ModelForge uses the documented direct-Postgres fallback.

Model lifecycle

flowchart LR
    Source{"Model source"}
    HF["Hugging Face browser<br/>search · quant · download"]
    Copy["Copy .gguf into<br/>MODEL_WEIGHTS_DIR"]
    Scan["Filesystem scan"]
    Register["Register metadata<br/>in PostgreSQL"]
    Entitle["Grant model<br/>to plans"]
    Warm{"Warm strategy"}
    Admin["Admin clicks Load"]
    First["First API request"]
    Spawn["Spawn private<br/>llama-server"]
    Resident["Resident in RAM"]
    Evict["LRU eviction<br/>when budget is exceeded"]

    Source -->|Hub| HF --> Scan
    Source -->|manual| Copy --> Scan
    Scan --> Register --> Entitle --> Warm
    Warm -->|manual| Admin --> Spawn
    Warm -->|LLAMA_AUTO_LOAD=true| First --> Spawn
    Spawn --> Resident --> Evict
    Evict -. reload on demand .-> Spawn

Copying or downloading a model file does not automatically expose it to customers. Registration creates the catalog entry (Hugging Face downloads can auto-register after verification), while plan entitlement determines who can call it.

Technology stack

Layer	Technology
Control plane	Next.js 16, React, Tailwind CSS 4, Auth.js
API gateway	Express 5, Zod, Prisma
Default inference	Prebuilt `llama-server` from llama.cpp
Optional inference	Rust, tonic gRPC, `llama-cpp-2`
Model discovery	Local GGUF scan + Hugging Face Hub API
Data	PostgreSQL 16
Queueing	Redis 7, BullMQ
Billing	Mock, Stripe, bKash, and Nagad adapters
Tooling	TypeScript, pnpm, Turborepo, Vitest, ESLint

Repository layout

apps/
  gateway/            OpenAI API, auth, quotas, metering, HF downloads, model pool
  web/                Customer and administrator control plane
  inference-engine/   Optional Rust gRPC inference backend
packages/
  billing/            Invoice calculation and payment adapters
  config/             Shared TypeScript and ESLint configuration
  db/                 Prisma schema, migrations, seed data
  engine/             Shared OpenAI schemas and error contracts
  platform/           Signing, policy, PII, SLO, RAG, and federation helpers
infra/
  docker-compose.dev.yml
  nginx-modelforge.conf   Production Nginx reverse proxy config
  systemd/                Example production service units
scripts/                  Binary fetch, model scan, diagnostics, E2E, benchmark

Prerequisites

Node.js 20+
pnpm 10+
PostgreSQL, either local or through Docker
A compatible .gguf model, or network access to download one from Hugging Face
Redis is recommended for production but optional for local development

The default llama-server backend does not require Rust, CMake, Clang, Visual Studio Build Tools, CUDA, or a system-wide llama.cpp installation.

Quick start

# 1. Install dependencies
pnpm install

# 2. Create local configuration
cp .env.example .env

# PowerShell equivalent:
# Copy-Item .env.example .env

# 3. Start PostgreSQL and Redis with Docker
pnpm infra:up

# 4. Generate the Prisma client, apply migrations, and seed
pnpm db:generate
pnpm db:deploy
pnpm db:seed

# 5. Download the official prebuilt llama.cpp CPU binaries
pnpm llama:fetch

# 6. Start the gateway and control plane
# Frees GATEWAY_PORT / WEB_PORT / GRPC_PORT first if something is still bound
pnpm dev

Before step 4, update .env with secure local values and set MODEL_WEIGHTS_DIR to an absolute path. Ports default to the 9000 series (9000 gateway, 9001 web, 9002 gRPC, 9100+ llama-server) so they do not collide with typical Next.js (3000), Express, or Laravel (8000) apps. Override with GATEWAY_PORT / WEB_PORT in .env, or free them alone via pnpm ports:free.

Service	Local URL
Control plane	http://localhost:9001
OpenAI API	http://localhost:9000/v1
Gateway health	http://localhost:9000/healthz

Development seed accounts

Email	Password	Role
`admin@modelforge.local`	`admin123`	Administrator
`demo@modelforge.local`	`demo123`	Customer

These credentials are for local development only. The seed also prints a one-time API key. Rotate all secrets and remove or replace seeded credentials before any shared or production deployment.

Add and serve a model

Option A -- Hugging Face browser (recommended)

Configure an absolute MODEL_WEIGHTS_DIR in .env.
Optionally set HF_TOKEN for private or gated repositories.
Sign in as an administrator and open Model Registry at /admin/models.
Search the Hub, select a repository, choose a quantization, and click Download.
Wait for verification to complete. Multi-shard GGUFs are queued together and registered only after every shard finishes.
Grant the model to the appropriate plans.
Optionally pre-warm it from Infrastructure at /admin/infra.

Downloads are host-side, resumable (.part files), SHA-256 verified when Hub metadata provides a hash, and limited by HF_MAX_CONCURRENT_DOWNLOADS / HF_MAX_DOWNLOAD_GB. Partial transfers continue even if the browser tab closes.

Option B -- Local filesystem

Copy a .gguf file anywhere below MODEL_WEIGHTS_DIR. Nested folders are supported.
Confirm discovery:
```
pnpm weights:scan
```
Open /admin/models, register the discovered file, and review its slug, quantization, context length, thread count, and pricing.
Grant the model to plans and optionally Load it from /admin/infra.

With LLAMA_AUTO_LOAD=true, the first authorized API request starts the model automatically.

# Inspect backend health, discovered files, and resident models
pnpm engine:status

# Warm a registered model by slug
pnpm engine:status your-model-slug

Subscriber experience

Surface	Path	Purpose
Usage overview	`/dashboard`	Throughput, latency, spend, and recent requests
Chat	`/chat`	Full-page streaming playground with generation settings
Floating chat bubble	all subscriber pages	Compact chat overlay; hidden on `/chat`
Core Inspector	`/core-inspector`	Opt-in one-shot diagnostic capture for the next request
Requests	`/requests`	Immutable execution history and cost debugger
Receipts	`/usage/receipts`	Signed usage proofs and verification

Reasoning models that emit <think> / <thinking> / <reasoning> tags are rendered with a collapsible "Thought process" block. The visible answer is shown separately, and Copy exports the answer only.

OpenAI-compatible API

cURL

curl http://localhost:9000/v1/chat/completions \
  -H "Authorization: Bearer mf_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-slug",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Explain mmap in one sentence."}
    ],
    "temperature": 0.2,
    "max_tokens": 256,
    "stream": false
  }'

OpenAI JavaScript SDK

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.MODELFORGE_API_KEY,
  baseURL: "http://localhost:9000/v1",
});

const response = await client.chat.completions.create({
  model: "your-model-slug",
  messages: [{ role: "user", content: "Hello from ModelForge" }],
});

console.log(response.choices[0]?.message.content);

Streaming uses the standard OpenAI Server-Sent Events format and terminates with data: [DONE].

Use "model": "auto" to let versioned routing policies select an entitled model based on cost, quality, and latency preferences.

Reasoning models

Some reasoning models consume part of max_tokens before producing visible assistant content. Use an appropriate token budget; a very small limit can produce an empty content field even though reasoning tokens were generated. All generated tokens count toward metering. The chat UI collapses inline thought tags so operators can inspect reasoning without burying the answer.

Inference backends

Backend	Compiler required	Recommended use
`llama-server`	No	Default local and production CPU serving
`grpc`	Yes	Optional Rust path for custom continuous batching

Select the backend with INFERENCE_BACKEND.

# Default
INFERENCE_BACKEND=llama-server

# Optional Rust engine
INFERENCE_BACKEND=grpc

The Rust backend is an advanced option. On Windows it requires an MSVC/Clang and CMake toolchain; on Linux it requires an equivalent native build toolchain.

Configuration

Copy .env.example to .env. The most important settings are:

Variable	Purpose
`DATABASE_URL`	PostgreSQL connection string
`REDIS_ENABLED`	Enables Redis-backed rate limits and BullMQ workers
`REDIS_URL`	Redis connection string
`GATEWAY_PORT`	Express API gateway (default `9000`)
`WEB_PORT`	Next.js control plane (default `9001`)
`GRPC_PORT`	Optional Rust inference gRPC (default `9002`)
`MODEL_WEIGHTS_DIR`	Absolute path to local GGUF storage
`HF_TOKEN`	Optional read-only token for private or gated Hugging Face repositories
`HF_MAX_CONCURRENT_DOWNLOADS`	Concurrent host-side Hub downloads (default `2`)
`HF_MAX_DOWNLOAD_GB`	Maximum allowed size for one GGUF file (default `100`)
`INFERENCE_BACKEND`	`llama-server` or `grpc`
`LLAMA_SERVER_BIN`	Optional explicit path to `llama-server`
`LLAMA_AUTO_LOAD`	Loads an entitled model on its first request
`LLAMA_REASONING`	Pass-through to `llama-server` (`off` recommended for chat UX)
`INFERENCE_TIMEOUT_MS`	Abort long generations (default `900000`)
`TOTAL_RAM_BUDGET_MB`	Model-pool RAM budget used for LRU decisions
`MAX_CONCURRENT_PER_MODEL`	Per-model concurrency ceiling
`INTERNAL_SERVICE_TOKEN`	Protects internal gateway routes
`JWT_SECRET` / `AUTH_SECRET`	Gateway and Auth.js signing secrets
`MODELFORGE_SIGNING_DIR`	Ed25519 usage-receipt key storage
`MODELFORGE_PII_REDACT`	Redact emails/phones/cards before inference
`BILLING_MODE`	`mock` or a configured live payment flow

Never commit .env. The repository includes .env.example with development placeholders only. HF_TOKEN stays server-side and is never exposed to the browser.

Authentication, quotas, and billing

Raw API keys are shown once; only SHA-256 hashes are persisted.
Every API key belongs to a customer with a subscription and plan.
Plans define model access, token quota, requests per minute, concurrency, and overage pricing.
Usage records include prompt tokens, completion tokens, model, latency, and an idempotency key.
Dashboard chat uses a session-scoped credential through the same quota, policy, metering, and receipt pipeline as API traffic.
BILLING_MODE=mock works without credentials.
Stripe can be enabled with its secret and webhook keys.
bKash and Nagad adapters provide extension points for Bangladesh payments.

Development and verification

# Repository checks
pnpm lint
pnpm typecheck
pnpm test

# Confirm the configured weights directory
pnpm weights:scan

# Inspect inference state
pnpm engine:status

# OpenAI SDK compatibility (gateway must be running)
MODELFORGE_API_KEY=mf_YOUR_KEY pnpm test:e2e

# Opt-in Core Inspector capture against a live subscriber key
pnpm test:inspector

# Basic concurrency benchmark
MODELFORGE_API_KEY=mf_YOUR_KEY pnpm benchmark

# Real llama-server integration tests
pnpm --filter @modelforge/gateway test

Production deployment (Ubuntu + PM2 + Nginx)

The recommended production setup uses PM2 as the process manager and Nginx as the reverse proxy. The project root is typically /var/www/modelforge.ai.

1. Server prerequisites

# Node.js 20+, pnpm, PM2
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs
npm i -g pnpm pm2

# PostgreSQL and Redis
sudo apt install -y postgresql redis-server

2. Clone, install, and build

cd /var/www/modelforge.ai

pnpm install --frozen-lockfile

# Create .env from the example and fill in secrets
cp .env.example .env
nano .env    # DATABASE_URL, JWT_SECRET, AUTH_SECRET, REDIS_URL, etc.

# Generate Prisma client, run migrations, seed
pnpm db:generate
pnpm db:deploy
pnpm db:seed

# Fetch prebuilt llama-server binary (Linux x86_64)
pnpm llama:fetch

# Build all packages and apps
pnpm build

3. Start with PM2

An ecosystem.config.cjs is included in the repository root. It starts four processes:

PM2 name	What it runs	Port
`modelforge-gateway`	Express API + llama-server process pool	`9000`
`modelforge-web`	Next.js control plane	`9001`
`modelforge-usage-worker`	BullMQ usage event persistence	--
`modelforge-invoice-worker`	Scheduled invoice generation	--

pm2 start ecosystem.config.cjs
pm2 save
pm2 startup    # auto-start on server reboot

Adjust APP_ROOT at the top of ecosystem.config.cjs if your deploy path differs from /var/www/modelforge.ai.

4. Nginx reverse proxy

An example Nginx config is provided at infra/nginx-modelforge.conf.

sudo cp infra/nginx-modelforge.conf /etc/nginx/sites-available/modelforge.conf
sudo ln -s /etc/nginx/sites-available/modelforge.conf /etc/nginx/sites-enabled/
# Edit server_name to your domain or IP
sudo nano /etc/nginx/sites-available/modelforge.conf
sudo nginx -t && sudo systemctl reload nginx

The Nginx config routes:

Location	Upstream	Purpose
`/`	Next.js `:9001`	Admin panel, customer portal, chat UI
`/v1/`	Gateway `:9000`	OpenAI-compatible API with SSE streaming
`/internal/`	Gateway `:9000`	Management routes, localhost only
`/_next/static/`	Next.js `:9001`	Static assets with long-lived cache
`/healthz`	Gateway `:9000`	Health check endpoint

gRPC (:9002) and llama-server ports (:9100+) are never proxied.

5. SSL with Let's Encrypt (optional)

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.com

Then uncomment the HTTPS server block in infra/nginx-modelforge.conf.

Production hardening checklist

Use unique, high-entropy values for INTERNAL_SERVICE_TOKEN, JWT_SECRET, and AUTH_SECRET.
Keep HF_TOKEN as a read-only Hub token with minimum scopes; rotate regularly.
Keep MAX_CONCURRENT_PER_MODEL low for CPU inference (1-2).
Set DEFAULT_N_THREADS near the physical-core count, not logical thread count.
Reserve 25-30% of system RAM outside TOTAL_RAM_BUDGET_MB.
Keep mmap enabled unless the storage environment requires otherwise.
Size INFERENCE_TIMEOUT_MS for your host's tok/s multiplied by max completion length.
Back up PostgreSQL regularly and treat model files as separately managed artifacts.
Remove or change the seeded dev accounts before opening to real users.

Modern platform features

Local-first vertical slices are enabled by default after pnpm db:deploy && pnpm db:seed:

Capability	Where to look
Hugging Face GGUF browser	`/admin/models`
Model load progress + eject confirm	`/admin/infra`
Chat playground + floating bubble	`/chat`
Immutable executions + cost debugger	`/requests`, `GET /v1/requests/:id`
Signed usage receipts	`/usage/receipts`, `/verify-receipt`, `/.well-known/modelforge-usage-keys.json`
Policies, budgets, auto-routing	`/policies`, `/budgets`, `model: "auto"`
Residency reservations + nodes	`/reservations`, `/admin/nodes`
SLO windows + service credits	`/reliability`, `/admin/slo`
Evaluations + canaries	`/admin/evaluations`
Knowledge ingest	`/knowledge`
Audit trail	`/admin/audit`
Opt-in Inference Core Inspector	`/core-inspector`

The Core Inspector is disabled during normal inference. A subscriber can arm a 10-minute, one-shot capture for the next request; it records privacy-safe pipeline, routing, runtime, token-batch, and performance events, then automatically deactivates. Prompt and response content are never retained. Per-token MoE routing and attention tensors are reported as unavailable unless an instrumented runtime adapter can provide genuine data.

Optional Redis workers for SLO rollups and evaluations:

pnpm --filter @modelforge/gateway modern-worker

Signing keys live under MODELFORGE_SIGNING_DIR (default ./data/signing) and are gitignored. Production deployments should swap the local file signer for a KMS/HSM provider behind the same SigningProvider interface.

Security model

flowchart TD
    Internet((Internet))
    Public["Public boundary<br/>Web + Gateway"]
    Internal["Internal boundary<br/>service token"]
    Inference["Inference boundary<br/>loopback / private network"]
    Storage["Persistence boundary<br/>PostgreSQL + Redis"]
    Hub["Hugging Face Hub"]

    Internet -->|HTTPS| Public
    Public -->|x-internal-token| Internal
    Internal --> Inference
    Internal --> Storage
    Internal -->|HF_TOKEN server-side only| Hub

Helmet and explicit CORS configuration protect the Express surface.
Public inference calls require a valid hashed API key.
Browser sessions are handled by Auth.js with role-gated admin routes.
Internal management routes require x-internal-token.
Model processes bind to 127.0.0.1.
Browser clients do not call private inference ports directly.
Hugging Face downloads are initiated by admins only; the Hub token and destination paths never leave the gateway.
Download destinations are constrained under MODEL_WEIGHTS_DIR and verified against Hub manifests before transfer.
GGUF weights, prebuilt runtime binaries, secrets, and generated artifacts are excluded from version control.

Security reports should not include raw credentials, API keys, model weights, or customer data in a public issue.

Files intentionally excluded from Git

Path	Reason
`.env`	Contains local credentials and secrets
`data/models/*/.gguf`	Large model artifacts with independent licenses
`data/models/*/.gguf.part`	Incomplete Hugging Face downloads
`data/signing/`	Local usage-receipt signing material
`vendor/llama.cpp`	Reproducibly fetched with `pnpm llama:fetch`
`node_modules/`, `.next/`, `dist/`, `target/`	Dependency and build outputs

Project status

ModelForge is an early public release with a working local-first modern control plane: Hugging Face GGUF acquisition, chat playground, immutable executions, signed receipts, policy routing, residency reservations, SLO credits, evaluations, knowledge ingest, Core Inspector diagnostics, and federation adapter boundaries. Payment integrations default to mock mode and should be validated against provider sandboxes before production use.

Issues and focused pull requests are welcome.

License

ModelForge is released under the MIT License.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
apps		apps
data		data
docs		docs
infra		infra
packages		packages
scripts		scripts
.cursorrules		.cursorrules
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
ModelForge_Cursor_Build_Instructions.md		ModelForge_Cursor_Build_Instructions.md
README.md		README.md
ecosystem.config.cjs		ecosystem.config.cjs
eslint.config.mjs		eslint.config.mjs
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
turbo.json		turbo.json

Latest commit

History

Folders and files

Repository files navigation