Add vLLM as a wrapped server backend (ROCm)

[ramkrishna2910]

Status: early rough draft

Tested only on gfx1151 (Strix Halo / Radeon 8060S). The backend does support gfx1150, gfx110X, and gfx120X as well but havent been tested yet. Feedback from people with other AMD GPUs is the main thing we need before treating this PR as ready to merge.

Summary

Adds vLLM as a WrappedServer backend using AMD's pre-built ROCm wheels from lemonade-sdk/vllm-rocm. Follows the same install/load/unload pattern as llama.cpp-ROCm.

Scope of testing

Verified end-to-end (install → load → inference → benchmark) on exactly one configuration:

GPU: Ryzen AI MAX+ PRO 395 w/ Radeon 8060S (gfx1151, Strix Halo)
OS: Ubuntu 24.04
Kernel: 6.18.22 mainline from kernel.ubuntu.com/mainline
amdgpu: built-in driver from the 6.18.22 kernel package; amdgpu-dkms 6.19.0 from ROCm 31.20 also installed but not actively loading
Models: Qwen3 (0.6B–8B), Qwen3.5 (0.8B–9B), Llama-3.2 (1B/3B), Phi-4-mini, Gemma-3-4B, both FP16 and AWQ variants

Not tested:

gfx1150 / gfx110X / gfx120X — the install flow fetches per-arch assets that exist but have never been exercised. Other architectures may have their own kernel/driver gotchas we haven't discovered.
Other distros — everything below assumes Ubuntu 24.04. The kernel and amdgpu setup on Fedora/Arch/openSUSE likely works but is untested.
Multi-user / batched serving — all benchmarks are single-user, one request at a time. vLLM's scheduler strengths (paged attention, prefix caching, continuous batching) are not exercised here.

Prerequisites for testing

Hardware

AMD GPU in one of: gfx1151 (Strix Halo), gfx1150 (Strix Point), gfx110X (RDNA3), gfx120X (RDNA4)
Kernel — strict requirement
You need a kernel with the CWSR (Context Wave Save/Restore) fix. Without it, any GPU dispatch triggers a GCVM_L2_PROTECTION_FAULT and the backend hangs. The verified path is mainline 6.18.4+.

Full doc: docs/gfx1151_linux.html.

If amdgpu-dkms is installed
The default Radeon repo (amdgpu/30.30) ships amdgpu-dkms 6.16.13 which overrides the kernel's built-in driver with a broken version. Either switch to amdgpu/31.20:

sudo sed -i 's|amdgpu/30\.30/|amdgpu/31.20/|g' /etc/apt/sources.list.d/amdgpu*.list
sudo apt update
sudo apt install -y amdgpu-dkms amdgpu-dkms-firmware
sudo reboot

Or uninstall amdgpu-dkms entirely — vLLM ships its own ROCm user-space, you don't need the DKMS package unless you also want to run other ROCm tools outside Lemonade.

Verify prerequisites

Kernel version

uname -r # expect 6.18.4 or newer

CWSR properties exported

grep -E "cwsr_size|ctl_stack_size" /sys/class/kfd/kfd/topology/nodes/*/properties

---

Test plan

1. Build

git fetch origin
git checkout test-vllm
./setup.sh
cmake --preset default
cmake --build --preset default -j$(nproc)

Test plan

1. Build
   git fetch origin
   git checkout test-vllm
   ./setup.sh
   cmake --preset default
   cmake --build --preset default -j$(nproc)

2. Start server

./build/lemond --port 8083

In another terminal, verify it's up:

curl -s http://localhost:8083/v1/health | python3 -m json.tool

3. Install vLLM backend
   First install pulls ~5 GB (split into two ~2.5 GB parts). Expect ~60 s on a fast link.

curl -s -X POST http://localhost:8083/v1/install \
  -H "Content-Type: application/json" \
  -d '{"recipe": "vllm", "backend": "rocm"}'
  {"backend":"rocm","recipe":"vllm","status":"success"}

4. Load a small model

time curl -s -X POST http://localhost:8083/v1/load \
  -H "Content-Type: application/json" \
  -d '{"model_name": "Qwen3-0.6B-vllm"}'
{"checkpoint":"Qwen/Qwen3-0.6B","model_name":"Qwen3-0.6B-vllm","recipe":"vllm","status":"success"}

First load takes 20–30 s (Triton JIT compile for the architecture). Subsequent loads are faster.

5. Run inference

curl -s -X POST http://localhost:8083/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3-0.6B-vllm","messages":[{"role":"user","content":"Say hi in 3 words"}],"max_tokens":30,"temperature":0}' \

Known gotchas

First-run Triton JIT: cold load of a new model size compiles kernels for your GPU, taking 20–350 s. Subsequent loads hit the on-disk cache.
huggingface-hub version conflict: fixed by forcing PYTHONNOUSERSITE=1 when launching vllm-server. If you still hit this, make sure ~/.local/lib/python3.12/site-packages/huggingface_hub isn't shadowing the bundled one.
Transformers version lag: the bundled vLLM 0.19.0 pins transformers <5. Models whose config.json declares model_type: qwen3_5_text (only some newer Qwen3.5 variants) won't load until a vLLM release that bumps the transformers pin. The model registry in this PR avoids those repos.
amdgpu-dkms 6.16.13 masking the built-in driver: see prerequisites. Uninstall or upgrade.

[superm1]

Why nightly TheRock SDK? I would think you are better off taking the tagged release (and then you can re-use with different backends like I'm doing for SD...)

[ramkrishna2910]

Yeah, plan is to move to tagged release. This PR should be in draft 😅

[ghost]

vLLM ROCm Burn Test — All 14 Models on Strix Halo (gfx1151)

Hardware: AMD Ryzen AI MAX+ 395, 128GB unified RAM, Radeon 8060S (gfx1151)
OS: Arch Linux (CachyOS kernel 7.0.0-1-mainline)
Backend: vLLM ROCm via this PR (built from source, test-vllm branch)
Method: 5 runs per model, 2 warmup runs, 200 max tokens, mean ± stddev
Date: 2026-04-14

Results — 12/14 PASSED

Model	Status	Mean tok/s	±StdDev	Min	Max	Load Time
Qwen3-0.6B-vllm	✅ PASS	116.7	±0.2	116.4	116.9	17.0s
Qwen3.5-0.8B-vllm	❌ FAIL	-	-	-	-	0.5s
Qwen3-1.7B-vllm	✅ PASS	25.1	±26.0	2.5	47.7	19.7s
Qwen3.5-2B-vllm	✅ PASS	44.0	±0.0	44.0	44.0	216.2s
Llama-3.2-1B-Instruct-vllm	✅ PASS	110.4	±0.0	110.3	110.5	38.4s
Llama-3.2-3B-Instruct-vllm	✅ PASS	50.5	±0.1	50.3	50.5	42.8s
Qwen3-4B-vllm	✅ PASS	25.4	±0.0	25.4	25.4	21.8s
Qwen3-4B-AWQ-vllm	✅ PASS	42.8	±0.0	42.8	42.8	75.6s
Qwen3.5-4B-vllm	✅ PASS	23.8	±0.0	23.8	23.8	331.7s
Gemma-3-4b-it-vllm	❌ FAIL	-	-	-	-	0.4s
Phi-4-mini-instruct-vllm	✅ PASS	25.1	±0.0	25.1	25.1	102.0s
Qwen3-8B-vllm	✅ PASS	12.3	±0.0	12.3	12.3	24.7s
Qwen3-8B-AWQ-vllm	✅ PASS	22.8	±0.0	22.8	22.8	81.2s
Qwen3.5-9B-vllm	✅ PASS	11.6	±0.0	11.6	11.7	309.3s

Notes

• First load per model size includes Triton JIT compilation (20-350s). Subsequent loads of same architecture hit on-disk cache.
• All measurements are wall-clock (request → response).
• --enforce-eager set by Lemonade's vLLM wrapper (no CUDA graphs).
• Server ran on port 8083 per PR test plan.
• Qwen3-1.7B shows high variance (±26.0) — may warrant investigation.
• AWQ quantized models show ~1.8x speedup over FP16 counterparts (Qwen3-4B: 25.4 → 42.8, Qwen3-8B: 12.3 → 22.8).

Failed Models

• Qwen3.5-0.8B: Immediate failure (0.5s load), likely unsupported config
• Gemma-3-4b-it: Immediate failure (0.4s load), likely architecture not yet supported in vLLM ROCm wrapper

Monster Burn (Queued)

Next up: Qwen2.5-72B-AWQ, Mixtral-8x22B, Qwen3-235B-A22B — testing large model viability on 128GB unified.

---

Tested on bare metal Arch Linux, no containers. Happy to run additional models or reproduce with different parameters. Great work on this PR — the wrapper cleanly solves the Triton HIP "invalid device ordinal" crash we were hitting running vLLM directly.

[superm1]

No. It's not a false negative. There are real problems that are incredibly difficult to debug. We need to flag it.

[superm1]

Please don't change any gfx1151 CWSR detection changes. The problems that can happen are very difficult to debug and will "appear randomly". The fixed kernels are all rolled out in Ubuntu 24.04 6.17 HWE, Ubuntu 24.04 6.14 OEM, Ubuntu 26.04. They're fixed in Arch and Fedora.

If someone is on something unique they need to upgrade.

[ghost]

Test Report — CachyOS / Kernel 7.0 / gfx1151 (Strix Halo)

Tester: @stampby
Hardware: AMD Strix Halo, Radeon 8060S (gfx1151), 128GB unified
OS: Arch Linux (CachyOS), Kernel 7.0.0-1-mainline
Build: ./setup.sh && cmake --build --preset default from test-vllm branch
Server: ./build/lemond --port 8083 per PR test plan
Method: 5 runs per model, 2 warmup discarded, 200 max tokens, mean ± stddev

All 14 Catalog vLLM Models — vLLM ROCm on gfx1151

Model	Status	Mean tok/s	±StdDev	Min	Max	Load Time
Qwen3-0.6B-vllm	PASS	116.7	±0.2	116.4	116.9	17.0s
Qwen3.5-0.8B-vllm	FAIL	-	-	-	-	0.5s
Qwen3-1.7B-vllm	PASS	25.1	±26.0*	2.5	47.7	19.7s
Qwen3.5-2B-vllm	PASS	44.0	±0.0	44.0	44.0	216.2s
Llama-3.2-1B-Instruct-vllm	PASS	110.4	±0.0	110.3	110.5	38.4s
Llama-3.2-3B-Instruct-vllm	PASS	50.5	±0.1	50.3	50.5	42.8s
Qwen3-4B-vllm	PASS	25.4	±0.0	25.4	25.4	21.8s
Qwen3-4B-AWQ-vllm	PASS	42.8	±0.0	42.8	42.8	75.6s
Qwen3.5-4B-vllm	PASS	23.8	±0.0	23.8	23.8	331.7s
Gemma-3-4b-it-vllm	FAIL	-	-	-	-	0.4s
Phi-4-mini-instruct-vllm	PASS	25.1	±0.0	25.1	25.1	102.0s
Qwen3-8B-vllm	PASS	12.3	±0.0	12.3	12.3	24.7s
Qwen3-8B-AWQ-vllm	PASS	22.8	±0.0	22.8	22.8	81.2s
Qwen3.5-9B-vllm	PASS	11.6	±0.0	11.6	11.7	309.3s

12/14 models passed.

Failures

1. Qwen3.5-0.8B-vllm — load failed immediately (0.5s). May be a model config or transformers version issue.
2. Gemma-3-4b-it-vllm — load failed. HuggingFace download error on .gitattributes. Likely needs HF auth token for gated model access.

Notes

• *Qwen3-1.7B had one cold outlier run at 2.5 tok/s (first bench run after JIT), subsequent runs were 47.4-47.7. The ±26.0 stddev reflects that single outlier.
• AWQ models show ~1.7-1.9x speedup over FP16 counterparts (e.g. Qwen3-4B: 25.4 FP16 vs 42.8 AWQ, Qwen3-8B: 12.3 FP16 vs 22.8 AWQ).
• Qwen3.5 models have significantly longer first-load times (216-331s) due to Triton JIT compilation for a new architecture. Subsequent loads from cache are fast.
• All variance is essentially zero (±0.0-0.2) once warmed up. Rock solid.
• Qwen2.5-72B-Instruct-AWQ is currently running as a bonus test (72B dense on 128GB unified memory, ~2.4 tok/s generation). Results will follow.

Environment

Hardware:     AMD Strix Halo, 128GB unified, Radeon 8060S (gfx1151)
OS:           Arch Linux (CachyOS)
Kernel:       7.0.0-1-mainline
lemond:       Built from test-vllm branch
vllm-rocm:    Installed via /v1/install (rocm backend)
Server port:  8083

Happy to retest anything or run additional models. Hardware is available.

[ghost]

vLLM ROCm gfx1151 Results — Strix Halo, 128GB unified

14 models tested, 12 passed. 5 runs, 2 warmup, 200 max tokens.

Model	tok/s	±stddev	Type
Qwen3-0.6B	116.7	±0.2	FP16
Llama-3.2-1B-Instruct	110.4	±0.0	AWQ
Llama-3.2-3B-Instruct	50.5	±0.1	AWQ
Qwen3.5-2B	44.0	±0.0	FP16
Qwen3-4B-AWQ	42.8	±0.0	AWQ
Qwen3-4B	25.4	±0.0	FP16
Phi-4-mini	25.1	±0.0	FP16
Qwen3-8B-AWQ	22.8	±0.0	AWQ
Qwen3-8B	12.3	±0.0	FP16
Qwen3.5-9B	11.6	±0.0	FP16
Qwen2.5-72B-AWQ	2.3	±0.1	AWQ (72B dense)

Full results + comparison with MLX Engine ROCm: https://github.com/stampby/bleeding-edge

[eddierichter-amd]

@ramkrishna2910

I followed the manual smoke-test portion of the vllm test plan against a local Lemonade server, and it worked out of the box without any extra code changes or request tweaking.

Command used:

curl -s -X POST http://localhost:8083/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3-0.6B-vllm","messages":[{"role":"user","content":"Tell me a story"}],"max_tokens":30,"temperature":0}'

Response:

{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "logprobs": null,
      "message": {
        "annotations": null,
        "audio": null,
        "content": "<think>\nOkay, the user asked for a story. I need to come up with something engaging. Let me think about a
simple yet memorable story.",
        "function_call": null,
        "reasoning": null,
        "refusal": null,
        "role": "assistant",
        "tool_calls": []
      },
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "created": 1776357013,
  "id": "chatcmpl-8704fee7a2990b81",
  "kv_transfer_params": null,
  "model": "Qwen3-0.6B-vllm",
  "object": "chat.completion",
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 30,
    "prompt_tokens": 12,
    "prompt_tokens_details": null,
    "total_tokens": 42
  }
}

The request completed successfully and returned a response from Qwen3-0.6B-vllm.

[superm1]

Generally speaking it looks fine now. a few minor nits on things that don't need to change but are being changed.

I do want to eventually get to the point we're not installing another copy of ROCm to make this work though.

[ramkrishna2910]

Generally speaking it looks fine now. a few minor nits on things that don't need to change but are being changed.

I do want to eventually get to the point we're not installing another copy of ROCm to make this work though.

Yes, agreed. We are using the pre packaged whls here. I would suggest sticking to this approach for the experimental release and eventually moving to the common rocm backend.