`AssertionError: CUDA_HOME is not set`

[SuperCowProducts]

yalihupokn@zoqu-endiman:~/Repositories/vM/vllm-vulkan$ uname -a
Linux zoqu-endiman 6.19.14-400.asahi.fc44.aarch64+16k #1 SMP PREEMPT_DYNAMIC Thu May  7 22:13:27 UTC 2026 aarch64 GNU/Linux
yalihupokn@zoqu-endiman:~/Repositories/vM/vllm-vulkan$ 

typescript.zip

[martinhoyer]

Also bumped into it. Looks like this is where it fails:

vllm-vulkan/install.sh

Line 157 in b5db06f

CXXFLAGS="-Wno-parentheses" uv pip install .

You can use the release wheel though (I'm also on F44 Asahi)

[SuperCowProducts]

@martinhoyer thank you for the tip, for some reason after

uv venv -p 3.12
. .venv/bin/activate
wget $(curl https://api.github.com/repos/ericcurtin/vllm-vulkan/releases/latest | jq .assets[0].browser_download_url)
uv pip install vllm_vulkan-0.1.0-cp312-cp312-linux_aarch64.whl

there is no vllm binary I can use to run the app:

(vM) yalihupokn@zoqu-endiman:~/Repositories/vM$ ls .venv/bin/
accelerate                  activate       activate.ps1      httpx            numpy-config   python        torchrun
accelerate-config           activate.bat   activate_this.py  huggingface-cli  proton         python3       tqdm
accelerate-estimate-memory  activate.csh   deactivate.bat    idna             proton-viewer  python3.12    transformers
accelerate-launch           activate.fish  f2py              isympy           pydoc.bat      tiny-agents   typer
accelerate-merge-weights    activate.nu    hf                markdown-it      pygmentize     torchfrtrace
(vM) yalihupokn@zoqu-endiman:~/Repositories/vM$ vllm
bash: vllm: command not found...
(vM) yalihupokn@zoqu-endiman:~/Repositories/vM$ 

Any ideas why this is the case? If I uv pip install vllm won't that completely fall back to CPU?

[SuperCowProducts]

So after installing standard vllm the Vulkan plugin appears to be active, so while I assume there will be some fallback to the CPU there should still be a performance boost:

(vM) [yalihupokn@zoqu-endiman vM]$ vllm
INFO 07-04 19:04:31 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-04 19:04:31 [__init__.py:46] - vulkan -> vllm_vulkan:register
INFO 07-04 19:04:31 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 07-04 19:04:31 [patches.py:134] Applied pure-Python fallback for compute_slot_mapping_kernel (vllm._C not available).
INFO 07-04 19:04:32 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-04 19:04:32 [__init__.py:46] - vulkan -> vllm_vulkan:register
INFO 07-04 19:04:32 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 07-04 19:04:32 [__init__.py:237] Platform plugin vulkan is activated
INFO 07-04 19:04:32 [patches.py:189] Registered gemma4 chat template fallback for base models (e.g. google/gemma-4-E2B) from /home/yalihupokn/Repositories/vM/.venv/lib/python3.12/site-packages/vllm_vulkan/template_gemma4.jinja.
INFO 07-04 19:04:32 [__init__.py:237] Platform plugin vulkan is activated
INFO 07-04 19:04:38 [importing.py:46] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
INFO 07-04 19:04:38 [importing.py:81] Triton not installed or not compatible; certain GPU-related functions will not be available.
usage: vllm [-h] [-v] {chat,complete,serve,launch,bench,collect-env,run-batch} ...

vLLM CLI

positional arguments:
  {chat,complete,serve,launch,bench,collect-env,run-batch}
    chat                Generate chat completions via the running API server.
    complete            Generate text completions based on the given prompt via the running API server.
    serve               Launch a local OpenAI-compatible API server to serve LLM completions via HTTP.
    launch              Launch individual vLLM components.
    bench               vLLM bench subcommand.
    collect-env         Start collecting environment information.
    run-batch           Run batch prompts and write results to file.

options:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit

For full list:            vllm [subcommand] --help=all
For a section:            vllm [subcommand] --help=ModelConfig    (case-insensitive)
For a flag:               vllm [subcommand] --help=max-model-len  (_ or - accepted)
Documentation:            https://docs.vllm.ai
(vM) [yalihupokn@zoqu-endiman vM]$