RTX 5090 + FP4 + Open WebUI via TensorRT-LLM (because VLLM made me cry at 2am)

[rdumasia303]

So… after a late-night slap fight with VLLM on Blackwell and FP4, I did the unthinkable: I got GPT5 to read the docs and tried this - NVIDIA’s own TensorRT-LLM. Turns out the fix was hiding in plain sight (right next to my empty coffee mug).

Repo: https://github.com/rdumasia303/tensorrt-llm_with_open-webui

Why you might care

5090 / Blackwell friendly: Built to run cleanly on RTX 5090 and friends.

FP4 works: Runs FP4 models that can be grumpy in other stacks.

OpenAI-compatible: Drop-in for Open WebUI or anything that speaks /v1.

One compose file: Nothing too magical required.

I haven't got multimodal models working, but

• nvidia/Qwen3-30B-A3B-FP4nvidia/Qwen3-30B-A3B-FP4

Works, and it's fast - so that's me done for tonight.

Apologies if this has been done before - but all I could find were folks saying 'Can it be done?' So I made it.