Support for GLM-5 (or 5.1/5.2) on Gaudi 2?

[tylerebowers]

GLM 5.2 is currently supported in vLLM 0.23, the latest here is 0.21. Wondering if support for these models is in the future roadmap?

[adobrzyn]

Hi! We're planning to release a vLLM 0.24.0 compatible version in early July, which moves us well past the current 0.21 base - and since GLM 5.2 landed upstream in vLLM 0.23, the model code will be present once we're on 0.24.0. That said, to set expectations honestly: we won't be specifically validating GLM-5 on Gaudi 2 for that release, so it may work out of the box or may need Gaudi-specific adjustments - please treat it as "will be available but unverified" for now.

[psun888]

thanks a lot, you guys are doing wonderful work. Gaudi 2 is a quite great alternative to H100, very powerful. With this project, AI becomes truly affordable. tonight I'll test GLM 5.2.

[psun888]

Tried GLM 5.2 on vllm-gaudi 0.21, failed on "NotImplementedError: Sparse Attention is not supported on HPU."

[psun888]

Tried with tag v0.24.0rc0 (https://github.com/vllm-project/vllm-gaudi/releases/tag/v0.24.0rc0), not working either.

[azavjalov]

Hello! I was able to run GLM5.2 on Gaudi2, but some optimization needed as it is at <3 tk/s right now, but it does work. The recipe is not trivial and I had to rework the model and make changes across the SW stack.

My config is

• 8× Intel Gaudi 2 (HL-225)
• Gaudi SW 1.24.0 (fw 62.6.2), kernel 6.8, Ubuntu 24.04
• vllm-gaudi v0.21.0 + patches (no luck with v0.24.0rc)
• Model: QuantTrio/GLM-5.2-Int4-Int8Mix (405 GB — W4A16-symmetric experts, W8A16 linears)

[marcuscmy]

Hello @azavjalov if you dont mind I want to ask a little bit about this. I was on the fence about purchasing some Gaudi 2 units for my company's inferencing needs. However, after renting an online instance for some testing with the Gaudi 3, I have found inference performance with the Qwen 3.5 397B-A17B to be quite poor.

Not sure if I have wrong expectations, or I am not using certain optimization techniques to get the most of the HPU. I have found the graph compilations take up quite a bit of time and also create a lot of inconsistencies for real life inferencing use cases.

Appreciate your expertise on the matter.

[azavjalov]

Hello, @marcuscmy, I am in a same position as you, I think - trying to evaluate on how useful Gaudi2 will be for our work running some large LLMs. Ideally something like GLM5.2 with 1M context, shared with 3-5 people for some deep coding/architecture tasks.

I just started looking into it, so it is hard to make any conclusions just yet, but I expect that unless the model is specifically adapted to Gaudi architecture, the performance won't be optimal.

[EvgC]

Hi @marcuscmy can you please provide your configuration during test
" Qwen 3.5 397B-A17B to be quite poor"? We can compare with our last time optimizations for this model.

[marcuscmy]

I tested the Gaudi 3 via Supermicro's Jumpstart program, so unfortunately I do not have access to the instance anymore, however I do have some statistics/notes left over, hope it helps:

OS: Ubuntu 22 LTS (Exact version unknown)
Gaudi Driver: 1.24.0 (Installed via the official script suggested in docs)
vllm-gaudi version: Release v0.19.1
vllm hash: obtained from the release clone as suggested

Run time commands:
#Without expert parallelism
VLLM_GRAPH_RESERVED_MEM=0.2 vllm serve Qwen/Qwen3.5-397B-A17B --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --gpu_memory_utilization 0.8 --block-size 128 --max-num-seqs 128

#With expert parallelism
PT_HPU_ENABLE_LAZY_COLLECTIVES="true" PT_HPU_WEIGHT_SHARING=0 PT_HPUGRAPH_DISABLE_TENSOR_CACHE=1 VLLM_GRAPH_RESERVED_MEM=0.2 vllm serve Qwen/Qwen3.5-397B-A17B --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --gpu_memory_utilization 0.8 --block-size 128 --max-num-seqs 128 --enable-expert-parallel

I will be checking whether I can retrieve the logs I have from LiteLLM for this model tomorrow.

[psun888]

With two 8x Gaudi2 servers, vllm-gaudi v0.21.0 and ChatGPT, I managed to run GLM 5.2 successfully. If anyone need leasing Gaudi2 servers, please let me know. 6x 400G internal connections for Gaidi2, and one 100G for public Internet connection with IPv4 or IPv6. Thanks for all the help. Gaudi2 is truly amazing!

[marcuscmy]

@psun888 Wow! That would be a dream to run! Can you share your benchmarks on its performance?

[psun888]

Yes, I'll get some benchmarks over this weekend and post them here.

[slavap]

@psun888 Any luck with benchmarks? Trying to decide if Gaudi 2 server is worth dealing with.

[azavjalov]

My best result on GLM5.2 with one node and 8x accelerators is 7tk/s output. I think it can do much faster with more debug/wiring, but I need a bit more time on the box which I don't have access to right now.

[psun888]

GLM-5.2-FP8 on Gaudi2: single-node TP8 works, two-node PP2 starts but inference is unusably slow

Summary

I tested zai-org/GLM-5.2-FP8 on one and two servers, each containing 8× Intel Gaudi2 HL-225 96 GB accelerators.

After applying several compatibility patches to vLLM-Gaudi 0.21.0:

• Single-node inference with TP=8 works using both the multiprocessing and Ray executors.
• Two-node Ray deployment with TP=8 and PP=2 can load the model and start the API server.
• The stock HPU V1 execution path does not transfer intermediate tensors between pipeline stages.
• Experimental PP transport patches allowed short requests to complete.
• Sustained two-node decoding was only approximately 0.05–0.1 token/s, making the implementation unusable.

This post documents the successful parts, failure points, and the apparent missing functionality in the HPU worker.

Hardware

Two identical servers:

• 8× Intel Gaudi2 HL-225 96 GB per server

• 16 Gaudi2 devices total

• Ubuntu 24.04

• Direct GaudiNIC links between the servers

• MTU 8166

• HCCL 16-rank all-reduce test passed

• Approximate HCCL test results:

  • Network bandwidth: 247 GB/s
  • Algorithm bandwidth: 132 GB/s

Servers:

server048055: xxx.xxx.48.55
server048056: xxx.xxx.48.56

The two-node HCCL and Gloo communication tests passed before testing vLLM.

Software

Model: zai-org/GLM-5.2-FP8
Model revision: ba978f7d347eaf65d22f1a86833408afdb953541
Model size: approximately 755.6 GB
Safetensor files: 141

vLLM-Gaudi: 0.21.0
vLLM-Gaudi repository commit: fbf22752
SynapseAI/runtime: 1.24.x
Firmware: hl-1.24.0-fw-62.6.2.0
Python: 3.12

The model was copied locally to:

/srv/models/GLM-5.2-FP8

The model files were verified to be identical on both servers.

Compatibility patches required before the model would load

GLM-5.2-FP8 did not run out of the box with vLLM-Gaudi 0.21.0.

The following model compatibility changes were required:

1. Force the Sparse Attention and MLA configuration to use the HPU MLA backend.
2. Bypass the unsupported indexer path.
3. Avoid loading the FP8 indexer wk_weights_proj tensor on a pipeline stage that does not contain the fused destination module.
4. Restrict sampler warmup to the final pipeline stage.
5. Prevent Ray from rewriting HABANA_VISIBLE_MODULES.
6. Explicitly call:

torch.hpu.set_device(local_rank)

These patches were sufficient for single-node execution.

Successful test 1: single node, multiprocessing, TP=8, PP=1

The following configuration successfully loaded the model and generated text:

vllm serve /srv/models/GLM-5.2-FP8 \
  --host 0.0.0.0 \
  --port 8000 \
  --distributed-executor-backend mp \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 1 \
  --dtype bfloat16 \
  --trust-remote-code \
  --gpu-memory-utilization 0.95 \
  --max-model-len 512 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 512 \
  --block-size 128 \
  --disable-custom-all-reduce \
  --served-model-name glm-5.2-fp8

Result:

Model loading: PASS
API startup: PASS
Text generation: PASS

The model used approximately 91 GB of memory per Gaudi2 device.

Successful test 2: single node, Ray, TP=8, PP=1

The same model also worked on one server using the Ray executor:

vllm serve /srv/models/GLM-5.2-FP8 \
  --host 0.0.0.0 \
  --port 8000 \
  --distributed-executor-backend ray \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 1 \
  --dtype bfloat16 \
  --trust-remote-code \
  --gpu-memory-utilization 0.95 \
  --max-model-len 512 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 512 \
  --block-size 128 \
  --disable-custom-all-reduce \
  --served-model-name glm-5.2-fp8

Result:

Ray cluster initialization: PASS
Model loading: PASS
API startup: PASS
Text generation: PASS

Example generated response:

GLM Ray single node working

One separate formatting issue was observed: the model’s internal thinking text was returned inside the normal content field before </think>. This does not appear to be related to Ray or HPU execution.

Two-node configuration

The intended two-node setup was:

Tensor parallel size: 8
Pipeline parallel size: 2
Total devices: 16
Pipeline stage 0: server048055
Pipeline stage 1: server048056

Launch options included:

--distributed-executor-backend ray
--tensor-parallel-size 8
--pipeline-parallel-size 2
--dtype bfloat16
--trust-remote-code
--enforce-eager
--no-async-scheduling
--gpu-memory-utilization 0.88
--max-model-len 512
--max-num-seqs 1
--max-num-batched-tokens 512
--block-size 128
--disable-custom-all-reduce

--enforce-eager was initially used to bypass pipeline-stage warmup failures and test actual runtime execution.

Successful part of the two-node test

The two-node Ray cluster successfully reported:

2 active nodes
16 HPU resources

The model loaded across both pipeline stages, and the API server started successfully:

Supported tasks: ['generate']
Starting vLLM server on http://0.0.0.0:8000
Application startup complete

Therefore, the following components worked:

Ray node discovery: PASS
Two-node process placement: PASS
TP=8 groups on each pipeline stage: PASS
Model partition loading: PASS
KV cache initialization: PASS
API startup: PASS

First two-node inference failure: missing pipeline activation transfer

The first actual inference request failed on pipeline stage 1 with:

assert intermediate_tensors is not None

On pipeline stage 0, another observed failure was:

AttributeError: 'IntermediateTensors' object has no attribute 'view'

The apparent cause is that the Gaudi HPU V1 worker calls the HPU model runner without implementing the pipeline receive/send logic present in the upstream GPU worker.

The upstream GPU worker performs logic similar to:

if forward_pass and not get_pp_group().is_first_rank:
    tensor_dict = get_pp_group().recv_tensor_dict()
    intermediate_tensors = IntermediateTensors(tensor_dict)

output = model_runner.execute_model(
    scheduler_output,
    intermediate_tensors,
)

if isinstance(output, IntermediateTensors):
    get_pp_group().send_tensor_dict(output.tensors)

The tested Gaudi worker did not appear to provide the equivalent runtime pipeline transport.

Experimental pipeline transport patch

I added an experimental HPU PP transport implementation that:

1. Receives intermediate tensors on non-first PP stages.
2. Passes them to the model forward call.
3. Returns IntermediateTensors from non-final PP stages.
4. Sends the tensor dictionary to the next pipeline stage.
5. Uses blocking transfers initially for correctness testing.

This moved execution past the original intermediate_tensors is not None assertion.

Second failure: async scheduler token propagation

With asynchronous scheduling enabled, the next scheduler iteration failed on PP0:

IndexError: list index out of range

The failure occurred at:

new_token_ids = req_data.new_token_ids[i]

The scheduler contained a request in:

req_data.req_ids

but req_data.new_token_ids was empty.

This appears to happen because vLLM omits the scheduler token payload for PP when asynchronous scheduling is enabled, while the Gaudi runner expects the token list to be present on non-final pipeline stages.

Disabling asynchronous scheduling with:

--no-async-scheduling

moved execution past this error.

Third failure: decode-input tuple handling

The first experimental implementation also incorrectly treated the result of _prepare_inputs() as the decode data object itself.

The Gaudi runner actually returns:

decode_data, dummy_decode_input_data_across_dp = decode_input_data

Correcting this allowed execution to enter the decode path.

Two-node short-request result

After adding pipeline transport, disabling async scheduling, and fixing the decode-data unpacking, a very short request completed with HTTP 200.

This proves that the experimental path could perform:

PP0 prompt forward: PASS
PP0 to PP1 prompt activation transfer: PASS
PP1 final-stage forward and sampling: PASS
Scheduler token return in synchronous mode: PASS
First PP decode iteration: PASS
Short HTTP request completion: PASS

Final result: decoding is unusably slow

Although short requests could complete, sustained decoding was extremely slow.

Observed server statistics repeatedly alternated between:

Avg generation throughput: 0.0 tokens/s
Avg generation throughput: 0.1 tokens/s

A 16-token request remained active for several minutes.

Approximate effective throughput:

0.05–0.1 generated token/s

This is not usable for inference serving.

The likely performance limitations of the experimental implementation are:

• --enforce-eager
• Blocking send_tensor_dict() and recv_tensor_dict()
• Synchronous scheduling
• No overlap between communication and computation
• No proper HPU-native nonblocking PP activation path
• No HPU graph or compiled execution
• Metadata serialization on every pipeline transfer
• Possible synchronization from repeated mark_step() calls
• No TP-aware sliced activation transfer or all-gather optimization

The repeated warning below was observed during PP transfers:

UserWarning: The given buffer is not writable
object_tensor = torch.frombuffer(pickle.dumps(obj), dtype=torch.uint8)

This warning itself was not fatal, but it indicates that Python object metadata serialization was being performed during every tensor-dictionary transfer.

Current status

Configuration	Result
Single node, MP, TP=8, PP=1	Working
Single node, Ray, TP=8, PP=1	Working
Two nodes, Ray, TP=8, PP=2, unmodified runtime	API starts, first request fails
Two nodes with experimental blocking PP transfer	Short request completes
Two nodes with sustained decoding	Approximately 0.05–0.1 token/s; unusable

Questions for the maintainers

1. Is pipeline parallelism with the HPU V1 model runner currently expected to work in vLLM-Gaudi 0.21.0?

2. Is there a newer branch or release containing HPU equivalents of the GPU worker’s:

irecv_tensor_dict()
isend_tensor_dict()
AsyncIntermediateTensors

pipeline execution path?

3. Should async scheduling automatically be disabled for HPU pipeline parallelism, or should the HPU runner support the direct sampled-token propagation path?

4. Does the current HPU runner support passing IntermediateTensors into model forward calls for non-first pipeline stages?

5. Is there a recommended vLLM-Gaudi version for running GLM-5.2-FP8 across two 8× Gaudi2 servers?

6. Are there known patches for pipeline-stage warmup, graph capture, or compiled execution with PP greater than 1?

Conclusion

GLM-5.2-FP8 can run successfully on one 8× Gaudi2 server with TP=8 after several model compatibility patches.

Two-node model loading and API startup also work. However, the tested vLLM-Gaudi 0.21.0 HPU V1 execution path appears to lack a complete native pipeline activation transport implementation.

A custom blocking implementation can make short PP2 requests complete, but generation performance falls to approximately 0.05–0.1 token/s. Native nonblocking HPU pipeline communication and correct scheduler integration appear necessary before two-node GLM-5.2-FP8 inference is practical.