Add heterogeneous pd docs #714
Conversation
Signed-off-by: Daniel Huang <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
There was a problem hiding this comment.
Pull request overview
This PR adds documentation for the heterogeneous PD (Prefill-Decode) disaggregation feature, which allows splitting model execution across CUDA prefill nodes and Gaudi decode nodes. The documentation covers setup requirements, installation procedures, service configuration, and verification steps.
Key Changes:
- Added comprehensive setup guide for CUDA+Gaudi multi-node systems
- Documented installation of NIXL with UCX support
- Provided launch configurations for prefill, decode, and proxy services
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Daniel Huang <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Daniel Huang <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Daniel Huang <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Daniel Huang <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Daniel Huang <[email protected]>
Signed-off-by: Daniel Huang <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Daniel Huang <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Daniel Huang <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
1 participant
Add docs for #711 feature. I'm not sure where the best place to put the docs, the placement is flexible.
This docker is reply on upstream PR merge for
Prefill(CUDA) -> Decode(Gaudi): #vllm-project/vllm#30275
Prefill(Gaudi) -> Decode(Cuda): #vllm-project/vllm#30448