Enable inc dynamic quant for MoE models #688
Conversation
Signed-off-by: mandy-li <[email protected]>
mandy-li
requested review from
adobrzyn,
afierka-intel,
iboiko-habana,
kamil-kaczor,
ksmusz,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski and
xuechendi
as code owners
✅ CI PassedAll checks passed successfully against the following vllm commit: |
| if weight_shape[0] / scale_shape[0] == self.block_size[0] and \ | ||
| weight_shape[1] / scale_shape[1] == self.block_size[1]: # block-wise | ||
| return dequant_block_fp8_weight_naive( | ||
| self.weight, |
There was a problem hiding this comment.
This check feels a bit fragile. Could we infer the block-wise scale by inspecting the quant_method instead?https://github.com/vllm-project/vllm/blob/408cf42f67dbcd50027fcd0f6ba35df83ced9107/vllm/model_executor/layers/fused_moe/layer.py#L1326 cc @xuechendi
Please add UTs and e2e test as well, thx!
There was a problem hiding this comment.
Is this copy from vllm-fork? If so, please paste original link
There was a problem hiding this comment.
No, partially from vllm-hpu-extension . For the channel-wise part to work, we need a bug fix in inc-fork. Please confirm if vllm-gaudi CI uses latest commit from inc-fork or not
There was a problem hiding this comment.
No, CI only uses public habana docker.
3 participants
This PR enables INC dynamic quantization for MoE models by adding dequant channel-wise weight to MoE OP