Skip to content

Commit 8f4c127

Browse files
[ROCm][Bugfix] Use BF16 MLA prefill for short prompts
Avoid FP8 rounding perturbations on short greedy-decoding requests while preserving the optimized AITER path for long-context prefill. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent beca88e commit 8f4c127

1 file changed

Lines changed: 3 additions & 1 deletion

File tree

vllm/v1/attention/backends/mla/rocm_aiter_mla.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1035,7 +1035,9 @@ def forward_mha(
10351035
prefill_metadata = attn_metadata.prefill
10361036
has_context = prefill_metadata.chunked_context is not None
10371037

1038-
if has_context:
1038+
# FP8 prefill can perturb greedy decoding for short prompts, where its
1039+
# speedup is negligible. Retain it for long, context-free prefill.
1040+
if prefill_metadata.max_query_len < 1024 or has_context:
10391041
return super().forward_mha(
10401042
q,
10411043
kv_c_normed,

0 commit comments

Comments
 (0)