Repository navigation
[xegpu] Add torch llama3 attention example - #307
Conversation
a3a027a to
c05a9cb
Compare
…ctor Leading parallel dimensions must be zero-padded.
c05a9cb to
240bf06
Compare
| Differences from the synthetic level1-97 SDPA: | ||
| * d_head = 64 (Llama3.2-1B), the per-head reduction dimension, within the | ||
| current <= 64 lowering limit. (Llama3-8B uses d_head = 128, which still | ||
| needs the new Intel-GPU head tiling.) |
There was a problem hiding this comment.
I think you could also reduce the reduction tile size to preserve registers. that way you may not need head tiling.
There was a problem hiding this comment.
Good point, for d_head=128 that might be sufficient.
| * Grouped-query attention: 32 query heads share 8 key/value heads | ||
| (num_heads / num_kv_heads = 4). K/V carry fewer heads than Q and are | ||
| broadcast across the group (enable_gqa=True). | ||
| * Causal masking (is_causal=True), as in autoregressive decoding/prefill. |
There was a problem hiding this comment.
here how is the causal masking handled? is masking applied to QK^t?
There was a problem hiding this comment.
Yes, causal masking adds a strict upper diagonal mask with -inf values (the rest being zeros) to the QK^T results. The subsequent -inf handling ensures that these values do not contribute to the attention.
| ``materialize_in_destination(X, memref.expand_shape(D))``. | ||
|
|
||
| WG-tiled attention writes its result in the GQA-expanded N-D layout, but the | ||
| output memref is the collapsed (N-1)-D layout, so a ``tensor.collapse_shape`` |
There was a problem hiding this comment.
I wonder if there is some existing transform op to handle this. I remember running into similar issues.
There was a problem hiding this comment.
Not that I know of. At the moment I can't really think of a generic way for fixing this upstream other than a specialized bufferization.materialize_in_destination transform like the one added here. The IR pattern essentially is:
func.func @main(..., %arg3: memref<4x32x512x64xbf16>) {
%4 = tensor.empty() : tensor<4x8x4x512x64xbf16>
%5 = scf.forall ... shared_outs(%arg8 = %4) -> (tensor<4x8x4x512x64xbf16>) { ... }
%collapsed = tensor.collapse_shape %5 [[0], [1, 2], [3], [4]] : tensor<4x8x4x512x64xbf16> into
tensor<4x32x512x64xbf16>
bufferization.materialize_in_destination %collapsed in restrict writable %arg3 :
(tensor<4x32x512x64xbf16>, memref<4x32x512x64xbf16>) -> ()
llama-bench_1B_attention.pykernel file which defines a Grouped-Query Attention (GQA) kernel following Llama-3.2-1B model specification.kernel_bench.pyroutine via the new-i/--inputflag.transform_to_flashattentionthereduction_tileargument is now a (zero-padded) vector to allow easier handling of inferred tile sizes.expand_output_destinationtransform op which rewritesmaterialize_in_destination(collapse_shape(X), D)intomaterialize_in_destination(X, memref.expand_shape(D)). This is required to avoid materializing a copy buffer for the layer result.contraction_normalization.