Skip to content

[xegpu] Add torch llama3 attention example - #307

Merged
tkarna merged 18 commits into
llvm:mainfrom
tkarna:xegpu-torch-llama3-attention
Oct 22, 2026
Merged

tkarna merged 18 commits into
llvm:mainfrom
tkarna:xegpu-torch-llama3-attention

Conversation

@tkarna

@tkarna tkarna commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor
  • Adds a KernelBench-like llama-bench_1B_attention.py kernel file which defines a Grouped-Query Attention (GQA) kernel following Llama-3.2-1B model specification.
  • The kernel is executed with kernel_bench.py routine via the new -i/--input flag.
  • The attention lowering schedules are updated to correctly lower the GQA layer. More IR canonicalization is required to fold 4D/5D expand/collapse_shape ops.
  • In transform_to_flashattention the reduction_tile argument is now a (zero-padded) vector to allow easier handling of inferred tile sizes.
  • Adds expand_output_destination transform op which rewrites materialize_in_destination(collapse_shape(X), D) into materialize_in_destination(X, memref.expand_shape(D)). This is required to avoid materializing a copy buffer for the layer result.
  • Minor fix in contraction_normalization.

@tkarna
tkarna force-pushed the xegpu-torch-llama3-attention branch from a3a027a to c05a9cb Compare October 16, 2026 21:11
@tkarna

tkarna commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

FYI @charithaintc @nbpatel

@tkarna
tkarna force-pushed the xegpu-torch-llama3-attention branch from c05a9cb to 240bf06 Compare October 19, 2026 18:19

@charithaintc charithaintc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

Differences from the synthetic level1-97 SDPA:
* d_head = 64 (Llama3.2-1B), the per-head reduction dimension, within the
current <= 64 lowering limit. (Llama3-8B uses d_head = 128, which still
needs the new Intel-GPU head tiling.)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you could also reduce the reduction tile size to preserve registers. that way you may not need head tiling.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, for d_head=128 that might be sufficient.

* Grouped-query attention: 32 query heads share 8 key/value heads
(num_heads / num_kv_heads = 4). K/V carry fewer heads than Q and are
broadcast across the group (enable_gqa=True).
* Causal masking (is_causal=True), as in autoregressive decoding/prefill.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here how is the causal masking handled? is masking applied to QK^t?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, causal masking adds a strict upper diagonal mask with -inf values (the rest being zeros) to the QK^T results. The subsequent -inf handling ensures that these values do not contribute to the attention.

``materialize_in_destination(X, memref.expand_shape(D))``.

WG-tiled attention writes its result in the GQA-expanded N-D layout, but the
output memref is the collapsed (N-1)-D layout, so a ``tensor.collapse_shape``

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there is some existing transform op to handle this. I remember running into similar issues.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I know of. At the moment I can't really think of a generic way for fixing this upstream other than a specialized bufferization.materialize_in_destination transform like the one added here. The IR pattern essentially is:

func.func @main(..., %arg3: memref<4x32x512x64xbf16>) {
  %4 = tensor.empty() : tensor<4x8x4x512x64xbf16>
  %5 = scf.forall ... shared_outs(%arg8 = %4) -> (tensor<4x8x4x512x64xbf16>) { ... }
  %collapsed = tensor.collapse_shape %5 [[0], [1, 2], [3], [4]] : tensor<4x8x4x512x64xbf16> into
      tensor<4x32x512x64xbf16>
  bufferization.materialize_in_destination %collapsed in restrict writable %arg3 :
      (tensor<4x32x512x64xbf16>, memref<4x32x512x64xbf16>) -> ()

@tkarna
tkarna merged commit 4c48432 into llvm:main Oct 22, 2026
3 checks passed
@tkarna
tkarna deleted the xegpu-torch-llama3-attention branch October 22, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants