Repository navigation
[xegpu] Add torch llama3 attention example #307
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
c0fce3b
5dcd7e2
19d4ae8
b45c8e5
3905608
0ba4306
57b5411
88a3222
975e6e8
58cc928
1583fb0
b7f77a0
2cfad21
d1c05ea
8fb68ae
240bf06
a85686e
46ca0ee
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| import torch | ||
| import torch.nn as nn | ||
|
|
||
|
|
||
| class Model(nn.Module): | ||
| """ | ||
| Llama3.2-1B scaled dot-product attention (GQA + causal), the realistic | ||
| counterpart to KernelBench level1-97. | ||
|
|
||
| Differences from the synthetic level1-97 SDPA: | ||
| * d_head = 64 (Llama3.2-1B), the per-head reduction dimension, within the | ||
| current <= 64 lowering limit. (Llama3-8B uses d_head = 128, which still | ||
| needs the new Intel-GPU head tiling.) | ||
| * Grouped-query attention: 32 query heads share 8 key/value heads | ||
| (num_heads / num_kv_heads = 4). K/V carry fewer heads than Q and are | ||
| broadcast across the group (enable_gqa=True). | ||
| * Causal masking (is_causal=True), as in autoregressive decoding/prefill. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. here how is the causal masking handled? is masking applied to QK^t?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, causal masking adds a strict upper diagonal mask with -inf values (the rest being zeros) to the QK^T results. The subsequent -inf handling ensures that these values do not contribute to the attention. |
||
|
|
||
| RoPE is intentionally NOT applied here; the rotary embedding is a separate | ||
| stepping-stone (llama-bench_4) because rotate_half is a slice+concat shuffle | ||
| on the innermost (head) dimension rather than a plain matmul/elementwise op. | ||
| Q/K/V are taken as already-projected inputs, matching level1-97. | ||
| """ | ||
|
|
||
| def __init__(self): | ||
| super().__init__() | ||
|
|
||
| def forward( | ||
| self, Q: torch.Tensor, K: torch.Tensor, V: torch.Tensor | ||
| ) -> torch.Tensor: | ||
| # Q: (batch, num_heads, n_ctx, d_head); K/V: (batch, num_kv_heads, n_ctx, d_head) | ||
| causal = False # NOTE handle causal masking in the lowering | ||
| return torch.nn.functional.scaled_dot_product_attention( | ||
| Q, K, V, is_causal=causal, enable_gqa=True | ||
| ) | ||
|
|
||
|
|
||
| # Llama3.2-1B attention config. | ||
| batch_size = 4 | ||
| n_ctx = 512 | ||
| num_heads = 32 # query heads | ||
| num_kv_heads = 8 # grouped key/value heads (GQA group size = 4) | ||
| d_head = 64 # per-head reduction dim; within the current <= 64 cap | ||
|
|
||
|
|
||
| def get_inputs(): | ||
| Q = torch.rand(batch_size, num_heads, n_ctx, d_head) | ||
| K = torch.rand(batch_size, num_kv_heads, n_ctx, d_head) | ||
| V = torch.rand(batch_size, num_kv_heads, n_ctx, d_head) | ||
| return [Q, K, V] | ||
|
|
||
|
|
||
| def get_init_inputs(): | ||
| return [] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| from mlir import ir | ||
| from mlir.dialects import ext, transform, memref, bufferization | ||
| from mlir.dialects.transform import DiagnosedSilenceableFailure | ||
|
|
||
| from lighthouse.dialects.transform.transform_ext import TransformExtensionDialect | ||
|
|
||
|
|
||
| class ExpandOutputDestinationOp( | ||
| TransformExtensionDialect.Operation, name="expand_output_destination" | ||
| ): | ||
| """Remove a reshape from a result's write path so it bufferizes in place. | ||
|
|
||
| Rewrites ``materialize_in_destination(collapse_shape(X), D)`` into | ||
| ``materialize_in_destination(X, memref.expand_shape(D))``. | ||
|
|
||
| WG-tiled attention writes its result in the GQA-expanded N-D layout, but the | ||
| output memref is the collapsed (N-1)-D layout, so a ``tensor.collapse_shape`` | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wonder if there is some existing transform op to handle this. I remember running into similar issues.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not that I know of. At the moment I can't really think of a generic way for fixing this upstream other than a specialized func.func @main(..., %arg3: memref<4x32x512x64xbf16>) {
%4 = tensor.empty() : tensor<4x8x4x512x64xbf16>
%5 = scf.forall ... shared_outs(%arg8 = %4) -> (tensor<4x8x4x512x64xbf16>) { ... }
%collapsed = tensor.collapse_shape %5 [[0], [1, 2], [3], [4]] : tensor<4x8x4x512x64xbf16> into
tensor<4x32x512x64xbf16>
bufferization.materialize_in_destination %collapsed in restrict writable %arg3 :
(tensor<4x32x512x64xbf16>, memref<4x32x512x64xbf16>) -> () |
||
| sits between the loop result and the destination. That reshape makes the loop | ||
| result and the destination differ in shape, so empty-tensor elimination cannot | ||
| fold the loop's output into the destination and bufferization materializes a | ||
| separate buffer and copies it out. Reshaping the destination up to the loop's | ||
| layout removes the reshape from the write path, so elimination fires and the | ||
| loop writes the output in place. | ||
|
|
||
| Only rewrites ``materialize_in_destination`` ops whose destination is a memref | ||
| and whose source is a single ``tensor.collapse_shape``; a no-op otherwise. | ||
| """ | ||
|
|
||
| target: ext.Operand[transform.AnyOpType] | ||
| result: ext.Result[transform.AnyOpType[()]] = ext.infer_result() | ||
|
|
||
| @classmethod | ||
| def attach_interface_impls(cls, ctx=None): | ||
| cls.TransformOpInterfaceModel.attach(cls.OPERATION_NAME, context=ctx) | ||
| cls.MemoryEffectsOpInterfaceModel.attach(cls.OPERATION_NAME, context=ctx) | ||
|
|
||
| class TransformOpInterfaceModel(transform.TransformOpInterface): | ||
| @staticmethod | ||
| def apply(op, _rewriter, results, state) -> DiagnosedSilenceableFailure: | ||
| targets = list(state.get_payload_ops(op.target)) | ||
| for func in targets: | ||
| _rewrite(func) | ||
| results.set_ops(op.result, targets) | ||
| return DiagnosedSilenceableFailure.Success | ||
|
|
||
| @staticmethod | ||
| def allow_repeated_handle_operands(_op) -> bool: | ||
| return False | ||
|
|
||
| class MemoryEffectsOpInterfaceModel(ir.MemoryEffectsOpInterface): | ||
| @staticmethod | ||
| def get_effects(op): | ||
| return ( | ||
| transform.only_reads_handle(op.op_operands) | ||
| + transform.produces_handle(op.results) | ||
| + transform.modifies_payload() | ||
| ) | ||
|
|
||
|
|
||
| def _reassociation(collapse: ir.OpView) -> list[list[int]]: | ||
| return [ | ||
| [ir.IntegerAttr(idx).value for idx in ir.ArrayAttr(group)] | ||
| for group in ir.ArrayAttr(collapse.attributes["reassociation"]) | ||
| ] | ||
|
|
||
|
|
||
| def _rewrite(func: ir.OpView) -> None: | ||
| materializes = [ | ||
| o | ||
| for o in func.regions[0].blocks[0].operations | ||
| if o.operation.name == "bufferization.materialize_in_destination" | ||
| ] | ||
| for mat in materializes: | ||
| source, dest = mat.operands[0], mat.operands[1] | ||
| if not isinstance(dest.type, ir.MemRefType): | ||
| continue | ||
| collapse = source.owner | ||
| if collapse.operation.name != "tensor.collapse_shape": | ||
| continue | ||
| expanded = collapse.operands[0] | ||
| expanded_type = ir.RankedTensorType(expanded.type) | ||
| with ir.InsertionPoint(mat): | ||
| dest_type = ir.MemRefType.get( | ||
| expanded_type.shape, expanded_type.element_type | ||
| ) | ||
| dest_expanded = memref.expand_shape( | ||
| dest_type, | ||
| dest, | ||
| _reassociation(collapse), | ||
| [], | ||
| static_output_shape=list(expanded_type.shape), | ||
| ) | ||
| bufferization.materialize_in_destination( | ||
| None, expanded, dest_expanded, restrict=True, writable=True | ||
| ) | ||
| mat.operation.erase() | ||
|
|
||
|
|
||
| def expand_output_destination( | ||
| target: ir.Value[transform.AnyOpType], | ||
| ) -> ir.Value[transform.AnyOpType]: | ||
| """Rewrite result ``materialize_in_destination`` to write in place.""" | ||
| return ExpandOutputDestinationOp(target=target).result | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you could also reduce the reduction tile size to preserve registers. that way you may not need head tiling.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, for d_head=128 that might be sufficient.