Skip to content

[Linalg][Transform] An Approach for Tiling and Fusing Dependent Reductions in Linalg - #274

Closed
charithaintc wants to merge 16 commits into
llvm:mainfrom
charithaintc:linalg_reduction_fusion_impl
Closed

charithaintc wants to merge 16 commits into
llvm:mainfrom
charithaintc:linalg_reduction_fusion_impl

Conversation

@charithaintc

@charithaintc charithaintc commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

This is the lighthouse-based implementation of the [Linalg][RFC] An Approach for Tiling and Fusing Dependent Reductions in Linalg.

The proposed structured.fuse_dependent_reduction_op is implemented as lighthouse transform extension. Major difference in the implementation include re-implementing IR utilities that are not available in python bindings (like the scf::tileAndFuseConsumer).

Fused attention example and nanoGPT examples are modified to use the new fusion extension. PR also provides isolated tests for the transform extension inside test/transform/test_fuse_dependant_reduction_ops.py. Implementation can be extended to other intricate usecases as the needs evolve.

Assisted by: Claude

…d Fusing Dependent Reductions in Linalg"

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@charithaintc

Copy link
Copy Markdown
Contributor Author

cc @tkarna

Comment thread examples/xegpu/nanoGPT_payload.py Outdated
Comment on lines +245 to +246
# A `linalg.softmax` would not do: its decomposition normalizes *before* the
# contraction, leaving @V reading the normalized P and breaking the chain.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this new input IR form now a hard requirement? I.e., linalg.softmax (or it's decomposed version) is no longer supported? I would rather try to make the lowering schedules more generic so that they can support different variants. E.g. #272 generalized the fused_attention schedule so that it works for the KernelBench level1-97 case too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a fix for this.

idea is to sink the normalizing divide over the contraction if its possible to prove that normalizing divide is invariant on the reduction dim of the contraction. So contract(A/N, B) -> contract(A, B)/N). worth exploring if this change can be upstream-able.

return ops[0].opview if isinstance(ops[0], ir.Operation) else ops[0]


class FuseDependantReductionOpsOp(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo dependant -> dependent
also elsewhere in this PR

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

@@ -0,0 +1,195 @@
"""Fusion of same-rank elementwise `linalg.generic` chains.

@tkarna tkarna Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this kind of elementwise-only fusion could be achieved by chaining existing ops. For example the reduction_schedule uses trace_producers, filter_elementwise, and structured_fuse_into_containing_op to fuse in all elementwise producers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point. but in my understanding structured_fuse_into_containing_op only works if the consumer is tiled first. do you have an example on it working at pure generic op (untiled) level.

for the reduction fusion to work we need the same rank elementwise ops fused before the tiling. this is an implementation requirement for now. reduction fusion depends on some pre-analysis of the elementwise part that are sandswitched between R1 and R2. fusion makes it more canonical and easier to analyze (have the R1 -> E -> R2 form).

I can think several approached to deal with this,

  1. add more customization to the upstream elementwise fusion pass so that it can, constrain fusion to elementwise ops only (reduction ops are unchanged). avoid fusion across different ranked ops (to help vectorizer). I believe these can be options to the existing pass to cater this use case.

  2. improve fusion to support the full elementwise fused form (R1 -> R2). I would like to avoid this but this is also possible. only reason I am against this is fusing elementwise naively makes the the vectorizer's life harder. it adds redundant broadcasts to match the ranks. This then require some fixes at vector level (which needs more improvement in upstream).

I would like to try option 1 and see what the community thinks. please share your thinking as well on this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, my thought was that we can implement such fusion logic without implementing one monolithic pass-like transform like this - in the schedule it is applied to the whole func. E.g., we can find a leaf elementwise op, tile it, then trace all the immediate elementwise producers, and fuse them in. The benefit is that this is more transform-dialect-like - we can choose which op chain is fused and which not.

We can return to this later too, I feel like we need to improve the fusion logic in general and should not use the monolithic linalg-fuse-elementwise-ops pass because, depending on the application, it can indeed fuse too much.

@tkarna tkarna Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding option 1, the upstream linalg-fuse-elementwise-ops pass has a comment that suggest deprecating the entire pass as it is too monolithic. I haven't checked whether those individual fusion patterns are available in transform dialect, and whether they would offer the granularity we need - but that would be the preferred solution.

https://github.com/llvm/llvm-project/blob/b2b38d4b335293edca03764dd2bcfa314a880933/mlir/lib/Dialect/Linalg/Transforms/ElementwiseOpFusion.cpp#L2791

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. I am aware of this comment. I think this pass was NOT supposed to be used in a production pipeline, the fusion is so naive. We need to expose its patterns in some meaningful way instead.

Comment on lines +61 to +65
The softmax is emitted in the flash-attention form -- ``max``, ``exp``, row
``sum`` and the ``P@V`` contraction as separate ops, with the normalizing
divide *after* the contraction -- rather than as a `linalg.softmax`. See
step 4 for why.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that the fused_attention_schedule is generalized to handle normal softmax IR, I think we can revert these changes to the attention payload gen routine. Starting with a linalg.softmax is a better test for the schedule than this custom IR form.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as discussed linalg.softmax has some issues that needs to be addressed before this transition. we can do this change in a separate PR.

Comment thread examples/xegpu/nanoGPT_payload.py Resolved
@charithaintc

charithaintc commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor Author

all comments are addressed. cc @tkarna

  • add a transform ext to move normalize div over the contraction.
  • add a transform ext to fuse same rank elementwise ops as opposed to fusing everything.
  • change attention payloads to match with KB attention style.

Comment thread examples/xegpu/nanoGPT_payload.py Resolved
narrow set, see there for what is in it and why. Ops that already carry a
non-default flag set keep it, so an explicitly-chosen weaker one is never
widened.
2. Rewrite `exp(a) / exp(b)` into `exp(a - b)`, trading a transcendental and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This transform seems to apply two unrelated transform. These should be implemented as two separate transforms.

# `exp(x - m)` as a separate `sub` and `exp`, and the fusion needs the whole
# term in a single op. This runs *before* the pass below so the term is formed
# by the narrow rewrite, which cannot fuse across a reduction.
func = transform_ext.fuse_same_rank_elementwise_chains(func)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed before, I feel like this transform op is too large monolithic pass-like op to address this issue. I think we can handle this elementwise fusion with the existing transform ops upstream and in Lighthouse.

# divide happens once per output element rather than once per (row, key) element
# and the chain takes the deferred-divide (flash) shape the reduction fusion
# folds into one loop.
func = transform_ext.sink_normalization_past_contraction(func)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a useful and necessary transform. Maybe we could start the merging by opening a new PR for this op only?

I wonder whether we need to apply this on the whole func, or if we can narrow the applicability more by passing in just some relevant op handle?

@tkarna tkarna left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After discussing this in our team we came to the conclusion that this indeed is quite a large PR and it would be easier to split it up to multiple PRs to help the review. Most of this looks solid but some ops could be redesigned and it's easier to do in smaller pieces.

@charithaintc

Copy link
Copy Markdown
Contributor Author

After discussing this in our team we came to the conclusion that this indeed is quite a large PR and it would be easier to split it up to multiple PRs to help the review. Most of this looks solid but some ops could be redesigned and it's easier to do in smaller pieces.

Hi @tkarna , Thanks for the feedback.

Here is the current plan for landing this PR.

  1. Transform extension for sinking a reduction-dim invariant scaling op over a contraction.
  2. Transform extension for fusing a given elementwise prod-consumer pair
  3. structured.fuse_dependent_reduction_op implementation:
    1. legality check + factorizability check for R1 and R2
    2. transformation logic
    3. tests
  1. can be a single PR or multiple PRs. please let me know if you have other suggestions.

for the time being I am closing this PR.

cc: @Garra1980

tkarna pushed a commit that referenced this pull request Sep 29, 2026
…ise fusion. (#286)

Current MLIR `linalg-elementwise-fusion` pass fuse elementwise ops
unconditionally. This can result in sub-optimal vectorization specially
when lower rank elementwise ops gets fused with higher rank ones. In
that case, the extra dims get broadcasted resulting in unnecessary
`vector.broadcast` ops. In any case, its useful to control elementwise
fusion in a more fine-grained manner.

This PR introduces `fuse_elementwise_op(prod, consumer)` which attempts
to fuse a given `prod` into a `consumer`. Original `prod` is not erased
by the op itself (cleanup is callers responsibility). The legality and
transformation logic follows `linalg-elementwise-fusion`'s
implementation itself.

This is part of landing #274 

**Note:** It maybe __easier__ to just introduce this op directly into
upstream rather than here considering the complexity of the python code.
Or at least provide some options/customizations to the
`linalg-elementwise-fusion` pass itself to circumvent the vectorization
issues.

Assisted by: Codex
tkarna pushed a commit that referenced this pull request Oct 7, 2026
…ons. (#295)

Given dependent reduction sequence in the form of `R1 -> E -> R2` where,
* R1 : is a tiled reduction loop.
* E : elementwise op that consumes at least one of R1's reduction result
* R2: consumes E and reduce it along same axis as R1.

This PR defines the legality conditions that must be met to fuse E and
R2 into R1 tiled loop. This is a crucial transformation in kernels like
attention and softmax.

Major conditions that must be satisfied include,
1. R1 and R2 sharing same reduction axis (innermost for now)
2. E consumes R1 in a broadcast way along reduction dim.
3. R1 use in E is `multiplicatively separable` (more details in
comments)

This PR is part of landing #274 

Assisted by: Codex, Claude
tkarna pushed a commit that referenced this pull request Oct 9, 2026
…dent reductions. (#297)

This PR introduces a new transform op `fuse_dependent_reduction_ops`
that takes tiled reduction and chain of elementwise followed by another
reduction and, fuse them into a single tiled online reduction loop.

Main motivation is is to capture `R1 -> E -> R2` linalg op sequence
where,
- `R1` is a generic tiled reduction with at least one result.
- `E` is an elementwise generic that consumes at least one output of
`R1` and uses it broadcasted over the same reduction axis as `R1` (This
recovers the rank of `R1`'s input)
- `R2` is a generic add reduction that uses `E`'s single output and
produce single output.

And fuse them into a single tiled loop. This type of fusion is not
always doable and legality is checked before applying the fusion.

PR does not integrate the op with any schedule yet and will be done in a
follow-up PR.

This is part of landing #274 

Assisted by: Claude, Codex
tkarna pushed a commit that referenced this pull request Oct 12, 2026
… dependent reduction fusion. (#298)

This PR updates xegpu schedules with `fuse_dependent_reduction_ops` to
fuse attention-like workloads instead of relying on manual
`replace_with_fused_attention` op.

Note that `apply_attention_tiling` also uses the new approach.

This is a part of landing #274 

Assisted by: Claude, Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants