Repository navigation
[Linalg][Transform] An Approach for Tiling and Fusing Dependent Reductions in Linalg - #274
charithaintc wants to merge 16 commits into
Conversation
…d Fusing Dependent Reductions in Linalg" Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
cc @tkarna |
| # A `linalg.softmax` would not do: its decomposition normalizes *before* the | ||
| # contraction, leaving @V reading the normalized P and breaking the chain. |
There was a problem hiding this comment.
Is this new input IR form now a hard requirement? I.e., linalg.softmax (or it's decomposed version) is no longer supported? I would rather try to make the lowering schedules more generic so that they can support different variants. E.g. #272 generalized the fused_attention schedule so that it works for the KernelBench level1-97 case too.
There was a problem hiding this comment.
I added a fix for this.
idea is to sink the normalizing divide over the contraction if its possible to prove that normalizing divide is invariant on the reduction dim of the contraction. So contract(A/N, B) -> contract(A, B)/N). worth exploring if this change can be upstream-able.
| return ops[0].opview if isinstance(ops[0], ir.Operation) else ops[0] | ||
|
|
||
|
|
||
| class FuseDependantReductionOpsOp( |
There was a problem hiding this comment.
typo dependant -> dependent
also elsewhere in this PR
| @@ -0,0 +1,195 @@ | |||
| """Fusion of same-rank elementwise `linalg.generic` chains. | |||
There was a problem hiding this comment.
I feel like this kind of elementwise-only fusion could be achieved by chaining existing ops. For example the reduction_schedule uses trace_producers, filter_elementwise, and structured_fuse_into_containing_op to fuse in all elementwise producers.
There was a problem hiding this comment.
good point. but in my understanding structured_fuse_into_containing_op only works if the consumer is tiled first. do you have an example on it working at pure generic op (untiled) level.
for the reduction fusion to work we need the same rank elementwise ops fused before the tiling. this is an implementation requirement for now. reduction fusion depends on some pre-analysis of the elementwise part that are sandswitched between R1 and R2. fusion makes it more canonical and easier to analyze (have the R1 -> E -> R2 form).
I can think several approached to deal with this,
-
add more customization to the upstream elementwise fusion pass so that it can, constrain fusion to elementwise ops only (reduction ops are unchanged). avoid fusion across different ranked ops (to help vectorizer). I believe these can be options to the existing pass to cater this use case.
-
improve fusion to support the full elementwise fused form (R1 -> R2). I would like to avoid this but this is also possible. only reason I am against this is fusing elementwise naively makes the the vectorizer's life harder. it adds redundant broadcasts to match the ranks. This then require some fixes at vector level (which needs more improvement in upstream).
I would like to try option 1 and see what the community thinks. please share your thinking as well on this.
There was a problem hiding this comment.
Yeah, my thought was that we can implement such fusion logic without implementing one monolithic pass-like transform like this - in the schedule it is applied to the whole func. E.g., we can find a leaf elementwise op, tile it, then trace all the immediate elementwise producers, and fuse them in. The benefit is that this is more transform-dialect-like - we can choose which op chain is fused and which not.
We can return to this later too, I feel like we need to improve the fusion logic in general and should not use the monolithic linalg-fuse-elementwise-ops pass because, depending on the application, it can indeed fuse too much.
There was a problem hiding this comment.
Regarding option 1, the upstream linalg-fuse-elementwise-ops pass has a comment that suggest deprecating the entire pass as it is too monolithic. I haven't checked whether those individual fusion patterns are available in transform dialect, and whether they would offer the granularity we need - but that would be the preferred solution.
There was a problem hiding this comment.
yes. I am aware of this comment. I think this pass was NOT supposed to be used in a production pipeline, the fusion is so naive. We need to expose its patterns in some meaningful way instead.
| The softmax is emitted in the flash-attention form -- ``max``, ``exp``, row | ||
| ``sum`` and the ``P@V`` contraction as separate ops, with the normalizing | ||
| divide *after* the contraction -- rather than as a `linalg.softmax`. See | ||
| step 4 for why. | ||
|
|
There was a problem hiding this comment.
Now that the fused_attention_schedule is generalized to handle normal softmax IR, I think we can revert these changes to the attention payload gen routine. Starting with a linalg.softmax is a better test for the schedule than this custom IR form.
There was a problem hiding this comment.
as discussed linalg.softmax has some issues that needs to be addressed before this transition. we can do this change in a separate PR.
|
all comments are addressed. cc @tkarna
|
| narrow set, see there for what is in it and why. Ops that already carry a | ||
| non-default flag set keep it, so an explicitly-chosen weaker one is never | ||
| widened. | ||
| 2. Rewrite `exp(a) / exp(b)` into `exp(a - b)`, trading a transcendental and |
There was a problem hiding this comment.
This transform seems to apply two unrelated transform. These should be implemented as two separate transforms.
| # `exp(x - m)` as a separate `sub` and `exp`, and the fusion needs the whole | ||
| # term in a single op. This runs *before* the pass below so the term is formed | ||
| # by the narrow rewrite, which cannot fuse across a reduction. | ||
| func = transform_ext.fuse_same_rank_elementwise_chains(func) |
There was a problem hiding this comment.
As discussed before, I feel like this transform op is too large monolithic pass-like op to address this issue. I think we can handle this elementwise fusion with the existing transform ops upstream and in Lighthouse.
| # divide happens once per output element rather than once per (row, key) element | ||
| # and the chain takes the deferred-divide (flash) shape the reduction fusion | ||
| # folds into one loop. | ||
| func = transform_ext.sink_normalization_past_contraction(func) |
There was a problem hiding this comment.
This seems like a useful and necessary transform. Maybe we could start the merging by opening a new PR for this op only?
I wonder whether we need to apply this on the whole func, or if we can narrow the applicability more by passing in just some relevant op handle?
tkarna
left a comment
There was a problem hiding this comment.
After discussing this in our team we came to the conclusion that this indeed is quite a large PR and it would be easier to split it up to multiple PRs to help the review. Most of this looks solid but some ops could be redesigned and it's easier to do in smaller pieces.
Hi @tkarna , Thanks for the feedback. Here is the current plan for landing this PR.
for the time being I am closing this PR. cc: @Garra1980 |
…ise fusion. (#286) Current MLIR `linalg-elementwise-fusion` pass fuse elementwise ops unconditionally. This can result in sub-optimal vectorization specially when lower rank elementwise ops gets fused with higher rank ones. In that case, the extra dims get broadcasted resulting in unnecessary `vector.broadcast` ops. In any case, its useful to control elementwise fusion in a more fine-grained manner. This PR introduces `fuse_elementwise_op(prod, consumer)` which attempts to fuse a given `prod` into a `consumer`. Original `prod` is not erased by the op itself (cleanup is callers responsibility). The legality and transformation logic follows `linalg-elementwise-fusion`'s implementation itself. This is part of landing #274 **Note:** It maybe __easier__ to just introduce this op directly into upstream rather than here considering the complexity of the python code. Or at least provide some options/customizations to the `linalg-elementwise-fusion` pass itself to circumvent the vectorization issues. Assisted by: Codex
…ons. (#295) Given dependent reduction sequence in the form of `R1 -> E -> R2` where, * R1 : is a tiled reduction loop. * E : elementwise op that consumes at least one of R1's reduction result * R2: consumes E and reduce it along same axis as R1. This PR defines the legality conditions that must be met to fuse E and R2 into R1 tiled loop. This is a crucial transformation in kernels like attention and softmax. Major conditions that must be satisfied include, 1. R1 and R2 sharing same reduction axis (innermost for now) 2. E consumes R1 in a broadcast way along reduction dim. 3. R1 use in E is `multiplicatively separable` (more details in comments) This PR is part of landing #274 Assisted by: Codex, Claude
…dent reductions. (#297) This PR introduces a new transform op `fuse_dependent_reduction_ops` that takes tiled reduction and chain of elementwise followed by another reduction and, fuse them into a single tiled online reduction loop. Main motivation is is to capture `R1 -> E -> R2` linalg op sequence where, - `R1` is a generic tiled reduction with at least one result. - `E` is an elementwise generic that consumes at least one output of `R1` and uses it broadcasted over the same reduction axis as `R1` (This recovers the rank of `R1`'s input) - `R2` is a generic add reduction that uses `E`'s single output and produce single output. And fuse them into a single tiled loop. This type of fusion is not always doable and legality is checked before applying the fusion. PR does not integrate the op with any schedule yet and will be done in a follow-up PR. This is part of landing #274 Assisted by: Claude, Codex
… dependent reduction fusion. (#298) This PR updates xegpu schedules with `fuse_dependent_reduction_ops` to fuse attention-like workloads instead of relying on manual `replace_with_fused_attention` op. Note that `apply_attention_tiling` also uses the new approach. This is a part of landing #274 Assisted by: Claude, Codex
This is the lighthouse-based implementation of the [Linalg][RFC] An Approach for Tiling and Fusing Dependent Reductions in Linalg.
The proposed
structured.fuse_dependent_reduction_opis implemented as lighthouse transform extension. Major difference in the implementation include re-implementing IR utilities that are not available in python bindings (like thescf::tileAndFuseConsumer).Fused attention example and nanoGPT examples are modified to use the new fusion extension. PR also provides isolated tests for the transform extension inside
test/transform/test_fuse_dependant_reduction_ops.py. Implementation can be extended to other intricate usecases as the needs evolve.Assisted by: Claude