Skip to content

Commit 9d4a7bb

Browse files
committed
Merge branch 'main' into xegpu_fused_attention_example
2 parents 531c6d4 + 794e215 commit 9d4a7bb

56 files changed

Lines changed: 2142 additions & 1827 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 8 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,8 @@ The upstream _lighthouse_ project needs to keep the three parts restricted to up
2828
A downstream fork of the _lighthouse_ project could extend any and all of the three parts to reach private repositories and tools, execute downstream schedules, load private dialects, etc.
2929

3030
The main purposes of this project, in chronological order, are:
31-
1. To **test and validate the existing assumptions in the upstream MLIR repository**, by encoding common ingress paths, transform schedules, pipelines, target differentiation and basic execution.
32-
2. To help MLIR developers **find common patterns on their pipelines**, and propose actions to merge and reuse upstream code for the same purposes.
31+
1. ~To **test and validate the existing assumptions in the upstream MLIR repository**, by encoding common ingress paths, transform schedules, pipelines, target differentiation and basic execution.~ ✔️
32+
2. To help MLIR developers **find common patterns on their pipelines**, and propose actions to merge and reuse upstream code for the same purposes. 🛠️
3333
3. Once common patterns are detected, to **discuss and agree on dialect design, canonical shapes, and common invariants**, to promote upstream and downstream collaboration on the same grounds.
3434
4. Build a solid base to guide downstream projects (open or closed source) to **fork this project and build on top of it**, making it easier to separate upstream/downstream parts and make it easier to upstream the delta to MLIR and/or the _lighthouse_.
3535
5. In time, this could eventually be the **seed for official upstream tooling that uses MLIR in production environments**, like Clang is to LLVM.
@@ -40,21 +40,15 @@ One key point in the proposal was to not hold _"load bearing"_ code in this repo
4040

4141
It should be fine to have schedule descriptions, aggregation transforms and passes that _use_ the upstream MLIR transforms and passes, but we should _not_ add actual transforms, dialects and passes here to _complement_ the MLIR story.
4242

43-
## Testing Framework
44-
45-
This project should have the same initial purpose as the LLVM Test Suite [https://github.com/llvm/llvm-test-suite], but for MLIR.
46-
4743
## Current Status
4844

49-
Since this is a new project, we're separating the work in branches that will eventually be merged to `main`.
50-
After the initial period, when we have a reasonable testing infrastructure, we can fork & create pull requests.
45+
As of June 2026, the project has *achieved the first goal* above (validation or assumptions), and has created key infrastructure to start the second goal: define common pipelines and develop reusable schedules.
5146

52-
The planned work is:
53-
* **Ingress**: PyTorch to Linalg extraction on branch [TBD].
54-
* **Harness**: Python harness to bind the MLIR tools, dialects and transforms together on branch [harness](https://github.com/llvm/lighthouse/tree/harness).
55-
* **Schedule**: MLIR schedules to combine into pipelines and combination strategy on branch [TBD].
56-
* **Runtime**: Dependencies, tools and environments to run on various hardware on branch [TBD].
57-
* **Build System**: CMake / Bazel magic to check dependencies, track repositories, run CI on branch [TBD].
47+
There are four main parts of the project:
48+
1. **Lighthouse:** The core Python modules that condense all the logic we want to wrap from the existing MLIR Python Bindings' API. This should not create new MLIR interfaces, just use them in a way that guides both Lighthouse user and MLIR developers.
49+
2. **Tools:** Command line tools that wrap the Lighthouse modules in a convenience, user-facing ways, such as `lh-opt`, `lh-run`, `lh-tune` and `kernel-bench`. This should be the entry point of new users, and the inspiration to developers of other tools using Lighthouse.
50+
3. **Examples:** Assorted examples that demonstrate how to use MLIR Bindings, Lighthouse, etc. For example, the [KernelBench](./examples/KernelBench/) example that uses the `kernel-bench` tool to go through all [KernelBench](./third_party/KernelBench/) models using various pipelines and supporting different targets.
51+
4. **Tests:** As with other LLVM tools, this has a LIT test that will test the tools above. Some of the examples also work with LIT tests, so the pre-commit test checks them all.
5852

5953
## Getting up and running
6054

‎examples/KernelBench/README.md‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# Kernel Bench Testing & Benchmarking
2+
3+
This example drives the tool `kernel-bench` to load, compile, run and benchmark Kernel Bench kernels.
4+
5+
Please review the [kernel-bench](../../../tools/README.md#kernel-bench)'s own documentation, for more information on its usage.
6+
7+
## Overview
8+
9+
There are three core components of this example:
10+
1. The lists of kernels, arguments, performance, etc (`level1.yaml`, `level2.yaml`).
11+
2. A `schedules` directory with descriptor files that are used with the kernels above.
12+
3. The `test-kernel-bench` tool that uses the kernel arguments and the schedules to run kernels.
13+
14+
Example:
15+
```bash
16+
# Runs kernel 2 from level 1 and compares the output with PyTorch eager for correctness
17+
$ uv run examples/KernelBench/test-kernel-bench.py --kernel level1/2_
18+
19+
STDOUT:
20+
Executing the module...
21+
Success: The output of the compiled model matches the reference output.
22+
Tolerance: 0.01
23+
24+
STDERR:
25+
26+
Return code: 0
27+
```
28+
29+
In the example above, `test-kernel-bench` will parse `level1.yaml`, take the arguments for kernel `2_...` and call `kernel-bench` with the appropriate arguments.
30+
31+
Any argument that is unknown to `test-kernel-bench` will be passed as-is to `kernel-bench`, so you can fine tune the execution without having to copy-and-paste command lines over.
32+
33+
Note that the `--kernel` option uses `startsWith`, so if you provide `level1/2` instead, it will execute kernels `2_...`, `20_...`, `21_...` etc.
34+
That's why the invocation above has the _underscore_ after the `2` (`2_`), so that only one is executed.
35+
36+
## Execution Modes
37+
38+
The `kernel-bench` tool has two execution modes: **Import** and **Torch Compile**.
39+
40+
### Import Mode
41+
42+
This mode will use Torch MLIR to import the MLIR module into a separate file, then execute that file.
43+
This is helpful when investigating the IR, injecting that IR into other runners, modifying the IR by hand, etc.
44+
45+
Since the IR generated has a return value (the function `forward` does), we need some adjusting to execute the function as a `main` function.
46+
Basically, `kernel-bench` runs some passes to convert return values to arguments and bufferizes it back into it.
47+
This is done automatically, regardless of the schedule you select to pass, and you don't need to do anything.
48+
49+
Input and output buffers are automatically allocated in Python and linked to the Module's execution.
50+
This is how we pass arguments and capture the return values.
51+
This is a _major_ problem in this mode, since some tensors are huge and can exhaust the host's memory.
52+
53+
This is the _default_ mode, and will be used in the command line above.
54+
For now, it's the only mode that supports benchmarking (option `--benchmark`).
55+
56+
### Torch Compile Mode
57+
58+
This mode will _compile_ the module using a Torch compiler backend that calls our MLIR importer, runs our schedules and returns the optimized MLIR module.
59+
The logic is supposed to be identical to the mode above, so any deviations should be treated as a bug.
60+
61+
The main differences between this mode and the import mode are:
62+
* The IR received by the optimizer is _whatever subgraph_ PyTorch decides to give us, which may be more than one. Each will be optimized separately.
63+
* There is no need to allocate extra tensors or handle return values, since PyTorch does all that for us.
64+
* We do not have an independent execution, so we can't yet run benchmarks with it.
65+
66+
This is the _expected_ mode that people will use in production.
67+
Once we can run benchmarks and have fixed all the bugs in this mode, it will become the _default mode_.
68+
69+
## Key Details
70+
71+
### Core Arguments
72+
73+
The key arguments in `test-kernel-bench` are:
74+
* `--kernel`: Specifies the kernel's `startsWith` name. Use to run one or more kernels.
75+
* `--benchmark`: Selects to run benchmarks (import mode only for now), and dispatches 100 iterations.
76+
The entry in `levelN.yaml` file must have the `gflops` set to calculate performance.
77+
* `--dtype`: Selects the data type for the kernel. Some kernels only support f32. Default is f32.
78+
* `--infer-shapes`: Uses the Kernel Bench module's own functions to infer input/output shapes.
79+
Default is to read from the YAML file. Some kernels are too big and the YAML has smaller sizes override.
80+
* `--target` and `--feature`: Decides which schedule sub-class to search schedules from.
81+
Featues are looked up first (ex. `avx512`) and if not found, the target (ex. `x86_64`).
82+
83+
### YAML Syntax
84+
85+
```yaml
86+
- kernel: level1/1_Square_matrix_multiplication_.py
87+
input_shapes: [1024x1024, 1024x1024]
88+
initializations: [rnd, id]
89+
output_shape: 1024x1024
90+
init_args: []
91+
gflops: (1024 * 1024 * 1024 * 2) / 1e9
92+
pipeline: matmul
93+
```
94+
95+
* `kernel`: Level and kernel name, to search inside Kernel Bench's submodule.
96+
* `input_shapes` and `output_shapes`: Dimensions and sizes for input/output shapes.
97+
* `init_args`: Arguments to initialize the module with (before compilation).
98+
* `initializations`: What type to initialize the inputs (`rnd`, `id`, `0`)
99+
* `gflops`: Calculation on how to get the number of floating point operations in this kernel (for performance measurement).
100+
* `pipeline`: Name of the sub-dir, replaces: `schedules/$target/$pipeline/$dtype.yaml`.
101+
102+
### Schedules
103+
104+
The descriptor schedules (YAML format) are stored in the root `schedules` directory.
105+
106+
They are organized in the following way:
107+
* Target agnostic schedules live in the root directory.
108+
* Target specific schedules, but not kernel specific, live in the target directory (`schedules/$target`).
109+
* Kernel specific schedules live inside a kernel type name (ex. `matmul`) inside a particular target.
110+
Different targets would have different schedules (parameters) for the same logic anyway.
111+
* Inside the target directory, there are multiple YAML files, one per datatype (ex. `f32.yaml`, `bf16.yaml`, etc).
112+
113+
Descriptor files can _include_ other files, and that works with relative paths, so you'll see the following in some pipelines:
114+
```yaml
115+
- include: ../lower.yaml
116+
```
117+
118+
So the `x86_64/matmul/f32.yaml` file will include the `x86_64/lower.yaml` with that directive.
119+
120+
_Note: In `kernel-bench`, you can **always** override the pipeline with the `--pipeline` option._
121+
122+
## Pipeline Development
123+
124+
The main reason for this example is to be able to quickly iterate through each kernel and fine-tune parameters and schedules for a particular target.
125+
126+
A reasonable cycle would be:
127+
1. Choose a kernel that doesn't run fast with its current schedule (or uses the default schedule).
128+
2. Create a name for the new schedule (for example, `elementwise`) and name it in the YAML file's `pipeline` key.
129+
3. Create that directory inside your target, and in there an `f32.yaml` file, perhaps copying from another similar pipeline.
130+
4. Run `test-kernel-bench` on that kernel, which will pick up the new pipeline and the correct arguments, and using the `--benchmark` flag, inspect the performance.
131+
5. Fine tune the pipeline YAML file, add new passes, transforms, change parameters, and run the benchmark again, and again.
132+
6. Once happy with the performance, commit the new pipeline and changes, and submit a new PR.
133+
134+
This also works when adding new targets.
135+
Copy and paste from an existing target, and adapt the schedules to your new target.
136+
Then benchmark each kernel on each data type to validate your choices.
137+
Submit a PR with the new target.
138+
139+
To reduce the load on reviewers, please submit PRs with small changes, and iteratively go through them upstream, rather than doing all the work downstream and then submitting a massive PR with all changes.
140+
This also reduces your work downstream while you're working, minimizing rebase work as the upstream tree evolves.
141+
142+
## Usage Examples
143+
144+
### Smoke Tests
145+
146+
Runs all registered kernels with the default schedule (lower-to-loops) on `Torch Compile` mode:
147+
```bash
148+
$ uv run examples/KernelBench/test-kernel-bench.py --smoke-test --torch-compile
149+
```
150+
_Note: This takes a loooong time..._
151+
152+
### Benchmark
153+
154+
Benchmarks a kernel with its default options
155+
```bash
156+
$ uv run examples/KernelBench/test-kernel-bench.py --kernel level1/1_ --benchmark
157+
158+
Running command: /home/rengolin/devel/llvm/lighthouse/tools/kernel-bench /home/rengolin/devel/llvm/lighthouse/third_party/KernelBench/KernelBench/level1/1_Square_matrix_multiplication_.py --pipeline /home/rengolin/devel/llvm/lighthouse/examples/KernelBench/schedules/x86_64/matmul/f32.yaml --benchmark --input-shapes 1024x1024xf32xrnd,1024x1024xf32xid --output-shape 1024x1024xf32x0 --init-args None
159+
160+
STDOUT:
161+
Running the benchmark...
162+
100 runs: 0.015572094917297363 seconds
163+
Success: The output of the compiled model matches the reference output.
164+
Tolerance: 0.01
165+
166+
Performance: 137.91 GFLOPS
167+
168+
STDERR:
169+
170+
Return code: 0
171+
```
172+
_Note: This will show the `kernel-bench` command line, verify the correctness of the output with PyTorch eager, show the mean time of 100 runs and calculate the performance if the YAML file has the `gflops` key set._
173+
174+
### Compare MLIR files
175+
176+
To compare the MLIR generated between Import and Compile modes, call both with the `-print-original-module` argument:
177+
```bash
178+
$ uv run examples/KernelBench/test-kernel-bench.py --kernel level2/11_ --print-original-module > import.mlir
179+
180+
$ uv run examples/KernelBench/test-kernel-bench.py --kernel level2/11_ --print-original-module --torch-compile > compile.mlir
181+
```
182+
_Note: there will be some verbosity in the text file, before and after the MLIR module._
183+
184+
### Debug the schedule
185+
186+
To debug the schedule you selected, you can use the `--print-mlir-after-all` option and see the evolution of your IR as you pass through your pipeline.
187+
If the pipeline crashes, you'll see exactly where it crashed and have the full context.
188+
189+
This isn't a `test-kernel-bench` options, and is an example of forwarding the arguments into `kernel-bench` directly.
190+
191+
```bash
192+
$ uv run examples/KernelBench/test-kernel-bench.py --kernel=level2/11 --print-mlir-after-all
193+
```
194+
_Note: You can also run this between import and compile modes to see the difference on their own evolution through the same pipeline._
195+
196+
### Dump assembly
197+
198+
To dump the assembly generated by the `ExecutionEngine` into a `.s` file, run the program with:
199+
200+
```bash
201+
$ uv run examples/KernelBench/test-kernel-bench.py --kernel level1/1_ --dump-assembly
202+
...
203+
Assembly written to 1_Square_matrix_multiplication_.s
204+
```
205+
206+
The file will be at the directory where you called the script, regardless of where the PyTorch model file was.
207+
208+
_Note: This simply calls `objdump` on the object file dumped by the `ExecutionEngine`, so YMMV on the output._

0 commit comments

Comments
 (0)