feat(rune/bench): add comprehensive benchmark suite #113
Conversation
Implements 162 benchmarks for forward/backward/JIT modes across 21 operations (elementwise, reductions, linalg, shape ops, neural ops). Following nx/bench pattern with ubench library: - Forward pass: direct operation execution - Backward pass: automatic differentiation via Rune.grad - JIT compilation: compiled execution via Rune.jit Includes PyTorch (torch.autograd) and JAX (jax.grad) comparison scripts for competitive analysis. Test configuration: - Sizes: [50, 100, 200] for systematic scaling - Dtype: Float32 - Measurements: 10 per benchmark with 3 warmup iterations System specs documented in README. JIT falls back to eager mode on Linux (Metal backend requires macOS).
There was a problem hiding this comment.
I think that this file is not really related to neither this PR nor Raven
There was a problem hiding this comment.
I have updated the link. You can check now.
|
@RidwanAdebosin I've merge the rune benchmarks in 94cfac0, putting you as a co-author as it builds on your work in this PR. I've decided to only include gradient benchmarks for now, since JIT is not ready yet, and the forward-pass benchmarks are already handled by Nx. I'm also only including comparative benchmarks against PyTorch, there's no good reason to track performance against Jax for now, we can do this when we measure JIT performance when it lands. Thank you for your work on this! |
|
@tmattio Thanks a lot! I’m glad to see the benchmarks merged and appreciate being added as a co-author. Makes perfect sense to focus on gradients for now. I’ll be happy to help once the JIT and JAX benchmarking work begins. |
3 participants
Why it matters
Performance of Rune’s backward pass is currently blocking some users. This PR adds systematic benchmarks to provide numbers for both forward and backward passes, enabling investigation and future optimization.
What’s in this PR
rune/bench/for:How to run
dune exec rune/bench/bench_rune.exe uv run rune/bench/bench_pytorch.py uv run rune/bench/bench_jax.pyThis PR implements the full benchmark suite as described in the task, covering all required operations, backward/JIT passes, and comparative scripts for PyTorch and JAX.
For more details, see the benchmark suite README.