Skip to content

Commit 119c4fe

Browse files
committed
softmax tiling example
1 parent 3a10839 commit 119c4fe

1 file changed

Lines changed: 27 additions & 8 deletions

File tree

‎docs/softmax_tiling.md‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -178,14 +178,33 @@ func.func @softmax_v0(%arg0: memref<64x1024xf32>, %arg1: memref<64x1024xf32>) {
178178

179179
**What happens:** The max reduction and sum reduction are fused into a single pass over the data using the "online softmax" algorithm. This algorithm maintains running max and running sum values, applying a correction factor when the max changes.
180180

181-
**Algorithm:**
182-
```
183-
For each element x in the row:
184-
new_max = max(current_max, x)
185-
correction = exp(old_max - new_max)
186-
corrected_sum = current_sum * correction
187-
new_sum = corrected_sum + exp(x - new_max)
188-
```
181+
**Why this works?**
182+
183+
Consider the reduction loop at an arbitrary stage $i$. We have access to:
184+
185+
$$m_{i-1} = \max(x_0, x_1, \ldots, x_{i-1})$$
186+
187+
$$s_{i-1} = \sum_{j=0}^{i-1} \exp(x_j - m_{i-1})$$
188+
189+
When we process element $x_i$, we first compute the updated max:
190+
191+
$$m_i = \max(m_{i-1}, x_i)$$
192+
193+
Now we need to compute the updated sum. The challenge is that $s_{i-1}$ was computed with respect to $m_{i-1}$, but we need the sum with respect to the new max $m_i$.
194+
195+
Using the property $\exp(a - b) = \frac{\exp(a)}{\exp(b)} = \exp(a) \cdot \exp(-b)$, we can rewrite each term:
196+
197+
$$\exp(x_j - m_i) = \exp(x_j - m_{i-1} - (m_i - m_{i-1})) = \exp(x_j - m_{i-1}) \cdot \exp(m_{i-1} - m_i)$$
198+
199+
Since multiplication distributes over addition, we can factor out the correction term from the entire sum:
200+
201+
$$\sum_{j=0}^{i-1} \exp(x_j - m_i) = \exp(m_{i-1} - m_i) \cdot \sum_{j=0}^{i-1} \exp(x_j - m_{i-1}) = \exp(m_{i-1} - m_i) \cdot s_{i-1}$$
202+
203+
Therefore, the corrected sum at stage $i$ is:
204+
205+
$$s_i = \exp(m_{i-1} - m_i) \cdot s_{i-1} + \exp(x_i - m_i)$$
206+
207+
This shows that we can maintain running max and sum values in a single pass, applying a multiplicative correction factor $\exp(m_{i-1} - m_i)$ whenever the max changes.
189208

190209
```mlir
191210
#map = affine_map<(d0, d1) -> (d0, d1)>

0 commit comments

Comments
 (0)