You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 119c4fe
Browse filesBrowse the repository at this point in the historyBrowse files
**What happens:** The max reduction and sum reduction are fused into a single pass over the data using the "online softmax" algorithm. This algorithm maintains running max and running sum values, applying a correction factor when the max changes.
180
180
181
-
**Algorithm:**
182
-
```
183
-
For each element x in the row:
184
-
new_max = max(current_max, x)
185
-
correction = exp(old_max - new_max)
186
-
corrected_sum = current_sum * correction
187
-
new_sum = corrected_sum + exp(x - new_max)
188
-
```
181
+
**Why this works?**
182
+
183
+
Consider the reduction loop at an arbitrary stage $i$. We have access to:
When we process element $x_i$, we first compute the updated max:
190
+
191
+
$$m_i = \max(m_{i-1}, x_i)$$
192
+
193
+
Now we need to compute the updated sum. The challenge is that $s_{i-1}$ was computed with respect to $m_{i-1}$, but we need the sum with respect to the new max $m_i$.
194
+
195
+
Using the property $\exp(a - b) = \frac{\exp(a)}{\exp(b)} = \exp(a) \cdot \exp(-b)$, we can rewrite each term:
This shows that we can maintain running max and sum values in a single pass, applying a multiplicative correction factor $\exp(m_{i-1} - m_i)$ whenever the max changes.
0 commit comments