Skip to content

Commit bddad40

Browse files
committed
add diagram
1 parent d1837ca commit bddad40

1 file changed

Lines changed: 271 additions & 0 deletions

File tree

‎docs/stage6.md‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
# Stage 6: Online Softmax (Fused Attention)
2+
3+
Process K/V in chunks to avoid materializing the full attention matrix.
4+
5+
## Matrix Layout Overview
6+
7+
```mermaid
8+
graph TB
9+
subgraph Input["Input Matrices"]
10+
Q["<b>Q Tile</b><br/>128 × 64<br/>(solid)"]
11+
K["<b>K Matrix (full context)</b><br/>4096 × 64<br/>(load in chunks)"]
12+
V["<b>V Matrix (full context)</b><br/>4096 × 64<br/>(load in chunks)"]
13+
end
14+
15+
subgraph Process["Chunked Processing"]
16+
K0["K Chunk 0<br/>64 × 64"]
17+
K1["K Chunk 1<br/>64 × 64"]
18+
K2["K Chunk 2<br/>64 × 64"]
19+
Kdots["..."]
20+
K63["K Chunk 63<br/>64 × 64"]
21+
end
22+
23+
subgraph Output["Output & State"]
24+
O["<b>Output O</b><br/>128 × 64<br/>(accumulated)"]
25+
State["<b>Running State</b><br/>m_i (max)<br/>l_i (sum)"]
26+
end
27+
28+
Q -->|Q @ K^T| K0
29+
K -->|Split into<br/>64 chunks| K0
30+
K --> K1
31+
K --> K2
32+
K --> Kdots
33+
K --> K63
34+
35+
K0 -->|scores| V
36+
V -->|V Chunk 0| O
37+
K0 -.->|update| State
38+
State -.->|correction| O
39+
40+
style Q fill:#bbdefb,stroke:#1976d2,stroke-width:3px
41+
style K fill:#e0e0e0,stroke:#757575,stroke-width:2px,stroke-dasharray: 5 5
42+
style V fill:#e0e0e0,stroke:#757575,stroke-width:2px,stroke-dasharray: 5 5
43+
style O fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
44+
style State fill:#fff9c4,stroke:#f57f17,stroke-width:2px
45+
style K0 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
46+
```
47+
48+
## Online Softmax Algorithm Flow
49+
50+
```mermaid
51+
flowchart TD
52+
Start([Start: Loop over chunks i=0..63]) --> Init[Initialize:<br/>m_0 = -∞<br/>l_0 = 0<br/>O_0 = 0]
53+
54+
Init --> LoadK[Load K chunk i<br/>64 × 64]
55+
LoadK --> QK[Compute Q @ K^T<br/>128 × 64 @ 64 × 64<br/>= 128 × 64 scores]
56+
57+
QK --> Scale[Scale scores<br/>scores * 1/√d_k]
58+
59+
Scale --> MaxChunk[Find max in chunk<br/>m_chunk = max(scores)]
60+
MaxChunk --> UpdateMax[Update global max<br/>m_new = max(m_old, m_chunk)]
61+
62+
UpdateMax --> ExpScores[Compute exponentials<br/>exp(scores - m_new)]
63+
ExpScores --> SumChunk[Sum exponentials<br/>l_chunk = Σ exp(scores)]
64+
65+
SumChunk --> Correction[Compute correction<br/>α = exp(m_old - m_new)]
66+
Correction --> UpdateSum[Update sum<br/>l_new = l_old * α + l_chunk]
67+
68+
UpdateSum --> LoadV[Load V chunk i<br/>64 × 64]
69+
LoadV --> RescaleO[Rescale old output<br/>O_old * α]
70+
RescaleO --> AccumO[Accumulate output<br/>O_new = O_old * α + exp(scores) @ V]
71+
72+
AccumO --> Check{More<br/>chunks?}
73+
Check -->|Yes| LoadK
74+
Check -->|No| Normalize[Final normalization<br/>O_final = O_accumulated / l_final]
75+
76+
Normalize --> End([End: Output ready])
77+
78+
style Start fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px
79+
style End fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px
80+
style UpdateMax fill:#ffcdd2,stroke:#c62828,stroke-width:2px
81+
style UpdateSum fill:#ffcdd2,stroke:#c62828,stroke-width:2px
82+
style AccumO fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
83+
style Normalize fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
84+
```
85+
86+
## Chunk Processing Detail
87+
88+
```mermaid
89+
sequenceDiagram
90+
participant Q as Q Tile<br/>(128×64)
91+
participant K as K Chunks<br/>(64×64 each)
92+
participant V as V Chunks<br/>(64×64 each)
93+
participant State as Running State<br/>(m_i, l_i)
94+
participant O as Output O<br/>(128×64)
95+
96+
Note over Q,O: Initialize: m_0=-∞, l_0=0, O_0=0
97+
98+
loop For each chunk i = 0..63
99+
K->>Q: Load K chunk i
100+
Q->>Q: Compute Q @ K^T → scores (128×64)
101+
Q->>Q: Scale scores
102+
Q->>State: Find max → m_chunk
103+
State->>State: m_new = max(m_old, m_chunk)
104+
Q->>Q: exp(scores - m_new)
105+
Q->>State: Sum exp → l_chunk
106+
State->>State: α = exp(m_old - m_new)
107+
State->>State: l_new = l_old * α + l_chunk
108+
V->>O: Load V chunk i
109+
O->>O: O_old * α (rescale)
110+
O->>O: O += exp(scores) @ V
111+
State->>State: Update m_old, l_old
112+
end
113+
114+
Note over O: Final: O_final = O / l_final
115+
```
116+
117+
## Visual Representation of Sliding Window
118+
119+
```
120+
Iteration 0:
121+
┌─────────┐ ┌────┬────┬────┬─────────────────────────┐
122+
│ │ │ 0 │ 1 │ 2 │ ... (64 chunks total) │
123+
│ Q │ @ ├────┴────┴────┴─────────────────────────┤
124+
│ (128×64)│ │ K Matrix (4096 × 64) │
125+
│ │ │ [Chunk 0 highlighted] │
126+
└─────────┘ └──────────────────────────────────────────┘
127+
↓
128+
┌─────────────────────┐
129+
│ Partial scores │
130+
│ Update m_0, l_0 │
131+
│ Accumulate to O_0 │
132+
└─────────────────────┘
133+
134+
Iteration 1:
135+
┌─────────┐ ┌────┬────┬────┬─────────────────────────┐
136+
│ │ │ │ 1 │ 2 │ ... │
137+
│ Q │ @ ├────┴────┴────┴─────────────────────────┤
138+
│ (128×64)│ │ K Matrix (4096 × 64) │
139+
│ │ │ [Chunk 1 highlighted] │
140+
└─────────┘ └──────────────────────────────────────────┘
141+
↓
142+
┌─────────────────────┐
143+
│ Partial scores │
144+
│ Update m_1, l_1 │
145+
│ Accumulate to O_1 │
146+
└─────────────────────┘
147+
148+
... continues for 64 iterations ...
149+
```
150+
151+
## Key Benefits
152+
153+
```mermaid
154+
mindmap
155+
root((Online<br/>Softmax))
156+
Memory Efficiency
157+
No 128×4096 matrix
158+
Only 128×64 scores
159+
~32x reduction
160+
Numerical Stability
161+
Running max update
162+
Prevents overflow
163+
Standard softmax trick
164+
Parallelism
165+
Each workgroup independent
166+
512 workgroups total
167+
16 batch×head × 32 tiles
168+
Hardware Optimization
169+
Tiles fit in registers
170+
DPAS instruction reuse
171+
Reduced memory bandwidth
172+
```
173+
174+
## Implementation Details
175+
176+
### Sub-chunking K and V
177+
178+
Each 64-column chunk is further divided into 4 sub-chunks of 16 columns:
179+
180+
```mermaid
181+
graph LR
182+
subgraph "K/V Chunk (64 cols)"
183+
K0[Sub 0<br/>16 cols]
184+
K1[Sub 1<br/>16 cols]
185+
K2[Sub 2<br/>16 cols]
186+
K3[Sub 3<br/>16 cols]
187+
end
188+
189+
K0 --> DPAS[DPAS Operations<br/>128×64 @ 64×16]
190+
K1 --> DPAS
191+
K2 --> DPAS
192+
K3 --> DPAS
193+
194+
DPAS --> Scores[4 partial scores<br/>each 128×16]
195+
Scores --> Max[Find max across all 4]
196+
Max --> Update[Update state & output]
197+
198+
style DPAS fill:#b3e5fc,stroke:#0277bd,stroke-width:2px
199+
style Update fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
200+
```
201+
202+
## State Variables
203+
204+
| Variable | Shape | Purpose |
205+
|----------|-------|---------|
206+
| `m_i` | `128×1` | Running maximum value per row |
207+
| `l_i` | `128×1` | Running sum of exponentials per row |
208+
| `O_i` | `128×64` | Running output accumulator |
209+
| `Q` | `128×64` | Query tile (constant per workgroup) |
210+
| `K_chunk` | `64×64` | Current K chunk being processed |
211+
| `V_chunk` | `64×64` | Current V chunk being processed |
212+
213+
## Comparison: Standard vs Online Softmax
214+
215+
```mermaid
216+
graph TB
217+
subgraph Standard["Standard Attention (Stage 4-5)"]
218+
SQ[Q: 128×64]
219+
SK[K: 4096×64]
220+
SQK["Q@K^T<br/><b>128×4096</b><br/>⚠️ Full matrix"]
221+
SSoft[Softmax<br/>128×4096]
222+
SV[V: 4096×64]
223+
SO[O: 128×64]
224+
225+
SQ --> SQK
226+
SK --> SQK
227+
SQK --> SSoft
228+
SSoft --> SO
229+
SV --> SO
230+
end
231+
232+
subgraph Online["Online Softmax (Stage 6)"]
233+
OQ[Q: 128×64]
234+
OK[K chunks:<br/>64×64]
235+
OQK["Q@K^T<br/><b>128×64</b><br/>✓ Chunk only"]
236+
OState[State:<br/>m_i, l_i]
237+
OV[V chunks:<br/>64×64]
238+
OO[O: 128×64<br/>accumulated]
239+
240+
OQ --> OQK
241+
OK --> OQK
242+
OQK --> OState
243+
OState -.correction.-> OO
244+
OV --> OO
245+
OQK -.exp.-> OO
246+
end
247+
248+
style SQK fill:#ffcdd2,stroke:#c62828,stroke-width:3px
249+
style OQK fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
250+
```
251+
252+
## Mathematical Formulation
253+
254+
For each chunk $i$:
255+
256+
```math
257+
\begin{align}
258+
\text{scores}_i &= Q \cdot K_i^T \cdot \frac{1}{\sqrt{d_k}} \\
259+
m_i &= \max(m_{i-1}, \max(\text{scores}_i)) \\
260+
\alpha_i &= \exp(m_{i-1} - m_i) \\
261+
l_i &= l_{i-1} \cdot \alpha_i + \sum \exp(\text{scores}_i - m_i) \\
262+
O_i &= O_{i-1} \cdot \alpha_i + \exp(\text{scores}_i - m_i) \cdot V_i \\
263+
O_{\text{final}} &= \frac{O_n}{l_n}
264+
\end{align}
265+
```
266+
267+
Where:
268+
- $m_i$ is the running maximum
269+
- $l_i$ is the running sum of exponentials
270+
- $\alpha_i$ is the correction factor for previous chunks
271+
- $O_i$ is the accumulated output

0 commit comments

Comments
 (0)