|
| 1 | +# Stage 6: Online Softmax (Fused Attention) |
| 2 | + |
| 3 | +Process K/V in chunks to avoid materializing the full attention matrix. |
| 4 | + |
| 5 | +## Matrix Layout Overview |
| 6 | + |
| 7 | +```mermaid |
| 8 | +graph TB |
| 9 | + subgraph Input["Input Matrices"] |
| 10 | + Q["<b>Q Tile</b><br/>128 × 64<br/>(solid)"] |
| 11 | + K["<b>K Matrix (full context)</b><br/>4096 × 64<br/>(load in chunks)"] |
| 12 | + V["<b>V Matrix (full context)</b><br/>4096 × 64<br/>(load in chunks)"] |
| 13 | + end |
| 14 | + |
| 15 | + subgraph Process["Chunked Processing"] |
| 16 | + K0["K Chunk 0<br/>64 × 64"] |
| 17 | + K1["K Chunk 1<br/>64 × 64"] |
| 18 | + K2["K Chunk 2<br/>64 × 64"] |
| 19 | + Kdots["..."] |
| 20 | + K63["K Chunk 63<br/>64 × 64"] |
| 21 | + end |
| 22 | + |
| 23 | + subgraph Output["Output & State"] |
| 24 | + O["<b>Output O</b><br/>128 × 64<br/>(accumulated)"] |
| 25 | + State["<b>Running State</b><br/>m_i (max)<br/>l_i (sum)"] |
| 26 | + end |
| 27 | + |
| 28 | + Q -->|Q @ K^T| K0 |
| 29 | + K -->|Split into<br/>64 chunks| K0 |
| 30 | + K --> K1 |
| 31 | + K --> K2 |
| 32 | + K --> Kdots |
| 33 | + K --> K63 |
| 34 | + |
| 35 | + K0 -->|scores| V |
| 36 | + V -->|V Chunk 0| O |
| 37 | + K0 -.->|update| State |
| 38 | + State -.->|correction| O |
| 39 | + |
| 40 | + style Q fill:#bbdefb,stroke:#1976d2,stroke-width:3px |
| 41 | + style K fill:#e0e0e0,stroke:#757575,stroke-width:2px,stroke-dasharray: 5 5 |
| 42 | + style V fill:#e0e0e0,stroke:#757575,stroke-width:2px,stroke-dasharray: 5 5 |
| 43 | + style O fill:#c8e6c9,stroke:#388e3c,stroke-width:3px |
| 44 | + style State fill:#fff9c4,stroke:#f57f17,stroke-width:2px |
| 45 | + style K0 fill:#ffcdd2,stroke:#c62828,stroke-width:2px |
| 46 | +``` |
| 47 | + |
| 48 | +## Online Softmax Algorithm Flow |
| 49 | + |
| 50 | +```mermaid |
| 51 | +flowchart TD |
| 52 | + Start([Start: Loop over chunks i=0..63]) --> Init[Initialize:<br/>m_0 = -∞<br/>l_0 = 0<br/>O_0 = 0] |
| 53 | + |
| 54 | + Init --> LoadK[Load K chunk i<br/>64 × 64] |
| 55 | + LoadK --> QK[Compute Q @ K^T<br/>128 × 64 @ 64 × 64<br/>= 128 × 64 scores] |
| 56 | + |
| 57 | + QK --> Scale[Scale scores<br/>scores * 1/√d_k] |
| 58 | + |
| 59 | + Scale --> MaxChunk[Find max in chunk<br/>m_chunk = max(scores)] |
| 60 | + MaxChunk --> UpdateMax[Update global max<br/>m_new = max(m_old, m_chunk)] |
| 61 | + |
| 62 | + UpdateMax --> ExpScores[Compute exponentials<br/>exp(scores - m_new)] |
| 63 | + ExpScores --> SumChunk[Sum exponentials<br/>l_chunk = Σ exp(scores)] |
| 64 | + |
| 65 | + SumChunk --> Correction[Compute correction<br/>α = exp(m_old - m_new)] |
| 66 | + Correction --> UpdateSum[Update sum<br/>l_new = l_old * α + l_chunk] |
| 67 | + |
| 68 | + UpdateSum --> LoadV[Load V chunk i<br/>64 × 64] |
| 69 | + LoadV --> RescaleO[Rescale old output<br/>O_old * α] |
| 70 | + RescaleO --> AccumO[Accumulate output<br/>O_new = O_old * α + exp(scores) @ V] |
| 71 | + |
| 72 | + AccumO --> Check{More<br/>chunks?} |
| 73 | + Check -->|Yes| LoadK |
| 74 | + Check -->|No| Normalize[Final normalization<br/>O_final = O_accumulated / l_final] |
| 75 | + |
| 76 | + Normalize --> End([End: Output ready]) |
| 77 | + |
| 78 | + style Start fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px |
| 79 | + style End fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px |
| 80 | + style UpdateMax fill:#ffcdd2,stroke:#c62828,stroke-width:2px |
| 81 | + style UpdateSum fill:#ffcdd2,stroke:#c62828,stroke-width:2px |
| 82 | + style AccumO fill:#c8e6c9,stroke:#388e3c,stroke-width:2px |
| 83 | + style Normalize fill:#c8e6c9,stroke:#388e3c,stroke-width:2px |
| 84 | +``` |
| 85 | + |
| 86 | +## Chunk Processing Detail |
| 87 | + |
| 88 | +```mermaid |
| 89 | +sequenceDiagram |
| 90 | + participant Q as Q Tile<br/>(128×64) |
| 91 | + participant K as K Chunks<br/>(64×64 each) |
| 92 | + participant V as V Chunks<br/>(64×64 each) |
| 93 | + participant State as Running State<br/>(m_i, l_i) |
| 94 | + participant O as Output O<br/>(128×64) |
| 95 | + |
| 96 | + Note over Q,O: Initialize: m_0=-∞, l_0=0, O_0=0 |
| 97 | + |
| 98 | + loop For each chunk i = 0..63 |
| 99 | + K->>Q: Load K chunk i |
| 100 | + Q->>Q: Compute Q @ K^T → scores (128×64) |
| 101 | + Q->>Q: Scale scores |
| 102 | + Q->>State: Find max → m_chunk |
| 103 | + State->>State: m_new = max(m_old, m_chunk) |
| 104 | + Q->>Q: exp(scores - m_new) |
| 105 | + Q->>State: Sum exp → l_chunk |
| 106 | + State->>State: α = exp(m_old - m_new) |
| 107 | + State->>State: l_new = l_old * α + l_chunk |
| 108 | + V->>O: Load V chunk i |
| 109 | + O->>O: O_old * α (rescale) |
| 110 | + O->>O: O += exp(scores) @ V |
| 111 | + State->>State: Update m_old, l_old |
| 112 | + end |
| 113 | + |
| 114 | + Note over O: Final: O_final = O / l_final |
| 115 | +``` |
| 116 | + |
| 117 | +## Visual Representation of Sliding Window |
| 118 | + |
| 119 | +``` |
| 120 | +Iteration 0: |
| 121 | +┌─────────┐ ┌────┬────┬────┬─────────────────────────┐ |
| 122 | +│ │ │ 0 │ 1 │ 2 │ ... (64 chunks total) │ |
| 123 | +│ Q │ @ ├────┴────┴────┴─────────────────────────┤ |
| 124 | +│ (128×64)│ │ K Matrix (4096 × 64) │ |
| 125 | +│ │ │ [Chunk 0 highlighted] │ |
| 126 | +└─────────┘ └──────────────────────────────────────────┘ |
| 127 | + ↓ |
| 128 | + ┌─────────────────────┐ |
| 129 | + │ Partial scores │ |
| 130 | + │ Update m_0, l_0 │ |
| 131 | + │ Accumulate to O_0 │ |
| 132 | + └─────────────────────┘ |
| 133 | +
|
| 134 | +Iteration 1: |
| 135 | +┌─────────┐ ┌────┬────┬────┬─────────────────────────┐ |
| 136 | +│ │ │ │ 1 │ 2 │ ... │ |
| 137 | +│ Q │ @ ├────┴────┴────┴─────────────────────────┤ |
| 138 | +│ (128×64)│ │ K Matrix (4096 × 64) │ |
| 139 | +│ │ │ [Chunk 1 highlighted] │ |
| 140 | +└─────────┘ └──────────────────────────────────────────┘ |
| 141 | + ↓ |
| 142 | + ┌─────────────────────┐ |
| 143 | + │ Partial scores │ |
| 144 | + │ Update m_1, l_1 │ |
| 145 | + │ Accumulate to O_1 │ |
| 146 | + └─────────────────────┘ |
| 147 | +
|
| 148 | +... continues for 64 iterations ... |
| 149 | +``` |
| 150 | + |
| 151 | +## Key Benefits |
| 152 | + |
| 153 | +```mermaid |
| 154 | +mindmap |
| 155 | + root((Online<br/>Softmax)) |
| 156 | + Memory Efficiency |
| 157 | + No 128×4096 matrix |
| 158 | + Only 128×64 scores |
| 159 | + ~32x reduction |
| 160 | + Numerical Stability |
| 161 | + Running max update |
| 162 | + Prevents overflow |
| 163 | + Standard softmax trick |
| 164 | + Parallelism |
| 165 | + Each workgroup independent |
| 166 | + 512 workgroups total |
| 167 | + 16 batch×head × 32 tiles |
| 168 | + Hardware Optimization |
| 169 | + Tiles fit in registers |
| 170 | + DPAS instruction reuse |
| 171 | + Reduced memory bandwidth |
| 172 | +``` |
| 173 | + |
| 174 | +## Implementation Details |
| 175 | + |
| 176 | +### Sub-chunking K and V |
| 177 | + |
| 178 | +Each 64-column chunk is further divided into 4 sub-chunks of 16 columns: |
| 179 | + |
| 180 | +```mermaid |
| 181 | +graph LR |
| 182 | + subgraph "K/V Chunk (64 cols)" |
| 183 | + K0[Sub 0<br/>16 cols] |
| 184 | + K1[Sub 1<br/>16 cols] |
| 185 | + K2[Sub 2<br/>16 cols] |
| 186 | + K3[Sub 3<br/>16 cols] |
| 187 | + end |
| 188 | + |
| 189 | + K0 --> DPAS[DPAS Operations<br/>128×64 @ 64×16] |
| 190 | + K1 --> DPAS |
| 191 | + K2 --> DPAS |
| 192 | + K3 --> DPAS |
| 193 | + |
| 194 | + DPAS --> Scores[4 partial scores<br/>each 128×16] |
| 195 | + Scores --> Max[Find max across all 4] |
| 196 | + Max --> Update[Update state & output] |
| 197 | + |
| 198 | + style DPAS fill:#b3e5fc,stroke:#0277bd,stroke-width:2px |
| 199 | + style Update fill:#c8e6c9,stroke:#388e3c,stroke-width:2px |
| 200 | +``` |
| 201 | + |
| 202 | +## State Variables |
| 203 | + |
| 204 | +| Variable | Shape | Purpose | |
| 205 | +|----------|-------|---------| |
| 206 | +| `m_i` | `128×1` | Running maximum value per row | |
| 207 | +| `l_i` | `128×1` | Running sum of exponentials per row | |
| 208 | +| `O_i` | `128×64` | Running output accumulator | |
| 209 | +| `Q` | `128×64` | Query tile (constant per workgroup) | |
| 210 | +| `K_chunk` | `64×64` | Current K chunk being processed | |
| 211 | +| `V_chunk` | `64×64` | Current V chunk being processed | |
| 212 | + |
| 213 | +## Comparison: Standard vs Online Softmax |
| 214 | + |
| 215 | +```mermaid |
| 216 | +graph TB |
| 217 | + subgraph Standard["Standard Attention (Stage 4-5)"] |
| 218 | + SQ[Q: 128×64] |
| 219 | + SK[K: 4096×64] |
| 220 | + SQK["Q@K^T<br/><b>128×4096</b><br/>⚠️ Full matrix"] |
| 221 | + SSoft[Softmax<br/>128×4096] |
| 222 | + SV[V: 4096×64] |
| 223 | + SO[O: 128×64] |
| 224 | + |
| 225 | + SQ --> SQK |
| 226 | + SK --> SQK |
| 227 | + SQK --> SSoft |
| 228 | + SSoft --> SO |
| 229 | + SV --> SO |
| 230 | + end |
| 231 | + |
| 232 | + subgraph Online["Online Softmax (Stage 6)"] |
| 233 | + OQ[Q: 128×64] |
| 234 | + OK[K chunks:<br/>64×64] |
| 235 | + OQK["Q@K^T<br/><b>128×64</b><br/>✓ Chunk only"] |
| 236 | + OState[State:<br/>m_i, l_i] |
| 237 | + OV[V chunks:<br/>64×64] |
| 238 | + OO[O: 128×64<br/>accumulated] |
| 239 | + |
| 240 | + OQ --> OQK |
| 241 | + OK --> OQK |
| 242 | + OQK --> OState |
| 243 | + OState -.correction.-> OO |
| 244 | + OV --> OO |
| 245 | + OQK -.exp.-> OO |
| 246 | + end |
| 247 | + |
| 248 | + style SQK fill:#ffcdd2,stroke:#c62828,stroke-width:3px |
| 249 | + style OQK fill:#c8e6c9,stroke:#388e3c,stroke-width:3px |
| 250 | +``` |
| 251 | + |
| 252 | +## Mathematical Formulation |
| 253 | + |
| 254 | +For each chunk $i$: |
| 255 | + |
| 256 | +```math |
| 257 | +\begin{align} |
| 258 | +\text{scores}_i &= Q \cdot K_i^T \cdot \frac{1}{\sqrt{d_k}} \\ |
| 259 | +m_i &= \max(m_{i-1}, \max(\text{scores}_i)) \\ |
| 260 | +\alpha_i &= \exp(m_{i-1} - m_i) \\ |
| 261 | +l_i &= l_{i-1} \cdot \alpha_i + \sum \exp(\text{scores}_i - m_i) \\ |
| 262 | +O_i &= O_{i-1} \cdot \alpha_i + \exp(\text{scores}_i - m_i) \cdot V_i \\ |
| 263 | +O_{\text{final}} &= \frac{O_n}{l_n} |
| 264 | +\end{align} |
| 265 | +``` |
| 266 | + |
| 267 | +Where: |
| 268 | +- $m_i$ is the running maximum |
| 269 | +- $l_i$ is the running sum of exponentials |
| 270 | +- $\alpha_i$ is the correction factor for previous chunks |
| 271 | +- $O_i$ is the accumulated output |
0 commit comments