From fba9d57802450ad43575742f6b4e2ca77aad90af Mon Sep 17 00:00:00 2001 From: Michal Adamczyk Date: Fri, 26 Sep 2025 08:29:20 +0300 Subject: [PATCH] Add Unified Attention docs Signed-off-by: Michal Adamczyk --- docs/assets/unified_attn/block_table.png | Bin 0 -> 12556 bytes .../unified_attn/block_table_annotated.png | Bin 0 -> 17652 bytes docs/assets/unified_attn/causal.png | Bin 0 -> 11522 bytes docs/assets/unified_attn/causal_sliced.png | Bin 0 -> 11630 bytes docs/assets/unified_attn/unique.png | Bin 0 -> 10438 bytes docs/features/unified_attn.md | 104 ++++++++++++++++++ 6 files changed, 104 insertions(+) create mode 100644 docs/assets/unified_attn/block_table.png create mode 100644 docs/assets/unified_attn/block_table_annotated.png create mode 100644 docs/assets/unified_attn/causal.png create mode 100644 docs/assets/unified_attn/causal_sliced.png create mode 100644 docs/assets/unified_attn/unique.png create mode 100644 docs/features/unified_attn.md diff --git a/docs/assets/unified_attn/block_table.png b/docs/assets/unified_attn/block_table.png new file mode 100644 index 0000000000000000000000000000000000000000..e932f89740a6eb449d92e5af75ece9cf63cf6c4f GIT binary patch literal 12556 zcmdUWc_5U1`|r?VOGx%D^&or7zV_IcP}b~Gk!_59%hF>D36*_&lO@|o%nV~0SyI*z zBV!qaG#E?v?R-bi`kwdqo^$>@e;C}p_kCaAYyDiG>nrxAffg+_J2eCXq1Dk=H-bRO zYatLaeah3|3c;U~6?`M}HqugszJaQ2k zy1cdFK76|0yrxF^j&{M(yN-?yNz{#o!yQ`{#K28q+P{BT8xd5Di? z2qe5%wHXM~Y;4gUjUH57DYmt*0TLi|(i###ZR=L|tW!V!nwKft z_g&`&_ArFILFrH(+FO}8*@%J6X#okcr`n=!7E?=!p16Jp?hPMQMf>V@aqKWRp8yYT zzP4u8IbKO@d|qlzd!Pk3%i^b)!7dk$JBMCTgg@(PwWh4q#l*^QG94_3wC>-=aOS=$ zcME6})XHRDR3fe}DZRwidDI!tyk3zn_2fWpO^P-iHPjP(;Eh{YoXhQr&x#vYAgEuP zPlXpVs%0`rR7lE;#oR=}3kH`y$puXgUmTK;yu#-puWr8O!J&DeB4F#v#?Aa)wpg96 z*rQ|beRKV)@8r$NXdK~YePUmK-13vQEV#EzaPd%ZHlYC8F%OMf88wuCk-*z{zSzb2 z?)Vi1nx;_3e*eRICM!H9MPYMdlHLz%UzO{!Ib|=_5O7rtrb1Glki8?U4PmtVdo7klEVYj`DN+g*O%ua?(@2=hff9eDeN9bZ@qEOqJL3Q z?wk*~_Xv2QVZrN}3=?tw-22&34z;{gw0mJr(G+24;mkcnf}EuBV<+X-eb#Y@q;W~Z zk(;ew`dA-gr0zd{2WQ9Rd#HVK&D8wv5YX%!IxkSL(o}g&yTp-1FTKS6>|eC^(&FE8ehyYGSVl?Kl6UG|}l^ zO7c|X;=A1u$E`O+T*Q`)Eex4K7mX5WP{GK>(&E!vZmnAI@bCY6SYPH z6N^esQfRt&L;}*$nj5LLFd@k=Lj1gMAFMT{?3vYi7qw+NN%UW7SPjWQ`mT@lpePnq zh>r2S7>r$|%J37Mf+cW+_mmuLCn!q>syi>}}s)*PUE+L2_8PKjs1~j^I zcQm>h1S~zu}+q#z$y(QekM`U5iI0(n~ zX*h1R9ABa(nzK(-O8#7Hy%yT9PD)z;maYCwHdSt(XLd{l71Rc2E{ns3VxkW`g~}>D zw>&wCSl{W#E>(86GNLoA0z)+(?nQAjiz39*6;x;trt?kMDotR^RFjlJ;+C9FbGt(z zOUu$Iy`mU?N&aCt^FU4Gj}T$>wYPJx@ii;<=%dh-Uy|~oPV^$|G3Y0$MTvV@S*bGv zo|b1&rUF6RA&Z4WIVP@W}UtycrzI(-q!kZ(puyNBL}&q4Tn4mbA7(?vT;23 zgb1I9{rmc`hu5?5*$M%em1|Bp=Q9g7FHPzcn>^;MZQI`47TfsY`ue-sZt5d_F~mWz zw5SR-xfrt$XCX4-fikaHewOWMPjRc}trarQmaVO>E)|xu^yd|D787wPnb(q8XebtD ze{6Vu{PJP2!N+FbyWOsgP)iR>`G)EUJ_6bG9K(p%Y-!a}rZ9Wpq{{W?v*LbB>evEY zypL^zxH_hzGRaC}ibwyB!9q(g(@~!;Lg~Wm+U0502}@7it`l=sn~32=QlQm@9kfgvB$U|_3wNSxbrFMf=qdvgxLPB`&r7&d`2~n zF`6d=*L)7rd*py9V$7AWN^7@?U-uAQB`ak1>QRQbFD^?MlnR|Z+dFaibZbDrdc^cG zby&HBHjb0N&4cEfAhI~(#GT^TPK9lJ$1Ki=jS3jKb& z>AbL}GfoS<=O57oLMKyiLcI4sRpPS5#Kh_Z*5AuJ1b?U+nltxw zZ}nnzq0Ga%i`9zaSF^n9U#CP{I{)?IRne~D2%_&cCx~r;9H{>dIkY!K^TN8EyFCD4 zz+J*SNKZt;>ZMhY08wnIH@Dx&zD+2EC%j%@27eE}fOp`x%tSKS+=wr+<|`Y3_8kUU zb(VS!o&)emOxn@XZD z7%jQj+Dn%b8UgrVz77hY4&9U1Ut6uJ#3FI|9AdY@eYK~_STa@|uy+R|i%JTwxb6-B zk<5AV4&1;x!WJo2Jl5hxXV%Mf#g-`E;SlK;BXKXCe@h*GoW5aRNrH1}7s1-nR3x$R zNZKa!WqPUAf`5=cYbq{N*_RA*{px?x$J^cQ?Qg|9uYitYHecv%?Cc1Ni;E8qthHfP z(rU${4%{sPMAG)x$-N!55+vOSHxnjbZaJ}RT^ZIJ;@rEGZ^rIY`0~$>kBDKuz~A^u z(i{UR2t<+T-(DM^5#VVmC%OOB*g~QkSg4kcT+DH5$x_H%d+g&bMm4ijx zXUSN?n!8lyHA);To5GK+G zn5hX*Ebtx%J5P)bkGE)_gQN@&)NIW5FiM+<`cAd^TK@20T*yWMCr|?hl9m`tt&+PQ zHp^!0(H;#ek2hHK2)>hE4O}ST@=5D%2M#FCHS0t`6`@M**TK(5=?LpQz4>D?X~|fw z!5s8&BpH-?aelMUE`@d7cThnWQ9km~?yz*Zk-WtKRWHULQb2xRdSii?J6c|E4S z!UqP16s>DgO92sbu4JIRsr`h-e-@f!RBHyx_km>CX99unDc7q+Sw&@bcJ{udW$yOw z{JE{dHyHgtKyewpZC?>KKuVF!{|lKv9IGz|1Ht0m!?;;)QA#s9l$f5+zS zZq}v-YCqrw<*e)gwU1Gdv8=;VOyEgH32Hx(`3j7x@GLW++c99HYFKaVRm20Pv=)HpIw*;>lI|**}GIfiviX+v7(-Sn`EN^T(-Szun zHe8fc)IWnZsUw+B=GW%-^68ARk<+MDp(}*8ome1HYpPll|7UbM_PZZ%qj{J9*&~7@NrlY zS77Tx04r~MpByU>SY0ij|5EZCiuPB2abDx`grB!WD&4wf!`YXx+fC|}9|=`d*MGM@ zpLYe&AoNjqR*Hd}sg9{d6|V?)o+>PRIfrq)X5hR*r~SWTjLKP^8O7eO`H2=$@_Ge4 zJ`Q!?2#v4D8aPcRBW#)3#Pmu%4xNHjlAaApe$K86o$QG3-lc~UU|q8&0gVR+7r0Y# zHS8S6djZWt&(mk<;Q@=@b3i%P5#d>vZ1jTHMC=v2^vq3x`ip;Wh5qg|MgJF0GvuPt zY!lF&L)hr66Yr_B`K+r}A^F!w-7xwW!%ynGR>h}%K6w6%0;JZ5^ssARmqu8ioCEfY zEdEYH6|#i|>B}S+B2v~SYPvd4EW|K@Jb9iR8g0TA6BR;dVb-;GMSZc?1bdu8Ap#7r z>cRp8PFfL`iPX3GIafe3VI+qRH;f2Hw-5OVm33>$uA2==uts81H#By%HG z46GK0?=!FvFh})lo0(?zHKUZm7lS5stX`PZuB$N<)Y&Gc0&}C)C=Q{MUt3}cK8wp+ z{;4;C_Bb_~ds0)2h%r^EeTHn=r+#>LONkd{2L-c^ro88kGUrW7^sm!)YbB_z2+LP( zrWF^Z{eD|J5uk_hPf9hyTqkyC^>5tem{@h)J1#|a%15FvfuGApg=by!z(^T>D6?p3 zXt0SFR9)d9ulnhj<1c@%NeIspWofBMHozKTV~j%^FE4)5_Zp-!MGa<5_P-gkm3f9c z=K|8zAZ4w^AaD2k_~cCijmu!SvLGBPSnnw;Y#*{iAQz~}WbUHj?P-N3rU;pSC?ing zi=mDVEtMXgjon>&Yc`BYUS8hX+S*mo@nL&&~iz*npvRCtUcXk*JMO zkIZEtYxs$iV{Ucy(1ZXdW{FRrdN+f1{tJ5ydMVMsIpQ)1B@rr++C7ab%-)V>2pL>E z0SQ;QfOqVPuP%In`@}j|(N#+~Ms&*Y*DM5?pq7ms`rg`9cb2+eU)Y_aD2bKETqk3u zxWf;Y-}8t4s58@cicMQRli>@DiwK+)#5*&|-^IBWi=Nao-6mhU!* z-uHD+DhB3?*KBTAM3016@a`%}8$JL?MaL;qEg&(mYL$1QbM*XxHSYZ*;GQDvV1pnm z1%{!${Rs_-X)ibGe`Qtq$JoQD`Fy!r%&_i#R}MuK=l)2ZJNkwUz(4PN1bdDnD@txj z!P!zxo1j%Qx8%!1AQO5rJ+|sYMl1%_#TrM_6}M<0;q4dkb2r+hYa4&8l2_Zut@rq( zN(zKN2oC;qk$2-pHOj$;(pAGBHYd*ps_JbK5HAe@EJd;$;#W9|>@}iaG(>$~a#)YG zk5I!S#J;YhBCqvKNf1`cHh*`}YSK&iF5w*)^K7~>6jY+6p=m`hBihExFQ(%(CkT-N z9JXatiENe62aGrvst>>OT{uO0ll0HXw6s1Ws=Rsgje@@a$2j&~!uXjr`n^ScxvBE+ zDowuO_Vbg zaJX@ez>AUCYO&fb<9G_lRzGOC#XEoK<2FIw!r?|2Y{T6Do9`z3GoB(Alor7g8&bSj zd&(3!Z{h`gr$acuy<~1&qA=Ivq6k`vy%Y)L7SVoBCgXR{F7~zT?dH!Vjm|O4#q#x$ zhsyDTCln%?bpBj~qTwCZ=AxhhjE?@gr;iBNu;LM0_Xr*QoOyfyyQbadgf1gY)YdQa z%AO>32(0r{-%Z*$%HGaT7b}mm8>hQw2wkMW%3T^V;HgcC=S@3aXYf%L|9(nKp~OsB zaOO&Z@8Zr-JnbX+wGHy!M@m?_<>eQKd9@R!#tI8p+VMSK?t?I(u*iGI70t8MA&4|# zZ9;|-3%OX5p0h&B415$$6U|~Gp38Qx)PMwnH(P_|U)kn%G_(zfs`mmogE;u;gAZ(A z0*z_eu_iSMJsx7Yugo$@Fb)<5UMKtMR{@(&i|b<_ zWiLm;XI{H{T0{kkx_qBoawttj!=}*z^w-AyL|qCqNFdWc)w4OlDWzsRfceVFLgJtg z%RN<{TaSG(O6;p<|iGE$xt*5=3TgVTLpL1Ou*u12~yh_nyCUf9N!1Vfx)(J z7ID}LNkn9zum(<>A?xyOzNLgw%FsXg;nj^kzGXf8u&h*Y`+&_00o4E{!f*BaIm16J zKfa;G=`1uk7jMYW(?L?*{%hE6X*iaq(mNI=F2Q1)(eio6eUL0KA-)@VDOHf^kfqpD z!jI|waeFGu4=y`!j2$qfG>9(ngE9jVAlXIxwxu;?T?$8Z0KIudfsaOQPI5Q+8_ass z9fqU#sxN(U_4Ige%}9#dTY;Qi@9v?{2Thlps|H{ zXPg?lH`MU6c~;70rBVOn9JqwJWt+^P#qs`7AqV&GPER z(~AYR$0B!wRfeArrGCfw?= z(K7zbrvS3&)UL@n0gg6oC6ymg6Y%!)yys~XhmOk$nqB1-Vt|Cq!^<{QYVl%W#qL2B z_eP?Z3VbGO26s8Ox3|;hQl0;+|A4g3gu`zRbkKB13--#K86u(K=Wm-nlw0awN^J*M zor!+H8SWUS0#hqX3Jcq8JC(^|uBFtx10bT=xom{r;n?w|qQ)!Bu=LZd3s}MB$^YrPeJ`)LIQX{=E+kvA>QZ&22q=>TFviNB8GgI1UGZo9Sx^67F&1j*mke%$#RQJ$fq_%OEhkxqIVj3ed( zNdLoc&(=;@lyq*V_zSucG=`P}_)#T=a1Q4^%^L4BxGXZ$^HYy6?TbXK6I#n@C@y-W z`LE9jS)|zYuuo^~meJXQG)4tnx9jRgOz^-d|D8+vl_}qS2Pb^r#K?1hQbr$=fYy{m&eNX2uoz2cQWNlom5suh&jZ1XLQXeMW);=r>jFT_ zdcl~~vW9*5tXmD5lJME|OrdO_cQ%LGgM^i_)g8Lxb@$adNvEu2EuRH~htGE1<-x=y zD9$tRf#MF|;i3V~*+XuB(bS_?vPtFX40kHx|cBt9?O-a}H=+x0H# zxi?&Qzi~CfK9D1wE(a&(!etp$Ve)s)Y;cmHXy1KIw>5lJ;(oTW_ejtC@H%&Y1t=7v zQ!MF!C-`2c%so>~-nvewrjqVjHK8}(avu7<#r;VV2!U{8Ac1Z9Qd`YT^Vu0ni;MDp zOl|nH!2M)-XTmvR0*BW8K2bq=Zsk>k~D}l_#lgyFX=Sud}XY099FVIAQ5TXU@43lu-{gXfb`)ZBrEH>_6HlU!wc8P z^@DC+rxr&F#n=i=x830c5n#*xigsDdRQug^x-Xu*W0?kexI`B6r{i@y&Y*v7Ui!)f zYot5EWr<8lt4uka3%!ia-Q_p4%+!qcKZ<+|k`HugRqX1uaan7kkNCaPoB1tdIx?c< z{<9!T{DPZj-35g+`|pT(FWc$8J~ejzJ(;;4BW76{SW6p~5xexNby8)%NF~%ZTWSiE zIuL(FrE-csrf@hq-8Ttmn@#yi0~3?70V*Nv=vc5DYHDd;wMWZAH5YI zUGt2X!{9S*bLehG8ke?v*IwsuA*T9HMbv$38)&7Nlm6o==-3t&Y;359>L*&v%|Xc@ z5gbr)BuX$a*W+E|2!cUk$sl;f;09%V;E^a5j;m!GX?2Cz#;(Gd{z;WAkTnc7jV;w@ zJR8>|zdk9bnEF!(hqeIAFcK4Mrk)(&m6wKgR;64^2tDy;yuU=h!JVIed$z91-E46W z$w?7(d5M0Y?hsP9LgI#ax?7dmLOZ5Czsh$MDF>+kZVy$EG+ua7*hQQHkJ~Vx)C%JM zlu*O_)iH!$W;3MTIfA24(L+rh7Um**lT$f^qSdQQ=1h7=_8u0r=O_TVxGG1+a+$J= z0LTvv8@4N5crAPKi_pm(4-4tYu>gLo;(~@Ra8>V*Xj-VOI%gj>vZAI{GMDS&-1CUl z*R=^+ADa~1q9w9F9?1U&e7gB%!-?f+qp2{@%+$Q>0p1ZA1ZVtPq1=y43t{xYTmRnN z*OXb8{y9AmXgaKG3lUuOAhGV((RyobLZCTXkU|FY;l&j2y&JppV`+0^Vt*&1Uy@LP z%`*Y+SFtS@{K*Jk1sK)4o)xS+S794BE26crc| zK@uPTNr(Z;^xp`vWk3`h$4ti;%`=sil{N2jN_w=}#ib*y82i3**Ss?|lG*4P#rOWF z-bN7-0DGe|tdu>dGmxje_>1FL?s)`00Q~S=sgaQRW2mJw?uwKpwYiSPc-bj9qif@5 zgxmlHv5u5{P-=kgWSj?_?j?j-zDc@|l87tIu>Kah!ZY|RMegI?iWAEsoUHK9N(M+H zAjXy+IlvJx6}=J6i23RrFwD)hla-a6%1g-WIA+8o&23A;SYWL0+@|=ZsUj=b%=C4- z8SG&_pv7KLyOLQO=o~-`s#8pW$L|*jJhOJ43Qc_zR#iD&MPitb(j|>*z0l2W*}5!G zGbagen6IBR)w7ZH%wayj8P5H+`Vl?MyU3w_*mgJI(U&XjJFw3;(5D6h0ZgNt%U9TT z{KeugIj@5VT${E@pLT0#c1w?Df;m&U#aIR%M(?;}U`vn3OLpdw#UBvN%T81v_=LE7N)zv5av5+T zhp@Eb%|E7r?(L5lTPEoO!@{zlUuws{jLF#@W$z9AXMl6k=_hI3AY+s|JJFT_j#`Bq zgB=VEM80|+5a}h`uLPh4$gm(0LKIT~rTe=Fw_3CoW>)uT%Zno|<2~(qwo|0nCdS}A zWB)d(>SW|?$n||^lWVC6sZB)6=my;?@)$*W#FTuQ7&h42b@gsKl-nwlL&g6SHiCvO z?JHI0<$Gm}#F^`Kh;QyHwU*yyKSgZ);iisdvg1s*Z{NC64Mf|13S1kY=yT>4o)htT zMYSRuS&>l-q_hWtumWRCZAK8XlU<7G#yZ=R^zulj$4G>|FOYbA;cUqC+19nx;F+Yv zcM4C%&K19Ppwt5K!gJF4m)KT{G*@qW+Zox}6yXBryWgN1{+h#+j9`G#tWOV1ZjPmW ziGcU~s?tCje=e@_iU|hNT$?zU5doxPtsJ@7xi)AQd%qSM_DNsdy^$q2e%kSb5yYuZ z*KF3jn$1Jc!qgmTYo{t&V1GpZR;{_6XJxG~@K#)JcM{d{cw%31^LS2H_jY6VoC`1Q z9!nAzn8LD%OUH*%PHy#mBlmw*5Gph9MOVnR;6bRj=iKkBIKV0<46<}gmYidi+2R)B za}*;M49gDg+5Y}E)#i3Lhg#7Ytl8E#^qK8gltb&gG%@s)!$&)dzm|v~r=)EorN>Wy zZH>1{v@4U=7mczYZp>smP$80{4`@CZ97WKOlBu$CQQiIB&0VX(vNtMiA=9n{A&iGn zH|zcP9LLiFaMpACbvK23E)BaRe7|eX!_oToYKVd;(F$}IdX*`JJ_)R4$#EM8&is{a z=fOIjPro)%0b_u_P6z%UoCO&5-wz0ujePU}NiX%g!|p23bqfG!0`T^Abh>_6CnF%$ zBK~Ru&aw9!|8s&0OJ)NPWsdpBQN}-KwqadXg61i;&Z~~0>kX}Uy9gZ&DysqJDQc}m z*_0HfXhBd3wyrnZuo6{eZL-Q*=RbDT>yl>Yn*&=F-+CjHg4ajrfGo{6%mTwfC9aoT z(0n}$9SZa68J0iZsD8xBf?)*=%qvYMbWNYywN`fOy2*?;O^suBO$AeMa@ab|GR}YB zcCpudziM+?Eo`H~>h}{69}QG&Wg2TX_&*2ju;kj6l;u%2E7b|@=j^yRes$l*`7Fs? z)q5Q!xyV>cMh<;OuZq=_A3w`=X3JeP3U`23hF3OKq`4>|OqJgDwq1~~5Z3c7O_hez z7?;(reCiP&^~l{qT1%aFGnZUzqHTb1#~n}Y4@0Oau&&11gWo)DO8GvkvTst2sZy#t zf!3>-Ip*qwi+-j0R!@!WNaNCn-Ie-&VMYf+RR@&Q2K!?QTZ&zMKdDHRXgx3ubJ>7< zREEuTbf8+-I|kA9b7N^wq1cq;lOxooVey88R zlT%%!b%rnwlSXD2EulD6ls2#Px_FcSkm(C%;e~eip?htxU0}3ijQFgv&9z(;Nw+t7 z?C*KaJeTlAmp|{sS4VGHen+PGWw!j}N?tvIMkmm~P*v>eAw_r;VZ@OcT?Yil2O5 zrCbB@FzUvsws?mk*tn76oQGbJGm80D<6-@h1Ay9^lX2e#^FHSh0oVyOwWw>P34_9e zYXh0K!fX>Px6m>PI}?Wp_om?(9p^!htWxfBoI8GSQ?qOdeY^gP`^;|E{P<({ckc{R z%TwZxwfZv}9Z<&qpF|@($d+_g@BJ4~Az=<^UCYYY>_-W7V?v*2%ogBOFI>QjJw~~f zSo7-R^d?bYxJ)n>+&D>8t+mM-d?y<@iKr8j29ekOTX_A2e;#U~a`Ma>p?^#mjnv|q zFMqB|AdP_Vo0PRB4JoCZm}$F|lo2R8q_w_sqs&MM#mKXT4N(aG-o7l!^BCdJvs*Tw z^Q|ZiIT2r^;d!;3dtjg@;hTfy7z?srSCn>#Ilv809p==Od4;y;X{j&=7Z}p6q>Y$Y zsy6GNQ~froU5tAfG1XL22&=clc5DoxHjFBYk)>M08LaBXzV`(Uh6djI+PnC&aHGGx z3|ns~4%lH)ePdKS;8I#kYv7}gKE<8;dz`Xoaxo292Y*F_M%&K8uF>Fsiqbc9o1Kv( zL5o!!T3D%&k&s|6kNuwYTzYK$)q{oLC%~%`rglt@DjvSt3y2F)2>XiYtrO=OhYKH{ zMNxFC*bavKRJKmW^;B*O{$!D;N6ayGzZk+!%V^B|_hxcSl>4`S8S%)g6 zwLb0yh5189F%i*}Q)9XBng7Da8P0j3m3hX0!gS*Xxg9w<`;w zR?fwOoa#@OY{P;U9~tcP4;{`0ZneRz-dD=(@+&&(8luqhz`u8BBSfH zs+zifn`Ip4xlk$cQ{T@SV8*Xy`dt=v?tE?kl7oCR7=&tU7KT^07_g-#Dh@|qmp$1W z;p=IKoPLo!Rq)Lt*S|x0>^)6;5zdj}sfdhwOpNmbjsmsE%XRoO0or%#kMf-hoLvOF z4c&M9K&HWqS5?~boqSgPUqX5z#}7_H@Z$gnMf+Xq#fG7NxLU9mio018?gvXgBW8YB z6qW}Eu~`cMeFGR50jkZqVmK0`p%9iAJ$uc4GtE|*2{9nj$h`OcC8-ZT6c}4A*iKoe z^AB|SV;C>mm;L{GIIL4hdN=2ohs&7*wK05;X2&;Ef!Fq)UvIW~qtZp&qjkFkoqv*P y>mLvg@=1&T_-}weODJen{(hJrfJ}zuBeDa=r}!KmsSn^5h>nJVI$ZV6Z~qIykNZ^s literal 0 HcmV?d00001 diff --git a/docs/assets/unified_attn/block_table_annotated.png b/docs/assets/unified_attn/block_table_annotated.png new file mode 100644 index 0000000000000000000000000000000000000000..6b54dd98535749edda739c1b5874140e7f72b826 GIT binary patch literal 17652 zcmcJ%cQjmG`!}v6Bm@ZwL6is~Q6qYdAbLnd3!_AfK6)>SAR?kgix$zMjcyn{dhbT> zj5b7@!5H2%a^Ls!e4pR5zU%kL`(~{y=FHjq?7h!bKcDLos_|TbjD(H^4-bz_Nm1@4 z9v(pz9v;5h6(Zouru&`Szz=-qmkLkuAboe%fe)7~Wz=Nw@Jb>`PfacZpRa-y^_=nW z$eVEg@vEY2ka&133yNz;Wp0OE zQn~wE`|#<2$x0+xmotQGp)nXwtM#c{^@pH9kvo=6EDEe!BaaQZZaguNW96d>yzB%c z@mztY;?v)wH)<57we?E#AJv=@l$?>&oRJ(BoSEz2EKS|m@H^ojeca%8Qr+?U5-~9` zq&&srEZ_TVWR%>sLKcM7&=LRp*W_%7!c|}w#WGibZR3q|TrvQD*xn@I0e%Q6tBY^I z_`%t3?(QJqE@MZb2H22`B!kszmnH)vV^?3_m#HZ!iStp(bHk#XGPf#8JiGwZ^BDHE zAHeW{@TZNy2{k^uJ5K-%&bSW72OjhScu?^TC#Aqtxi%yuGL%5yVH-v5s2$O(IFh8Z z^Kb{fWoU$#uK*0~F=-jsKzP?r5eIDF1Lm+OTOT`0H@H7XQ1t{iOX%LK_p5Mpbri>1 z?7jlZ%QZf5!v@W4=OWGFK%?aQ*pv{$WJv8lsdLb~Iq9V3uQ97|Jqtx$rn>%`&#_^c zY3G{!5=3{iD-C5>Z3JwOqorhJ{p!M=taO8QjJyb?tX{u^py4cS}5p|t-rf19rtd`1}dXFp0~*- zwc*Url=FOhQqSqWW zU7)9R3$fT<@}c^Me!0_xaJ8g%>s06V)w2~1pSpr+8$)=!NPcs<_@1ns#zXRi z9gr8}K#!8j%+=_(oEbF*B);h_f$y$OYx_y=yk@Q#E!)ce&Yf0iATLZ->WcpG&8X7P z`nTjztV|&2m9M;Xu?%lFe`wtvQJCn$ZlCL?NUyKKUOE!;BB>`Q59g$2d!)wp96zyU z*i+z*y$c}4qrYD+!?}By??O}hjJwEnxvMoZD)X?OXmnapTF`u)+iKj-Mg8{R=A>z^ zT2RL3^HSH?!rI4Lh1p9y*G@d>e6`iRYN<%brp-GI)px4MXJZWNtTH?W;naH4QKd4d zaYtmCx6k&JX;BfmNPf5War6Y-GVz%$o=R!jCxn=Y2sjJbvTroAFWtIMWNZ>>ve?bn zGYy%7TJ<0;nl;c!jE1nZ>Xu+zsRl1PB(z2YjrFp8>N*1!fpKz@V)G2=pstq9W{apm z);p8)!u4GFsP6WqkY-kuEx(LDDn^G^Lj|U?I^-JcY;cpI#>f{mZCbb^F76cAi78|n zJcYZTo>n~fRb6St>kFudKOY)CFVjh0|NUFT&CP9k7PW9(w0*pJAj+O>k2l7uPIy(7 zBTz=tm=q7TF5vl|hXk+q@N=fpY(h959*+=CLLXBp73DZL!2EvV731U7wf9%SX`Vym zpckpJ^MT9tC{994Yj<~@t>aY`n9{Jw&7*E|;m|q+p3m^-gJ|z&D{~pu?^9Z{N}gIn zAe(r2^g#h($ZVSqG=UYDT12kXqZb-(TX61= zZJXPsVunSnYoDyec#9k{32lnYeT80O9!;xVYG(fM8aQV$g2thP_Dh=nsN3$@b4qYY zg_CKso%CIgL!l%*iQnV&~eTQAu-k(H1 zH7u=#GLAQnH{R~9Fy{1KLep*Mt;1iPs@5%}O4jUMqz`=No{=X%JM^258LuP5dN4QR zlO`pPt)UaA8aS?0jlV=`ZZOA0HeAo4w2-A8@n^73ZD;yO`=#EndW#-|--Iwr;#6mg zO^EBzEMpoXL?1dnv01jQz8Q1|TZE3j`Ak+-{~?}tb`vnZ1C;NO<*Kg^L5~gZd&e+c zABa?$6sBy;oYH*Y)ak7LOA<@s$sCGRph?gp!MtAge;N9W=?Y65DEVb-_M1};9^-np zd4jhJ#OmuzuL!0(%h_}Y6}cvfBk}lSl-PC`2X#6`WD+L!AJV(N3q1^!F$pyAygqyz zgUXk_Zf1D*sxMeQ-MZ(>QrJXFU;yd-h|33WvvO}@Ec|&t)SsDh7ipx+vMuoJM+Xu4 zYlrw4#LFs{ZxXvmUQRNrLt+AKb|d0YOs}5zKE=upw=nj30+Cx}P?<%gY;J{Ze$|+G z-+NZYl57lo=0;_iXD3xY3wgPpJjikq9sM4y*%D!3<22bEZ;R}@Geeh!W(-H>(0~tM zrwh_&NI>v3bB!{+iWxI{w@PzH(VDntN_!_M`scU2kNB3uugaX(jH*O0yXHP{gw%vuDXEr4ktmcE*aS7IK3lgu>HP2KUS!q$|Ic;Ri2zQ@|oD` zD}521%Oo3n`Z-BgD)JR2ZRA)ni(jn{AW@hona}Jbk1=Ou4NcRVw{E$ViAA~P?om~D zNQQW!sP+e9bHpShzNVxwK%u@D9-r>hbCH>W#(&qEzf>s4v+ixvfAP*gcW6!mnfv3mF5v`ZsK|k5AXYB0v@qn@gBTUEa9xmRJ*0BAE7`X zbK{bMO9+$77VRT>j=Sq_o7gFge^v6NGE`@FUv2wkqgS)K*6_=9)IFnzdnjozapeao zOY~_Ps@=Vc)bYmjX>bEiPR$U~`FzSdpVaaGm8!Kb60sxW;LTer0fM&BK#KDq&QOEn z0da$?#k1#@5M$r@6qX%Hu(JL$2ykhs7TxssLt?>q9`;CB%L<8^7&ddz!nlR|=aszt!&&zsU7hl;GL;a+)v(*o@})J;PBaZ4s$;t{ z5a^G6Ex*4=-|$SUn+rLYc275VVDI-V>2N*I?|9tapX>95>0ut`Y;Epi?SNEv5vJ2x z`aE`{6yXo)$Uo1o3EJ0}zQ2c>qdG3b%v@$+s+o*$2kl45)h`>>sALZy;673(uWOHD zS%iJO1q0vjH_>3JJon(I#Z&ryFQwbtOkdWipkYG^m9Kf;#O|asod&vhplgvODD=R< z8Tm$MgkMukgez6mg76$~1B%99KhEelHzKyC;b7fd4?897tKnDm(=Wsqnd9^HteaD+ zqIRoK`aA)PMTPF|(MTP~EN82GC!ei&|r)&k+$F z^s`v^l{06?bB^kD&xjbf-=OA~>2s)ux`-3gJ|lDp-)`t@8QMn&h zs=8}w%Cbu~XTQ(yfS}~zd7i#0);2>%zFUM4mvRHl{G6}ZtF$v6bW;63vM83xyZc=H z@X^M~q3J7j$v%wqL0n=4mA+iFokephnQI0(-=#7@y5jDR@Oeg!ajku1{tWZMK>iw;t=WewhW1KH=;#yLiZM(_T6 z{yWMF-5WFESoFl{ajSbn(X|h*Ra&pdx&?ZkdtpM_;KO=1?^^|o7Lr3Zc%@I!dufa( zZ#|8O!>5Fq!oH!aVx{0QsWnooCuf_R4}G6Y?^h};n>Xy$lRn;;-I<@CKYDCCJ~NYb z*|8@ePypd6rP@S>1{`Up2NQI+|&1)K&EUh@Sfv)E^9(uy;=H? zR~&Mmu3nWY-FNcrKb%9NGHccA91hVD-q8{prc4jJ1?#r@8&HU@J@)fS_l~8x>dtT13P_G|Dm~Vf6R2A)+?-ogXibdU(IRDsOPAdv$-zLR8jJzUYORt z;Bh!wO1b{7f5DXagmldb)t-OD+dCf+lJ>4c&gWl381J1Qg4m^@meOt<{zvjl>wSIQ z+rB#0L*up1pLj^vndc#^$HVd_^-P}K{txt}ZQ@f8B#ztE&y;<9 zpgSY{n;K$x{ZN(>up2&uWl}0q=d`H?=gIML&=-@fGWx1E=Zix+UDEzToX;Zik4Mif zrLppR&ga7vu`84PjHh`%h$R1MWT?-Mis3z@GR%s(9xr@pLo~ua&gb0H-}%_1{O}m! zKFhhiF9za7D5POL_l}BAD74Ucb~3$o zk)v@v{`*NQyU~^QLNy#eZO*Mm3!h-S&K=7S_nD?&`c@bXYY$M=>n~qM1&;3zvwT)4pt3@Cp!b1+N0=*iq5BeIuu2t3Qf|r1A1WHu@}=Q>z|RR_{3S zEf@c#>?+~8SrHc}StP6oYgut_%|NI1m^ClLXK_nk?cl6p2T5PQg}R4ztwD)6HoaXN zKC9>ji-!|+_@l5|0VBwj^C1TCeb~fTU(1H|zJ~{63)<^g-$_IjL;kiGUl=MZ+^+zZ zGV1_>$3CkX!z<{QiQx2}u?VwU=Ouy|GYG)s-(r3B{js&>ss2ZF?$<%z@9rbc7KXe0 za$!~+wkI9z&*z>Tt)e3Wi%>PAQsu67bY|>oyBW& zN5a%9uk~jT_HN8i9xSv`7EIC^z>$@mMf(bzWmf*v7C&c?eRSR?2=kF4ww+VYZtLWY zq!**2p1n;*G*O_t>MJYaB{6U7^6bK_=wq;+Mow_=ZfZ>=wFWQuw5_K-if#KL0nY$F z9re@z5B|xmdh&rG+8cc~5A^M_e9xBJjIORfc~s|*g#F$#YFN=70Ksv~A1ca+5stV# zi6uWc={GXEJTOc&qc?kwfXuHk^DvBL2UeEbv?y^`*r{E6C|&_Qujdqr5S>L_r|Gr| zoZqQEe>(?fmw+LZ5%ru8(hRKJsNI}b6-8v;gZbVEtLE~8>h z>^#@)9I-^2hr%hDCNG-T$|s1Z(+_KRzl?@VnoD4hA$SJY(#|r6Da!R}2zazvX{au* ze3zSf|Iys7f5<`dQ~~m;mmSmK4gDHeyn!qj^?iTnSD_+5%zNHib-n<@&5zA%I5*wv zEAj%3_P@fUjR|}{8}zuo`f}C>F$m62e@*PU&MZFjZ6C>#cRcQDKMk{o#y);3!0TPT z8FAJ#v(Zpt1>!7@a%1u@4{Nll5h~~^D-xq4cKsb^FQy^`vbUeysoK0!r5h@NR56q2 zB!bE9Jo&{E<)_>tS!kg+Lam-6YLaRZA55R-J~I+T`CN9#rgW;&-ybdVHC1j$$I`@O zqO47mZ{4S=LQC?wZA66hXs|5O(yNRN*8BaaZ`ZTpD}}lX@sNx`jIUe#VS&2)dX5B>LTEyC`*JACPvi{CzSd}Er(A+uSevkXXlfq|b>kY00XQg6~ zpAOILe3PtzZ&O87c5HkmrD4FQ;e4@NeWb*dUgTYGP#HA6HT=n2ISu`prsm~}_gTBm zN`%yrvECS7-tb2&Cb-v)APaPW@Y$&G&5sq{0p!cyEYYE98_Kav9DcA`io%TQt_cAz z14PcKjHxT%*p%dHhlfV^b0%&5RmV4BrycQMXw-gM`Hw_|=LlGn%jtKS-hcK3{_chB zTKRfA_3rd8aeZlpZjJ6*msMZJ`AN!Xy_BWzwwf5YFxxlkQ0!!Uqdf;3TvE*IcQW`p zZ`f(ngvW1QHY2uvOeSYFZ$02KC{a~5&t6PY&-oj~>V6HG&t{U8?ko!8?z7eL&~mue z`VBa_o?0v`f!r^DG)zb0*m4duHsJ~+Uvdd#=GWub-Fo(IMY%YD7N2a_cDWkcK^rkP@7pqyn!U+ab3?+3w5pzR~bGe+2n~O{Qw_)OZj{E zA>^0k**zVw0N(}{n@~5W^+7+2d!TpBGDS&7=ectkhXfj7Bx$ImmgYSyNQI5W9f|hw zQRTRx`W?o^ZRO?yBGDAKdGFcNg^D2kL#;G77makHmnpqZpq--N9M9vj2p-B8b7$Fe zCymoUfQTYaaffCX1rkm2`VTKXE8|$(%jsm1eI@WrR^}ETVBLaX`*|m+M>t( zb5{o>V*M7C;(|s?Dz)|CqJzgK4Lg(YJ6Egsr%%_5zIo4hwz~w3bEZDVaOz14O%zp* zCEvHD^JRqA&ZoiZVwezAjIA7wy1jfN;iwpp{G4vq44%u2k9yzAM7fr_>kpR^S@(-F z`kg0@Yra+f?(;qs87$=WdXKFW@Qov=GoZ25B~C$9((<$Sif2D}##mIPWrivDh|eC- zs+&a8Rz{Kw4F^;UPy7&%=iN1Ak*8DodVkS8k&97DM-P5HD1XBxi=Ql^x9GT{z}MjX zFzgI5`00(9y1nm=MZk!09%1!~c#%u|`1NC@cG*mx+V8G{I|@6p#1ZQy7LmSAOPn(f z`)4%Ar8x};x_D#0zCI0ZSoJ%4Jqwxb$G7@@5^I)^mTaYhEmFpx&=CoatpB(oNcN3x z)y0tz$v$<^vg@gD=->p7vnFcK@&Cpt0^|5VZe}x^W@y9|*te;3ip6dxZTe0GWY#4gbc;^6l=@cK0_kJ_M-GW0tj zP4&rC3E(OC9wv0UE$tWVvrxwPn4~dg*wxp`lrPRrKZW|B95QJJeYa{10kWSXJ6A_y zvbsxz8 z7wmZ$Y%(3<7KejSwx(a!k&TQ#x7--orXd4?^3XrvS)j|-F)D{+LWhUeQs~KAXX`sF z@7Q`tpgD0l(cd(l7Dc*D(B9V810aJc4V<-oQ6-sP;ZkWZwzXscBlt-re23`|RtKb*ytENbclRXW@j{-ZiVl8&cOBiJ6Ths0x8j zBbA?cuTXZA4X-vyR&6h@SiIA7+YzsSI|ziF2-Q00akFBfbQL#a<~(zqTkgvRdlkwCmfCOnV=i zN%OfzaM_BvPF_Et<~8s#L}&ox`8Bkh!k)RU+sxsnhx*;mY$fL}d%t)-#J0q6k>;D) zxmEyV9he7r$^*O3QK_mDId07S3FQvk%giV_D;C4J&c#Qi`M9^S{{*l)zu7)h?S@Bb z8?9nWT2kxOZYWpTe%i7I;o;5q+yL|5x%YzCn&wWw0gu%{Ojb?yYHns*uwqf90LUr2 z1TqG|Ykha{p_)#e6{F7E+KLO18HRd~)bLp=$=jUJ6}i2E;9pMX0=A0VIv~7KE^6NO z?nS+So{s$bbciFGLz%6@#PP%;Fr&sT{=>u*{(eObo`dU(4@0lLzSh_BkJ-$%vzmSPUg1n~W#NMrEG+(K?05GMEXKF9=n}C#O23 z?G)em*o~E`5wreh6yk`oJ^ki`!Vkc?hcVB)y|~?NY{8jcxyP>spk9q!e<;rj=>ibc z$(b2$%q#o+=4RQ6i3vyVDTTE2hX&`3g%AO^{t9R{WkA`O4~1d%`CT14aa^uSFgaC? zCDO?|;F;M}HYJ8jE&_5_1GpHVimgZW9@{WryCR zM;2B$NG-=;w7>;mqs2t@cz6H>X>U0JroR+@Gu;fJ2xN8P9nx=8m6Cy3NEi!OHRsoKFn>2=8kqyW377z=B*KxBXx5c4+a+7sCsF- zZ_oC-YpYU9@_nxOatGGWG`4kQrb=54ul5ZmdYfWw(=%=DWC2Mvz7H(m*hv)CDwmre z5kGAQ_VTkIKO@Zs$ELnH5SCiD+)RItu1U802}qu*d?T_zqjW(dnBAV? z?d2ZV-U@^0#$J>LigT1L$Xh5JcvLYueBcqkm-I&zzZ@?YUg^*_+|? zKdTnPt(xAR>cKDF#iOIKI=!28zY1L8SEU)N;^vgArFCuDpciZHdDe(59|+cC_$*L; z{DB3K4%#bKDOoa=k?(hkfcO0R?!24s?e=KmcXBTTN(LRMfU}Il+10-!U!l$6AIV2N zh#v`_lGAsJaIHFI>+8$e$(B0Tsych853<@ywQ+D}!?P?rW~_BOmj!ggqk9H&sT}Hz zA%nbH2f^U6ZR8N|me>H4+fGf_h15#Y8zQoGQs7?$YQ6;I?|0B$g-i4!^HR^V#2s9+ zO8zSwKn-U5Pc|Sxy!O~y2jB_oQ`8Fc-P@ZX0zx~wbEOpK%D|!0SO?Xd0F<|Sq>3)o5*_==|BWh+d=NQkg)R(IEICOFpb~1dp|_O?;gfgD4eWY}(5rTP z0+F*nWKsc57`FzN@hp8f-jj0M3jN_7KuqMvS9JV(t;(7j9^NpyWv!1LQxT5RiX0 zF17AN8#ZKg?UuK9fx^?NxK$i?QM`Jq{WU~C1CaYlYT#I%I>Tqmad#I=49EbPPwv!P zbm$)bRAUldQa>;BtyLv-UB{@xQjg1UD8BVEJ>Z;*d#_Y6RHmWK&)9fqg?$?86JheH;fbQVz`i4fN$%Hg8TJ6(D=Y zzGAcT=_IAc^-sqkN~m zT`0SAgHrf+ks^{f+L39x{JCn*jkXm6>fF`5KdBw}cufLqQqLUlVJ=TX2<_eb4{Y3U<>Ng@NqBYQYmtwxFA0QFcxD;muKc(6EH2?Y?cpcK%sv7%;t9U3u? z%(ufN9|d!%v#yweT-DIN`sA(d!?h{jE~v>MzmJab+@Jn>R)Jm!A*AwQP_zx_cQR!K zVfR}uA9z#QUvgorSk>7Cg^WFeG7C?HRb~EuL;>&!;smxgdmm&hXad|AI;rTRyKKLP zDN%Lmb3fRMGwtK=EY+>eoP$v8+!z_{d$>t}XN(oQ14eMWJYj0;yZV^c3}vb$+QAk1 zvsk);dicYNl@Vn>QNbQ@ewdG9w^Xg^U!%U00;5t4^a(XXe{4|L+h6nd5buIV>ib_Q zv4ZxfbBp6I_Y4B%X#uprOwLNf zPotZj5a~&Z8L{^8Uky*E0FDeVdjs6aZQDY&vRAUBl>ZMHcJ7CuKje<6rszzhmwyRM>It1Xc zZ#zw#Xz68T)4%c!ap>Hen&LJQi3^XLNeZap&q}MGW`YG2n=&ma3+fB#p|(7X)udPTokVR3R=T)G)e6K_WTJ< z#=>0GG}jP#@jWB!G8QlT3w5&rr*60C1Y9zH%1&qxaen(O_a@4S&Z@fP^1($xiEUkA zU|>3@dPPl*K2L;qs=$19bvSfVmEKFCxquA1uyQmFdvNEU>e2s}!2j;TQUDa#0=Ih& z;3h9vxh!1i37vS=_6=OZRk!m{hc3ze`Jt^xN|K?gG2D}QTA9bHoHxGgkX+*~jx;lZ z?LX6@t8p3zK}KdD`tw{=ybj000FAZ6*TJ#)k~q&;3R$Mp`Kd};)tpr1mgR<1uMwD+ zyTsN-_GaJzqj(hzM1}!jIa6gEz^?JQZ~Q4^6_ZlaRe#SMJ+fTz;Sz9zCh%s+*yzbw z$$dMApAdjLKNn9~-KQ%Oqmkq42a>O`j3v-bpQ}q%uX}rfXJmXNwtn1GWsTy=*z(gZ zVHH{~y#wTOBK|a)6!tvT6Y&CSEz9*sCM-&8udiKwkrGK0q<9;49Ev2S(x>*OWs(6D z)pOIOu~+Tki|*rCLfmj;F~dzIE0YKm~TJ z46E;3u&iHRvL~B-+3-7r25@4yEW|eUIR(FpCZ@e7c(Mr$)b(kj+9j1S>ywe2sN!cP z_w`r+9}B_6IUohdS;fJpMuLhinRdqD;c^}$3Id!E{nF> zd~Nn?uUT=FSlD{v!j=79MPn{i1sJv|Ab>MYa)E{Y@|%puo?w(*ba^oMX&A2S`raOF zdKY?RR}+B?L;--p+I@EA?-4dGBJ(FsqU5aynw|STZ=7|SEIKun!pZmz?^?l@DH36m z6F4!}&zSNBXS4)Q$h~Jy7-u1hbN$4-a=WaY@%q=PQtU*KP{LPCPL#@IQGIY9&X4hi zA!z?Tj2Loa1BNHrSkWB2gVxP&eDsQ|iPm2hog(#E0;vh#UzE-Ss(Qv~EXuJCj$!uR z0@c)gLL-;)#(5;^Z-+o6Vj;?|zqIaX+%T|juSmpc2B9xG=d2Dx|tXX>N8GgJrUI{z*E*#L~Y zHKQu_HGllC`_sP+UyXO`-UY-KuM9Y@)#&UX)ib9SUS#@?`t}$nOm4U&8P}j zxYjz14A9^A#YoRgO94CLlLbPYsI_qti^I0Rd{r|(JVxY8gk37*)%_Ih{fttE8C-jJ_3i{Hw7G5+|5y^y6x9% z9Q`zNe#{T1d5X2~M;k_dkpiz+F-yV*jkUBr{ea}YQFIMwH-?L9BJ2n|6`>3n*bL;3 zwCOZ&gMsR|@pInHpZ@d&n>Q~Af-}iJ&2@IeV{XD?%6y52%5x>>k3pVjFJ zW=FKm-1dB3`CfYgq%V6UZuPp~+?dat;f>?#N+%WE;zSBaVBeE}d zhi57o`thZQ5Wv#C9M<7^dKS_**|A?5a=;}8d#8Zs3Mhjw1n4WKs;@T#d58VhnvZW^ zdW$1A)0dvoZoF3l@Zi;x!YYV*vb-S1_vP2tKTAK+S;qOyGZUVEB@E+me-xI-?XXF{ zyldYEFp7P$Ie;Uq7exswic)Mp^FNd{Uwy|H^Ny@K#lCa2+F?_wYI0A4WWDj^c!jui zkJmYOn&~$gp|)u(0uC61L{cUrV?IS7yKzzS!x-}La z*JIP{?fsjN)5%bJT*E&Kinyy~SD648l#t~P2J^YBkJVUi5r<0B2_)WPq{ASDao?$0?liRrcj+ zTrKEJaU7~?4XP7obiZ}O6EI{PI<^Yi4y8-;<%gYo#)gi@h0L2vc z`3CGdVC6cP_nVWT@G2v2k3j)bGJeTQamZMe8#}SxEJBqcagi>7*6~pRxPwn<*_XJkM#U^eGzHK4y;RiY>4^%p zWVyRUfEVBjfT?jntuV9F4P~L%oqvGMv0F~znN~cG$#-n@`q1IaCI*H4h>%Gw-OqOy zfHC_D06++&(9MOY>OvhSQnsIxsh+-H}et?rSox#z8q*5te6(& z<>xqS>U{%T)8qHr-c4QC8+r-j(G_%uAca^so^_zwbH;SnbV5bqkyoOxkNjKztI-aF zH#{ffTk`umagKRCLg53n8R^B>hB_+h6^RZH*SE%RRV4AUYV9k#+cI0yOBsl4<%65rVReg(|kG(JWztCQQw{6znb_2X^%9H~* z3s*S(A9Tb6qQn$qUmzPur2Pwgl@9Eg%z(K4w2>l8G}SyT^-g5h$vv6 z1Hw>UY~F$Eo00wEz+7N*K)~aGkPhkpfrp>FII2DfMirV9$17rapkC zXjZ+cXeM#B?A zcYyNopk+-Xs%=t=Hg@kAc|7eECn`w98jq82@81y4c0P_j{Sew8u6E()b# z!3hBM{>QVXnoYcy6$FU4F-!VB-aah@J^;K0@YTE`Y|@^~x2+LusLu&3U%?X%Jpe1USw;qE z?n9pI1orXAlSQr^vJA%B_OC0`z9~VUcmw=MiGM=f;<)g=LV(%(d$B;nTS}~o{Z_U- z?&cMIyuuZr)VoOQf=%Fkq~ol!`Rl~C`M9TC5t23q#19_KKr)`vV}(~j1U8UY3v5(8GkqWi5tPd z@3Gjs-rn&uY!&-9y`a(mjm`R3we-K`S${aP|CVY!cnucsR-|K#Muc9H;VT*B|BJ7} zS(@h3!_%X-11>-|0J;Q_tKt`b+7!mOx3}X8g8?cSt3KJy7Jd?);3(`Tt!lGDUPU6HDvB7yLslTb=KTwU8FPP##c#3QM;!P(65Wop&@PsEIL2jJ! zw@T+Cy1I!1_peq1=HVK@O$07hKL)hq&W0F%V9y0v968a(!P@Vd9-{keP@*cm1vVg? zI+Xz|IsxDb^k%nm8@Jx&$pc9|UJTC6cdv}oP-VT@GjM{g*SEyCZdU>96|gxCz%XkI zQa~Ps%7u%9?`7PL7sW;A7J#si)p~-Hrs!5%xZ_vI?vLFqdJiR69^w9|`|Ab}x)S5l z6t3+x$w-S*8@i_Kb$CRR3Aj)IMD#k7S|jjO#QT6S>d%=%$^1dGxRM?97Q=pV=;a3i)vBy@#?yB90di2D6lFLU z3>zRv&Ff!ylhzEH@%ZdO@`ns58sO||idF$Kzl%jZ0O*qzj_D0%0e9K~hU!f0hRrBf zSgXnj)xHR{EQpPyza+koVWn3Y=GbRr=514;m(~Nq;U=K zZ`2oDLBSt~eh+T|IJUn*Nr?Zvw$8U%V)XC08W4a$B2e@Ox%`C`7iG==#yfnHv0&FV zC*BNutJR+-V}X1D^aTqt0n#iBVmFSQc=>bF>>57~64b_Z-~Nvd-ixNYY4g}_IK)J0 z(tI0QYgc?Ew)oiI;YTC);8YCro2v?do-$sV=8mDKe~#(=wcxypRAY_WWu?DnE^?^4BGHhz&V2tsZ(}#C}guOM_?6sMfKJc z28|8cxzZ?>*m0Ih>;AY)6H{X~=wJrqPvy;`@0_1B+O-A`__4@~aL_c|CV#W5y1(fq z$;cqe^J4B$E5(Uxka^?F<%6PW3V@{eQqY{dn*de*1hP3K z`1$;2;9!6I(9f)LM1)XM5b=ZcCuW}LYpVkj1TDMf0Tc8Gxs|LHioPP$&Bd{N(tCz4 z$az3re{HJ`*w#@z+d+QfrH9oC<%2Y148j@shF%h-4%9W$E0q}<9$N|Z?Jpnsv6T3| zH^5B?a1JP`cHiLcc1!8AJ^b5MAM^aqPj1=aV9xG?^*>hSQXeBeG&axwizR_@lFUQ- z@nn9L2+KDyn{R~z1&VTd^1g)IZxX%H!orayO;3K+?tD399S}7~ztKVSqo<6e6Ny%k zf_D(nU(0yPb-CX30fNAsXqmU( zBeWNdvWX0z?T(P_1=Bzs$XG}~jr=DG8tG8B1K0le?#jp{EOlYkvzPQAmqbdSsePHx zYWZAnU3dt9gVjz|zWK^%7yVvw=oiSayZ6Y0H?l)uaS71#!kp^N@NL4^?)%v)GeuND ze`^cX;K;s3@aWQCI-g?3WcskgY*~N)&EqJnS->e$AFkrN>v# z={Ze6?~bD_vqmcq9^JlbI>ZfL2)bw&Y)I>}S4cGr^K+WPK@cs~Fd(R-Kdw)Z`06o2 zvN5#}`7gTe`Q^)(pbXflNE~00`WoA|I}zX*9e~FA6czqlRzt4*Jn}}dW6GTgicu@p z7vDa{o0lG?4dWmfmE?S&fT{Aua=7;JgRZ4)+iznUW+;;`z)%cO^lX>P<3K09MFTs4I}b+mQ+!z2)mQ`wGkeY& z`nP4PN>It{$lB)>-P1dl$e{21>K+*^us7BYC~zvx5Lc75@Bs~XPAX>(2!X}vMuknX}Q_rbOI`19{`Pq3Rh>9|J5caqh1t{*k_!S}GIw%jg#s3Tn n{!wAzyNkcE0X&zPx;CCVyWkYuWAa6ha!$COb zSpxu1g33c#U2n6sN#o$c0?JF%$i%LxfmxTELiCnmybfZsJ3m;qbTeuZ4@pxy4+2+f zG>aczOQzJa9~1JOZ9Y5A3OqfuZK#&|e(Et1SvLC_CdR?t{{wl1J~#v4Xei%;3k2ys z_?PQe)~%t}UJ6?!;02K-7qEY0y9@w@ zK|!GL|9*uVrM;-lLVonR=BC#@f3!74DO2s_>SjoxjDtf06pOi;v=+*J+~1jXRM^Y0 zAa)1=ACe$uB%ZU(J>ad7y12XADFNraYyhOU8FP8rPMVvPKAGn;87`Y(s5=PdD;r9y z)M`8Ca(Y5WPso$gtUS4@ArZ)nW<>J0+r~lXy&lbYikpkN?$+*)?f$*LWb*fAbYaCW(VQWo$CBD6YUmfQFMn&eX~vE7Yx=hlA9@=jk}%n0 zTahgxzx_>6XH`O)fumqT`bGNe+OOkh29afR)RyW+4Uq}@UMDEOR`+$U>t^&~4*7YA z#pK|WQu(O4mg>dKl`SE^fI`eirQRdU6B`+}VNplgDw)Z>fuHjh@l~SXzLUaI2xyvz zkaWM%u>1Gr0mkLpnDyGs1mng#qhZMi!MS@59s`3xObn)R?z=yP94FGCOpLX@MUaJ2 zzk!~Bnx+Ik#^#--o&lpHhRF%k&W7UwsZG+b7eV)<(QD5wabPBhA^^e5t`qjG{qqi_ zVnc2c}>5{Oyc$-vZgT&(|-I!KEa@e%4=9ymu7~nN74{2M)?Mj zoW+J7)CosVY#z7yz3rWJQdTCyA(_T|UMxmd{ieVTCLGC|B~OKPe77^)%C_mP`H}_FqoCjiaf}Ni0V7h}19~0t*O_%&s z(5zi=P3(@lbbz~q_u8&^`|c@fDUEx3*s9M(1{OKgXjj}Bl?K~*e8ca) zIn&g-t0}pSC_|_6I{Jbs@YIscR2;c1|x=E8x!q3o}K!2Mm4{o{878ipwFTVxQy_;!TX zUZYQj_D%2lMP&WwT&T65d}b^5SJE``lesaQzT(fhn@qb?4RugCo#99#0w240v+F#q z1uM#7j=z{#A3_$~{M#t?3!>f)VNCt^MR?EFbGAiN8DpfY$f(NtH22VD?4g zcVAR8u1%C4V@FmKe(lGmH(E3Y)-jOm`Qh_)jR@d*nI(ovK_Vxkf!pU}6?v#UVQ ze>bQ(jSbx|M9ERSG^}sH{ z@eYPxTrHJ=1b>$ZE zl?Vd@`S4M{Af4wgNT%%MJ3w#CRK22iCC6|GPL;C?2-tRCw|X*E*me>Gc}R!6S%%NQ z%15UhiDEtwQAoIzysl;0Bq-59`t3;!IopS@#=Xha(B?uX$;ZLQS%O#L9(kwA?Sgpe zWoQ7BxrppzQyfwfE(*cP5p~Y zXJIK!M)!J52pgU>`7s~A(rs#}D9PSm6!6-e5|41R(w`J_|Ew;AEhgKc({!=TCyc>;*4O}gTe8|L}$>zPBgnXm6c*o(S8 zK^M_G4>~L2BcgNn?2e^ysUZy*rXo_@$Vgrx-GzR}Iz(q_BpX#}0iw9bl_+Zfa=o`2 zquFw!|3}R3tm0)1aVMV)vuuCUPGlw{^1$EKu*BKnCQ~hXCpj(VI;(N$nr1at$g8%=S$zPi`B=(R_J5bXk?a-*)K6O(u<^wfW(?@f80^ zjeoYUqtn=WCXD^q%0UC20NEdVmOxdH)@%xB4LI8@XK6mBow)kw*Z%g=9~gEv@P*9a z_Snc}4sLQ}czHv`4M2Tpk4W1!U3V>HRnHgB;~sf<1KA9noPGz!#g-ch?G>@?|8r1%5g^m<$sD zIJndR0K@&4Yuow;x><7vdIn;?*D#f{0&--)fpsE1{Fhq8z{h<6a8e(=CPDsClVwmh z09*8+MYS5sxt|la8uAx&1TzF5NLdXC@Ol38s@xS@h!n8kQZuwPVgr;3aY$mKYXR;j zTEob=N4(4Z-x$lsa39Qb#ninr*G1*1geO#gH1X2DxLv+b=Q^#wGUB2%Ze6nry}I<$hjxvA4&; z*@GLzbIYPNGAfpz)dFH(UBjrY8%VeELIKJ^Db7c6_xuL>TNbizLs)%r^}ZMr{es*& zm%01uN28Zx%XokTPtdqH{6g(I^leH}v(0!OLJ3n3Yjde~0UJYLa~0gJL@T&`!pml# zvg3^7Vmo`T-n4i`Vw?@mgY&Jts)SO0=%Nd44{2}L-eTG5@-)ev2CikM=Ki>nh%I{` zsS{5C_<%EgC@j)wA^>UZb#~0J%Q^k^sL3U;TkM7!ILWzlf(+>Y>(z39e*^vH5Xshj z+UU_?94)xJbF+Kz%k5*z4R?$b5M->g`!xb@a9CLwY97da1wP~$i1g>2_suwG`(R`V z>1lpeyYg=j{rd$mF%`GN>=Kd0I?Om1aS&0M=O%bq^r(9zW}TFf*ZPnkNEnQ+7N<%$kbM;chi94)^5Vm!ApR#Wcw9 z*B9aY$BXWHzH*76A;$wwc<^D5d=jgR%UBaGPpCSj^X9O9xvtRrnjC=pUN&&Z77;<~ z*j~W}Aa}qU^Nn(GjuyUWFH1Zk@~9dKV|@nz6R(1xA_|`6{qefWZf{-h0m_$gxc$;z zutA$y73ky3kdR9sqUNr+djJoNE*lue#xbxZYUm7CjfCbfPttM?Xjd(E8Y&UQDB?VL z3jV&@9dNuhM$gBImdP>ZyW*qF8E-b-+J8A9My9*3iAks zbscO6)znPs)_ilHp`e0#1CoTZpqb3mjVy_Q@(y1KvRaF}rL$|~(LgU`9HE-H$kxxC zh$)}9b+_u92rzsynk=v1u820}H$>akv<}AlHmSR#97t^X{=i7io+Pj?ZbK$zUnNqx zm(D{=pW~Mp+aAk_D5q_&`S6IjGAobyCM}+OlE+}QzX%2Pd^%U~d6eZxzkV##xk82! zly4yuwT-?D&q#*-fiGi=#vCLT##i8O2qlc3p|(#sWJ;lSYQ6iqNpu+SCI8;TXU)HU zW6!56qRcNSkF?vtkd~KuNk{++KCX-3jwH(lEHiPduQ?hJCR=vYK5b!2g3~ z&T{fcOjs>`PL8Kg=x$TuPn&PoZ-YB;G6w25O z3J4clM7Ka$A!z2AA}UaRh=&7A;p1>SM7)MFZ?K+?E}Dyx0-KjX(-`~M-WoRHoG-<# z49hZOS^+7ZF7?Ym{w)ief!K03DB@O2by3g3O<0fq%*i`v9Dq^21({N)>XdUmJ30PG zJWT`;Qkd6Ux_p~cv8?wEDUc4%(jg)RL#;NU#p_P!CZJVEzF&!1?0hoc5iK?Mm5ap| zGyMQ^$~eH=!(4*Gq{)OtFmVs5Fux-!yL z*O9mf9G4Ahbp%64b@tI{e*Avu!gThCx;@bu}ePFoL$lm*(t# z>rL>7#i>t<@gnY+b^0~itnV?^arr>!rzY2X5ywNDX&DZM%Od%wD{C~2B{ayod;4A~ zcs3ofhOQt>!-vsopI&jzmh&|sYHYs%MfY#BSOw}1k>%Y2nI8#qEp+zi>2ZU}EU=EO zUO0;wxPZlekU;#tEtH*dPbS_gelLFM_}Bq5_dvs)$@vuSl}%>wl*EquqUvNHJ7@k)vWeS zqJUKaQsamarW1Va}%1%hbJS?JSDUnH#H+lzwvX(;rFn#T#Rmsv}>8bT20HY!v9+4UO!d%PO1JLwi8 z$ulH$qprKj&pu2d>Kc*0o|4hCxWxxQ?4`edceaFG1*VwK_*$n0#Tp&RrQTvvSZtaQ zc<;9?vP@N`_<*h{;!gO_jsvnzN4NR>)wkiZA)8-$D@=XtqkW8roR)V9+=|E+PB&jkPBr@XOFP%(S4Fp>RR-98R* zTI!y6qZX=iA9@}K<8okze7Jz4#?^`l6x)c}4l@9U%izN-^~q~x7CP&aqA|-&tWY+> z`$Y)k&Z{pVS>0hKQ$5rW5Er|I1*cl6>{wLsM49+wx&HWdbjk*K?imF1|G|M z7h%M}RM6nN!h#G3Nnj9Dlurvt8y&XYat&)9>_KSFaTB$4uRokDRH6a-g^49KgW>zK zG-M@DP`W@>1!K}-8I!*=kXNU-I}`%U$C(Ckm7A z!rddsNNh~~%r8n6-Z&70tr_9l7jycw<}Cg^zg;$@PAbp4g1=I6S9s);Y|WX4{*e+m zC8x$hn89CDdb!OMu$_^Aa^#GWs+9?BgAsY^ToXqSZbL}Lq_M*RqD-g_wc7un?$KwS z1~W%dQikM43`#t-EgJ`=lw)rJ;Dzb}PETwlL!!QRIDNk~rc-Y?YC)e`jv8or3EIcn z_%^pBxv81ZCldgsS=Rn4$Ikk1#P{}^#fh)BCOt}7F!&@Y{Q-+!4fTk_cE(03|25GUwm|)^Pd0x6fkv zbk*ro@)BHO@UFE5u{4kXO+;Z;Xb`l#akFRLf_KoidqaxlVYo?Yrm^La`zS zz`|IN!XPP0KNj;@FT`Xu&7?AGi0Z^!+HM91m|iS;N?JJoc~nn+wzEJ8!enJ|=&HN+ za)x-fXTB<6j*bo|&-+qEi5b*E-m^-Tzbac7^cJCB7vn>sJ6E zq`d^ok2?rWF$wf+h`fGS>)szNC{pKybL$-`9x9b(x1OXEzP%u3!8uet9QTN1N72aX zF~!_m(&SLA!$qBnD~5oc{eehkSE?1&ssg#BIxzmrJnXUA4uLT{zqUaB?y$;pPT((S z;#Xo^{RvT%!!1`|1B%;CjyD5-`-33taJ?C8OND}dIgZhF0h0+j@T~SC2w+v`K>!lb zqI(u(h7O+2Nmb6qW=XCExwe4>L^&tSC=TD_oU$abg0ymehJYR0=FJoktl1s-9Qfj> zF7WqVuckwS>y=vS!i=OW-A~JMJ-X`3{4qkdWIoarX2P9IyoRTJo*{6qJEJ&&449n{b03rRVe& z)(jEvf4|pU!^+&6rzln2_2Bt#wPqKU0Laln={+&hlCvOC+yGP=7`J84Yqp?2 zbXVAP#h$4$>UpX)<0T-d2q|fyB8a&62*Ry0tx9etfnNfvEg;3qEsGoFe8&W|6zYFt z%Tpc`l&^opGOBHizQo`42CV8jwyF!dUB0!=PXbY$;hpdrz4_Cq%FvVqVV2MjjFC%^ zIiIz9F^Gomdi&EeJ>PVkMYt&A0G!rL@P=|-=un65(3G#Xr$j|XQqoFYMwOBi%p@;! zn&t8-uiTq$$ne&yj_%=5w#u zsXF_meDHuPsgPw>{8;<4hl#9Ie)6*-8sz;ypeOFUYCGSAqXLoIMz#&?s9+Pm^!sY- z)H|RKu=ZX?@It4unEcz#$40MAxZvE3pT85!N8Y+RH`sK&vn4c%vxfd!{qA#cKv7Ww6^Ms_ zj_{3+-k&=?A^M`$^J~p*rNU%mr`YhPjz)5`;)XIE{44bk&qZVJJl2I9FO!T}{_1x7 zwj%b_^F#{~X?ckN2y*O}wD2jbVi_iGb46# zU?h(7JN%9)4~$p-hnF9wdt`Ku+!lH7<0io*FnKkVO7>v1K+M(C8 zQJJ62X)_E40J<2T7OuQvwvEww?JgnTgFw*IP>^(4Q=^#O*A9mu4l9uTszg2uK23eW z2OF$23*Sm@*shlGR-=gZRAw03`;CDm1xtxwbdW2@<)kb4s^F9Wvnrn0`taVWjxn zz-t%gw5kWL>(=8s5B^CW-mO*%YlTjlT8-&vC@p4hH`HdJ#LBm?!SqW>q-W(9Y9Xf_ zKLA7AN5F5XWsFYf=^|BVAgmL1^5MjG&vC3DJjOcal~d_K1oS>9Jo7}2vw4?87DNo& zKTUcs2B&=VTKUmMSpC-K4y7GIHQ8v*I^dRNKgG{b>$G$=$P=`LN_W*|RPRP_a})XM z(GOVUjN|hh(pCc_ZYtBX7h};(?yu*2&s^uA6{Ub+YoIa870z|POzgF%XW0hj3FiI~ z8d4G9&p<6>M$e?f|Wk?U!>yor^Rck}j;zh5*rT2abjX&!v>TFl!F8(dTl~NatvxFPuN}*+!DDu&hMs71Zl&NfVGkUU*bN>@kpr8xSKxy4HcjP7B+%;ZSw&$Y zgL|$3K{@q$e&0n~uH$0TxzQ;AOdaPJ)gqAw<5;wv$oCy-{Wl`8E~Z zB>zxjVpQksb0f_)cxM?13maK&8z^IDJ+s?GrUXkY6y6#FQ6Q;6Szr1c^N8L3H^Z}| zDAsqO$Z}ouhDh(4!B|YnmbNncZ9DThBDV4+3fi|gfHtU4{Sa{X8%BN_waleZuf#7h zPcAY;g~^=tbA~gN#A<(GCr*sd#yH;Cz&qh`+ESx_)qc9g?ITE zG#^mRZ+<^;1H3J|PQP`FM7&gRr3^C*XlJsQoaivKhB}O)AO;2KiKA%>|7nQJGD}52!7;U&#pti z*L}BqKVW>G8618Hc6USLh8GaD&ulgx$?x_Nc@a<G1%e7f0s)lxX2~lm9TtY zTwd6(twCen*L7n(e!*101HSXceU+a@jwJ;EP!JgQRBz(p-?0oKz|I43-umXiGie5` zsFR}`?W)6~bhasKT1G~qRt6i%-KJGCRO1QW>2w=7z)!GjavAC%ZKi?GWs@i!C6tt3 zbXoqFqtmB&l&VrJP~rm@S{#`>kr8dl{eHT)pn7AG3|euomu)F^t{FMkW(fqDgDYLB zW?mrB+~AQ;+*$|MI7ZTdl~Y) z2R&;_7-U47RvJThYa62sC4I@K6SiJW6-)MA!JPXdVL)yC?hM!xx1c+)7}%8IE^RUX zksO?qbLi~GyV?%gq|QgrSU2A9ww2aM7A`c zg(7dYC_TP#GnG4Uwn}rrr@dyjnAhm8(b8;8ak)xe+oxplRv*}EU|Y2oTwc#zylqr$ z<1YaEy3ck5O)V!V->^yI0E3d_*l+VFVG*ilXKj_iptWG+2<{`+mo+sc_?d`89okfD zQOH=%oMg9@J1gKyHuvtJV>rb)-tdX4S27D8+xX0bFCS)Z>>}m-(ih1_GrQfShDMuX zSO_;992@^nJ+RVs+c>ft6wbi)3_bq@yBc72rg|B8cItos^R2_(`MiKa?Y+Lv9qU3V zTCbbIdjDN*Va+>>I6ttWo$LIk+TwtCf~|=APwKlw*=wXU)f3V#vW)RGNV2bW`R*4$ z4OmE=Z>>8~glNguwND@;t~%ES!<+0Cip&01bQK`BkQWefpoI|B`+m4E#Jh(Ulsmw= z*kwt^tZ`bZKz(p+(9>YChuV+ciC0n$B8I&i||}Olod|oEBM(1d6i(T zA*`~<^ekb+cqlbP>e&{4&_hse9E#E^EAH#_v`F30{*W@rv47ovi)`ccGn4>O0Y|fn z_cby5FJ4!e{5^ttVhQ7zdihGIMZP%G)Mq7kuPWQL(&e`oiH*(WzZ+P@>}dHr9_pcI zhWX8nIgOlov~Ip%C*%xi~&+irkQmdBD_C}jO|4T>}q=p?hP(_6=1!NzcPo-euf zlSrmCiD-oOcpGdKR3fsHnb>Wyy|b1NJRq%maI@4F1V2%`T3lD2sopQQWxk$n+`6>w&)-C+7kHpilN6-7*{; zZUX20pL)OdgzZWoPTtP7khIcw`)Wp4X$7D;FpYO}0AL0tSJ^37_;_+htMxXvI1|}D zS1q;cuU02SUopkli8C$FY~dI|!b!ODgQr(d1RL;2^GlQCuijVXn`&LCeDzo{tm`EP zLpYV1hE@@5FR$v_b*J&SPy1zCib{~)SEl5?mrY|^oF35ZdwFungJcpE*1Ahah6v$kYb+h{3Iivh>N3Q?PA}!(~r%H zZ{p^1-^}hJ46?oge|jkMb7fGMiTkO-QG?&FBOf-q0#Fa47hjhgb8)d_otW7Y_vcX? z6|r;fQhnez?~ZrxS9Q2T;C*!>zwKU?k%8X3_LHFBIbjD^G9SSCn135^Z+) z)BNrw-NpFSj>~`x0eEAEMp2Sq+s@A+J7aqUo5zMq&*HeBv&S68;}z(DKgsgYPpn%M zevoEp5S5yLw0?V{8=lkH_wv&0Egvt*8QHXwE+kfBLrQ;JbYQC-6{8bszbps=4U&8O z_)6e{T`w;z9YwE*15i$9ZGL&y^vg-?%FTiAuQnRC~8QZ+*74lcx-Kcx|U)o(CEFs@gwvx#77dEjGia> zmood$F3H%^|B+Ko!FuDL2L~i*Ip{8!pD|;5%&D$&7}#(FwK3jz;`=K^DIRy_|38l* zaC)oF7yF*U=H5$>#Y(7A8tRADp4|=^;A7u$^x;n5B0&=Y5U@MHf&tFWe2irB_VNv2e|LSLtrc|FB09}ISiJ_asf&CY zxfsDpx2YZh%k)lhSPfo48Oqn)*f7I*R_Ia~#(^6I`faDIjdR|xeDK$%qI{RdNvmy$ z=a!`2u`6t`-+dv9*1*jQI;X2%B#BTfln7YmY#!MjNS$4#AQZ{T&D7*EBviW3x`_)Y zli+a2pS2e;s_&LU1#^R$HYO!YgzeV4u`^w+i?yMz)t-N(Paj5<>6*swg~ISFo=AHI zr$8(Z(EFvmG~#Nxd7OH#0I&$gf#)eq&t1uTDg>o1&WEVj8kq9Snnn$H??49{<@;v~ zQ#`;P?dN?;CB4yT*;Qg70Bk0~1pMTyTNjkpJrCl_srW#MlLKBA)mZ!Rt49OCamup@Eod?45`hF*-jIwWUSL z)opyh`zo%!u)=P2M%zc*9ccW0|A6pnN%R$^Ow711isH*T+BPMmyDJFkFP+d=hgYB@ z2iEf9R&$ytOkaY}#35{O@gF%He^d6*WSWleM*Kj6h0pWHGq{$(fzN+)aNd7Ex)Cyo zZVpT*=THvt2lXbeneW0=_Y*zABY>?HB5iwtr?>e3ZXw%bT%02aS~@fBV~o_gTk!)t PMGUAYXgn;Ivk3klDJDLE literal 0 HcmV?d00001 diff --git a/docs/assets/unified_attn/causal_sliced.png b/docs/assets/unified_attn/causal_sliced.png new file mode 100644 index 0000000000000000000000000000000000000000..99ca33fa1973424bce3fef669ad59d75244cc8ac GIT binary patch literal 11630 zcmZvCc|27A_y4se38hfh(pWOXL|Ld z8SqN>WcMxbACa?$f(($?$-Dr5A$@l5!94&#M^GLblY!qU929k(0f4&k^p6N@|J4)# zcuADx?rFLsS4JPWW{+*Gt#4eF_w>^@=QXCj8jwlnKq`8pfM$!OzT<4eL->tyXJ9}e z_s3X*d@_%;iE3J)+`ZbAaWlgUqcz`(a7PV!^;J8{cexskU~yAPei_9~r_Ev+utCFPJfc)cqyv1+&Cu@SF8-*{ zyVjjrN_Fd^RL?-+ui^IoP2UqwVINGgc9q%soNQZ2?9IoAfR~)+- z>N~%RqQ%dqyT6Z{O60oNv&=)VuD6+)s?qaJD^t+hsgh9OtY@dM%1TH<+PhRvD8*DW z5W4zye;p^sYMXSUx<_Ct!&nC$ z<;B*lRFh9uGgK`J*cfHhSKU8TU9yptn0sK?QfTw}a566UVA$DC!nJ5QDSx*}t;}z; zysHbpb$frc+RoQ*#@XF*e_c0K5OlE~vW`s0VyVtDlqWo7R?afg8d^t}OItfES-D(1 zb9sK~nKc&-(zQ}9zS(1@v%bo))<(fRyxcqYtK5FI&F=PiW&4&O1p8KY#k<-itZ9TX zA9Zqd<&xqDOI(-Uljlsy%=MJzK!0wm)vXZQoXpAx&BGCHlT>F}zUo>}6+4nTlx3$q;Gn0_mMm zm~2&kvdSIy970uxIk0L|5|f)6RO+g{<0;hqW~X{AMKeN0@1Dduz7wz+NRRpJiv5HU4JAz?7J-VCUzQ3R`3We_V!*w*qR^ z>ijcj%p7W?W@Jhhshc-vIkez~+1`UukCkYgl=JdNgbqxR5gaQ8XN%{y^XN>pKISWo zyKU=fn`JJM3mv^JWqT%s#R!epN@R{5FVu*<^2w`uM>`{p@zY2%NE2Ugatkok)huQ9 z4|0DS)y^S`r)%r0Fk z$xW$ULI$98;+!NzHQJ*Ym}Ii{-Jl@fRRXqJ2DXzUWE+p?k$etbtV zEs$rXRTH{_PEX*VhDCEeq{*WzwaBP3?5~f-t8KCE-I3fe{2<-C%Ojd*XW2*jj@KcG zk|JbxujWR@lw^vL8k!DnWkXpxP_{j!(0f=^79xTqH6zBsV$s!e663*awGmC@`TLmc z!ky|I!||2M@q?7r@oBG5H_&cwWgraV_&%F>tk2rS+k(Z1d+`vaU<5xl%rE>Bjn!= z?oq$R{ea(SUHJBaR=QJ8j4b<#)SFh$Xg4TXXIZ{~q z>a8D##Z{ISZ|fcS#MS#gzG!-PuCV5Gj-;nKvM1KZH2J2rayUl@CN?WtNQ?@E6*2k% z%hM12Zd56+Yqm3GZfM1^w5>tc>J%%PN&F}89Yz{FUuf z@McCXG01MKhbS>nabX6pAoBdW=$J^X#l8%es+96G(ZIm=35&L;3oTY7+dU35=;~&fTu#``$XNvBpAQ-(p9(?(=q#WUCm{60vJx5^K ztcTL_7Q>*)l%zYFZPyD2b%rK1QU8cZ{*I9{oV2$Fxc1#;}c%Bn-$i+ zJLH&%2AXlNsuR9+InfJ*ljm8~i}O}uae&MFRd~*WG`*O$YDA*=OmAQ3JzrYcfktnE}>}2ugQi>4URKi-c&okNS}_ z`&o8G{-qZ&-2y7^@ygwI+?sJOe*EY#(`DhCb935U(_kTz4xR|IOK3OcbWYI6=ylEz zyde=0(HqQw=2_(Yd;Nr?XPo()Tx2pSO2ntHm+mjS+CVj=agq>45XzRh`iI!GWWD z+<9WvUA=*LMG{1hp-AMAYRHrOhr|Wol3YC^O$h{fo);0kv7f*}E6cST_9m{zj)&t3 zKeVyHlfQN#`fKB)`jL^5dmde{8-I4jw8!UlZ1ww?5^8yO4EJ{lMlRo3goTCEPFS;V z0CT3>-}3w-NT?La<Oz7j?;!)a>1#R<_gx00Q zd#tl&8f~*bzjPD=&nfCDYuVMjDGaY!(zA@2l{=w$zx0G+Fb}7@&^^CLoT;bGEvlQr znkZRAQV&fn4Sg;pyhWNxN(sDn`e;Tpe?E{wuY}q_x3P``$G%eQvMwRnX?o`b+F|nW z^k-V|=V?f=tp+B#8xGgtD|=)w+TKgtuK40XPm}PeUxDwC*&B;tlX@iyW0es$ zlVqI=CrR2~buI<(i#&}jsnnzQ|9la%1~srSLDrNz@~AC6arJLiHGfJ0aq$z4uicCkx63(iq=LW^ zJKPCm@C_=dvuTG$70kXL$ZbEq9X;^tg?KwoE~-OT?sNzY91_WO?{etekNQ?_%Ch0M ztJN*ICAni}rt81wx`6S-De2shUALT$+9MJ5bo$v_Iman*wYu$a5OZY6OIfFW7MT>I zVwW46G$KQfudZs0Ue{@~Dy%Gp4jk!hToVzAF1-JSU8~jwcj|A`tV1Sja9$kQgia2R z-zOo2%9uSqd*j=IW8RB1GCr85)t~^BOOobyZbF9jkXi7JZ}CJ^PJk~_ zJ&M6eNrsX%ZUX(|uwV*mKOL-U(tn0%?Jnq(XV%R7v~XgjBGtmDDjYiBCJ#obp71#e zQ0ttK)Df<^yNsn7NJtj(^XZ$7l}fWyKJgf1Of^lQ8f3ke6?Kf@vT91Ms z2UG^Kx^=Pa;C*{@#oS4WJo&s$S<%<51{XhonLlnzeK+3u*Il+&lh{}3rPkb(nO?zAE%z zl3QC8=Tf>7>WqF_1NSY_EmUCeWj^p_Fy9uXXbk%HK;IBrHVi`W4^Z_{M&QEez=b7e zNP58g_lB!ou1P4{SqDth9^i9Yx*M{1O$582=#R}@Q(4mZ*kZcHvuWbI^7pP(2_$65 zT$LOt(8e7;e@xUmzC|Az=xCAS(gLB7%~H{Km^-#YbIZ#oeN#5wI$szMd)@K z)Rw&{;WC|1bmb9a8*v`pO?B$;vl=JcF(@p2QZo(ym=Y^Mjfu{w>2b{_(MV$skKsPm zC=7Y6!SpeMrwou&f0X06=W+MJ1hIb}h`?mBtRR?E$FcpC09!pjO)u#x*zEGYFTe4W^c*Ep z$hc9-VRor}nDvcW!~GNf$aE0?uPK!#f#3dt4hX5*cw1M88)N%#llot};qHR0 zC^cS%`3=k%rJo00pW-C1gz9cV@^v7QsGgD+$}m|{-;dX}R!0INz$pS9gW<=w(*!$4s%E~+wV@9fRYH4P&vyZQ6!oahM3GG1 z1XyfKzrgpp2{Ls@W)TU*C`vyOf=WdCc4$c>S*K@LO@g*(4jeHiqZAwRh=med#`=dsGn1|WwyL}LTIynW7pM>(6^D_-t+sG13d$9Pn zim(97P#ErEk+|m6EhJ7n*a$Ass;xk*{BNSZIXX4Af8 zIA&XTA3fyDb8n74zJ#_mfzil~_;o2{7XEu%^tJKzZX@QS1?}L}nq*gM9W5A=E7Ef> z`G>WfixT^ORP33eZ`6#?mCmE+s>&=0@(D(AE6lc5@BQ&IgMo;9Lxsgg}T&n;KTAn0|O8 zE-{KI%Sp_m>)(-v-BjO?p!kp1MbXc8=)PwFw82DzZNLBpVW%YY4?#-mZ~R%XSK^mD zyo}3*$3qg84Ici4y5EDTEiB}17VeJIAMo+d;{AkXGqx36NQ-f&DH0B*$a(L2l!0+$ zoUU@pDrzZ%enXrpbVRJSp;`ESf#cQp1?n_rc;~B^gq*liGwR_x(L920a*L6P)ee-c z&Q*${QBQo55Z59N(+%8a&H?2YB+ep-H@dZ7xx1#KR9Zj-)pw3P#4yR% zw$J)oe}6HN!Ald&QAlaH`OFbtlRJE2J^^R%;qH`A7_h|kS*k$uh3>}9Bc-D8w)jz^ ziE~@&BL(L*KpD(Chc|Na{?bFJE1!W!QaqT9m`@d<^jiR?~#pz5dX)E3T4JWoj-};Klpydf-K=s^Qh&kuY00} zn3VrgPc%PVKf~ME`?X`TU7H#D!qtd~vs!Q(C+A%>&%To{moN<+ul@ks5kXQ_n?qiY zO5~}%LQX5+6Cai}X{hlaO&6l29OKju&FXt5a>@rQ(*EIt0^>|g{_Qe zktqHhnG5qYVIo=06;6&yFppfboH%*LgPy6+;-}QZa+PTy_3!(k$m>H*-&WJ6M`i3s ze|}%^lVQrR^y^ycCQZDyad`0ttTKmF;G>dys!*2fO;oZFj@VZp)S;&Lp~t@}baJmN z->CZ3;ycS`aFZ3)5zr!O<#jDChWiNnG7J=wk}2O8q#AIO`0)gvDG0g`XF4>x3>EFR zJPQnbC9j9ZLv$4rbs1yuFrlR+WsdHChM$x`o*GSHe1;sw^s&mxM7THt@DCMo_wu^? zwa-(8IoYxur~Pi1gGKqV zLk*HR{QTGApnE{N+CnAMdYPfu+UW`@nD%&8Az2^=GHiZH1&{i!VKZ2cKo1=+5QE1{ zeNTR4lj^8fa|v^k7S5Lfg5(neAj3p%Hej*-}rv9951H%^;} z?G2&X^DYz|E4bM49`S|9=3<;R)XgFRXX|DAv&$%<0LE$-p z*g{L;#Xw%{w6Jh9rn18l#K>Nycw5(>U4mot_b!3V{`>*7Kgch=KeL%2=QG5WzIlxE zu9P&Ia^6P2d{bGQh#UW<$ckfT*;YT@j4Ai-t7)X^Xf$R)&|yAT^XEf63omartrM&_ z+twwsPmWP$)oE@wUsVeYx>?ShW6~QMJu+;wNEaMW@ze6ndi8rUEfT2B_da~#PE#VX z7-L}3H;tuD$Ag2J=&hNYk`Qh)abCOyG1l9oi}|j(Ye*#5#@yh!h~F%+E{>HH&LoVs zuGR<+v2i*=4`nsd5>KUsbh&zaf&Z7v6d z#Bc-t%3vyg`NE1LOzoz+kR3T-5c`lkUz~#D{3Tr-y_oJDVWAX$I8d(rhp@Z`57Vt! z49uOQ;uW7Yl5DCgK$!&Lz{Wr8!oG7n@^qY|8MCHu)2^~qvUt|V3_MoXn$U3COb;)5 z#vR-d8?s^?zwTag_kzk=!#_(Q=CpCAO%in=0xC1uiojaUp!8{h=b_q{Hky!c(ZTw; zJTySLIhdj2-U~qKXF4t!c~hB^5bI{@(gRF3B^@-EjvoSg1JQP}YJn}!~rW6)`Zr5J9c4 zi?pAbBl8#9W_O?Ai|j75<2L%D#D|QD;H*bkTf%VCTF-=_4FopaB`bz3_drrf40fZM|19|>RbFRz7aj-ySB%S z-uqQ6HHscLH>>XIb{@A*S@5~JsU#eE#jqFMw~f*2GK*=FZv%mnSxh{}04LqXf zlL6Bi*RJ|CyAO)V*3jiX?NPFLc|Wr*qrZp8AM4;s1!ICjyI)MqKqyS8G|_uCwIz{V zD1Gspv(^xlb(PcTKy6ZoBrKc;4eM=+U}c+GcjUaG!Zlu=mzLw1mkKio0b{31aVYQV zzMYD-W{618wGIeN)dnLb03Q4;%J!|>N>Oo$WqPb;kKmnC1tA2bWGJt@4W4AmnI%BN z(YlQ|+;rwA8_`ZETNree=*i-AEEc^mq+m<)EsB<`HVEWIpdCZs7Z{_2&L3t$80+X6 ztLZ)aWb935&H%q@Kn0{#W-X#W$YB4k(OD_45`I}hy=&u;Fg^E^5=KHD+wp^mf zEWKjyVuO+H&3rx@wl^}&{AT%cDf!ccf&lAAxhPLQlZID_3|X{e{?nqPJ~BQ)x@_x) zd#-!(aaZZ$6~H0Z^8&_3>(@%Gpf=N=MBSRb5jr`X_n?$Cl2naWK2z5=nIGPa@i)0O ztJOvJ*2B`Fuw=cobnE6j`5KX-b4((de8Fa={qSU414Zfy;VtWS>uBLPeukTF;WLB~ zh5*z3&{j@-O#GdKLn(jPZE*6(pKNR>0#IIC_Gx}GE|T!7#OqWLx2|DU691how}$%| z?)W#^t3lz6UXu$EVJrM96R@)5}(q-{-!A!Em-GIT^>8+ zXTBlk_wKbv!bDUzUhQZ_=&;amR?ukXijm7&aL1y)8Elo-_U6br?tZaOjo@Z{hcx1c z)spoVJ3Rl=FWKqETvpP2v|Iq*f9ff_w5DwnZ#{N>u{1Q2OMWTie0zTg#mr`Ubd}wZgKOOKeT!smGbp@1iBJ79zfKLH1me8f=z$T=Z?WzgFFnyUz9x#7ORr%Di-YPu+ zHP4rLn(}Ym@ZDB}S5ir+`PJLw2b)}w45KXD9p|gB#j29%Wa>Y0wH>f($LYOY0?sFn z|Br4u9|1<)fgqdT;fH)7P=)oa{JzR<>!O>VdM4}9i>*wN45D_uC;XwbB;`-5uWarM zrNkd+*G{u-Bhv|e`y(6qp|tV5xArFigM438ja5K9_L-K)#FZQ#qpq6ww@*3Gd>=FQ zYtnou4f0fOsiz!h4vgxg z4?6c2T!C`$f7BMV(q`1l(pvPd)*vGu0Fb)jzO99BI}1gCvHRA8W!~EaSqcz}eq_#3 zX}9Uc4fLiYKqAIM9*z6|If__&R>^>lY!Mj4dd&6lravMs1Tm^5oY)hko!h z;oSTz%XvE?nB(*JqZxzaPKAwfbelJ?ZCubWc&M9e^_dQ)yyQ5X-HPf6=cpWTM_1-n z-t}j(Guuf?z;v=r8zVV0Y*_6;G1cWFMyyMTwt3P3mP!aUvP~@lb{v=8*yTe?Ny`}Z zez`tAFH)v&%l}u-4-XRpC(#2ZVNl$!QC=;#R@qLTduC$baeT4B=6pN59#`i6lo)R= z%KnequQ2<_t_b#BW=#f#Igl2Qj~Vctd8q%k^vc-^Y&YHi)rwR3{?TKK-00{TxM)l# zMnDsOt113JsZ zH7h(1*&lN33pm=trt0o0#WUk%K@@m=_1tM<4jc&RWWpqpK4a|z+b6Ks-A|#`Z9C{u zs@x<)uNp+|FBpiRAytP|5d1plv~1=K$1eb#W@|e;u8;1ziZJ)uy2sfyTG`542l!mw z6dMiM?0-q_?{nvirWY3dz6b?t#!x@6nP_^?;O}eUY!JbpKQ@9cy$%@M1o0DV&BQ5G zQ-Z|4+dfZZAY4gB^L)&%5!4e)Gd@4;OHwSm?(Z`%^zD_QHIu7}ge)K;7|8Ge8W+5Q zb2(qOB1R4|S!~x5hYS)>~RxGx?S*F$2W`6L^xiGU#&?ieA-1 z)v`P9(3DLyU9t5zrL+gx+#$6U%{Bv~fCYGf+=S&SUbBo-HVs|#to%o(St@QD*xXLD zELviAF#mJ>+&K7s{EWla2IR%LaDdg7JnSj$CXM<{`j4L**=!lf@dBg(5^F4cSmgoIIS=1 zI}%IV;{J~??ZWbpvZBK9fR5qXDY!}uKybO3X{+dfhD{vm4Qx;(k?Z08U6lCL19=Ei z1nc_YSL9;j!>?xKYy0G;7Zz$N&1RnG-s%@3E$F&AAOvw|c@yM8fdpJljFC14KuN_F z!mR6uIx@iUKiZRo#vDXozAHv3=Wvgqw~e5*vMgzC``^+^57_y@x+L`dy#@;l7L-=Z zKq~!dlMGH}PyMmj<2_r>z1*+Dg> zYa0d_u$?BV;aB)oAqjB(K;A{r+2R!u;*Q4b5B2ptN`}yKyJ_Tqm62Pg^MrYhDTRy9 zdKs73q+R3J1=0B?Fxn=62Eekrrv?a!nqO`*`mz8A%Upxz_lG07b}OT+p6M9@dh%;B zBtkAyu@fJ&!td?#=@!%VHcv$xs!}cgt#_6Du6I4D*kvkzRVeoMU*#_BOW)_;PSKYh z@h|j=L$(cNby=FhrmrOXQe$E7gJauuBIRR(H^fwv*q%b`~y(A@4nipKB} zsA#;_mK}JcP1PZcFe7IR8(hn?#p6=wWeRpAB|aA9%YU;`NJ_OO?ffSfaHEdBOC|nT|6@S{8yi7}B8j*C zfR8{(APvs(aacZvpeN&&p>0qcfT;VKI>qy2;+UoM@+8(EU2E zIV{{`tk22#s`x0_+~IPI#NYJ-=0i(LiQr|$;=C9M3h)V~U0wmC_xM(IKeq_aX;9tC zGP5&Cq3k#FP>0-ORe4&!@_tT&-#TFEJpuwM^QR?0tEa{{uefn{R;AI+AVw;W$94zd z&6Bxlef(Ws^jZ!sabIYGVOen4vQM#V&Ev!Y`bTj}+P3lxpGw~BNbH0LO{3npv~MQA zbn&9@WWGgYKoW?!G*(TD6rIXTUr91GSC|fljluKl*Ck7KsPIB8!(Te-x>-+yVXoD+B{?`+sv}i)a%+;*}xaC zZ~WLt%$iNzv7iCXrlt#@Bj3J~{U2=I2)aE`2l%Yvkk|hLt*#mQQS3l0o|x)o8oVcquqLc7y1X z4{84WJ9Vlix};`>*j-o@dSqFqX!B?0ChbEMI@P|j3H25mH-clBlaCq_9W8v4N^{f_ zfj{){i^3J15$QU(NsBkjuLn!T*GT-w@2y#}`}Oxfy4F*WS?A&?c;@OrOtS|bVHys*`C0_dSL-(BAn&RrqN52}7wz0apW$aE?(tSsU_bvmA z?Cyd;M^z?bZ?7s3k0}W?CueWkN$56ozK$AP7mj#PTnOjch0noSAd?uKkmh#&t}0`e za+_Xx7k#Ku^I}NuJmu&-J(+*p1q|al#52TfIa@ctmp$}}^Kf~Rn*8Tp&wit*_{Y77 zx&8>evt6QgY*lh!zs5T?lrzctif2xyR@oT%%(VLa@?$G!?KE3KK2AQqhoy66brn{Y zj{E}teb+hK0u+N~lrICBvfxKu({=ZM+BW_aKS*@J;>R!vFtKatoxM{d=H77T621G0 sEhy)ow(Wp@g!_8*|I_esHS&aJZh!ZafE#uM?2!hPcahRRl!k!4H_=Dl>^_j5nb`##U}d*9FJ{bRVUYtA{Y<2=sq`8|&Fd!8{@%?x?> z3GD-cK)gl}Jqr+s0|R^l_wWEu^lU}EfqxwS7KT?q*shb`fg3Kj%chq>pvr{(n>V?E z`@O$IVE!P`!RDP0N1IRKZ4l_Jy^-E!tDqYTG!l01BjwZ5F?y7(m8N;je88i9F?U5y z=Xot2TzYK(qPEk%r?YnL}&MdKh3W^cIm-$|FgA25D@6{9xy0Uj0^! zDXP9E4ojuUC4~xr-pFzqRoG-0+=HYWVZgRfxyrvDLcZ3e)cB$}oSgT7n{Kr?^1EyA z*HUuF3hY>N1}v$5%7zY19;n$h1O{yAbu_=s)SU~IM#_WUsByO2Wf&B{N>q_=38~H~ zp55LZi82&}>85mev!x&KJgw=~7xiFNh8wpDL^2J<_6|a>Rbwdl{=j_|8sloxP@~nt z`&T(X(+)%K;)RU)*X!z?TFmwbs&Rad10K1*O>{^wRxs4dDE?&VjP>08V-4_e{x^(6 z-U!*dHf?m8HnMQr@lmql3EP@oL7|hLIZo=-1*KLrlyhw5VzFgTWk*>J4!lVa%X#%J!M9R} zAa^A#24k#*OnQrnN3RtJoMFlIw$)diddreXEIr+oH|W&+dh4XUd*)m?ulSSdT|*$w zb_?2kqg!m)2ixb+qbU~X@Hz&)9`vJi4>(rSxGWRuVwrn#QJJN<6B5{M&(`sk2j}@g ziwS?g1@^%7G%0P%n*tBGV_Ubl|8>&t3SKVEF{`5I8^T5wgBBw zqDf0)c+LC?3634d#wCr816O;wFd$zWV_1ayN62JJHPsE8dftH3>Bxs+?Iu|_7MDim ze$b7S*8QltUNhnL9-HOAAuod1q#OVvQn*3m?xkC=5c_wg4*y596&lj0^l^uhBH{B7fyQe0_4t*LcZNa z@Vg~Y)p!d6H7LZi2MHD76VL2eIJ>H`Kit-O9FEg57yYzFbC19q?=s2?gM^hMW;W|X zLWd4lQicj1Y`>R45%RD%nA>aGpR%=3Iv3Qk8u>H`^bTaVj^f4!vfM)UBz&OU^g@7% zr5C(OA~SYIPi(Ki_iJsoIj4HG?%KM>-BiF(ETjS*g}b}0Mq1*PyliGnz+3&T3aNY= zO|LePo~xOTDJn*x&(EWpYB){9Qy=yJf;x zS}=mSqGM-?%i-kLC576PzF{92lnX8f$O*$Cx3kfCa(p9WqF`y-L~$3N(7}Ul2`8&L zG+Su9vE6B3aH1U*)MAr^{4*Pc{@1z&{JZ;+;WiP>4+H)S7s|KS-puzbXrT&DQl|~6 zK?S3+N_4E~lm2p9i7v7w{EKuP-tlA-S6@S?TvEr{$BbzYU-KsD9r==c5%VM+&d;8r2qPoN$TAAaKxwTa9Cp{r8@idp9V>?WoNZ&yD2+;ccy5Aun z)tAYP4^7MM0dd1ikXv^~0??$3MAZ+RS1jAzt|2eyJ71m}8@5|>fZN`g3k{=RkT22j z2rkswcL(3IIJ}k>CvD=jm@>#|kAmj7DQa@nA+V!g=YM}$De>fD8XYv@+7MSXdPBp{ zlBD`Wd(;M=;=XnNWb&8YckZb@K96aanT3r9W{YC-76r@6RPFI$moJ&{1l?6;Lijp< zj!Nn7P8|ww-6eeQlhlukW`SsvR}>v)$N>pDInG(0s5xnicAROyEIicb?1CmyvoESo zP7VBQX<1wI^t@6FW3T-jT{(XWxk9i}^tO+tC@5}!-atmn3zd)ev1*mPG6~U3x962Z z?hm8S3`xfvvCa#9Q8eFcgSJg#ctBhWHeM6dr1Gzm&p)#kxM8LidpQLHfBNCTB4USL zWS>2695(je>Y0*Gx3*XlJRQGg@tN1kV$xJi|KMdVdwn2Jd7Q0BF_wo<$OoARf(bap znbC_J71w|<5IP4Y9lx)l>Sc^YTrqpw^c31bzd9bTH|+Kn;5((8gLMX$kPY`E7YHmV zv%F221UUR5K`BVjzLEB3p}@$uJ4AcSqzvYoW_ruGL}RzF zwn>zK=BdMJgfKZ<#)A`bWtX-?#$|w_!m#fHUy)r-3iNDCJ`)F?dY&bc)ael!sxViQ zi~Cz4YZm(XVVy=;yi16R1!Qef;{_9#l3L$J z2rvs~*qh&k^|~0Myb~%KN|m`QCKjy_g9{d6Hkdcx=5v6NOu7wcU+p)Q%&yN51*_9) zxy!qGuJm;fxYG-L8?U2<>s#`ygFZ?H$5lTM>QOufqy|#44W`^;mvL|YReb)Yao#{% z83~7e;yN>c+*`T~F}65sUYFx4m|%r@T{fwh6&rbpzcl4_444pqB=D4xF8w>o_kg&Y z<=E$>p)TETuBdO_I#q9$>Q!oV2WV{%T3uW3fGMA;a8#eZ6R$!D%jNFUO-^-cL2+TX zPWba&>N4DM=56*h+IT^Y=6J#8$d+F^bhALqJRBCWGDmpT*gDYWFkds;EzHJcmo#=I zj|6HCqsUJN1TFJSDzxs$$W+k=s9|7&t?PGGH%rqk<$=U3834?T`y5%x=k;;`ywI7U zFP8q9;ewMLAqpp~Bxz0wGL5eN3+}G3kX!#1XFODnKP%Cv>Qs3*br2}lzs9FQ)VGsK za)lw{;aUjTFJuGnaw~28Hp2O5LHu_K`z0z*`ZoK>`UBk?wJd6boOwO1+nYe;d)ixe zHi&A{H}$ci>Ad}#`%oYjHYld4h}0>fRlsZRoBU3(DjFopBFd@wZ>D43mLxg0Cfbm; z3qc({Y-H)KyOkcjC6?(p@~MR%(Y%Cs@3v@o8W^wNL}6+wE)3OfS94eCtzDdC9`) zC9j98QQ)WqRD~qtF6R!=&~}MF23gZtZ7SiLZIr_)nabDDV?`3+E-Wqv-hwy2;gK1l z5l`g0^>Tyx6Pe|t+~uKOnp`(~9Zhdccmv(&*QuogID?@&8+WlD(2vT}`8w$A zOP2s26O^6YSxM0nu8U5l*-3j~UQ^$h%cjdsIz#qR}kqK!C(TAuMT z>SF$5Dz(5-Cu8DgKulWodXF8tD zW;b~GABi7cgP$29sOki#OUm{xca5>}JymiyyMNRwW0)U{%cD}%mbFEF(x7(vX!}6) zJin5>%$!5c56`^f-<(~L6`@_eR0(o85XyZQr|{0*IqEt7!Vz zMCVMH(w0a!!R1UTjEQS>RPNk9=@v9t1*9Nn#s%0@HFPO2{FRPfsKYHpX?g1zkFCSJ zbI5wg1{z1-G<+Es#V{ozZPxU4k1rGMVUx7A3LK|ZT@!>$SRB)XMWq^=wF^svEsXVY zi_}V<`PKT-GKw;eRM+GTrckYUyll&;{-tOZy}oTRJuKnK#D$r`($LT@J6CkPvgQsK z0U7j`(E46PBl`k*WTSeZ@mFrlDWgzY6(C%o(zw4D?rto#zKy`zbMG5N`zc|6CFt`* z+BZ9y-Ji38eEmntRkm>q@Ztd7{hQOc_(7Kve_ZOFI*v}sT6kIik=4!_OxG?$vWle?Nu3}a$Pi(2Pz!aO`_S_#>t$=dB7_;SA)dzdAM|8jT$Lv;}z zpOyj{#_Mlcn3dLMA6>>`CD$RX1$Db4vhp_vq})Du@gpX3 z<-0Z)BP1WFzDu9d^-VaCv9sP7oFP*Dm&Bk2XdyI|fVeLd7mRy|wZP+_IvgXQ%iK~}jyK0wvO zguyalP%K*@E}Ykhi(2v=nXvfKQOx(dF3wr9HGIPywdfj-DlBodgNg>ZJ!dQ+udUb7 z`%RD9&-O_P(!TJ8(smQHowsTy_xlP-RK7@oFsK$R&N+xN?T8o6shD106SEZ|u5m6n z^fZa#A9(u~Z=jTFcEyj~j>j;DSlM${7PD*LW=2wup%!L-_I$3xvii95l{_TbUe4gt z^?e1~Ypsjq;YJSyf33$=-9B#|kNwS8>_tkFxo!cXr|!*8IQa-Df?t$_!*=x zD??aVd9N_aq5C%yaxO3@*lyvd@JY(@@=I2sY_-k<3TbmUkhYZ#Cn8MfSTC^ud_+ zN^=Jed_J>E!cTl}jN49KeofuQ>-ED~1JM()%ceS&icHXK^10bC9y0#HW<-R6pBWJe zPZ*oZj^jWJ!shzAqX^WVpr(q z;nzty71Jz7TxprP{X*=oihejGJUDxMKj^LG-vVRS0w@xua-+`P_TUi(gDtf?olJ9B z#91<*%lb<>9q0d|83N>V=x;lJ^Iamy{M+L8wkUx+hfnKbrWOMhvUsGai6qHM_f&GyE#w$U)VZ!RPt?_#uA4=G4E+%58xD z2a>bQucD%jUpD2H1@uJQg?r}oR^l)G;ndNxNu)gfIp=&A)3n2aUrXk6vgiINuD6_J z4>5Y$-U`Pn`|i?AJ$ZH>0ZVvFE{1$K>9x#i&of!7+QUnnP8`H|uwcW)@=$hzOXGf#AW^t9hLlUsuCOh>0Eo5x!>un!@5(ra1U#LGMGga4ht9F72jlp3R4 zbm3XGyH1#H@~3w_KNC*%^Pb(II8C*{7a+aGlPiOo$)BL&|s)x z@=EjRt{i4ypguAcPXt|mfhn#+7#X+oXA0@3f#=Fk-xON!{I#8N2oiZ_sT{|aLNl4{ zPxAM^Y?D}uTj2Qa=H+;++P)8h53E80-l;58fgOhcn_sUbYr*v4lX*#rqGyK%!?UxU#ej1Yx%#^mMd31?W zGbz+ivKysxOD-vec$daALY$iQ-5J*)aSHH7kV@~-qCEH0Wu}rjBMUf}?&IwOhXF_Z zBdnIQ3?(Em_s?T^fqt30UX%$QtW3Hnn=eOFaRo111Fa>=K$ygKNaP!OyPPf0o?&n6 ztk0mw4mP591Ii7cWp8f>Zn&J-s@Y^v9GBZf(RHCz+%&5|PA7TFt+T3h$x%76)LI}X zH)l~M7AyaYJ|ns3RmS0IEyaa$mf|%}cCZHZvql|JRc*Ff=1S?@iSpT0Fzy6eUiOwe zGEms+*G!9AJ)5^7shqk=((7X+~Jok)} z_gdVUm&G_LE=lTVAU=I?A&ooAPyPZ>B)6v+4J%q-n@kX}2gbF+p$}qx6#hzfuE(SJ zF))>Y5pqhGNuh09?&Fp6%8%9VpLGsZxH5WjhbM%jAKNSwNFd+3`(SBS?4iAqJJ_hW zRPqv>l#m$<5s#^_xD5aSB=9NnbiGK5L;RzmYUz4x%xJjhuAP}%dE2obf?u2-=@3rK z$vY%XQo;s~ANQ+qJ|eeUeRHVj4Q@#P5iRbgcddXrbzD@Pdd5>r5LfOdkLY~!#r{%P zppL!b&amva!i+ieXv;YaH~!*}p?Oj&YSORr^4i(&?6$xiu5n5~bi@dox-6Si<9@<( z{5ar0qa{htA+b)Ki(d?a9+{hfT^5pFbDj$?g$?c_NKcff-W99T--)v3anEjQM8;93 z+#hZyyiO|52S4^a7{zs67ZLi6NY99&O@1$rJ6Z%k9I>{!l5Q|Bv-Gl*cB8k%9TJ~$ zWi6NH@sNF$;dz#g9wZSe~w9fEC| ze0_1hYCV=!BoVnAZm$NN%}ziWh;i5?-frHs@I?ug8lPrV=R zpYxiCHxWU^Ub$x~SI?81vkFakw=a&1)QfEj{u#e0(y24UfDfV`R>(Z z7|KRsJ;6->7WL%t^o^=H$ZF^H8a>b5K)cJKAFanA`L#&f&28z5rD(7(GN~YR3d(zm9gt}Kj zsA90J`Z`24+I{kjXL!gR$8NjS3w}o&ky%T{io{n9#HEX(?RLm+<`|I4-yhx^c}_^{ ztG6$+5`e^Cm6AHOa_^)k8#d{Q_Bl{vKC~g00pKr|!%+t@D2QB~IY$0-f9qeih5;_U zoOaejF<}|f*#|+UR(s?;?0kTCh248yp_FFwbEunzXH~8@{3r$FPY)Sh@tok_cMSuA z?8qogq20Q7dbwUj$@;q~tZBu=y<+pDptpIN?0eTU*h#@8E;ryes!vp7y;7%|p7PfG zp-eq&mhyaZ0kUOx^o6Ium$|Ad*7^fe`(Urut&N%_e>?g=L@hgzA2@8)qDL)s@k>|AkUu1nDkCLgRM^oC%5;S zekbA|-i8ZY{WkV)Zxq*0-uaJ()l~{OJVx_{e!9!B@IQ%kwzVc=d z(|{=e@KVJq=R=iEq0Awr%D5h=^#O{fN{EKe+>PZ!ZWf}uA{A(Nh$8ae7u0tx!fS1^ zKD51mnKI>B5V~KBbMGhkqdS5l!$@#*HVd5~ zSI>vNPxq#JIA02K`5R!r4i;oB3~~9@u;u(}NNRon%0d)ecfk5w>%x+CP8_gYps|AA z|A$KVt*KK`MlaYC*fKVzRZ9`(Eh}UEw`e=sX}7d!Hv4(jLLIOb9iUL42Gh(>;ZCYf zY3WS1KU0>M0~H$6=qn2lK4w&X+YD2(xtK2Zfo;|msCh|@!!v_%aZu_vha9)9xo4L= zI@b&1)kkUJqNY?2PviLYad|lQo3r*(Bv76M!yi!GNNpuV2%u;xgR0Q#Z;3aJ^P}{3 zy`6HP*0hTlL-gQqZYJDuBF6ei|I0|LMCQRnXREtJ%9HS=YhUr6d;e^&A)8epege&57&xdGdpNT#9w-3kra`AhP-W&ZE%QMbet@9#!uMjT2r`vrPj^7J>COyTSvfKk6dUKIA`>Uz-r23(71?hYRd{3ZD zzL&Vmhwr4pt)nxynpIDy&Y;h@Y3$U}UPl%EbI*{4BaYdhJ3)w4Ddu_)(m3ptGItd_ zJ~65Uc;^0p2Nt-V+W!oeOG45L%jw-YH?|6;Z==uTgaAra?LR415fkYo+LQ;X8_>0u zOFwHJf>Z<#7EYIu2d<^@!^V&aHdRH{mD?YEgq^)wbo|puv=ohFP?6O9)3my#*(X0QPDiLOYv?bz$E$C(ew=s=D&<(6OuD&}^rZn{Rm8RVq$eG&OnJ$AW%DqK zQ(M8YB$o!E&Nn~nN|%sK&4>t#vGn`rA8T^nGu8F6BuAShB%rQ{HMKsEKf1v)HZ}X{ z9Diz#F|_WnUZo^=@NeE3KXH*v$Wy)vnbaCvFxxH_$GsSR>24>~nC9<8d-1a|QMHo( z;=@6cu^VdIdF<&)NAi z=;D6A$1SilPejx! zrk1U5OOv}ZT;d`I=9Uv%dbJEQ*}52t_xo@`s`bon-ivgfXi4?twt;rv@T>y~*&qLu zFIF4IG_^}LD%;Fvbt6i1Zl+@knp|Kvvl%QrqY5bE-whT-uQ5tphKCJ~I5LKJRMm!G z$7=qlg#TLk{>5Iu7O4_EK$RALsFQ+AeIHpa%0u1qY>*?={Nr%bKQe0+7nLE#u6NY- zcN0$|bo$__ohGKCwWY=uwWiHVh<(i6)Pd{2eR(MdP1o3v`g4Z%HgC?rZ!pMV)2tYQ zP^GkxG-CLT5{(-+;!{ngsB)U?3cPOrpIMGDae1wIUexlh#@8kv}?&u5<-Dh$lMTo1RwAdHQh87j|kV zG5*pUlUYC$CN}IY`T8{|Cb6Gu@ycRfr zWgjM)pcQhzC*>H_waJK2-nE*ILK@XLT$RbS@hd;i4+mIDnT!AP%j*eJelhp~hL57G zk)_uQ%0;1IJwEwO+*?C_#JQ>iiMm&-Bwi2I0(oTuNd6aR&Hdpnid3F9{F|_E+Ozu> z%)c-c**hBwxt~ZqIv&*gHl?@noO^%oSF5r;}ZcRLqUl-3u(@-`Q6+t(`~2Z;)Os{x;VHyc>|tfaToWe^k#v c;%w3A&+lbFpF literal 0 HcmV?d00001 diff --git a/docs/features/unified_attn.md b/docs/features/unified_attn.md new file mode 100644 index 00000000..3f5a862d --- /dev/null +++ b/docs/features/unified_attn.md @@ -0,0 +1,104 @@ +--- +title: Unified Attention +--- +[](){ #unified_attn } + +## Overview + +Unified Attention is a new attention backend introduced in vllm-gaudi plugin. It brings several benefits when compared to previous approaches: + +1. properly handles shared blocks in case of contiguous kv-cache +1. allows running both prefill and decode tokens in a single batch (aka mixed batches) +1. allows running all query tokens 'flattened', i.e. without the need of a separate seq_len dimension + +The name comes from the fact that it combines several previous algorithms into a single implementation. Since it's a completely new attention backend, not all features are yet supported. + +## Main idea + +To get the main idea behind the algorithm, let's work on a concrete example. Assuming: + +* block_size=4 +* 4 samples in a batch, out of which 2 are prefills and 2 are decodes, with query lengths=[8, 4, 1, 1] and context lengths=[0, 4, 6, 4] +* we're using scaled dot product attention: +$$\text{Attention}(Q, K, V, B) = \text{softmax}\left( s \cdot QK^\top + B \right) V$$ + +![](../../docs/assets/unified_attn/block_table.png) + +We can observe two things: + +1. some of the blocks are only used by a single token, and some are shared +1. some of the key values have been just calculated and are available alongside queries and don't need to be fetched from the cache + +In a naive implementation we would just multiply whole query times key and value and use appropriate bias to mask unused fields, but that would be very inneficient especially for decodes where usually we have only a single token per sample in a batch and there's almost no overlap between used blocks. We could slice the query and key into chunks and multiply only those regions that have relevant data, but that's currently difficult to achieve due to technical reasons. Instead we can divide the work into 3 separate parts and merge the results at the end. + +![](../../docs/assets/unified_attn/block_table_annotated.png) + +## Splitting softmax + +The main trick that unified attention utilizes is spliting and merging softmax values. Softmax is defined as: +$$\text{softmax}(x_i) = \frac{e^{x_i-c}}{\sum_{j} e^{x_j-c}}, c = max(x_i)$$ +The problem here lies in the denominator as it contains the sum of all terms. Fortunately we can split the calculation into two separate softmax and then readjust the results and combine them. Let's say we have: +$$z_1, z_2\text{ - local softmax results} \\ c_1, c_2 \text{ - local maxima} \\ s_1, s_2 \text{ - local sums}$$ +We can then calculate: +$$c = max(c_1, c_2) \\ adj_i = e^{c_i-c} \\ s = s_1 * adj_1 + s_2 * adj_2\\ z_i\prime = \frac{z_i*s_i*adj_i}{s} $$ + +This way we can calculate parts of softmax and later readjust and recombine the values into the final result. There are two other tricks that we can use. Since we're going to divide by the global sum anyway we can skip dividing by local sums followed by multiplying by local sums during readjustment and keep intermediate 'softmax' values without division. Additionally since readjustment is multiplication by a constant we can utilize the facts that: +$$(sA)B=s(AB) \\ [A; B; C+D] \times [A; C+D; E] = [A; B; C] \times [A; C; E] + [A; B; D] \times [A; D; E] = [A; B; E]$$ +and move softmax readjustment after multiplication by V in attention calculation. + +## Causal Attention + +Causal attention is used to calculate attention values between currently computed Q, K and V. Since we data has been recently calculated, we don't need to fetch it from kv-cache. Prompt lengths are usually much longer then max_num_seqs. This means, in practice, we don't need to distinguish which tokens are used in prompts and which in decodes and use the whole Q relying on attn bias to mask out unnecessary tokens. Since we're using all query tokens one after another it works similarily to merged prefill feature. Here's an example how the computed causal bias might look like: + +![](../../docs/assets/unified_attn/causal.png) + +One optimization that is used here is that we can divide query into equal slices that use different lengths of key: + +![](../../docs/assets/unified_attn/causal_sliced.png) + +This way we can skip parts of the computation where index(key) > index(query). In the current implementation slice size is constant and is set to 512 based on experimental results. + +## Shared Attention + +Shared attention is used in cases where a context block is used by multiple tokens. This is usually the case when we have a prompt with parts of the context cached or in case of decode when multiple samples share a common prefix. Since shared blocks are used more then once we fetch them all and we multiply them with all the query tokens. Usually the number of shared blocks is relatively small compared to whole kv-cache that's why it's better to fetch them instead of relying on tricks like contiguous_pa. The main difficulty is creating the shared_bias. + +## Unique Attention + +Since we know that each block is used by upmost one token, we can use two optimizations: + +1. compute attention per block instead of per query token +1. use a contiguous slice of kv-cache instead of fetching individual blocks + +First optimization allows better handling of batches with large differences between sequence lengths. For example, if we have two samples in a batch, using [4, 12] context blocks respectively, instead of padding the block_table to the highest number of blocks we can use flattened list of blocks. This way the amount of compute we need scales with the sum of blocks_used instead of bs * max(num_blocks). This is a simplified diagram that shows how it works + +![](../../docs/assets/unified_attn/unique.png) + +The main difficulty in this approach is that several blocks might be used in a single query token and thus we cannot compute softmax directly. Fortunately we can utlize the same approach to calculate softmax in parts and then readjust. + +Second optimization comes from the fact that in case of decodes, most of the time we only need to fetch the block once and since we're going to fetch most of the kv-cache anyway we might just use a contiguous chunk instead. This optimization is optional from unified attention's algorithm point of view, but currently unified batch creation assumes it's turned on by default. + +## Merging intermediate values + +There are 3 optional code paths that unified attention code can take: + +1. causal attn +1. shared attn +1. unique attn + +Each of those code paths returns a triplet either with (local_attn, local_max, local_sum) or with (None, None, None) in case that path is skipped. The last step is to combine partial values, readjust them and combine them together using the previously described method. + +## Unified/Mixed Batches + +One of the main benefits of unified attention is that it doesn't distinguish between prompt and decode tokens and the whole attention pass can be computed by a single function without breaking synapse graphs. This means that we no longer need to do any kind of preprocessing of scheduler output (like sorting and separating prompts and decodes). Which code paths should be active in unified attention is based on the presence of a particular bias tensor in attention metadata: + +* causal_bias => causal attention is enabled +* shared_bias => shared attention is enabled +* unique_bias => unique attention is enabled + +This means that there are 8 possible code paths that we can take. This is reflected when printing particular configuration that is being run. For example phase string of "csu" means that all 3 code paths are used whereas '--u' means that only unique attention is being run. + +Most of the model forward code relies only on query_len. Two other dimensions come into play when calculating unified attention - num_shared_blocks and num_unique_blocks. In reality, when contiguous_pa is enabled for unified attention (which is currently enforced) num_unique_blocks is kv-cache slice size that we need to use (this depends on max(block_id) currently in use). Last missing part is whether to include causal_attn or not. This depends on existence of prompt samples in the batch. If there's at least a single prompt in the batch we enable causal attention. The last part beside model forward is dependent on the number of logits that we want to fetch as not all token logits should be passed to sampler. This is usually padded to max_num_seqs, but the code allows creating a more detailed bucketing scheme in the future. + +To sum up, a single model execute can be characterized by a following tuple: + +*(phase, query_len, num_shared_blocks, num_unique_blocks, num_logits)*