From 7cd2c2be14a320122691463b003b1522f47b893c Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Mon, 27 Jan 2025 15:51:33 +0100 Subject: [PATCH 01/19] add google dataflow docs --- .../assets/dataflow-inqueue-job.png | Bin 0 -> 50837 bytes .../google-dataflow/dataflow.md | 31 ++++ .../google-dataflow/java-runner.md | 20 +++ .../google-dataflow/templates.md | 30 ++++ .../templates/bigquery-to-clickhouse.md | 141 ++++++++++++++++++ .../images/logos/dataflow_logo.png | Bin 0 -> 8305 bytes docs/en/integrations/index.mdx | 1 + sidebars.js | 22 +++ 8 files changed, 245 insertions(+) create mode 100644 docs/en/integrations/data-ingestion/google-dataflow/assets/dataflow-inqueue-job.png create mode 100644 docs/en/integrations/data-ingestion/google-dataflow/dataflow.md create mode 100644 docs/en/integrations/data-ingestion/google-dataflow/java-runner.md create mode 100644 docs/en/integrations/data-ingestion/google-dataflow/templates.md create mode 100644 docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md create mode 100644 docs/en/integrations/images/logos/dataflow_logo.png diff --git a/docs/en/integrations/data-ingestion/google-dataflow/assets/dataflow-inqueue-job.png b/docs/en/integrations/data-ingestion/google-dataflow/assets/dataflow-inqueue-job.png new file mode 100644 index 0000000000000000000000000000000000000000..8c56eece62960bf3cddcb573fcdbc51af2c5efec GIT binary patch literal 50837 zcmcG$Wk6g_wk`~WgbtPlLLfK<3GVKVJHegc!Ce}c#yz+PcXuZc+=9CWm*B3qGxwc2 zGxwV_%%3mm&EDO+yjHDR@~jG$mlZ=n#6g6CfkBZF7gm6Qc?E$!4**_4|Au9gTtjcL z_6lMkn6eSPUFa8mBQ*(Q85tNV=raHYHqaCX{!bFEG?|;xm|)C2=t4 zB~g=+Cla!@Ga_Q6W1wRo;X@=MBI2<#H0Dwe7X2$b^cye97Y7F$E_!-rXJDB+6RoULiPRQC4TBifw|4hwang99a zUm1Dm|8)JIhT{K@^WT1j4m2Mk5B)zTjSsP!MwSZ(MgT@aSWw9Y_8<+xT}cG1n;x?4 z7WmWcNV!NUHx#i>4()q@K7x{xKu5q!ApFZ5DY3Uvj_-&jR#%EP?77|YE zDH~dzYqj9Btq0vXcKfYmiF1LZpW@!|&whT<`jFnCLVhg7tOrNKypDSA6GWO72(*5I zP6YeRCujv8z4vHheeS<0l{A>rH>KXHKdzX4i6CM1i4nF?C(bXFrE%vRZF?BbPx2qI+s=VtvT$Ae zxWk_xo<%a?hd0D7ufS$VPp>f`IBaGm&i()v%e}XnY|T#bvFKz{bMc%QNiYo)QXiSH z%7~w9q>VJu*bAcw_C_7I+k=-g!8DYs_=fPAN`(sj{V^0|EG$a-UuN3;YRsP?WUMA> zXnpCdG1bG1qHv@F&wSY6JTW0Qli6aCeb+0!;^R3In~RP1O+?`YmNY#EeJV^N9ApB5 zeUt=Nq|Y_70vPnEk3))&&V(4gRHqSy1Oob%7`FseM07^QA#Y1OOIJi&@a}W~b$_C_ z?AFWf3fU@_Nld1`bi6Hos019Jk;nq$(t|)(hf5PTtL;C6F{HADgDnOhlN-IB+SEQA zy?pudX8$K2f5cXC;w#F!L+tT&oK=nI$r;YiTka)s5FZfoSrpJYX$JJdVMAwJNBXV_ z@Hz~L#y!8Nud@wW1;?^m6Z8|}(I^)y#fQCT)_Ak&@OXcN^p!?kK_Z2%bi!deTr83x zCijUhX36ccZ>ib&f;=`8N$rX|J(m`KX39uc-Qf5zAz|2)YlRUSz^e2Sq^Erg?MWR$U|>mSRC7A$ z=5#A=Z>o4;iQ9Q|d~>kM7P3mu*aHUkl~<^H?X zcO-+GBHIhMkX;CSAeAgsqR5|kKM_=iB5TZKJn%7&N~I?E%VdR*bSjH_-f$wN(O~=- z@enE=&&RVZ`luKRIf;1|)0~h~q9WBRv_`wl5b5OmK&>VR2{!9FDl9s!*jFff=C)sk zlWVKoH>p}_L#fuY!*Rq5zmgfvlr5cJ-QU<9E`l2!_v=cun`P(g%|A5SZ-x*7h{N!B zg6fWD3(vf_r5{`!$;_uqBzZKbHS4Wnwllm`6iDUhfQg>?yrNY_>osOK4SWevAH)oo zTV9UWm>Rqu!5XU`aH?`*rjSj2>%tvJsXbs7>in);yP15#WF)$D!MT9JPEeuD!qjwP zk5M9;#Q!%8us2USjrdqeGK08vvOq;dgT-8bq0RJ6C9c zWp4%pAby5*pDpq}%sEEuH4_8iePXW*Cosx*D7jQPJynAPB;H`VPB9Q1Lt(jFAvaDc zan2XfLwCD>bfW>0OlAp2&8{=QvU-2B|AiK0BoFHUE)=*Cb?fI3!Hf2|yBut~z}>j~ zz2J%up$G!aRU1tV9^ThW2Q$=~so1*>5jvl;n0Bn*Wq8JWC6p_{DeLsdQ3uj!wI~JL z^C%or3J%WpJhAmxVN%zs#R@Y%7|cs26{~g7X|;UhN4`;4$dx3cjl-J2kK2~7s;pc*o+ha_@x`Z| zee3oZ%Y}OClbnE-Z!HeDZ?NA|8fsm!>#7paa60nEQLBFS@oS66A6VEP6Gc%@=ghHw zf>n?M-zybskSUB2xzlL2X_?Iw6?WFw{#=mD;QYXk@}4|2fhKNiI2o}Am&<1LTkJ<2 zE&s*^>i0~BKi7!`!#cSvx?IFnj#9)Cs!AH1kJ1t2>Q!7luOo~~uZ&g8wdvjBzFcgX zYv=0w+i|VO6;Jokw?{7>sg-i8E`EPWI_lk-SrQ}I?rS_8Oh0?vDQjpGR2u5cKen} zP2TN=AaduuC)sVs<-uHW=dyB?cJtNF4ctq_zRLC9S{uW`j_#o?+67Xvh=Z@|nb=z- zjlC&XNBl(ZnT;!g1^d1khnq23{WA0*5Cwv+*&)7u-|dS$m$ll^b8Rv~zUHSB zKhYElLEyqqts$|__Nf$U#7`fd+oz5by+`3QcU)mP7e-^Mx!pO0L;1jV+bRYRsWG~u zk43!xdmkg;LJYgG`6QGY>I*EodY%lMaQpxUCra|2FHgODBl`RboPsMJZ!h*;-)D}M z1Z>7oh2fXDoai%|U*cg-AJ(9X%;d>Xw|Y+G4J$|oQ(>zXKqS}8^oG*8YR#Kc7E2vI z3jGq+i+)*NbmBK&aFS5+>N~==!{(my!XAt1hdhZG^4eu5A%kwb%agyNX)#8gmAqWSz2$m=I1dZr-!t+rFH~Q^dyTQWEI=t16!Mm` zq^pW}=zV9QqwB2%ekTg?I#Z!7q$FOP_A?KzLGzRgKVWUTaD76dRYTcy{Q49A-jIbe z>u?H1q^$3;b&q^N7jYn&?sA1C5CY`;T@F4Yb=;fu^#D?#u5GD6sRCe|AmTq`-wP6= z?_Ykn%az;=IV5#Xe3km@i9sp6(LvF=-EOl8y>o<`n3m>V=R4fZCIs=u>1F_P;n8F? zJ?3ni9#Hpwp_x6bUOiw219|P}GZuZoK+Nxj0dG%_oU^C&m^8DJ9A3ApUCaB(=j5`E zJNPW8=#yWSD&-o)tY*`7n>#~DJFDMc)HxhNS%QaXJk5hD(tBcJV!QKM#{Q%afe!As zlxZSO4!h?Ly(8KRneO1Z8gO-o-$%u~3LCrGM0$>lQy4BHTsEu8JN*o5RVQj#vE0GL zPmk&Bx}lO>AJ`uU)3_4|yuQqzzqi|;w(KwXk&8*KYRZ_iX%M-KSAn{=)Y;jsJW0R= zKHZqs;lNe|tiNeAr&cW=Bi@E?M2ClT(c@%t@zUYNH9`TA_cCUM@-~C9y=d^oSt21j zHF3(~&5d583D&QkNG&ya%XkwX6ZsU!sUp82##dOIjHJiueGu)>M%gc}aHcTaI%u3n z0_?`qXpat}_GQ>Vje9lP zZB3?gV9WJk|43jqyNN8E$kkq$R=A4h;u2mQ?eZFvpo*o>?->o=Q8v7EWI;hh{5nGa zAlD<4_)#%mX6BcKD;ZO*^^8hj2=iTs)XwDYM4cnxyGn~1Bdv6teCFgLYY;nq`*llm zKYkvPO4jUz9B-yb#;|6cS#+Mqf{H?u((5#W^^X@jk4<5>znyOm1BqR=3CP@AXR_Vk zBa3mMqEmlN{>Gr-idl5AOC`yvYdsy8$8QZ*!>8{;Oi*Tf`(}HgWH%d)Ck*dh1og9$ zZI%pC^eK$3wLBg|Acx1LuU~Dn>USYPCy)jz=YCZUW>X6wIzOdeB6%%{LTE{o4gf_* z?zjxIvc5v?xfH}^1E575++N_CNvFov$p8Z1#gd9M<;YoUZto72YSwS;O__h?v~Ljm znh&1MA5CZFDbMbW7=%bDZumK`)XBXoHCjZ^kDI{+O^dLOpo1pKWK+e%uHqhi90(*o zOT}=7pSmF83iTzw(%T)$;HRIfGKw=CU=DtsCb;1|(IE2fK_G??$a)06QH%hAHgw-1 z#$OYYRDkaMvtNzR^QR1NIiVi)P;*eXUYhdTCh%Ea%h-cuh6}uU#E&-yz;eMpJyYK@hZ{=s@{j zKY}-*DkYsd0gnq&Cx>A8B(@+5F6;E2Ov-JbbPAWqlio`a;slPVS>9l63Vp=q8|orX zB z$g`R%bV`odU|j*=l#gd0d^iMPKsM2!@3!A^Xh{BJRo?6X(o0eDA;0IiI8=)yvUx2J zL|ioQ`ue<;z*`j{4?IN_H3mJWRSCfPNO}9puj=5&hdtkj7Ld1Zlqv%g#`7}|(1Lpl zfxDWoF#Inj1{0dL#WytzQ4L)+O-{YG_*U%MZ-YlbcTm)I9Og zrW{4;S#~7=r$Y=iWl&=$y?(xD@N}?Is1 z9q)ZeN7(Fe|5W0VA|NgvRAW9Po9=eV2|gQU=N%yMm2dZa*w{?ajju7CaIK-o2t>m^ z+2(wIvaKFz;{c{9hF+m!cMOcLqXf=hA9IHDaUV+j3;8aSdRw1`yOkHV?{h6kiYQ9t zW)*&zjE)X)G`3cV1nW$5VV2Du6MSbJjph9D9?nH)?9$(a^&m+G(`4W<()U&g=$dI? z57=gK-C$@v*u4cL+4w-$l`KXBl>b^RHhjWi`2x%z{ZRI9S&8_RQ2?O zmecdH?(ImL)~TV4aaZ+SpG>3Yew&VL98cxj;S>-$MU`dz`u1VpZm4soDo?5q!5kCh zNf@68EM>LqNkrzSI6H2|X(M#Fq>AAD`Np0!2DhG|A+G-6kRwCpu6%!8b9D1|_^wL5 zbTo;q@NHi-Pp8%NnRE5Y=nT~FQ47w&m~B8 z>ul1eJw|kzsGC7nl7;t_r1niVDp#0RF=CA%O0qsgEh8Iwd~T&U;_?CZ;(?Xf_m21x zh5KuL3Zycm-DkNqo8~ex!kfA(_`O9B_=sU}q2BiQ2< z3RjYhj-$dx>`HCTN?clzVWxf#Rf*|hO1-DsuDU$=AHAg0mG^nAj6RX*B1#mNWiZ~E zV2%zSX0s1(iBF8bR2Z21**xZ|La*CSaTmGr71a1u>UL6Zh7jSnw>p{JNPodT?2d)Q zjj2DD?H*i}{zYlD2#jj;gr$o2+p^fTcS-q(qvcA06v-saVGG>lK292(Q{cKysybT8 zc!n1{lA-XGSIcXE8I%-Twxyw45(K-g zzYZQ7Jg$5eN^FpE{#qE!2FTDHgcfmgPskP%iAb~!W3e*a?OSVj`Pxe^qv*@bFZ}TrNu+Bo?b0gL+ zIzA$f>6vl{jC0<`+bYjA6??9Ha%Nb;;jnhT(QdF)wI7Hg4_?(o?|maI4 zGo8|7?2r!OEN0d$r{0qlIU|oew%k5g9?IZt?BYti@vK?4j@Vn6{ymLR>DjfO;(F$J z5niy@cr#V~xhOJk@$!o5R2XOS_lvm_M=C9PXWFI<(~YB@*3YMdS$hXJN@?E{8xKX4 za3wXGO=dpv8S`({KV8dk=I$LeuuX3-Zx4r-twe` zD(JVjL55}6nS~U6dUzu2v^&cXqZ1h-GIJEsms@&Xd zSryDZe|6Z3Fus`r1Xp>IgOBJR_Z&pR-JJB@eImRx6&m^9QQ>*(WS)I9KD(_jY5-!xcZHFJe zTvp0%d#cYm9jVP2EZ-7TtORVP1zHd6^W#MemTyGA!t87jeYhyeN=Ulk6crNfUyU6v z{6uc5#hFG|R%|lWf<&}4gsVpCTTmrY`Q?-}#uNxnRh5e+| z+Xoh}E~6XI`S(dDBE*_PQ@cpc=d~`y+D_oDF=Zv;_A7_js!_vH zT(-}_eD@Dq2ag#)I zF0~)@t7%TniYu&bEjo@UJoCpWU$#%ER6oholw&2X9B-~*h5jsj1x~b_zGpbvN%?Yn z8Vda{$ywgEuNOSk3ut>mP#&#A-+YIkg;D%;&~BtPRr@n@yZ60`p~_09Z<8;lwd(!j z^;MSpRj+Ad_MX<>nS`_V{xnre>0%yFC2#E&oggpDiWh?gQcC(1C`!lXG~>Zh#3XHC zzjk!nKVu(JHp%+$?=A}C_VV7 z-@0jQD($y)J_A`m8-kYye|tY2QQlV5H-_V#Dt3!rR*1ae(VCGA?$WJ;($UKp zLN0BYeZ{`%Xf#mwT~}m()ZtBzMsS5Q>HwND-)w-aE@2o4-2&=Th2zgo&P-aEH0rb^ zDVNc1=W4$C%k7YT0`kSC2ycuy>=2$M%%z{APSTFaqJe4-VXVeWgh$$U5&p(*o(>!} z4)-EDebv6`u$8*f;l5G+t7+WMq^9FRt$Z{o4@_=H%e+$s$!gkK^jg~Ov55%8ut&g- z)^`1F#2ct#c;kF`LLn3M68w0xP3E|FOekuf)9idS5QO%@@b`QzQ~@(KoyZHHEcn!A zF+G9HZq--tDZ7RpJ$>J7s%V$NvGO5I?4uzJt;KaCgidC&ta*`cN?BQO?{WRpQp!Cu zS;sdMPO5Bf$_4kgSDfH#OEy|`e12!m<%gR@ho{|wu!jn7nPbLnu*_?wiFobUX!}u# z8&F}6$CAxhw`&P3{dRnEZfAEgX_4U7rI62l;j5&Lsd)9xR4!|!^C+xq(BdxpVa_4= zd_CnxNrWV>WWhtm>n&=ci==C@-P{XaxitRrZxq9*!uP2ck13!qQYK5kFhiI z%qx^NpwF_x6~tw+Q&YwO}JibC7w6X8a5n-|^o*tPUBtYH>_BzY{Sj zdQ8bav~)%u`dWness-mn2Z!sd#JzBtwv8dxoN{yQrH}LJcV6xZmt-i<^lgxY zdEOT8TaAsV~#uDC1jE+ zPJ^AK=?-7tW!7u3p~yV^j>0aVFLvcU|0KrGq~f|gqU(Sn%{8~({r-HTa|-_qzbo4vN$*nbx(SpEbgZjB#OOO4fxswQ*~?TS zGrj_apxsF0+!az>n5rK64^m0?I`8R-Z7Z8>(PY44$RUw13EW%i(6(Cex@a$7uN6G9 zJ!R}^nGTJ$8rF697IM$5q}FqH$}8!AJhf4E=sK%)nTP|VVd4Ldtb1qP3ySgTCehPF z4d-F7JrP+l{X87?3k{y!UMF3awj3zu9L~S1TB<`k%KOAmsaWT$RQ^e7+3o4YYZS*t zm(BfZ&-LN@oU7i9>syAQ40yPRj!^oxj~DwhGrm7FpPky&ku>$Q z3t&2wQH$^rM|5pmr`=s+N*w3WitR4(m65wv%0sE%xHMPwaB_Vu@~7cZrSJ^EOaH15 zq!=Jc0S=#?$`2a>mhATFfUfBhGquqcYDt#R0U+E!>}$o?6>J=rN>shcbZhiWJp|+9 zpj%B=ycwmq0vs|$^JruO)obrS?-xhp1_0+u_XXu8zwVM3{N~um)dY$zl!>R~)-1Hfdh9Mv1C(c^-$2XCHKxsG19xldl_e z>NZ38a@a?@VtL!7TQ5qI-4>mGPw|?8B)#LZO)wW^TmiKeZ_Ui6lip~RM23zQH3klT z%*-(wKKEU25K}&)mCH~r{(>l14Bn^Xb~ojo7(;-t0cBjy(D zuaDICvQMp-Rfp&PN1kG797F6e%@=TY>kYT$O|B1Z1^R$a!>$*vH$7Wr*p_@ymI3MR zf!G9>D@=LFc?3XZ?LiGE)%(06S^FTXDFhP* zc6D02Q?U|BSg_j`il`%jclK;Eonx)0dx6Hd<2cM$no1Xz)k_DVlx<_H!QOX-#0NwS znX*s)_KgbX%<{X zoT)4ox&7Ll8jO#e`pUWlQg|2~o#@pkx&y&MZe?Y;*=Ila(j9{8Mqgy%a`7d%Kc;nY zZ?4UOG?AB6fY*3QbA;(mUwQ?5JLNGkduP3hv`LFRWRer54Z4N;!pPpn__~>|@VZ@! z*ISiK)t8m`ee=;B=nl<1-SqZ3-7_yAoMmj6_)7XHGP(Y{o>aZ+IPvQ;m)w>RoR9Q3 z-F8nVrvo+GJ%_zX2`tBjI!lFSC(GUW+F}}wnvb9pBrH0br-!SI%RFx9!&KR;%LQvG zXd*Hwx@DgE3lsprWWOct;{w0GP+#tTf6~jO*9q@)EgZ@T8>aiBSc}HphR@w4?1rIT z=V+(Bc5fYlKLVTmhuIfo5g|UBQbfV&0tK}`vU{Rmf#(;~b4o2sIaEsXjJ&S`KG=Rw z&T!-_YAH3%N*1$jZWXgOrN=ezVs$m|=zyVXW=NfJ-z>CKX6^eW^k!`%nFkp^7i0Dz zP}Bf`(}NdI?E9wE|4DmVqQl~-GNlRO^VF}kp-ME~lw;c;t;?LjEtzQ8l#3PcIk|^+-O~A1rnJE*i#fEY`}#t? zHLpj~SHO4q2li-hn#t6nkW5idjJ5L4+o04SVpO@dpT*9boSP>*d+HA+>qht_qV1Z> z_s)=}^S~(uV#ZIw^LEDpvF;qxzSVbK8su&&)0U|ef2lmwuoDGk@YsY-C~JfA)hGDN=x$|7E1n>fC@GjLtMn!{|*HnG~5^_@OZW8_o(2F`9k z$Kp8=t=JAT3d;;-bYpn@I)%5CVtfb2|$dxB%sg!;zJZx70^%lunuC!?tOg_R_rk4CZ%8@%SR7whr`3S9q7Uh9igOV(E4fr$OP4 z4I^D{@caAwWegCHgChjFK`UW)vAc)Tc|Ri0_L3G&uXKAAAX;}@TGxT~kL&W-deZrz zFyeq7 zPK=4v-KC|`-sCRTqLOld3Omj6FQ_lTuqP}Dis3QnPh`v{a-V{F1`HQ2e=B4O+ZO7U zr-%3XzI+n5vEkd?tk5kh2C8fh#7FxfypgCjma#NtvsnB%Q?9M5gX*|3S*ZAxNm;T# zfxg`G?(#Rc;pTwyS61_Prej%+q;lz6sCXP7zZ~~44aF8}H`moccY&>Y%3R5~uf8vl zW>0@1zlJCAIa%*DKf?d;Dl^PzAU1kyIAt(LBF1nqo>rkui&Jx~{ozuR_LZJUJdJve zP$24QHV z7zD6W7n8on@(CF)uGfj!$h^K4Tv?!svqjP3>4f;(Hk5sJ{G1wB6zA09Iz9gzjG%CYtp{P=(eb)xcS-yJ`-|VLc)UtviipO3hrYMijdL%WEX?! z7MNwIWVA>fq4Mo0P*^q$4MwhwO7e@y7a7&dJ71));=KF4%A2sA#Sb0RWtWkMmvEBE zkTiKe30!+8FCrPGlA*`czfeHlygoK^C`bNbcVhk(+HI^QA8;Z9Kf?FN1UzklTr>)} zw1_t5*{N8HTGtkEe3418zkPpW5Kp({-pv+3H14KBU%?;1eSrnB^e19b4(~r3s3%sC z_Yk`}NECX*bky%0RPU9McH$>*T*}>c`G7{NwRV5PyHXUt@9NDgK)1O>zyKa%J-~*r*Qp47Zr$b)3zB5k=s8Kx#0q#a;x()l5uLe z$u0i4PEZ6FglmvOiI=~^;!Q83=!5y?z_Ht}{KK^)YO$-Y7rZ7MYWWe^`pw)WB{x5M zc;E@{zo+`!3FFx7HYi3qInj*>vj-If?TpKc#D71+PS82t=*aC zJ1)qqVA*revx@F7nzIIv5tAIGLkIJ7sdzh z03ln2qFZUrP6HH#SJS)kw3>7;5Il10YB9pxPJc$^G1~ZOTJ$6TWT1L&W<_)`Y*+E+dbWdCaUhk6Hg69ie0HSkWxQ6S-L89?RwIe^ zm5b_QDzoBt*ONGr{A;2X+#kbND_;BxXqq)&GGAtDH#>;{A-=wy{Iiu9>8b!w;2Xke zFwfZ#^XYFF6Vh%TM1=`_rXyzy)dRZLekGC~Eseal7LU%cbI_jgSzuA$JyI zrA!#0{_mi@F<(TaEi|?fGcR`yhE6pSV_Y23FFruza-gh=TU9LMnQ zmV}w`L`6i4s=Wb8wYXl7czV4;C3gF5b*BXtXgx$DL=ffea-rI7SU$;QrPurYU&e(^ z=Idldcgs3?IGBQ*3pTzxE1xO~L7Vi?S9MN-UPZXl>Yuv-rt@t796rTXy>+5v27Pm@GFqf&f2$!0g9(9O7!`k7A=znHBxJTgeOUdZRH_VWHn8Zm0Gk6J2Fl$sl*6#-%&k z{rd5T^B1V4{ha;dbmj*Dmm^D^@T=n*9dGxNE!l^cj>M4o097E@SCk~M=(0dk`^Owm ze`6VQ;t|mLiI007p*$Y5$kWf7&_*Z+mL)*HWt!Ee=v<7Ao zZOaSlY3?R=#&;i)33$Hru^In%*(qxKeSnJ+&Hk$OtC$;Yrp`+rjA-tX^#|Ff7I{e1 zt)ea-OLx$WqVxX5VH*|<(hOW0FY&EDvSD%K0EOD?v5GXRs#h%ucEy*8R8)3-r=_Yd zb$s&hUzy@9NvRzT9EfB#5WP|#r7XGE!v~c7`lbx1%8TrGSIWXA5&7n^)!5VM*o@V} z`AM}73!)W#va=JigNsdlTe-K4HE19L40&y1edxB)ht}xyTO;TDj%}D0r~SaZvsCh_2pJ2G7S_RUN4`^AwiBtUC;N0tzIfLx>R3s4krtP)K*}Y?7xBgVlm_I;BR+kwp7m`gSqsQ$Ge+{5UyQkmj#{sJDP^`!(3M_G= zpMVZ()L$i2DZR011a1sLq9a2*<_UEE7ZH6a?Byl8?KXk$Yl?UCy|3svSrJycsY|%NbG@py zDxsX>B+PEw6vBJgyG#U3WCN1#Kivq+mX0MEd24{9m?|Jkd&8aF3>`2_=KUvi984T~ zsyE)~HwtFPj+psQ1I>tp%&Y3+tUTFYYxPE1D^YQRPl?)jsI>5?lo?{{TM$hG?K@ZF zu0QThQHnuq1G1A+CcZ=iHY@AviWD|VKvo&cG(w9YGIYYW zYrnO7MyKdz->81-pZU`!OkAUsBlK-N#SMopE34|>-iR}9Y8HIOE~2(?J{Uhzw5LUZ z_vv>Te=LGkWQ35H%>ko~THqnUpMro0*P{BS=PmZGpKtUfmAn%s zEXk_QaU8W)Uq(h4mF*#KasR+-AN_RVx8ql}?uHtk1FTtPg%|u7g&f{{jNX}X!WC5H zPCoQ*oY*3c*{{HlG3h7Rrs;NMKWDG6ksx5%sgAI2%~e#{In2uAE}Sr;V4Wf+G8QJp zX8Twv@N>PVvzqpdauS?rb0g|BQzojVFiSCqH;eJyZA9f^C>Xkk%a)NY*DmDAAs#)< z7Obi_-!O3YcwJUH+;U0~m%{lcgQe4ElXV@oOQlQB#)d~T?$-DOl7#C zJM|H^%8|_J^yCq6HZVFqxH-<&sWP1~rqrwpqtR**5C6mjviAD|6;K5+L00iPJzkWG zc~G3A(*k^^WL#&9hqN!7rN*GH*XV>V5TvDCt|L=r61lj512!2=*JiU`&U&ig2lfA6 ztW-pI6eqpdn~sCx!w_2_?`U;MUbG!hwDLa+yHOBBv>$FShKM=IVB0a&T3jm8sj$8< z<_Y*Pl)rxa>wK4i-esPJ5rv6k_z0DeEcOqX7VwYU$N5yxZ7+Nt_WpjUR+Zs!+ z%eOEOwPTy30SthG4xnYI?60DSFD0Hpr){yzf#CrM!1O<{2VNqrZ@pEGi{_6-0D|&r z3kc>gM6``}i}+MpZCF71ZL3HfD&PZ4$ul=_jIoQH5 z^^sfaRem>6IZu8&1>pJ+@EMmty@eB$1HnIEGplIDn&22szvyk7rz(Eu5-`k1J*8CM zR4ZxN6*%lBw`DVX)}DJhawBnp4Y@+;xfKxiNt`{gufG5%^Q_)RAf`>>D2Y?Qpdfn( zP~Ne9r_C%q{J0bQNg6DdX|fe_!MrCKIv22inH$pEqO9yY*fiGlGwqB4++GKceXyIwvNIXW^8HRab`0ZNDakKlrY(p;{j}K8t*b$~qJFoj*U5#oq z>{eN$;eLgYCq~*7(!u`6yccgGtlLo>KbSwfyr6TRFc=Dkj<^DTwr>qPhG*BO`tOr5 zM?k!Zh4#^i>&yra5A_1%WOvDBvjlC00#qI27A59%ngR3$c z_HrAtzki^uH{M(%*X&9!0I0#?V)LA2%om8NPA+!;N&HrS{4=KL0Rn5Cw1@I`k6NX< zk|tt0;UsJrXH&w@u_8#^$QF$rnn-B1Mds2Qu0KGS8wvP&m247}6 zFKQ7%he0O|dA!`4u_m{K=wVVDGEzT%ivCdg zyWy06U9X2y-jMNpITNJacj)L@Eht|HzUE3<-pv%N%8lpAm}}N$2#Pm;_w$Mh#S*Hm z|J0kGQ6-m^AcPvY6Xh`7GPa73hc~_dEgjuGYSb%*%~Y^Zuj|#R$jT!X@-mxxwNt5s z*BMISjUCzRYrhU?jE~W3htE82zV%95oseugkNRi=H10(1#C@tzQB0*ojd*)BLp9xc z&6bt;uc3}R3g^RP>}2xMBIV*e2W&Se*ws%`e5r&ta@BOQSe5=gpL>JF zQWGmkM=OXA*4@EP@0`esBAF*?UNk@J(|J0xYF5+ z&)E3lZ6})N@lOFW6#78hfS;PyU(EnLpHsgGI~rYrv+d6Ht8K0{;&WYOPVk!(!hm%BO(02L!kd~ACmLTJX%p-{5Kymgm#s?p7)IQ7KA(x! z3e~LzD}?SH8>+=-kQJ->%efP4{$1q(DN-mK^bj+RViO#~%9lL?Oxo(K+=na3~3K=kA?fq_l^}IrG`xlqf z4fXCj6wH?_$e9VWuF>C#(mn3Xblw+$klh(bN+yK$Hv4aW7lTP-{9vmtWcT=8DAiGp z^fBI{E)aFZDf*hrZCVCZgsRv(c!jyrx3dPPJO7|kASwHMh#VnW#Zzh5Rrk-sHGHID zy%V<^(G&|;WS)|RQZCU0MUpTNj-7idLuO$TcR!pro@XgLB|4vV#lEZ2^&HgC&FHQw zGwnUxH!)8Jw&TqhRAxPn=}vXY6C~!Z5HCM^)NKbc5#wC}yInhS6vs8e%-X8>r^mx^ zRss0q-F2C0@52Z5!u#|H5F!TPqtSOi=dI0s;<|GzU|#1o8yH7AO+~yhf%<1S3qnoO z_VK3bq0adTKd8xCK0~GyL=dT}_ltpm`jFaP>Q@R|*?S1O$*;)lGsb11h&uHd&f-5t z>k2GL6#E`y0sW{h;X9m_H232$2L(8Wp3Kg~E&z9+EBhX-%Vp_0A9OY2wQiSezt7kz z?F5bt^5mzdXSZ@#V;Y<&fz2hdpZSIhA|lAN&sdAKUb?^#$%WU6 zob|XbBb5VUE#&iitQB}K1Ta}z>(3_>3P5Jkzh=ONhZq-Y=cl(!tkZ%|`%ztag|i8c zS|}X%`AF22{4ThUT1scyb9&x@WZWnF7$S)^$={f0)LK!=x#8OAO<% z8eRvXRayNu<3$CcJN&*(CJ@@G9_5=_+X+CuQ@55&X0glOo+?(&)z2+5OO1VpUINKF zTv~~OeSy$_VLr17#XuXb2RMGEQf}bZfWX-h3u4SV6A6dBMSNO4e~ApSrq*t%`OH&{ zrR)AFa+BHlPzxF+5w`D&174J{IjBF}ri?Vvx1njkrAim4xSF=`i-Ns1u2!O;zVt8UwAD{ctRtF{nY}e)p9eOA&maAC2hEW(B1ejL z)PiKYj~9g=r+@Dl&aPFzu)ANxS7**?u>ANU#>*k^wGME;%BYyK#A^WmJ zP*Chh|?1=BtiIZd8xn6`i<27hx&UG8AuMJeXFv>_Jt>CAY&eq7*xd| zscXNW6cQ0cXk^c=UUA7y&jt8A;CNE|-QAzna4+UPTKJ@PCjKut+SMQ!9`R$7BS-HZ zBUJ-s&Vk7NdnuTiewPsWnlt*lS-#|pIT7_=P3(#EF1M3HAE|tLCobAJY{kGj%}z4x zcnB}S1g8K3VQ92u5W{0|wt=uSeh;o>0$u54K#&!|VkA5=tQQ!NwEhoi@?Wa|N9ZuH z8Qy)5_-%9k*3B3`_F6Qaizbz5tIiVD^Cw4{6{?OAH;S##_CVSlY6+c^Zx?< zn@_wR1TB(?;WdMy)@L28_>lipzPpGw0jP~+(EZsk3~?d>C>R_u3hBH4^YCi#@;|_F zO!Y0Pe?i{<+Yy3=g2CmzwE*w_3-mg)xxP{5eN4K4!NdN(-~XGh0N@|s_y(u~``^|| zI@XyZ{`a#0{xMn5hgE85K6jD$n`gsyqXfL6!0@+3Siar=p&fsJrHcjykyBSaKi%H& za1bbn9HtRS^Xhr1^>qjoM6Mfc)b{Ut`OiHA68!;@_xXQj{4dBz6`J-YgyxLwxqv_- z9~j~&Qh|ZLk;Qe=AH;S>WM9{R4leld4hj&j6iM%WzIa$dAOICLB9{7Prnq>6FFg7y zSTTAC#CD5E>fg-z-$xr=+y$N^K)$nii^t$UCHs@!2VN9PfpE>MJlfADg@{3u{{K)a z(JFjqQuX!#Qd%=v3-x#A=TBaMB}4;!6a?u19GdgG-BEf$u>b|sxwbO2^-b*i8wNZ} z4wayeBVYN%9cK3NevR&L)3k)4HA|}01a2*HN+n7nblOcD$3R-m+Bk*W!2zi3g>QA` zusOTZD-n}7Y}uT~{fp>ZMwdH6~_`foh z4tzL?MTw147L_psaGlj0$i!0n4P#%))$|`X@p=zRqYI;w@lYZeeao>HBFd33hGbpNJ7~c%8W2DJ{ci%=sxx z*flOG`FCnh*dMxtM?mPM!Kkfolr=VEVj0|ON-CTiDZB!>>}$A|3l9bVdq(+--v=^J zzRoV#M|HdER22}wd?tzv((noaK|{vlgE1&X&>`fq$)wO2{5Yt`N7MywRi?&nlj=36 zoiLS30nt>Ea;@b`3piA(*;D3pV}uq7i#FaJd?pFCN|8!@WtfCov*Ms3`;s@On}+i> zyH#FKuu!{qvUDRb|blODKVF zhjf*|cCAl7hOl(|H?4q<;Up%;~A{XGQ~7&_G{wcsQt z4vo3ZHS9-c@_K}W0!dB&_yEp5by2Yk{F9-Fc~HxJ8?TalP0wk+Ax|<)+Ujw>Q>o#e z%>P8c`?M)=Q~4q1R+`>5JpV+ksWK>*Sj0$1YUrMJ2IR$_n}41rwl7!uAI)1+$s5+`;4h z^6bsfwucCOCgqE59-7mo+PU>-?bmXjSA2k!3Ljkw$VH{q8trobs0Yw^ z!K1foTSG~?P})%_R-p|6$x1Y~Gm`USyF4szA4JA7HpV}GB#adN_q*sxv;fF5=AT1rECUl=wl%$T# z$_PXm)8PhC&y;x}n2xnHDwU`SV%k$C)|avaWDjjQLm zkCy7PFyQ^woJq!>rBL7l&E4fu4CY`0^q2!v_)aCifbh*fF3sYlCcdGkHe~!$ze}A? zL;&b=8fw)6?{+`!PAxUH=J`{90DM1t4N~1s(3xC7K)sa&Qx8{+`VNu;gJFI^?yioC zO^keL)T>Cxoy4e#upgixCcEonz1`W$Y=Lt_0Fzjbdwayp=v5dP{(oiJ03|q2Amnj! z+e(-a19gN7S#P-{5=H*_=kDZ*>F(^V`ptRFvPNGLgKd>ri>IU&<>-3@KyMvX6WpEH zR@4Nr2MHWc3SzT=0ANxn$#45SgkVzV;kK{&8_)xi+)EW9A;nPi#yh_XKJ6w4stqk| z^;)}H8ECXUC#1^W7+_>l0u}C0kJ<;%e~o4LIwUrU;j*%%!1@k)`oFX1qEAi+~|6wZL z;o_dxooprnJ!;Me4LD~M3Xfo=$Ar}FBPwgwU!k&9)+>BV9qANuKN$VBevkfsodR>H zdrvH#+MF&J5mc&H3357Gw#4=7`-HR4g@-|c&a)|-%0aAJu2XBZ)$(T>ScLEXUjz3% zKU658L@Km`*-NDZJg2qN7f-QB4mQc?oaJ#?3}w6uVrA|M?@ zcS}e+bayx0i*v4Xo^zl3e_iK#`@9?2^O>2w_u6ay*0+9xNxM-?-UIf>7b%>9bfKG} zu3App04azaNEZ;80rU7~PxygB$n@vBgX3|7B#~Q;!3N<8;MY3jX`96r=zSoi=pWVk|T3 zWua9`#|_L05V};PW#+wRcm8Cf_^B`o#9iTA{I1RWUN|!q1$8+Ci!mjx!%}FPu$MGQ zOd0s$l|pBT>znK=0z1`(!`M5ZF{mA59Ov$z=7Z1)X7Ewol$%o)H(Y!>0&og3w-w!; z>M7IvqI*QbJ>I(~p8jCFYSz6xniY>^z<&RNh^5SIFtOHovj;%NrkZYoLS7f;S^KlG z){~`DwJw+>IsHQ?YlW98Zl5v)-8lH`+)bYbYDElWmId6NEkq@+7w!_GMH(586b)gKv!=$&LR|jinfDED2TJ#pPIjF zd@&0GosO$5iVp5lbWUVl5~8O6qN<^2fcpM2e&YY@>*HrJ=bWoyrr%$ij5!i>BFPz5 z)@{}=ejp6w1iRPBa;r)8(;Zf=qQcfVkINrBwPyQ=tnh*ap-{%gsEhU=`+lUR6NTCu z%xq5>_2tKkw7(KhlB-3I3+36)-VB$S+4{+-%(4o{y?J5DKlNwPS(Vb~>_*2eZ_{!( zt2xJAHxUfr_V=xMowUtlSIYN3w*ci>y^AhH9OmNh?=LujXDUYZz9WL@&y?%0EHO|t zIHuzZ?F%@qWwmGH>L1jWv90Nk9tNLymq<4_i2o$DtDERN95+9l{N+7GWY zfLC2rsI_ha>&W!B;b1jwujDmU0gG%VKM3!+e1>2RuDNhhxW1LFN;$4SVlt@he zOuahT9c~{k($lFYzx}O+-I4J}z^XU-=So;GUwjoPo+`O);rpL}oxQzM7ijFXOV6Hq z_1$t8#cfTN%0-d#Sx@sxAZa2&@t`bN*}veXd!(WbLoZy#+4*|vtfwpAoPn9Cm{g72 z;UPMHd@7Ia)JM%?eLW9x!-wt+o~M6Y)xWA!Q}?ZWteW+nUP)Q9A~|tejO2030O>L> zjbkf@F5dwbbkzb=iThn2A-!srultRIIqMgY6Vc=M0M%-Fze7>C(VI}N@MCb#c0D>oR2Vo*Yo@UfI;}nvD4k$ShK)3 zQ1CDXRD3hp$&1S8x_Lw*-tb4q7Y2bQo8!gf%WZWwO}_X2V{F+JfxM|mL>c63SuoOy zFBzT#EmYE=$gQBb{%haR8pl=T>5B@6$6MgvtY=y6!ehUZs+E1N_Ctg_#tnR!L1|I7 zjfzb(bIIFrXEAckv!*ipi}jTv3a147fms_x`AAkA{giLX0Ab?3%pv*xOb>`#ae-d~ z9&ytz!WB$~G+z7Xwy1*lcHBi_Z-QBho#IMOdt`H@F-><2e0mEf9hZLW*t&nh9HFH^ zGRLUmwp+Z}9m)xjL552AN8As9$I#x8x;?J$nOIt6NWXT0aE%w~;c@M?lH6N-l#S6* zxaci;=L-;)i&6TB;nbxd|zMX^lpEt5kUTOAYI~UvZ<`n zeCYGV8&Y9U-eIB^jTJD#`a|FE;r0MXFoJ|fImF>=uNAvO7TFMcfmkw6gCUTi1^&Wf zSP~$rd9vqjrAS7s8Lq}_Y<{ykKzLSf^kPN?`e?tfr5AgbSFjf%6tWp{iA{+ft)|(x2~JhxkE5R3Ve{TkCLZnE_aW z=&iTc@}_M=?2c{y58`9>-f7!oEuh{X$+G@}5M+#_{jvkU%JHy@&_iN> z^bCtQl~yr1OF~c_>R@VKB5llrfmsQ>^JD(d#B}Rc7h}`-vx^?a;x#FlNRz_7&x1fN zjBL0`q1vQ6Al&28l={wetKTy=>4mLBlG66PEb zd@=!-(a4KE4u2_U=A({~+iW#*|ND~Iu#&f=#1Vc0he7%1Z5W})XA*B9WRRpv=S@v7 zpC$>&XPKZ!PdwcXbb7a}5PI`Xc(E1A>HL)vV_=R9C#-zp!Z~*@nqh`Yn%B=4wQ_~O z#p4}F26_W++1Z=9yxeUx?!z|9e!)bIoD7EAR05k7j(DtL0+)q)J+8j!-WI3fDlq%y zJ$x2-zb##c+^vmCny-E0)!t1j;XAe&*6RF7y9iGTnih(83&I`Tq$Dl4Jr8%e1C@{! z{$afSS6s5~71D`+%(37v7EKkY&eGFo4wcc)J_B_!O{@Mf9_?=#d~mx!0GH$*ZBY;3?W%9(hiDxOs^jL z;$?anFqLkP_>tZGT0CYdoyYl{+chFbK`JYQzb)S>k9)490zC>wa=eP5=cH> ztoPq8u0J8B;eVXZWjan{Gn#mv+K@MvTlAXi)Y>@t}NV@rPSHEidou5SQ{nyzv=5i7p-(ZHodV zI3@*RiW&tGGJg?ah`dWN)0-SZxkcGgJOy_vYXGaot=DvNz?c`G%3tiY{gm&Ki>{Sh z+rDF>lre&yWV=O+8tt04r-7AYf$=_9AgC{=Eb~J;-Y$iSs9;`7nq9HLypOi0wHnSA z(3!0$3$?PvgN|V2bPqZ$)C^iHTg-SK zLjjo!ia_Zie(_%R;%MF2bD--1X6lgjRZtZwu}qG!ir{{rP4(__{mSyQnyyq5aV&1Usl_*&fAOHmda8V1MlFpTaKXru@TYSL z!Tav2E|8eN@yGWlZQL$)zG;_zA!o%vBJ8LVr{j;HeRt4YEe-5hqec7Bv^SMjU#sTz z_WC#n*Y-Q?t_)j)*FA_SAN_Y%a;7=6KKt1Br$EV* zi|cr$9$mRO161Ywh%g4ejY63*Q}QWk3O3 z*dFgo-Is2Y#UV-PMX6o|VkN-@VYHYh8M+FW4zN}2Q7`f+5|b}0x+PnTVqHnr9AA{~ull(>+$_y;mR)2G*XQx@Nr>tM z0XU(~NHD{BY|Ng$0pGWNs%As%Tv07KNd6(+(O~ox|?wiN$ zSrEFKf5l7x{u&yB23A^eY+$=s{fT8kfov#$$a0qml|<_Z5~0MJ@USV$q zD>=G3_*{R2HBf0H8-H(tsxvAiGP$Rv0;iV(C^wGz0~J0KF6E&w7;?kC$&rj5{-S*Q zJ4+O51~{(vGW4Di%KHYmY@d<75RRZ4>w~#f!UK(RiwH$3#85~uU@zn+>qi=;_jvv# ztjsO|l559uB~SQkt^lkDNPeOpb@i$ydwyEHwU`$C+vcJP=RVb1(+DkC5#Max!2tNd z&_s-PfvAg^fu@J6bSg!MPw(rC-2PmN{foWG-=m)?Vj#aH=wgacdc~3K1ATE^Bkm#3 zcg7O!H!G6m&GX;l%^e?#IFkSO=}JbafK{AmKoS&2YcoWWUSrpwf~G9N0MTh9z(G+3N34n91E_VD08 zos@V~t+|D^Ch8uA@y%30o*{NG;fe_k%6fV<$WYAmLQ{|nAE0&u38 z*`^(TG2H*{k%SW7ai$&4Z|VQ}P5H|IS1BK16+EFiY>z>fm|9Q^>2y0lF=vbk4eLP;- z{~Z1WaYDi%A_!n~Iz>DU z7;Gzdeobau`p=BEpVVD3PsG^;LrF2!B2;wOTEF_~HNH<$deihmE{Q{SBT1eXxT~ai zx5%FG+w6Q%cI5iWwo#Y3U9!_8q)tnUy?*9?yb6U|1tQSsq(8afHWV+p{ea4C*K<8tNRbl;< zlb^NRC<-a@vcfV$jcN+?vnjTby>rw-jyG+r3kZlzPgSpArTs_DDaK zEWkY{*+C@HvmZ0Hjs}P;W=R*)dP>uSyz=cqqW{&218}~930{1NfUSL6>WX<8F2;*5 zUjKq6RlgzGMR!MM##5gDXBS~$=04+q#XhauaGN@p)+3_IxpLiVcdd|A5=5S_Tn{~3 zn*ve@7THGtz@vRB0kOui&HfMb!~#(E4?Nw}|A*(&PwXuKpzct{=Es`(%4BX!u`;=U zLG@E>dd(6v_>hwGUCcfPM{m)0 zJsX|4aC;~Ga{qOPyjc&Kg)MeLLu%;ae-)RDWDo*-eJG;z@U8lxJg zwyb^r2BQ|XU9=@+uaQw0D|ee-4bzFFbQ0}B4Uw;VpK{aya(ova_yIqXWif@;DiY8s zV$2W5{sDRBz|#0d=mTI-jTA~9mgVJ=SkmL^H99Jz?h?~)C`;ZpBOMEvoo9Q%CNPLN zWPvQ}N+a|hZ%^ax4I^kM5TW`pjn7dYBy&&|I##l|9hlSr&<|{w+qF5gt8CJR@!i>m zWarNI7fV5=IdDStS_59we|8{A-<^Kg#peeh+gsn4-(Jd*dmX9eholL4R6b_a*9D~O z2A7`h6l|dlZL-2L31C4oj|f-_Z?Cky8yK(Trf)l2YzIIl$nxHN>m6Q555(EI@(De+ z59P9CFfcKH51xWT;JNG0%qJfZJKBDstPI)-6pi04;<(h0Hksa6-igXT(qeDw*18|R zzb-1$sVYJS-44>$DlErMK*azY-*|V9@4@KR@CU#UCQ~LIkJkFQ)4pwnpwlb^T%MmQ zZ$W~Bzb*4?aQ%*4H9^5_wL85Q&nE00Egcdtxe9(@DKL)Et9=e!>P;FfJTc$E(twY2 zMv(#c`(=@ePZ9u8i|hUS&R@Dc>FJWvSN-Ra!k&kZh>ngz-13(XB0Wl7W?!Y`3=8VhTh-Gq+PxTn zinnZQEeuOibNpsV)U@@_SFKVDG3}@gcfX=6#87VwNLYnNA}`?GhY0O_Dl0EI6Ni4{ zF#Q0~;XjGXcZ`z=#oa7)i&qWX|2&7C;SCd3$rSzp$oPh>+m^NkoT0rB(r&E}M!stm zyXQ#)tU3{JIPyUt8`<3dYszWQ{bac`8oP~A`s=N9K1Z6CYgdpn;VaC_C9Ryy)~qQ# zGZNKis^ibg&1pHtM8s|s2=2e8owaKB3rhN}ObZp|{>g@u>9eA}K&u&fbQ~iy9PmAol= z7l2KHz!4?+1xQ6JXEje(FA$;u1Oq?JGc9#}c%(1eawc$Q2@4h_V9`bG@$2V@x3;@X zbA=2Iyxwaza_ccA_tezX#4$v1p5Wk6OQPjHmQ=l+2>b9MG}PbkGvwKaxAz~>P*Z*! zE^%_bUKQvS+-az1#dk@HbQI3~bBc(pacn$Aq|_XyI==hiwZ`6;4`j0 z?n&21nCnPq5Y`Vo5~?XREjRk*r9}|t82yszlisI>a!Cl4*4NiA>%X!}^iZ@6Ufc_{3y23sX%D|MdGG{{>dSe9^)&Zos$0Y(e8KgUPWD6=J%)&*@47Xm zCjFm&pT4eDMay+X0^~-Wl;|pioCN^HPv8MiNy3VDb06uZXVCqv`9CrNAJzJ+VZ~cZ~08Cgm$s=7Ylr+^N@sg10DL_m=aEjy1Z-RLgFXIK=j*e%b+#btGUf-dc z+MyMbXJU&2dJQ7dPILn@Ev5+Pi!cWn2l*ARPL}Vz(M((ao&oP%B@$DuK1Fy~>&cUe z(Gm{gIU{ZBOtk=L@u#&Re6+w4%36@*mT|p9bYd9q44{l{G<#CG&YtLJjZm{BZZ7AV zLmN2+rwWmbF&nWikQCW+N|KlZ!a4D4Z#H9`+q+qhU7<^T+^tCkGn!p4SPinrW7~<6 z=CNyd@!NPDNZ)FPhfUW(l7s(drw4XsInPzkUSA65bl0OD;@X2WuF+j)L5>R3N>_Zb{ z3Dk8z#+7#v>*^<3f(Opn?n)h8?o^)YBqjGwjS4B8P1o!eEOp1qadf!5qhg7jRqtbG zF4kqZ`RWSS@vzX;DH5MYj>nD@XJl%fQC)jVL)A5qEF$kEIEZ7iW@C}@2hK$Hcmd6j zI$(dMW4HdqEIZZyI=^Ve7Mb{PyHmGYnQ#RtC#&UVble1CR=Jl--u1Z-8Z)E zxW7n%GpG9HqpIL!X8PW&Ho%?rr@#oCDo=hWJQYqEG1a1x)OaA<$J1a10SM+ zt);xMgmAdO5iq&aW=O0Qk|-ImAY*L$@F|}|E~Z9TS&Jw}AUZq*8U$l;8-pnIh(^wN z?APr!lsSju1@86G77{+dEGS+($|h41QqrCyQ%ta#OT{z&MU>GD5gW~cr}B!?!jL%w zX4Xt3f1NxMFGnSmZM}hrH%kkkvG#;K^`6k@5kFF2p@al6_*meno{m(0X3E>1*9soF z$x3tfe|CzDFe=Bcwyj=SO5i41mGg)^{mj~UHQd0;7c0y=NWbh#w?a?t^9h?e>#mvI zG2ROD2USz*7Nm@6j9hVK(XK4pu+pgt8TurmnO55#>8VbP8%vOvNH+R-Iqm)!r zfjk2-G9?3zeg;ln=+FKuRHqG&Z<0!2dRT+}l52x#NMHYI`eGcfjIsKo#e4)vJ*Ui4 z?92fhS4fRRW)EamfN!O0%+XJx-`E}NK z&2nGuBI>u#F`o+0Sd*Ux5-mfQ|LD2?S?+EAX(I4?ICdOp4X1~00K++@EYfDn?@Two zMn@@#i4^xp{Ql_I8}#{{G|F85`z_4jU%Yd@m!%>-f;{kX&~So1uyH=tltRs`$Ho7s zC|6sJ7p4D6{Pn82<4`G|nrt|;V`-IORN>xnA;diN7kBESA(!94|@WWU5G&64W7 zwnaRL4nsA~j!jo$GpD+XK%AkG;dA3wv)6*Y&C%nNLA!DwPC>|F5_+t;I2r!3kykvF zRIm(Egu$PcHaDqI><@M3-GX*N8(168j2O-924p+?y*N}OIYul)tpx|S6o{Z!@M6`Q zVd2jb7?kM_L^!N3&lQN_<7{sOH`^!#8aZY>m=JYzo9jzULS2!e4GSL_x85y9`ccp5 zI5iCwDXjZLiB64oglz9-EYT^7rh5nw-1mku?LId~vS>A$mgaEfl2AhF$I`SD;}Ujg z$O-;uWK&`CO^l+qZ+~(&o(fd4F#8On%+Jh6ib{H9K%Nej;}zKMRcA-4)Suj3?skd= z<zCJ|_E`|a#Uj3X)KZ1KvDSFYZGYt-qN{2|oYoe*YOI!qcgU;t|f5gs988 z82!N6jD93P0jmx_+P?1!kqn(Zc}=q}ba}62 zpLqGjGLb$Hh8=nXjo@K?9R~S4vAV$snv4t$zV1yaxrDDYMvHjDd<5L3!vlL&f-lfs zQCdPH(HM?bbCB#=%^jha;(vS2uF^dDyO91qwcQ6~a}N%4KeLRB$op>b`SS=b7;4VV z5?N?vY-aLc9pf?sGh>#S{HJz`XyRX-f%ldh?I^>e&1|yqZsGY4GmgTwWzI@b35P|patD!EZ2ettwCC;#%+hK`5zO#4*s?9LVW3M zgkq=VxUJ30Kpt_sXEc;;Y6$zts~&^Iha}maS3P#f6lR{Z(KC2XQT)xFn40#}d0z%q z`!q)Q%=Hs|aYLi-Z7ZWlQIf*}H_c((fk6_?sjDq_ITpl>?fjSr+G!1P=x^2K&CP;) z4pVf=l?41w9~bqmH_!%to7SzKRwXs(Ja@aQt_i$qqd1W(PGqmosLrUuRYeMu@GW6O zd!c)k!_NO^P4bJ8OGh{%tI|tZma&4_@E_BN-8dX{*z;eIInz)z3Vwe?bUcd&(mWb3 zPG}TVJkD+L;vi!FVIU)K1|)(@`b^BzbcbE~;n3^c#csxnV3CCsJ%uhL>9`sZ{-9Q( zRmGHcK8{k}c-RiVX|gRv_Hv4Vcl(DY%svw>L@T7z9!fKn7b*N`c0Sb}(SvhXv*2;3 znn64EO=4caW$kr(##w^om-u>P#z!z|eutKyxhCJO!)<+`W{(q@wvBfDe630;*K~AT z(98*m%|AW%2*E9udmZ;)h!tK=25p<)$>14CF=IKqTTvsO; z$E16Ml;k3i3c&~EZf?Tq1dmS9de^%?3yQYQ5s8s5RDD&Fw3g zm169WiXOBoRc@(g1q7(=D<&(3pkg}mD-yCRpyD68<46Y$F!PA_Zl|h?xcz*aTYTBd z$&j`$-t5NtEjcM^j!neIpwgHO&zYuN(9wbdiQNF%jTB&F)}i_MgN&|N7epENoD=I# z&vGTkEy1=^KLZU_Y9bP#GgEp*=L2i@;&i{hc8g+UjN6^!9sb#4((oDHC+)Ee_11V%jJpt_ZhImlELV7|wXV$4Rz61v^LHCE4cR zb(wDnsjHkn8FB7b@Wpy0=m*(e?l}Tt+o_s&U1;e>5RB0r*w4LA&OSkxP`)A-w1!OB z7pPbAWjGcG4@E%DJ-2t-16Puv3hP4xU3#W1?d-TI+!&_V;y`J@QcgUrMng5#TnErV zmj&yp#?v^5gvxu4bSSR!2T=j11g0O*b>d^pxLy+N2nnypx(X5!=?&I5n_rF&pB!W; zl=szNuDOD8co^&ch7^h@g>|l${;i{zCsm_92*3GPAh;s9q|u za8%B1Z4emxRH?zINvux(}%*=VAcztGx z{is8vpNUT1nMdvo*e0>6HJxi5=~M#casDr)WS^^bZ_Lap zyowqJ%3M)eILQ-oPmJsI6Ex)U=m@K)AVcD4YjQ+5$w|~5^^Itt$LOwUigWf(zWI)2 zi|yX37D|juc`(5PXKqtzc}|!{)X-0;Rr8?JUIiKgaVY?sH6M4Y3O^DWt%xkoEK<=& zf}x-PxfBeTaos^I63z3Sl+9hzkJfh`_8j6fkTsT%9B}ZH&%4)}PJHF~a6*Ks6){@Y85Cz3p}<=I9B2-)`mFKNR_FLRVD&e`aun(U zeqyNQ+69(-Teo-hF2D=bFq>BRVV9_%zfLQ!FrAPLq5IeSrLC&kb@B34soD`+>{rd+ zgQZK*G3#3h(Nf3sA@4!O*UmvDE>1pa9j9vhSrv^YI5dZ~E_-dmff`D7pBNUQ`VRM4 zwMDr=IsRyr66b>$+j5)HXCqa%$)jTd-D|@4det6Wqt}mZp32NjYi}o}_s}9G8ibsE z%}toN3^eMo_s?9nn?_k>gcFZND(tZE{(P{kyAp!Gja;i!1AiWI0}x z05*CPP&-9}tVBWNJ&t!bLMwc^tXXfH{kt9CX68U~0Y|7d>M+v3(D0@{YSNT%m3G?rXuFA zEWMu$|I@GI#YEbmbRyn!()v&D{M$F?q98Fq@`AtTx`7h#|GVh__G|umq{#x>zYlH+qbmjJ%@j}LbJKKLhBT+geUgy?0$YFhPs2bLUGXCbBL z`|6XW&Z#fIdMZ;~ONA#*I6D_rGnFvqd4jaTP?Nr7-hr1uGM@m{?{c6W#`Ze(yD9ph zq!p@VyopM${KpK*B5Ir;pyh5R0*okRla(AX+_UMj@nRehBsE@-t!rm>HkEhunrsU2 zer_8PCidwl*=D$fOfUKEG}gJ>4tdR5S{m;5xz|yvrgr&-Z<-d1jG*-VS^G3E5rNgL z*UQ?*3hAPv6s=N|0qAbm?We=^xKSx9r>Ho&?Y+*S3nTvj*jMsnfc;4mV1YPrTV3fLzZL&|3;T%)P*=^voPT5iOw*hnuAVZ70MheL!Md+UAlv1M=E`E_3IaSlibkojdpvC2Q@O26 z-309Bd>_-PQjkBbRFb`JIw|o(T~?w4O6OZE{T%wmRz0c#f=XSj4bm5XzkssBNZn)K zffZl^K<04Dtrzfc zum`y(8Gl!hOJsBFmn`^l6}pYO9%FO5_8uIg=K7@KHf(AvwjAqV_x2=9aBmJ}%x*-6 zJlE0miRCA4B3OAJVzzb`&bDbLtUx}X+7IjQThT@m&%2&Cop0Kk?WqLinQln9vqZeaCd z!6f0v$W;lUnWi9|ap+|RhNQzL{zg`xoAdk#Qa<%NzFL51y~%UqYkuKgKtby%=sGhM zCCblod9+Ton&ucK+0%PRQ9nS%!hj3dTk-1pTpxQ+{xVg=%_>W>6whZ&&vP2RL#|JR z!#Tp9|G}`29nfu)dO;mCfeeJ}jJA!}l~2Ca+!`Ah~UAgAVCzc zVX1z7Q}KR|6j=GmCpfZ?L3@E^Uc~|uXwzNx5D-gX=5`ZER)TQhH$Xs24?=3f%UI52 zsx6PesGGhh0FG1ttd6X8;G_{mynrZc`22l+5nX`F0>SW5zV--7p8;O#VI>)VF9;^? z+kK7Gl^%r`@=@XO!tyGB8a@i>RFrC;fsb2G&)$`vb~)|RTI`sLCECSs>*eOsCCA$= z1HC#puC5LD87AB04Mx?$gPPyYHC!3+ou*x#z0;HN zP`Cc5++?B4lxWWijdyzRUk?O2a>MP7M>bxpvh=ejeQA6e!Prl2+V6Sq{;3;x$pE;m zvu3s7k~^>QbMEvk&?qSGdr@tV&Ba~k5a6Hu8!VijZ*_RFSv)<^1YAJ6($VD1(ZUx6 zU|4{j>dWxMfjEHA^XlR|=^;TVO5tY<;p)Zu^}fGCD2^I$uT=(wFSnk%@hFL&IB8JF zEFnXS6%rVpKpljy_9V`M>o^Y-P@2dxP%*mBI8z`|IRHqsJeX^&Yr;zWUk}`y1#y)? zOQE*}*olt|?cvj&R4%Iel!QTL?8Y5kc}e;VWc*IhXLWFZ*B$`Pau48PnC=2#f!U@z zK}+94>Z9Fypa8+iB5$Zdy~k;#sYMCxOS>u9YAnsFbNdsm2}jSpX4D@cv`E_Go2bJw zk^wTS1L@oIV9uifp#g~kJJnQnu%0~N)Z!^TjhRmNdz4*-^LYf^Kvqz;SB^(`b=`ue zYN-XC)k$vcZxy~M+H*=j6P!RO$oKb8<5QtLCHV;{rabKyZl12T#=GP+KzOQ`E=_W4 z3x_{pWT>jBmcaisWV4!@lnU&kghOAF$rk z!1PCXsMMMZ=mzy!uBB_M!HoK`>)T)1;ajp1;9#?6i-5~cj5eoPYDXuvdcFJ1qu5vy z5v2?x8D(9M{ueX`Qb=+mY{|f?DzuZ}T=a_|=pNDa$9QBY46ZpJ`5HTB`Hzv_MSu@y z6_j!*S&J_Iy288|^4h(=UUTP#o9~m8Xuea!*-s_R!B4ZtYG>a_a+vnL)xDHLiJGL-xJ58` z7|IY6k*BBPC-%{3PDhuybmtP)F^LSnZd@*LBCeKi@cicg$F*aj{$L1u+Nz*->Xl=+ zP~_l84ym%zubBHQoxMdE7d*puWVVEcKZ;l$zeU(i@PBpmwz2TEXvh*$zDYZs^(NZW zj_{nQO4Fj?XIE-3dZD^#*U#S+!^^Oa?Zw;)*`~eczDlg#<^B zAbuf#bGC@hSfmR$br-G|>7Kw*Y(c>U&7NkdoGAJKod0+$RGd;g0jvmJkp;rmth00U|o8NytiS0 zO4gYL7fBQu? z(x(o3Q7{Vclh`WV5WbC=ARlN+N59k<10~r|hAr3f>fM$VB6;m+9~D1aM5aU-!8J9Y=%daD@#W+81qNOVDH6|xx4p+?@--Q&^ZTx7KSn@0S6RC4>$h}>+ zW5Wnm(!g++T%FbXijo%Jyn_1Nq!5) zm$w>{oYXyg+H8ELXAgG=;p`7~S*cl_ndp5|?F{2?rYv|UJtw$12{0=yi^Fto_ED%9 z9>=RiCzXBfD0N0%-V2K{8T2HjT21U{F^y5BYFDy_Po zI=nAU$CzNi3gt9A4EcbZ)v)93lkXdh* zP2Tdfh-=S`>E$*hiYGbMVI=QCDNcU|`?HSnbB`rGylUkg)u*I5LC|5S zaK_c!0Iqtb%qO7^t6g-((P7)7k2)D+l2_5kw->#abIWj+5af;Bk3dBIgoldm)4>({ z26919MQBqAXq&E3#nVrUDh(q#H|JwSspn$oNW0Xp%dmVg08#l-OmDIieX!ENXB7B- z;ZncpHwzf!{D}|72=2*!TVB~+){7mOQ+Y5FMr>-T4Xi_qTy}TJur0MD=}O; z1(Lq)FwQsW1K}h#)na$CeADc5<~eV39{dD30DkRGZ6oKaD|F8kNets50fBabJZL#B z5=58m-_m1b2-9i7V-*Yh$rCs4u$~1zHFLt?V14Kld*DyO<#XdHP?{F?TH0HNO1;K- z-OLjykSl*AK1#oMXg{(kU?>*D!weFuJV%8*0+bI-pfOq%5>EOLY)+vB%^loU;W8PVt{A5RQ-gM^Eos(*bt2bCaZfU# zHaK2@osgTHOGgE5t~znwh?HP!5US>@p8V>La#aBqsn@2FZUN9ZoT$$~?ZzQP;hSxO zDxt~-p8G+VB(CSnNiuQeHgQpqU++HGFZAj7P~PKQn+H;j7}`wFVT6L9%zJ=GNXpDb zpoaTruedPzxm?yfd79?l?r@$$_XK;6^;A9i27O%Lnn;ASIH+?A@;Q&%`VqWhadSm1M4o&a&Qj%MpU{>Tthw~*}I8am6EKBP{$x6@14mN zNKLJ>jD$s>D*ougO;Ig{Ym-EG-tUtOarG(K?n^;|)MFd5YPO3o#WUmla`p^4SlMV{yv7cf(Nwa5 zvdi=8>i`j)5NJQI7Cg*-=FLlYE>pM?-y!P59b{~oHqez*q9%u2BhZ4)*(+xFqjFdO~ zTlw@H)$}*JU-iIJkJ-pIeORYyL!GqJl5}n^cr@EGFHXrfj{^vx!E3N~x1V^QJJO(J zuEh7ZNLw!tZ=U2>4Kbqfz=a;YOQ`=LZ#lnL*!lX@fX1tJUY} zJAqU!cvNZ308qGVlyExZsPdLm+MKGEZ5){_wa-=zRi8^*W>v?wo_Z+&ZV0!8ei%r( zBxl+>TMkk?1b*|%)dmW4xpV2L zZ*@tsTlKOeZNP$m66!S(ZTE7fwU1uP+?jt3O{43Qjk3DLT#$Gx_(BCO?Qh+fbnHJl zprEr9zwYoN)YTtKQC#C~$>Q7Xyz?^ijVDKkm*UMoVMXqdYH6+!fzQ2O>1k2zd|NZ_ zKO%Ot_XZdo5T%%dQ+IZGnAbu!fhP-s>-B0s!6tmw=iBiE|2@zObqQDJpHH<2qtN`UA>6duY?5r@(oz*U%1$W za`jk_7O;S7xF5nr(3R=a&xS{q$dfRo<)`Pk)MX=W!SoR%N}wbLU5LLaQtPKnG?gCB zjsdM_etHdol+I9`03#XKWlqffpj)PLxdiTgXN%<1jXyGPX2xdPN`Fx02$KL(D_DUn?Agfyc}-E+0rCT@-0dU@MfdI%mTB0K8q385L^5Kh%>%T zrfp<(0PiA!QGaS@VG3uL7@j)oyG_GQKS#r$G{4r5H&}Q`wvGE(kIa@_W6%O*2y3f4 zZ4-OHrQmc+Pc)0~i&N-S?8p515`*K*a+KDnXMRJe*ss%IVDnw4iTT3?UURY4PbxujQ5w>MjDw%-A>a z(C}uyxs|=%QR0_ePnV8au8Qy<0qkCh<>k-eEDm)Kky+rMb$N4|%Dt*N#Elj9alogTlGs~2m4X6ky0BM1Y1)8Ah=oiA93-Qpp9 zVW_yDHbJ|TQZFTlBxE=qhcwdKt`Ey z1A#U`Sw*IFLK~CxTvEI$i6-WDC@*Ax-_@wJF-!GoiybPEzyK=rE1R+MFdv(NU&G2| zyPOZ@EOk=w>2asKQ(_9)EgmgEEq)LP20E!^#AR!^c0MgPe{oSm+)?WKN_I~&K@*i*bSSxy@^^Bd9YW|#}36|tN4y&uXB-G5S`7e0KvIReUZE^+!8*L%u`Jm z-b2jfr{=~0J*)^#2BktD(w;fC->=um$_7PE-dfmuk!?zT@6X~jFV5b>-7bIsgf+~b z4+|m-k(I8rfvG)*~z>$ z`&KS}{jfV+X6kcoHbV9680!N6EuUZR=B6tavR`H=TxDJ+2H!_^^alrf3#JQbS~PCE ze%d84uvnDkwBJej43P|1%S&UROuk2_6cWuY&tB)rW}m`?t=A_!@G!|4=_ABz8~eFU zV{$Gg#++*GCUMUFs+y?o+R=XEmmd=)?AF4qP{r=?+@^_gwY!UbPUbk=BKP2fZ3-As zCwavUwjH?H&?7;;hc~&DzgNU}q4H}7?LE)&;}f<54RgzdZ^etM9rnOjEuLzUVTgFE z`UbwPgRv%m)^u+XAHfReu$%?rjPP95yV6#pK;19Kn8rY3F5rHPNvu+i++_K+9|#+A z2vgcZ64ahW=+1Odia}QTBQ4zTR@3had(|s7$X<62O_TOwUi|v8ec>haPSl3Srnsov zV(nye?VjK)&N91YQlPqe)QyFC-KRs(g-r;?LSqc1=FY`|)@a||KC>TxkPkbI7RFR& zZ(#gaa1`Ai+z0yC3+<$&+*pZ&$22loqGM=3+4*b4SI@WjK5f~U^4viNB7YLErWJ=; z_eID<;CnTd#Mynua?Ztu?l}TUksFklb>uXOP4KX(e~DWjcd@2Shl?56KuNvdLD7IG zDtDcO9vaKP^93$D{&s}eD~;**P<`h%$1oNA#VoCIn7A29D7*XxIZd<2x^3#U3qsFm z=qjsLjy;~hj0-|piM9)A?Zt4sO{i%SSWJwCjvSE)P-GW{2SXfQ)lYyho;8WckJMN%*p6aO7Y=F;qzYUkMaQ+gN zu6t7bL#V>*R}#C9A?Eg;Vpf~j^Ky|gH^wBI=p@bTjmX4}-2n;n;j_&)it5fN*|ouz zOuvgMd2h>)Y95k|w{dM%894CCZAPR%^(>V<=3SHY@@7~po$^y|&28)pT_M2a)otrC zp58RR4dF(}xp~y#3oCh*YXt~6c6?DY-!GD@{&FgsFPD2pS=k8}4W#C!u)blp7+=wk z7F3IV95lO5rs`$pzy6N(OKT6;E}oHHd*=~V-a?v*6)FO1YN9#KIQ8{Z7;-m{dq~o& zi_c^b`zwK(2~3DjSW86i?pse&x<15~Q-9}fA;}0-kS`5NdV|jS$nlFx4p(%sR$@S( zg=IhEDo04F`Xg8rjUbw=H`2|zZ9{B=Ub1^=f=3!H%-{so0 zxAScHObE@OdCld>9ZDzF&Wg!Q>4LWr8nLe++IBV#FNOc)^@T=_9yOm6}?HAfVeOuLw2OF)Zwzz8TTaz)|e zvN=8C&k2uyG`QsZD^742o;bMTp&nWn=VxX0Pa;TJ)&zrI6oBPQ z+8rlh@7Y`3BbvDz?sC#NvJ5^2o?4lLNCfm%79Pne-+{`Ckmm+`wgL#GMM3c1P^}U> z!}zM3cN`N>=$Bc|zeIuPR;Q%>FYCXLmQ0eCa{R@hL$eQ1vB7bo(YV*W9-bfmHZQ8= zvWz);6bH}mRbIWOm zR7Kt_C!5tjwRgk6s~;{XiXb|EYL0nLq=8B>mK!B79bAs;63Efkr$d&Nz$N`!{Ddqs z4efEF48Mqfo+#6!;4_w>E;tW~WXTo1X)J(xlJ5Q7Eb8&l;t@pxn>DNW(tsN+%H#c< zKhqUMJ+6vV^NzQqiin*WLUr~U$9nSkjgZmDT3NEJn-%UnRF9yqJaSWR5Ik{Nh#(%# zygS#*eQl|5GrMy>f`od$s4G?N<{Zg`rBgH(Ta>q#V}*(I6RG<4SCpx@^ZN833Qm)_ zUDWzME3^=n)xJ6>GyGmb=kpV0Mr6a?)##E=j&(``CU0#SaMo6++sKf6O!x-b@ zI6))CO(Pp$gQ>rjs5EWBHn#V~w&fL1c|sqWjqUTd#`V4X|2Wy`(qC1)TkkY&3|t5Y z36AAb3nph_rE6~Qks!d-gZjsuyKCbX%V5qO+AQ~jz~hzTgMsxfixq^3nLIamJXgx0s!0?{J$-Kfcj+YqXEwU*e26Sz{|RE3=1on!TRYQ2A+QiU$fd}{ z=Qui%$~=aKCCG8azKJrH;jrUWQxE8G-~tgAmD#55r|lVo7nGac{WnJHolGY2HGk7s zgLt><>{(#S)#czKsWJYCQwPTQ_4J1xz#UmT69*k<%}y;oRf9a;!)5gtTZ|IkxG(M1 zB;On&uRpvq_7f-*KF8v{Ufy1=UN+f`?Jobx7GF=_K-d$|f%z4a>F_m67?pPMs&?y` z(UA3!UzYeM4x{K<)CnykqLxFP!`w)8wcfwPG9I-6!nTq?IGa)H(IWLpKcmGq@}=OS zLOY=?2!s@J`on&^G5ae_`&pgcm?ki61Q~fLExq19DwB1fa}k}z$O5A^+7)+}%83Y` zPbvQmblHTj{K8#{yt042o%c{Y$Iq)(EMKDI1^EUe&Qp2DMSiept1^gTQK@6|^cXm! z1R?z0T)!I@c;0_no!TilMK;ZTaLvR&e6gg#{@yr0WhP>{WN?4yl#sV~giY?MO>kk?}=uGyYOSO8wgO>m+)iZ`2 z|DFJKaP`HEOHcM(LEL;jZ694=V9WMsk0VaE^Q4=}tiOCo z{r!9JD2QMx{=WKOdH~H;#j+uiS!+ZD%<^&ZTT@a~uaiHBqT^ToQP-4?HJDTllP?Y;tgQ8S#uAy;_Ju0)(k)k zdJae&b`TP(6bobWQNf2UeMi^J>7iMa-B5e?64<;se9}33beKt(4rT=Vg)cAaiMrem zRj**dLq&TP2^&r6T7rH0;I+e#BQyNXO-Qy8Jhax6l3{1>!MW&& zoG+eewiuaT-|@cKaY4qBW~FWX3u+WJ%v|ZcYh|jL3tIJZtW=BJd{uknx<4h2W!PC3wRQ_|-Y~|I2gix?v_t9SE}5E0QUi!< z*Q{H5`rM@!eY+yr+%-GBt=Ni1k>96YKh6Y>Yd^JTX=@K~Z`@v@>qvNnR)ul&QB{tp;e z5HrhbI$4!hdSrT79EsApK9ItG!BIo-IseO*{I_q0goEzqa)|r84cF-XUpO)8cZ{!t zwcj;9$k6$(fA=paFYESI3WR2K=0@TFf!ksadLSo#E$@Nb+q?Py<(mFYvx(dTpjY;7 z;VIC|`EPISub1%mZ{CJV7qLYZW}s~q0Y0Tnjx|}gnIwB{mreCa7!iFcRpg-xYK@9T zsSboqqgs?KM-%+3pPi6!B=Oc*ppqC-{CZAsW-c=#)p#s@<`C=e80;>KPp{6ZO($nR z=BDj?yINGAD5<7Kk2*(^LBdG^!##`I=co)jvdb8n9}_IIYuxf*|3nj38%0DeRt zFmzc=%E$d07u-_T6K^d3=DBD-q)(Rh7Hs}r?LH+lVH2Y>KF^p`hH+P}TZ1!Q+s#7r zqP^+GTn!g9cAwOpzn=e^&S8d;qt#3Oo75X55%E|{10yWI)tU+@gEWBljavT`5B)m( z^sMKj`XDqPeULE8&&sMe&STQ*=cto4@HvdQ58wfofY>*(LcY`;y=&1aREi0V`U&pjemIMDZ z6HLvLq6-5s)ns)GU`RLk4ia|)K#A=^JAIUI{Q9(uiQg zgrber`gL88Ug5sevC*dTilSLZBhWC1S)|2(wH#s+fjt7()XnQfh)}N(v&6MSveurd zn^Fz0T^@0E@Gsqczq3iAw?G}?e%xsC>r#C7^mNJGX0anuP94)zs|wb~Az>{hi42u| zpt!SyV!m(?Vb}7j{B&0~R)r2`T+08miz{1A^+RTt?x>!;~(^EWJc=$!A?s zK$>iHZ(+69X+*Sz?8h0bbN%Zsi%~FSk-BkjNNtYB59G~yT!83Cm9~JhpQUqn*``_& z`$=o)#{0{AuojRk-YvS6MP_XvCVlCuKbxp7yzIKK+yJyql1{#&3b>iqtAX^c3c$XF z#Z0G^-2o|+-)BI}?p3&D#0+jcqP4DDVxlHqv)A&llid{Wg0^X}_As+t?bYlp19^E9 zuNXm+aws{E;W(XuZsRd{+1()SqY&J^!Ol8hECL8%^IdC&kR}x-H9~lk?2-xrLRM!v zb$YbB&TrkHW_oxTayE09#2Cb-bAsuush4;bz+|vjySWz3Q1;}W_gYmK)`@G3To_R@h?#i8|FRZfSB_)DSI>Zw_=KNs5IgAICBo(W1O%Lv znTX(a%{7eB0|@KGiDt&#o}5U%ET#caM<+N(jf)xvOnWcU9)W)ILU1t;erOjP0q*S} z?c)5T;wxDQURKoxpp;Ub_pKUMHN<9zZgh1o$Ver>fK}QIGEIiQGihrzAm3+W;>UK& zddWqG7CL^5qss}FcHLjegMJ)AjCa@)Ib8XTW0w~bdMZK9*H>x_N~i>VpY*Qju&bl-%=8q>DuNTwq-a-Id&NvwWV zWN$1`44+F)HsTWa){a!Fqe!jz{#^!}W9qPgf zife05@N`fcX74rTr;SwPu$6{=O{6bP1%ln3djq1PJEOFAkv-x+$I{ri8We&C>Z4yg z`Wg;D(}E}uL`TKD01O5%g96|FbL!RHt`6~lhFvSc?aMRU*%{(ghTjYcHp1mp$IQ zoO&-j?MJ2sJ)Q|6#-a^#Ar;vU(Nq=_Pzj4^`xi*e=Bb;a>+mckTovu`f?*+8E^ai9 z-`u`yDvnWZjTK0}bxYF4Z-j3smI;!Y{86mN8RCGIleNa`YXaa4hJ22_VyWL6o&hFo z9)Kh%T;&bz1XAEBY@_Jj#?274jsYPHpHut`R3=WqZPx=9I=enw#O@`;MxcT*-=USs z6`+&pvP&vX$_I2?%}Mn}yb@hi``~vy69-{h&>pnUE^Dad=Nf8#RaO@mpIAqQl6k{a zpI*9VT*U!oSsY9$=CR8;cw%bJ&cVaY3KAEIA@@Nx2_ar&WBPv zI)@SZg_?IL=IumwkzkF!mI<+!r~<)1-uvJ69E`&GI{630H{ssDXTz|&Tus`)XU>)j zpM|!QJ<6X3MiNL3_F)fz31pMD!T*^J7qsYr1DJBRsW}O6< z!+>jgZO1~ubX=fMjEwU}=EZbCW36rJ(}Yg1&m*Ez^+sRBTI*XE3)Fm*hbAni7RS#X zw^=TkQY*%6T4~$T5T^2AB(bBzM=Gr_A}3q5%|TQu#VR9D}t ztoym1u^cfQjBY8+b)EI9q-GMR4PtHI+yBbiZxBz`7qWZ)@vB0C_3k){0ZmQi><9OC znO0o;AEy$J8k)X)$$Q^Hd)JivbkaP!Zbpw&cFijN?jL)B zna?&(rJ!Tm<8D1+%@+>@#~bXy2>=&3h|~cXuSww`QBz1vlE1xiP&AwpVRXqNrJuR{ zq-P?*NCJd^o9Ez6b6hN@%SI0w!iSRw4)}O6D;@>A{NwpOWy4yRH@dVg^tE5RTUW&z zzFs*~<&4YSA)uE*lwXS4R3g=PZ`l;DeG6$?n=g6(+gxS~+|lnTJf_lrW*-97R}}>D z6y7h^+?frdCpeS{u>yJIcDY?Zs+hPddA4`IIm_Xhr|I?fmGAFUr*6fH;P(F32RE6M z0q_8HZwLKuN$0o|mO&3LlP+`6E7rB!fv82EXgeNM!j*eRE*c$lrY+JC#L>EB-=&ehfHF9-ZyNMDV1)J zWS)fY1cs51=;H#T!oCZ$)*4Me>cs9LIbJW+GM|5Wg1XthC>q&8X8JyKcH{aeqI{0n ziRDim92_wiwjtl>OcBDSXvo7|uy9OPy6_aiX?>%9d`a89nlDKDX?eAxErr?%$D0mCB*)QBFI=-xHor)E zJ>(gj9t*eD0Gfy|&&yLRf6ba!QOsWMF?@0TlN+8EleN3q9KL*d%tw&GLzrirhbHb~ z*9{;+As4@rwcp57!`DzcLxkO8ToAEp@EOj`nLgBB5olnffwk4T>;Gc>D6tOz4RI6q z+Io6|biQE1ovl)YDE#2_`e5X@-FeVxRW1*&_D0!ELRC}0#d~dUbAt&jV|8{7fi_F0 zU~hhOnztVMQMp@X$kHH(9`|_Wc}Wq@5GmMZhlak7xanY7hhM?!&bQLcVT;<5>8^$; z#Tw|a-~W=BNR-)1ChFtLXU|Yf?^plufQoS*70Mf54e;^vfu1vR}jfBWm^j zVLM#+y=&<>xkghE_HVx<-w8>odzQ_9zm@fyZ^|sh;oEs0= zK{wuqT)wVvhJ@SD7Kh?v{e|R8P=Qy_H`*n;lJ;KKJIj(pCptBTf9fn?8&$rIrG8zr z`05*N!27H%+sDk`z?+ZhSAGZ&T1I|u?wSBBk6EBz>0|yJ21I>~pzO6lm)U-Q>kh6! z1efh(T^TL@W*z>yMXeajh+oldHGaLc~NTI`B8x&PpSaK$;&pz z@DI>rqY&|*n{lfMVl{DKy+(RD&2YZu7Ykm}aXFqX21|98nItg9@vC_5R)S6mf;a)l zyfnKQ(ZG_5ry(yWVssi#amN&B`ZC$zXMm}X#&)sOyXqTplLN6T2~G0LIhA4wX2Y*Ov`-?qPm zJGGaS$i~D7E0!Hok!9>2Ud9|nd?B%dFm{xTu;qBQhkiCCbyBPrAk84cuK)+%?>@|p zmI(DcvKXQpiZ2A)vra2{pUrr8;LKxmmfW#OJ&4ScM6#0$X77nHuBbDr(*%!zakCcU zGJldoaS4#$$?qXIwIq)U*4bN^-$a%|hik00xH=fTV(FzYJ}V4whWVrTt701X{iW!M?_i`2pfv{#tO z<;~_bl-i1Uun^-d0FT8^=Xv-pX)8=yZnE!&s|)qFn>&Izf;rjRi#Zab!$h}D&~1F+v4XC3W{}#u!0sdR*n~3N+%0F$&%i`uIDZ_Ol*su{>Hg$Rv=v48-V=^QOL_xT^ zlwB61rLW5tT&7YuEYX;ez)V(3^4-4K^l+O1KNTwr^F_ZBNa}LqU8SQYK@W$nwzVV4|4+`fPsVDM-o)b zz8`Mtnd8SRbfk_ObVKsXF8ff^Rp>!Ws4)8yRfJB#X?2@m``Gr?nFRpkvBdlO4pq`$CvF*N1| zai-M86ihE&+nMrb=iKM9z89J-g|ZQen7fXFILUd__H}tnLiXbJ@aG;qA3m5V`GTrv zee}hh{a+nSM4pf#`vvIfz*F9X05tE?=I1bo|G*=xhaH2u5(~ zO(E?N8E^Vo?y_348DFE!6}0NFwB%|hX&2r_FhH|o1|!rP)gOx>0xC#$UWo=ktY0#G zet{*=0r!U@AcDOwu8e||wgC?>ahb^KG&dP{M)d8~?E34w&CxWXLq4}>&pyIk%jwKs zW9`EDV+p<>rAO!yov%9B4FmAc6lpKJw+yrhw(@wBh_f~+ImwX=q5^9_^7F6K^T&!mY%%Ho;1 zQuz1s6oaNI(q^6qK91fEyP*?N3ciI?kz5Wojjvd0dh*NWBHelJT*mVJsbXtpU%|qM zeI$9=n0r&FO@NH;Bmkc++D3aTe-cv`z-tY?Z&-svym(Hg*U;)>iIx-%WJ(<|N&S^8< z+CpKyb+gc|8H^B3!J|f)lk>EJ2G;uL#QS7{;U@FZbEZIZP6Uk_-$K{&hqEg|R+etd zGf$#j-Um%GP!+mqNDQ@%(T~j+}QAqdMgZGDVz2N%{DDR z{s*NVU(POwCk9dUdAEBr4dXVl`PZijwh`#-#OU~ea>4$>Uam)50(JYSvzfXZ`zlLj zm7YW}3o3j~O1eLeSkgXhP@s!rk9~C@>yHo+^ODS&aUeYr?&_GkLVQh1PTJJSdFoRg zf}MpsE0>MTEhkQB^(^XWWts0#O_(%a`)I+-i@a_NLk>`c>^yfnA z7@d+mckf>e(vx9QZASwk85wP=rhyi+!(s<7^dKReM%F*!2PmMevhl*=suQ; zG=u21Jm&6J*0=wgQ-jY84P5rYBK!9BA5G{CmZTyU$d@DkZn*sOvRqmM(5*v`heDS> z3WDI+K0g+2zu9!GPFC`Swb^f^9-WQ~r8D{Gn){y&E^=vL4v{;P+$B_HxJQ~MT3~D2 z@6ps-1Dvk+ePwL_Eg7JlR0J0M?Mx z--_PfF4NzyWr#V*B3EwtWvU6;IbX+RlXTTj?TiB7p9KMD`@%;%eJ-uV4Isao>aI=Y zW>bN&i2regfFoK@#h+5+zrMFWb#jr*TKgvOD={=@R6;Np#c%SB+X+<@7@P*b!t**#N62{xfxB0G;(>&!ITcJD2Bbo z*Y%_chHo6yT3a8;un&6K%Km>J$$$Jtmotc&nCcb>+_02=q~n#07olkng>1RcKH);+ zl5rdu&!u{PdbCy*m;qM+A;+9RZ%p~|f%ik76)KH`g6l!jC38Lv$n?LBxBvKm{*u;m~6MdxkUIl8^?M(Ay1tBM0L-as9S-|nUCwm}GiW*O&NL(^#ye2T6c~oY6=bqh2++`|E<;G^)Qt!r8Gm8?6W(R1H zs!N)**sJK&mxLF|M}#+(YceyM6aYPbGhJjweD$Tf#^+Elj`8hBu~zQKTd@*Wv+dTG z!!R6L^+5q7r3||1EW1#&0(Oh`nYVT}KTvQz>b~n{tniAwYKe;5{&F;R!0EHR zRV~{?1G`>T`uXAJ)W(%7*gsU{WgbYZTXu`fEzE}bg1iGI3k>pTsL+iqvk-Kxci*YU zuujx0#ev*aFp~|*>hJ^25-W93VkG4<#4Ko{y@rH8eD@$F=5UI7=>gQ^$&`1h1~or{ zT)3^jo1Ye>Mx7@2fHyN4DPc z0-26p+OG56JeGY)@iI;~QtBKwQo`tdJv+?in>AP&Z7CjkuQ&Ee1Eeh_Zkt3Wf|~fp zKA4LVe7=hFhwLZI!!O!vgeuC`AkoUM)q+eg=TrvFV&U7T}Em#An85#kj#7Xfv0+M z&$}Qu3q()}%&*rogHAyK-vgS)58i~&h8n6yO7%ZGhLr5BBeoNGP3$Sfn!lYIG`Xjv zr;4)Co+fzD{H?j_AIH#3@l>>kNrNG5X4gu@pG;XIU$XX`8Hx1)`{%N=q3>{7BMBsf zKh|0Lsm4>|MmmN0DU^wek>9*d#Z4t(5eK{iZO~H`L4i2foaQCo_jzpWH7nX!c2wqJ z5YQ$C5|}Qg=MFr<*s+D(0m&+FBcxZ=UBK?xW03kFjGm@DH$WxxGIgcayU6F!U$@l% zbnW7}gX2E0!9yVUON)fAvf8>~VQ$`#b9rbx7p~Up+`uCRvFAvwo%tSfc-Xs1KD)_! zhg92#?^S^TT*@{8p6vMuwqpbS@LFh)Y6{#S;AH*q1p~sFj2oAqS0}S>9Sli0-uZZD z^wS~)>JTnNIlVgCG89@W2R$D4x(>2^A3zkk=tHm6Zsoo_5WfH;TOB(YTJ_#P(g0+K z;{b4o&#SP8zAvVsa(t@W(gmM|op4ZaCK1nHj&Bpn?o>>=FT?&qx$S>nT9o&35*|W> zQ!=@_y|EZ^vN`y}Pe)3OKfhwTUwqDeiMVi8lzujw<8V)=umtupd72c2(Q>-=O!uvs zLUZ(qUT=TP9ECpGb#tlhf~T5s2CVRU6RrNuAX4LFvl!~|XXmRP@VTy6KjhNWXS4Rzi%*c9*|S|hZQNc6nI8k QSHPEwf~I`A?9<@?2lZfjMgRZ+ literal 0 HcmV?d00001 diff --git a/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md b/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md new file mode 100644 index 00000000000..72118979c4b --- /dev/null +++ b/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md @@ -0,0 +1,31 @@ +--- +sidebar_label: Integrating Dataflow with ClickHouse +slug: /en/integrations/google-dataflow/dataflow +sidebar_position: 1 +description: Users can ingest data into ClickHouse using Google Dataflow +--- + +# Integrating Google Dataflow with ClickHouse + +[Google Dataflow](https://cloud.google.com/dataflow) is a fully managed stream and batch data processing service. It supports pipelines written in Java or Python and is built on the Apache Beam SDK. + +There are two main ways to use Google Dataflow with ClickHouse, both are leveraging [`ClickHouseIO`](../../apache-beam): + +## 1. **[Java Runner](./java-runner)** +The Java Runner allows users to implement custom Dataflow pipelines using the Apache Beam SDK `ClickHouseIO` integration. This approach provides full flexibility and control over the pipeline logic, enabling users to tailor the ETL process to specific requirements. +However, this option requires knowledge of Java programming and familiarity with the Apache Beam framework. + +### Key Features: +- High degree of customization. +- Ideal for complex or advanced use cases. +- Requires coding and understanding of the Beam API. + +## 2. **[Predefined Templates](./templates)** +ClickHouse offers predefined templates designed for specific use cases, such as importing data from BigQuery into ClickHouse. These templates are ready-to-use and simplify the integration process, making them an excellent choice for users who prefer a no-code solution. + +### Key Features: +- No Beam coding required. +- Quick and easy setup for simple use cases. +- Suitable also for users with minimal programming expertise. + +Both approaches are fully compatible with Google Cloud and the ClickHouse ecosystem, offering flexibility depending on your technical expertise and project requirements. diff --git a/docs/en/integrations/data-ingestion/google-dataflow/java-runner.md b/docs/en/integrations/data-ingestion/google-dataflow/java-runner.md new file mode 100644 index 00000000000..379b5689005 --- /dev/null +++ b/docs/en/integrations/data-ingestion/google-dataflow/java-runner.md @@ -0,0 +1,20 @@ +--- +sidebar_label: Java Runner +slug: /en/integrations/google-dataflow/java-runner +sidebar_position: 2 +description: Users can ingest data into ClickHouse using Google Dataflow Java Runner +--- + +# Dataflow Java Runner + +The Dataflow Java Runner lets you execute custom Apache Beam pipelines on Google Cloud's Dataflow service. This approach provides maximum flexibility and is well-suited for advanced ETL workflows. + +## How It Works + +1. **Pipeline Implementation** + To use the Java Runner, you need to implement your Beam pipeline using the `ClickHouseIO` - our official Apache Beam connector. For code examples and instructions on how to use the `ClickHouseIO`, please visit [ClickHouse Apache Beam](../../apache-beam). + +2. **Deployment** + Once your pipeline is implemented and configured, you can deploy it to Dataflow using Google Cloud's deployment tools. Comprehensive deployment instructions are provided in the [Google Cloud Dataflow documentation - Java Pipeline](https://cloud.google.com/dataflow/docs/quickstarts/create-pipeline-java). + +**Note**: This approach assumes familiarity with the Beam framework and coding expertise. If you prefer a no-code solution, consider using [ClickHouse's predefined templates](./templates). \ No newline at end of file diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates.md b/docs/en/integrations/data-ingestion/google-dataflow/templates.md new file mode 100644 index 00000000000..72478b2ef18 --- /dev/null +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates.md @@ -0,0 +1,30 @@ +--- +sidebar_label: Templates +slug: /en/integrations/google-dataflow/templates +sidebar_position: 3 +description: Users can ingest data into ClickHouse using Google Dataflow Templates +--- + +# Google Dataflow Templates + +Google Dataflow templates provide a convenient way to execute prebuilt, ready-to-use data pipelines without the need to write custom code. These templates are designed to simplify common data processing tasks and are built using [Apache Beam](https://beam.apache.org/), leveraging connectors like `ClickHouseIO` for seamless integration with ClickHouse databases. By running these templates on Google Dataflow, you can achieve highly scalable, distributed data processing with minimal effort. + + + + +## Why Use Dataflow Templates? + +- **Ease of Use**: Templates eliminate the need for coding by offering preconfigured pipelines tailored to specific use cases. +- **Scalability**: Dataflow ensures your pipeline scales efficiently, handling large volumes of data with distributed processing. +- **Cost Efficiency**: Pay only for the resources you consume, with the ability to optimize pipeline execution costs. + +## How to Run Dataflow Templates + +As of today, the ClickHouse official template is available via the Google Cloud CLI or Dataflow REST API. +For detailed step-by-step instructions, refer to the [Google Dataflow Run Pipeline From a Template Guide](https://cloud.google.com/dataflow/docs/templates/provided-templates). + + +## List of ClickHouse Templates +* [BigQuery To ClickHouse](./templates/bigquery-to-clickhouse) +* GCS To ClickHouse (coming soon!) +* Pub Sub To ClickHouse (coming soon!) \ No newline at end of file diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md new file mode 100644 index 00000000000..1e50c79eba5 --- /dev/null +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md @@ -0,0 +1,141 @@ +--- +sidebar_label: BigQuery To ClickHouse +sidebar_position: 1 +slug: /en/integrations/google-dataflow/templates/bigquery-to-clickhouse +description: Users can ingest data from BigQuery into ClickHouse using Google Dataflow Template +--- + +import TOCInline from '@theme/TOCInline'; + +# Dataflow BigQuery to ClickHouse template + +The BigQuery to ClickHouse template is a batch pipeline that ingests data from BigQuery table into ClickHouse table. +The template can either read the entire table or read specific records using a provided query. + + + +## Pipeline requirements + +* The source BigQuery table must exist. +* The target ClickHouse table must exist. +* The ClickHouse host Must be accessible from the Dataflow worker machines. + +## Template Parameters + +
+
+ +| Parameter Name | Parameter Description | Required | Notes | +|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **jdbcUrl** | The ClickHouse JDBC URL in the format `jdbc:clickhouse://{host}:{port}/{schema}`. | ✅ | Don't add the username and password as JDBC options. Any other JDBC option could be added at the end of the JDBC URL. For ClickHouse Cloud users, add `ssl=true&sslmode=NONE` to the `jdbcUrl`. | +| **clickHouseUsername** | The ClickHouse username to authenticate with. | ✅ | | +| **clickHousePassword** | The ClickHouse password to authenticate with. | ✅ | | +| **clickHouseTable** | The target ClickHouse table name to insert the data to. | ✅ | | +| **maxInsertBlockSize** | The maximum block size for insertion, if we control the creation of blocks for insertion (ClickHouseIO option). | | A `ClickHouseIO` option. | +| **insertDistributedSync** | If setting is enabled, insert query into distributed waits until data will be sent to all nodes in cluster. (ClickHouseIO option). | | A `ClickHouseIO` option. | +| **insertQuorum** | For INSERT queries in the replicated table, wait writing for the specified number of replicas and linearize the addition of the data. 0 - disabled. | | A `ClickHouseIO` option. This setting is disabled in default server settings. | +| **insertDeduplicate** | For INSERT queries in the replicated table, specifies that deduplication of inserting blocks should be performed. | | A `ClickHouseIO` option. | +| **maxRetries** | Maximum number of retries per insert. | | A `ClickHouseIO` option. | +| **InputTableSpec** | The BigQuery table to read from. Specify either `inputTableSpec` or `query`. When both are set, the `query` parameter takes precedence. Example: `:.`. | | Reads data directly from BigQuery storage using the [BigQuery Storage Read API](https://cloud.google.com/bigquery/docs/reference/storage). Be aware of the [Storage Read API limitations](https://cloud.google.com/bigquery/docs/reference/storage#limitations). | +| **outputDeadletterTable** | The BigQuery table for messages that failed to reach the output table. If a table doesn't exist, it is created during pipeline execution. If not specified, `_error_records` is used. For example, `:.`. | | | +| **query** | The SQL query to use to read data from BigQuery. If the BigQuery dataset is in a different project than the Dataflow job, specify the full dataset name in the SQL query, for example: `..`. Defaults to [GoogleSQL](https://cloud.google.com/bigquery/docs/introduction-sql) unless `useLegacySql` is true. | | You must specify either `inputTableSpec` or `query`. If you set both parameters, the template uses the `query` parameter. Example: `SELECT * FROM sampledb.sample_table`. | +| **useLegacySql** | Set to `true` to use legacy SQL. This parameter only applies when using the `query` parameter. Defaults to `false`. | | | +| **queryLocation** | Needed when reading from an authorized view without the underlying table's permission. For example, `US`. | | | +| **queryTempDataset** | Set an existing dataset to create the temporary table to store the results of the query. For example, `temp_dataset`. | | | +| **KMSEncryptionKey** | If reading from BigQuery using the query source, use this Cloud KMS key to encrypt any temporary tables created. For example, `projects/your-project/locations/global/keyRings/your-keyring/cryptoKeys/your-key`. | | | + +## Source and Target Tables Schema + +In order to effectively load the BigQuery dataset to ClickHouse, and a column infestation process is conducted with the +following phases: + +1. The templates build a schema object based on the target ClickHouse table. +2. The templates iterate over the BigQuery dataset, and tried to match between column based on their names. + +
+ +:::important +Having said that, your BigQuery dataset (either table or query) must have the exact same column names as your ClickHouse +target table. +::: + +## Data Types Mapping + +The BigQuery types are converted based on your ClickHouse table definition. Therefore, the above table lists the +recommended mapping you should have in your target ClickHouse table (for a given BigQuery table/query): + + +| BigQuery Type | ClickHouse Type | Notes | +|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [**Array Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array_type) | [**Array Type**](https://clickhouse.com/docs/en/sql-reference/data-types/array) | The inner type must be one of the supported primitive data types listed in this table. | +| [**Boolean Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#boolean_type) | [**Bool Type**](https://clickhouse.com/docs/en/sql-reference/data-types/boolean) | | +| [**Date Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#date_type) | [**Date Type**](https://clickhouse.com/docs/en/sql-reference/data-types/date) | | +| [**Datetime Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#datetime_type) | [**Datetime Type**](https://clickhouse.com/docs/en/sql-reference/data-types/datetime) | Works as well with `Enum8`, `Enum16` and `FixedString`. | +| [**String Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#string_type) | [**String Type**](https://clickhouse.com/docs/en/sql-reference/data-types/string) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). | +| [**Numeric - Integer Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Integer Types**](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). The template will also convert unassigned Int types if used in ClickHouse table (`UInt8`, `UInt16`, `UInt32`, `UInt64`). | +| [**Numeric - Float Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Float Types**](https://clickhouse.com/docs/en/sql-reference/data-types/float) | Supported ClickHouse types: `Float32` and `Float64` | + +## Running the Template + +The BigQuery to ClickHouse template is available for execution via the Google Cloud CLI. + +:::note +Be sure to review this document, and specifically the above sections, to fully understand the template's configuration +requirements and prerequisites. + +::: + +### Install & Configure `gcloud` CLI + +- If not already installed, install the [`gcloud` CLI](https://cloud.google.com/sdk/docs/install). +- Follow the `Before you begin` section + in [this guide](https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates#before-you-begin) to set + up the required configurations, settings, and permissions for running the DataFlow template. + +### Run Command + +Use the [`gcloud dataflow flex-template run`](https://cloud.google.com/sdk/gcloud/reference/dataflow/flex-template/run) +command to run a Dataflow job that uses the Flex Template. + +Below is an example of the command: + +```bash +gcloud dataflow flex-template run "bigquery-clickhouse-dataflow-$(date +%Y%m%d-%H%M%S)" \ + --template-file-gcs-location "gs://clickhouse-dataflow-templates/bigquery-clickhouse-metadata.json" \ + --parameters inputTableSpec="{bigquery table id}",jdbcUrl="jdbc:clickhouse://{clickhouse host}:{clickhouse port}/{schema}?ssl=true&sslmode=NONE",clickHouseUsername="{username}",clickHousePassword="{password}",clickHouseTable="{clickhouse target table}" +``` + +### Command Breakdown + +- **Job Name:** The text following the `run` keyword is the unique job name. +- **Template File:** The JSON file specified by `--template-file-gcs-location` defines the template structure and + details about the accepted parameters. The mention file path is public and ready to use. +- **Parameters:** Parameters are separated by commas. For string-based parameters, enclose the values in double quotes. + +### Expected Response + +After running the command, you should see a response similar to the following: + +```bash +job: + createTime: '2025-01-26T14:34:04.608442Z' + currentStateTime: '1970-01-01T00:00:00Z' + id: 2025-01-26_06_34_03-13881126003586053150 + location: us-central1 + name: bigquery-clickhouse-dataflow-20250126-153400 + projectId: ch-integrations + startTime: '2025-01-26T14:34:04.608442Z' +``` + +### Monitor the Job + +Navigate to the [Dataflow Jobs tab](https://console.cloud.google.com/dataflow/jobs) in your Google Cloud Console to +monitor the status of the job. You’ll find the job details, including progress and any errors: + +DataFlow running job + +## Template Source Code + +The template's source code is available in ClickHouse' +s [DataflowTemplates](https://github.com/ClickHouse/DataflowTemplates) fork. \ No newline at end of file diff --git a/docs/en/integrations/images/logos/dataflow_logo.png b/docs/en/integrations/images/logos/dataflow_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..fa55e09bd2d94120636a27acd05a7da6b28cee4e GIT binary patch literal 8305 zcma)icUTi&(C#J#2wkd_P^5QIK$=u(g3?8b3P|WJpdeC0K$>(AgLDw-(jheIQUnoD z>C(j@9i-Q9@pr#}?)~TH+2_gbp0j7?%$a%TojLnTUsscYl$jI&0E#4Qc8 zbBV%PdDF9&&7UU<_=gw{c<}$nq1lEcwvv9Q=>rS;BSMO|0Gbnm)NOmzlBkH;p-BcA zl?U;U_hyC+u<@*~GdLi z&n3^$!4T~%H#^_uqMX2r8L8kX+ z51(ZeJLhIgDOpE*+%sSarz~ZD%^HzXkEYch#-FHADOsndu08J=sW%7A!8aPu-A*DS zh-#c>Laozq_=ad22Rf1O?(+%SJWQh)X{gojt!2V2=&=C`r)o<-kgYmppWZMjdoON= zt;xU-vIU%)S)^f)QXu@P-iF@fAx)^&=Y3vG%DAtA`re3;lc8rb3pg}n%oY%|Ijgea z`1z{i?P?Y9fJE6@ZwFw!*%$i~#TT8EQ`_^1CAiEC8o#b-8dBn-GKK^q1B+R9w`hMG zy*?{zfIdJ1$P;RlMFxu@dM3L(FIUb;O02DhFO^-jSV-4HhACbx ztG0vN&k^-c3`2)#($;vV;^K)qH43JJW2a%bcEE^2i=*I!zMg zU3eTQ`ZzrwTAGyyh|Fp{gR^v2RPx-M5RoV6lCh(W|9wJB*=~T#wz&bQe{Z~8hd`$) zhFUI^8j-S13%p>S&SHp~3G*)7!kZjs4J#NA#$5&Ero&O3x|M{k&9DfzdFon0DQ)?S z5U%<9qJdE6^^eyaR>?6gN*%-EE;T1bs<;8b;mg%|8kc}C^{x^?UFIiMpV-t7p{_bE zDG*S+dWxn@L1aI9TMS>tm6`4`6Wls1TM9eb$;b?v(iKingQKQSaceR=l_NBmZhz{s z?l4?->5YG4NZqU`04qLRD~|5TbH3uZ`0=rg6GI`hrq z!}e+j_jiQ-OiFFQ$HW9h#k<8o%_l=XrbC?@HWfm(Qsl0d2aB`liMr)MrQ_PLi(_fW zPdyFqd$9$$u=$f&(roLOjmv`<=Zh~so>!Yqbva*^IUH^;lbWvD5W5E~Kg(JoA!hPc zFbI95cYKAE?tJfbZ$$R=kR2bNo^WEdv~cG3^nQERojh6eDb-(G#8|9Dq`6o>LlQ3_ zQu+D7;oIPomLq!h$<#B`2S$_SJa6t*UyLaYq#-7D{Qn9mu1&328Wb{k`lT=KmFz6k z3V}HAQY<|0E>$8YFg#PlDmWI%xj5tfE5VC&-)&FqPL|HngRB*7S>4kU8q4GLo&EUB zS&hx4bnQ(w&h@5830vq#0yX>GJq~!K*NU4{jn6VHMK>>0JDoS*)+y;*bXGif2K1E7 zGxeAL9M<5wXoTR>uZ$wG|7h zxLL;KGo{g)QPtl4{+CZr+h01?)P)S2^6A0>k5$*)6Pmmma#)SoHkgLn`}VeMqEUK&VbpoL8V3QfpI3zt(yS-j7OK2=E)-86q>r&VFtVN|mVfRai`e8_M*dhERkSG|9+#ddOF)9~%D4KOes=SHNj6i+G|>S0C6XpPk_B z0*8-ZdNQ`$j4?{?rq-jgjE#o_2!DqP=K%(x4ck`$(kb&r7X5*zB$*ljq^=N*w<-i{ zI+0zSNa|eoh?UrUzkM!tx#YGyG=`Z1lv!Rp@aM>z%(Wv-KT-J)D^2))a`;Hr!VaK;L88DT@8 z*&<1s^gfCB9YWmgp`_2gXYv?bBFEiqEnWaf8+~AXu^-`hLXd0YnBJ1qCgwhn;(2Dc z<;0tPeJL)SvK(!(>lJDb40GLrqI^%H^1-K&>A@ARbRgB`mZXmdV8+ww!pc5G%DAl0 z!mml*-j_9v0ADO4EpFYHZ&~DXomSk&|E)eG`+I;eXZiEF&;e_KGz?zpZkg$n`%N}F z$UH84D3(h7yYA#-A$4*mLP zVj$0g`KKq^th8=?|Fr<4ibM1(<%fcA9)rM}5^lCw|KZ)xF1y5bG#vHw$8Xh`H*Ro~ zk6s@R3&7aXs?TOtFelbZ`{^Kyem2*grE!%bWG z+50E<-UJgcfqo+HjCgP>XMQ&iR`xiJ2&kzuNxtP$p`=*7 zrOK=S`LLqOQW~TOOK5E2sZ51x#7t1DuPZa?Mjb+w&Rl{|0{1QQr5Ru3m}(geQCR7i zWr^T#gNW?5bK1`Z0#|z-^l($GjOKic$}&ykBt>tS&w=|v?62t9mLQUvtRuS4;MwsQ zFG%`y6J_{wMJw~%gsEf#a^ciJZ6f$;qH9y-{$H4z9`)@`dnN<@)}u8G5BICSH4C+m z^atH=Rq0GjD^<@Ap9PV&cwr!*WPAFzU=CHFlSa>z+YEezHYLlEBB;K<@^2StiP403Z5>r&^0Ao zB^4ZR72(`XbsK<1lm8OA1JDKTfa^VKgDthcDLb_lO6xI?%W?S04+ROGu|>slNu=(< z_3ozd+w_)PnM0`4g+~INQeU!*6fw8u*{@UyA07*}QePQ1d{~d*cl0+MCdcR=RK}*4 z9(262D0^`j6zzHlo-MPaUvXlI{1lp^d`|#37s&*SoUj!9Bya1}l}#~rVo7}i`GvCJ z)&Iiqu<|O0h0Wc!5v-I`6^ZGtJ}T~C2uO3H%9EXY^DwT#F z!tr->^Oj$a1l5Hq7_x?6?8oM5roM?x!xBW#H(Zj*&~x+z`6VlEGn$rMn0n}TN6s|& zsPeu;G1XkcT2kg}-EVPEE|D&~q`ix5Mz=m#RYsFn|JigEG=Aci74oDZ`rc~Em6K{Q z;fWY`aIKC8tUu8*H_n^}W`5Dac1m%!RaDmUy|9{F+Dbge@aSNcxBtk=P4)-4IFrt= z&3Fl!dw{NOe^4foN8M9YZztO}lp$O9^tukL%|Yb4b)cs`F}80$!`XnP^5BTTWPygd z@H6GsqfeZW2)4AJ9gLaP!rUD+a^@uz_jX7K>7SX3*A3pOSAj7e>I^S__aeJ+8LsQl zqPRYf=-d7|;IQc>=XNFNRHY>g{~~45ddA92*)2B^MT7YGK8v+Bd$PQ!#XqaLNYTny zePKC9^4RJtj6(u-+|5A{r4e_}?XwrzjODV2E-ni=KR#Qtc&A3aB?r23tdO5~BU~84 zSqx0nDS4fc1PM6(j#6=2aZ3M)=(U9?n*cI+LLEYq#qm2Z!R_55HY?k7VJNAMAE5jS zw9Uz*t8tIiIAn8^M_w!WyH4)K+~GtO!^OjH0$WAr0=_3{%AFeYO_}g_9*e1ie6W~U z+dfyZ3;(`v;UysKSW1}wUU@~H7iu(dBd)71sqD*q9-JXafl4zs$I*tw%yXAS}8w1jyT11}x@VW#RQUCaw z-OSMO{X?hD@EFBbS(jyHIwlzhz3gt17m-{($HP15;l~>vg$`ynLN4reJ&3gcn1Mw; zcG8s_`zGncz5J2$YTXK6Mg^t?z&q5|M4{Y3;w!$jqpk{Q`RUZUFnAqYk8b*CSw_zq z>%s>`NC(WhkdsTETLAVmcNW73iFMA8=AFM|lj|Aenx!*`hQPRA#1ZpT%jR&h zCg;Y|_nYMq{;&*TRm$k%*1s`ja1$9B41b26;$p;XTIU#g%yQTMlEWPi3<{SxkrC6M z7Os2N8Ih-8 z)|zT#6DfC)^-FzT6ul>aIvW*6Z_XC>D>^9W5C2|1YQkb`&cTs+CfsVN+0lV7*^@=& zDr~$n?fd+s9;18>epeVD+&Yy-MqwzI{Kx@w-$ZvKoeI<4BgvMX_^@t5%az0CFUKP* zf;YKhlx07-!J!-W*T@3S4TyEgZv@v=S^t*-_ycW>ek^m`0Mx)Aa{Ar<8)L(@z?nyW zOtk1JixQEmm|xMNu13ZTRr@fR#WUqmBE&N3-pj>&+vf-HNHL&Rd*j*kWQ2QHr!r}v zZeBG+ST#&OTc`zXWz7oMOk<~~Z}&0;?a?OsUoqt;XR{(5Zsc!22cqfziw$K*`;{fq^bWo zY_ZhucWca~Mal4t;W1m+nB`H-tKr_BH3bMI`o_UN@s4)TWjTfoe^|8DMDy)%skEWW zX=9Ou!Xu4sdfoeOKTO7Lr8-Iv{`iUVO0lo<+9Q&Qhli+diGx=b#MNPpt88+83~vlY zsWCaXOS)wa4LqLnAfL{^g(5yL6THIQ>aaZWxYYacVOq72>$)u z^lIDD%UTEbQqghMH6jopGp4v?nx1DA6kL0tUE~K%eMcAI9fh9iWvGeW$2vWfF#B`rs#=ZSCPl}4{47CzMQW#=AFvGW@MVX;+e&ca^6T!8JT zJ4q@wkAQM-f9eMs`KT4ccTb$lDi!rT2LjjyuV#Yqz*;)ulm99;`wLQ*dJa)e)r?>k zS^CQ(CPUn)dM2q-FH=W2*!YjJA`ghuRsEc$j`z=STj8@pcsL`EyKKd~_2jr5)&kWL zwNC-(yRCUcN;0pK+McB9>#O+bNPT60Xo+&mWOYEh(1Yi3pcXpA(Pe62WSE_UJ(ls% z8JC*w#&c#5v^FNLC+iuoh-YClkxDiW634P)?OqfmrlcozV{a3o$u=y^j!jaN`49g~ z)AHBWqRgQ$ps{*o-E1>fzqrC|<>c>Tj$tMk0m1wi@E)<5&ZzR~?&Y`RIau#XicS^6 z_0kcUzzUQ9csbanvDwwMyOj*L_YEH&WXzO7{hc;_x4{w~OzsjjJm^c(O$P>xEFL=) zNQHbv)9|oBMn(k3GbYbUP&M zdvdo2n9x3dy*rmoif3etb@l05#`mQj7Y>H$72hvXuo^f0if@J*85aggfeAiH7du9r zzmK08Gm5!Tl38$x)lleJYQKHFFlEfH?B`s&C5;Ao^AXkS`Yn~jnZu>Izsa@2V$Q#Z z26A#G_GXv0F}ieu+M={`lfrg*Sq=C1mtoJeTDzTVwd=(qh?bdtoY8yE0gMd^k3E;Q z@|cum3g?v2ej@wsaquh#e=xtC#&13W;XW@subz;|`}ayE@hm^S+O{>Umq9q=A#Yoi zU)lDjTOfVz(!p=QXW4rv!kV&Z9(dH96p4c=4Sh?FESU0uZP^D4j|O*DBEkr0kNw9{ zD_Oam0%Km!N=OSWDqswv#4w%WdWf_A<(%h0|t;0zMo>8dDSuwWol z^A}?6u@vudrO09)(KB-~nBcdBo~fXCOEnSPJ=ciSf_a9#jNCd4LUfMhC#28ww6>Q= z-@S%FhUu=05Sru3^2w*fvLeBvvwx%RNUec>mhxvh8VE-5Y=Q zMJ>uQyQQ59EEmst#tKaVAR8<_4z$Ha)MFo*-hlxJlEV+<2Em*xq=f1)Nu_(=ufE~i zDxCUX-MYju7fHsP`xXu`wKhNXi`lXbns{&b>~h>y|JQTH7doAqV`t8JW7M1gFvP60 zp5vynn8XQfID>HUE%&~6M+J`_@piO5QCfYmY42p{YW^lKg*u& z)pCTgg8p@02@@JWvW(FGM^N7hN~+Z$wCN#j=o5mPuXR^g1JShy^B31vJXHw+4vPw! z=m>x#Fc33(G}FqOa#0PHQ>#>XxHPKMk@dvS2;;B+Awacl?O8uVuc~ z#81dCvj_*ohZ`Id@eRG$r&KtlnQJN5l_+!8CIaF`_@P`WnERBF^<9fNttt0Ewf90? z+WQ1}p;b;W2dXw^wJG$`TR{Lx7MT+dkJ#XpB=y=e!&ps;u^U}ZMW2pLz3oHMttGHT z#^4)4lAvPTe4+hT6!;;+AL})Om<=$mT(@Sqgg7^=;C*7)wRcnHjQM1MGA-fs|N&1+sdgDtC-RysLC?pHR%SGCu+1xkQ|<^JiM+4kzGk81V0 z@tpR8r;y1w`p8riRNc(KLhc|al7Wj+{<8G?WufVc|ALrujQWwO^~sXW3lgRb5udEm zz_#smB|1Ocj0HQd2guhVGuQ1y0*{akN@qQE(EH@A2Njm%Ud~*KpL+1yVp(5EoAQl5 zkE2=8$i0BPw|(HN@KGi9Tbr#7jSIYNCEiogrR5#x>j{wu#Dzu6igsfs7 zRPPK%)IXA3o&YJw+_Nn0c)|bOSzvmZ(fbO7t?b-d=0W$z=a|9E3YC|DHH)4C!g#1b z!LULz$O$0rZ1<)*xV4M@aH1n7%x8hRE=vu7wc_;|ujJW0t@{`uuFm~_%|<{iZYyF% zFDA&Peb9^_s%+QHN+}ror)d_fH+pImC}N&0udqK}57@%rxE(JF0A6IGnl11v8y&@N ztxv`m+8QqFv_qq7$*jayVA&G5XF%B-+wK+*ucswkt05l@ zly#|u^{U|Jc7{CL)U|MDJC*mqZ9_SJGE>Sv*iQm55UnvXJQ0F0u8_uf#*W$cSZiv* z(y8!_xWNWE-jIFcO+Aw zMr8&7ngzp4wKg-9m1-7ME2n(yQbpeX-jqef+IUxjBE0#Bs9o2hhJKHK0nHJ|1p1AY zfIhF4`vN??XX{ zcYx7Aj*zacv{j4zgqrY6N2Jz*{pxszs3`#nL#b<~k*2#NT)!o)Skb^|E*q=yzvIHbC@x(R$T7j_A42rbx$pp#-b+a4>r9uiQVB^6F9 zQC)C09_g3!!;-zi$jW>MrAfGy##EFWy5Irwi@02u%NGl?SHZii!oLRywpad}ERQ}f z2Rxts4XY}Tj5*ND@PV8rsBaG8E;9fQG5l``0dDA0dl1RtY2Y>aSZ$Awm&3|(L%R*y zbmcFny?xn>A4&ruK>05igUI#Z->l(UFzt}VaNSx^w3>g4e!>w|A+B2oUSF<#HwimMe0*(&eixh=2e011 zAeixf$euY~u;z^FK7YB!_@$%vQU}>gJ|eWJG*Cy*5&*_UMPkdGx2^s-5$y131R7sS z*ihncAwNe?ZDmzi(oA*!&>La5u)?Dp3-WM&8;5w-@F~<^&o;-uXZ>9xOv;o2qBVrZ zEF6eBJ7|i@Y}?bS#R{k%ybatd9%{p3)Rbl1XhBdFf8^?MyTo0<=LtzX7qgC3@!Te) zFo88{3f1Y!aj?}CFx`4Z4++R#?jJpv%m|c+3AZE;L^^&8E8@Uc?@{|Yi~qjf;J(xFx84QI2eBsfw!s71GE6_wST+V z{PLH(;Znh{92aC1fM+yeJkcVaVcB@GEaDh+1*Y|Data ingestion|Open-source workflow management platform for data engineering pipelines|[Github](https://github.com/bryzgaloff/airflow-clickhouse-plugin)| |Apache Beam|Beam logo|Data ingestion|Open source, unified model and set of language-specific SDKs for defining and executing data processing workflows. Compatible with Google Dataflow.|[Documentation](https://clickhouse.com/docs/en/integrations/apache-beam),
[Examples](https://github.com/ClickHouse/clickhouse-beam-connector/)| +|Google Dataflow|Dataflow logo|Data ingestion|Google Dataflow is a serverless service for running batch and streaming data pipelines using Apache Beam.|[Documentation](https://clickhouse.com/docs/en/integrations/google-dataflow/dataflow)| |Apache InLong|InLong logo|Data ingestion|One-stop integration framework for massive data|[Documentation](https://inlong.apache.org/docs/data_node/load_node/clickhouse)| |Apache NiFi|NiFi logo|Data ingestion|Automates the flow of data between software systems|[Documentation](/docs/en/integrations/nifi)| |Apache SeaTunnel|SeaTunnel logo|Data ingestion|SeaTunnel is a very easy-to-use ultra-high-performance distributed data integration platform|[Website](https://seatunnel.apache.org/docs/2.3.0/connector-v2/sink/Clickhouse)| diff --git a/sidebars.js b/sidebars.js index 6dbc49efd9e..79e8261b7c8 100644 --- a/sidebars.js +++ b/sidebars.js @@ -839,6 +839,28 @@ const sidebars = { "en/integrations/data-ingestion/etl-tools/airbyte-and-clickhouse", "en/integrations/data-ingestion/aws-glue/index", "en/integrations/data-ingestion/etl-tools/apache-beam", + { + type: "category", + label: "Google Dataflow", + className: "top-nav-item", + collapsed: true, + collapsible: true, + items: [ + "en/integrations/data-ingestion/google-dataflow/dataflow", + "en/integrations/data-ingestion/google-dataflow/java-runner", + "en/integrations/data-ingestion/google-dataflow/templates", + { + type: "category", + label: "Dataflow Templates", + className: "top-nav-item", + collapsed: true, + collapsible: true, + items: [ + "en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse", + ], + }, + ], + }, "en/integrations/data-ingestion/etl-tools/dbt/index", "en/integrations/data-ingestion/etl-tools/dlt-and-clickhouse", "en/integrations/data-ingestion/etl-tools/fivetran/index", From 80803a9b0929b7ec6257b65ef70ce334ba0d759f Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Mon, 27 Jan 2025 16:26:03 +0100 Subject: [PATCH 02/19] Fix extra space --- .../google-dataflow/templates/bigquery-to-clickhouse.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md index 1e50c79eba5..61f1d0ca11e 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md @@ -137,5 +137,4 @@ style={{width: '100%', 'background-color': 'transparent'}}/> ## Template Source Code -The template's source code is available in ClickHouse' -s [DataflowTemplates](https://github.com/ClickHouse/DataflowTemplates) fork. \ No newline at end of file +The template's source code is available in ClickHouse's [DataflowTemplates](https://github.com/ClickHouse/DataflowTemplates) fork. \ No newline at end of file From 44017ea5b3c17db65cf1f7ac76f79a78d1e92299 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Mon, 27 Jan 2025 16:49:27 +0100 Subject: [PATCH 03/19] exclude words from spell check --- scripts/aspell-ignore/en/aspell-dict.txt | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/scripts/aspell-ignore/en/aspell-dict.txt b/scripts/aspell-ignore/en/aspell-dict.txt index 561b53b46e8..5a5a039f13e 100644 --- a/scripts/aspell-ignore/en/aspell-dict.txt +++ b/scripts/aspell-ignore/en/aspell-dict.txt @@ -3452,3 +3452,23 @@ znode znodes zookeeperSessionUptime zstd +DataFlow +Dataflow +DataflowTemplates +GoogleSQL +InputTableSpec +KMSEncryptionKey +clickHousePassword +clickHouseTable +clickHouseUsername +insertDeduplicate +insertDistributedSync +insertQuorum +jdbcUrl +linearize +maxInsertBlockSize +maxRetries +outputDeadletterTable +queryLocation +queryTempDataset +useLegacySql \ No newline at end of file From a11a2ef10072808a63ceb65b07ad9749c9961789 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Tue, 28 Jan 2025 10:36:34 +0100 Subject: [PATCH 04/19] rename folder name to images --- .../{assets => images}/dataflow-inqueue-job.png | Bin .../templates/bigquery-to-clickhouse.md | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) rename docs/en/integrations/data-ingestion/google-dataflow/{assets => images}/dataflow-inqueue-job.png (100%) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/assets/dataflow-inqueue-job.png b/docs/en/integrations/data-ingestion/google-dataflow/images/dataflow-inqueue-job.png similarity index 100% rename from docs/en/integrations/data-ingestion/google-dataflow/assets/dataflow-inqueue-job.png rename to docs/en/integrations/data-ingestion/google-dataflow/images/dataflow-inqueue-job.png diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md index 61f1d0ca11e..33c82b61c00 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md @@ -132,7 +132,7 @@ job: Navigate to the [Dataflow Jobs tab](https://console.cloud.google.com/dataflow/jobs) in your Google Cloud Console to monitor the status of the job. You’ll find the job details, including progress and any errors: -DataFlow running job ## Template Source Code From 316bbad2aeaca4c854dc291122187321b8832816 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Tue, 28 Jan 2025 11:00:01 +0100 Subject: [PATCH 05/19] put settings in `` --- .../google-dataflow/templates/bigquery-to-clickhouse.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md index 33c82b61c00..8a3f6d2d3c2 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md @@ -64,7 +64,6 @@ target table. The BigQuery types are converted based on your ClickHouse table definition. Therefore, the above table lists the recommended mapping you should have in your target ClickHouse table (for a given BigQuery table/query): - | BigQuery Type | ClickHouse Type | Notes | |-----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [**Array Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array_type) | [**Array Type**](https://clickhouse.com/docs/en/sql-reference/data-types/array) | The inner type must be one of the supported primitive data types listed in this table. | From 3ec5b5982353ea930a0a5581abbcfeca72e88935 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Tue, 28 Jan 2025 11:00:08 +0100 Subject: [PATCH 06/19] put settings in `` --- .../templates/bigquery-to-clickhouse.md | 36 +++++++++---------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md index 8a3f6d2d3c2..e67c4982b12 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md @@ -25,24 +25,24 @@ The template can either read the entire table or read specific records using a p

-| Parameter Name | Parameter Description | Required | Notes | -|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **jdbcUrl** | The ClickHouse JDBC URL in the format `jdbc:clickhouse://{host}:{port}/{schema}`. | ✅ | Don't add the username and password as JDBC options. Any other JDBC option could be added at the end of the JDBC URL. For ClickHouse Cloud users, add `ssl=true&sslmode=NONE` to the `jdbcUrl`. | -| **clickHouseUsername** | The ClickHouse username to authenticate with. | ✅ | | -| **clickHousePassword** | The ClickHouse password to authenticate with. | ✅ | | -| **clickHouseTable** | The target ClickHouse table name to insert the data to. | ✅ | | -| **maxInsertBlockSize** | The maximum block size for insertion, if we control the creation of blocks for insertion (ClickHouseIO option). | | A `ClickHouseIO` option. | -| **insertDistributedSync** | If setting is enabled, insert query into distributed waits until data will be sent to all nodes in cluster. (ClickHouseIO option). | | A `ClickHouseIO` option. | -| **insertQuorum** | For INSERT queries in the replicated table, wait writing for the specified number of replicas and linearize the addition of the data. 0 - disabled. | | A `ClickHouseIO` option. This setting is disabled in default server settings. | -| **insertDeduplicate** | For INSERT queries in the replicated table, specifies that deduplication of inserting blocks should be performed. | | A `ClickHouseIO` option. | -| **maxRetries** | Maximum number of retries per insert. | | A `ClickHouseIO` option. | -| **InputTableSpec** | The BigQuery table to read from. Specify either `inputTableSpec` or `query`. When both are set, the `query` parameter takes precedence. Example: `:.`. | | Reads data directly from BigQuery storage using the [BigQuery Storage Read API](https://cloud.google.com/bigquery/docs/reference/storage). Be aware of the [Storage Read API limitations](https://cloud.google.com/bigquery/docs/reference/storage#limitations). | -| **outputDeadletterTable** | The BigQuery table for messages that failed to reach the output table. If a table doesn't exist, it is created during pipeline execution. If not specified, `_error_records` is used. For example, `:.`. | | | -| **query** | The SQL query to use to read data from BigQuery. If the BigQuery dataset is in a different project than the Dataflow job, specify the full dataset name in the SQL query, for example: `..`. Defaults to [GoogleSQL](https://cloud.google.com/bigquery/docs/introduction-sql) unless `useLegacySql` is true. | | You must specify either `inputTableSpec` or `query`. If you set both parameters, the template uses the `query` parameter. Example: `SELECT * FROM sampledb.sample_table`. | -| **useLegacySql** | Set to `true` to use legacy SQL. This parameter only applies when using the `query` parameter. Defaults to `false`. | | | -| **queryLocation** | Needed when reading from an authorized view without the underlying table's permission. For example, `US`. | | | -| **queryTempDataset** | Set an existing dataset to create the temporary table to store the results of the query. For example, `temp_dataset`. | | | -| **KMSEncryptionKey** | If reading from BigQuery using the query source, use this Cloud KMS key to encrypt any temporary tables created. For example, `projects/your-project/locations/global/keyRings/your-keyring/cryptoKeys/your-key`. | | | +| Parameter Name | Parameter Description | Required | Notes | +|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `jdbcUrl` | The ClickHouse JDBC URL in the format `jdbc:clickhouse://{host}:{port}/{schema}`. | ✅ | Don't add the username and password as JDBC options. Any other JDBC option could be added at the end of the JDBC URL. For ClickHouse Cloud users, add `ssl=true&sslmode=NONE` to the `jdbcUrl`. | +| `clickHouseUsername` | The ClickHouse username to authenticate with. | ✅ | | +| `clickHousePassword` | The ClickHouse password to authenticate with. | ✅ | | +| `clickHouseTable` | The target ClickHouse table name to insert the data to. | ✅ | | +| `maxInsertBlockSize` | The maximum block size for insertion, if we control the creation of blocks for insertion (ClickHouseIO option). | | A `ClickHouseIO` option. | +| `insertDistributedSync` | If setting is enabled, insert query into distributed waits until data will be sent to all nodes in cluster. (ClickHouseIO option). | | A `ClickHouseIO` option. | +| `insertQuorum` | For INSERT queries in the replicated table, wait writing for the specified number of replicas and linearize the addition of the data. 0 - disabled. | | A `ClickHouseIO` option. This setting is disabled in default server settings. | +| `insertDeduplicate` | For INSERT queries in the replicated table, specifies that deduplication of inserting blocks should be performed. | | A `ClickHouseIO` option. | +| `maxRetries` | Maximum number of retries per insert. | | A `ClickHouseIO` option. | +| `InputTableSpec` | The BigQuery table to read from. Specify either `inputTableSpec` or `query`. When both are set, the `query` parameter takes precedence. Example: `:.`. | | Reads data directly from BigQuery storage using the [BigQuery Storage Read API](https://cloud.google.com/bigquery/docs/reference/storage). Be aware of the [Storage Read API limitations](https://cloud.google.com/bigquery/docs/reference/storage#limitations). | +| `outputDeadletterTable` | The BigQuery table for messages that failed to reach the output table. If a table doesn't exist, it is created during pipeline execution. If not specified, `_error_records` is used. For example, `:.`. | | | +| `query` | The SQL query to use to read data from BigQuery. If the BigQuery dataset is in a different project than the Dataflow job, specify the full dataset name in the SQL query, for example: `..`. Defaults to [GoogleSQL](https://cloud.google.com/bigquery/docs/introduction-sql) unless `useLegacySql` is true. | | You must specify either `inputTableSpec` or `query`. If you set both parameters, the template uses the `query` parameter. Example: `SELECT * FROM sampledb.sample_table`. | +| `useLegacySql` | Set to `true` to use legacy SQL. This parameter only applies when using the `query` parameter. Defaults to `false`. | | | +| `queryLocation` | Needed when reading from an authorized view without the underlying table's permission. For example, `US`. | | | +| `queryTempDataset` | Set an existing dataset to create the temporary table to store the results of the query. For example, `temp_dataset`. | | | +| `KMSEncryptionKey` | If reading from BigQuery using the query source, use this Cloud KMS key to encrypt any temporary tables created. For example, `projects/your-project/locations/global/keyRings/your-keyring/cryptoKeys/your-key`. | | | ## Source and Target Tables Schema From 83007eaad6ec84969f5a1f18677f72a62e271f80 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Tue, 28 Jan 2025 11:00:31 +0100 Subject: [PATCH 07/19] add a file level specific exclusion --- scripts/aspell-ignore/en/aspell-dict.txt | 22 +------------------ .../google-dataflow/aspell-dict.txt | 4 ++++ 2 files changed, 5 insertions(+), 21 deletions(-) create mode 100644 scripts/aspell-ignore/en/integrations/data-ingestion/google-dataflow/aspell-dict.txt diff --git a/scripts/aspell-ignore/en/aspell-dict.txt b/scripts/aspell-ignore/en/aspell-dict.txt index 13f85f89f2b..043657ad46e 100644 --- a/scripts/aspell-ignore/en/aspell-dict.txt +++ b/scripts/aspell-ignore/en/aspell-dict.txt @@ -3452,24 +3452,4 @@ znode znodes zookeeperSessionUptime zstd -DataFlow -Dataflow -DataflowTemplates -GoogleSQL -InputTableSpec -KMSEncryptionKey -clickHousePassword -clickHouseTable -clickHouseUsername -insertDeduplicate -insertDistributedSync -insertQuorum -jdbcUrl -linearize -maxInsertBlockSize -maxRetries -outputDeadletterTable -queryLocation -queryTempDataset -useLegacySql -Okta +Okta \ No newline at end of file diff --git a/scripts/aspell-ignore/en/integrations/data-ingestion/google-dataflow/aspell-dict.txt b/scripts/aspell-ignore/en/integrations/data-ingestion/google-dataflow/aspell-dict.txt new file mode 100644 index 00000000000..df817c4e44d --- /dev/null +++ b/scripts/aspell-ignore/en/integrations/data-ingestion/google-dataflow/aspell-dict.txt @@ -0,0 +1,4 @@ +DataFlow +Dataflow +DataflowTemplates +GoogleSQL From a09a8449b3d9af9a9a99a0dc8bac4a4b62848b80 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Tue, 28 Jan 2025 12:40:17 +0100 Subject: [PATCH 08/19] add troubleshooting section --- .../templates/bigquery-to-clickhouse.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md index e67c4982b12..b795c3683d4 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md @@ -134,6 +134,16 @@ monitor the status of the job. You’ll find the job details, including progress DataFlow running job +## Troubleshooting + +### Code: 241. DB::Exception: Memory limit (total) exceeded + +This error occurs when ClickHouse runs out of memory while processing large batches of data. To resolve this issue: + +* Increase the instance resources: Upgrade your ClickHouse server to a larger instance with more memory to handle the data processing load. +* Decrease the batch size: Adjust the batch size in your Dataflow job configuration to send smaller chunks of data to ClickHouse, reducing memory consumption per batch. +These changes might help balance resource usage during data ingestion. + ## Template Source Code The template's source code is available in ClickHouse's [DataflowTemplates](https://github.com/ClickHouse/DataflowTemplates) fork. \ No newline at end of file From 4474f6fcc0112ab7d2cb954d828ed308226e633d Mon Sep 17 00:00:00 2001 From: Bentsi Leviav Date: Tue, 28 Jan 2025 13:08:31 +0100 Subject: [PATCH 09/19] Update docs/en/integrations/data-ingestion/google-dataflow/dataflow.md Co-authored-by: Mikhail Shustov --- docs/en/integrations/data-ingestion/google-dataflow/dataflow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md b/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md index 72118979c4b..d83aefe5858 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md @@ -9,7 +9,7 @@ description: Users can ingest data into ClickHouse using Google Dataflow [Google Dataflow](https://cloud.google.com/dataflow) is a fully managed stream and batch data processing service. It supports pipelines written in Java or Python and is built on the Apache Beam SDK. -There are two main ways to use Google Dataflow with ClickHouse, both are leveraging [`ClickHouseIO`](../../apache-beam): +There are two main ways to use Google Dataflow with ClickHouse, both are leveraging [`ClickHouseIO Apache Beam connector`](../../apache-beam): ## 1. **[Java Runner](./java-runner)** The Java Runner allows users to implement custom Dataflow pipelines using the Apache Beam SDK `ClickHouseIO` integration. This approach provides full flexibility and control over the pipeline logic, enabling users to tailor the ETL process to specific requirements. From 74db9c4136e7ef051b8727718ab6ccabc6bce897 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Tue, 28 Jan 2025 13:10:10 +0100 Subject: [PATCH 10/19] replacing dataflow based on alphabet --- docs/en/integrations/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/integrations/index.mdx b/docs/en/integrations/index.mdx index 888bfa7a772..66768d9dbc5 100644 --- a/docs/en/integrations/index.mdx +++ b/docs/en/integrations/index.mdx @@ -242,7 +242,6 @@ We are actively compiling this list of ClickHouse integrations below, so it's no |------|----|----------------|------------------|-------------| |Apache Airflow|Airflow logo|Data ingestion|Open-source workflow management platform for data engineering pipelines|[Github](https://github.com/bryzgaloff/airflow-clickhouse-plugin)| |Apache Beam|Beam logo|Data ingestion|Open source, unified model and set of language-specific SDKs for defining and executing data processing workflows. Compatible with Google Dataflow.|[Documentation](https://clickhouse.com/docs/en/integrations/apache-beam),
[Examples](https://github.com/ClickHouse/clickhouse-beam-connector/)| -|Google Dataflow|Dataflow logo|Data ingestion|Google Dataflow is a serverless service for running batch and streaming data pipelines using Apache Beam.|[Documentation](https://clickhouse.com/docs/en/integrations/google-dataflow/dataflow)| |Apache InLong|InLong logo|Data ingestion|One-stop integration framework for massive data|[Documentation](https://inlong.apache.org/docs/data_node/load_node/clickhouse)| |Apache NiFi|NiFi logo|Data ingestion|Automates the flow of data between software systems|[Documentation](/docs/en/integrations/nifi)| |Apache SeaTunnel|SeaTunnel logo|Data ingestion|SeaTunnel is a very easy-to-use ultra-high-performance distributed data integration platform|[Website](https://seatunnel.apache.org/docs/2.3.0/connector-v2/sink/Clickhouse)| @@ -254,6 +253,7 @@ We are actively compiling this list of ClickHouse integrations below, so it's no |Chat-DBT| |AI Integration|Create ClickHouse queries using Chat GPT.|[GitHub](https://github.com/plmercereau/chat-dbt)| |ClickHouse Monitoring Dashboard||Dashboard|A simple monitoring dashboard for ClickHouse|[Github](https://github.com/duyet/clickhouse-monitoring)| |Common Lisp|clickhouse-cl Logo|Language client|Common Lisp ClickHouse Client Library|[GitHub](https://github.com/juliojimenez/clickhouse-cl)| +| Dataflow|Dataflow logo|Data ingestion|Google Dataflow is a serverless service for running batch and streaming data pipelines using Apache Beam.|[Documentation](https://clickhouse.com/docs/en/integrations/google-dataflow/dataflow)| |DBNet|Airflow logo|Software IDE|Web-based SQL IDE using Go as a back-end, and the browser as the front-end.|[Github](https://github.com/dbnet-io/dbnet)| |DataLens|Datalens logo|Data visualization|An open-source data analytics and visualization tool.|[Website](https://datalens.tech/),
[Documentation](https://datalens.tech/docs/en/)| |Dataease|Dataease logo|Data visualization|Open source data visualization analysis tool to help users analyze data and gain insight into business trends.|[Website](https://dataease.io/)| From d4d64f0110b3de910e98b4c32e81011beb27baba Mon Sep 17 00:00:00 2001 From: Bentsi Leviav Date: Tue, 28 Jan 2025 13:14:38 +0100 Subject: [PATCH 11/19] Update docs/en/integrations/index.mdx Co-authored-by: Mikhail Shustov --- docs/en/integrations/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/integrations/index.mdx b/docs/en/integrations/index.mdx index 888bfa7a772..b7108e76824 100644 --- a/docs/en/integrations/index.mdx +++ b/docs/en/integrations/index.mdx @@ -242,7 +242,7 @@ We are actively compiling this list of ClickHouse integrations below, so it's no |------|----|----------------|------------------|-------------| |Apache Airflow|Airflow logo|Data ingestion|Open-source workflow management platform for data engineering pipelines|[Github](https://github.com/bryzgaloff/airflow-clickhouse-plugin)| |Apache Beam|Beam logo|Data ingestion|Open source, unified model and set of language-specific SDKs for defining and executing data processing workflows. Compatible with Google Dataflow.|[Documentation](https://clickhouse.com/docs/en/integrations/apache-beam),
[Examples](https://github.com/ClickHouse/clickhouse-beam-connector/)| -|Google Dataflow|Dataflow logo|Data ingestion|Google Dataflow is a serverless service for running batch and streaming data pipelines using Apache Beam.|[Documentation](https://clickhouse.com/docs/en/integrations/google-dataflow/dataflow)| +|Google Dataflow|Dataflow logo|Data ingestion|Google Dataflow is a serverless service for running batch and streaming data pipelines using Apache Beam.|[Documentation](/docs/en/integrations/google-dataflow/dataflow)| |Apache InLong|InLong logo|Data ingestion|One-stop integration framework for massive data|[Documentation](https://inlong.apache.org/docs/data_node/load_node/clickhouse)| |Apache NiFi|NiFi logo|Data ingestion|Automates the flow of data between software systems|[Documentation](/docs/en/integrations/nifi)| |Apache SeaTunnel|SeaTunnel logo|Data ingestion|SeaTunnel is a very easy-to-use ultra-high-performance distributed data integration platform|[Website](https://seatunnel.apache.org/docs/2.3.0/connector-v2/sink/Clickhouse)| From 5afe3a8e4c475378c9659bae3e2fac2b9ee6b2b2 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Tue, 28 Jan 2025 13:36:33 +0100 Subject: [PATCH 12/19] move links from headers --- .../data-ingestion/google-dataflow/dataflow.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md b/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md index d83aefe5858..8beb39adbf1 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/dataflow.md @@ -11,19 +11,19 @@ description: Users can ingest data into ClickHouse using Google Dataflow There are two main ways to use Google Dataflow with ClickHouse, both are leveraging [`ClickHouseIO Apache Beam connector`](../../apache-beam): -## 1. **[Java Runner](./java-runner)** -The Java Runner allows users to implement custom Dataflow pipelines using the Apache Beam SDK `ClickHouseIO` integration. This approach provides full flexibility and control over the pipeline logic, enabling users to tailor the ETL process to specific requirements. +## 1. Java Runner +The [Java Runner](./java-runner) allows users to implement custom Dataflow pipelines using the Apache Beam SDK `ClickHouseIO` integration. This approach provides full flexibility and control over the pipeline logic, enabling users to tailor the ETL process to specific requirements. However, this option requires knowledge of Java programming and familiarity with the Apache Beam framework. -### Key Features: +### Key Features - High degree of customization. - Ideal for complex or advanced use cases. - Requires coding and understanding of the Beam API. -## 2. **[Predefined Templates](./templates)** -ClickHouse offers predefined templates designed for specific use cases, such as importing data from BigQuery into ClickHouse. These templates are ready-to-use and simplify the integration process, making them an excellent choice for users who prefer a no-code solution. +## 2. Predefined Templates +ClickHouse offers [predefined templates](./templates) designed for specific use cases, such as importing data from BigQuery into ClickHouse. These templates are ready-to-use and simplify the integration process, making them an excellent choice for users who prefer a no-code solution. -### Key Features: +### Key Features - No Beam coding required. - Quick and easy setup for simple use cases. - Suitable also for users with minimal programming expertise. From a522b2cf7455124c57582a5925df3886cec123b5 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Tue, 28 Jan 2025 14:42:37 +0100 Subject: [PATCH 13/19] Add github tracking issues for gcs and pubsub --- .../integrations/data-ingestion/google-dataflow/templates.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates.md b/docs/en/integrations/data-ingestion/google-dataflow/templates.md index 72478b2ef18..7c358f9851c 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/templates.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates.md @@ -26,5 +26,5 @@ For detailed step-by-step instructions, refer to the [Google Dataflow Run Pipeli ## List of ClickHouse Templates * [BigQuery To ClickHouse](./templates/bigquery-to-clickhouse) -* GCS To ClickHouse (coming soon!) -* Pub Sub To ClickHouse (coming soon!) \ No newline at end of file +* [GCS To ClickHouse](https://github.com/ClickHouse/DataflowTemplates/issues/3) (coming soon!) +* [Pub Sub To ClickHouse](https://github.com/ClickHouse/DataflowTemplates/issues/4) (coming soon!) \ No newline at end of file From ab21da2978e44411801a8f50eac6da8632fd491a Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Tue, 28 Jan 2025 15:35:28 +0100 Subject: [PATCH 14/19] add beam parameters and link dataflow parameters to it --- .../data-ingestion/etl-tools/apache-beam.md | 63 +++++++++++-------- .../templates/bigquery-to-clickhouse.md | 5 ++ 2 files changed, 43 insertions(+), 25 deletions(-) diff --git a/docs/en/integrations/data-ingestion/etl-tools/apache-beam.md b/docs/en/integrations/data-ingestion/etl-tools/apache-beam.md index c7fcaaa93aa..13555309b1b 100644 --- a/docs/en/integrations/data-ingestion/etl-tools/apache-beam.md +++ b/docs/en/integrations/data-ingestion/etl-tools/apache-beam.md @@ -97,31 +97,44 @@ public class Main { ## Supported Data Types -| ClickHouse | Apache Beam | Is Supported | Notes | -|--------------------------------------|------------------------------|--------------|----------------------------------------------------------------------------------------------------------------------------------------| -| `TableSchema.TypeName.FLOAT32` | `Schema.TypeName#FLOAT` | ✅ | | -| `TableSchema.TypeName.FLOAT64` | `Schema.TypeName#DOUBLE` | ✅ | | -| `TableSchema.TypeName.INT8` | `Schema.TypeName#BYTE` | ✅ | | -| `TableSchema.TypeName.INT16` | `Schema.TypeName#INT16` | ✅ | | -| `TableSchema.TypeName.INT32` | `Schema.TypeName#INT32` | ✅ | | -| `TableSchema.TypeName.INT64` | `Schema.TypeName#INT64` | ✅ | | -| `TableSchema.TypeName.STRING` | `Schema.TypeName#STRING` | ✅ | | -| `TableSchema.TypeName.UINT8` | `Schema.TypeName#INT16` | ✅ | | -| `TableSchema.TypeName.UINT16` | `Schema.TypeName#INT32` | ✅ | | -| `TableSchema.TypeName.UINT32` | `Schema.TypeName#INT64` | ✅ | | -| `TableSchema.TypeName.UINT64` | `Schema.TypeName#INT64` | ✅ | | -| `TableSchema.TypeName.DATE` | `Schema.TypeName#DATETIME` | ✅ | | -| `TableSchema.TypeName.DATETIME` | `Schema.TypeName#DATETIME` | ✅ | | -| `TableSchema.TypeName.ARRAY` | `Schema.TypeName#ARRAY` | ✅ | | -| `TableSchema.TypeName.ENUM8` | `Schema.TypeName#STRING` | ✅ | | -| `TableSchema.TypeName.ENUM16` | `Schema.TypeName#STRING` | ✅ | | -| `TableSchema.TypeName.BOOL` | `Schema.TypeName#BOOLEAN` | ✅ | | -| `TableSchema.TypeName.TUPLE` | `Schema.TypeName#ROW` | ✅ | | -| `TableSchema.TypeName.FIXEDSTRING` | `FixedBytes` | ✅ | `FixedBytes` is a `LogicalType` representing a fixed-length
byte array located at
`org.apache.beam.sdk.schemas.logicaltypes` | -| | `Schema.TypeName#DECIMAL` | ❌ | | -| | `Schema.TypeName#MAP` | ❌ | | - - +| ClickHouse | Apache Beam | Is Supported | Notes | +|------------------------------------|----------------------------|--------------|------------------------------------------------------------------------------------------------------------------------------------------| +| `TableSchema.TypeName.FLOAT32` | `Schema.TypeName#FLOAT` | ✅ | | +| `TableSchema.TypeName.FLOAT64` | `Schema.TypeName#DOUBLE` | ✅ | | +| `TableSchema.TypeName.INT8` | `Schema.TypeName#BYTE` | ✅ | | +| `TableSchema.TypeName.INT16` | `Schema.TypeName#INT16` | ✅ | | +| `TableSchema.TypeName.INT32` | `Schema.TypeName#INT32` | ✅ | | +| `TableSchema.TypeName.INT64` | `Schema.TypeName#INT64` | ✅ | | +| `TableSchema.TypeName.STRING` | `Schema.TypeName#STRING` | ✅ | | +| `TableSchema.TypeName.UINT8` | `Schema.TypeName#INT16` | ✅ | | +| `TableSchema.TypeName.UINT16` | `Schema.TypeName#INT32` | ✅ | | +| `TableSchema.TypeName.UINT32` | `Schema.TypeName#INT64` | ✅ | | +| `TableSchema.TypeName.UINT64` | `Schema.TypeName#INT64` | ✅ | | +| `TableSchema.TypeName.DATE` | `Schema.TypeName#DATETIME` | ✅ | | +| `TableSchema.TypeName.DATETIME` | `Schema.TypeName#DATETIME` | ✅ | | +| `TableSchema.TypeName.ARRAY` | `Schema.TypeName#ARRAY` | ✅ | | +| `TableSchema.TypeName.ENUM8` | `Schema.TypeName#STRING` | ✅ | | +| `TableSchema.TypeName.ENUM16` | `Schema.TypeName#STRING` | ✅ | | +| `TableSchema.TypeName.BOOL` | `Schema.TypeName#BOOLEAN` | ✅ | | +| `TableSchema.TypeName.TUPLE` | `Schema.TypeName#ROW` | ✅ | | +| `TableSchema.TypeName.FIXEDSTRING` | `FixedBytes` | ✅ | `FixedBytes` is a `LogicalType` representing a fixed-length
byte array located at
`org.apache.beam.sdk.schemas.logicaltypes` | +| | `Schema.TypeName#DECIMAL` | ❌ | | +| | `Schema.TypeName#MAP` | ❌ | | + +## ClickHouseIO.Write Parameters + +You can adjust the `ClickHouseIO.Write` configuration with the following setter functions: + +| Parameter Setter Function | Argument Type | Default Value | Description | +|-----------------------------|-----------------------------|-------------------------------|-----------------------------------------------------------------| +| `withMaxInsertBlockSize` | `(long maxInsertBlockSize)` | `1000000` | Maximum size of a block of rows to insert. | +| `withMaxRetries` | `(int maxRetries)` | `5` | Maximum number of retries for failed inserts. | +| `withMaxCumulativeBackoff` | `(Duration maxBackoff)` | `Duration.standardDays(1000)` | Maximum cumulative backoff duration for retries. | +| `withInitialBackoff` | `(Duration initialBackoff)` | `Duration.standardSeconds(5)` | Initial backoff duration before the first retry. | +| `withInsertDistributedSync` | `(Boolean sync)` | `true` | If true, synchronizes insert operations for distributed tables. | +| `withInsertQuorum` | `(Long quorum)` | `null` | The number of replicas required to confirm an insert operation. | +| `withInsertDeduplicate` | `(Boolean deduplicate)` | `true` | If true, deduplication is enabled for insert operations. | +| `withTableSchema` | `(TableSchema schema)` | `null` | Schema of the target ClickHouse table. | ## Limitations diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md index b795c3683d4..f24432811f9 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md @@ -44,6 +44,11 @@ The template can either read the entire table or read specific records using a p | `queryTempDataset` | Set an existing dataset to create the temporary table to store the results of the query. For example, `temp_dataset`. | | | | `KMSEncryptionKey` | If reading from BigQuery using the query source, use this Cloud KMS key to encrypt any temporary tables created. For example, `projects/your-project/locations/global/keyRings/your-keyring/cryptoKeys/your-key`. | | | + +:::note +All `ClickHouseIO` parameters default values could be found in [`ClickHouseIO` Apache Beam Connector](/docs/en/integrations/apache-beam#clickhouseiowrite-parameters) +::: + ## Source and Target Tables Schema In order to effectively load the BigQuery dataset to ClickHouse, and a column infestation process is conducted with the From adaa9f8a98118ad0e2534bd0579f132a795a2816 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 5 Feb 2025 13:48:39 +0200 Subject: [PATCH 15/19] change links to relative --- .../templates/bigquery-to-clickhouse.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md index f24432811f9..a133bd83e6e 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md @@ -69,15 +69,15 @@ target table. The BigQuery types are converted based on your ClickHouse table definition. Therefore, the above table lists the recommended mapping you should have in your target ClickHouse table (for a given BigQuery table/query): -| BigQuery Type | ClickHouse Type | Notes | -|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [**Array Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array_type) | [**Array Type**](https://clickhouse.com/docs/en/sql-reference/data-types/array) | The inner type must be one of the supported primitive data types listed in this table. | -| [**Boolean Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#boolean_type) | [**Bool Type**](https://clickhouse.com/docs/en/sql-reference/data-types/boolean) | | -| [**Date Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#date_type) | [**Date Type**](https://clickhouse.com/docs/en/sql-reference/data-types/date) | | -| [**Datetime Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#datetime_type) | [**Datetime Type**](https://clickhouse.com/docs/en/sql-reference/data-types/datetime) | Works as well with `Enum8`, `Enum16` and `FixedString`. | -| [**String Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#string_type) | [**String Type**](https://clickhouse.com/docs/en/sql-reference/data-types/string) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). | -| [**Numeric - Integer Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Integer Types**](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). The template will also convert unassigned Int types if used in ClickHouse table (`UInt8`, `UInt16`, `UInt32`, `UInt64`). | -| [**Numeric - Float Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Float Types**](https://clickhouse.com/docs/en/sql-reference/data-types/float) | Supported ClickHouse types: `Float32` and `Float64` | +| BigQuery Type | ClickHouse Type | Notes | +|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [**Array Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array_type) | [**Array Type**](../../../sql-reference/data-types/array){:target="_blank"} | The inner type must be one of the supported primitive data types listed in this table. | +| [**Boolean Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#boolean_type) | [**Bool Type**](../../../sql-reference/data-types/boolean) | | +| [**Date Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#date_type) | [**Date Type**](../../../sql-reference/data-types/date) | | +| [**Datetime Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#datetime_type) | [**Datetime Type**](../../../sql-reference/data-types/datetime) | Works as well with `Enum8`, `Enum16` and `FixedString`. | +| [**String Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#string_type) | [**String Type**](../../../sql-reference/data-types/string) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). | +| [**Numeric - Integer Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Integer Types**](../../../sql-reference/data-types/int-uint) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). The template will also convert unassigned Int types if used in ClickHouse table (`UInt8`, `UInt16`, `UInt32`, `UInt64`). | +| [**Numeric - Float Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Float Types**](../../../sql-reference/data-types/float) | Supported ClickHouse types: `Float32` and `Float64` | ## Running the Template From c7fd4885ddc6686c378c5ad43680cd91bffc285f Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 5 Feb 2025 14:03:42 +0200 Subject: [PATCH 16/19] apply spelling for specific file --- scripts/aspell-dict-file.txt | 5 +++++ .../data-ingestion/google-dataflow/aspell-dict.txt | 4 ---- 2 files changed, 5 insertions(+), 4 deletions(-) delete mode 100644 scripts/aspell-ignore/en/integrations/data-ingestion/google-dataflow/aspell-dict.txt diff --git a/scripts/aspell-dict-file.txt b/scripts/aspell-dict-file.txt index 4a7515800ab..f7999c3e701 100644 --- a/scripts/aspell-dict-file.txt +++ b/scripts/aspell-dict-file.txt @@ -320,3 +320,8 @@ westus intra --docs/en/cloud/manage/backups.md-- slideout +--docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md-- +DataFlow +Dataflow +DataflowTemplates +GoogleSQL diff --git a/scripts/aspell-ignore/en/integrations/data-ingestion/google-dataflow/aspell-dict.txt b/scripts/aspell-ignore/en/integrations/data-ingestion/google-dataflow/aspell-dict.txt deleted file mode 100644 index df817c4e44d..00000000000 --- a/scripts/aspell-ignore/en/integrations/data-ingestion/google-dataflow/aspell-dict.txt +++ /dev/null @@ -1,4 +0,0 @@ -DataFlow -Dataflow -DataflowTemplates -GoogleSQL From 1cebe756cc6c336d1caa9996afa633dea1a66078 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 5 Feb 2025 14:05:19 +0200 Subject: [PATCH 17/19] add more words --- scripts/aspell-dict-file.txt | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/scripts/aspell-dict-file.txt b/scripts/aspell-dict-file.txt index f7999c3e701..124ab564be7 100644 --- a/scripts/aspell-dict-file.txt +++ b/scripts/aspell-dict-file.txt @@ -325,3 +325,9 @@ DataFlow Dataflow DataflowTemplates GoogleSQL +--docs/en/integrations/data-ingestion/google-dataflow/templates.md-- +Dataflow +--docs/en/integrations/data-ingestion/google-dataflow/dataflow.md-- +Dataflow +--docs/en/integrations/data-ingestion/google-dataflow/java-runner.md-- +Dataflow \ No newline at end of file From fc9a4486e871aa8b2ffb3a1bd348b58ef678f96f Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 5 Feb 2025 14:47:16 +0200 Subject: [PATCH 18/19] add more words --- scripts/aspell-dict-file.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/aspell-dict-file.txt b/scripts/aspell-dict-file.txt index 124ab564be7..ddc313a9698 100644 --- a/scripts/aspell-dict-file.txt +++ b/scripts/aspell-dict-file.txt @@ -325,6 +325,7 @@ DataFlow Dataflow DataflowTemplates GoogleSQL +linearize --docs/en/integrations/data-ingestion/google-dataflow/templates.md-- Dataflow --docs/en/integrations/data-ingestion/google-dataflow/dataflow.md-- From f41d9df4b9b1095ce70a1f7cca36016606590242 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 5 Feb 2025 18:49:06 +0200 Subject: [PATCH 19/19] remove problematic chars --- .../templates/bigquery-to-clickhouse.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md index a133bd83e6e..ee8ff6e9aa7 100644 --- a/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md +++ b/docs/en/integrations/data-ingestion/google-dataflow/templates/bigquery-to-clickhouse.md @@ -27,7 +27,7 @@ The template can either read the entire table or read specific records using a p | Parameter Name | Parameter Description | Required | Notes | |-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `jdbcUrl` | The ClickHouse JDBC URL in the format `jdbc:clickhouse://{host}:{port}/{schema}`. | ✅ | Don't add the username and password as JDBC options. Any other JDBC option could be added at the end of the JDBC URL. For ClickHouse Cloud users, add `ssl=true&sslmode=NONE` to the `jdbcUrl`. | +| `jdbcUrl` | The ClickHouse JDBC URL in the format `jdbc:clickhouse://:/`. | ✅ | Don't add the username and password as JDBC options. Any other JDBC option could be added at the end of the JDBC URL. For ClickHouse Cloud users, add `ssl=true&sslmode=NONE` to the `jdbcUrl`. | | `clickHouseUsername` | The ClickHouse username to authenticate with. | ✅ | | | `clickHousePassword` | The ClickHouse password to authenticate with. | ✅ | | | `clickHouseTable` | The target ClickHouse table name to insert the data to. | ✅ | | @@ -69,15 +69,15 @@ target table. The BigQuery types are converted based on your ClickHouse table definition. Therefore, the above table lists the recommended mapping you should have in your target ClickHouse table (for a given BigQuery table/query): -| BigQuery Type | ClickHouse Type | Notes | -|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [**Array Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array_type) | [**Array Type**](../../../sql-reference/data-types/array){:target="_blank"} | The inner type must be one of the supported primitive data types listed in this table. | -| [**Boolean Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#boolean_type) | [**Bool Type**](../../../sql-reference/data-types/boolean) | | -| [**Date Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#date_type) | [**Date Type**](../../../sql-reference/data-types/date) | | -| [**Datetime Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#datetime_type) | [**Datetime Type**](../../../sql-reference/data-types/datetime) | Works as well with `Enum8`, `Enum16` and `FixedString`. | -| [**String Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#string_type) | [**String Type**](../../../sql-reference/data-types/string) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). | -| [**Numeric - Integer Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Integer Types**](../../../sql-reference/data-types/int-uint) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). The template will also convert unassigned Int types if used in ClickHouse table (`UInt8`, `UInt16`, `UInt32`, `UInt64`). | -| [**Numeric - Float Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Float Types**](../../../sql-reference/data-types/float) | Supported ClickHouse types: `Float32` and `Float64` | +| BigQuery Type | ClickHouse Type | Notes | +|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [**Array Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array_type) | [**Array Type**](../../../sql-reference/data-types/array) | The inner type must be one of the supported primitive data types listed in this table. | +| [**Boolean Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#boolean_type) | [**Bool Type**](../../../sql-reference/data-types/boolean) | | +| [**Date Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#date_type) | [**Date Type**](../../../sql-reference/data-types/date) | | +| [**Datetime Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#datetime_type) | [**Datetime Type**](../../../sql-reference/data-types/datetime) | Works as well with `Enum8`, `Enum16` and `FixedString`. | +| [**String Type**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#string_type) | [**String Type**](../../../sql-reference/data-types/string) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). | +| [**Numeric - Integer Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Integer Types**](../../../sql-reference/data-types/int-uint) | In BigQuery all Int types (`INT`, `SMALLINT`, `INTEGER`, `BIGINT`, `TINYINT`, `BYTEINT`) are aliases to `INT64`. We recommend you setting in ClickHouse the right Integer size, as the template will convert the column based on the defined column type (`Int8`, `Int16`, `Int32`, `Int64`). The template will also convert unassigned Int types if used in ClickHouse table (`UInt8`, `UInt16`, `UInt32`, `UInt64`). | +| [**Numeric - Float Types**](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric_types) | [**Float Types**](../../../sql-reference/data-types/float) | Supported ClickHouse types: `Float32` and `Float64` | ## Running the Template @@ -106,7 +106,7 @@ Below is an example of the command: ```bash gcloud dataflow flex-template run "bigquery-clickhouse-dataflow-$(date +%Y%m%d-%H%M%S)" \ --template-file-gcs-location "gs://clickhouse-dataflow-templates/bigquery-clickhouse-metadata.json" \ - --parameters inputTableSpec="{bigquery table id}",jdbcUrl="jdbc:clickhouse://{clickhouse host}:{clickhouse port}/{schema}?ssl=true&sslmode=NONE",clickHouseUsername="{username}",clickHousePassword="{password}",clickHouseTable="{clickhouse target table}" + --parameters inputTableSpec="",jdbcUrl="jdbc:clickhouse://:/?ssl=true&sslmode=NONE",clickHouseUsername="",clickHousePassword="",clickHouseTable="" ``` ### Command Breakdown