Deployment Patterns

[ForeverAngry]

Hi all, this is my first time here. So I tried the docker demo and got a general feel for the project. Does anyone have any suggestions or examples on how this should be deployed in prod (i use aws and kubernetes typically), as in should it be implemented with some rest client so that the jar file can be invoked when needed - or is there a better path to do this?

Thanks for letting me post the question! I'd love to hear some thoughts!

[the-other-tim-brown]

I guess that depends on how you want to use the tool. Currently, we're running this as a "meta sync" tool after each commit in Hudi with this tool. If you're already running some code in a jvm, then your best bet may just be to import the jar and call the sync methods similar to what the linked sync tool does for Hudi users.

If you want to run this periodically without interfering with existing code/deployments, then I think running as a kubernetes job on a schedule would be easiest to get started and lowest overhead for you. If you want to go this route, we can work together to build a docker file and yaml for the deployment. I think others would also find this useful for syncing their tables periodically.