Kyuubi and Spark in k8s

[hanna-liashchuk]

Hi there!

We are running kyuubi in k8s cluster as well as spark. Hence, our spark driver is running in the server pod.
I'm wondering, how do you handle the cases when spark.driver.memory can take more than expected and force the server to restart? E.g.
We have 1 pod with Kyuubi server with 10gb memory
and we have 10 users that created 10 engines with 2gb driver memory each. As a result, the memory requested by spark is larger than the pod has

Any ideas are appreciated, Thanks

[zwangsheng]

Have you considered booting the driver pod outside of the kyuubi server pod?

In spark, provided two deploy-mode: client & cluster, the essential difference is whether the driver starts at the submit node.

So from your description, it seems that you adopt the client mode. We recommend the kyuubi on kubernetes scenario to use the spark cluster mode.

FYI, https://spark.apache.org/docs/latest/running-on-kubernetes.html#cluster-mode

[hanna-liashchuk]

Interesting, it is known that cluster mode is not suitable for interactive sessions, that's why for example Jupyterhub is not working in cluster mode. Since Kyuubi sessions are pretty interactive, I assumed that client mode is required. I searched the documentation and it has only one sentence about deploy modes :( anyway, I will try it an let you know

Just curious, is cluster mode support was added later on? I found a PR that added cluster mode for YARN ... Nothing about k8s

Thanks

[hanna-liashchuk]

I've tested the cluster mode and it works great, thanks!

I created a small PR to save some time for other people like me :)
#5227

[pan3793]

Supporting cluster mode is one of the major advantages of Kyuubi over Spark Thrift Server / spark-sql :)