spark.submit.deployMode not work

[imperio-wxm]

When spark.submit.deployMode=cluster is configured, the log shows that kyuubi's spark-submit submission command has taken effect, but the parameter on spark-web is still client

Log

2023-04-10 14:30:04.791  INFO 6 --- [engine-launch-log] o.a.kyuubi.jdbc.hive.KyuubiConnection    : 	--conf spark.submit.deployMode=cluster \

Spark-web

[pan3793]

When spark.submit.deployMode=cluster is configured ...

How?

[imperio-wxm]

resource manager is k8s and found that the client is specified in the official entrypoint.sh of spark. I don’t know if it has anything to do with this.

[pan3793]

it's by design, the Spark on K8s cluster mode can be judged by the following two configurations.

spark.submit.deployMode=client
spark.kubernetes.submitInDriver=true

[pan3793]

For more implementation details, please read code org.apache.spark.deploy.SparkSubmit

[imperio-wxm]

spark.kubernetes.submitInDriver
Do you mean that only setting spark.submit.deployMode=cluster cannot take effect, need set spark.kubernetes.submitInDriver=true? But I can't find this parameter on spark official website: https://spark.apache.org/docs/latest/running-on-kubernetes.html

[pan3793]

No.

The user interface is unified, spark-submit --conf spark.submit.deployMode=cluster or spark-submit --deploy-mode cluster is sufficient. But the internal implementation is a little bit hacky in K8s cluster mode compared w/ others.

I mean, a running Spark application that runs on K8s, whose deploy mode can be judged by two configurations.

[yaooqinn]

It might be just a display issue, you can check whether there is a driver pod or not

[imperio-wxm]

The driver pod is available. Maybe it's a web display problem as you said