[QUESTION] Spark versions compatibility for operator version v1beta2-1.6.x-3.5.0

[kraj007]

• ✋ I have searched the open/closed issues and my issue is not listed.

Please describe your question here

As per documentation base spark version for operator version v1beta2-1.6.x-3.5.0 is 3.5.0.
What is mean by base spark version here ?
We are using spark 3.4.1 (cant upgrade to 3.5.0 as of now), we just tested once example application with operator version v1beta2-1.6.x-3.5.0 and spark version 3..4.1. So far no issues. Does it mean , operator operator version v1beta2-1.6.x-3.5.0 supports spark 3.4.1 also ?
If this is not recommended , which operator version we should use ?

Provide a link to the example/module related to the question

Additional context

[ChenYi015]

@kraj007 In Spark operator, it will use spark-submit script to submit spark applications to kubernetes, and this spark-submit script comes from Spark 3.5.0 (for operator version v1beta2-1.6.x-3.5.0). In most cases, it will be fine, since the spark-submit changes slightly between v3.4.x and v3.5.x.

[yuzhouliu9]

@ChenYi015
To confirm, the SparkApplication spec sparkVersion: 3.4.1 still defines the spark version to use for a run.
It's just the spark-submit script used to submit SparkApplication is the updated one?

[jacobsalway]

@yuzhouliu9 correct, the Spark version you've used in your application image is what it'll run with. The sparkVersion field on the CR actually isn't used for any submission logic right now.