[FEATURE] Support Spark 4.0.0

[josecsotomorales]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

The goal is to enable the Kubernetes Spark Operator to support Apache Spark 4.0.0, which is currently in preview mode. This update will allow users to leverage the latest features and improvements in Spark, enhancing performance and compatibility with evolving data processing requirements.

Describe the solution you would like

I propose updating the Kubernetes Spark Operator to support Apache Spark 4.0.0. Given that there have been no significant changes on the Kubernetes side in Spark 4.0.0, this update should be straightforward. It will involve ensuring compatibility with Spark 4.0.0 APIs and conducting comprehensive testing to ensure that the Operator continues to manage Spark clusters effectively within Kubernetes environments.

Describe alternatives you have considered

Since there are no significant Kubernetes-specific changes in Spark 4.0.0, maintaining a separate branch for this preview release is unnecessary. Directly updating the main branch would simplify development and deployment processes for users eager to adopt the latest version.

Additional context

Apache Spark 4.0.0 introduces several enhancements that could benefit Kubernetes deployments, including performance optimizations and expanded API capabilities. By updating the Kubernetes Spark Operator to support Spark 4.0.0, we can provide our users with immediate access to these improvements, ensuring that our tool remains at the cutting edge of cluster management technology.