SparkKubernetesOperator : FileNotFoundError: [Errno 2] No such file or directory

[elzteam]

I am getting this error when using SparkKubernetesOperator operator in 2.8.4
File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 127, in manage_template_specs
template_body = _load_body_to_dict(open(self.application_file))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: "apiVersion: sparkoperator.k8s.io/v1beta2\nkind: SparkApplication\nmetadata:\n name: demo-pyspark-app\n namespace: test-bdp-dev-001\n annotations:\n sidecar.istio.io/inject: 'false'\nspec:\n type: Scala\n mode: cluster\n image: 'test


.azurecr.io/test/bdp/bdp-spark:3.3.0-108'\n imagePullPolicy: IfNotPresent\n mainClass: org.apache.spark.examples.SparkPi\n mainApplicationFile: 'local:///opt/spark/examples/jars/spark-examples_2.12-3.3.0.jar'\n arguments:\n - '200000'\n sparkVersion: 3.3.0\n sparkConf:\n spark.eventLog.dir: /tmp/\n spark.eventLog.enabled: 'true'\n spark.eventLog.logStageExecutorMetrics: 'true'\n spark.jar.ivy: /tmp\n spark.kubernetes.driver.pod.name: demo-pyspark-app\n spark.kubernetes.executor.podNamePrefix: demo-pyspark-app\n spark.dynamicAllocation.enabled: 'true'\n spark.dynamicAllocation.shuffleTracking.enabled: 'true'\n spark.dynamicAllocation.minExecutors: '1'\n spark.dynamicAllocation.maxExecutors: '15'\n spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path: /var/log/app_logs\n spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName: OnDemand\n spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.sizeLimit: 2Gi\n spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.storageClass: cps-managed-disk-delete\n driver:\n annotations:\n sidecar.istio.io/inject: 'false'\n coreLimit: 1000m\n coreRequest: 55m\n memory: 512m\n serviceAccount: dev-bdp\n executor:\n annotations:\n sidecar.istio.io/inject: 'false'\n coreLimit: 1000m\n coreRequest: 55m\n memory: 512m\n labels:\n version: 3.3.0"

<>.. Why is it doing.. Can you please help..
Here is the yaml and dag code..
Note: When i was using 2.7.1, i used to redirect the yaml file contents to a string variable and use to pass that string variable the sparkoperator.. Its no longer working in 2.8.4 .. so i am passing the file path of the yaml.. (in the code thats the reason i commented.. # yaml_file = yaml.dump(yaml.load(open(args_dict["application_yaml_file"]), Loader=yaml.FullLoader))
# print(yaml_file)... i wish i can still pass that as a string (not to divert the discussion)

[schobe]

@elzteam Is there any solution for this problem? I am facing the similar error, might be missing some task config for operator? I am using astro-runtime:11.3.0 image which is based on Airflow 2.9.1.

[gassuncao-ccm]

Same error here.
If someone found the solution, please share with us