Run soda-core locally against a (local) delta table

[sherl0ck-]

Hi,

I'm trying to run checks against a locally defined delta table.

from soda.scan import Scan
df = spark.read.format("delta").load("employees_delta")
df.createOrReplaceTempView("employees_delta")
# I can confirm that the table indeed exists and can be accessed e.g. using spark.sql()

scan = Scan()
scan.set_data_source_name("my_spark")
scan.add_spark_session(spark, data_source_name="my_spark")
scan.add_configuration_yaml_file(file_path="/Users/jjovan/data_eng_spark/work/soda.yml")
scan.add_sodacl_yaml_file("/Users/jjovan/data_eng_spark/work/checks.yml")
# Note: I tried different orderings here, to no avail

scan.execute()

This fails with this exception

Query execution error in 40.my_spark.employees_delta.aggregation[0]: 'NoneType' object has no attribute 'sql'
SELECT 
  COUNT(*) 
FROM employees_delta
  | 'NoneType' object has no attribute 'sql'
  | Stacktrace:
  | Traceback (most recent call last):
  |   File "/Users/jjovan/anaconda3/lib/python3.10/site-packages/soda/execution/query/query.py", line 145, in _execute_cursor
  |     cursor.execute(self.sql)
  |   File "/Users/jjovan/anaconda3/lib/python3.10/site-packages/soda/data_sources/spark_df_cursor.py", line 16, in execute
  |     self.df = self.spark_session.sql(sqlQuery=sql)
  | AttributeError: 'NoneType' object has no attribute 'sql'

[00:16:26] Metrics 'row_count' were not computed for check 'row_count > 0'

My soda.yml configuration:

data_source my_spark:
  type: spark_df

checks.yml

checks for employees_delta:
  - row_count > 0

Versions as seen in pip list | grep soda

soda-core                              3.4.4
soda-core-spark                        3.4.4
soda-core-spark-df                     3.4.4

[tools-soda]

CLOUD-9157