'default_runtime_data_connector_name not found in batch_identifiers for the RuntimeDataConnector

[BStefansen-lightbox]

Describe the bug
default_runtime_data_connector_name is not being added as a key to self._batch_identifiers

  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\great_expectations\datasource\data_connector\runtime_data_connector.py", line 240, in _get_batch_definition_list_from_batch_request
    self._validate_batch_identifiers(
  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\great_expectations\datasource\data_connector\runtime_data_connector.py", line 406, in _validate_batch_identifiers
    self._validate_data_connector_level_batch_identifiers(
  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\great_expectations\datasource\data_connector\runtime_data_connector.py", line 436, in _validate_data_connector_level_batch_identifiers
    if not set(batch_identifiers_keys) <= set(self._batch_identifiers[self.name]):
KeyError: 'default_runtime_data_connector_name'

To Reproduce

flow.py

from prefect import flow
from prefect_great_expectations import run_checkpoint_validation


@flow
def flow(data_asset_name):
    validations = {
        "validations": [
            {
                "batch_request": {
                    "datasource_name": "local_data",
                    "data_connector_name": "default_runtime_data_connector_name",
                    "data_asset_name": data_asset_name,
                    "runtime_parameters": {
                        "path": f"../data/{data_asset_name}"
                    },
                    "batch_identifiers": {
                        "runtime_batch_identifier_name": "foobar"
                    }
                },
                "expectation_suite_name": "tx_isp_shp"
            }
        ]
    }
    
    run_checkpoint_validation(
        checkpoint_name="isp_file_submission_checkpoint",
        checkpoint_kwargs=validations
    )

if __name__ == '__main__':
    flow(
        "tx_fb_mock.zip"
    )

great_expectations.yaml

datasources:
  local_data:
    data_connectors:
      default_inferred_data_connector_name:
        class_name: InferredAssetFilesystemDataConnector
        default_regex:
          group_names:
            - data_asset_name
          pattern: (.*)
        module_name: great_expectations.datasource.data_connector
        name: default_inferred_data_connector_name
        base_directory: ../data
      default_runtime_data_connector_name:
        assets:
          my_runtime_asset_name:
            class_name: Asset
            module_name: great_expectations.datasource.data_connector.asset
            batch_identifiers:
              - runtime_batch_identifier_name
        class_name: RuntimeDataConnector
        module_name: great_expectations.datasource.data_connector
        name: default_runtime_data_connector_name
    class_name: Datasource
    module_name: great_expectations.datasource
    execution_engine:
      class_name: PandasExecutionEngine
      module_name: great_expectations.execution_engine
config_variables_file_path: uncommitted/config_variables.yml

isp_file_submission_checkpoint.yaml

name: isp_file_submission_checkpoint
config_version: 1.0
template_name:
module_name: great_expectations.checkpoint
class_name: Checkpoint
run_name_template: '%Y%m%d-%H%M%S-isp_file_submission_checkpoint'
expectation_suite_name:
batch_request: {}
action_list:
  - name: store_validation_result
    action:
      class_name: StoreValidationResultAction
  - name: store_evaluation_params
    action:
      class_name: StoreEvaluationParametersAction
  - name: update_data_docs
    action:
      class_name: UpdateDataDocsAction
      site_names: []
evaluation_parameters: {}
runtime_configuration: {}
validations: []
profilers: []
ge_cloud_id:
expectation_suite_ge_cloud_id:

Expected behavior
The self._batch_identifiers variable should already be populated with a value for self.name aka default_runtime_data_connector_name, and therefore should not be raising this exception

Environment (please complete the following information):

• Windows 10
• Great Expectations Version: 0.15.26
• prefect-great-expectations==0.1.0
• prefect==2.4.0

Solution?
I had to change the source code and add self._batch_identifiers[self.name] = batch_identifiers to line 127 of C:\Users\BStefansen.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\great_expectations\datasource\data_connector\runtime_data_connector.py to get everything to work.

[austiezr]

Hey @BStefansen-lightbox ! Thanks for raising this; we'll review this behavior and be in touch.

[BStefansen-lightbox]

@austiezr I figured out the problem. It has to do with my yaml configurations. I don't have the time right now, but I will post my findings on Monday.

[BStefansen-lightbox]

@austiezr

Here was the solution. The code was looking for batch identifiers for the assets defined in the data connector rather than during runtime. The asset config defined in the data connector was overriding anything defined in runtime, which was throwing the exception above.

TLDR; remove the asset config defined in the data connector yaml

great_expectations.yaml

      default_runtime_data_connector_name:
        class_name: RuntimeDataConnector
        batch_identifiers:
          - runtime_batch_identifier_name

flow.py

        validations = {
            "validations": [
                {
                    "batch_request": {
                        "datasource_name": "local_data",
                        "data_connector_name": "default_runtime_data_connector_name",
                        "data_asset_name": data_asset_name,
                        "runtime_parameters": {
                            "path": f"../data/{data_asset_name}"
                        },
                        "batch_identifiers": {
                            "runtime_batch_identifier_name": data_asset_name
                        }
                    },
                    "expectation_suite_name": flow_config["test_suite"]
                }
            ]
        }

[AFineDayFor]

thank you for following up with us. i moved this to Discussions to use as a later point of reference if anyone else comes in with a similar issue.

We appreciate you oodles and thank you very much 🙇