add e2e reconcile test for databricks source #2145
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,6 +7,7 @@ | |
| import re | ||
| import sys | ||
| import time | ||
| import webbrowser | ||
| from collections.abc import Mapping | ||
| from pathlib import Path | ||
| from typing import NoReturn, TextIO | ||
|
|
@@ -652,9 +653,11 @@ def reconcile(*, w: WorkspaceClient) -> None: | |
| ctx.workspace_client, | ||
| ctx.installation, | ||
| ctx.install_state, | ||
| ) | ||
|
|
||
| _, job_run_url = recon_runner.run(operation_name=RECONCILE_OPERATION_NAME) | ||
| if ctx.prompts.confirm(f"Would you like to open the job run URL `{job_run_url}` in the browser?"): | ||
|
Collaborator
There was a problem hiding this comment. I see where you are going which this, but let us not do this given our overall goal of going away from user prompting to UI.
Collaborator
There was a problem hiding this comment. Check the implementation in
Contributor
Author
There was a problem hiding this comment. then my refactoring makes sense, the prompting is where it should be in the cli and not in the runner that should not know anything about it.
Contributor
There was a problem hiding this comment. This looks like a move: hoisting it from inside the Overall I'd prefer to just drop the prompt and leave the existing logging of the URL: every terminal allows the job to be opened by just clicking on it. That's not the point or focus of this PR, however, which is why I'm neutral on leaving the UX as-is for now. I assume this was moved to make some of the tests easier in some way?
Contributor
Author
There was a problem hiding this comment.
Removing it made the tests easier that I dont have to mock it since the real implementation suspends the execution waiting for a user input. I moved it to the cli because it only belongs there |
||
| webbrowser.open(job_run_url) | ||
|
|
||
|
|
||
| @lakebridge.command | ||
|
|
@@ -668,10 +671,11 @@ def aggregates_reconcile(*, w: WorkspaceClient) -> None: | |
| ctx.workspace_client, | ||
| ctx.installation, | ||
| ctx.install_state, | ||
| ) | ||
|
|
||
| _, job_run_url = recon_runner.run(operation_name=AGG_RECONCILE_OPERATION_NAME) | ||
| if ctx.prompts.confirm(f"Would you like to open the job run URL `{job_run_url}` in the browser?"): | ||
|
|
||
| webbrowser.open(job_run_url) | ||
|
|
||
|
|
||
| @lakebridge.command | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| from databricks.labs.lakebridge.config import ( | ||
| ReconcileConfig, | ||
| DatabaseConfig, | ||
| ReconcileMetadataConfig, | ||
| LakebridgeConfiguration, | ||
| ) | ||
| from databricks.labs.lakebridge.contexts.application import ApplicationContext | ||
| from databricks.labs.lakebridge.reconcile.recon_config import RECONCILE_OPERATION_NAME | ||
| from databricks.labs.lakebridge.reconcile.runner import ReconcileRunner | ||
|
|
||
| TABLE_RECON_JSON = """ | ||
| { | ||
| "source_schema": "test_source", | ||
| "target_catalog": "sandbox", | ||
| "target_schema": "test_target", | ||
| "tables": [ | ||
| { | ||
| "source_name": "diamonds", | ||
| "target_name": "diamonds", | ||
| "join_columns": ["color", "clarity"] | ||
| } | ||
| ] | ||
| } | ||
| """ | ||
|
|
||
| recon_config = ReconcileConfig( | ||
| data_source="databricks", | ||
| report_type="all", | ||
| secret_scope="NOT_NEEDED", | ||
| database_config=DatabaseConfig( | ||
| source_catalog="sandbox", source_schema="test_source", target_catalog="sandbox", target_schema="test_target" | ||
| ), | ||
| metadata_config=ReconcileMetadataConfig(catalog="sandbox", schema="reconcile"), | ||
| ) | ||
| config = LakebridgeConfiguration(None, recon_config) | ||
| source_catalog_or_schema = ( | ||
| recon_config.database_config.source_catalog | ||
| if recon_config.database_config.source_catalog | ||
| else recon_config.database_config.source_schema | ||
| ) | ||
| filename = f"recon_config_{recon_config.data_source}_{source_catalog_or_schema}_{recon_config.report_type}.json" | ||
|
|
||
|
|
||
| def test_recon(ws): | ||
| ctx = ApplicationContext(ws) | ||
| recon_runner = ReconcileRunner( | ||
| ctx.workspace_client, | ||
| ctx.installation, | ||
| ctx.install_state, | ||
| ) | ||
|
|
||
| ctx.installation.save(recon_config) | ||
| ctx.installation.upload(filename, TABLE_RECON_JSON.encode()) | ||
| ctx.workspace_installation.install(config) | ||
|
|
||
| run, _ = recon_runner.run(operation_name=RECONCILE_OPERATION_NAME) | ||
| result = run.result() | ||
| assert result |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's not do this. I'll provide more context in the review summary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I need this convenience in my day to day.
labs testerrors out on my machine and I didnt really get how it works