Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add storeManifest boolean property to allow persisting manifest.json between executions #45

Open
anna-geller opened this issue Aug 17, 2023 · 0 comments
Labels
area/plugin Plugin-related issue or feature request enhancement New feature or request

Comments

@anna-geller
Copy link
Member

anna-geller commented Aug 17, 2023

Feature description

dbt allows rebuilding only models which have changed since the last run:

dbt build --select state:modified+ --defer

To know which models have been modified, the user would need to store the manifest.json.

We can store that in the KV store automatically to allow that pattern.

Possible implementation

if the user sets the storeManifest to true, then Kestra would:

  • store the manifest file in the KV Store after each run
  • before starting a new run, load the manifest file so dbt can compare the existing one with the new "state" of the dbt models
@anna-geller anna-geller added the enhancement New feature or request label Aug 17, 2023
@anna-geller anna-geller added this to the v0.17.0 milestone Dec 5, 2023
@anna-geller anna-geller modified the milestones: v0.17.0, v0.22.0 Apr 10, 2024
@anna-geller anna-geller removed this from the v0.22.0 milestone Aug 20, 2024
@anna-geller anna-geller added the area/plugin Plugin-related issue or feature request label Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/plugin Plugin-related issue or feature request enhancement New feature or request
Projects
Status: Backlog
Status: Planned
Development

No branches or pull requests

1 participant