Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support publishing only authorized views #5429

Open
whd opened this issue Apr 24, 2024 · 1 comment
Open

Support publishing only authorized views #5429

whd opened this issue Apr 24, 2024 · 1 comment

Comments

@whd
Copy link
Member

whd commented Apr 24, 2024

Now that almost all artifact deployment is handled via Airflow, the only thing we need to deploy from Jenkins are authorized views. I wrote some custom logic to do that in https://github.com/mozilla-services/cloudops-infra/pull/5478 but it would be nice to support it via bqetl natively.

I think I started putting this together in https://github.com/mozilla/bigquery-etl/tree/only-authorized but didn't have time to circle back to it.

┆Issue is synchronized with this Jira Task

@whd
Copy link
Member Author

whd commented Oct 3, 2024

Another consequence of having proper support for this is that it'll make cleaning up deleted authorized views easier. Currently when a PR like https://github.com/mozilla/private-bigquery-etl/pull/462/files lands that deletes some such views, they are not cleaned up by Airflow artifact deployment because they are authorized. I could probably add a cleanup step to https://github.com/mozilla-services/cloudops-infra/blob/master/projects/data-shared/Jenkinsfile.bigquery#L258 to do the cleaning but I didn't want to accidentally delete all the other non-authorized views when I added that, which I rate as having a medium chance of occurring.

So for now, if we're deleting authorized views we (probably DSRE) will need to delete them manually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant