Use custom service account for search dataform

[emmalowe]

GCP is enforcing a new access control model for Dataform called "strict act-as mode". As part of this, GCP is disabling the ability for Dataform instances to be run using the Default Dataform service Account, so all Dataform workflows must switch to a custom service account, which the Dataform Service Agent is given permissions on. The custom service account must include the iam.serviceAccounts.actAs permission to configure Dataform workflows.

Steps covered here:

• Move creation of custom service account "dataform-sa"
• Change BigQuery internal project permissions from default service account to custom service account
• Give default service account permissions to impersonate custom service account
• Add custom service account to repo set up (to use this as the default account for running workflows)
• Give custom service account secrets permissions to connect to our dataform Github repo

See:

• Background on "strict act-as" mode: https://docs.cloud.google.com/dataform/docs/strict-act-as-mode
• GCP guidance on granting Dataform the required access: https://docs.cloud.google.com/dataform/docs/access-control#grant-dataform-required-access
• Service account migration guidance: https://gov-uk.atlassian.net/wiki/spaces/GIAT/pages/5133271058/Dataform+Service+Account+Migration

Note that we don't need read permissions on other projects. GA4 data is written to our project from a different project - that project will need to add write permissions to our project when they come to switch their accounts:
https://github.com/alphagov/govuk-infrastructure/blob/main/terraform/deployments/search-api-v2/events_ingestion.tf#L27

[emmalowe]

Since this is Terraform, I assume the approach to testing is to get this merged and then test it out on integration before applying the changes in production.

[aaronfowles]

The terraform plan looks like it failed with what could be a transient error. Might be worth triggering a retry from the UI to see if that resolves...

[emmalowe]

The terraform plan looks like it failed with what could be a transient error. Might be worth triggering a retry from the UI to see if that resolves...

Thanks @aaronfowles ! I reran the plan and it looks like it's working now.

[hannako]

This makes sense to me based on the confluence documentation and my reading of the gcp docs. But as discussed with @emmalowe in person, our roll out approach will be to merge, apply the change in integration and confirm all looks as expected before applying the change in production