DRAFT: Use the Dataform Service Account for search api v2

[hannako]

What

Following changes to the Dataform API (see why section below) we need to use a service account with the predefined role of User Service Account to orchestrate our data pipelines. We need to remove permissions from the Dataform service agent.
Jira: https://gov-uk.atlassian.net/browse/SCH-1670

WIP 🚧

• I've granted the relevant role the existing Dataform service account. I don't think it was being used. It had no role assigned.
• I've given the Dataform service account analytics_write permission for the events ingestion.

The instructions state that we also need to remove permission for the dataform service agent. I'm not sure how best to do this.

• We would need to remove it from events ingestion here. Once that's merged, would we follow up by adding a google_project_iam_member_remove block?
• Do we need to replace the dataform service agent with the Dataform service account in these blocks?
• The dataform service agent is being used here. But this membership doesn't seem to be referenced anywhere, so I'm not sure what it's doing or what action to take.

Why

See here

Grant explicit act-as permissions for Dataform security enhancements

We’re writing to follow up on our previous communication and inform all new and existing customers that we’re making changes to our security model in Dataform. Starting January 19, 2026, security enhancements in the Dataform API will change how workflows are run and what service accounts users can use.

This change affects all users and scheduled workflows of Dataform, BigQuery Notebooks, BigQuery Pipelines, and BigQuery data preparations, unless their permissions are already set up in a way outlined below. To help you prepare for the upcoming security enhancements, we have also released a new diagnostic tool.

We understand that these changes may require some planning and decision-making. Therefore, we have provided additional information about the tool and the changes below to guide you through the transition.

What you need to know
Key changes:

This update enforces a new access control model known as strict act-as mode. It affects the following resources:
Workflows need to be scheduled to run using either a custom service account or a user’s Google Account. Running workflows using the Dataform service agent will no longer be allowed. Existing Dataform, BigQuery Notebook, BigQuery Pipelines, and BigQuery data preparation workflows using the Dataform service agent will stop running.
Users who update release configurations in Dataform or configure workflows in Dataform, BigQuery Notebook, BigQuery Pipelines, and BigQuery data preparation need to have the iam.serviceAccounts.actAs permission on custom service accounts used in those workflows.
For Dataform repositories not connected to a third-party git repository, automatic releases will be disabled.
New diagnostic tool:

We have introduced a new log-based diagnostic tool in Cloud Logging to help you identify and resolve potential permission issues before the changes take effect starting January 19, 2026. For more information, review our documentation on Using strict act-as mode.

Timeline:

January 19, 2026: Act-as check will be enforced for all newly created repositories.
Between April 29 and July 31, 2026: We will gradually enforce the strict act-as mode for existing repositories.
What you need to do
Action is required before January 19, 2026, for new repositories and before April 29, 2026, for existing ones:

Switch all workflows using the Dataform service agent to use a custom service account. This applies to all scheduled workflows for Dataform, BigQuery Notebook, BigQuery Pipelines, and BigQuery data preparation.
Ensure that the appropriate principals have the Service Account User role (roles/iam.serviceAccountUser) granted on the custom service accounts in Identity and Access Management (IAM). This role contains the iam.serviceAccounts.actAs permission. Users without the iam.serviceAccounts.actAs permission will be unable to create new schedules or manually invoke workflows using the service account.
Grant this role to the Dataform service agent on custom service accounts used in workflow configurations in Dataform, scheduled notebooks, pipelines or data preparations in BigQuery.
Grant this role to all users who need to use custom service accounts to create or modify release configurations, workflow configurations or schedules.
Grant this role to the custom service accounts that call the Dataform API to start workflow invocations (for example, using Cloud Composer).

[emmalowe]

I've been digging into this a bit today.

We would need to remove it from events ingestion here. Once that's merged, would we follow up by adding a google_project_iam_member_remove block?

When I looked at this before, I found it challenging because in the console I couldn't see what roles the default service accounts actually had. Thanks to Gemini, I learnt there is a checkbox labelled "Include Google-provided role grants" that let's you see this.

https://console.cloud.google.com/iam-admin/iam?project=search-api-v2-integration

Looking at "View by roles" under "ga4-write-bq-permissions", I noticed that the "default" service account with this permission isn't the one for our GCP project. So I'm not sure that we need to change the permissions here. (Someone might have to, but I'm not sure how this plugs in.)

Wherever we do change permissions, the IAM View by roles tab with the "Include Google-provided role grants" should hopefully give us a good indication of whether we need to run anything extra to remove the role explicitly.

Do we need to replace the dataform service agent with the Dataform service account in these blocks?

I'm pretty sure this is what we need to change.

We're getting warnings when running the workflows listed in the dataform.tf file in the cloud logs that are telling us we need to update our permissions.

These docs on creating a Dataform repo in GCP also tell us these specific roles need to be granted.

I think we also need to change the service account in the repo settings. There might be an extra line in the Terraform when we set up the repo we can add for this and without it, it's using the default. (I've run out of time to check this before my next meeting, but want to write my thoughts down before context switching!)

I noticed in the settings, at least in Integration that we have "actAs permission checks" off right now. We should probably turn these on so it's easier to surface if there's anything else missing.

The dataform service agent is being used here. But this membership doesn't seem to be referenced anywhere, so I'm not sure what it's doing or what action to take.

I think this relates to the service account under the Dataform repo settings, which I mentioned above.