Skip to content

Use custom service account for search dataform #3800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
emmalowe wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Use custom service account for search dataform #3800

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6e249c3
Move setup of custom dataform service account
emmalowe Mar 4, 2026
1756975
Use custom service account for search dataform
emmalowe Mar 4, 2026
04fce38
Remove unhelpful comment
emmalowe Mar 18, 2026
e6323a6
Add read permissions from search Dataform to GA4
emmalowe Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions terraform/deployments/ga4-analytics/project_iam_binding.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,25 @@ resource "google_project_iam_binding" "gds_log_alert_writer" {
"group:govuk-performance-analysts@digital.cabinet-office.gov.uk"
]
}

locals {
Copy link
Contributor Author

@emmalowe emmalowe Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we only need to read from 2/17 datasets in the GA4 Analytics project, I've added these as dataset permissions instead of project permissions (which would give access to all the datasets). This might be overkill.

I've chosen to use iam_member instead of iam_binding for two reasons:

  1. I'm not confident how dataset permissions interact with project permissions, and I didn't want to wipe out any permissions for anyone else. From what I've read, dataset binding and project bindings together are additive, so that shouldn't be the case, but I'm not sure.
  2. It seems like some of GA4 Analytics permissions have been set up via ClickOps and again, I don't want to delete permissions for anyone else. But maybe I need to do more work to check what is there and rectify any issues because we probably should use binding if we can (see point 1).

If the permission set up looks okay, I'm not clear on how this new code should be organised. I've seen locals.tf files elsewhere. Maybe we should also have a separate file for dataset permissions 🤷🏻‍♀️

members = [
"serviceAccount:dataform-sa@search-api-v2-integration.iam.gserviceaccount.com",
"serviceAccount:dataform-sa@search-api-v2-staging.iam.gserviceaccount.com",
"serviceAccount:dataform-sa@search-api-v2-production.iam.gserviceaccount.com",
]
}

resource "google_bigquery_dataset_iam_member" "flattened_dataset_reader" {
for_each = toset(local.members)
dataset_id = "${google_project.project.project_id}.flattened_dataset"
role = google_project_iam_custom_role.gds_bigquery_read_access.name
member = each.key
}

resource "google_bigquery_dataset_iam_member" "events_reader_and_lister" {
for_each = toset(local.members)
dataset_id = "${google_project.project.project_id}.analytics_330577055"
role = google_project_iam_custom_role.gds_bigquery_read_and_list_access.name
member = each.key
}
13 changes: 13 additions & 0 deletions terraform/deployments/ga4-analytics/project_iam_custom_roles.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,19 @@ resource "google_project_iam_custom_role" "gds_bigquery_read_access" {
title = "GDS BQ read access"
}

resource "google_project_iam_custom_role" "gds_bigquery_read_and_list_access" {
Copy link
Contributor Author

@emmalowe emmalowe Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm adding this custom role, because for some of our pipelines that use a table wildcard, we need the bigquery.tables.list permission, which isn't in gds_bigquery_read_access. This also might be overkill, since I could just add bigquery.tables.list to gds_bigquery_read_access.

Copy link
Contributor Author

@emmalowe emmalowe Mar 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think because I'm only adding a list permission (rather than create or delete), it's not worth making a separate role, but I'm curious to know what other people think.

description = "Permissions to read and list BigQuery datasets and tables"
permissions = [
"bigquery.datasets.get",
"bigquery.datasets.getIamPolicy",
"bigquery.tables.get",
"bigquery.tables.getData",
"bigquery.tables.list"
]
role_id = "GDS_BQ_read_and_list_access"
title = "GDS BQ read and list access"
}

resource "google_project_iam_custom_role" "gds_bigquery_saved_query_writer" {
description = "Permissions to create, update and delete BigQuery saved queries"
permissions = [
Expand Down
50 changes: 34 additions & 16 deletions terraform/deployments/search-api-v2/dataform.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
# Create a service account for Dataform
resource "google_service_account" "dataform_service_account" {
account_id = "dataform-sa"
display_name = "Dataform Service Account"
project = var.gcp_project_id
}

variable "search_dataform_github_repository_url" {
description = "URL of the GitHub repository to link with Dataform"
type = string
Expand All @@ -14,22 +21,16 @@ resource "google_secret_manager_secret_iam_member" "member" {
project = var.gcp_project_id
secret_id = "github_search_v2_api_dataform_ssh_key"
role = "roles/secretmanager.secretAccessor"
member = "serviceAccount:service-${var.gcp_project_number}@gcp-sa-dataform.iam.gserviceaccount.com"
}

# Create a service account for Dataform
resource "google_service_account" "dataform_service_account" {
account_id = "dataform-sa"
display_name = "Dataform Service Account"
project = var.gcp_project_id
member = google_service_account.dataform_service_account.member
}

# Create Dataform repository with GitHub integration
resource "google_dataform_repository" "search_api_v2" {
provider = google-beta
name = "search_api_v2"
project = var.gcp_project_id
region = var.gcp_region
provider = google-beta
name = "search_api_v2"
project = var.gcp_project_id
region = var.gcp_region
service_account = google_service_account.dataform_service_account.email

git_remote_settings {
url = var.search_dataform_github_repository_url
Expand Down Expand Up @@ -106,22 +107,39 @@ resource "google_dataform_repository_workflow_config" "search-monthly" {
}
}

# BigQuery cross-project permissions
# Permissions for Default Dataform Service Agent to impersonate our custom Dataform Service Account
# This allows Dataform to set up and run new workflows
resource "google_service_account_iam_binding" "dataform_sa_impersonation" {
service_account_id = google_service_account.dataform_service_account.name
role = "roles/iam.serviceAccountUser"
members = [
"serviceAccount:service-${var.gcp_project_number}@gcp-sa-dataform.iam.gserviceaccount.com"
]
}

resource "google_service_account_iam_binding" "dataform_sa_token_creator" {
service_account_id = google_service_account.dataform_service_account.name
role = "roles/iam.serviceAccountTokenCreator"
members = [
"serviceAccount:service-${var.gcp_project_number}@gcp-sa-dataform.iam.gserviceaccount.com"
]
}

# Service account permissions to access BigQuery
resource "google_project_iam_member" "bigquery_data_editor" {
project = "search-api-v2-${var.gcp_env}"
role = "roles/bigquery.dataEditor"
member = "serviceAccount:service-${var.gcp_project_number}@gcp-sa-dataform.iam.gserviceaccount.com"
member = google_service_account.dataform_service_account.member
}

resource "google_project_iam_member" "bigquery_job_user" {
project = "search-api-v2-${var.gcp_env}"
role = "roles/bigquery.jobUser"
member = "serviceAccount:service-${var.gcp_project_number}@gcp-sa-dataform.iam.gserviceaccount.com"
member = google_service_account.dataform_service_account.member
}

resource "google_project_iam_member" "bigquery_data_viewer" {
project = "search-api-v2-${var.gcp_env}"
role = "roles/bigquery.dataViewer"
member = "serviceAccount:service-${var.gcp_project_number}@gcp-sa-dataform.iam.gserviceaccount.com"
member = google_service_account.dataform_service_account.member
}
Loading