Issue with Concourse authenticating with GCP Workload Identity Federation to check/pull Artifact Registry images

[cal-nic]

Hi,

Hoping this is the right place to post this..

My organisation has recently shifted towards the use of Workload Identity Federation (WIF) within Google Cloud Platform instead of our previously used method of authentication/access via service account keys.

We are running Concourse v7.13.1 hosted on a google kubernetes cluster, and for the most part have been successful with the transition to WIF. We have bound our concourse workers to the relevant kubernetes accounts, and they are successfully able to impersonate our designated service account for concourse/WIF. Here's a link to our current concourse repo/WIF implementation: https://github.com/ONSdigital/blaise-concourse/blob/main/concourse/iam.tf

The issue we're having is when we reference our own images hosted within GCP's artifact registry, concourse seems to be doing an initial 'check' on these images, presumably to check they exist. Previously we would pass in the authentication details as part of the image resource, which would work fine. However, if we remove these keys and try to use WIF, it seems to be unable to authenticate. The following error occurs:

Then when you drill further into the above (I've removed some of the project specific references jic the links look weird!):

No matching credentials were found for "europe-west2-docker.pkg.dev"
ERRO[0000] checking origin europe-west2-docker.pkg.dev/repo-here failed: initialize transport: GET https://europe-west2-docker.pkg.dev/v2/token?scope=repository%3Arepo-here%3Apull&service=: DENIED: Unauthenticated request.
Unauthenticated requests do not have permission "artifactregistry.repositories.downloadArtifacts" on resource "projects/project-here" (or it may not exist)

We've done the obvious, e.g. making sure the service account that we're trying to use has correct permissions. It just doesn't seem to even get to the point where it tries to use WIF.

These images I mentioned are just 'base' images, containing tools such as python/gcloud as our kubernetes cluster that concourse runs on is using "Container-optimised OS", meaning we've been unable to run gcloud commands to force the authentication, as the image doesn't actually have gcloud on it. We'd also like to avoid building the images as part of the jobs if possible, just to keep run times down.

I've been on numerous calls with Google's support, and after some back and forth/troubleshooting they have suggested the following:

The core of the issue appears to be that Concourse is not leveraging the Application Default Credentials (ADC) made available through the standard Workload Identity credential flow via the GKE metadata server. You can find more details on this standard mechanism in the official documentation: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity#credential-flow

Concourse seems to be working fine with everything else using WIF, it's just this one particular aspect of GCP (artifact registry) where we're facing this block.

I'd really appreciate some support/suggestions, more than happy to provide more information too just let me know.

Cheers

[taylorsilva]

The repo you linked to is private (https://github.com/ONSdigital/blaise-concourse/blob/main/concourse/iam.tf) it currently returns 404 when I try to visit.

There is not enough information here to provide any guidance on what to try next. Some immediate questions/thoughts:

• it looks like you're pulling the images via the task config's image_resource field. What resource type are you using? registry-image (https://github.com/concourse/registry-image-resource) or docker-image (https://github.com/concourse/docker-image-resource/)
• Depending on which resource type you're using, we'd then need to investigate to see if either resource knows to use the GCP metadata server. I don't think either of them do TBH.
  • Registry image uses the AWS metadata service because we leverage the AWS SDK. The same is not happening for GCP though.

---

As the maintainer of Concourse I am providing commercial support for Concourse as a way to support the ongoing maintenance of the project. If you think that's something your org would be interested in you can find more details here: https://concourse-ci.org/support.html

[cal-nic]

Thanks for the reply, apologies - forgot that our repo for concourse is private.

We are pulling images via the task config's image_resource as you say..

The current working implementation which passes through the service account keys:

As you can see there this is using the registry-image type. When we tried to use WIF by removing the credits, that's when we got this error:

We also tried docker-image, where we experienced the 'same' issue (image check failed), but with a different error:

For a little more context - at first we thought it was permissions that was causing the issue, so we tried adding various permissions around Artifact registry to the service account that we were trying (initially artifact reader/writer, then admin - we even just gave the account a blanket owner permission for troubleshooting.

I then also read something about potentially the 'default' compute account being used, so I did the same with that just incase it was a permissions thing. Unfortunately, neither worked.

If you think that neither will use the GCP metadata server that makes sense, if that does turn out to be the case - is this something that could be added? or are we better off looking at alternative ways?

Really appreciate your help, and if I can provide any more information please let me know. If there's any setup/config snippets from our repo that might be helpful I can provide them since I can't share the repo itself.

Cheers
Cal

[taylorsilva]

okay, I am NOT an expert in this and haven't set this kind of stuff up before, but I think the IDToken credential manager is what you're looking for: https://concourse-ci.org/idtoken-credential-manager.html
Idtoken was added in v7.14.0

In your GCP project, you would setup Concourse as an identity provider: https://docs.cloud.google.com/iam/docs/workload-identity-federation-with-other-providers

Then in your pipelines you'd define a var_source:

var_sources:
- name: myidtoken
  type: idtoken
  config: ...

Then your resource configuration would look like this:

- name: myimage
  type: registry-image
  source:
    password: ((myidtoken:token))
    # other fields go here

If you're on 7.14 there's nothing you need to do, IDtoken is setup by default. All the setup should be on the GCP side to support. If your Concourse is private you might have issues, see #9358 which should resolve that.

Hope that helps!

[cal-nic]

Hey,

I shall take a look into this as it sounds promising! Thanks again for looking into it, much appreciated :)

Cheers
Cal