Digest immutability issue with quay.io/nginx/nginx-unprivileged image

[aydosman]

Describe the bug

When using the NGINX Unprivileged Docker image from quay.io, it appears that during releases the digests of all tags are updated. As a result even when attempting to use a specific pinned version (e.g., quay.io/nginx/nginx-unprivileged:1.27.3-alpine@sha256:3092a71e4222a73893547dad52bbcf259582a6dd40c8b616e5bf44bcca3f01ff), the digest changes, making it impossible to pull the original image.

Expected behavior
The digest for a pinned version should remain immutable and unchanged, allowing users to reliably pull specific versions of the image regardless of subsequent releases.

Additional context
We observed that other related images such as the following, behave as expected with tags and digests remaining consistent on release:

nginx-prometheus-exporter
nginx-ingress
nginx-ingress-operator

Is there a reason why the digests for the nginx-unprivileged image change across releases? Or is this an oversight? This behaviour significantly impacts our environments.

[aydosman]

@alessfg -Any updates on this? I am asking you directly because I see that you are running the GitHub Action.

[mahesh-hegde]

I am observing this as well. Furthermore, I cannot search the old digest in the "untagged images" section of the ghcr because of this workflow. So this prevents me from even referring to this digest at all (like ghcr.io/nginxinc/nginx-unprivileged:1.27.3-bookworm@sha256:3092a71e4222a73893547dad52bbcf259582a6dd40c8b616e5bf44bcca3f01ff).

So it's not ideal for reproducibility.

I think it's inevitable the SHA would change because docker cache would not always be available and some OS dependent details (like the OS packages themselves, or anything that is randomly generated) may upgrade.

However is it required that you need to release image for a given nginx version eg: 1.27.3 multiple times and overwrite the existing package?

Or can we keep older digests?

Thanks

[alessfg]

I really don't have a good solution for this issue right now. Images are rebuilt on a weekly basis to avoid potential security issues and keep the base image updated.
If I update the image, the digest is also going to change no matter what. I will note that this is also the case with our core NGINX image and probably almost every other image out there, but they just happen at a much lower rate than what you see in this image.

Re the second issue mentioned here -- once upon a time I didn't cleanup untagged images, but that lead to issues with the image repositories getting full. I guess I could explore only deleting untagged images that are "x" time old, but I don't really have a timeline for when I could start looking into it. PRs are always welcome if any of you have any ideas!

[stevehipwell]

@alessfg it's a best practice to reference container images by their digest even if the tags are immutable, for mutable tags using digests is essential. This is primarily a security concern around the supply chain, but it's also required to have a deterministic system and matching pods across nodes.

Given the current pattern you're using if you add an additional <major>.<minor>.<patch>-<n> tag when building where <n> is incremented from the last build (next_n="$(crane ls --omit-digest-tags ghcr.io/nginxinc/nginx-unprivileged | grep -E '<major>\.<minor>\.<patch>-\d+' | jq -sRr '[split("\n") | .[] | select(. != "")] | sort | .[-1] | match("[0-9]+$").string | tonumber | . += 1')"). Then when you move the <major>.<minor>.<patch> tag the digest will no longer be cleaned up as it will still be tagged.

[alessfg]

Right, but this doesn't remove the issue of some registries being hard capped to only a set amount of images before running into issues. Not only that, but the goal is to keep these images and the tags identical to the core Docker NGINX images, and changing the tag pattern would break that.

Changing the cleanup algorithm to be time based instead of automatically deleting any untagged images is probably the way to go, but I need to find some time to dig into it.

[stevehipwell]

Right, but this doesn't remove the issue of some registries being hard capped to only a set amount of images before running into issues

@alessfg no it doesn't but it's not attempting to do so; it does fix the issue of this image not being consumable by industry best practices. Dealing with cleanup is a separate concern, but one that's easy enough to do as a GitHub Actions CRON job; for example a nightly check using crane to list the tags and if they're close to the limit (which is what, and are we talking tags or layers or total data?) clean up some of them based on a rule (e.g. remove a batch of the oldest <major>.<minor>.<patch>-<n> tags).

Not only that, but the goal is to keep these images and the tags identical to the core Docker NGINX images, and changing the tag pattern would break that.

The NGINX Docker library images aren't removed once published, surely that's the most important compatibility feature? That said the pattern I proposed above isn't mutually exclusive to your desire as it's not modifying your current tag pattern but is purely additive.