[Bug]: Overwriting the last tag referencing a manifest makes it inaccessible by digest

[Meisterlala]

zot version

v2.1.21

Describe the bug

Overwriting the last tag referencing a manifest makes that manifest immediately inaccessible by digest in the same repository. Subsequent requests to /v2/<repository>/manifests/<original-digest> return HTTP 404 with MANIFEST_UNKNOWN and the manifest is not pullable.

Impact

My Kubernetes workloads reference images pinned to their SHA256 digests. When I publish a new image under the same tag (usually latest), existing workloads still reference the previous digest. If a pod subsequently restarts or is rescheduled and Kubernetes needs to pull that image again, the pull fails, leaving the workload unable to start (ErrImagePull / ImagePullBackOff). This breaks my canary-rollouts.

To reproduce

1. Start Zot locally at localhost:5000 with docker

services:
  zot:
    image: ghcr.io/project-zot/zot:v2.1.21
    ports:
      - "127.0.0.1:5000:5000"

2. Push an image

docker pull alpine:3.20
docker tag alpine:3.20 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

3. Push another image with the same tag

docker pull alpine:3.21
docker tag alpine:3.21 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:3c81aa9a3d770b316568f4499e30461a5cd3fbd7180bd89e28e34894c7845832

4. Pulling original image by digest fails

docker pull localhost:5000/myimage@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e
# Error: failed to unpack image on snapshotter overlayfs: unexpected media type application/octet-stream for sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e: not found

curl -i "http://localhost:5000/v2/myimage/manifests/sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e"
# HTTP 404: MANIFEST_UNKNOWN
#  {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"description":"This error is returned when the manifest, identified by name
 and tag is unknown to the repository.","reference":"sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e"}}]}

Server logs: log

Expected behavior

I expect the original manifest to remain pullable by digest after moving its tag, especially with GC disabled. OCI permits (under Definitions->Tags) manifests with zero tags, and other tested registries preserve access in this scenario.

docker pull alpine:3.20
docker tag alpine:3.20 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

docker pull alpine:3.21
docker tag alpine:3.21 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:3c81aa9a3d770b316568f4499e30461a5cd3fbd7180bd89e28e34894c7845832

docker pull localhost:5000/myimage@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

My Expected behavior, that the original manifest remains pullable by digest after its last tag is overwritten, works in these registries:

• Distribution v2.8.3 and v3.0.0
• Harbor v2.15.2
• Quay v3.15.8
• olareg v0.2.2

These results can be reproduced locally in docker using the attached registry-digest-retention.py (AI-generated):

python3 registry-digest-retention.py --all

Screenshots

No response

Additional context

Additinal findings:

Content remains stored, manifest lookup fails

GET /v2/myimage/manifests/<original-digest> -> 404 MANIFEST_UNKNOWN
GET /v2/myimage/blobs/<original-digest>         -> 200, original manifest bytes

Multiple tag behavior

If another pushed tag in the same repository still references the original manifest, it remains pullable by digest. The failure occurs only when the last remaining tag referencing that manifest is overwritten with a different image. The tag's name and age do not matter.

Deleting a tag keeps the digest

Deleting a tag though the api does leave the manifest pullable by digest. This was implemented in #2626

Affects single-arch and multi-arch images

The failure was reproduced with ordinary image manifests and multi-platform image indexes. In the multi-platform tests, the original index and its old platform manifests became inaccessible through the manifest endpoint.

Affects previous versions

I could reproduce this in v1.3.9, v1.4.3, v2.0.0, v2.0.1, v2.0.2, v2.0.3, v2.0.4, v2.1.0, v2.1.18, v2.1.21. So its likely not a regression

Changing GC settings does not help

The failure was reproduced with GC enabled and disabled, and with GC enabled plus deleteUntagged: false. Access was lost immediately after the overwrite, without waiting for GC or retention expiry.

Workarounds

There are some ways i can get the behavior i want. But these don't seem intended.

Keep a backup tag before overwriting the original tag

docker pull alpine:3.20
docker tag alpine:3.20 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

docker tag alpine:3.20 localhost:5000/myimage:backup-tag-to-tag1
docker push localhost:5000/myimage:backup-tag-to-tag1
# Pushed with sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

docker pull alpine:3.21
docker tag alpine:3.21 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:3c81aa9a3d770b316568f4499e30461a5cd3fbd7180bd89e28e34894c7845832

# Succeeds: backup still references the original manifest.
docker pull localhost:5000/myimage@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

Download the blob and upload it again

docker pull alpine:3.20
docker tag alpine:3.20 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

docker pull alpine:3.21
docker tag alpine:3.21 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:3c81aa9a3d770b316568f4499e30461a5cd3fbd7180bd89e28e34894c7845832

# Fails
docker pull localhost:5000/myimage@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

# Retrieve the existing manifest bytes from Zot.
curl -fS \
  http://localhost:5000/v2/myimage/blobs/sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e \
  -o original-manifest.json

# Re-register those exact bytes by digest.
curl -fS -i -X PUT \
  -H 'Content-Type: application/vnd.oci.image.manifest.v1+json' \
  --data-binary @original-manifest.json \
  http://localhost:5000/v2/myimage/manifests/sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

# Success
docker pull localhost:5000/myimage@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

Delete the tag temporarily

docker pull alpine:3.20
docker tag alpine:3.20 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

# Delete only the tag, not the manifest by digest.
curl -fS -i -X DELETE http://localhost:5000/v2/myimage/manifests/tag1

docker pull alpine:3.21
docker tag alpine:3.21 localhost:5000/myimage:tag1
docker push localhost:5000/myimage:tag1
# Pushed with sha256:3c81aa9a3d770b316568f4499e30461a5cd3fbd7180bd89e28e34894c7845832

# Success
docker pull localhost:5000/myimage@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e

[andaaron]

Thank you for reporting this, will look into fixing it.