[Bug]: on-demand sync from an ECR pull-through cache fails for freshly pushed images with a provenance attestation child

[ohookins]

Describe the bug

We're using zot as an on-demand pull-through mirror in front of an AWS ECR pull-through cache, and we can't reliably pull freshly built images through it. The pull fails even though the actual platform image resolves fine directly.

After digging into it, the culprit is the digest-prediction walk that runs before a copy. It does a ManifestGet on every child in the image index, including the BuildKit provenance attestation entry (the unknown/unknown platform child). Right after a build, ECR's pull-through cache will serve the index and the real linux/amd64 manifest, but it hasn't materialized the attestation child yet, so that one GET errors out and the whole pull is aborted.

So in practice zot can't sit in front of an ECR pull-through cache for any image that carries a provenance attestation, which is basically anything built by BuildKit with the defaults these days.

To reproduce

Our setup:

• An ECR pull-through cache (upstream that materializes images lazily, on first real pull).
• zot as an on-demand sync mirror pointing at that pull-through cache.
• A multi-arch image built by BuildKit with default provenance. The index looks like this:

mediaType: application/vnd.oci.image.index.v1+json
manifests:
  - platform: linux/amd64      mediaType: application/vnd.oci.image.manifest.v1+json   # the real image
  - platform: unknown/unknown  mediaType: application/vnd.oci.image.manifest.v1+json   # provenance attestation

and the unknown/unknown child is an in-toto attestation manifest:

mediaType: application/vnd.oci.image.manifest.v1+json
config:    application/vnd.oci.empty.v1+json
layers:    [ application/vnd.in-toto+json ]

To hit it:

1. Build/push a fresh image with this shape so it isn't materialized in the pull-through cache yet.
2. Immediately docker pull it through zot.
3. It fails, with a manifest-not-found error that references the attestation child's digest.

A plain docker pull --platform linux/amd64 straight at the ECR pull-through cache works at the same moment, because it only fetches the platform manifest it needs and never touches the attestation child.

Expected behavior

I'd expect zot to serve the image the same way that direct docker pull does — an as-yet-unresolvable attestation child in the index shouldn't block the pull of the actual platform image.

What I found

The skip check computes the would-be OCI digest before copying, to decide whether the image is already synced. In v2.1.20 that's predictOCIDigest in pkg/extensions/sync/oci_digest_predict.go, reached via RemoteRegistry.GetOCIDigest (remote.go) from BaseService.computeLocalStoredImageDigest (service.go).

fetchManifestNode walks the index and ManifestGets every child, with no special-casing for attestation/referrer children, and any single failing child kills the whole thing:

for _, desc := range manifestList {
    childRef := imageRef.SetDigest(desc.Digest.String())
    child, err := fetchManifestNode(ctx, regClient, childRef, false, walkState)
    if err != nil {
        return nil, err   // aborts prediction, so the pull fails
    }
    ...
}

Against a fresh image in an ECR pull-through cache, the attestation child is the one that isn't materialized yet, so this fails despite the requested platform manifest being available.

I also tried PreserveDigest: true, since it skips the predictOCIDigest walk. It does avoid the attestation-child GET, but then computeLocalStoredImageDigest calls RemoteRegistry.GetDigest, which is a ManifestHead on the tag — and a HEAD against the ECR pull-through cache doesn't trigger materialization of a fresh tag either, so it also fails. So neither PreserveDigest setting gets a fresh image through when the upstream is an ECR pull-through cache:

• PreserveDigest: false → predictOCIDigest GETs the attestation child → fails.
• PreserveDigest: true → ManifestHead on a not-yet-materialized tag → fails.

A possible fix (up to you)

One option that would have solved it for us: during the prediction walk, tolerate a child that can't be resolved when it's an attestation/referrer-style entry (unknown/unknown platform, in-toto/empty-config manifest) rather than aborting the whole prediction — skip it, or just treat prediction as "can't skip, go ahead and copy". Falling back to a plain copy whenever prediction fails would also work, and a config knob to exclude attestation children from the walk is another angle.

I don't want to prescribe the approach though — you'll have a much better sense of the digest-correctness implications than I do, so I'll leave the call to you. Happy to help test or put up a PR if that's useful.

Related

Looks related to but distinct from #2836 (OCI image index support for on-demand sync from ECR). In our case the index and the real platform manifest do resolve; the specific thing that breaks is the attestation child of a freshly pushed image not being materialized in time.

zot version

v2.1.20

Environment

• zot as an on-demand sync (pull-through) mirror.
• Upstream: an AWS ECR pull-through cache.
• Multi-arch images built by BuildKit with default provenance (index carries an unknown/unknown in-toto attestation child).

[rchincha]

@ohookins thanks for filing this issue.

No great answers here.
Either we take a pessimistic/conservative view and reject an image if child deps are not present. This is the current behavior.
(or)
we take a optimistic view and accept an image even if child deps are not present yet. But then how do we know if this image will ever be valid!

[andaaron]

It's basically the same case as #3128

[andaaron]

Just merged #4410. Please let me know if this solves the issue you are facing.