on-demand sync: stale .sync staging after killed sync causes all by-tag manifest GETs to hang

[jvallery]

zot version: v2.1.20 (-events-imagetrust-lint-metrics-mgmt-profile-scrub-search-sync-ui-userprefs binary), filesystem storage on an NFS mount (Synology, nfsvers=4.1, hard), dedupe+gc on, sync extension with 8 onDemand registries.

What happened (production, 2026-08-25): an in-flight on-demand sync of a ~10 GB multi-arch image (docker.io/gpustack/runner) was interrupted (client connection cut at our HTTP writeTimeout, then the pod was restarted while a second demand for the same image was in flight). After restart, a stale staging tree remained at <repo>/.sync/<uuid>/ (~18.5 GiB). From then on:

• Every by-tag manifest GET on EVERY repository hung indefinitely (each triggers an upstream freshness sync that appeared to queue forever — even for tiny, already-cached images on unrelated repos/registries).
• By-digest manifest GETs and /v2/_catalog remained fast (local store healthy).
• The process was near-idle (few mCPU), no Copy layer / sync progress log lines, no errors logged; readiness/liveness stayed green.
• Upstream (Docker Hub) and storage were verified healthy from the same host at the same time.

Recovery: stop zot, delete all .sync staging dirs, start zot → by-tag GETs immediately behaved normally (~2 s freshness syncs) and re-warms of the same big image completed fine (with a plain post-download commit/verify phase of up to ~12 min for ~10 GB on NFS, which is expected and distinct from the hang).

Suspicion: the on-demand sync worker pool / per-image demand channel can be exhausted or deadlocked when stale staging from a killed sync is present (possibly interacting with image already demanded, waiting on channel bookkeeping across restart), starving every subsequent freshness sync. A startup sweep of orphaned .sync dirs (or ignoring them for demand accounting) would likely prevent the state entirely.

Config highlights: onDemand: true x8 registries, maxRetries: 3, retryDelay: 1m, maxRetryDelay: 10m, SyncTimeout default 3h, storage gc 1h. Happy to provide fuller logs/config on request.

[rchincha]

| an in-flight on-demand sync of a ~10 GB multi-arch image (docker.io/gpustack/runner)

^ is this an AI model use case?

Wondering if the following would fix this!
#4347

[andaaron]

Can you please provide the information below?

Timeline and recovery

1. Exact order of events after the first sync was interrupted: when did by-tag GETs start hanging relative to the pod restart? (Before restart only / after restart only / both?)
2. During the hang after restart, was anything still trying to pull docker.io/gpustack/runner (or the same large image) again?
3. Recovery was stop → delete all .sync → start. Could you confirm whether a restart alone (orphans left in place) or deleting .sync alone (no restart) was ever tried? If not, would you be willing to note that for a future occurrence?
4. Rough free disk space on the NFS volume while the ~18.5 GiB orphan existed (and after deletion)?

Logs during the hang

For a few hung by-tag GETs (ideally one for the large image and one for a tiny unrelated repo), please share log lines around:

5. "trying to get updated image by syncing on demand"
6. "image already demanded, waiting on channel" (or referrers equivalent)
7. "starting on-demand image sync" / "sync: syncing image"
8. "skipping image because it's already synced" / "Copy layer" / "pushing synced local image"
9. Any credential-refresh or upstream error lines

Especially useful: did hung unrelated tags log only (5), or also (6), or (7) with nothing after?

Config

10. Paste the extensions.sync section (redact secrets): all 8 registries’ urls, onDemand, content prefixes, maxRetries / delays, reqConcurrent if set, downloadDir, credentialHelper.
11. Were any registries catch-all (empty / very broad content) so every tag GET tries that upstream first?
12. HTTP server readTimeout / writeTimeout (and any ingress / client timeouts) that cut the original pull.

Scope of the hang

13. Did by-tag GETs hang for repos that sync from non-Docker-Hub registries as well, or only images that would hit the same upstream as gpustack/runner?
14. With a hung by-tag GET in flight, did a by-digest GET for that same tag’s manifest digest still return quickly?
15. Did /v2/_catalog and readiness stay healthy for the entire hang window?

[andaaron]

Does this reproduce now that #4383 and #4410 have been merged?