[sfc0324]

What: a small module + CLI that checks every adjacent level pair of an OME-Zarr pyramid and reports coarse chunks that have no object although the next finer level has data under them. zarr returns fill_value for those chunks, so viewers, loaders and annotators see blank papyrus while the data exists one level down.

• vesuvius/src/vesuvius/data/pyramid_occupancy.py: check_pyramid(store_url, anon=...) and expected_coarse_occupancy(...). It reuses zarr_chunk_index.build_chunk_occupancy (with use_cache=False) for the listing, so it adds no new dependencies and downloads no chunk data. It reads only .zattrs, .zarray and object listings. Scale factors come from the multiscales scale transforms, or from the level shapes if those are missing. Different chunk sizes per level and partial edge chunks are handled.
• vesuvius/src/vesuvius/scripts/check_pyramid_occupancy.py: python -m vesuvius.scripts.check_pyramid_occupancy [--anon] [--json] STORE.... It exits with code 1 if any hole is found, so it can gate an upload or run in CI.
• vesuvius/tests/data/test_pyramid_occupancy.py: 7 tests on synthetic local v2 stores (consistent pyramid, single missing chunk, / separator, empty coarse level, edge clipping, CLI exit code).

Why: this is the gap behind #1965 and #2092. Existing checks look at one level at a time, so they cannot see a coarse level that is internally fine but missing chunks over real data.

Evidence (run 2026-10-14, public bucket, ~8 s total for both stores):

$ python -m vesuvius.scripts.check_pyramid_occupancy --anon \
    s3://vesuvius-challenge-open-data/PHerc0343P/volumes/20250521134555-8.640um-1.2m-116keV-masked.zarr \
    s3://vesuvius-challenge-open-data/PHerc1299/representations/predictions/surfaces/20260309130042-surface-20260413222639-surface-m7-L2-th0.2.zarr
...PHerc0343P...  level 0 -> 1: HOLES 123/1737 e.g. [[0, 1, 1], [0, 1, 2], [0, 1, 3], [0, 2, 0], [0, 2, 1]]
...PHerc0343P...  level 1 -> 2 .. 4 -> 5: ok
...PHerc1299...   level 0 -> 1 .. 3 -> 4: ok
...PHerc1299...   level 4 -> 5: HOLES 1/3 e.g. [[2, 0, 0]]
exit 1

These match #1965: level 1 has no objects over the oversized level-0 region. They also match #2092: level 5 chunk 2/0/0 is missing.

Scope / limits: zarr v2 only, the same limit as zarr_chunk_index. A level that can't be indexed is reported as SKIPPED and never counted as holes. A hole means "a chunk is expected and no object exists". It does not decode chunks to prove the finer data is non-fill. The full standalone audit (oversized chunks, v3/sharded stores, gap verification by decoding) is at https://github.com/sfc0324/zarrsweep (MIT).

Tests: pytest vesuvius/tests/data vesuvius/tests/utils → 80 passed, 1 skipped. This includes the test_no_silent_swallowing ratchet. The new module adds no broad except, and a missing .zattrs is the only case that returns None.

AI disclosure: prepared with an AI assistant. I ran and checked the code, tests and S3 output above.

[vercel]

@sfc0324 is attempting to deploy a commit to the scroll Team on Vercel.

A member of the Team first needs to authorize it.