[CVasilopoulos]

In one sentence: write_tifxyz(path, surface, overwrite=True) now replaces the surface files and keeps every other file in the segment directory, as QuadSurface::saveWithUuid does, instead of deleting the directory and putting back only x/y/z.tif, meta.json and mask.tif.

One real example: Starting with a copy of PHercParis4 segment 20231007101619 (the 45.532 µm tifxyz) with an ink label next to it, I rewrote the surface in place with s = read_tifxyz(seg); write_tifxyz(seg, s, overwrite=True). The label was the published 2.4 µm ink prediction for this segment, resized to the surface's full resolution and saved as 20231007101619_ink.png. On main the call returns normally and the label is gone: list_labels() goes from 1 label to 0. With this PR the label is still there, byte-identical, and the surface files are rewritten as before.

Before: TifxyzWriter.write writes the surface into a sibling temp directory, then runs shutil.rmtree on the existing directory and moves the temp directory into its place. Anything the writer does not produce itself is deleted without a warning: label images, generations.tif, approval.tif, overlapping.json, trace.json.

After this PR: just before the swap, every regular file of the old directory that the new write did not produce is copied into the temp directory, so the swap keeps it. Files the writer does produce are replaced by the new ones. This is the same rule QuadSurface::saveWithUuid follows when it overwrites a segment ("Preserve auxiliary files from the existing directory before replacing"). Writing to a new directory, and overwrite=False, are unchanged.

Proof: upstream main 5432721dc against this branch, same segment copy, same call, same container. Look at the 20231007101619_ink.png row and the label lists.

segment: PHercParis4 20231007101619 at 45.532 um, tifxyz grid (137, 285), full resolution (2740, 5700)
label: published 2.4 um ink prediction (ds8 JPEG, 13518 x 6505) resized to 5700 x 2740 -> 20231007101619_ink.png

### main: s = read_tifxyz(seg); write_tifxyz(seg, s, overwrite=True)
exit code 0
  | labels before: ['20231007101619_ink.png']
  | labels after: []
  | x/y/z written from the surface: True
files before -> after:
  20231007101619_ink.png       6228195 B 1ea2297f1614   -> MISSING
  mask.tif                   -                          ->      2238 B 1c3d30b71736
  meta.json                        427 B 9abe01aa8b77   ->       350 B 628ab0b84578
  x.tif                         156314 B 8f4a198ed107   ->    148059 B 0730dfd627ea
  y.tif                         156314 B 29fe10c068a9   ->    148173 B 798f90587709
  z.tif                         156314 B 6985d4cd2562   ->    138270 B 634ae7c190a3
other files: 1 before, 0 after, 0 changed
leftover temp dirs: []

### fix: s = read_tifxyz(seg); write_tifxyz(seg, s, overwrite=True)
exit code 0
  | labels before: ['20231007101619_ink.png']
  | labels after: ['20231007101619_ink.png']
  | x/y/z written from the surface: True
files before -> after:
  20231007101619_ink.png       6228195 B 1ea2297f1614   ->   6228195 B 1ea2297f1614
  mask.tif                   -                          ->      2238 B 1c3d30b71736
  meta.json                        427 B 9abe01aa8b77   ->       350 B 628ab0b84578
  x.tif                         156314 B 8f4a198ed107   ->    148059 B 0730dfd627ea
  y.tif                         156314 B 29fe10c068a9   ->    148173 B 798f90587709
  z.tif                         156314 B 6985d4cd2562   ->    138270 B 634ae7c190a3
other files: 1 before, 1 after, 0 changed
leftover temp dirs: []

### tests/tifxyz/test_writer_overwrite.py against main
FAILED tests/tifxyz/test_writer_overwrite.py::test_overwrite_keeps_companion_files
1 failed, 1 passed in 1.32s

### tests/tifxyz/test_writer_overwrite.py against fix
2 passed in 1.51s

### all of tests/tifxyz on this branch
45 passed in 3.02s

The rewritten x/y/z.tif, meta.json and mask.tif (the writer adds one; the published segment has none) have the same hashes on both sides. The only difference is the label: deleted on main, untouched here.

Why / where this is useful:

• The sheets tracer re-traces into an existing directory with write_surface(..., overwrite=True) and deliberately keeps the surface uuid across re-traces, so the directory is treated as one persistent surface; every re-trace still wiped whatever else had been put in it.

• autoreg_mesh/infer.py writes its prediction to <save_path>/<segment uuid> with overwrite=True.

• Rewriting a segment in place is the natural way to refresh its meta.json from Python: the writer recomputes the bbox from the valid points, which makes the read-time fix from tifxyz: recompute the bbox from valid points when the stored one carries the -1 marker (refs #1618) #1731 permanent. Ink labels are often hand-drawn and the only copy, and losing them raised nothing.

• I personally verified that the example and proof above were produced by this PR on the stated data.

Details

• Change: three lines in vesuvius/src/vesuvius/tifxyz/writer.py TifxyzWriter.write, in the overwrite branch only: before shutil.rmtree(original_path), regular files of original_path that do not exist in the temp directory are copied there with shutil.copy2. The original directory stays intact until the copies are done, so a failure leaves it as it was.
• Test: new vesuvius/tests/tifxyz/test_writer_overwrite.py. One test puts a label PNG, generations.tif and overlapping.json next to a written surface, rewrites it with overwrite=True, and checks the three files are byte-identical, x.tif holds the new coordinates and no temp directory is left. The other checks that mask.tif, which the writer produces, is replaced by the new surface's mask rather than carried over. The first test fails on main; both pass here, as do the existing tests/tifxyz tests.
• Same limits as the C++ writer: subdirectories are not carried over (the C++ save copies regular files only), and a carried-over label keeps its old size if the new surface has a different grid shape. TifxyzReader.discover_labels(validate_shapes=True) already flags label shapes that do not match the grid.
• Data: the tifxyz grid of published segment 20231007101619 rendered at 45.532 µm (137 × 285, scale 0.05, full resolution 2740 × 5700), the segment used in vc_obj2tifxyz: write scale as cells per voxel measured from the grid, fail on an empty grid #1781, Resampled tifxyz grids: don't keep vertices interpolated from a missing vertex (merge_concat_runs) #2067 and labels_to_zarr: use the resized label on the first run instead of writing an empty volume #2088. The label is the published ink-detection/downsampled/...-2.4um-...-ds8.jpg for the same segment (13518 × 6505, same aspect ratio as the surface), resized with INTER_AREA and saved as an 8-bit PNG.
• Environment: Python 3.12 in python:3.12-slim, Linux x86_64; numpy 2, tifffile, opencv-python-headless 5.0.0. Not tested on Windows or macOS.
• Duplicate check: searched open and closed PRs and issues for write_tifxyz, TifxyzWriter and tifxyz writer overwrite. vesuvius: tifxyz write_metadata recomputes the bbox from valid vertices #1606 (closed) and tifxyz: recompute the bbox from valid points when the stored one carries the -1 marker (refs #1618) #1731 changed the bbox, UUID alignment #2077 the uuid; none touches the overwrite branch.

Evidence script (evidence.py)

import hashlib
import os
import shutil
import subprocess
import sys

SEG = "/w/data/seg-20231007101619-45um"
INK = "/w/data/tw-src/inkpred-2.4um-ds8.jpg"
OUT = "/w/build/tw/ev"
LABEL = "20231007101619_ink.png"

CHILD = r'''
import hashlib, sys
import numpy as np
import tifffile
from vesuvius.tifxyz import read_tifxyz, write_tifxyz

seg = sys.argv[1]
surface = read_tifxyz(seg)
print("labels before:", [l["filename"] for l in surface.list_labels()])
write_tifxyz(seg, surface, overwrite=True)
after = read_tifxyz(seg)
print("labels after:", [l["filename"] for l in after.list_labels()])
same = all(np.array_equal(tifffile.imread(f"{seg}/{c}.tif"), getattr(surface, "_" + c)) for c in "xyz")
print("x/y/z written from the surface:", same)
'''


def sha(path):
    return hashlib.sha256(open(path, "rb").read()).hexdigest()[:12]


def listing(d):
    return {n: (os.path.getsize(os.path.join(d, n)), sha(os.path.join(d, n)))
            for n in sorted(os.listdir(d)) if os.path.isfile(os.path.join(d, n))}


def prepare(name):
    import cv2
    root = f"{OUT}/{name}/20231007101619"
    shutil.rmtree(f"{OUT}/{name}", ignore_errors=True)
    shutil.copytree(SEG, root)
    ink = cv2.imread(INK, cv2.IMREAD_GRAYSCALE)
    cv2.imwrite(f"{root}/{LABEL}", cv2.resize(ink, (5700, 2740), interpolation=cv2.INTER_AREA))
    return root


sys.path.insert(0, "/w/tw/main/src")
from vesuvius.tifxyz import read_tifxyz

s = read_tifxyz(SEG)
s.use_full_resolution()
print(f"segment: PHercParis4 20231007101619 at 45.532 um, tifxyz grid {s._x.shape}, full resolution {s.shape}")
print(f"label: published 2.4 um ink prediction (ds8 JPEG, 13518 x 6505) resized to 5700 x 2740 -> {LABEL}")

for tree in ("main", "fix"):
    root = prepare(tree)
    before = listing(root)
    env = dict(os.environ, PYTHONPATH=f"/w/tw/{tree}/src")
    r = subprocess.run([sys.executable, "-c", CHILD, root], capture_output=True, text=True, env=env)
    after = listing(root)
    print(f"\n### {tree}: s = read_tifxyz(seg); write_tifxyz(seg, s, overwrite=True)")
    print(f"exit code {r.returncode}")
    for line in (r.stdout + r.stderr).strip().splitlines():
        print("  | " + line)
    print("files before -> after:")
    for n in sorted(set(before) | set(after)):
        b = f"{before[n][0]:>9} B {before[n][1]}" if n in before else "-"
        a = f"{after[n][0]:>9} B {after[n][1]}" if n in after else "MISSING"
        print(f"  {n:<26} {b:<26} -> {a}")
    kept = [n for n in before if n not in ("x.tif", "y.tif", "z.tif", "meta.json", "mask.tif")]
    lost = [n for n in kept if n not in after]
    changed = [n for n in kept if n in after and after[n] != before[n]]
    print(f"other files: {len(kept)} before, {len(kept) - len(lost)} after, {len(changed)} changed")
    print("leftover temp dirs:", [n for n in os.listdir(os.path.dirname(root)) if n.startswith(".tifxyz_tmp_")])

AI-assisted, human-directed.

[vercel]

@CVasilopoulos is attempting to deploy a commit to the scroll Team on Vercel.

A member of the Team first needs to authorize it.

[CVasilopoulos]

Marked ready for review. Re-checked on current main (e2254eb): it merges cleanly, including #2110's change to tifxyz/writer.py, and tests/tifxyz/ passes on the merged tree (48 passed, including the two new tests in test_writer_overwrite.py; Python 3.14, with imagecodecs installed for LZW).

AI-assisted, human-directed.