[ChaseHendrick]

Draft only because I'm at villa's open PR limit; I'll mark it ready when a slot frees.

In one sentence: Adds volcodec_ng as a lossless zarr chunk codec in VcDataset and VC3D. On 12 held-out CT crops it made files 8.1 % smaller than vc-delta3d 0.2.0's LPC mode, and on one thread it decoded 1.18x and encoded 1.40x as fast, faster on every crop.

No quality loss: volcodec_ng is lossless only. Every voxel of every crop came back byte for byte.

Motivation:

vc-delta3d LPC is villa's best lossless codec on CT. When I opened this PR, volcodec_ng only beat it on write speed, and its files were about 30 % larger. I kept working on the coder until it beat LPC on size too, without giving up speed. I fixed the test before picking the crops: smaller files than LPC, and encode and decode each at least as fast as LPC, one thread, same machine, on crops nobody had tuned on.

One real example: A 256³ crop of PHerc0500P2 (open data volume 20250821110041, 0.55 µm, origin z 4096, y 4352, x 1536) is 16.8 MB raw. vc-delta3d LPC stores it in 4.16 MB and volcodec_ng in 3.90 MB. Both decode back identical to the source.

The same crop, slice by slice: the original, the slice after a volcodec_ng round trip, and their difference.

Before: For lossless chunks villa has vc-delta3d (delta by default, LPC since 0.2.0), volcomp at q 0, and the general byte compressors (Blosc, zstd, lz4, about 1.0 to 1.6x on CT). VcDataset reads vc-delta3d arrays; they're written by scripts/recompress_zarr.py or the Python package, and VC3D's chunk cache writes delta.

After this PR: A zarr v2 array with "compressor": {"id": "volcodec_ng"}, or a v3 array with the codec {"name": "volcodec_ng"} (usually inside sharding_indexed with 128³ inner chunks), opens, reads and writes through VcDataset, so VC3D and the tools built on it can use it. createZarrDataset(..., "volcodec_ng", ...) makes a new one (uint8, 128³ chunks). The streaming fetcher keeps its chunks verbatim in the disk cache as .volcodec_ng. volcomp and vc-delta3d are untouched.

Proof: 12 held-out 256³ crops from 10 public scans, 3 each at 0.55, 1.13, 2.4 and 8.64 to 9.36 µm. A fixed rule picked them after the codec was frozen, away from every crop used in development. Data: Vesuvius Challenge open data, CC BY-NC 4.0.

Lossless option	Size ratio, pooled	Encode MB/s	Decode MB/s
vc-delta3d 0.2.0, LPC	3.77x	84.9	284.4
volcodec_ng (this PR)	4.10x	119.1	334.4

Speeds are MB/s of raw voxels on one thread, pinned to one core of an AMD EPYC 7R13 (AVX2), gcc 13.3. Both codecs were timed in the same process, interleaved, best of 9 per crop. LPC timed twice agreed with itself to within 1 %. vc-delta3d was built from its v0.2.0 tag with its own CMake (Release), as villa fetches it. By resolution, volcodec_ng gets 4.26x, 6.23x, 4.32x and 2.88x at 0.55, 1.13, 2.4 and 8.64 to 9.36 µm, and LPC 3.99x, 5.54x, 3.98x and 2.65x.

Why / where this is useful: Anywhere villa stores uint8 CT losslessly. Recompressing or uploading volumes, writing volumes from C++ tools through VcDataset, and VC3D's chunk cache all get smaller files without slower reads, at least on x86-64 machines with AVX2.

• I personally verified that the example and proof above were produced by this PR on the stated data.

How it differs

• volcodec_ng predicts each voxel from up to 40 already decoded neighbours. They're chosen per 128³ chunk from 82 candidates in the current row, the three rows above and the two previous slices. The weights are fitted by least squares and stored in the chunk.
• It sorts voxels into 16 activity classes and corrects the prediction with a bias per class. It codes the residual with rANS, one static 12-bit table per class.
• It decodes in a wavefront: row (z, y) is rebuilt at step y + 4z, so up to 32 rows are rebuilt together as 16-bit SIMD lanes (AVX2 on x86-64, NEON on Arm, plain C elsewhere). Prediction is 16-bit fixed point, like LPC, and the decoder is integer only, so all three paths write and read the same bytes.
• LPC also fits a least-squares predictor. What it doesn't have is context modelling of the residual and the per-class bias correction, and in my development tests those made up most of the size gap.

How I tested it

• The coder in this branch is byte-identical to the one I timed: 712 checks, 0 failures, on 64 development chunks with GCC (AVX2 and plain C) and Clang, plus ASan/UBSan, and Arm NEON under qemu on synthetic chunks. The timing run checked a pinned golden stream on its own host before timing anything.
• The villa branch e1770003 on main 131d9ac (it also merges cleanly onto e2254eb): a minimal volume-cartographer build with GCC 13.3 on Ubuntu 24.04, and its 4 codec tests pass (test_vcdataset_volcodec_ng with 5 cases, plus volcomp, shard index and compressors). I haven't built the full villa tree, macOS or Windows.

Limits

• The speed lead is measured on x86-64 with AVX2. On an Apple M1 Pro (NEON), on development crops, decode was 0.82x LPC's speed and encode 1.11x, so on Apple silicon LPC still decodes faster.
• One thread only; multi-threaded throughput wasn't measured.
• The 0.55 and 1.13 µm crops come from scans that also had development crops (new positions only), so those groups test new parts of known scans.
• uint8 and 128³ chunks only, like the rest of volcodec_ng.
• The codec name stays volcodec_ng with no new config key. New writes use this coder; arrays written with the earlier version of this PR still read.

[vercel]

@ChaseHendrick is attempting to deploy a commit to the scroll Team on Vercel.

A member of the Team first needs to authorize it.

[jrudolph]

Thanks. Before your latest changes, I would have said that we settled on the 0.2.0 LPC solution. But you iterated fast ;) Adding context modelling would have been the next thing I tried as well. 5-10% more compression + decoding speed benefits might be worthwhile integrating. Can you open up a PR against vc_delta3d instead, we won't create yet another compression branding and the latest changes are really an incremental improvement on the LPC idea? I'll have a look on Monday.

[ChaseHendrick]

Moved to vc-delta3d as bruniss asked: ScrollPrize/vc-delta3d#6

It's a new ctx predictor there, so VC3D reads it through its existing vc-delta3d support and doesn't need the separate volcodec_ng codec from this PR. Closing this one.

[ChaseHendrick]

Thanks. Before your latest changes, I would have said that we settled on the 0.2.0 LPC solution. But you iterated fast ;) Adding context modelling would have been the next thing I tried as well. 5-10% more compression + decoding speed benefits might be worthwhile integrating. Can you open up a PR against vc_delta3d instead, we won't create yet another compression branding and the latest changes are really an incremental improvement on the LPC idea? I'll have a look on Monday.

Thanks! It's up now. Yeah, it's an incremental step on your LPC, so it's a 3rd predictor next to Delta and LPC. It integrates some context modeling work. Anything that isn’t lossless 128^3 uint8 also falls back to exactly the LPC output.