Zipstore in Icechunk

[MosGeo]

I am thinking about using icechunk instead of plain zarr. I really like the snapshot concept in it. My use case would be mostly local data. Would it be possible to implement a zipstore.

Problem

Currently, there is no single file format for icechunk. Zarr haz the zipstore. Working with localstore is not ideal for large datasets.

Proposed Solution

Implement icechunk zipstore. Other possible implementations is sqlite store but it looks like v3 has deprecated and removed the store.

[d0rianb]

Hello, I would also be interested in this feature.
Is there any plan to integrate it? If not, would it be possible to contribute by adding it?

[paraseba]

The way Icechunk achieves its consistency guarantees is through very careful use of modern object store operations. It's not possible to do this in a single file in object store. If we are thinking about a local filesystem solution, there are more powerful operations, going all the way to using a full database.

Icechunk main target is not local filesystem but very large object store repositories, but if somebody is interested in implementing this, we can discuss. It would be a very significant amount of work, and we should design very carefully so maintenance of the new code doesn't slow down other targets.

Initially, I'm skeptical about this path, but open to hearing more about it.

[ianhi]

Some use cases I can think of for local:

1. HPC compute (potentially multi-user so icechunk very useful, but not cloud storage)
2. Several local applications in bio (especially with high speed microscope writes)

[rabernat]

Let's separate the following two things as distinct use cases:

• Exporting an Icechunk snapshot to another Zarr store. This should be possible for any potential Zarr store target, including ZipStore (see Repo export #1356 ). Once a repo is exported, it loses its version history, transaction capability, etc.
• Storing a full Icechunk Repo on a local filesystem. This works today, minus some consistency checks. Storing a whole repo in a single file would be very infeasible (like trying to store a whole local Git repo in a single file). However, there are many easier things we could do to make IC work better on local files.