Add C API and Julia bindings for Zarr Store#1722
Conversation
Create the icechunk-c workspace crate with Cargo.toml, cbindgen.toml, and a minimal lib.rs placeholder. Add workspace-level edition = "2024" setting and add icechunk-c to workspace members (not default-members). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add thread-local error string with return codes and icechunk_last_error() extern C function. Add global tokio runtime with block_on helper for bridging async operations to synchronous C API calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add IcechunkStorage opaque type with in-memory and local filesystem constructors, plus a free function. Both constructors handle errors from the underlying async storage creation. Includes tests for roundtrip creation/free, null path handling, and null-safe free. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add IcechunkStore opaque type with icechunk_store_open (creates a new repository and writable session on a branch), icechunk_store_free, and icechunk_store_is_read_only. The store struct keeps the Repository alive alongside the inner Store to prevent storage from being dropped. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the C FFI function for reading Zarr key values via malloc-allocated buffers. Properly classifies StoreErrorKind::NotFound as ICECHUNK_ERROR_NOT_FOUND. Includes a test for the not-found case. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the C FFI function for writing Zarr key values from caller-owned buffers using bytes::Bytes::copy_from_slice. Includes a set-then-get round-trip test that sets zarr.json metadata, writes a chunk, and reads it back to verify data integrity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add icechunk_store_exists (returns 1/0/negative) and icechunk_store_delete (returns ICECHUNK_SUCCESS/negative) FFI functions, with a test that sets metadata + chunk, verifies exists, deletes the chunk, and verifies not-exists. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add IcechunkStoreListIter type and icechunk_store_list, icechunk_store_list_prefix, icechunk_store_list_dir, icechunk_store_list_next, and icechunk_store_list_free functions for iterating over store keys via the C FFI. The iterator wraps a boxed async stream behind a std::sync::Mutex, with block_on used to drive iteration. Includes tests for empty store listing, populated store listing, prefix filtering, directory listing, and null argument safety. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add build.rs that uses cbindgen to auto-generate include/icechunk.h from the extern "C" functions. Fix cbindgen.toml export prefix to avoid double-prefixed type names (IcechunkIcechunk -> Icechunk). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Exercise the entire C API end-to-end from a plain C program: storage creation, store open, set metadata, set chunk, exists, get with roundtrip verify, list, delete, and free. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Julia package with: - LibIcechunk module: low-level ccall wrappers (JLL-style library path pattern) - Icechunk module: high-level Julia types with finalizers and Dict-like interface - IcechunkStorage/IcechunkStore types with automatic resource cleanup - Dict interface: getindex, setindex!, haskey, delete!, keys, get - 27 passing tests covering storage, store, CRUD, listing, filesystem Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New ffi.rs module with parse_c_str, alloc_c_bytes, alloc_c_string - Deduplicate CStr parsing (8 call sites -> helper calls) - Extract wrap_list_stream for list/list_prefix/list_dir - Extract collect_iter_keys test helper - Use i32::from(bool) for boolean-to-int conversions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace unwrap()/expect() calls with proper error handling to satisfy workspace clippy lints (unwrap_used, expect_used) run with -D warnings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Hi @asinghvi17 - apologies for the long silence on this PR. This PR generated a lot of excitement and discussion from our team. We deeply want to support the Julia community, and having C bindings unlocks some much more than just that. Like you, we are also very excited about the potential of AI tools to accelerate development. However, as you know, the rapid growth of agents poses complex and nuanced questions for the open source community. There are many benefits AND also many concerns. Rather than just looking at this PR in isolation, we decided it was best for us to actually come up with a policy. Here it is: https://icechunk.io/en/latest/ai-policy/ You can read the full policy, but the core principle is pretty straighforward:
Your description of how this PR was created:
...does not create the impression that you fully understand what is happening here. Therefore, the burden is shifted to our team in terms of reviewing and understanding these 2000 lines of code. We have an extremely high quality bar for this project. We're committed to building something that organizations like NASA can depend on for critical, production work. We stand behind every line of code and every paragraph of the docs. Accepting this PR in its current form wouldn't allow us to maintain that same standard. (Not because the code is bad! Just because we can't actively affirm that it's good.) However, as mentioned above, we really like what this PR unlocks, and we want to support it. There are basically two paths going forward.
Doing 2 now does not preclude doing 1 later. If we choose to go with 2, we would be happy to host the project via a standalone repo as part of the Earthmover org and bless this as an "official" Icechunk project. We're very grateful for your work and we want to collaborate. 🙏 Hopefully our position makes sense to you. |
2 participants
Hey! A while ago I had opened #381 to ask for a C compatible API. I threw Claude at it now and this is what it generated. I'm happy to split the Julia port out into a separate PR if you feel that makes things easier.
The idea here is that there is an
icechunk-ccrate which exposes a C API for icechunk, which is then wrapped by theicechunk-jlpackage. In practice, if this is merged we will have anIcechunk_jllin Yggdrasil, which is Julia's binary building infrastructure. AnIcechunk.jlwould depend onIcechunk_jllinstead, though we could override that locally for CI purposes in this repo.I had this implemented in a blocking way for the C API since that seemed simplest, but happy to pursue another path if that's what folks here would want.
🤖 Generated with Claude Code