Rust API: `Store::list_dir` does not differentiate between keys/prefixes

[LDeakin]

For example, I would expect the following to output "c/" and "zarr.json".

icechunk/icechunk/src/zarr.rs

Lines 2073 to 2075 in d2cced7

	let mut dir = store.list_dir("array/").await?.try_collect::<Vec<_>>().await?;
	dir.sort();
	assert_eq!(dir, vec!["c".to_string(), "zarr.json".to_string()]);

Also, consider outputting a struct holding keys/prefixes in separate Vecs instead.

[paraseba]

@LDeakin we are a bit limited by what other implementations of the Zarr store do. The current behavior reproduces what RemoteStore does today in zarr-python. In general it's not clear from the zarr spec what these stores should return for listing. Also the list methods don't provide a great API, forcing the stores to return always both metadata and chunk keys.

What are you trying to do? Can you use a higher level API like Repository?

[LDeakin]

In general it's not clear from the zarr spec what these stores should return for listing:

Relevant part of the spec for list_dir:

Output: set of keys and set of prefixes
For example, if a store contains the keys “a/b”, “a/c”, “a/d/e”, “a/f/g”, then list_dir("a/") would return keys “a/b” and “a/c” and prefixes “a/d/” and “a/f/”

The current behavior reproduces what RemoteStore does today in zarr-python

Fair enough. I am seeing the same thing with LocalStore too. I'd argue that zarr-python and icechunk are non-conformant, but the abstract store API section is not actually intended to be normative. So who cares? But in my view, list_dir is far less usable if it does not differentiate keys and prefixes (as the spec intended).

What are you trying to do?

Attempting to wrap the icechunk::Store for zarrs.

Can you use a higher level API like Repository?

Perhaps, but the API for icechunk::Store looked most similar to the zarrs storage traits.

[paraseba]

Makes sense @LDeakin . Would it help if we add a new method that has the behavior you want? Maybe we can wrap each individual result in an enum that differentiates between keys and prefixes. I wouldn't want to modify list_dir to maintain compatibility with what other Stores are doing.

[paraseba]

@LDeakin: How about #286 ?