zarr-python's consolidated metadata implementation violates the spec

[d-v-b]

Previously, additional fields in metadata were allowed as long as they were JSON objects with a must_understand: false key: value pair. Zarr-python's consolidated metadata implementation complied with this requirement (see script below).

The recent redefinition of extra fields in metadata documents added the requirement that such extra fields have a name key which is a string. zarr-python's consolidated metadata does not contain a name key, and so it is out of spec.

As consolidated metadata is used heavily by xarray users, this is a very high-impact change. The recent spec refactor has thus made many (most?) zarr v3 xarray datasets technically out of spec.

# /// script
# dependencies = [
#   "zarr==3.1.0",
# ]
# ///

import zarr
from pprint import pprint
import json

store = {}
zarr.create_group(store)
consolidated = zarr.consolidate_metadata(store)
pprint(json.loads(store["zarr.json"].to_bytes()))
"""
{'attributes': {},
 'consolidated_metadata': {'kind': 'inline',
                           'metadata': {},
                           'must_understand': False},
 'node_type': 'group',
 'zarr_format': 3}
"""

I think we should treat this as a regression in the spec. A fix could be:

• clarify that readers may ignore any additional field that is a JSON object with a must_understand: false key value pair, no matter what other keys that object has.
• remove the requirement that top-level extra fields in array / group metadata objects have a name field if they are JSON objects.

Without these two changes, or changes that achieve the same effect, a large volume of zarr data is out of spec, and we need to fix that.

[d-v-b]

I'm working on something that could conceivably be an additional, must_understand: False array metadata field, so it would be really helpful to know if this is actually permitted by the spec!

[maxrjones]

I wanted a bit more context around this change but find it difficult to track conversations across the ZEPS/specs, so I had claude create this detailed context document - https://hackmd.io/8cBgorBeR4Sv5hrZ5g5NSw. This approach has issues (including overconfident assessments), but did help me get started thinking about this.

Unlike @d-v-b, I would not have interpreted the new name field requirement as applying to consolidated metadata. The relevant section in the context doc is copied below. tl;dr: it's ambiguous whether top-level keys with must_understand=False are extensions, which matters because only extensions need a name field. @joshmoore are you able to clarify the intent of the name field requirement? Is it meant to apply to top-level keys like consolidated metadata?

---

Relevant section from claude's document

A close reading of the v3.1 spec reveals that the name requirement may not actually apply to zarr-python's consolidated_metadata key. The "Extension definition" section is structured as follows:

1. It begins by listing the 5 extension points: data type, chunk grid, chunk key encoding, codecs, and storage transformers.
2. It then describes two ways to encode extensions within those extension points:
   • Objects: {"name": "<name>", "configuration": {...}} — e.g., inside the codecs array
   • Short-hand names: Bare strings like "gzip" — equivalent to an object with just a name key
3. The qualifier "If using an object definition" means "as opposed to using a short-hand name" — it distinguishes between these two encoding forms for extensions within extension points.

zarr-python's consolidated_metadata is not inside any of the 5 listed extension points. It is a top-level metadata key whose value happens to be a JSON object. Under this reading, it is not an "extension definition" in the spec's sense, and the name requirement would not apply to it.

The must_understand subsection does bridge to top-level keys, but only with:

"Use of must_understand=False to add top-level keys is discouraged in favor of the explicit use of extension-points."

This discourages top-level keys with must_understand: false but does not state that they require a name field. The ambiguity lies in whether the broader must_understand paragraph — which says implementations must fail on unrecognized "metadata fields" unless they have must_understand: false — implicitly brings all top-level keys under the extension definition rules (including the name requirement), or whether it preserves the V3.0 behavior where any JSON object with must_understand: false is valid regardless of its structure.

This interpretive question is central this issue: if the name requirement only applies within extension points, then consolidated_metadata may not actually violate the v3.1 spec — but the spec text is ambiguous enough that reasonable readers disagree.

[maxrjones]

zarrs also does not include a name for consolidated metadata (e.g., https://github.com/maxrjones/test-zarrs-consolidated-metadata/blob/main/output.zarr/zarr.json) which further motivates treating this as a regression or clarifying that the restriction does not apply to top-level keys.

[d-v-b]

I think this section unambiguously requires that inline consolidated metadata (an extra key in the metadata object) be treated as an extension:

All other keys found in the metadata object MUST be interpreted following the :ref:Extensions section <extensions_section>.

The extension definition clearly states that if an object is used, the name field must be a string:

If using an object definition, the member name MUST be a plain string which conforms to :ref:extension name <extension-naming>. Optionally, the member configuration MAY be present but if so MUST be an object.

Contra claude I don't see an ambiguity here. <extra field> : <object> key : value pairs in metadata objects MUST have a "name" : <string> key : value pair. consolidated metadata does not.

[maxrjones]

I think this section unambiguously requires that inline consolidated metadata (an extra key in the metadata object) be treated as an extension:

Thanks, I missed that section as did claude. Unfortunately I now agree that 3.1 unambiguously made inline consolidated metadata out of spec.

Do you know of other examples of previously compliant data that are now out of spec? I'm not sure if it'd ever be possible to reach a consensus on the future of #309 such that it would be a practical solution, but I'm still curious whether formalizing consolidated metadata would be a theoretical alternative solution.

[d-v-b]

I would guess the reason claude missed that section is because the heading "unknown" is inaccurate. Additional keys like consolidated_metadata are not unknown.

[d-v-b]

Do you know of other examples of previously compliant data that are now out of spec? I'm not sure if it'd ever be possible to reach a consensus on the future of #309 such that it would be a practical solution, but I'm still curious whether formalizing consolidated metadata would be a theoretical alternative solution.

consolidated metadata is the only one that comes to mind. And I don't think we want to bake consolidated metadata into the spec. It was spec compliant, and recent changes broke that. IMO we should do the backwards-compatible thing and restore the old language.

[maxrjones]

IMO we should do the backwards-compatible thing and restore the old language.

I agree. Thanks for talking through the counter arguments to help me get on the same page.

The quote "this PR with its intent of being nonbreaking" from #330 (comment) does make it sound like an accidental breakage, which will hopefully make it more straightforward to consider a regression and revert via #373.

[d-v-b]

ping @zarr-developers/steering-council

[rabernat]

I support reverting / updating the spec change which retroactively made the consolidated metadata out of spec.

[normanrz]

I am very sympathetic to adding consolidated metadata to the spec.

I don't think I agree with the interpretation that the earlier version of the spec allowed for consolidated metadata.

The array metadata object must not contain any other names.
Those are reserved for future versions of this specification.

This says that no additional fields are allowed, because they are reserved to enable backwards-compatible evolution of the spec itself.

The next sentence

An implementation must fail to open Zarr hierarchies, groups
or arrays with unknown metadata fields, with the exception of
objects with a "must_understand": false key-value pair.

does not refer to additional non-spec fields, but defines the behavior for implementations that don't understand fields that have been added in later versions of the spec.

[d-v-b]

I totally agree that the original v3 spec language was confusing re: extra fields. But I think many people read "X is not allowed... here's when X is allowed" as ultimately defining a condition when X was allowed. That's how we got inline consolidated metadata in the first place.