Use zarr v3 and updated ome-zarr-py

[will-moore]

Uses zarr v3 (via ome-zarr-py 0.12.0)

To test:

• Export:
  • $ omero zarr export Image:6001240
  • $ omero zarr masks Image:6001240
  • $ omero zarr export Image:9836998 --format 0.4
  • $ omero zarr masks Image:9836998 # Masks will also be zarr v2 (0.4)
• Import:
  • $ omero zarr import --target 1094 https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr

[will-moore]

Check above failing due to mypy suggests that zarr v2 is installed rather than zarr v3.

src/omero_zarr/raw_pixels.py:49: error: Module "zarr" has no attribute "Group"  [attr-defined]
src/omero_zarr/raw_pixels.py:49: error: Module "zarr" has no attribute "open_group"  [attr-defined]

But we are installing ome-zarr-py from ome/ome-zarr-py#413 which specifies "zarr>=v3.0.0",.
In the build logs above, I see several "Installed..." blocks, but none of them have any "zarr" libs: e.g.

Successfully installed attrs-25.3.0 bleach-6.2.0 cfgv-3.4.0 distlib-0.3.9 docutils-0.21.2 exceptiongroup-1.3.0 filelock-3.18.0 flake8-7.2.0 identify-2.6.12 iniconfig-2.1.0 mccabe-0.7.0 nodeenv-1.9.1 pep8-naming-0.15.1 platformdirs-4.3.8 pluggy-1.6.0 pre-commit-4.2.0 pycodestyle-2.13.0 pyflakes-3.3.2 pygments-2.19.1 pytest-8.4.1 pytest-django-4.11.1 pytest-mock-3.14.1 readme-renderer-36.0 restructuredtext-lint-1.4.0 restview-3.0.2 typing-extensions-4.14.0 virtualenv-20.31.2 webencodings-0.5.1

So I can't see where any version of zarr or ome-zarr-py is installed?!

I can try to add to "test-requires" deps, or even to .omeroci/py-setup but it seems strange that regular dependencies are ignored?

[will-moore]

Still seeing:

mypy.....................................................................Failed
- hook id: mypy
- exit code: 1

src/omero_zarr/raw_pixels.py:49: error: Module "zarr" has no attribute "Group"  [attr-defined]
src/omero_zarr/raw_pixels.py:49: error: Module "zarr" has no attribute "open_group"  [attr-defined]
src/omero_zarr/masks.py:39: error: Module "zarr" has no attribute "open_group"  [attr-defined]
src/omero_zarr/cli.py:29: error: Module "zarr" has no attribute "open_group"  [attr-defined]
Found 4 errors in 3 files (checked 9 source files)

I am now trying to specify that mypy checks depend on zarr >=3.0.8 in .pre-commit-config.yaml like this:

       - id: mypy
       # This is for checking tests/* (everything else has a stricter check
       # below), but since there's no include it's run on everything
+        additional_dependencies: [
+          zarr >=3.0.8,
+        ]

but I can't commit this change because I get:

[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/Users/wmoore/.cache/pre-commit/repo660mey3l/py_env-default/bin/python', '-mpip', 'install', '.', 'zarr >=3.0.8')
return code: 1
stdout:
    Processing /Users/wmoore/.cache/pre-commit/repo660mey3l
      Preparing metadata (setup.py): started
      Preparing metadata (setup.py): finished with status 'done'
stderr:
    ERROR: Could not find a version that satisfies the requirement zarr>=3.0.8 (from versions: 0.2.2, 0.2.3, 0.2.5, 0.2.6, 0.2.7, 0.3.0, 0.4.0, 1.0.0b3, 1.0.0b4, 1.0.0b6, 1.0.0, 1.1.0, 2.0.0a2, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.4.0, 2.5.0, 2.6.1, 2.7.0, 2.7.1, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.9.4, 2.9.5, 2.10.0, 2.10.1, 2.10.2, 2.10.3, 2.11.0a1, 2.11.0a2, 2.11.0, 2.11.1, 2.11.2, 2.11.3, 2.12.0a1, 2.12.0a2, 2.12.0, 2.13.0a1, 2.13.0a2, 2.13.0, 2.13.1, 2.13.2, 2.13.3, 2.13.6, 2.14.0, 2.14.1, 2.14.2, 2.15.0a1, 2.15.0a2, 2.15.0, 2.16.0, 2.16.1, 2.17.0, 2.17.1, 2.17.2, 2.18.0, 2.18.1, 2.18.2)
    ERROR: No matching distribution found for zarr>=3.0.8
Check the log at /Users/wmoore/.cache/pre-commit/pre-commit.log

This seems to be because the python version is not 3.11 which is required for zarr >3.08.
But I can't see where to specify the python version?

ChatGPT tells me: "Unfortunately, you cannot directly specify the Python version used in .pre-commit-config.yaml for a hook like mypy. Pre-commit hooks run in isolated environments (usually via virtualenv) that are based on the system Python or the Python interpreter pre-commit was installed with."
My local python environment where I'm trying to commit this change is py3.12.

As in #171, getting the build to pass is the hardest part of the PR! Any ideas?

[joshmoore]

I think when I've run into this, I've installed a pre-commit into a separate Python environment with the right version.

[will-moore]

I guess there's a few questions/issues I'm having:

• I don't see any version of zarr or ome-zarr-py being installed in the builds above before mypy is run. Is mypy run on the code before the omero-cli-zarr branch (and dependencies) is installed? The same is true for "green" builds, such as https://github.com/ome/omero-cli-zarr/actions/runs/14618374687/job/41011823388 (previous PR) - How is this passing if zarr 2.x isn't installed?
• If zarr IS already being installed before mypy is run, then the version of zarr should be specified by the setup.py and I would expect it to use zarr>=3.0.0 for this branch unless restricted by the python version used by omero-test-infra. I don't see anything failing, so either it's not being installed, or the failure to install it isn't showing up, or it IS being installed silently. How do I find out the version of python used by omero-test-infra, and can I control it? I can't see anything in the omero-test-infra docs.
• I am in a 3.12 python environment with pre-commit installed (is that what you mean), but that doesn't seem to be used by pre-commit:

(omeroweb_zarrv3) M-002366:omero-cli-zarr wmoore$ python --version
Python 3.12.9
(omeroweb_zarrv3) M-002366:omero-cli-zarr wmoore$ pip freeze | grep commit
pre_commit==4.2.0
(omeroweb_zarrv3) M-002366:omero-cli-zarr wmoore$ git commit -m "Add zarr to mypy additional_dependencies in pre-commit-config.yaml"
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/Users/wmoore/.cache/pre-commit/repo660mey3l/py_env-default/bin/python', '-mpip', 'install', '.', 'zarr >=3.0.8')
return code: 1
stdout:
    Processing /Users/wmoore/.cache/pre-commit/repo660mey3l
      Preparing metadata (setup.py): started
      Preparing metadata (setup.py): finished with status 'done'
stderr:
    ERROR: Ignored the following versions that require a different python version: 2.18.3 Requires-Python >=3.10; 2.18.4 Requires-Python >=3.11; 2.18.5 Requires-Python >=3.11; 2.18.6 Requires-Python >=3.11; 2.18.7 Requires-Python >=3.11; 3.0.0 Requires-Python >=3.11; 3.0.0a0 Requires-Python >=3.10; 3.0.0a1 Requires-Python >=3.10; 3.0.0a2 Requires-Python >=3.10; 3.0.0a3 Requires-Python >=3.10; 3.0.0a4 Requires-Python >=3.10; 3.0.0a5 Requires-Python >=3.10; 3.0.0a6 Requires-Python >=3.11; 3.0.0a7 Requires-Python >=3.11; 3.0.0b0 Requires-Python >=3.11; 3.0.0b1 Requires-Python >=3.11; 3.0.0b2 Requires-Python >=3.11; 3.0.0b3 Requires-Python >=3.11; 3.0.0rc1 Requires-Python >=3.11; 3.0.0rc2 Requires-Python >=3.11; 3.0.1 Requires-Python >=3.11; 3.0.2 Requires-Python >=3.11; 3.0.3 Requires-Python >=3.11; 3.0.4 Requires-Python >=3.11; 3.0.5 Requires-Python >=3.11; 3.0.6 Requires-Python >=3.11; 3.0.7 Requires-Python >=3.11; 3.0.8 Requires-Python >=3.11
    ERROR: Could not find a version that satisfies the requirement zarr>=3.0.8 (from versions: 0.2.2, 0.2.3, 0.2.5, 0.2.6, 0.2.7, 0.3.0, 0.4.0, 1.0.0b3, 1.0.0b4, 1.0.0b6, 1.0.0, 1.1.0, 2.0.0a2, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.4.0, 2.5.0, 2.6.1, 2.7.0, 2.7.1, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.9.4, 2.9.5, 2.10.0, 2.10.1, 2.10.2, 2.10.3, 2.11.0a1, 2.11.0a2, 2.11.0, 2.11.1, 2.11.2, 2.11.3, 2.12.0a1, 2.12.0a2, 2.12.0, 2.13.0a1, 2.13.0a2, 2.13.0, 2.13.1, 2.13.2, 2.13.3, 2.13.6, 2.14.0, 2.14.1, 2.14.2, 2.15.0a1, 2.15.0a2, 2.15.0, 2.16.0, 2.16.1, 2.17.0, 2.17.1, 2.17.2, 2.18.0, 2.18.1, 2.18.2)
    ERROR: No matching distribution found for zarr>=3.0.8
Check the log at /Users/wmoore/.cache/pre-commit/pre-commit.log

The error doesn't say what version of python it's using, but it rejects with Requires-Python >=3.10 suggesting it's using python 3.9.

[joshmoore]

$ cat .git/hooks/pre-commit | grep "INSTALL_PYTHON ="
INSTALL_PYTHON = '/usr/local/anaconda3/envs/z/bin/python'

$ which pre-commit
/Users/jamoore/.local/bin/pre-commit

$ head -n 1 $(which pre-commit)
#!/Users/jamoore/.local/pipx/venvs/pre-commit/bin/python

$ /Users/jamoore/.local/pipx/venvs/pre-commit/bin/python -V
Python 3.12.8

The Python used is specified in the .git/hook. If you have a new pre-commit, you can re-install the hook.

[will-moore]

Thanks @joshmoore - that answers the last of my 3 questions I think.

The jobs above are failing on the mypy check being run by omero-test-infra, so that's where we need to fix the python version.

How do I find out the version of python used by omero-test-infra, and can I control it?

[will-moore]

Updated to use the Release Candidate of ome-zarr-py 0.12rc1 but no change in build errors (no change was expected).

src/omero_zarr/raw_pixels.py:52: error: Module "zarr" has no attribute "Group"  [attr-defined]

etc.

Need to update omero-test-infra (I think)!?

[will-moore]

@jburel I'm not sure that I see any difference with the most recent build based on your branch at https://github.com/jburel/omero-test-infra/tree/python3.12. The errors look the same. Does my previous commit look correct?

I'm not sure of the best way to check if your branch is being used for the last build?
Both the last build and the one before appear to be using the same omero-server docker? e.g. this is found in both:

FROM docker.io/openmicroscopy/omero-server:5@sha256:f9a8c6049d3ea4176186

[will-moore]

Tried running locally:

git clone https://github.com/jburel/omero-test-infra/ .omero
cd .omero
git checkout origin/python3.12
cd ../
.omero/docker cli
[+] Building 6.9s (6/6) FINISHED                                                                                                                                     
 => [omeroclizarr_omero internal] load build definition from Dockerfile                                                                                         0.0s
 => => transferring dockerfile: 206B                                                                                                                            0.0s
 => [omeroclizarr_web internal] load build definition from Dockerfile                                                                                           0.0s
 => => transferring dockerfile: 216B                                                                                                                            0.0s
 => [omeroclizarr_omero internal] load .dockerignore                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                                 0.0s
 => [omeroclizarr_web internal] load .dockerignore                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                 0.0s
 => ERROR [omeroclizarr_omero internal] load metadata for docker.io/openmicroscopy/omero-server:5                                                               6.8s
 => CANCELED [omeroclizarr_web internal] load metadata for docker.io/openmicroscopy/omero-web-standalone:5                                                      6.7s
------
 > [omeroclizarr_omero internal] load metadata for docker.io/openmicroscopy/omero-server:5:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: no match for platform in manifest sha256:f9a8c6049d3ea41761865ebd61962fde2f69afbe57200c7287c4dd8b8ef005e3: not found
Warning: No resource found to remove for project "omeroclizarr".

I get the same error with openmicroscopy/omero-test-infra too (nothing to do with jburel's branch)

[will-moore]

Excluding to avoid conflicts while we focus on Zarr v2 and import workflow at #180

[EDIT] removing exclude 26th Sept...

[will-moore]

@dominikl - This is another option you could use to generate OME-Zarr v0.5 data for testing...

$  omero zarr export Image:6001240
$  omero zarr masks Image:6001240

[dominikl]

Thanks Will good idea!

[dominikl]

Exported an image to v0.5 zarr, trying to read it with zarr-java, got this error:

com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "consolidated_metadata" (class dev.zarr.zarrjava.v3.GroupMetadata), not marked as ignorable (3 known properties: "zarr_format", "node_type", "attributes"])

Indeed there's a consolidated_metadata:

...
  "zarr_format": 3,
  "consolidated_metadata": null,
  "node_type": "group"
...

Is that part of zarr v3 spec, in which case I should open an issue on zarr-java?

[will-moore]

Seems no consolidated_metadata in https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#group-metadata

But with zarr==3.1.1 and just:

>>> import zarr
>>> g = zarr.create_group("test")
>>> g.attrs["ome"] = {}

I get:

{
  "attributes": {
    "ome": {}
  },
  "zarr_format": 3,
  "consolidated_metadata": null,
  "node_type": "group"
}

[dominikl]

So this is a problem of the zarr python package? How about the latest version 3.1.3?

[will-moore]

Same behaviour in 3.1.3

[will-moore]

I opened an issue at zarr-developers/zarr-python#3521