Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
b0d6b3d
Basic read example working (no labels)
will-moore Oct 26, 2024
da8c32f
cli_tests passing
will-moore Oct 30, 2024
19b89a8
Passing all 6 test_io.py
will-moore Oct 31, 2024
a954161
Passing tests/test_io.py and test_node.py
will-moore Nov 4, 2024
80f6e01
Include dtype in group.create_array()
will-moore Nov 4, 2024
e568911
Uncomment labels spec. Fixes test_ome_zarr.py download
will-moore Nov 4, 2024
b49ecc8
Fix test_scaler
will-moore Nov 4, 2024
18abe02
Add dimension_separator to existing v2 data .zarray to fix test_upgra…
will-moore Nov 4, 2024
86142c3
Fixed test_write_image_dask
will-moore Nov 4, 2024
31584bf
Pin zarr==v3.0.0-beta.1
will-moore Nov 7, 2024
daa3546
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 7, 2024
fa29ccc
Remove python 3.9 and 3.10 from build.yml
will-moore Nov 7, 2024
8fc02b4
Remove unused imports
will-moore Nov 7, 2024
29890b8
remove fsspec from .isort.cfg
will-moore Nov 7, 2024
35bc979
mypy fix
will-moore Nov 7, 2024
75ba690
Use Blosc compression by default
will-moore Nov 11, 2024
52aceb0
Black formatting fixes
will-moore Nov 11, 2024
55d4ba9
Use group.array_values() for iterating arrays
will-moore Nov 11, 2024
0ea21bc
Use zarr_format=2 for zarr.open() in test_writer.py
will-moore Nov 11, 2024
7fc113b
Fix return type RemoteStore | LocalStore
will-moore Nov 12, 2024
94f7ace
Support reading of Zarr v3 data
will-moore Nov 12, 2024
d140c6d
Hard-code zarr_version=2 in parse_url()
will-moore Nov 12, 2024
f7b5f98
Use read_only instead of mode when creating Stores
will-moore Nov 13, 2024
c527c77
Pin zarr-python to specific commit on main branch
will-moore Nov 13, 2024
d8d5378
Fix test_write_image_compressed
will-moore Nov 13, 2024
2138160
Support READING of zarr v3 data
will-moore Dec 9, 2024
af2648d
Merge remote-tracking branch 'origin/master' into zarr_v3. Use zarr v…
will-moore Dec 9, 2024
1ea9e1a
Check that PR is green IF we skip test_writer with 3D-scale-True-from…
will-moore Dec 9, 2024
7754774
Bump dependencies including zarr==v3.0.0-beta.3 in docs/requirements.txt
will-moore Dec 9, 2024
499531f
Specify python 3.12 in .readthedocs.yml
will-moore Dec 9, 2024
c0fe50d
Merge remote-tracking branch 'origin/master' into zarr_v3
will-moore Dec 17, 2024
e021c13
Merge remote-tracking branch 'origin/master' into zarr_v3
joshmoore Dec 18, 2024
0a8d0b4
test fixes
will-moore Jan 13, 2025
c953723
Merge remote-tracking branch 'gh/zarr_v3' into zarr_v3
will-moore Jan 14, 2025
4f2a4b1
Merge remote-tracking branch 'origin/master' into zarr_v3
will-moore Jan 14, 2025
50e43c1
Rename zarr.storage.RemoteStore to FsspecStore
will-moore Jan 14, 2025
6c4ba92
_blosc_compressor() helper and other zarr-python fixes
will-moore Jan 14, 2025
872ce11
Use zarr_format=2 for download dask.to_zarr()
will-moore Jan 16, 2025
ebea6a4
don't pass storage_options to da.to_zarr()
will-moore Mar 14, 2025
7c13096
Allow extra .zattrs in test_astronaut_download
will-moore Mar 14, 2025
34d8636
Merge remote-tracking branch 'origin/master' into zarr_v3
will-moore Mar 14, 2025
a3bb7bd
Handle ZarrLocation(localstore)
will-moore Apr 7, 2025
69add1d
Add deprecation warning for get_json()
will-moore Apr 7, 2025
80b9407
Cleanup comments
will-moore Apr 7, 2025
1380c07
Remove skip - tests now passing
will-moore Apr 7, 2025
1459a76
Add print to debug test fails
will-moore Apr 7, 2025
25507ee
comp.get_config() expects 'typesize': None
will-moore Apr 7, 2025
c6abf3a
Docs use v0.5 example for ome_zarr info
will-moore May 23, 2025
82bb977
Merge remote-tracking branch 'origin/master' into zarr_v3
will-moore May 23, 2025
30f6c2f
Fix creation of test plate for test_finder()
will-moore May 23, 2025
80ecc2b
ome_zarr finder handles Zarr v3 data
will-moore May 23, 2025
cd025ed
Fix test expected compressor config
will-moore May 23, 2025
7d15112
enforce fmt specified in parse_url()
will-moore May 26, 2025
1c3bc46
Don't check version in parse_url()
will-moore May 27, 2025
5d6c4d3
Check version isn't v0.5 for all write methods
will-moore May 27, 2025
f286f5a
Update tests
will-moore May 27, 2025
19f2f62
Fix parse_url(p, mode='w') overwriting data
will-moore May 28, 2025
d428c68
Remove try/except in parse_url()
will-moore May 28, 2025
a813ae2
Revert parse_url(..fmt=FormatV04()) not needed
will-moore May 28, 2025
bd86dc4
Check plain json written is v2 zarr
will-moore May 28, 2025
b668b78
Update docs python examples
will-moore May 28, 2025
d595cdb
Add test_read_v05()
will-moore May 29, 2025
32524b7
clarify rc1 version in docs
will-moore May 29, 2025
e63990a
parse_url() should't try to open an array
will-moore May 29, 2025
142b30b
Specify fmt at parse_url() instead of at write_image() etc
will-moore Jun 11, 2025
6dacb33
Update docs examples
will-moore Jun 11, 2025
aa1d2e4
Support writing OME-Zarr v0.5
will-moore Jun 11, 2025
a0f238a
Use FormatV05 in more test and fix da.to_zarr() writing v3
will-moore Jun 12, 2025
e4fc856
Remove unused get_json() to improve code coverage
will-moore Jun 12, 2025
bbcc87e
Remove unreachable PlateLabels class
will-moore Jun 12, 2025
8ee1d6e
revert change to untested __check_store() in scale()
will-moore Jun 12, 2025
2f3277d
Increase test code converage of utils.py
will-moore Jun 12, 2025
24f1ef4
Fix v0.5 write_plate_metadata() and add v0.5 to tests
will-moore Jun 13, 2025
838c366
Fix v0.5 write_well_metadata() and add v0.5 to tests
will-moore Jun 13, 2025
4b074d8
Fix write_label_metadata() for v0.5. Add tests
will-moore Jun 13, 2025
d954ae4
test_two_label_images with v05. add_metadata() util
will-moore Jun 17, 2025
3a5eee8
Update docs example code
will-moore Jun 17, 2025
9c9249c
Add --version support to ome_zarr create
will-moore Jun 17, 2025
8152817
download cli supports v0.5. Added tests
will-moore Jun 18, 2025
3f4da90
Update docs. Fix well and plate metadata
will-moore Jun 18, 2025
58989a7
Update other docs/python examples. Fix dimension_names in write_image()
will-moore Jun 18, 2025
584216d
Fix test_cli.py on Windows
will-moore Jun 18, 2025
65acce2
Rename _check_format() to check_format()
will-moore Jun 23, 2025
3897e0c
Remove unused line from docs python.rst code
will-moore Jul 15, 2025
2df5e63
Add tests for compressor options
will-moore Jul 17, 2025
84e1fbc
Don't assert 'codecs' bytes endian
will-moore Jul 17, 2025
8e709fe
Update python.rst re v0.5 support info
will-moore Jul 22, 2025
f277c32
Typo fix
will-moore Jul 22, 2025
1cf5273
Fix pytest.skip() message: compressor fails with dask
will-moore Jul 23, 2025
788ee4e
Remove 'compressor': None from storage_options in test
will-moore Jul 23, 2025
1a5b773
Rename --version to --format
will-moore Aug 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .isort.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[settings]
known_third_party = dask,fsspec,numcodecs,numpy,pytest,scipy,skimage,zarr
known_third_party = dask,numcodecs,numpy,pytest,scipy,skimage,zarr
multi_line_output = 3
include_trailing_comma = True
force_grid_wrap = 0
Expand Down
2 changes: 1 addition & 1 deletion .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ version: 2
build:
os: ubuntu-22.04
tools:
python: "3.10"
python: "3.12"

# Build documentation in the docs/ directory with Sphinx
sphinx:
Expand Down
2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ ome-zarr-py

Tools for multi-resolution images stored in Zarr filesets, according to the `OME NGFF spec`_.

See `Readthedocs <https://ome-zarr.readthedocs.io/>`_ for usage information.
See `Documentation <https://ome-zarr.readthedocs.io/>`_ for usage information.

Documentation
-------------
Expand Down
4 changes: 2 additions & 2 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
sphinx==7.1.2
sphinx==8.1.3
sphinx-rtd-theme==3.0.2
fsspec
zarr
zarr>=v3.0.0
dask
numpy
scipy
Expand Down
16 changes: 9 additions & 7 deletions docs/source/cli.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ Use the `ome_zarr` command to interrogate Zarr datasets.

Remote data::

ome_zarr info https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr/
ome_zarr info https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr/

Local data::

ome_zarr info 6001240.zarr/
ome_zarr info 6001240_labels.zarr/

view
====
Expand All @@ -47,25 +47,27 @@ download

To download all the resolutions and metadata for an image use ``ome_zarr download``. This creates ``6001240.zarr`` locally::

ome_zarr download https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr/
ome_zarr download https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr

Specify a different output directory::

ome_zarr download https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr/ --output image_dir
ome_zarr download https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr --output image_dir

create
======

To create sample OME-Zarr image from the `skimage <https://scikit-image.org/docs/stable/api/skimage.data.html>`_
data.

Create an OME-Zarr image in coinsdata/ dir::
Create an OME-Zarr image in coinsdata/ dir using 'coins' method in OME-Zarr latest version or v0.4::

ome_zarr create coinsdata
ome_zarr create coinsdata.zarr

ome_zarr create coinsdata.zarr --format 0.4

Create an rgb image from skimage astronaut in testimage dir::

ome_zarr create testimage --method=astronaut
ome_zarr create testimage.zarr --method=astronaut

csv to labels
=============
Expand Down
2 changes: 2 additions & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ ome-zarr-py

Tools for reading and writing multi-resolution images stored in Zarr filesets, according to the `OME NGFF spec`_.

NB: The default version of OME-Zarr written by ``ome-zarr-py`` is ``v0.5``, which uses ``zarr v3``. OME-Zarr v0.5
is not yet supported by all OME-Zarr tools. See the documentation for more information on how to write other versions.

Features
--------
Expand Down
88 changes: 57 additions & 31 deletions docs/source/python.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,18 @@ of 2 in the X and Y dimensions.
Alternatively, the :py:func:`ome_zarr.writer.write_multiscale` can be used, which takes a
"pyramid" of pre-computed `numpy` arrays.

The default version of OME-NGFF is v0.5, is based on Zarr v3. A zarr v3 store is created
by `parse_url()` below. To write OME-NGFF v0.4 (Zarr v2), use the `fmt=FormatV04()` argument
in `parse_url()`, which will create a Zarr v2 store.

The following code creates a 3D Image in OME-Zarr::

import numpy as np
import zarr

from ome_zarr.io import parse_url
from ome_zarr.writer import write_image
from ome_zarr.format import FormatV04
from ome_zarr.writer import write_image, add_metadata

path = "test_ngff_image.zarr"

Expand All @@ -28,10 +33,11 @@ The following code creates a 3D Image in OME-Zarr::
rng = np.random.default_rng(0)
data = rng.poisson(lam=10, size=(size_z, size_xy, size_xy)).astype(np.uint8)

# write the image data
# Use fmt=FormatV04() to write v0.4 format (zarr v2)
store = parse_url(path, mode="w").store
root = zarr.group(store=store)
write_image(image=data, group=root, axes="zyx", storage_options=dict(chunks=(1, size_xy, size_xy)))
write_image(image=data, group=root, axes="zyx",
storage_options=dict(chunks=(1, size_xy, size_xy)))


This image can be viewed in `napari` using the
Expand All @@ -41,18 +47,18 @@ This image can be viewed in `napari` using the

Rendering settings
------------------
Render settings can be added to an existing zarr group::
Rendering settings can be added to an existing zarr group::

store = parse_url(path, mode="w").store
root = zarr.group(store=store)
root.attrs["omero"] = {
add_metadata(root, {"omero": {
"channels": [{
"color": "00FFFF",
"window": {"start": 0, "end": 20, "min": 0, "max": 255},
"label": "random",
"active": True,
}]
}
}})

Writing labels
--------------
Expand All @@ -64,10 +70,11 @@ The following code creates a 3D Image in OME-Zarr with labels::
import os

from skimage.data import binary_blobs
from ome_zarr.format import FormatV04
from ome_zarr.io import parse_url
from ome_zarr.writer import write_image
from ome_zarr.writer import write_image, add_metadata

path = "test_ngff_image.zarr"
path = "test_ngff_image_labels.zarr"
os.mkdir(path)

mean_val=10
Expand All @@ -76,19 +83,20 @@ The following code creates a 3D Image in OME-Zarr with labels::
rng = np.random.default_rng(0)
data = rng.poisson(mean_val, size=(size_z, size_xy, size_xy)).astype(np.uint8)

# write the image data
# Use fmt=FormatV04() to write v0.4 format (zarr v2)
store = parse_url(path, mode="w").store
root = zarr.group(store=store)
write_image(image=data, group=root, axes="zyx", storage_options=dict(chunks=(1, size_xy, size_xy)))
write_image(image=data, group=root, axes="zyx",
storage_options=dict(chunks=(1, size_xy, size_xy)))
# optional rendering settings
root.attrs["omero"] = {
add_metadata(root, {"omero": {
"channels": [{
"color": "00FFFF",
"window": {"start": 0, "end": 20, "min": 0, "max": 255},
"label": "random",
"active": True,
}]
}
}})


# add labels...
Expand All @@ -104,18 +112,19 @@ The following code creates a 3D Image in OME-Zarr with labels::
labels_grp = root.create_group("labels")
# the 'labels' .zattrs lists the named labels data
label_name = "blobs"
labels_grp.attrs["labels"] = [label_name]
add_metadata(labels_grp, {"labels": [label_name]})
label_grp = labels_grp.create_group(label_name)
# need 'image-label' attr to be recognized as label
label_grp.attrs["image-label"] = {
write_image(label, label_grp, axes="zyx")

# we need 'image-label' attr to be recognized as label
add_metadata(label_grp, {"image-label": {
"colors": [
{"label-value": 1, "rgba": [255, 0, 0, 255]},
{"label-value": 2, "rgba": [0, 255, 0, 255]},
{"label-value": 3, "rgba": [255, 255, 0, 255]}
]
}
}})

write_image(label, label_grp, axes="zyx")

Writing HCS datasets to OME-NGFF
--------------------------------
Expand All @@ -125,6 +134,7 @@ This sample code shows how to write a high-content screening dataset (i.e. cultu
import numpy as np
import zarr

from ome_zarr.format import FormatV04
from ome_zarr.io import parse_url
from ome_zarr.writer import write_image, write_plate_metadata, write_well_metadata

Expand All @@ -144,6 +154,7 @@ This sample code shows how to write a high-content screening dataset (i.e. cultu
data = rng.poisson(mean_val, size=(num_wells, num_fields, size_z, size_xy, size_xy)).astype(np.uint8)

# write the plate of images and corresponding metadata
# Use fmt=FormatV04() in parse_url() to write v0.4 format (zarr v2)
store = parse_url(path, mode="w").store
root = zarr.group(store=store)
write_plate_metadata(root, row_names, col_names, well_paths)
Expand All @@ -154,7 +165,8 @@ This sample code shows how to write a high-content screening dataset (i.e. cultu
write_well_metadata(well_group, field_paths)
for fi, field in enumerate(field_paths):
image_group = well_group.require_group(str(field))
write_image(image=data[wi, fi], group=image_group, axes="zyx", storage_options=dict(chunks=(1, size_xy, size_xy)))
write_image(image=data[wi, fi], group=image_group, axes="zyx",
storage_options=dict(chunks=(1, size_xy, size_xy)))


This image can be viewed in `napari` using the
Expand All @@ -177,11 +189,9 @@ the data is available as `dask` arrays::
from ome_zarr.reader import Reader
import napari

url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr"
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr"

# read the image data
store = parse_url(url, mode="r").store

reader = Reader(parse_url(url))
# nodes may include images, labels etc
nodes = list(reader())
Expand All @@ -207,26 +217,32 @@ Writing big image from tiles::
import os
import zarr
from ome_zarr.io import parse_url
from ome_zarr.format import CurrentFormat, FormatV04
from ome_zarr.reader import Reader
from ome_zarr.writer import write_multiscales_metadata
from ome_zarr.dask_utils import resize as da_resize
import numpy as np
import dask.array as da
from math import ceil

fmt = CurrentFormat()
# Use fmt=FormatV04() to write v0.4 format (zarr v2)

url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/9836842.zarr"
reader = Reader(parse_url(url))
nodes = list(reader())
# first level of the pyramid
dask_data = nodes[0].data[0]
tile_size = 512
axes = [{"name": "c", "type": "channel"}, {"name": "y", "type": "space"}, {"name": "x", "type": "space"}]

def downsample_pyramid_on_disk(parent, paths):
"""
Takes a high-resolution Zarr array at paths[0] in the zarr group
and down-samples it by a factor of 2 for each of the other paths
"""
group_path = parent.store.path
group_path = str(parent.store_path)
img_path = parent.store_path / parent.path
image_path = os.path.join(group_path, parent.path)
print("downsample_pyramid_on_disk", image_path)
for count, path in enumerate(paths[1:]):
Expand All @@ -246,10 +262,16 @@ Writing big image from tiles::
dask_image, tuple(dims), preserve_range=True, anti_aliasing=False
)

options = {}
if fmt.zarr_format == 2:
options["dimension_separator"] = "/"
else:
options["chunk_key_encoding"] = fmt.chunk_key_encoding
options["dimension_names"] = [axis["name"] for axis in axes]
# write to disk
da.to_zarr(
arr=output, url=image_path, component=path,
dimension_separator=parent._store._dimension_separator,
arr=output, url=img_path, component=path,
zarr_format=fmt.zarr_format, **options
)
return paths

Expand All @@ -270,16 +292,18 @@ Writing big image from tiles::
row_count = ceil(shape[-2]/tile_size)
col_count = ceil(shape[-1]/tile_size)

store = parse_url("9836842.zarr", mode="w").store
store = parse_url("9836842.zarr", mode="w", fmt=fmt).store
root = zarr.group(store=store)

# create empty array at root of pyramid
zarray = root.require_dataset(
zarray = root.require_array(
"0",
shape=shape,
exact=True,
chunks=chunks,
dtype=d_type,
chunk_key_encoding=fmt.chunk_key_encoding,
dimension_names=[axis["name"] for axis in axes], # omit for v0.4
)

print("row_count", row_count, "col_count", col_count)
Expand All @@ -296,7 +320,6 @@ Writing big image from tiles::
zarray[ch_index, y1:y2, x1:x2] = tile

paths = ["0", "1", "2"]
axes = [{"name": "c", "type": "channel"}, {"name": "y", "type": "space"}, {"name": "x", "type": "space"}]

# We have "0" array. This downsamples (in X and Y dims only) to create "1" and "2"
downsample_pyramid_on_disk(root, paths)
Expand All @@ -313,7 +336,8 @@ Writing big image from tiles::
write_multiscales_metadata(root, datasets, axes=axes)


Using dask to fetch::
Using dask to fetch. Here concatenate lazy "delayed" source of tiles into a full image.
When that dask data is passed to write_image() the tiles will be loaded on the fly::

# Created for https://forum.image.sc/t/writing-tile-wise-ome-zarr-with-pyramid-size/85063

Expand All @@ -323,9 +347,11 @@ Using dask to fetch::
from dask import delayed

from ome_zarr.io import parse_url
from ome_zarr.writer import write_image, write_multiscales_metadata
from ome_zarr.format import FormatV04
from ome_zarr.writer import write_image, add_metadata

zarr_name = "test_dask.zarr"
# Use fmt=FormatV04() in parse_url() to write v0.4 format (zarr v2)
store = parse_url(zarr_name, mode="w").store
root = zarr.group(store=store)

Expand Down Expand Up @@ -374,7 +400,7 @@ Using dask to fetch::
# This will create a downsampled 'multiscales' pyramid
write_image(dask_data, root, axes="czyx")

root.attrs["omero"] = {
add_metadata(root, {"omero": {
"channels": [
{
"color": "FF0000",
Expand All @@ -389,7 +415,7 @@ Using dask to fetch::
"active": True,
},
]
}
}})

print("Created image. Open with...")
print(f"ome_zarr view {zarr_name}")
Loading
Loading