The xarray-compatible zarr output only contains last item

[j08lue]

#135 added an option --write_xarray_compatible_zarr to write explicit coordinate arrays along with the image transform / crs attributes (see also #66).

The intention was that this output should be compatible with that you get when not setting this flag, but also be readable with standard CF client libraries like Xarray.

However, the writer function currently uses xarray.DataSet.to_zarr(..., mode="w", group="chronic_heat/osc/v2"), i.e. the group is constant.

This causes to_zarr to delete the group every time a new array should be written into it, leaving at the end of subsequent writes to the same group only the last array.

You can reproduce this with the following command:

CEDA_USERNAME=<username> CEDA_PASSWORD=<password> os_climate_hazard degree_days_indicator --store ./uk_12km_dd_28 --source_dataset UKCP18 --source_dataset_kwargs "{'resolution':'60km','domain':'uk'}" --gcm_list "[ukcp18]" --scenario_list "[rcp85]" --threshold_list "[]" --threshold_temperature 28 --central_year_list "[2030,2040,2050,2060,2070,2080,2090]" --central_year_historical 2005 --window_years 20 --write_xarray_compatible_zarr true --dask_cluster_kwargs "{'n_workers': 1, 'threads_per_worker': 1}"

The expected behavior is the following structure:

/chronic_heat/osc/v2/
├── longitude
│   └── 0.0.0.0
├── latitude
│   └── 0.0.0.0
├── mean_degree_days_v2_above_28c_ukcp18_rcp85_2030
│   └── 0.0.0.0
├── mean_degree_days_v2_above_28c_ukcp18_rcp85_2040
│   └── 0.0.0.0
...
├── mean_degree_days_v2_above_28c_ukcp18_rcp85_2090
│   └── 0.0.0.0
└── spatial_ref

but only mean_degree_days_v2_above_28c_ukcp18_rcp85_2090 is present.

[joemoorhouse]

Hi @j08lue,
This is a problem of paths I think. The path of the HazardResource is meant to be the location of the (3 dimensional) Zarr array that is the hazard indicator data set. The path is also used by physrisk to get hold of the Zarr array. This means that in order to write a DataArray with co-ordinates and still follow that convention, the DataArray needs to be written a level higher.

In other words, actually the above was not my intent: the coordinates are meant to be under separate prefixes.

This means that we require the addition of an extra level into the path when store_netcdf_coords is moved from False to True. I could not think of another way to do this that keeps the rule that 'the path points to the Zarr array containing the data'.

In the mean degree days case, the fix is to change the path to something like:

path="chronic_heat/osc/v2/mean_degree_days_"
+ f"{above_below}"
+ "_index_{gcm}_{scenario}_{year}"
+ "/mean_degree_days"

When writing with store_netcdf_coords=True, the data array gets written to e.g.
/chronic_heat/osc/v2/mean_degree_days_v2_above_28c_ukcp18_rcp85_2030 and the Zarr array is in:
/chronic_heat/osc/v2/mean_degree_days_v2_above_28c_ukcp18_rcp85_2030/mean_degree_days

This assumes that the name of the data is mean_degree_days.

[joemoorhouse]

That was the original intent. But I think that perhaps this is confusing :)! Maybe there is another way we could go: the rule could be instead that the data is always written to path (no writing to parents or similar). So either the DataArray or the Zarr arrays get written to path depending on the mode and we define a convention for naming the Zarr array containing the hazard indicator data (e.g. data). We then change physrisk so that it will look in the child data folder instead if store_netcdf_coords=True.

This is actually back to how we previously we writing the XArrays in terms of path.

As I write, this seems an increasingly better option...