ability to re-set variable attributes - ODS2.6 compatibility

[Kate-Willett]

I want to be able to cmorise anomalies rather than actual values. In ODS2.6 it specifies 3 things in order to do this:

1. the variable name should have 'anom' appended, e.g., hussanom.
2. the 'long_name' variable attribute should contain the climatological reference period, e.g. 'near-surface (~2m) specific humidity monthly mean anomalies from the 1991-2020 climatological reference period'
3. the 'units_metadata' variable attribute has to be added to specify that the variable is a difference e.g., "units_metadata": "specific humidity: difference".

Item 1. is ok - this is now a variable in obs4mips_Amon.json.
Item 3. is ok - we can add >cmor.set_variable_attribute(varid, 'units_metadata', 'c', 'specific humidity: difference') within the script to cmorise the data.

Item 2. is not currently ok - script fails (and highlights some related issues). "long_name" is specified for each variable within the obs4MIPs_Amon.json which is set as the table_id within the cmorisation script. It is not possible to reset this using cmor.set_variable_attribute(..). I have tried - it results in an error in the logfile stating that this attribute must be set by cmor.variable(). However, the only way to do this is through the table_id. The "long_name" set for the variable in obs4MIPs_Amon.json is generic and rightly should not state a specific climatology period because the same variable name could be used by other products that use different climatology periods. So, in order for us to comply with ODS2.6 (and ensure that users know which climatology period was used) there needs to be a way to overwrite the 'long_name' variable attribute.

Relatedly, some of the variable attributes set within obs4MIPs_Amon.json are not correct for all products. For example, the 'cell_methods' attribute is set to "area: time: mean" for hussanom (and most/all variables?). For my insitu product (and likely others) this is not true - it should be "time: mean over station month, area: mean over gridbox". I create a time mean for each station first, and then a mean of all the station months for that gridbox. The order of calculation can make a difference.

Solution?

• Is there a way within cmor to overwrite variable attributes already set by the table id?
• If not, should those attributes that are likely to be different for different products be removed from the table_id (obs4MIPs_Amon.json and equivalent) tables? This would definitely be 'long_name' and 'cell_methods', and possibly 'cell_measures' and 'comment'. I see that 'valid_min' and 'valid_max' are specified as "" within the obs4MIPs_Amon.json table and are able to be reset using cmor.set_variable_attribute() within the cmorisation script. THIS OPTION IS MY PREFERENCE
• Failing the above, then we need another way to record the climatological reference period information and this needs to be changed in the ODS too? This could be done in the 'units_metadata' variable attribute which could then be set to 'specific humidity: difference from 1991-2020 climatological mean'.

[Kate-Willett]

I have just seen that the newest update to CMOR now allows long_name to be overwritten: PCMDI/cmor#891

Hurrah! I will update my version of cmor and hopefully all will now be well.

What about cell_methods though?

[gleckler1]

@Kate-Willett Hoping the CMOR update enables overriding long_name for this purpose.

cell_methods has not received the attention it deserves for obs4MIPs. Since the CMIP6 CMOR tables have been leveraged, the cell_methods entries for model output have been carried over and may not be consistent with observational variable equivalents. It would be good for the steering panel to take this up, with the CMIP7 guidance on grids offering a starting point. With the CMOR update can the cell_methods be overridden?

[Kate-Willett]

Thanks Peter. I've now got my script working nicely with cmor 3.13.2. This doesn't allow cell_methods to be overwritten. I may be able to engage / raise an issue on the PCMDI/cmor repo to open up discussion on that but we should probably discuss as obs4MIPs steering panel first. If this does happen then we can add a brief pointer to 'good' use of cell_methods in the ODS.

It did take a bit of jiggling to get cmor 3.13.2 to work as there were some dependencies between it, python_abi, netCDF4 and hdf5 that had to be worked out. This also picked up several cmor changes that require global attributes to be specified as objects or arrays - I had to edit the license and activity_id entries to make them arrays in the obs4MIPs_CV.json. I've done all this in my branch. It will be good to work with you and sort out what of that might want to be pulled through to master at some point.

The good news is that I think I'm making progress at least.

[Darwyn72]

@gleckler1 @Kate-Willett Might be worth putting something in the next obs4MIPs agenda to discuss with the rest of the Panel. By the way @Kate-Willett how did you update your copy of CMOR? Mine is on a local Mac and I can't seem to get it to update beyond 3.10

[Kate-Willett]

@Darwyn72 Good idea on steering panel discussion.

To get cmor to update I updated the obs4MIPs.yaml file that I was working with to point to 3.13.2 instead of 3.9.0. In order to get it to work / resolve conflicts I had to unspecify the versions for netcdf4 and python. I then deactivated my original environment and created my new one and it worked. I've copied my .yaml file below. Presumably the 'channels:' will be specific to your set up? Possibly, someone else in the Met Office did something to update the versions that we can access internally as there are many cmor users other than me. I have to admit to not being super aware of how these things work.

name: cmorobs4mips
channels:

• conda-mo-internal
• conda-forge

dependencies:
# Core

• python==*
• pip>=24.0
• setuptools>=71.0

# Analysis packages

• cartopy==0.23.0
• geopandas==1.0.1
• matplotlib==3.9.2
• netcdf4==*
• numpy<2 # for tensorflow compatibility
• pandas==2.2.3
• scipy==1.14.1
• xarray==2024.6.0
• xcdat==0.7.2
• cmor==3.13.2

# Dev packages

• pytest==8.3.3
• ipykernel==6.29.5

[taylor13]

If you run into problems installing/running cmor, you might raise an issue at https://github.com/PCMDI/cmor/issues . Hopefully, someone will have time to respond.

[gleckler1]

@Kate-Willett @Darwyn72 I am unfamiliar with this yaml. I have successfully installed the latest CMOR via:

conda create -n MYENVNAME -c conda-forge xarray cmor xcdat

however I am experiencing some strange behavior so I have just posted an issue as Karl suggested.

Kate, feel free to push your branch to this repo so that we can discuss any issues. It sounds like you may have manually modified obs4MIPs_CV.json which is routinely generated, so we'll have to make sure your changes can be included without being overwritten. But progress!!!! :-)

[Kate-Willett]

@gleckler1 I see that the obs4MIPs_CV.json is written by the writeJson.py, and that the info is stored in the individual obs4MIPs_xxx.json files within the master branch. In my branch I have made changes to the _Amon, _source_id, _license .json files, presuming that those are used to create the _CV.json and pull through the correct info when cmorising. The 'activity_id' entry in the obs4MIPs_CV.json will need to be changed which I think is line 682 in writeJson.py - this needs to be:
obs4MIPs_CV['CV']['activity_id'] = ['obs4MIPs']
instead of
obs4MIPs_CV['CV']['activity_id'] = 'obs4MIPs'

I've made a note of the changes I've made to the .json files, and that should all be tracked in the various commits too.

I'll push this up now so hopefully we can work it all through together.

[gleckler1]

@Kate-Willett @Darwyn72 @taylor13 Great progress. I'm still thinking about some of the things on your branch. Having noticed how json files are modified via writeJson.py may have made you think how this process could be improved. That is the plan, once it becomes clear how best to latch onto CMIP7. For now, what we have is a bit clunky when trying to introduce new capabilities, but is workable for the most part.

I don't understand the need for brackets in obs4MIPs_CV['CV']['activity_id'], but I presume that also is a new feature of CMOR and that there is good reason behind it.

The prototyping of site data has also been impacted by the CMOR update. In addition to the attributes you have already identified, I think a case can be made (at least for obs4MIPs) that it would be useful if we could modify the grid_label with cmor.set_variable_attribute(). Perhaps we should start a little document justifying of our recommendations for minor (hopefully they are) modifications to CMOR, and run these by the WIP. Like you I think this would be the smoothest path (there are other, clunky, options), and I'm hopeful that the CMOR developers might be able to address these.

[Kate-Willett]

@gleckler1 @Darwyn72 @taylor13 On the brackets, that was in response to an error message that said that the activity_id (and license) must be either object or array. I spotted that most things within the json files are specified as either dictionaries or lists but activity_id and license were just strings. Hence adding the brackets to turn it into a single element list - which fixed the error.

I have a bit of time over the next week or so but life will get very busy Feb/March so please do let me know anything specific I can do to help with this.

For the list of variable attributes that we would ideally like flexibility to overwrite I would like to add cell_methods because it is important to know how an obs dataset has been processed. cell_measures might be another one although I'm not entirely sure I understand what that one is. You have mentioned grid_label too. The cmor team were pretty quick to make the change for long_name so I'm reasonably optimistic this will be possible.

[gleckler1]

@Kate-Willett I'm ready to merge your branch. I will also modify the src/writeJson.py file so that both the activity_id and license entries in the obs4MIPs_CV.json are saved with square brackets every time the tables are updated. As we have discussed above, I will also draft an cmor issue making a case for expanding the overwriting use of cmor.set_variable_attribute().