Description, verification, and handling of parameter file metadata

[ekluzek]

We had some discussion about understanding the parameters with multiple dimensions on the parameter file. The metadata is constructed to follow the CF conventions.

https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html

But, not everyone is aware of those conventions and how they work.

The extra dimensions on the file include these for one of the latest files:

netcdf ctsm60_ciso_cwd_hr_params.c250311 {
dimensions:
        pft = 79 ;
        ndecomp_pools_max = 8 ;
        segment = 4 ;
        variants = 2 ;
        string_length = 40 ;
        ntill_stages_max = 3 ;
        ntill_intensities_max = 2 ;
variables:

The "coordinates" attribute for each variable says what coordinate variable describes a given dimension. For example, pftname is the coordinate variable for the "pft" dimension. For multidimensional data they are listed in the "C" order they are in the file with space as delimiter.

There is also a variantnames coordinate variable for the "variant" dimension.

        char variantnames(variants, string_length) ;
                variantnames:_FillValue = "" ;
                variantnames:long_name = "Description of variant names" ;
                variantnames:units = "unitless" ;
.
.
.
 variantnames =
  "water                                   ",
  "carbon                                  " ;
}

So we need coordinate variables for the dimensions: segment, ndecomp_pools_max, ntill_stages_max, and ntill_intensities_max. The "string_length" dimension doesn't need a coordinate since it's just the length of the strings in the metadata. Although I suppose a coordinate variable could be added that just says that.

The following variables are without the "coordinate" attribute and should have it added:

	double bgc_initial_Cstocks(ndecomp_pools_max) ;
	double mimics_fmet(segment) ;
	double mimics_initial_Cstocks(ndecomp_pools_max) ;
	double mimics_kint(ndecomp_pools_max) ;
	double mimics_kmod(ndecomp_pools_max) ;
	double mimics_kslope(ndecomp_pools_max) ;
	double mimics_mge(ndecomp_pools_max) ;
	double mimics_vint(ndecomp_pools_max) ;
	double mimics_vmod(ndecomp_pools_max) ;
	double mimics_vslope(ndecomp_pools_max) ;
	double bgc_till_decompk_multipliers(ntill_stages_max, ndecomp_pools_max, ntill_intensities_max) ;
	double mimics_till_decompk_multipliers(ntill_stages_max, ndecomp_pools_max, ntill_intensities_max) ;

Another CF convention that is used is for variables that are "flags" as they are either logical variables or mean one of a short list of options. For example, for c3psn you have:

        double c3psn(pft) ;
                c3psn:_FillValue = NaN ;
                c3psn:long_name = "Photosynthetic pathway" ;
                c3psn:units = "flag" ;
                c3psn:valid_range = 0., 1. ;
                c3psn:flag_meanings = "C4 C3" ;
                c3psn:flag_values = 0., 1. ;
                c3psn:coordinates = "pftname" ;

The last set of CF attributes used for some variables is: comment, and valid_range.

Definition of done:

• Release spreadsheet with cesm3.0
• Decide what things should be done here and what a timeline might be
• Add a note on this to the User's Guide?
• Add coordinate variable with the names to all of the list of variables with dimensions (with same name as the dimension)
• Add a coordinate variable for the rest of the dimensions
• Add an attribute for each variable that gives the general process the variable applies to from the spreadsheet
• Add a comment to each variable that only applies when a given parameterization is on, that says what needs to be turned on for it to be used
• Add a checker that CF conventions are followed? (there are some available to use)
• Add our own such checker?
• Add a tool that can query a case or compset and tell you which parameters are active
• Add some checking to query_paramdata for these kind of things?
• Have the code read the metadata and either use it internally, or abort if it isn't what's assumed in the code.

[samsrabin]

One thing that contributes to confusion, as you mention, here is the fact that we have dimensions such as variants with no coordinates. Variables with such dimensions do have coordinates, but they're different. E.g.:

double mimics_desorp(variants) ;
	mimics_desorp:coordinates = "variantnames" ;

When working with a paramfile in xarray and you look at paramfile_dataset["variants"], xarray will just make up values (array([0, 1]) here):

<xarray.DataArray 'variants' (variants: 2)> Size: 16B
array([0, 1])
Coordinates:
    variantnames  (variants) object 16B ...
Dimensions without coordinates: variants

[samsrabin]

@ekluzek I also think that every dimension (except string_length) should have a variable associated with it. Ideally that would also serve as the coordinate variable. We could implement this, I think, by simply changing the name of the existing coordinates to match their dimension. So, e.g., the variantnames variable would just become variant. Thus the names would become the data, which would avoid us needing to invent arbitrary data values.

[samsrabin]

I think the problem @adrifoster was describing is that the Fortran code indexes directly into some variables using Fortran-based indexing. We should avoid this as much as possible, instead retrieving those indexes by searching for the name value of interest in the dimension/coordinate array. (This would be a one-time thing that happens upon read of the paramfile, so very low cost—not like we're doing a string comparison for every PFT at every timestep.)

[ekluzek]

Nice discussion. Hmmm. I think possibly there should be two two coordinate variables for each dimension. One is the name to describe them (since they are discrete axes as described in the CF convention). The other would be an indexing coordinate which could be the same as the dimension name (so pft for example). This would be zero based to pft, and 1 based to match the Fortran code convention.

It sounds like if that were done, xarray would have an indexing that would match the Fortran code right?

[ekluzek]

I think the problem @adrifoster was describing is that the Fortran code indexes directly into some variables using Fortran-based indexing. We should avoid this as much as possible, instead retrieving those indexes by searching for the name value of interest in the dimension/coordinate array. (This would be a one-time thing that happens upon read of the paramfile, so very low cost—not like we're doing a string comparison for every PFT at every timestep.)

This would be a much more robust way of handling the data on the files. What happens right now for pftname accomplishes the same thing -- but opposite to what you suggest here. The index names for pfts are figured out by finding the index on the paramfile based on the name. Hence, the indices for pft -- match the index on the file. So I think this method is sufficient for pfts.

But, you are right other dimensions should be handled similarly with making sure the index in the code matches the name in the file.

Doing this consistently for all dimensions -- would mean the model would match however it's done on the file and robustly do the right thing rather than silently screwing everything up, if the file ended up being changes. So this would be a good set of improving robustness to the model.

[samsrabin]

Hmmm. I think possibly there should be two two coordinate variables for each dimension. One is the name to describe them (since they are discrete axes as described in the CF convention). The other would be an indexing coordinate which could be the same as the dimension name (so pft for example). This would be zero based to pft, and 1 based to match the Fortran code convention.

It sounds like if that were done, xarray would have an indexing that would match the Fortran code right?

We'd just need one coordinate variable—the names, not integers. Doing that would give xarray an indexing that matches the Fortran code, as long as we incorporated my second suggestion where the Fortran code never indexes based on hard-coded integers.

[katierocci]

As I talked about with @samsrabin and as a newer user, being able to see what the different dimensions of a parameter mean in words (e.g., for rootprof_beta, water and carbon rather than 1 and 2 or 0 and 1) would be really helpful for users I think. Adding documentation to the users guide about how to see the names of a parameters' dimensions would also be helpful (e.g., via Sam R's new parameter tools). Currently, I figure out dimension names with xarray in python but even figuring that out was harder than expected.

Also, as mentioned by @ekluzek, clarifying when parameters are only used for a specific model component (e.g., mimics_cn_r for the MIMICS soil sub model) would be helpful for new users, even if that information is just added to the long_name (For example, for mimics_cn_r, it could be "C:N of copiotrophic microbes, only for MIMICS"), or somewhere else in the metadata