netcdf to grib with cdo: use own parameter file to define names and units

[gisi90]

Current setting uses
cdo -t ecmwf
to convert grib to netcdf files which seems not to be recommended for ML files (https://confluence.ecmwf.int/display/OIFS/How+to+convert+GRIB+to+netCDF).

This comes with a mix in variables (upper case/lower case, missing short names, unphysical units e.g. 0-1).

It is possible to provide own tables to cdo that define names and units. File paratab.txt has to look as follows:
26 cl Lake cover [(0 - 1) ]
27 cvl Low vegetation cover [(0 - 1) ]
28 cvh High vegetation cover [(0 - 1) ]
29 tvl Type of low vegetation []
30 tvh Type of high vegetation []
31 ci Sea-ice cover [(0 - 1) ]
32 asn Snow albedo [(0 - 1) ]
....
124 emis Surface emissivity [dimensionless ]
125 tec Vertically integrated total energy [J m**-2 ]
126 Generic parameter for sensitive area prediction [Various ]
127 at Atmospheric tide []
128 bv Budget values []
129 z Geopotential [m2 s-2 ]
130 t Temperature [K ]
131 u U velocity [m s**-1 ]
132 v V velocity [m s**-1 ]
133 q Specific humidity [kg kg**-1 ]
134 sp Surface pressure [Pa ]
135 w Vertical velocity [Pa s**-1 ]
136 tcw Total column water [kg m**-2 ]
137 tcwv Total column water vapour [kg m**-2 ]
138 vo Vorticity (relative) [s**-1 ]
139 stl1 Soil temperature level 1 [K ]

Cdo identifies variables in grib-files based on their parameter ID only, table2Version is ignored. This means if variables of a table different from the standard one (i.e. 128) are used their conversion to netcdf requires a new adjusted parameter file, e.g. paratab228.txt:
29 i10fg Instantaneous 10 metre wind gust [m s**-1 ]
246 u100 h100 metre U wind component [m s**-1 ]
247 v100 h100 metre V wind component [m s**-1 ]
239 u200 h200 metre U wind component [m s**-1 ]
240 v200 h200 metre V wind component [m s**-1 ]
(in this case ID 29 in table 228 differs from ID 29 in table 128)

This means variables of table 128 should not be mixed with variables from other tables in one grib file. For that case, own grib files with own conversion and the right parameter file must be used when converted with cdo.

For the variables of table 128 a template with all variables with paramter IDs 1 to 255 is available.

Name in the parameter file is the long name. Default shortnames and adjustment or possibilities to define them needs to be investigated.

Integration of own parameter files can help to reduce some lines of code required up to now to adjust units.

[joernu76]

Mmmh. long names are not that useful to us. We'd need to be able to set the standard_names to really save code.

[gisi90]

Suggestion for workaround: set long_names as desired, delete old attributes "standard_name", rename the attribute long_name to short_name using ncrename. Would be two additional lines only after cdo conversion with own parameter table:

ncatted -O -a standard_name,,d,, infile.nc
ncrename -a .long_name,standard_name infile.nc

[joernu76]

That might work. I am not fully convinced that this is a significantly better approach as we still have to fudge the files... Would you prepare a draft?