Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ECCODES ERROR: unable to convert endStep in stepUnits #335

Open
kevinrosa opened this issue Mar 14, 2023 · 5 comments
Open

ECCODES ERROR: unable to convert endStep in stepUnits #335

kevinrosa opened this issue Mar 14, 2023 · 5 comments

Comments

@kevinrosa
Copy link

kevinrosa commented Mar 14, 2023

This may or may not be the right place to discuss this issue but it seemed related to some other previous discussions regarding time dimensions) and ECCODES ERROR message.

I'm working with WRF model GRIB2 output files. I receive the message ECCODES ERROR : unable to convert endStep in stepUnits but it does not actual raise an exception. And from what I can tell, the time variable that is returned in the xarray DataSet seems accurate.

  • I have 1 file per timestep, with 30 minutes between files. The ECCODES message appears only when reading the files on the half-hour, not the hour (i.e. *_30_d02.grb2).
  • The message is repeated 16 times per xr.open_dataset() call.
  • Opening with backend_kwargs={'time_dims': ['valid_time']} reduces this to 8 messages per open.
  • If I create an *.idx file, the ECCODES message is only returned on the first open, and not returned on subsequent opens once the index file has been created.
    • There is some nuance to this, however. If I open a file using filter_by_keys={'typeOfLevel': 'surface', 'stepType': 'instant'}, then I'll get the ECCODES message when trying to open the same file using {'typeOfLevel': 'heightAboveGround', 'level': 10}.
    • Then, if I try to open for a different height level e.g. {'typeOfLevel': 'heightAboveGround', 'level': 2}, I won't get the error.

The message originates from this line: https://github.com/ecmwf/eccodes/blob/develop/src/grib_accessor_class_g2end_step.cc#L293

Example usage:

fname = 'wrf_wrfout_2022-07-02.00.00-0024_30_d02.grb2'
ds = xr.open_dataset(fname, engine="cfgrib", filter_by_keys={'typeOfLevel': 'heightAboveGround', 'level': 2}, backend_kwargs={'indexpath': '', 'time_dims': ['valid_time']})

Example data file: wrf_wrfout_2022-07-02.00.00-0024_30_d02.grb2.zip
(Note, I zipped it since .grb2 files are not supported by github file attachments.

I've spend some time trying to figure this out and would be interested in any possible explanations or suggestions. At this point, I'd be happy just to convince myself that the resulting DataSet is correct and then figure out how to suppress the ECCODES message -- I've tried setting logging levels and catching warnings, but nothing has worked yet.

@shahramn
Copy link
Collaborator

@emfdavid
Copy link

emfdavid commented Oct 2, 2023

So based on @shahramn reference I take it that WRF is incorrectly using a reserved attribute type?
There is a great deal of existing operational WRF output from NOAA which generates these errors
gs://high-resolution-rapid-refresh/hrrr.20230928/conus/hrrr.t00z.wrfsubhf01.grib2
Can this be reconciled in eccodes?

@shahramn
Copy link
Collaborator

shahramn commented Oct 4, 2023

We are working on a feature to support sub-hourly GRIB2 data. This is still in early stages but in the future the output of grib_ls will look something like this:

% grib_ls -n time wrf_wrfout_2022-07-02.00.00-0024_30_d02.grb2
stepType      stepunits     startStep     endStep       stepRange
instant       m             1470m         1470m         1470m
...
accum         m             1440m         1470m         1440m-1470m

So for data encoded in minutes, we will add the step unit to the stepRange key so that this becomes, e.g., "0m-30m" for an accumulation from 0 to 30 minutes (it will, however, remain as e.g., "0-2" - so without the unit - for an accumulation from 0 to 2 hours).
We will keep you posted

@emfdavid
Copy link

emfdavid commented Oct 4, 2023

@shahramn is there a github issue I can follow or some other place to track progress?
Thank you for your work!

@shahramn
Copy link
Collaborator

shahramn commented Oct 4, 2023

Best thing to do is watch the ecCodes GitHub repo for new releases here:
https://github.com/ecmwf/eccodes/releases

Also you can watch the "History Of Changes" in the ecCodes home page:
https://confluence.ecmwf.int/display/ECC/History+of+Changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants