-
Notifications
You must be signed in to change notification settings - Fork 340
Enable using instantaneous outputs for GDD generation #3445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
samsrabin
wants to merge
12
commits into
ESCOMP:b4b-dev
Choose a base branch
from
samsrabin:gddgen-python-h2i
base: b4b-dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
821c6d3
utils.py write_output(): Save to netCDF4, then try to convert to netC…
samsrabin 820f674
Add unit tests for grid_one_variable.py. Failing.
samsrabin f71e27c
create_filled_array(): Replace np.NaN with np.nan.
samsrabin b5b9bdd
Complete unit testing of create_filled_array().
samsrabin 315e65e
import_and_process_1yr: TEMPORARILY? look for h2a files.
samsrabin 9a9ac56
Add h2a testmod.
samsrabin 873587a
vegtype_str2int(): Handle 0d vegtype_str.
samsrabin 4f4cc6b
Use DASK_UNAVAILABLE and log about it.
samsrabin 6d6c741
Reformat with black.
samsrabin 0c47024
Add previous commit to .git-blame-ignore-revs.
samsrabin f1bbda2
pylint: Delete unused imports.
samsrabin d620a16
Add h2a version of 10x15 RXCROPMATURITY_Lm61 test.
samsrabin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,11 +2,14 @@ | |
|
|
||
| import logging | ||
| import os | ||
| import stat | ||
| import sys | ||
| import glob | ||
| import string | ||
| import re | ||
| import pdb | ||
| import tempfile | ||
| import subprocess | ||
|
|
||
| from datetime import date, timedelta, datetime | ||
| from getpass import getuser | ||
|
|
@@ -191,6 +194,7 @@ def write_output(file, file_in, file_out, file_type): | |
| """ | ||
|
|
||
| # update attributes | ||
| logger.info("Updating metadata: %s", file_out) | ||
| title = "Modified " + file_type + " file" | ||
| summary = "Modified " + file_type + " file" | ||
| contact = "N/A" | ||
|
|
@@ -205,11 +209,51 @@ def write_output(file, file_in, file_out, file_type): | |
| description=description, | ||
| ) | ||
|
|
||
| logger.info("Writing: %s", file_out) | ||
| # mode 'w' overwrites file if it exists | ||
| file.to_netcdf(path=file_out, mode="w", format="NETCDF3_64BIT") | ||
| file.to_netcdf(path=file_out, mode="w", format="NETCDF4_CLASSIC") | ||
| logger.info("Successfully created: %s", file_out) | ||
| file.close() | ||
|
|
||
| logger.info("Trying to convert to NETCDF3_CLASSIC: %s", file_out) | ||
| if convert_netcdf_to_classic(file_out): | ||
| logger.info("Conversion succeeded") | ||
| else: | ||
| logger.info("Conversion failed, perhaps because nccopy wasn't found in your shell") | ||
|
|
||
|
|
||
| def convert_netcdf_to_classic(filepath): | ||
| """ | ||
| Try to convert netCDF to netCDF-3 Classic format using nccopy. | ||
| """ | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
|
||
| # Get temporary file path | ||
| dirpath = os.path.dirname(filepath) | ||
| with tempfile.NamedTemporaryFile(delete=False, dir=dirpath, suffix=".nc") as tmp: | ||
| temp_path = tmp.name | ||
|
|
||
| try: | ||
| subprocess.run(["nccopy", "-3", filepath, temp_path], check=True) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Use |
||
| copy_permissions(filepath, temp_path) # nccopy doesn't preserve permissions | ||
| os.replace(temp_path, filepath) | ||
| return True | ||
| except subprocess.CalledProcessError: | ||
| return False | ||
|
|
||
|
|
||
| def copy_permissions(src_file, dst_file): | ||
| """ | ||
| Copy file permissions from src to dest | ||
| """ | ||
| # Get the mode (permissions) of the source file | ||
| src_mode = os.stat(src_file).st_mode | ||
|
|
||
| # Extract the permission bits | ||
| permissions = stat.S_IMODE(src_mode) | ||
|
|
||
| # Apply them to the destination file | ||
| os.chmod(dst_file, permissions) | ||
|
|
||
|
|
||
| def get_isosplit(iso_string, split): | ||
| """ | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nccopyis available in shell before the initialfile.to_netcdf()call. If it's not, just skip straight to old behavior of"NETCDF3_64BIT".file.to_netcdf(), this time straight to"NETCDF3_64BIT".There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @samsrabin I'm just wondering the reasons behind the convert to classic being done here?
I don't know of any compelling reasons to prefer classic over other formats. And classic has some pretty strong restrictions. And it seems preferable to me to write it out in the preferred format from the get go rather than convert. But I would like to hear the thinking here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I think I meant for this all to be netCDF-3 64-bit. Will fix.