Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chmod of download path fails if someone else already created the directory #94

Open
ledm opened this issue Mar 23, 2023 · 2 comments
Open
Assignees

Comments

@ledm
Copy link
Collaborator

ledm commented Mar 23, 2023

This line fails if you try to run over a job that someone else already created.

os.chmod(outputFold, st.st_mode | stat.S_IWGRP)

Same issue for this line:
https://github.com/valeriupredoi/bgcval2/blob/main/bgcval2/download_from_mass.py#L480

    outfile.write(download_script_txt)

Trying to overwrite a file that someone else owns.

@ledm
Copy link
Collaborator Author

ledm commented Mar 23, 2023

Actually, that second one is the wrong line. This is the problem one:
https://github.com/valeriupredoi/bgcval2/blob/main/bgcval2/download_from_mass.py#L486

When it get copied over, not when the local copy is made.

@ledm
Copy link
Collaborator Author

ledm commented Mar 23, 2023

We can't hide this behind a try/except: pass statement, as if the file exists already, then no one seems to be able to overwrite it.

The problem there is that the automated mass download script executes all the files that are younger than one month - meaning that someone has requested this data within the last month, so lets make sure it gets downloaded.

The problem will just keeping the existing file copy is that if I request to download some job, lets say u-aa012 once, but never again, the age of the download script will be my old file. When that file is older than a month - downloads will stop.However, if someone else wanted to keep u-aa012 up to date, they can't do it!

My solution is to add usernames to the download script filename. The downside is that the same job may be requested to be downloaded more than once - but this happens in series so the second one is unlikely to find any new files and will complete quickly!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants