Skip to content

Add file size verification to POSC plugin to prevent incomplete uploads #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
patrickbrophy merged 6 commits into from
Dec 19, 2025
Merged

Add file size verification to POSC plugin to prevent incomplete uploads #127

patrickbrophy merged 6 commits into from
Dec 19, 2025

Conversation

@patrickbrophy
Copy link
Contributor

@patrickbrophy patrickbrophy commented Dec 17, 2025 •
edited
Loading

This PR improves the robustness and user experience of POSC uploads by addressing two related issues: incomplete uploads and premature file visibility.

The PR does the following:

  • POSC now uses the oss.asize attribute, when provided, to determine the expected size of an upload. This allows POSC to detect and guard against incomplete uploads using the expected size as an advisory check.
  • File creation behavior has been adjusted so that POSC no longer creates an empty file at the final destination before an upload completes. This prevents empty files from briefly appearing in directory listings during active uploads.

@patrickbrophy
Fix OSS passthrough and set proper mode
8430580
@patrickbrophy
Implement atomic writes
ecabae7 
There are two parts to this commit, the first being that we are spoofing
creates because we are handling it in the call to Open. The second part
is using the `oss.asize` to give a hint on how big the upload is so that
we can ensure that the file is only commited when all expected bytes are
on disk.
@patrickbrophy
Persist if and only if we expected size matches the actual size
6dac741
@patrickbrophy
Clear posc file name in all paths of Close()
94f2035
@patrickbrophy patrickbrophy added bug Something isn't working enhancement New feature or request labels Dec 17, 2025
@patrickbrophy @github-actions
Fix linter issues
df8de8b 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@patrickbrophy
Copy link
Contributor Author

patrickbrophy commented Dec 19, 2025

@bbockelm This is ready for your review

bbockelm
bbockelm approved these changes Dec 19, 2025
View reviewed changes
Copy link
Collaborator

@bbockelm bbockelm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@patrickbrophy patrickbrophy merged commit e65236c into PelicanPlatform:main Dec 19, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bbockelm bbockelm bbockelm approved these changes

Assignees

@bbockelm bbockelm

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@patrickbrophy @bbockelm