Skip to content

External block storage for JSON/TOML backend via file system or S3 #1789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
franzpoeschel wants to merge 36 commits into
base: dev
Choose a base branch
from
Draft

External block storage for JSON/TOML backend via file system or S3 #1789

franzpoeschel wants to merge 36 commits into from

Conversation

@franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Sep 30, 2025 •
edited
Loading

TODO

  • error handling
  • can we use ADIOS2 as a block storage backend?
  • do not always try creating the bucket, only if we need to
  • JSON merging (MPI)
  • async IO
  • (probably for later PR) non-matching load operations
  • merge script: read file names from stdin, not from command line args
  • ensure that restarting will not overwrite old blocks

@franzpoeschel franzpoeschel force-pushed the s3 branch 2 times, most recently from 54db435 to ec2d404 Compare December 5, 2025 17:29
@franzpoeschel
wip first commit
766cd9e
@franzpoeschel
Some cleaning
c2e53d5
@franzpoeschel
Hmm maybe usable
6a6330f
@franzpoeschel
Quickly use this in the JSON backend
4f41e60 
bin/openpmd-pipe --infile samples/git-sample/data%T.bp --outfile data.json --outconfig "$(echo -e 'json.dataset.mode = "template"\njson.attribute.mode = "short"')"
@franzpoeschel
Better and generalized handling for datatypes
7301e94
@franzpoeschel
structure for aws
ec47d7d
@franzpoeschel
first untested implementation for S3
c17553c
@franzpoeschel
Reordering
39679a0
@franzpoeschel
continue restructuring
6ff99e4
@franzpoeschel
Some first little MPI awareness
1a47575
@franzpoeschel
WIP: Config for external block storage from JSON
97b8020
@franzpoeschel
Add configuration
b2ccf98
@franzpoeschel
Add option to init AWS API
2a66841
@franzpoeschel
Add verifySSL parameter
2a1c520
@franzpoeschel
Add TODO comment
99669e3
@franzpoeschel
Add meta information object
57a0505
@franzpoeschel
Prepare reloading ext block storage from old file
a5d89c1
@franzpoeschel
Reload config when reading from a JSON file
7b045fe
@franzpoeschel
WIP: Read from EBS
6860212
@franzpoeschel
Base implementation for get()
7a75578
@franzpoeschel
Untested read impl
aa17b83
@franzpoeschel
Basically working reading
ba1d0ff 
needed also support for availableChunks
@franzpoeschel
cleanup
4381409
franzpoeschel
franzpoeschel commented Dec 8, 2025
View reviewed changes
src/IO/JSON/JSONIOHandlerImpl.cpp
else
{
// sic! initialize the deferred json config as a new
// tracing object
Copy link
Contributor Author

@franzpoeschel franzpoeschel Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Return warning upon unused keys

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

api: new additions to the API backend: JSON backend: TOML

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@franzpoeschel