Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

♻️ Allow for file uploads/downloads to be async #6079

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

chrisjsewell
Copy link
Member

Note, this PR is currently dependent on aiidateam/plumpy#272


Currently, a possible bottleneck for workers (running potentially 1000s of processes asynchronously) is the upload/retrieval of calculation input/output data from an external compute resource (e.g. HPC).
This runs in a blocking manner, i.e. all other async tasks have to wait until all the input or outputs are fully uploaded/retrieved.

This could be made asynchronous, either at the "file level" - relinquishing control to the event loop after each file upload/download, or even at the "byte level" - relinquishing control after each "chunk of a file" has been uploaded/downloaded.
(For other transports, like FirecREST there are even other aspects of async to consider.)

This particular PR does not actually implement any async behaviour for uploads/downloads, it merely modifies the engine API to allow for implementations of the Transport interface to achieve this.

The PR changes the following functions/methods to async:

  • execmanager.upload_calculation
  • execmanager.retrieve_calculation
  • execmanager.retrieve_files_from_list
  • Calcjob.run
  • CalcJob._perform_dry_run
  • CalcJob._perform_import

However, all of these are not intended for use by the user, hence I would suggest this is backwards compatible.

@chrisjsewell
Copy link
Member Author

chrisjsewell commented Jul 9, 2023

As discussed with @giovannipizzi (who suggested it)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants