Stefans updates #6
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
So that slurm functions are run in their own thread/process.
The experiment tracker assumes it's based on inputs, config and overrides. So if we want to batch submit a bunch of input permutations and we want each one tracked, we need to use a distinct driver for each one, hence the Hamilton within Hamilton. This means that we are not able to create a single DAG (driver chaining is a roadmap item) to view it all. Otherwise: - does anything need to be parallel within the inner DAG? A possible feature to help here would be to make @subdag take in distinct adapters... or we implement driver chaining... Anyway let me know what you think. If you want a single DAG we could use parameterized_subdag, but that would come at the cost of visibility with the experiment tracker, unless a whole permutation of inputs is a single experiment. There is always of course modifying more of hamilton to help here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds some fixes, and then one idea for batch.
See the last commit message for more details.