refactor: refine shell and scheduler arguments #805
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- move `Scheduler.options` to `QueuedScheduler` and rename (with deprecation warning) as `directives` since it only applies to schedulers that use a job submission system - remove `Scheduler` attributes `shell_args` and `shebang_args` (and class variables `DEFAULT_SHELL_ARGS` and `DEFAULT_SHEBANG_ARGS`) and replace with `Scheduler.shebang_executable`, which will typically not be used, and is a way to override the shell's executable and executable arguments in the jobscript shebang line. - remove unused `DEFAULT_SHELL_EXECUTABLE` class variable from `Scheduler` sub-classes - add `Shell.executable_args` and include this in `Shell.executable`
dkfellows
approved these changes
Apr 23, 2025
There was a problem hiding this comment.
Subject to one question, this is fine.
The question: what's the way of handling adding extra bits to a shebang line? The classic example of that is running a script interpreter with PATH searching using: #!/usr/bin/env interpreter though I've also seen an occasional need to pass other options there too. It's more common with Perl scripts I suppose...
In any case, as long as there is a documented answer, whatever it is, I'm happy with this PR.
|
Thanks Adam. I made a new nenv and installed hpcflow using On CSF3 using SGE it has worked :) Using SLURM, it doesn't. I updated the config like this: But the workflow wouldn't submit. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Scheduler.optionstoQueuedSchedulerand rename (with deprecation warning) asdirectivessince it only applies to schedulers that use a job submission systemSchedulerattributesshell_argsandshebang_args(and class variablesDEFAULT_SHELL_ARGSandDEFAULT_SHEBANG_ARGS) and replace withScheduler.shebang_executable, which will typically not be used, and is a way to override the shell's_executableandexecutable_argsin the jobscript shebang line.DEFAULT_SHELL_EXECUTABLEclass variable fromSchedulersub-classesShell.executable_args(e.g. for using bash in--loginmode) and include this inShell.executable@gcapes please could you test this PR on CSF3 and check if it resolves your issue?
Recommended changes to configuration files
If the
schedulers.[scheduler-name].defaults.shebang_argsis used in a configuration file, this will need to be removed and added to the shell's newexecutable_argsconfiguration, e.g.shells.bash.defaults.executable_args, which should be a list of strings. An example change is shown here.Recommended changes to workflow template files
The
Schedulerargumentoptions(for verbatim scheduler directives) has been renameddirectives. Theoptionskey can still be used, but a deprecation warning will be printed.