Remove need for user home directory entirely.

[mulroony]

Allow the Cylc run root to be configured outside $HOME

Problem

Cylc currently uses $HOME/cylc-run as the root for installed and running workflows.

The existing [install][symlink dirs] configuration can store workflow data on another filesystem, but it still requires a corresponding path or symlink under $HOME/cylc-run. This does not cover sites where:

• home directories are unavailable on workflow or job hosts;
• home-directory inode or metadata quotas are restrictive; or
• site policy requires runtime state to be kept outside home directories.

In these cases, the entire Cylc run namespace needs to live at another location rather than being reached through symlinks in $HOME/cylc-run.

Proposed solution

Allow CYLC_RUN_DIR to override the canonical Cylc run root:

export CYLC_RUN_DIR=/path/to/cylc-run

cylc install my-workflow
cylc play my-workflow

If CYLC_RUN_DIR is not set, Cylc would continue to use $HOME/cylc-run with no behavior change.

The override should apply consistently to installation, workflow discovery, scheduler paths, cleanup, remote commands, and generated job scripts. Cylc should automatically forward it to scheduler run hosts and job platforms.

An explicitly supplied alternate directory, such as cylc workflow-state --run-dir, should continue to take precedence.

Scope and constraints

• This would configure one run root per Cylc process environment; it would not add discovery across multiple run roots.
• All Cylc commands targeting workflows in the alternate root would need the same CYLC_RUN_DIR value.
• The configured path would need to be accessible at the same location on scheduler and job hosts.
• Existing [install][symlink dirs] behavior would continue to work, with symlinks created from the configured run root.

Relationship to existing symlink support

PR #4177 removed the old per-platform run-directory setting because symlink directories provided that storage-placement functionality. This proposal is narrower: it changes the canonical run namespace itself for installations where $HOME/cylc-run cannot be used at all.

[oliver-sanders]

Hi @mulroony 👋, thanks for your interest in Cylc.

This does not cover sites where:

The situations you've outlined above are familiar to us. I believe that Cylc's existing functionality does cover these cases but may be missing the finer details of your setup. I have included some details of how to manage these situations below, let me know if they make sense to you.

Configurable run directories are a very awkward feature to support. We do have experience of what would be required to implement this as this is a feature we used to support in Cylc 7. Unfortunately, this proposal would still leave Cylc dependent on $HOME/cylc-run in a small number of situations which are hard to resolve, notably remote-init. Cylc 7 solved this problem by requiring alternate cylc-run locations to be configured on the scheduler host, which could then use this mapping to template remote commands issued by the scheduler with the appropriate paths. To support this in Cylc 8 would require something similar, however, this functionality was fragile and difficult to maintain, so we decided instead to require a $HOME directory. This way we have a consistent path for accessing a Cylc workflow across all of the platforms it may be installed on, but allow sites to symlink some or all of the actual contents of the cylc-run directory to the desired location(s).

---

home directories are unavailable on workflow or job hosts;

For job hosts with no $HOME, we provide a pattern for setting the run directory using the global init-script configuration:

https://cylc.github.io/cylc-doc/stable/html/reference/config/writing-platform-configs.html#platform-with-no-home-directory

Workflow hosts must have a $HOME for various reasons besides locating the cylc-run directory.

home-directory inode or metadata quotas are restrictive; or

site policy requires runtime state to be kept outside home directories.

The symlink dirs feature does cover these scenarios. While it can be used to symlink bits of workflow runs onto different mounts, it can also be used to symlink the entire workflow run.

For example, this configuration:

[install]
    [[symlink dirs]]
        [[[localhost]]]
            run = /some/other/mount

Would result in an arrangement like this:

$ tree ~/cylc-run
|-- workflow1
|    |-- run1 -> /some/other/mount/cylc-run/workflow1/run1
|    `-- run2 -> /some/other/mount/cylc-run/workflow1/run2
`-- workflow2
      |-- run1 -> /some/other/mount/cylc-run/workflow2/run1
      `-- run2 -> /some/other/mount/cylc-run/workflow2/run2

I.e, one directory per workflow and one symlink per run. It would take an extremely large number of workflow runs to saturate the strictest inode quota.

Under this arrangement all workflow state is stored on the alternative mount. The $HOME/cylc-run directory is just a bare skeleton of symlinks.

But if a more extreme solution is desired, the entire $HOME/cylc-run directory itself can be manually symlinked, i.e:

$ tree ~
|-- cylc-run -> /some/other/mount/cylc-run
...

Cylc (incl, cylc-uiserver, cylc-tui, etc) will be fine with this.

[mulroony]

@oliver-sanders , thanks!

Yeah we saw that option but it still requires a home directory which creates some issues for the way we intend to use Cylc and goes against how we structure all our projects / operational runs due to some laws and regulations related to user privacy. We see significant benefit from being able to locate all workflow and project data with no dependency at all on the home directory of a user (even if it is a service account).

I admit to being new to this code base so happy to look into any edge cases where my patch would not work. We tried to structure this PR so the default behavior was preserved but provide an option to avoid a home directory.

[hjoliver]

home directories are unavailable on workflow or job hosts;

@mulroony - are you unequivocally stating that you have workflow hosts (i.e. where you will run Cylc schedulers) that do not have user home directories?

We have encountered home-less job platforms (where Cylc needs to submit jobs to run) before, which perhaps makes some kind of sense, but not this. Given the purpose and history of user accounts and associated home directory spaces in *nix systems, it's hard to understand why anyone would do that - but maybe I'm missing something.

[oliver-sanders]

Yeah we saw that option but it still requires a home directory

Only to hold a single symlink, no actual data needs to be stored there there so wouldn't expect that to cause issues with data regulation?

Workflow nodes:

Would require a home directory with a single symlink. But, that said, it is probably possible to fiddle the value of `$HOME` in the `cylc` wrapper script in order to relocate the `cylc-run` directory on workflow nodes.

Job nodes:

Do not require `$HOME` directory.

happy to look into any edge cases where my patch would not work

I think you would probably be able to get this approach to work for one particular Cylc deployment, say a homogenous deployment where schedulers run directly on the login node. But I think it would be very difficult to get this feature to play with the range of configurations and deployment patterns Cylc supports.

I think we would need to implement this functionality as it was at Cylc 7 for it to work reliably. I.e, the run directory location for each install-target is defined in the Cylc configuration on the scheduler node. Cylc then templates this into remote paths for all future operations on that platform.

I haven't gone through in detail, issues I'm aware of so far:

1. Cylc remote utilities rely on ssh/rsync landing in the $HOME directory on the remote host, so we only need to specify a relative path (i.e, cylc-run/<workflow-id>) to access it. These utilities are run from scheduler nodes (where Cylc schedulers run) OR from user nodes (where users run Cylc commands). Both of these nodes would need to know the location of cylc-run directories on all other nodes in order to function (as relative paths would no longer work under this plan). The opposite possibility would be to replace all rsync usage with bespoke Cylc commands able to implement bespoke cylc-run location logic.
2. This can only work for heterogenous setups. This is because CYLC_RUN_DIR is added to the ssh forward environment variables list. This means that the CYLC_RUN_DIR variable local to one node, is forwarded onto all remote nodes used by the workflow. This makes the assumption that the cylc-run dir is in the same location on the remote node. This might make sense if you are trying to use a login node as a scheduler host - so the whole deployment is within one install target - but will break at install target boundaries.
3. Getting this to work across install-target boundaries would break scenarios where user names are not consistent across platforms. Relying on rsync landing in the $HOME directory saves us having to construct install_target:run_dir directory mappings, but it also saves us having to construct platform:user mappings. We don't need to template user names into paths at any time in Cylc, we defer this to the ~/.ssh/config.

[mulroony]

@mulroony - are you unequivocally stating that you have workflow hosts (i.e. where you will run Cylc schedulers) that do not have user home directories?

We have encountered home-less job platforms (where Cylc needs to submit jobs to run) before, which perhaps makes some kind of sense, but not this. Given the purpose and history of user accounts and associated home directory spaces in *nix systems, it's hard to understand why anyone would do that - but maybe I'm missing something.

@hjoliver, no they have home directories but in some situations we have multiple accounts that need to run / monitor the same workflow and having the dependency on the home directory to do that creates trouble, confusion, and / or additional work for us. We also have rules around accessing user home directories so we prefer to keep everything related to our workflows outside those directories.

[mulroony]

Yeah we saw that option but it still requires a home directory

Only to hold a single symlink, no actual data needs to be stored there there so wouldn't expect that to cause issues with data regulation?

Workflow nodes:
Would require a home directory with a single symlink. But, that said, it is probably possible to fiddle the value of $HOME in the cylc wrapper script in order to relocate the cylc-run directory on workflow nodes.
Job nodes:
Do not require $HOME directory.

happy to look into any edge cases where my patch would not work

I think you would probably be able to get this approach to work for one particular Cylc deployment, say a homogenous deployment where schedulers run directly on the login node. But I think it would be very difficult to get this feature to play with the range of configurations and deployment patterns Cylc supports.

I think we would need to implement this functionality as it was at Cylc 7 for it to work reliably. I.e, the run directory location for each install-target is defined in the Cylc configuration on the scheduler node. Cylc then templates this into remote paths for all future operations on that platform.

I haven't gone through in detail, issues I'm aware of so far:

1. Cylc remote utilities rely on ssh/rsync landing in the $HOME directory on the remote host, so we only need to specify a relative path (i.e, cylc-run/<workflow-id>) to access it. These utilities are run from scheduler nodes (where Cylc schedulers run) OR from user nodes (where users run Cylc commands). Both of these nodes would need to know the location of cylc-run directories on all other nodes in order to function (as relative paths would no longer work under this plan). The opposite possibility would be to replace all rsync usage with bespoke Cylc commands able to implement bespoke cylc-run location logic.
2. This can only work for heterogenous setups. This is because CYLC_RUN_DIR is added to the ssh forward environment variables list. This means that the CYLC_RUN_DIR variable local to one node, is forwarded onto all remote nodes used by the workflow. This makes the assumption that the cylc-run dir is in the same location on the remote node. This might make sense if you are trying to use a login node as a scheduler host - so the whole deployment is within one install target - but will break at install target boundaries.
3. Getting this to work across install-target boundaries would break scenarios where user names are not consistent across platforms. Relying on rsync landing in the $HOME directory saves us having to construct install_target:run_dir directory mappings, but it also saves us having to construct platform:user mappings. We don't need to template user names into paths at any time in Cylc, we defer this to the ~/.ssh/config.

Thanks, I will dig into these tools / issues and see if anything jumps out as a solution that would work with our PR. Give me a few days and if you know of any other dependency on the hmoe directory outside the cylc-flow repo I am happy to look at that too.

[oliver-sanders]

in some situations we have multiple accounts that need to run / monitor the same workflow and having the dependency on the home directory to do that creates trouble, confusion, and / or additional work for us

Sorry, I'm not quite following, we have multiple users monitoring/controlling the same workflow, I don't think the filesystem where the cylc-run directory is located poses a barrier to this.

Are you perchance trying to enable a group of people to monitor/control the same workflow by giving them a shared cylc-run directory with write permissions?

Feel free to drop in on the developers chat (link in the README) if you want to catch us in real time.

[hjoliver]

@mulroony - here's how we do it at my site:

• production workflows run on "role accounts" (not normal user accounts), with run-dir symlinking to a data area
  • the role account authorizes operations team members to interact with its workflows via (Cylc Hub launched) Cylc GUI; and the workflow logs record who did what
  • and operations team members can sudo to the role account if needed
• normal users run their own private workflows on their own user accounts, with run-dir symlinking to a data area
• normal users collaborating on projects run workflows on their own user accounts, with run-dir symlinking to a shared project data area, where all project members can read the workflow files
  • the workflow owner can choose to authorize colleagues to view or interact with their workflows via Hub-launched Cylc GUI; and the workflow logs record who did what
  • (however, there are a few things that still require logging in as (or sudo to) the workflow owner)

@hjoliver, no they have home directories but in some situations we have multiple accounts that need to run / monitor the same workflow and having the dependency on the home directory to do that creates trouble, confusion, and / or additional work for us. We also have rules around accessing user home directories so we prefer to keep everything related to our workflows outside those directories.

Given that you do have home directories, I think we need to look at exactly what trouble this causes for you. I wonder if you're really looking at the production "role user" scenario (where no one person "owns" the workflow) but are not in a position to get such accounts created? Or the project workflow scenario (where there usually is a natural owner, but others need to see progress and outputs and maybe have some control) but are not in a position to get such shared project areas created? Or if you were not aware of Cylc's other-user authorization capability?

(Note we just need to understand exactly what the requirements are, and whether or not the status quo really can't handle them, before considering a significant change that will require ongoing maintenance and support even if the default behavior remains the same).