A UIS / Hub service for managing workflow runs.
At present, Cylc workflows are manually started, manually monitored throughout the life of the workflow, manually updated (cylc vr) when changes are made, until they are manually stopped when no longer needed.
While this approach will continue for high-stakes deployments where such oversight is desirable/necessary, today's increasingly automated work patterns don't always fit this model. It would be convenient if Cylc itself could provide some level of automation over the management of the lifecycle of workflows to better suit CD working patterns for lower-stakes, more hands-off deployments.
For example, a workflow's lifecycle might look like this:
- Checkout a working copy of a workflow (
git clone ... ~/cylc-src/my-workflow).
- Start it (
cylc vip).
- Update working copy with changes (
git pull upstream/main).
- Apply changes (
cylc vr).
- Stop workflow (
cylc stop).
An alternative, approach to this might be:
- Register a workflow repository with the UIS.
- Working copy automatically checked out.
- Repository automatically monitored for changes, and updated live.
- Workflow automatically restarted if it were ever to go down (e.g, critical infrastructure failure).
- Unregister the workflow with the UIS.
- Workflow automatically stopped, run archived and working copy removed.
This could potentially be orchestrated via a REST API (convenient for making one-off changes), or via a configuration file (convenient for more infrastructure as code type solutions), e.g:
workflows:
- repo: github.com/my-company/my-workflow
name: local
options:
template variables:
- mode: operational
- archive: true
- domain: regional-atmos
- repo: github.com/my-company/my-workflow
name: global-atmos
options:
template variables:
- mode: operational
- archive: true
- domain: global
- repo: github.com/user/research-workflow
name: test/trial
options:
optional configs:
- mode-research
- site-my-site
- grid-n1234In conventional deployments, the lifecycle of the deployment is effectively coupled to the lifecycle of the VMs. In more cloudy deployments, lifecycle is coupled to that of the containers it orchestrates, the longest-lived of which is the Hub/UIS.
As such, the Hub/UIS becomes the top-level service (the thing the people running the orchestration need to ensure is always running when needed) making it the more natural place for such functionalities to lie.
| implementation |
pros |
cons |
| Hub Service |
|
- Per-deployment not per-user (requires authorisation)
- Cloud deployments are generally single-user, so might not use the hub at all
|
| UIS Service |
- Per-user (authentication easy, existing configuration file)
|
- Would require an external service (e.g, a hub service) to ensure the UIS is always running (note hub-less deployments will already have this)
|
Related / upstream issues:
A UIS / Hub service for managing workflow runs.
At present, Cylc workflows are manually started, manually monitored throughout the life of the workflow, manually updated (
cylc vr) when changes are made, until they are manually stopped when no longer needed.While this approach will continue for high-stakes deployments where such oversight is desirable/necessary, today's increasingly automated work patterns don't always fit this model. It would be convenient if Cylc itself could provide some level of automation over the management of the lifecycle of workflows to better suit CD working patterns for lower-stakes, more hands-off deployments.
For example, a workflow's lifecycle might look like this:
git clone ... ~/cylc-src/my-workflow).cylc vip).git pull upstream/main).cylc vr).cylc stop).An alternative, approach to this might be:
This could potentially be orchestrated via a REST API (convenient for making one-off changes), or via a configuration file (convenient for more infrastructure as code type solutions), e.g:
In conventional deployments, the lifecycle of the deployment is effectively coupled to the lifecycle of the VMs. In more cloudy deployments, lifecycle is coupled to that of the containers it orchestrates, the longest-lived of which is the Hub/UIS.
As such, the Hub/UIS becomes the top-level service (the thing the people running the orchestration need to ensure is always running when needed) making it the more natural place for such functionalities to lie.
Related / upstream issues: