feat(fireactions): derive each pool's labels from a host_vars canary flag - #35
Merged
Conversation
…flag A host carrying `fireactions_canary: true` in a `host_vars/` file beside the fireactions inventory now drops the shared labels on all three of its pools for a flat `self-hosted` + `fireactions-canary`, taking it out of the shared pool for the length of a rollout window so a kernel or VMM change can be proven on one host before the other three get it. THIS COMMIT IS INERT. It lands the mechanism and no canary: no `host_vars/` directory is created, so all four hosts still resolve to exactly today's labels. Verified by rendering `fireactions_pools` per host on both trees and diffing - the config the template emits is identical for all four, so nothing restarts and no capacity moves. Creating the file is the on switch, and it is deliberately not committed ahead of a window, because the unattended apply picks it up within minutes of it landing on main. A FLAG RATHER THAN THE LABELS THEMSELVES, and that indirection is the whole design. `install-fireactions.yaml` loads `fireactions_pools` through `vars_files:`, which outranks inventory `host_vars` - so a `host_vars` file setting `labels` or `fireactions_pools` would be read, beaten, and apply `ok` while changing nothing. `fireactions_canary` is a name `vars_files` never defines, which is why it survives to be read from an expression there. The condition is extracted to `fireactions_canary_on` rather than repeated inline: the three pools must differ on their shared label list and must not differ on the test, and nothing in CI renders these files to catch it if they did. `| bool` is what makes a quoted "false" or "no" mean false rather than merely non-empty. All three pools rather than only `fireactions-small`: leaving medium and large on the shared set would keep the host serving ordinary org traffic on an unproven configuration for the whole window, and the runner-image build targets `fireactions-medium`. Flat rather than tiered, because closing a window is then one org-scoped string to assert gone instead of three - at the cost, now documented, that the three tiers become indistinguishable on a canary host. Also swept, because this change made both stale: the fireactions inventory's comment claiming every host runs the same definitions, and README.md's tier table and org-wide capacity arithmetic, which now has a § "The canary" carrying the figures, the restart-kills-jobs warning and the placement trap - the flag must sit beside the fireactions inventory, since `pve-vms/all/` holds only `all_vms` and this play runs against `fireactions_all`. `yamllint`, `ansible-lint --offline` and `ansible-playbook --syntax-check` pass; the two changed YAML files pass ansible-lint's `production` profile with no skip-list. ADR-0156 §A, §B, §C; ADR-0165 §A, §B.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each Fireactions pool's
labels:is now derived from afireactions_canaryflag rather than being a literal list, so one host can be taken out of the
shared pool for the length of a rollout window — letting a kernel or VMM change
be proven on it before the other three hosts get it.
This PR is inert
It lands the mechanism and no canary. No
host_vars/directory is created,so all four hosts still resolve to exactly the labels they carry today.
Verified rather than asserted:
fireactions_poolswas rendered per host againstboth
mainand this branch and the results diffed — the configconfig.yaml.j2emits is identical for all four hosts. Nothing restarts, norunner re-registers, and no capacity moves.
Creating the flag file is the on switch, and it is deliberately not committed
ahead of a window, because the unattended apply picks it up within minutes of it
reaching
main— merging the file is the operation.README.md§ The canarydocuments how to open and close a window.
Why a flag and not the labels
install-fireactions.yamlloadsfireactions_poolsthroughvars_files:, whichoutranks inventory
host_vars. Ahost_varsfile settinglabelsorfireactions_poolsdirectly would therefore be read, silently beaten, and applyokwhile changing nothing at all.fireactions_canaryis a namevars_filesnever defines, so it survives to beread from an expression there. That is the whole design — it is the one shape in
this play that does what it appears to do.
The condition is extracted to
fireactions_canary_onrather than written inlinethree times: the three pools must differ on their shared label list and
must not differ on the test, and nothing in CI renders these files to catch
it if they drifted.
| boolis what makes a quoted"false"or"no"meanfalse rather than merely non-empty.
What a canary host looks like
All three of its pools drop their shared labels for one flat set,
self-hostedfireactions-canary:runs-onfireactions-smallfireactions-mediumfireactions-largefireactions-canaryAll three rather than only
small, because leavingmediumandlargeon theshared set keeps the host serving ordinary org traffic on the unproven
configuration for the whole window — and the runner-image build targets
fireactions-medium. Flat rather than tiered, because closing a window is thenone org-scoped label to assert gone instead of three.
The cost of flat is real and is now written down in the README: on a canary
host the three tiers are indistinguishable, so a 12 GiB job can land on a
4 GiB
small.Docs swept, because this change made them stale
Still true of the definitions; no longer true of the labels they resolve to.
README.md's tier table and the org-wide32 / 4 / 4arithmetic assumed everyhost is on the shared labels. It gains a § The canary carrying the figures
above, the warning that the flip restarts
fireactionsand kills that host'sin-flight jobs, and the placement trap — the flag must sit beside the
fireactions inventory, because
pve-vms/all/holds only theall_vmsgroup and this play runs against
fireactions_all. Ahost_vars/next toall/would be read by nothing and the apply would go green having changed nolabel.
Checks
Run locally on the versions
lint.ymlpins (ansible-lint 26.6.0,ansible-core 2.21.2,yamllint 1.38.0):yamllintover every tracked YAML — clean.ansible-lint --offlineon the two changed YAML files — passes theproductionprofile with no skip-list. The full-tree run reproduces the twofindings already present on
mainand adds none.ansible-playbook --syntax-checkoninstall-fireactions.yaml— passes.real YAML list (not a string) through
to_nice_yaml.No host was contacted and nothing was applied.