Skip to content

feat(fireactions): derive each pool's labels from a host_vars canary flag - #35

Merged
kholisrag merged 1 commit into
mainfrom
feat/1036-canary-flag
Aug 6, 2026
Merged

feat(fireactions): derive each pool's labels from a host_vars canary flag#35
kholisrag merged 1 commit into
mainfrom
feat/1036-canary-flag

Conversation

@kholisrag

Copy link
Copy Markdown
Owner

Each Fireactions pool's labels: is now derived from a fireactions_canary
flag rather than being a literal list, so one host can be taken out of the
shared pool for the length of a rollout window — letting a kernel or VMM change
be proven on it before the other three hosts get it.

This PR is inert

It lands the mechanism and no canary. No host_vars/ directory is created,
so all four hosts still resolve to exactly the labels they carry today.

Verified rather than asserted: fireactions_pools was rendered per host against
both main and this branch and the results diffed — the config
config.yaml.j2 emits is identical for all four hosts. Nothing restarts, no
runner re-registers, and no capacity moves.

Creating the flag file is the on switch, and it is deliberately not committed
ahead of a window, because the unattended apply picks it up within minutes of it
reaching main — merging the file is the operation. README.md § The canary
documents how to open and close a window.

Why a flag and not the labels

install-fireactions.yaml loads fireactions_pools through vars_files:, which
outranks inventory host_vars. A host_vars file setting labels or
fireactions_pools directly would therefore be read, silently beaten, and apply
ok while changing nothing at all.

fireactions_canary is a name vars_files never defines, so it survives to be
read from an expression there. That is the whole design — it is the one shape in
this play that does what it appears to do.

The condition is extracted to fireactions_canary_on rather than written inline
three times: the three pools must differ on their shared label list and
must not differ on the test, and nothing in CI renders these files to catch
it if they drifted. | bool is what makes a quoted "false" or "no" mean
false rather than merely non-empty.

What a canary host looks like

All three of its pools drop their shared labels for one flat set, self-hosted

  • fireactions-canary:
runs-on Outside a window With one canary
fireactions-small 32 24
fireactions-medium 4 3
fireactions-large 4 3
fireactions-canary 0 10

All three rather than only small, because leaving medium and large on the
shared set keeps the host serving ordinary org traffic on the unproven
configuration for the whole window — and the runner-image build targets
fireactions-medium. Flat rather than tiered, because closing a window is then
one org-scoped label to assert gone instead of three.

The cost of flat is real and is now written down in the README: on a canary
host the three tiers are indistinguishable
, so a 12 GiB job can land on a
4 GiB small.

Docs swept, because this change made them stale

  • The fireactions inventory claimed every host runs the same pool definitions.
    Still true of the definitions; no longer true of the labels they resolve to.
  • README.md's tier table and the org-wide 32 / 4 / 4 arithmetic assumed every
    host is on the shared labels. It gains a § The canary carrying the figures
    above, the warning that the flip restarts fireactions and kills that host's
    in-flight jobs, and the placement trap — the flag must sit beside the
    fireactions inventory, because pve-vms/all/ holds only the all_vms
    group and this play runs against fireactions_all. A host_vars/ next to
    all/ would be read by nothing and the apply would go green having changed no
    label.

Checks

Run locally on the versions lint.yml pins (ansible-lint 26.6.0,
ansible-core 2.21.2, yamllint 1.38.0):

  • yamllint over every tracked YAML — clean.
  • ansible-lint --offline on the two changed YAML files — passes the
    production profile with no skip-list
    . The full-tree run reproduces the two
    findings already present on main and adds none.
  • ansible-playbook --syntax-check on install-fireactions.yaml — passes.
  • Both branches of the conditional resolved per host, and confirmed to produce a
    real YAML list (not a string) through to_nice_yaml.

No host was contacted and nothing was applied.

…flag

A host carrying `fireactions_canary: true` in a `host_vars/` file beside the
fireactions inventory now drops the shared labels on all three of its pools for
a flat `self-hosted` + `fireactions-canary`, taking it out of the shared pool
for the length of a rollout window so a kernel or VMM change can be proven on
one host before the other three get it.

THIS COMMIT IS INERT. It lands the mechanism and no canary: no `host_vars/`
directory is created, so all four hosts still resolve to exactly today's
labels. Verified by rendering `fireactions_pools` per host on both trees and
diffing - the config the template emits is identical for all four, so nothing
restarts and no capacity moves. Creating the file is the on switch, and it is
deliberately not committed ahead of a window, because the unattended apply
picks it up within minutes of it landing on main.

A FLAG RATHER THAN THE LABELS THEMSELVES, and that indirection is the whole
design. `install-fireactions.yaml` loads `fireactions_pools` through
`vars_files:`, which outranks inventory `host_vars` - so a `host_vars` file
setting `labels` or `fireactions_pools` would be read, beaten, and apply `ok`
while changing nothing. `fireactions_canary` is a name `vars_files` never
defines, which is why it survives to be read from an expression there.

The condition is extracted to `fireactions_canary_on` rather than repeated
inline: the three pools must differ on their shared label list and must not
differ on the test, and nothing in CI renders these files to catch it if they
did. `| bool` is what makes a quoted "false" or "no" mean false rather than
merely non-empty.

All three pools rather than only `fireactions-small`: leaving medium and large
on the shared set would keep the host serving ordinary org traffic on an
unproven configuration for the whole window, and the runner-image build targets
`fireactions-medium`. Flat rather than tiered, because closing a window is then
one org-scoped string to assert gone instead of three - at the cost, now
documented, that the three tiers become indistinguishable on a canary host.

Also swept, because this change made both stale: the fireactions inventory's
comment claiming every host runs the same definitions, and README.md's tier
table and org-wide capacity arithmetic, which now has a § "The canary" carrying
the figures, the restart-kills-jobs warning and the placement trap - the flag
must sit beside the fireactions inventory, since `pve-vms/all/` holds only
`all_vms` and this play runs against `fireactions_all`.

`yamllint`, `ansible-lint --offline` and `ansible-playbook --syntax-check` pass;
the two changed YAML files pass ansible-lint's `production` profile with no
skip-list.

ADR-0156 §A, §B, §C; ADR-0165 §A, §B.
@kholisrag
kholisrag merged commit 43d0365 into main Aug 6, 2026
1 check passed
@kholisrag
kholisrag deleted the feat/1036-canary-flag branch August 6, 2026 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant