Skip to content

ci: gate releases on an approved environment and sign, attest and SBOM release images - #1985

Open
arnaubennassar wants to merge 12 commits into
developfrom
ci/release-gating-attestation
Open

arnaubennassar wants to merge 12 commits into
developfrom
ci/release-gating-attestation

Conversation

@arnaubennassar

@arnaubennassar arnaubennassar commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🔄 Changes Summary

Addresses the release-path finding in agglayer/issues#112: today any v*.*.* tag push publishes production images to ghcr.io/agglayer/aggkit with no review, a moving X.Y tag that a later tag silently overwrites, and no signature, SBOM or verifiable provenance. After this change no single account can publish a production image.

  • release.yml is now preflight -> approve -> build -> publish.
    • preflight (read-only) runs the new scripts/release/check_tag_ancestry.sh. The ref must be vX.Y.Z or vX.Y.Z-<pre>, and the tagged commit must be an ancestor of origin/develop or an origin/release/* branch. It also refuses to continue if the version is already published, so versions are immutable.
    • approve uses the release environment (required reviewers, prevent self-review). Nothing is pushed to the registry before a reviewer approves, because build pushes by digest.
    • build produces each platform image with sbom: true and provenance: mode=max.
    • publish creates the manifest lists, signs each with cosign (keyless) and adds a GitHub build-provenance attestation pushed to the registry. It is the only job with id-token: write and attestations: write. Top-level permissions: {}, no job has contents: write, and the new actions are pinned by commit SHA.
  • Branch builds move to ghcr.io/agglayer/aggkit-dev (build-push-docker-image.yml). Production images are only published by release.yml. Tag naming of branch builds is unchanged. No workflow in the repo consumes the old ghcr.io/agglayer/aggkit:<branch>_... tags.
  • Tags policy: published tags are exactly X.Y.Z[-pre], X.Y.Z[-pre]-no_shell and X.Y.Z[-pre]-dev. The moving X.Y, X.Y-no_shell and X.Y-dev tags are no longer published, and there is no latest. Branch and PR tag types are removed from the release metadata. Deployments should pin an exact version or, better, a digest.
  • Production manifest fix: the digest artifacts of the production variant were selected with a pattern that also matched the -dev platform images, so the production manifest list could include dev images. publish now selects only the platform images of its own variant.
  • New scripts/release/check_release_settings.sh: a read-only (GET only) audit of the GitHub controls the release path relies on (tag ruleset, release environment, branch protection, teams, package ACL). It prints PASS/FAIL/MANUAL per control and exits non-zero on any FAIL (--json is supported).
  • Docs: docs/release_verification.md (operators: cosign verify, gh attestation verify, SBOM inspection, pin by digest) and docs/release_admin_runbook.md (admins), linked from docs/SUMMARY.md and docs/release_lifecycle.md. Images published before this change are unsigned.

⚠️ Breaking Changes

  • 🛠️ Config: none
  • 🔌 API/CLI: none
  • 🗑️ Deprecated Features: the moving X.Y, X.Y-no_shell and X.Y-dev image tags are no longer published. Consumers following them must pin an exact X.Y.Z version or a digest.

📋 Config Updates

  • 🧾 None. No aggkit configuration changes.

✅ Testing

  • 🤖 Automatic:
    • bash scripts/release/check_tag_ancestry_test.sh covers a commit on develop, on release/0.11 only, on an unrelated branch, a malformed tag and a non-tag ref.
    • bash scripts/release/check_release_settings_test.sh uses a stub gh and covers all-pass, a missing tag ruleset, an environment without prevent-self-review, and the package ACL reported as manual.
    • shellcheck scripts/release/*.sh is clean.
    • actionlint on both workflows: see the note below.
  • 🖱️ Manual: the real release path needs the GitHub settings below and a tag push, so it is not exercised by this PR. The runbook describes a first-release dry run with an rc tag (the run waits on approve, the author cannot approve, a tag on an unreviewed branch fails in preflight, cosign verify succeeds).

actionlint note: arm-runner-2204 (and amd-runner-2204) are self-hosted runner labels that actionlint does not know by default. This is already the case on develop. With those labels declared in an actionlint config, release.yml is clean. The remaining shellcheck info/style findings in build-push-docker-image.yml are pre-existing and untouched by this PR.

🐞 Issues

  • Fixes the release-path finding in agglayer/issues#112

📝 Notes

Settings that must be applied by admins before the next release (step-by-step in docs/release_admin_runbook.md; verify with scripts/release/check_release_settings.sh). Without them the gates in this PR are not enforced:

  1. Teams release-approvers and release-manager.
  2. Environment release: required reviewers release-approvers, prevent self-review, deployment policy limited to tags v*, no admin bypass.
  3. Tag ruleset on refs/tags/v* (creation, update, deletion and non-fast-forward restricted; bypass only release-manager; organization admins may also bypass, and the audit log records it).
  4. develop and release/* require a PR with at least one approval, no direct pushes, force-pushes or deletion.
  5. GHCR package aggkit: disable inherited repository access and grant Actions write only to agglayer/aggkit; configure aggkit-dev after its first build.

Tag workflows run the workflow file from the tagged commit, so the ancestry check is only a guarantee together with items 3 and 4. Residual risks (colluding maintainers, other workflows with packages: write) and an optional further hardening are in the runbook.

arnaubennassar and others added 5 commits November 17, 2026 10:25
Verify that a release tag points at a commit reachable from develop or a
release/* branch, and expose the version without the leading v.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Branch builds from the build-push workflow now go to ghcr.io/<owner>/<repo>-dev
instead of the production package, so every tag and manifest list from this
workflow lands in the development package. Production images are published by
the release workflow only.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Checks the tag ruleset, release environment, branch rules, teams and
GHCR package settings through read-only GitHub API calls.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Split the release workflow into preflight, approve, build and publish jobs:

- preflight checks that the ref is a release tag whose commit is on develop
  or a release/* branch, and that the version is not already published
- approve waits on the protected release environment before anything is
  pushed to the registry
- build publishes only exact version tags (no moving X.Y or latest tags)
  and adds an SBOM and mode=max provenance to every image
- publish creates the manifest lists, signs them with cosign keyless and
  pushes a GitHub build-provenance attestation for each

The production manifest list now contains only the production platform
images; previously it also picked up the -dev images. Runs for the same tag
no longer cancel each other.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Document the published image tags, how to verify signatures and build
provenance, and the repository settings that protect releases.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@arnaubennassar arnaubennassar self-assigned this Nov 17, 2026
@arnaubennassar

Copy link
Copy Markdown
Contributor Author

@claude

@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @arnaubennassar's task in 1m 25s —— View job


Review of #1985: release gating, signing and attestation

The overall design is sound. The preflight, approve, build, publish split puts the approval gate before any registry push. Permissions are minimal, with id-token and attestations confined to publish. The new actions are SHA-pinned. The prod-manifest fix is correct: the *-dev / non--dev filter in the Create manifest list step now separates the variants. The jq regex changes (\\. escaping, test() anchoring on -dev$) also look right.

I read the diff and the scripts but did not run anything, so none of the findings below were run. I could not check that the new action SHAs match their tags (cosign-installer v4.1.3, attest-build-provenance v4.2.2) because network commands needed approval. Please confirm them against upstream.

🔴 Required

  1. check_release_settings.sh never checks "no admin bypass" on the release environment. The PR description and runbook step 2 require can_admins_bypass: false, but the script has no reference to can_admins_bypass (grep count is 0). check_environment (scripts/release/check_release_settings.sh:125) therefore reports PASS while an admin can skip the approval gate, which is the main control here. Add a check on .can_admins_bypass == false. Add it to the fixtures and tests as well: testdata/pass/repos_acme_widget_environments_release lacks the field, so it needs "can_admins_bypass": false, plus a failing case.

🟡 Suggested

  1. The signing job runs on a self-hosted runner. publish (release.yml:222, arm-runner-2204) holds id-token: write, attestations: write and packages: write. Anything else that runs on those runners (for example PR jobs, if the runners are shared or not ephemeral) could tamper with cosign, the Docker config or the OIDC token. Manifest creation and signing do not need arm hardware, so I would use ubuntu-latest for publish. If that is not possible, confirm the runners are ephemeral and dedicated.

  2. The branch-rule check gives false FAILs and misses gaps.

    • check_branch_rules reads rules/branches/<b>, which only returns ruleset rules. Runbook step 4 says classic branch protection is equivalent, but a repo using that gets FAIL. Fall back to branches/<b>/protection, or document that only rulesets are supported.
    • It only checks the newest release/* branch, not all of them.
    • matching-refs and rulesets are fetched without --paginate, so with more than 30 release branches "newest" can be wrong.
    • Bypass actors on the branch ruleset are not inspected, though the runbook says "do not add bypass actors".
  3. Tag-ruleset bypass is inconsistent between the docs and the script. The PR description says bypass is "only release-manager", but the script (check_release_settings.sh:114) and fixture rulesets_7 accept OrganizationAdmin too. The runbook also allows it. If admin bypass of tags is acceptable, say so in the PR description and runbook. If not, drop it from the script.

  4. The tag regex is looser than semver. check_tag_ancestry.sh:21 accepts v01.2.3 and v1.2.3-rc.01, which docker/metadata-action's semver type rejects. The run would then pass preflight and approve and fail in build, after a reviewer has spent an approval. Tighten it to semver (no leading zeros). Add -dev-style prerelease handling too: v1.2.3-dev is a valid prerelease, but its prod tag 1.2.3-dev collides with the dev-variant tag of v1.2.3. Reject prereleases that equal or end in -dev.

  5. A partial failure in publish cannot be recovered by re-running the whole workflow. The manifest is pushed before signing. If cosign sign or the attestation fails, the version is published but unsigned, and a full re-run fails preflight as "already published". The docs should say: use "Re-run failed jobs" (which skips preflight), or cut a new version. Ideally sign before the tags become visible, or document that unsigned versions must be abandoned.

  6. The preflight registry check has no login. docker buildx imagetools inspect runs unauthenticated. If the aggkit package is ever private or internal, it returns unauthorized, and the script (correctly, failing closed) aborts every release. Either state that the package must be public, or log in with packages: read. The "published check" also only treats the text not found as absent, which is brittle but fails safe.

  7. The new tests are not run in CI. Nothing in the Makefile or workflows calls check_tag_ancestry_test.sh, check_release_settings_test.sh or shellcheck scripts/release/*.sh. Add a small job or a make target, otherwise they will rot.

🟢 Optional

  • on: workflow_dispatch: is still present in release.yml. It is harmless because preflight rejects non-tag refs, but removing it narrows the surface.
  • check_tag_ancestry.sh does its own git fetch while actions/checkout with fetch-depth: 0 already fetched branches and tags. The extra fetch is redundant, and it also depends on unauthenticated read access to the repo.
  • Signing and push-to-registry add sha256-<digest>.sig and referrer tags to the package. The "tags are exactly X.Y.Z[-pre]..." wording in the docs could note that signature and attestation artifacts also appear.
  • check_release_settings.sh:149: problems="${problems% ;}" is a no-op. Clean up the trailing-separator handling.

Template summary

🚀 What's New

Release publishing is now gated: release.yml runs preflight (tag format, commit ancestry on develop or release/*, version not already published), then approve (required release environment review), then build (SBOM and max provenance per platform), then publish (manifest lists, cosign keyless signature, GitHub build-provenance attestation). Branch builds move to ghcr.io/agglayer/aggkit-dev. Adds scripts/release/check_tag_ancestry.sh and check_release_settings.sh with tests, plus docs/release_verification.md and docs/release_admin_runbook.md.

🐛 Bug Fixes

The production manifest list no longer includes -dev platform images (digest selection is now per variant). The image-name regex now escapes the dots in \d+\.\d+\.\d+.

📋 Config Updates

None. No aggkit configuration changes. Admins must configure the GitHub settings listed in the runbook (teams, release environment, tag ruleset, branch rules, package ACL) before the next release.

⚠️ Breaking Changes

The moving X.Y, X.Y-no_shell and X.Y-dev image tags and latest are no longer published. Consumers must pin an exact X.Y.Z or a digest. Branch builds are no longer published under ghcr.io/agglayer/aggkit.

--- · branch ci/release-gating-attestation

arnaubennassar and others added 5 commits November 17, 2026 16:09
…ase tags

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tors

Check the release environment for administrator bypass, all release/* branches
(paginated) and bypass actors on the branch rulesets. State that only rulesets
are read and that organization admins may bypass the tag ruleset (audited).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ements

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@arnaubennassar

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I checked the action SHAs against upstream (sigstore/cosign-installer v4.1.3 is 756fe2d2…, actions/attest-build-provenance v4.2.2 is 4d101475…; both match), and pushed fixes in 867add30, 88d02077 and 473efcdc. Point by point:

Required

  1. Admin bypass: fixed. check_release_settings.sh now fails when the release environment has can_admins_bypass other than false. The pass fixtures carry the field and a new admin_bypass fixture and test cover the failure.

Suggested
2. Runner: fixed. publish now runs on ubuntu-latest.
3. Branch rules: fixed. The calls are paginated, every release/* branch is checked, and a ruleset with bypass actors on those branches now fails. Only rulesets are supported (not classic branch protection); this is stated in the runbook and the script header.
4. Tag bypass: kept as is, with the wording corrected. Organization admins may bypass the tag ruleset and the audit log records it; the runbook, script comment and PR description now say so.
5. Tag regex: fixed. Strict semver (no leading zeros) and prereleases that are dev or end in -dev are rejected, with tests.
6. Partial failure in publish: documented. The runbook says to use "Re-run failed jobs" or cut a new version, and not to trust a published version that is unsigned. I did not reorder signing before tag creation, since the tags are created by the same manifest push that signing refers to.
7. Anonymous published-check: documented. The runbook states the aggkit package must stay public for it. It fails closed otherwise.
8. Running the new tests in CI: not done in this PR. It would mean changing the Makefile or other workflows, which this change deliberately does not touch. Better as a small follow-up.

Optional

  • workflow_dispatch is kept: preflight rejects non-tag refs, and it is useful for re-running a release on an existing tag.
  • The extra git fetch in check_tag_ancestry.sh is kept: it makes the script correct when run outside a full-depth checkout, and SKIP_FETCH=1 covers the tests.
  • The note about signature and attestation artifacts is added to the docs.
  • The no-op problems line is removed.

The preflight job pinned actions/checkout to v5.1.0 while the build job
uses v7.0.1. Pin preflight to the same commit so both jobs run the same
checkout action.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant