Automated release builds

[joshuagl]

Description of issue or feature request:

Automate releases to reduce potential for human error and protect against developer machine compromise.

Current behavior:

Current release process is very manual and runs on the local machine of a developer making the release: https://github.com/theupdateframework/tuf/blob/7731738590f291041c06b3c525e248d245cb1f33/docs/RELEASE.md

Expected behavior:

Release process is automated as much as possible and, ideally, happens in a hermetic environment on a hosted build service.

At a minimum we should script releases, but even better would be to define a plan to meet SLSA 2 or above https://slsa.dev/levels

[jku]

The first step seems to be research:

• what's the state of the art in other projects (especially ones publishing to pypi and github)?
• how do hosted build service and developer signing work together?
• should we make "fully scripted build process" a reality first? this does very little to protect against malware but might reduce human error
• for "hosted build service" idea, what are the practical steps to get there?

[joshuagl]

A big part of this is indeed figuring out, not only the state of the art but also where they violate best practices (standard practice != best practice). Developer signing is the big wrinkle we've run into when thinking about automating this in the past.

For hosted build service, we almost certainly will continue to use GitHub Actions. This might be a starting point: https://github.com/slsa-framework/github-actions-demo

Agreed that a scripted build/release process (make release?) is the correct starting point.

[jku]

https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ I'm not saying we should do this (because of the signing issue) but nice to see that python.org is willing/trying to define "the right way to do this"...

[joshuagl]

In future a PyPI build farm, developer signing of PyPI with thresholds (PEP 480), or sigstore trust delegations might make this easier, in the near term some options:

• automated workflow with an offline developer verify and sign. We'd need verifiably reproducible artefacts first Verifiably reproducible build artefacts #1269
• developers sign releases i.e. git tags, automation ingests those tags, verifies the signature, then builds and signs dists with a release key

[joshuagl]

Having discussed with one of the SLSA creators, I think the SLSA way to achieve automated builds and have developer signing of releases is.

1. Automate the build
2. Have the builder (i.e. GitHub Actions) produce SLSA provenance, perhaps based on https://github.com/slsa-framework/github-actions-demo
3. Define a policy for what is expected and acceptable in the provenance file, documenting what to check the provenance file against the policy
4. Developers take the output of the automated build (sdist, wheel and provenance file) and if the provenance matches the policy, they sign the release

In future, we could either

• move signing to an automated system with a project secret (not individual developer keys)
  OR
• develop tooling to be run locally which checks a project policy against the provenance file and signs with the developer's key if the provenance file matches the policy

I think a shared secret managed by a service is the way to go, but we might want to wait for sigstore trust delegations for that?

[joshuagl]

For moving the signing to an automated system, it might be as simple as having a new project signing key with the public key stored as a file in the git repo (where our users can easily find it) and the private key is stored in a GitHub secret.
There's even WIP support in sigstore/cosign to generate a key and set up the secrets in GitHub here sigstore/cosign#848

I think we don't want to have individual developers and their PGP keys sign releases, I like binding everything to the GitHub organisation. But if we do want that, we could have automated signed releases by the project which are also approved/signed by the developer who triggered (tagged) the release.

[jku]

private key is stored in a GitHub secret

This is is an issue with normal Github repository secrets as anyone with the ability to create branches in the project practically has the ability to leak the secrets.

Defining a deployment environment might be a way to solve that but I've not looked into those.

[jku]

in the near term some options:

• automated workflow with an offline developer verify and sign. We'd need verifiably reproducible artefacts first Verifiably reproducible build artefacts Verifiably reproducible build artefacts #1269
• developers sign releases i.e. git tags, automation ingests those tags, verifies the signature, then builds and signs dists with a release key

so in practice (let's assume we have ways to verify that build artifacts match):

• developer makes the release changes in git. Release changes are reviewed, merged and tagged (git tag --sign)
• CI release build is triggered by correctly signed release tag: it builds the wheel and tarball and makes them available as build artifacts
• first verify possibility is here: developer can confirm that CI build artifacts match the local build artifacts. We should have a script I suppose
• developer manually approves pypi upload, CI uploads to pypi.org
• second verify possibility is here: developer (and/or a github CI job) can confirm that build artifacts match what is being served by pypi.org. We should have a script for this as well I suppose

The advantage here is mostly auditability: A malicious maintainer can still create a release with something malicious in it, but now github content should reflect that.

is this correct?

[joshuagl]

in the near term some options:

• automated workflow with an offline developer verify and sign. We'd need verifiably reproducible artefacts first Verifiably reproducible build artefacts Verifiably reproducible build artefacts #1269
• developers sign releases i.e. git tags, automation ingests those tags, verifies the signature, then builds and signs dists with a release key

so in practice (let's assume we have ways to verify that build artifacts match):

• developer makes the release changes in git. Release changes are reviewed, merged and tagged

checking we mean git tag --sign?

• CI release build is triggered by correctly signed release tag: it builds the wheel and tarball and makes them available as build artifacts

is the release made (listed on https://github.com/theupdateframework/python-tuf/releases) at this point, or gated on the below verification, along with the PyPI upload?

• first verify possibility is here: developer can confirm that CI build artifacts match the local build artifacts. We should have a script I suppose

This verification prevents tampered changes from being uploaded to PyPI.

• developer manually approves pypi upload, CI uploads to pypi.org
• second verify possibility is here: developer (and/or a github CI job) can confirm that build artifacts match what is being served by pypi.org. We should have a script for this as well I suppose

This verification detects tampering by PyPI. If we sign our PyPI uploads, the verification can be a signature check.

The advantage here is mostly auditability: A malicious maintainer can still create a release with something malicious in it, but now github content should reflect that.

is this correct?

that's right, we want provable linkage between the source in the repo and the artefact being installed. Ultimately we want to prevent tampering, and having documented and verifiable linkage between source and artefacts is necessary to develop a tamper proof process.

[jku]

checking we mean git tag --sign?

Yes that is what I meant (will edit comment).

is the release made (listed on https://github.com/theupdateframework/python-tuf/releases) at this point, or gated on the below verification, along with the PyPI upload?

Hmm, have not considered this yet. I suppose combining github release and pypi upload makes sense, so both gated behind maintainer approval.

The approval btw is something I believe a github actions deploy job can do: we define a "deploy environment" that can have a list of reviewers who are authorized to approve deployments as well as related environment secrets like pypi credentials)

[joshuagl]

The approval btw is something I believe a github actions deploy job can do: we define a "deploy environment" that can have a list of reviewers who are authorized to approve deployments as well as related environment secrets like pypi credentials)

That's very neat!

[jku]

• first verify possibility is here: developer can confirm that CI build artifacts match the local build artifacts. We should have a script I suppose

This verification prevents tampered changes from being uploaded to PyPI.

On second read: this not completely correct I think as tampered changes could still be uploaded to pypi if the infrastructure is tampered cleverly enough... but I think the first check still probably makes sense: a bit of assurance that we are not trying to upload something non-reproducible to PyPI

[joshuagl]

True, I could have couched my language a little more or made my implicit threat model (GitHub itself isn't compromised) more explicit.

[jku]

On more thing to remember: our -dev requirements are currently unversioned. That should not be the case if we're trying to control the environment.

See #1813