Document the new release process for Rustup #3844
Conversation
The Rustup release process has historically been a manual process that involved copying files from S3 to the local machine and back to S3. This introduced a high risk of human error. When modifications to the existing release script became necessary, the decision was made to automate the release process (see rust-lang#3819 for details). The documentation in the dev-guide has been updated to cover the new release process, which is fully automated to produce `beta` releases using GitHub Actions and the [promote-release] tooling. [promote-release]: https://github.com/rust-lang/promote-release
|
Here's a rendered version of the file: https://github.com/jdno/rust-rustup/blob/new-release-process/doc/dev-guide/src/release-process.md |
doc/dev-guide/src/release-process.md
Outdated
| ### Release Artifacts | ||
|
|
||
| The release artifacts are produced by CI when a new commit is merged into the | ||
| `stable` branch, and they are uploaded to the `dev-static` bucket on S3. There, |
There was a problem hiding this comment.
I think we should move this to a different bucket. CI shouldn't have direct access to the static buckets IMO, it also complicates any intermediate processing (e.g. signing or recompression).
There was a problem hiding this comment.
That's fine with me. How about we create a rustup-builds bucket that stores the artifacts, similar to the bucket for rust-lang/rust?
There was a problem hiding this comment.
The config for the new bucket has been created here: rust-lang/simpleinfra#428
|
|
||
| After merging the PR in the previous step, update the commit SHA checksum in | ||
| `rustup-init.sh` to match the latest commit on `master`. Submit the change in a | ||
| PR and merge it. |
There was a problem hiding this comment.
This step sounds surprising to me. If someone updates this to some other commit, how will we notice? Is there an opportunity to adjust the script as part of CI publishing it?
There was a problem hiding this comment.
@Mark-Simulacrum This is the most painful point for me personally during the past 2 releases (v1.27.0 and v1.27.1, for which I actually handled the release process), and I have no idea why this have to be the case.
Well, when we migrated to GitHub Merge Queue in v1.27.0, we have almost broken the whole workflow (which relied on merges instead of rebases and thus assumed shasum stability; to keep the coherence with the old workflow, a 2nd PR is required with GHMQ enabled for script traceability, see #3653 (comment) for more context). The current description is the closest I could get from the "spirit" of the original.
I'd love to see this step being automated. As long as the resulting rustup-init.sh script is traceable via the shasum when it's published onto rustup.sh etc., I personally don't see a particular reason we cannot leave a placeholder on master for example.
There was a problem hiding this comment.
We can look into automating this step, but I would do that after refactoring the release process as-is. This is already quite a big change and I don't want to slow down the implementation or add additional complexity to it.
There was a problem hiding this comment.
Yeah, for sure. Mostly just flagging it since it sounded off.
| the right version number when run `--version`. | ||
|
|
||
| Once the beta release has happened, post a new topic named "Seeking beta testers | ||
| for Rustup $VER_NUM" on the [Internals Forum]. |
There was a problem hiding this comment.
We've automated this for rustc, maybe can reuse that for rustup...
There was a problem hiding this comment.
Happy to create a tracking issue for this so that we can look into it once we've refactored the tooling.
| Once the official release has happened, prepare and push a tag on the latest | ||
| `stable` commit. | ||
|
|
||
| - `git tag -as $VER_NUM -m $VER_NUM` (optionally without `-s` if not GPG |
There was a problem hiding this comment.
Should we do this in promote release? We do for rust and cargo (also always signing the commits with the official key).
There was a problem hiding this comment.
@Mark-Simulacrum Will be cool. So far I have to ping t-release before bedtime, and manually do the tagging the next morning 😓
There was a problem hiding this comment.
If I understand correctly what this mean, I agree it would be cool.
Currently I have an automated process to make sure the script that I download from rustup.rs is the same as a specific tagged release from this repository. Until a few hours ago it was v1.27.1. Now it should be a commit between e2d9e7e and master that would be tagged v1.28.0. But there is no such tag yet.
If I understand correctly, this tagging step is supposed to happen after the file is distributed. And this comment suggests to do it at the same time. Is that right?
Co-authored-by: rami3l <[email protected]>
doc/dev-guide/src/release-process.md
Outdated
| Before making a release, ensure that `rustup-init.sh` is behaving correctly, | ||
| This document describes the process for making a new release of Rustup. It is | ||
| split into the two sections. The first section explains the release process, | ||
| while the second section documents for maintainers how to make a release. |
There was a problem hiding this comment.
"documents how to make a release"? The "for maintainers" part reads a little awkwardly.
(Also "make" feels a little vague here, can we make that more precise?)
There was a problem hiding this comment.
I've rewritten the into a bit, but I'm not sure how much better it is now. 😅
|
Thanks for working on this! |
Based on a discussion[^1] with the release team, the decision has been made to create a new S3 bucket for Rustup build artifacts. This enable us to revoke the access that Rustup's CI has to the release bucket, improving security and reducing the risk of malicious or accidental compromise of Rustup releases. [^1]: rust-lang#3844 (comment)
|
@jdno what's your plan for driving the new release process to completion? We're starting to think about how/when we want to publish our next release so it would be good to understand how much of a dependency we have on you and what that means for the schedule. |
|
I've started working on it, but I can't give a good estimate on how long it'll take. My hope is to get most of the implementation work done in the next two weeks. I don't want to be a blocker, though, so my work is strictly additive right now. The current process will continue to work until we have a full replacement. So regarding dependencies or the schedule, there shouldn't be any. |
|
I'd strongly recommend adding roll back process for when a deployment causes failure like what happened in 1.28.0. I'd like to echo and amplify the comments in #4211 about rolling back to make sure that there's more than one voice calling for this practice. The 1.28.0 release highlighted that Rustup is a load bearing piece of software. Having a failed deployment cause many downstream failures like this should have resulted in a rollback within about an hour, not a roll forward a day later. I base this opinion having participated (as an owner and as someone affected by) in many post-mortems at a large internet company. The consensus engineering best practice is to rollback first, and think about how to solve the greater problem second. It's rare that this advice should be ignored, and there doesn't seem like there's a any good rationale that's been expressed here why that should have been so here. The users that migrated to using the new rustup changes would generally have been a very small number compared to those who were affected by this, and those users were actively aware of the changes. The impacted users were generally not made aware until this caused failure in their systems. I want to explicitly state this comment is not intended to throw shade at all on anyone involved in 1.28. It is only meant to constructively improve the situation going forward. |
@joshka I second this. In fact, I have already mentioned it on Zulip:
I added that rustup could be modified to make use of the Let's leave this thread for the release-related work: I've made #4240 for the rustup-related part of this problem. |
The Rustup release process has historically been a manual process that involved copying files from S3 to the local machine and back to S3. This introduced a high risk of human error. When modifications to the existing release script became necessary, the decision was made to automate the release process (see #3819 for details).
The documentation in the dev-guide has been updated to cover the new release process, which is fully automated to produce
betareleases using GitHub Actions and the promote-release tooling.