Figure out noble upgrade cadence plan

[legoktm]

Description

Instead of upgrading every single instance at the exact same time (once we push a deb), I think it would be better to do some sort of staged rollout.

My proposal would be that on package upgrade, each instance generates a random number (1-5) and stores it somewhere. In theory we've now split all the securedrop servers into 5 groups.

Then, in another file we ship with the package (possibly the upgrade script itself) we have a number we control. if we set it to 1, we'll upgrade ~20% of servers. Then we can do another deb package release to bump it to 2 to upgrade ~40% of all servers. And so on.

I also think this mechanism should be split for both app and mon. We should upgrade all mon servers to 100% and then do all the app servers.

[legoktm]

Also, for instances with hands-on administrators, we can give them a heads up and let them manually run the migration script before our auto/forced migration.

[nathandyer]

(Early thoughts, not fully formed)

I like the idea of admins being in control of the migration, unless there's a situation where there's not a hands-on admin and we run up against the deadline.

What about an alternate approach that might look like:

0. We select a hard deadline date for the auto upgrade (for arguments sake, March 1st). Prior to March 1st, Admins can manually kick off the upgrade from an Admin Workstation. The mechanism for which might be:

1. We publish the package with the noble upgrade script on a new (temporary?) Apt server

2. We update securedrop-admin on the Admin Workstations with a securedrop-admin noble-upgrade command, which essentially adds the new Apt server to the sources list on both app and mon.

3. Admins can manually update before the deadline date

4. When the deadline happens, we promote the packages to the normal apt prod servers and "force" the update

5. We retire the temporary Apt server

[zenmonkeykstop]

is the the idea behind phasing it to have some level of feedback as to how it's going and to not have a massive volume of support requests if things go wrong? If so, (building on @nathandyer's proposal) we kindof get that already if we give folks the option to migrate ahead of time and get the feedback of the first ones off the ice.

[zenmonkeykstop]

We don't need temporary apt servers or anything tho - we can ship the changes packaged as normal and just have EOL checks.

[legoktm]

is the the idea behind phasing it to have some level of feedback as to how it's going and to not have a massive volume of support requests if things go wrong

Yes, and (if things go poorly) we shouldn't take down every single SecureDrop all at once.