MGS driven SP components left in invalid state should have a way to recover from failed updates

[karencfv]

Specifically in the case of the RoT, it could be left in an un-updateable state. In the case of there being a bad signature check on the alternate RoT image, either the pending-persistent or just the persistent boot preference will need to be set to the good image before proceeding.

We could add a variant to UpdateAttemptStatus called RestoringComponent (or similar) and have apply_update set the status to this new status if the component needs to be set to a different state before an update. This could happen after a precheck.

Needs oxidecomputer/hubris#2050 to fully work

To be able to differentiate whether an RoT has a mismatch with the active version and persistent boot preference or transient/pending boot preference are not empty due to a failed update or an ongoing update we'll need oxidecomputer/hubris#2066 which will be available soon.

[davepacheco]

It seems a little hard to reason about what to do in this situation because it's hard to imagine how we got into this situation without a bug in the planner or executor.

[karencfv]

There's additional context in this comment thread.

Also, it's worth noting that a non-empty transient_boot_preference/pending_persistent_boot_preference or a mismatch between the active slot and persistent_boot_ preference doesn't necessarily mean there is an ongoing update. We need a way to handle the case where it's a failure. The device needs to be reset before proceeding.

There are other ways a device update may fail, @lzrd has more context on those.

[davepacheco]

Is this a dup of #8349? Both of these seem to talk about identifying the case of a failed update due to a signature mismatch and then responding to it.

[karencfv]

Ha! yes. Now I'm remembering that I wrote this one with the intention of it superseding #8349, but clearly I forgot to close the other issue 😅 . There are now more comments on the other one. Which one do you think I should close @davepacheco ?

[davepacheco]

I'd keep this one. Even though there's more discussion in the other one, I think this one better reflects something actionable (recover a device from a bad update). That said, my gut is to do nothing here for the foreseeable future -- it's hard for me to see how a situation where the control plane attempted to deploy the wrong artifact to a device wouldn't be a support call.

[karencfv]

I'd keep this one. Even though there's more discussion in the other one, I think this one better reflects something actionable (recover a device from a bad update)

Cool, done.

That said, my gut is to do nothing here for the foreseeable future -- it's hard for me to see how a situation where the control plane attempted to deploy the wrong artifact to a device wouldn't be a support call.

That makes sense. Also, we don't have the mechanisms from the Hubris side in place yet.