"Split Mode" operation

[chuckadams]

If we get blocked by upstream, we'll want some procedures in place to handle this. Some endpoints like /events or /browse-happy can be left broken. Theme and plugin info can stop passing through, and rely only on what AspireSync feeds it (Sync can run anywhere). And some like plugin and core update checks MUST continue operating in some fashion: having the user deactivate AU leaves them open to supply chain attacks like those that have already been perpetrated by upstream.

When we go fully-federated, we'll want the system to react and hopefully self-heal, but if not, fail a little more gracefully than 5xx errors. For now, some automated alert followed by a checklist and some scripts (e.g. artisan commands) will do. We don't have to adapt instantly, I just don't want to get caught with our pants down.

The name "split mode" descends from netsplits in IRC, but I'm open to a better term. Make it catchy 🙃

[asirota]

Is this like a toggle to flip into this mode or does AC auto-negotiate into this mode when upstream is blocking?

Some Catchy names

• kernel panic
• upstream down
• worg block

[chuckadams]

How we get into split mode is implementation detail, but I imagine there would be at least a manual toggle, as well as some conditions that can be automated, such as x% of requests failing for n minutes. The conditions are definitely worth hammering out, but right now I'm thinking more about how we would operate in that mode -- both the software and the people. Checklists, etc. When we have procedures in place, we can run some fire drills and try them out.

"upstream down" has a nice ring to it, but might be inaccurate if upstream isn't actually down... "Degraded mode" perhaps? Not very catchy, but it does have clarity. Might be more than one degraded mode possible, but that's also something to think about.

[toderash]

"unreachable truth" or "unknowable truth" or ;-) "canon misfire"

not a proposed mode name so much as a general description of the problem.

Think of it this way: if for any reason (dns, routing, remote site borked, etc) the canonical source of truth for a package can't be reached, you must (a) issue a warning, and (b) ask your trusted peers if they've got a more recent copy than you do.

error conditions include domain not resolved, no route to host, remote return of 40x or 50x errors, and (lastly) network timeout or unrecognized response.

We might set a threshold of n errors in n minutes to rule out temporary internet hiccups before checking alternate sources. For a distributed/federated model, this is not a degraded mode, it's necessarily by design. Right now we're thinking mostly of w-org as holding most of the sources, but there are already plugins that have left w-org and we've committed to serve first-party ones not on w-org, so the solution cannot be w-org specific, though it must also deal with w-org.