K8SPSMDB-486: Fix cluster crash on losing majority due to downscale (#695)

* K8SPSMDB-486: Fix cluster crash on losing majority due to downscale

Co-authored-by: Viacheslav Sarzhan <[email protected]>

Author: egegunes

docs/architecture/decisions/0002-allow-downscaling-in-a-safe-manner.md

@@ -0,0 +1,46 @@
+# Allow downscaling replicaSets in a safe manner
+
+* Date: 2021-06-15
+
+## Status
+
+* Status: Accepted
+
+## Context
+
+Currently there are no mechanisms to stop users downscaling MongoDB replicaSets
+in a manner that causes cluster failure. For example, if a user has 7 members
+of replicaSet and downscales it to 3 members, the cluster is going to lose
+majority (4 members) and the operator won't be able to run `replSetReconfig`
+because of that.
+
+## Considered Options
+
+* Compare current statefulset size with replicaSet size in CR and downscale one
+  by one on each reconciliation until reaching the target size.
+* Compare current statefulset size with replicaSet size in CR and if the target
+  size breaks the majority throw an error.
+* Rather than populating replicaSet members based on a set of pods that are selected
+  according to labels, we could fetch pods one by one in respect to replicaSet size and
+  thus removing the excess pods from replicaSet config *hopefully* before they
+  become inaccessible.
+* ~Use [admission controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) to validate size updates.~ For every CRD, there can be only one webhook in endpoint in a cluster. PSMDB Operator doesn't work cluster wide
+* Document this issue and let the end users decide
+
+## Decision
+
+Chosen option: We will compare the current statefulset size with the replicaSet size in
+the CR and downscale one by one on each reconciliation until reaching the target
+size.
+
+## Consequences
+
+Users can downscale their clusters to any size they like in a single step.
+
+### Negative Consequences
+
+We will be mutating the replicaSet size field in CR to downscale one by one. If
+CR is updated by operator before the replicaSet reaches the target size, we'll
+overwrite the user's changes on size field. For instance, in the `writeStatus`
+method, we're trying to update the status subresource but if it fails we update
+the whole CR.

docs/architecture/decisions/index.md

@@ -3,5 +3,6 @@
 This log lists the architectural decisions for PSMDBO.
 
 - [ADR-0001](0001-record-architecture-decisions.md) - Use Markdown Architectural Decision Records
+- [ADR-0002](0002-allow-downscaling-in-a-safe-manner.md) - Allow downscaling replicaSets in a safe manner
 
 For new ADRs, please use [template.md](template.md) as basis.

pkg/controller/perconaservermongodb/psmdb_controller.go

@@ -225,6 +225,11 @@ func (r *ReconcilePerconaServerMongoDB) Reconcile(request reconcile.Request) (re
 		return reconcile.Result{}, err
 	}
 
+	err = r.safeDownscale(cr)
+	if err != nil {
+		return reconcile.Result{}, errors.Wrap(err, "safe downscale")
+	}
+
 	if cr.ObjectMeta.DeletionTimestamp != nil {
 		err = r.checkFinalizers(cr)
 		return rr, err
@@ -498,6 +503,26 @@ func (r *ReconcilePerconaServerMongoDB) checkConfiguration(cr *api.PerconaServer
 	return nil
 }
 
+func (r *ReconcilePerconaServerMongoDB) safeDownscale(cr *api.PerconaServerMongoDB) error {
+	for _, rs := range cr.Spec.Replsets {
+		sf, err := r.getRsStatefulset(cr, rs.Name)
+		if err != nil && !k8serrors.IsNotFound(err) {
+			return errors.Wrap(err, "get rs statefulset")
+		}
+
+		if k8serrors.IsNotFound(err) {
+			continue
+		}
+
+		// downscale 1 pod on each reconciliation
+		if *sf.Spec.Replicas-rs.Size > 1 {
+			rs.Size = *sf.Spec.Replicas - 1
+		}
+	}
+
+	return nil
+}
+
 func (r *ReconcilePerconaServerMongoDB) getRemovedSfs(cr *api.PerconaServerMongoDB) ([]appsv1.StatefulSet, error) {
 	removed := make([]appsv1.StatefulSet, 0)