K8SPSMDB-1219: PBM multi storage support (#1843)

* K8SPSMDB-1219: PBM multi storage support

Operator always supported multiple storages but didn't have native
support for having multiple backup storages until v2.6.0. In operator we
were reconfiguring PBM every time user selects a storage for their
backups/restores different than the previous storage. This was causing
long wait periods esp. in storages with lots of backups due to resync
operation.

Another limitation was forcing users to have only 1 backup storage if
they want to enable point-in-time-recovery. In case of multiple
storages, PBM would upload oplog chunks to whatever storage is last used
by a backup/restore and this would make consistent recovery impossible.

PBM v2.6.0 added native support for multiple storages and these changes
introduce it to our operator:

* User can have one main storage in PBM configuration. Any other
  storages can be added as profiles.

Main storage can be found in:
```
kubectl exec cluster1-rs0-0 -c backup-agent -- pbm config
```

This commit introduces a new field `main` in storage spec:
```
storages:
  s3-us-west:
    main: true
    type: s3
    s3:
      bucket: operator-testing
      credentialsSecret: cluster1-s3-secrets
      region: us-west-2
````

If user only has 1 storage configured in `cr.yaml`, operator will
automatically use it as main storage. If more than 1 storage is
configured, one of them must have `main: true`. User can't have more
than 1 storage with `main: true`.

If user changes main storage in cr.yaml, operator will configure PBM
with the new storage and start resync.

Any other storage in `cr.yaml` will be added to PBM as a profile.

User can see profiles using cli:
```
kubectl exec cluster1-rs0-0 -c backup-agent -- pbm profile list
```

When user adds a new profile to `cr.yaml`, operator will add it to PBM
but won't start resync.

**`pbm config --force-resync` only start resync for the main storage.**

To manually resync a profile:
```
kubectl exec cluster1-rs0-0 -c backup-agent -- pbm profile sync <storage-name>
```

If user starts a restore using a backup in a storage configured as
profile, operator will start resync operation for profile and block
restore until resync finishes.

Note: Profiles are also called external storages in PBM documentation.

If user has multiple storages in `cr.yaml` and changes main storage
between them, operator:
1. configures PBM with the new main storage.
2. adds the old main as a profile.
2. deletes profile for the new main storage.

If user configures `backupSource` in backups/restores:
* if `cr.yaml` has no storages configured, operator configures PBM with
  storage data in `backupSource` field. This storage will effectively be
  the main storage until user adds a storage to `cr.yaml`. After a
  storage is configured PBM configuration will be overwritten and
  `backupSource` storage will be gone.
* if `cr.yaml` has a storage configured, operator adds `backupSource`
  storage as a profile.

* Oplog chunks will be only be uploaded to main storage.

User can use any backup as base backup for point-in-time-recovery.

* Incremental backup chains all need to be stored in the same storage.

TBD after #1836 merged.

---

Other significant changes in operator behavior:

* Operator now configures automatically configures PBM on a fresh
  cluster.

  Before this changes, PBM was not configured until user starts a
  backup/restore after deploying a fresh cluster. Now, PBM will directly
  configured with main storage in `cr.yaml` and resync will be started
  in the background.

  There's a new field in `PerconaServerMongoDB` status:
  `backupConfigHash`. Operator will maintain hash of the current PBM
  configuration and reconfigures PBM if hash is changed. Fields in
  `spec.backup.pitr` are excluded from hash calculation, they're handled
  separately.
* If `PerconaServerMongoDB` is annotated with `percona.com/resync-pbm=true`,
  operator will start resync operation both for main storage and
  profiles. Resync for profiles are started with the equivalent of `pbm
  profile sync --all`. These resync operations will be run in the
  background and will not block the reconciliation.
* If a backup that has `percona.com/delete-backup` finalizer is deleted,
  operator will only delete oplogs chunks if it's in main storage.

* fix users

* fix repeatedly sending backup commands

* fix users

* address review comments

* assert err in main storage unit test

* don't use break labels

---------

Co-authored-by: Viacheslav Sarzhan <[email protected]>

Author: egegunes

build/Dockerfile

@@ -7,6 +7,7 @@ RUN go mod download
 
 ARG GIT_COMMIT
 ARG GIT_BRANCH
+ARG BUILD_TIME
 ARG GO_LDFLAGS
 ARG GOOS=linux
 ARG TARGETARCH
@@ -16,7 +17,7 @@ COPY . .
 
 RUN mkdir -p build/_output/bin \
     && GOOS=$GOOS GOARCH=${TARGETARCH} CGO_ENABLED=$CGO_ENABLED GO_LDFLAGS=$GO_LDFLAGS \
-       go build -ldflags "-w -s -X main.GitCommit=$GIT_COMMIT -X main.GitBranch=$GIT_BRANCH" \
+       go build -ldflags "-w -s -X main.GitCommit=$GIT_COMMIT -X main.GitBranch=$GIT_BRANCH -X main.BuildTime=$BUILD_TIME" \
            -o build/_output/bin/percona-server-mongodb-operator \
                cmd/manager/main.go \
     && cp -r build/_output/bin/percona-server-mongodb-operator /usr/local/bin/percona-server-mongodb-operator

cmd/manager/main.go

@@ -35,6 +35,7 @@ import (
 var (
 	GitCommit string
 	GitBranch string
+	BuildTime string
 	scheme    = k8sruntime.NewScheme()
 	setupLog  = ctrl.Log.WithName("setup")
 )
@@ -69,7 +70,7 @@ func main() {
 	ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
 
 	setupLog.Info("Manager starting up", "gitCommit", GitCommit, "gitBranch", GitBranch,
-		"goVersion", runtime.Version(), "os", runtime.GOOS, "arch", runtime.GOARCH)
+		"buildTime", BuildTime, "goVersion", runtime.Version(), "os", runtime.GOOS, "arch", runtime.GOARCH)
 
 	namespace, err := k8s.GetWatchNamespace()
 	if err != nil {

config/crd/bases/psmdb.percona.com_perconaservermongodbs.yaml

@@ -321,6 +321,8 @@ spec:
                           required:
                           - path
                           type: object
+                        main:
+                          type: boolean
                         s3:
                           properties:
                             bucket:
@@ -18790,6 +18792,8 @@ spec:
             properties:
               backup:
                 type: string
+              backupConfigHash:
+                type: string
               backupVersion:
                 type: string
               conditions:

deploy/bundle.yaml

@@ -1017,6 +1017,8 @@ spec:
                           required:
                           - path
                           type: object
+                        main:
+                          type: boolean
                         s3:
                           properties:
                             bucket:
@@ -19486,6 +19488,8 @@ spec:
             properties:
               backup:
                 type: string
+              backupConfigHash:
+                type: string
               backupVersion:
                 type: string
               conditions:

deploy/crd.yaml

@@ -1017,6 +1017,8 @@ spec:
                           required:
                           - path
                           type: object
+                        main:
+                          type: boolean
                         s3:
                           properties:
                             bucket:
@@ -19486,6 +19488,8 @@ spec:
             properties:
               backup:
                 type: string
+              backupConfigHash:
+                type: string
               backupVersion:
                 type: string
               conditions:

deploy/cw-bundle.yaml

@@ -1017,6 +1017,8 @@ spec:
                           required:
                           - path
                           type: object
+                        main:
+                          type: boolean
                         s3:
                           properties:
                             bucket:
@@ -19486,6 +19488,8 @@ spec:
             properties:
               backup:
                 type: string
+              backupConfigHash:
+                type: string
               backupVersion:
                 type: string
               conditions:

e2e-tests/custom-replset-name/conf/some-name.yml

@@ -12,6 +12,7 @@ spec:
     serviceAccountName: percona-server-mongodb-operator
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/custom-users-roles-sharded/conf/some-name-rs0.yml

@@ -63,6 +63,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/custom-users-roles/conf/some-name-rs0.yml

@@ -63,6 +63,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/data-at-rest-encryption/conf/some-name-unencrypted.yml

@@ -12,6 +12,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:main-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret
@@ -82,4 +83,4 @@ spec:
         resources:
           requests:
             storage: 1Gi
-    size: 3
+    size: 3

e2e-tests/data-at-rest-encryption/conf/some-name.yml

@@ -12,6 +12,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:main-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/demand-backup-physical-sharded/conf/some-name-sharded.yml

@@ -11,6 +11,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/demand-backup-physical/conf/some-name-arbiter-nv.yml

@@ -13,6 +13,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/demand-backup-physical/conf/some-name.yml

@@ -13,6 +13,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/demand-backup-sharded/conf/some-name-rs0.yml

@@ -11,6 +11,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/demand-backup/conf/some-name-rs0.yml

@@ -11,6 +11,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/expose-sharded/conf/some-name-rs0.yml

@@ -11,6 +11,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/functions

@@ -68,7 +68,9 @@ fi
 KUBE_VERSION=$(kubectl version -o json | jq -r '.serverVersion.major + "." + .serverVersion.minor' | $sed -r 's/[^0-9.]+//g')
 
 log() {
+	set +o xtrace
 	echo "[$(date +%Y-%m-%dT%H:%M:%S%z)]" $*
+	set_debug
 }
 
 set_debug() {
@@ -198,15 +200,14 @@ wait_backup_agent() {
 	local agent_pod=$1
 
 	set +o xtrace
-	retry=0
-	echo -n $agent_pod
-	until [ "$(kubectl_bin logs $agent_pod -c backup-agent | egrep -v "\[ERROR\] pitr: check if on:|node:|starting PITR routine|\[agentCheckup\]" | cut -d' ' -f3- | tail -n 1)" == "listening for the commands" ]; do
+	local retry=0
+	echo -n "waiting for pbm-agent to be ready in ${agent_pod}..."
+	until kubectl_bin logs $agent_pod -c backup-agent | grep "listening for the commands"; do
 		sleep 5
 		echo -n .
 		let retry+=1
 		if [ $retry -ge 360 ]; then
-			kubectl_bin logs $agent_pod -c backup-agent \
-				| tail -100
+			kubectl_bin logs $agent_pod -c backup-agent | tail -100
 			echo max retry count $retry reached. something went wrong with operator or kubernetes cluster
 			exit 1
 		fi
@@ -341,6 +342,7 @@ wait_restore() {
 	local target_state=${3:-"ready"}
 	local wait_cluster_consistency=${4:-1}
 	local wait_time=${5:-1780}
+	local ok_if_ready=${6:-0}
 
 	set +o xtrace
 	retry=0
@@ -351,6 +353,10 @@ wait_restore() {
 		echo -n .
 		let retry+=1
 		current_state=$(kubectl_bin get psmdb-restore restore-$backup_name -o jsonpath='{.status.state}')
+		if [[ ${ok_if_ready} == 1 && ${current_state} == 'ready' ]]; then
+			echo "OK"
+			break
+		fi
 		if [[ $retry -ge $wait_time || ${current_state} == 'error' ]]; then
 			kubectl_bin logs ${OPERATOR_NS:+-n $OPERATOR_NS} $(get_operator_pod) \
 				| grep -v 'level=info' \
@@ -364,7 +370,7 @@ wait_restore() {
 			exit 1
 		fi
 	done
-	echo
+	echo "OK"
 	set_debug
 
 	if [ $wait_cluster_consistency -eq 1 ]; then
@@ -460,10 +466,17 @@ deploy_minio() {
 		kubectl_bin create svc -n ${OPERATOR_NS} externalname minio-service --external-name="minio-service.${namespace}.svc.cluster.local" --tcp="9000"
 	fi
 
-	# create bucket
+	create_minio_bucket operator-testing
+}
+
+create_minio_bucket() {
+	local bucket=$1
+
 	kubectl_bin run -i --rm aws-cli --image=perconalab/awscli --restart=Never -- \
-		bash -c 'AWS_ACCESS_KEY_ID=some-access-key AWS_SECRET_ACCESS_KEY=some-secret-key AWS_DEFAULT_REGION=us-east-1 \
-        /usr/bin/aws --endpoint-url http://minio-service:9000 s3 mb s3://operator-testing'
+		bash -c "AWS_ACCESS_KEY_ID=some-access-key \
+		AWS_SECRET_ACCESS_KEY=some-secret-key \
+		AWS_DEFAULT_REGION=us-east-1 \
+        /usr/bin/aws --endpoint-url http://minio-service:9000 s3 mb s3://${bucket}"
 }
 
 deploy_vault() {
@@ -848,11 +861,20 @@ compare_mongo_cmd() {
 	local suffix="$4"
 	local database="${5:-myApp}"
 	local collection="${6:-test}"
+	local sort="$7"
 
-	run_mongo "use ${database}\n db.${collection}.${command}()" "$uri" "mongodb" "$suffix" \
+	local full_command="db.${collection}.${command}()"
+	if [[ ! -z ${sort} ]]; then
+		full_command="${full_command}.${sort}"
+	fi
+
+	log "running ${full_command} in ${database}"
+
+	run_mongo "use ${database}\n ${full_command}" "$uri" "mongodb" "$suffix" \
 		| egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:' \
 		| $sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/' \
 			>$tmp_dir/${command}${postfix}
+
 	diff -u ${test_dir}/compare/${command}${postfix}.json $tmp_dir/${command}${postfix}
 }
 

e2e-tests/mongod-major-upgrade/run

@@ -19,6 +19,8 @@ function main() {
 	kubectl_bin apply -f "${conf_dir}/client.yml" \
 		-f "${conf_dir}/secrets.yml"
 
+	apply_s3_storage_secrets
+
 	desc 'install version service'
 
 	cp $test_dir/conf/operator.main.psmdb-operator.dep.json ${tmp_dir}/operator.${OPERATOR_VERSION}.psmdb-operator.dep.json

e2e-tests/multi-storage/compare/find-2nd.json

@@ -0,0 +1,13 @@
+switched to db myApp
+{ "_id" : , "x" : 100500 }
+{ "_id" : , "x" : 100501 }
+{ "_id" : , "x" : 100502 }
+{ "_id" : , "x" : 100503 }
+{ "_id" : , "x" : 100504 }
+{ "_id" : , "x" : 100505 }
+{ "_id" : , "x" : 100506 }
+{ "_id" : , "x" : 100507 }
+{ "_id" : , "x" : 100508 }
+{ "_id" : , "x" : 100509 }
+{ "_id" : , "x" : 100510 }
+bye

e2e-tests/multi-storage/compare/find.json

@@ -0,0 +1,8 @@
+switched to db myApp
+{ "_id" : , "x" : 100500 }
+{ "_id" : , "x" : 100501 }
+{ "_id" : , "x" : 100502 }
+{ "_id" : , "x" : 100503 }
+{ "_id" : , "x" : 100504 }
+{ "_id" : , "x" : 100505 }
+bye

e2e-tests/multi-storage/conf/backup.yml

@@ -0,0 +1,9 @@
+apiVersion: psmdb.percona.com/v1
+kind: PerconaServerMongoDBBackup
+metadata:
+  finalizers:
+    - percona.com/delete-backup
+  name: backup-minio
+spec:
+  clusterName: some-name
+  storageName: minio

e2e-tests/multi-storage/conf/restore.yml

@@ -0,0 +1,10 @@
+apiVersion: psmdb.percona.com/v1
+kind: PerconaServerMongoDBRestore
+metadata:
+  name:
+spec:
+  clusterName: some-name
+  backupName:
+  pitr:
+    type: date
+    date: