diff --git a/.github/styles/config/vocabularies/Percona/accept.txt b/.github/styles/config/vocabularies/Percona/accept.txt index 6476c4ed..0e131f3b 100644 --- a/.github/styles/config/vocabularies/Percona/accept.txt +++ b/.github/styles/config/vocabularies/Percona/accept.txt @@ -28,3 +28,5 @@ S3-compatible uncomment CAs Ceph +Alibaba +Alibaba Cloud diff --git a/docs/details/authentication.md b/docs/details/authentication.md index fa020efa..10287579 100644 --- a/docs/details/authentication.md +++ b/docs/details/authentication.md @@ -17,13 +17,13 @@ etc. as the `mongo` shell or `mongodump` command does. The `pbm-agent` processes should connect to their localhost `mongod` with a standalone type of connection. - ```{.bash data-prompt="$"} + ```bash pbm-agent --mongodb-uri "mongodb://pbmuser:secretpwd@localhost:27017/?authSource=admin" ``` Alternatively: - ```{.bash data-prompt="$"} + ```bash export PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@localhost:27017/?authSource=admin" pbm-agent ``` @@ -32,15 +32,15 @@ etc. as the `mongo` shell or `mongodump` command does. === "The `pbm` CLI connection string" - ```{.bash data-prompt="$"} + ```bash pbm status --mongodb-uri "mongodb://pbmuser:secretpwd@mongocsvr1:27017,mongocsvr2:27017,mongocsvr3:27017/?replicaSet=configrs&authSource=admin" ``` Alternatively: - ```{.bash data-prompt="$"} + ```bash export PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@mongocsvr1:27017,mongocsvr2:27017,mongocsvr3:27017/?replicaSet=configrs&authSource=admin" - $ pbm status + pbm status ``` Replace the `pbmuser:secretpwd` with the credentials of [the user who owns the pbm process](../install/configure-authentication.md#create-the-pbm-user) @@ -86,28 +86,28 @@ Specify new values in MongoDB connection URI string as follows: === "The pbm-agent connection string" - ``` + ```bash pbm-agent --mongodb-uri "mongodb://pbmuser:secretpwd@localhost:27017/?authSource=admin&readConcernLevel=local&w=1" ``` Alternatively: - ``` + ```bash export PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@localhost:27017/?authSource=admin&readConcernLevel=local&w=1" pbm-agent ``` === "The `pbm` CLI connection string" - ```{.bash data-prompt="$"} + ```bash pbm status --mongodb-uri "mongodb://pbmuser:secretpwd@mongocsvr1:27017,mongocsvr2:27017,mongocsvr3:27017/?replicaSet=configrs&authSource=admin&readConcernLevel=local&w=1" ``` Alternatively: - ```{.bash data-prompt="$"} + ```bash export PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@mongocsvr1:27017,mongocsvr2:27017,mongocsvr3:27017/?replicaSet=configrs&authSource=admin&readConcernLevel=local&w=1" - $ pbm status + pbm status ``` Supported values are: @@ -131,14 +131,14 @@ For [Kerberos authentication :octicons-link-external-16:](https://docs.percona.c Specify the following string for MongoDB connection URI: -``` +```bash PBM_MONGODB_URI="mongodb://%40@:27018/?authMechanism=GSSAPI&authSource=%24external&replSetName=xxxx" ``` Note that you must first obtain the ticket for the `pbm` user with the `kinit` command before you start the **pbm-agent**: -```{.bash data-prompt="$"} -$ sudo -u {USER} kinit pbm +```bash +sudo -u {USER} kinit pbm ``` Note that the `{USER}` is the user that you will run the `pbm-agent` process. @@ -147,7 +147,7 @@ Note that the `{USER}` is the user that you will run the `pbm-agent` process. For [authentication and authorization via Native LDAP :octicons-link-external-16:](https://docs.percona.com/percona-server-for-mongodb/latest/authorization.html#authentication-and-authorization-with-direct-binding-to-ldap), you only create roles for LDAP groups in MongoDB as the users are stored and managed on the LDAP server. However, you still define the `$external` database as your authentication source: -``` +```bash PBM_MONGODB_URI="mongodb://:@:27017/?authMechanism=PLAIN&authSource=%24external&replSetName=xxxx" ``` @@ -170,7 +170,7 @@ When using [AWS IAM authentication :octicons-link-external-16:](https://docs.per The MongoDB connection URI string then looks like the following: -``` +```bash PBM_MONGODB_URI="mongodb://:@:27017/?authMechanism=MONGODB-AWS&authSource=%24external&replSetName=xxxx" ``` diff --git a/docs/details/azure.md b/docs/details/azure.md index acc36186..ed86f711 100644 --- a/docs/details/azure.md +++ b/docs/details/azure.md @@ -26,32 +26,32 @@ For either method you need a storage account. 1. Install the [Azure CLI :octicons-link-external-16:](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli). After the installation, the `az` is available for you. 2. Sign in to Azure CLI: - ```{.bash data-prompt="$"} - $ az login + ```bash + az login ``` 3. Create a Resource group if it's not created for you: - ```{.bash data-prompt="$"} - $ az group create --name --location + ```bash + az group create --name --location ``` For the list of available locations, run: - ```{.bash data-prompt="$"} - $ az account list-locations + ```bash + az account list-locations ``` 4. Create a storage account: - ```{.bash data-prompt="$"} - $ az storage account create --name --resource-group --location --sku Standard_LRS + ```bash + az storage account create --name --resource-group --location --sku Standard_LRS ``` 4. Create a blob container: - ```{.bash data-prompt="$"} - $ az storage container create --account-name --name --public-access off + ```bash + az storage container create --account-name --name --public-access off ``` ??? example "Expected output" diff --git a/docs/details/gcs.md b/docs/details/gcs.md index 0b7a0cad..c039551c 100644 --- a/docs/details/gcs.md +++ b/docs/details/gcs.md @@ -30,14 +30,14 @@ To use GCS, you need the following: 2. Create a bucket - ```{.bash data-prompt="$"} - $ gcloud storage buckets create my-gcs-bucket --location=US + ```bash + gcloud storage buckets create my-gcs-bucket --location=US ``` 3. Verify the bucket creation - ```{.bash data-prompt="$"} - $ gcloud storage buckets list + ```bash + gcloud storage buckets list ``` After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup). diff --git a/docs/details/minio.md b/docs/details/minio.md index 11994f16..1c6b0cfe 100644 --- a/docs/details/minio.md +++ b/docs/details/minio.md @@ -12,20 +12,20 @@ This document provides an overview of MinIO as the closest S3-compatible storage 2. Configure the `mc` command line tool with a MinIO Server - ```{.bash data-prompt="$"} - $ mc alias set myminio http://127.0.0.1:9000 MINIO_ACCESS_KEY MINIO_SECRET_KEY + ```bash + mc alias set myminio http://127.0.0.1:9000 MINIO_ACCESS_KEY MINIO_SECRET_KEY ``` 3. Create a bucket - ```{.bash data-prompt="$"} - $ mc mb myminio/my-minio-bucket + ```bash + mc mb myminio/my-minio-bucket ``` 4. Verify the bucket creation - ```{.bash data-prompt="$"} - $ mc ls myminio + ```bash + mc ls myminio ``` After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup). @@ -105,16 +105,16 @@ Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path 2. Set the `SSL_CERT_FILE` environment variable to that file's path on each host where `pbm-agent` and PBM CLI are running: - ```{.bash data-prompt="$"} - $ export SSL_CERT_FILE=/etc/ssl/minio-ca.crt + ```bash + export SSL_CERT_FILE=/etc/ssl/minio-ca.crt ``` If this variable isn't set, PBM uses the system root certificates. 3. Restart `pbm-agent`: - ```{.bash data-prompt="$"} - $ sudo systemctl start pbm-agent + ```bash + sudo systemctl start pbm-agent ``` 4. Verify that your custom certificate is recognized. Check PBM logs for successful storage access. @@ -122,8 +122,8 @@ Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path Alternatively, you can turn off the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: -```{.bash data-prompt="$"} -$ pbm config --set storage.minio.insecureSkipTLSVerify=True +```bash +pbm config --set storage.minio.insecureSkipTLSVerify=True ``` !!! warning diff --git a/docs/details/oss.md b/docs/details/oss.md index 5144f542..dae6457c 100644 --- a/docs/details/oss.md +++ b/docs/details/oss.md @@ -25,24 +25,24 @@ You can create a bucket via the [Alibaba Cloud Management Console :octicons-link 1. [Install](https://www.alibabacloud.com/help/en/oss/developer-reference/install-ossutil2#DAS) and configure the Alibaba Cloud OSS client. After the installation, the `ossutil` command line tool is available for you. 2. Specify the region: - ```{.bash data-prompt="$"} - $ ossutil config + ```bash + ossutil config ``` Press Enter until you see the prompt `Please enter Region [cn-hangzhou]:` and specify the desired region. 3. Create a bucket: - ```{.bash data-prompt="$"} - $ ossutil mb oss://your-bucket-name + ```bash + ossutil mb oss://your-bucket-name ``` Replace `your-bucket-name` with the desired name for your bucket. 4. Verify that the bucket is created: - ```{.bash data-prompt="$"} - $ ossutil ls + ```bash + ossutil ls ``` After you created a bucket, apply the [necessary permissions](storage-configuration.md#permissions-setup) for the user identified by the access credentials you plan to use with PBM. diff --git a/docs/details/s3-storage.md b/docs/details/s3-storage.md index 90808b4b..3acd9ddb 100644 --- a/docs/details/s3-storage.md +++ b/docs/details/s3-storage.md @@ -9,6 +9,7 @@ This document provides overview for the native AWS S3 services. To use MinIO and [Configuration example :material-arrow-down:](#configuration-example){.md-button} + ## Storage bucket creation To create a bucket, do the following. @@ -17,14 +18,14 @@ To create a bucket, do the following. 2. Create an S3 bucket - ```{.bash data-prompt="$"} - $ aws s3api create-bucket --bucket my-s3-bucket --region us-east-1 + ```bash + aws s3api create-bucket --bucket my-s3-bucket --region us-east-1 ``` 3. Verify the bucket creation - ```{.bash data-prompt="$"} - $ aws s3 ls + ```bash + aws s3 ls ``` After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup). @@ -149,5 +150,59 @@ retryer: This upload retry increases the chances of data upload completion in cases of unstable connection. +<<<<<<< HEAD +======= +## Data upload to storage with self-signed TLS certificates + +Percona Backup for MongoDB supports data upload to S3-compatible storage service over HTTPS with a self-signed or a private CA certificate. This feature is especially important when you use services like MinIO, Ceph, or internal S3 gateways that don't use certificates signed by public Certificate Authorities (CAs). + +Providing a whole chain of certificates is recommended to ensure the connection is legit. The `SSL_CERT_FILE` environment variable specifies the path to a custom certificate chain file in PEM-format that PBM uses to validate TLS/SSL connection. + +### Usage example + +Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path and your S3 endpoint is `https://minio.internal.local:9000`. To use self-issued TLS certificates, do the following: + +1. Ensure the cert file is in PEM format. Use the following command to check it: + + ```bash + cat /etc/ssl/minio-ca.crt + ``` + + ??? example "Sample output" + + + ```{text .no-copy} + -----BEGIN CERTIFICATE----- + MIIC+TCCAeGgAwIBAgIJANH3WljB... + -----END CERTIFICATE----- + ``` + +2. Set the `SSL_CERT_FILE` environment variable to that file's path on each host where `pbm-agent` and PBM CLI are running: + + ```bash + export SSL_CERT_FILE=/etc/ssl/minio-ca.crt + ``` + + If this variable isn't set, PBM uses the system root certificates. + +3. Restart `pbm-agent`: + + ```bash + sudo systemctl start pbm-agent + ``` + +4. Verify that your custom certificate is recognized. Check PBM logs for successful S3 access. + + +Alternatively, you can disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: + +```bash +pbm config --set storage.s3.insecureSkipTLSVerify=True +``` + +!!! warning + + Use this option with caution as it might leave a hole for man-in-the-middle attacks. +>>>>>>> Removed prompt sign from code blocks diff --git a/docs/features/multi-storage.md b/docs/features/multi-storage.md index f772073e..14657f4c 100644 --- a/docs/features/multi-storage.md +++ b/docs/features/multi-storage.md @@ -38,14 +38,14 @@ storage: To upload the configuration profile to PBM, use the [`pbm profile add`](../reference/pbm-commands.md#pbm-profile-add) command and specify the path to the profile. -```{.bash data-prompt="$"} -$ pbm profile add /path/to/profile.yaml +```bash +pbm profile add /path/to/profile.yaml ``` To show the information about the external backup storage, use the [`pbm profile show`](../reference/pbm-commands.md#pbm-profile-show) command: -```{.bash data-prompt="$"} -$ pbm profile show +```bash +pbm profile show ``` See the full list of the configuration profile management commands in the [pbm commands](../reference/pbm-commands.md) reference. @@ -54,8 +54,8 @@ See the full list of the configuration profile management commands in the [pbm c To make a backup to an external backup storage, pass the profile name with the `--profile` flag for the `pbm backup` command. For example, to run a physical backup and store it in the MinIO storage defined via the `minio` configuration profile, run the following command: -```{.bash data-prompt="$"} -$ pbm backup -t physical --profile=minio --wait +```bash +pbm backup -t physical --profile=minio --wait ``` ??? example "Sample output" @@ -76,8 +76,8 @@ Before you start, make sure that `pbm-agents` have the read permissions to backu 1. List backups by running the `pbm list` or `pbm status` commands. - ```{.bash data-prompt="$"} - $ pbm list + ```bash + pbm list ``` The output shows the backup names and timestamps. External backups are marked with an asterisk: @@ -96,8 +96,8 @@ Before you start, make sure that `pbm-agents` have the read permissions to backu 2. To make a point-in-time restore, you must explicitly pass the backup name for the `pbm restore` command: - ```{.bash data-prompt="$"} - $ pbm-restore --time= --base-snapshot + ```bash + pbm-restore --time= --base-snapshot ``` 3. After the restore is complete, do the required post-restore steps depending on the restore type. @@ -109,8 +109,8 @@ You can delete backups from an external storage only by name. Run the `pbm delete` command and pass the backup name: -```{.bash data-prompt="$"} -$ pbm delete-backup 2024-06-25T10:54:55Z +```bash +pbm delete-backup 2024-06-25T10:54:55Z ``` ## Implementation specifics diff --git a/docs/features/physical.md b/docs/features/physical.md index b4ff1767..abb518b0 100644 --- a/docs/features/physical.md +++ b/docs/features/physical.md @@ -156,8 +156,8 @@ To configure physical restores with a fallback directory, use either the PBM con You can start the restore with a fallback directory directly using the `--fallback-enabled` flag: - ```{.bash data-prompt="$"} - $ pbm restore --time