From 5a6bdc991ae7d745f2f3db45593e66bbc27e0339 Mon Sep 17 00:00:00 2001 From: Ivan Groenewold <9805809+igroene@users.noreply.github.com> Date: Wed, 3 Dec 2025 07:52:35 -0300 Subject: [PATCH 1/7] Update oplog-replay.md Add info about using (epoch, ordinal) pair --- docs/usage/oplog-replay.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/usage/oplog-replay.md b/docs/usage/oplog-replay.md index 325b9fbd..13e814fe 100644 --- a/docs/usage/oplog-replay.md +++ b/docs/usage/oplog-replay.md @@ -33,6 +33,11 @@ After you [restored a physical backup](restore.md), do the following: ```bash pbm oplog-replay --start="2022-01-02T15:00:00" --end="2022-01-03T15:00:00" ``` + + !!! note "" + + Another possibility is using MongoDB’s (epoch, ordinal) values directly. + For example, it is possible to use `pbm oplog-replay --end “1764576382,0”` or `--end “1764576382,100”` so it’s possible to specify up to the exact operation you need to apply in the oplog. 4. After the oplog replay, make a fresh backup and enable the point-in-time recovery oplog slicing. From 2af4ed82fe8ef5e744607a9e4cee1afb0ad3b834 Mon Sep 17 00:00:00 2001 From: Ivan Groenewold <9805809+igroene@users.noreply.github.com> Date: Wed, 3 Dec 2025 10:43:42 -0300 Subject: [PATCH 2/7] Update docs/usage/oplog-replay.md Co-authored-by: Anastasia Alexandrova --- docs/usage/oplog-replay.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/usage/oplog-replay.md b/docs/usage/oplog-replay.md index 13e814fe..5d54785f 100644 --- a/docs/usage/oplog-replay.md +++ b/docs/usage/oplog-replay.md @@ -36,7 +36,7 @@ After you [restored a physical backup](restore.md), do the following: !!! note "" - Another possibility is using MongoDB’s (epoch, ordinal) values directly. + Another possibility is to use MongoDB’s (epoch, ordinal) values directly. This enables you to replay oplog up to a specific operation. For example, it is possible to use `pbm oplog-replay --end “1764576382,0”` or `--end “1764576382,100”` so it’s possible to specify up to the exact operation you need to apply in the oplog. 4. After the oplog replay, make a fresh backup and enable the point-in-time recovery oplog slicing. From 56b9446e44ff9ed2b6b3dff2178171ff4b840e32 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Wed, 3 Dec 2025 17:06:29 +0100 Subject: [PATCH 3/7] Added a section with the explanatio n how to define time for oplog replay and how PBM processes ops --- docs/usage/oplog-replay.md | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/docs/usage/oplog-replay.md b/docs/usage/oplog-replay.md index 5d54785f..5a334e2d 100644 --- a/docs/usage/oplog-replay.md +++ b/docs/usage/oplog-replay.md @@ -13,13 +13,25 @@ By replaying these oplog slices on top of the backup snapshot with the [`pbm opl Use the oplog replay functionality with caution, only when you are sure about the starting time from which to replay oplog. The oplog replay does not guarantee data consistency when restoring from any backup. However, it is less error-prone for backups made with Percona Backup for MongoDB. +## Ways to specify time for oplog replay + +PBM uses MongoDB's timestamp format for oplog replay, which provides operation-level resolution. Each oplog operation is identified by `(epoch, ordinal)`, where `epoch` is the Unix time in seconds and `ordinal` distinguishes multiple operations within the same second. The specified operation is always included in the replay. + +You can define the oplog replay stop point in two ways: + +1. **By ISO timestamp**: + Specify an end time as an ISO timestamp (for example, `2025-01-02T15:00:00`). PBM converts this to `(epoch, 0)` and includes that operation. Use this method for convenience when the first operation within a specific second is sufficient. + +2. **By MongoDB timestamp tuple**: + Specify the stop point as `epoch,ordinal` (e.g., `1764576382,20`). PBM includes all operations up to that exact operation. Use this method when you need precise control over which specific operations within a second to include. This is especially useful when multiple operations occurred at the same time. + ## Oplog replay for physical backups !!! note "" Starting with version 2.2.0, oplog replay on top of a physical backups made with Percona Backup for MongoDB is done automatically as part of [point-in-time recovery](pitr-physical.md). -This section describes how to manually replay oplog on top of physical backups with Percona Backup for MongoDB version 2.1.0 and earlier. +This section describes how to **manually** replay oplog on top of physical backups made with Percona Backup for MongoDB version 2.1.0 and earlier. After you [restored a physical backup](restore.md), do the following: @@ -27,17 +39,21 @@ After you [restored a physical backup](restore.md), do the following: 2. Run `pbm status` or `pbm list` commands to find oplog chunks available for replay. +3. Run the `pbm oplog-replay` command and specify the `--start` and `--end` flags. See [how you can specify the time](#ways-to-specify-time-for-oplog-replay). -3. Run the `pbm oplog-replay` command and specify the `--start` and `--end` flags with the timestamps. + === "Use timestamp" - ```bash - pbm oplog-replay --start="2022-01-02T15:00:00" --end="2022-01-03T15:00:00" - ``` + ```bash + pbm oplog-replay --start="{{year}}-01-02T15:00:00" --end="{{year}}-01-03T15:00:00" + ``` - !!! note "" + === "Use `epoch,ordinal`" + + For a fine-grained precision which exactly operations within a second to include, specify the values for the `--start` and `--end` flags as `epoch,ordinal` tuples. - Another possibility is to use MongoDB’s (epoch, ordinal) values directly. This enables you to replay oplog up to a specific operation. - For example, it is possible to use `pbm oplog-replay --end “1764576382,0”` or `--end “1764576382,100”` so it’s possible to specify up to the exact operation you need to apply in the oplog. + ```bash + pbm oplog-replay --end “1764576382,100” + ``` 4. After the oplog replay, make a fresh backup and enable the point-in-time recovery oplog slicing. From 41197f9ede58287840208fc3e47d0cf4bd6d683e Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Wed, 3 Dec 2025 18:02:46 +0100 Subject: [PATCH 4/7] Updated after the review --- docs/usage/oplog-replay.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/usage/oplog-replay.md b/docs/usage/oplog-replay.md index 5a334e2d..768da904 100644 --- a/docs/usage/oplog-replay.md +++ b/docs/usage/oplog-replay.md @@ -20,10 +20,10 @@ PBM uses MongoDB's timestamp format for oplog replay, which provides operation-l You can define the oplog replay stop point in two ways: 1. **By ISO timestamp**: - Specify an end time as an ISO timestamp (for example, `2025-01-02T15:00:00`). PBM converts this to `(epoch, 0)` and includes that operation. Use this method for convenience when the first operation within a specific second is sufficient. + Specify an end time as an ISO timestamp (for example, `2025-01-02T15:00:00`). PBM converts this to `(epoch, 0)` and includes all oplog operations within that second. Use this method when you want to include all operations that occurred until the specified second. 2. **By MongoDB timestamp tuple**: - Specify the stop point as `epoch,ordinal` (e.g., `1764576382,20`). PBM includes all operations up to that exact operation. Use this method when you need precise control over which specific operations within a second to include. This is especially useful when multiple operations occurred at the same time. + Specify the stop point as `epoch,ordinal` (e.g., `1764576382,20`). PBM includes all operations up to that exact operation. Use this method when you need precise control over which specific operations within a second to include. ## Oplog replay for physical backups From c65dedcf05acd854046dec81c046ce988d94b1c0 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Wed, 3 Dec 2025 20:05:46 +0100 Subject: [PATCH 5/7] Removed confusing sentence --- docs/usage/oplog-replay.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/usage/oplog-replay.md b/docs/usage/oplog-replay.md index 768da904..a06141e5 100644 --- a/docs/usage/oplog-replay.md +++ b/docs/usage/oplog-replay.md @@ -20,7 +20,7 @@ PBM uses MongoDB's timestamp format for oplog replay, which provides operation-l You can define the oplog replay stop point in two ways: 1. **By ISO timestamp**: - Specify an end time as an ISO timestamp (for example, `2025-01-02T15:00:00`). PBM converts this to `(epoch, 0)` and includes all oplog operations within that second. Use this method when you want to include all operations that occurred until the specified second. + Specify an end time as an ISO timestamp (for example, `2025-01-02T15:00:00`). Use this method when you want to include all operations that occurred until the specified second. 2. **By MongoDB timestamp tuple**: Specify the stop point as `epoch,ordinal` (e.g., `1764576382,20`). PBM includes all operations up to that exact operation. Use this method when you need precise control over which specific operations within a second to include. From 654ead3e7a6c0d5183d840caa6928c629da43eb1 Mon Sep 17 00:00:00 2001 From: Ivan Groenewold <9805809+igroene@users.noreply.github.com> Date: Tue, 16 Dec 2025 09:47:51 -0300 Subject: [PATCH 6/7] Update configure-authentication.md --- docs/install/configure-authentication.md | 73 +++++++++++++----------- 1 file changed, 39 insertions(+), 34 deletions(-) diff --git a/docs/install/configure-authentication.md b/docs/install/configure-authentication.md index 9a805edb..03816233 100644 --- a/docs/install/configure-authentication.md +++ b/docs/install/configure-authentication.md @@ -2,7 +2,7 @@ Percona Backup for MongoDB uses the authentication and authorization subsystem of MongoDB. This means that to authenticate Percona Backup for MongoDB, you need to: -* [Create a corresponding `pbm` user](#create-the-pbm-user) in the `admin` database +* [Create a corresponding `pbm` user](#create-the-pbm-user) in the `admin` database for each replicaset * [Set a valid MongoDB connection URI string for **pbm-agent**](#set-the-mongodb-connection-uri-for-pbm-agent) * [Set a valid MongoDB connection URI string for `pbm` CLI](#set-the-mongodb-connection-uri-for-pbm-cli) @@ -10,7 +10,7 @@ Percona Backup for MongoDB uses the authentication and authorization subsystem o !!! info - Do this step on a primary node of each replica set. In a sharded cluster, this means on every shard replica set and the config server replica set. + Each PBM agent connects to its local node, so do these steps on a primary node of each replica set. In a sharded cluster, this means on every shard replica set and the config server replica set as well. 1. Create the role that allows any action on any resource. @@ -48,26 +48,16 @@ You can specify the `username` and `password` values and other options of the `c To list all the host+port lists for the shard replica sets in a cluster, run the following command: ```javascript - db.getSiblingDB(“config”).shards.find({}, {“host”: true, “_id”: false}) + db.getSiblingDB("config").shards.find({}, {host: true, _id: false}) ``` - The replica set name at the *front* of these “host” strings will have to be placed as a “/?replicaSet=xxxx” argument in the parameters part of the connection URI (see below). - ## Set the MongoDB connection URI for `pbm-agent` -!!! info - - Execute this step on each node where `pbm-agent` is installed. - -!!! important - - Each **pbm-agent** process needs to connect to its localhost `mongod` node with a standalone type of connection. Avoid using the replica set URI with **pbm-agent** as it can lead to unexpected behaviour. - - For sharded clusters, **pbm-agent** on each shard member also need to be able to connect to the config server replica set. The agents auto-discover the config server URI by querying the `db.system.version` collection. +!!! info - Note that the MongoDB connection URI for `pbm-agent` is different from the connection string required by pbm CLI. + Execute this step on each node where `pbm-agent` is installed. -The `pbm-agent.service` systemd unit file includes the location of the environment file. You set the MongoDB URI connection string for the `PBM_MONGODB_URI` variable within the environment file for every **pbm-agent**. +The `pbm-agent.service` systemd unit file includes the location of the environment file. You set the MongoDB URI connection string for the `PBM_MONGODB_URI` variable within the environment file for every **pbm-agent**. ??? tip "How to find the environment file" @@ -93,12 +83,12 @@ The `pbm-agent.service` systemd unit file includes the location of the environme [Install] WantedBy=multi-user.target ``` - + === ":material-debian: On Debian and Ubuntu" - Edit the environment file `/etc/default/pbm-agent` and specify the MongoDB connection URI string for the `pbm` user to the local `mongod` node. + Edit the environment file `/etc/default/pbm-agent` and specify the MongoDB connection URI string using the `pbm` user making sure to connect to the local `mongod` node only. - For example, if `mongod` node listens on port 27017, the MongoDB connection URI string will be the following: + For example, if `mongod` process listens on port 27017, the MongoDB connection URI string for PBM agent will be the following: ``` PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@localhost:27017/?authSource=admin" @@ -106,9 +96,9 @@ The `pbm-agent.service` systemd unit file includes the location of the environme === ":material-redhat: On Red Hat Enterprise Linux and derivatives" - Edit the environment file `/etc/sysconfig/pbm-agent` and specify the MongoDB connection URI string for the `pbm` user to the local `mongod` node. + Edit the environment file `/etc/sysconfig/pbm-agent` and specify the MongoDB connection URI string using the `pbm` user making sure to connect to the local `mongod` node only. - For example, if `mongod` node listens on port 27017, the MongoDB connection URI string will be the following: + For example, if `mongod` process listens on port 27017, the MongoDB connection URI string for PBM agent will be the following: ``` PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@localhost:27017/?authSource=admin" @@ -116,6 +106,14 @@ The `pbm-agent.service` systemd unit file includes the location of the environme To improve the security of the file, you can change its owner to the user that is configured in systemd, and set the file permission so that only the owner and root can read from it. +!!! important + + Each **pbm-agent** process needs to connect to its localhost `mongod` node with a standalone type of connection. Avoid using the replica set URI with **pbm-agent** as it can lead to unexpected behaviour. + + For sharded clusters, **pbm-agent** on each shard member also needs to be able to reach the config server replica set over the network. The agent auto-discovers the config servers URI by querying the `db.system.version` collection. + + Note that the MongoDB connection URI for `pbm-agent` is different from the connection string required by pbm CLI. + ### Passwords with special characters If the password includes special characters like `#`, `@`, `/` and so on, you must convert these characters using the [percent-encoding mechanism](https://datatracker.ietf.org/doc/html/rfc3986#section-2.1) when passing them to Percona Backup for MongoDB. For example, the password `secret#pwd` should be passed as follows in `PBM_MONGODB_URI`: @@ -128,24 +126,31 @@ PBM_MONGODB_URI="mongodb://pbmuser:secret%23pwd@localhost:27017/?authSource=admi !!! info - Execute this step only on a host at which you will use `pbm` CLI. - -!!! important - - The pbm CLI needs to connect to the replica set that stores PBM Control Collections. The MongoDB connection URI format is different from the one required by `pbm-agent`. + Execute this step only on the hosts where you will run `pbm` CLI commands for backup or restore. - In a non-sharded replica set it is simply that replica set. In a sharded cluster it is the config server replica set. +Set the MongoDB URI connection string for `pbm` CLI as an environment variable in your shell. This allows you to run `pbm` commands without specifying the `--mongodb-uri` flag each time. -Set the MongoDB URI connection string for `pbm` CLI in your shell. This allows you to call `pbm` commands without the `--mongodb-uri` flag. - -The following command is the example how to define the MongoDB URI connection string for a replica set with the replica set members `mongors1:27017`, `mongors2:27017` and `mongors3:27017`: +=== "Replica Sets" + + In a non-sharded replica set, point PBM CLI to the replica set. The following command is the example how to define the MongoDB URI connection string for a replica set with the replica set members `mongors1:27017`, `mongors2:27017` and `mongors3:27017`: -``` -export PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@mongors1:27017,mongors2:27017,mongors3:27017/?authSource=admin&replSetName=xxxx" -``` + ``` + export PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@mongors1:27017,mongors2:27017,mongors3:27017/?authSource=admin&replSetName=xxxx" + ``` + +=== "Sharded Clusters" + In a sharded cluster, point PBM CLI to the config server replica set. The following command is the example how to define the MongoDB URI connection string for a sharded cluster with the config servers `mongocfg1:27017`, `mongocfg2:27017` and `mongocfg3:27017`: + + ``` + export PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@mongocfg1:27017,mongocfg2:27017,mongocfg3:27017/?authSource=admin&replSetName=xxxx" + ``` -For more information about what connection string to specify, refer to the [pbm connection string](../details/authentication.md#mongodb-connection-strings) section. +!!! important + + The pbm CLI needs to connect to the replica set that stores PBM Control Collections. The MongoDB connection URI is different from the one required by `pbm-agent`. + +For more information about the connection string to specify, refer to the [pbm connection string](../details/authentication.md#mongodb-connection-strings) section. If you are using external authentication methods in MongoDB, see [External authentication support in Percona Backup for MongoDB](../details/authentication.md#external-authentication-support-in-percona-backup-for-mongodb) section for configuration guidelines. From f293dfe7d114b112b5104b980ce5f95beceb2fd5 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Tue, 23 Dec 2025 13:26:28 +0200 Subject: [PATCH 7/7] Minor wording and style edits --- docs/install/configure-authentication.md | 37 ++++++++++++++---------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/docs/install/configure-authentication.md b/docs/install/configure-authentication.md index 03816233..5ca6874c 100644 --- a/docs/install/configure-authentication.md +++ b/docs/install/configure-authentication.md @@ -10,7 +10,10 @@ Percona Backup for MongoDB uses the authentication and authorization subsystem o !!! info - Each PBM agent connects to its local node, so do these steps on a primary node of each replica set. In a sharded cluster, this means on every shard replica set and the config server replica set as well. + Each PBM agent connects to its local node. To configure authentication, create the required user and role **on the primary node** of each replica set. + + - For a standalone replica set, perform these steps on its primary node. + - For a sharded cluster, perform these steps on the primary node of every shard replica set, as well as on the config server replica set. 1. Create the role that allows any action on any resource. @@ -40,12 +43,12 @@ Percona Backup for MongoDB uses the authentication and authorization subsystem o }); ``` -You can specify the `username` and `password` values and other options of the `createUser` command as you require so long as the roles shown above are granted. +You can change the `username` and `password` values and specify other options for the `createUser` command as you require. But you must grant this user the roles as shown above. !!! tip - To list all the host+port lists for the shard replica sets in a cluster, run the following command: + To view all the host+port lists for the shard replica sets in a cluster, run the following command: ```javascript db.getSiblingDB("config").shards.find({}, {host: true, _id: false}) @@ -54,10 +57,10 @@ You can specify the `username` and `password` values and other options of the `c ## Set the MongoDB connection URI for `pbm-agent` !!! info - + Execute this step on each node where `pbm-agent` is installed. -The `pbm-agent.service` systemd unit file includes the location of the environment file. You set the MongoDB URI connection string for the `PBM_MONGODB_URI` variable within the environment file for every **pbm-agent**. +The `pbm-agent.service` systemd unit file includes the location of the environment file. You set the MongoDB URI connection string for the `PBM_MONGODB_URI` variable within the environment file for every `pbm-agent`. ??? tip "How to find the environment file" @@ -86,31 +89,35 @@ The `pbm-agent.service` systemd unit file includes the location of the environme === ":material-debian: On Debian and Ubuntu" - Edit the environment file `/etc/default/pbm-agent` and specify the MongoDB connection URI string using the `pbm` user making sure to connect to the local `mongod` node only. - - For example, if `mongod` process listens on port 27017, the MongoDB connection URI string for PBM agent will be the following: + Edit the environment file `/etc/default/pbm-agent`. Specify the MongoDB connection URI string using the credentials of the `pbm` user you created previously. Ensure that the pbm-agent connects only to its local mongod node by providing the URI in the following format: ``` PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@localhost:27017/?authSource=admin" ``` -=== ":material-redhat: On Red Hat Enterprise Linux and derivatives" + * Replace with the actual credentials you assigned to the pbm user. + * `localhost:27017` ensures the agent connects only to the local `mongod` instance. + * `authSource=admin` points authentication to the `admin` database, where the pbm user was created. - Edit the environment file `/etc/sysconfig/pbm-agent` and specify the MongoDB connection URI string using the `pbm` user making sure to connect to the local `mongod` node only. +=== ":material-redhat: On Red Hat Enterprise Linux and derivatives" - For example, if `mongod` process listens on port 27017, the MongoDB connection URI string for PBM agent will be the following: + Edit the environment file `/etc/sysconfig/pbm-agent` Specify the MongoDB connection URI string using the credentials of the `pbm` user you created previously. Ensure that the pbm-agent connects only to its local mongod node by providing the URI in the following format: ``` PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@localhost:27017/?authSource=admin" ``` + * Replace with the actual credentials you assigned to the pbm user. + * `localhost:27017` ensures the agent connects only to the local `mongod` instance. + * `authSource=admin` points authentication to the `admin` database, where the pbm user was created. + To improve the security of the file, you can change its owner to the user that is configured in systemd, and set the file permission so that only the owner and root can read from it. !!! important - Each **pbm-agent** process needs to connect to its localhost `mongod` node with a standalone type of connection. Avoid using the replica set URI with **pbm-agent** as it can lead to unexpected behaviour. + Each **pbm-agent** process needs to connect to its localhost `mongod` node with a standalone type of connection. Avoid using the replica set URI with **pbm-agent** as it can lead to an unexpected behavior. - For sharded clusters, **pbm-agent** on each shard member also needs to be able to reach the config server replica set over the network. The agent auto-discovers the config servers URI by querying the `db.system.version` collection. + For sharded clusters, **pbm-agent** on each shard member also needs to be able to reach the config server replica set over the network. The agent auto-discovers the config server's URI by querying the `db.system.version` collection. Note that the MongoDB connection URI for `pbm-agent` is different from the connection string required by pbm CLI. @@ -130,7 +137,7 @@ PBM_MONGODB_URI="mongodb://pbmuser:secret%23pwd@localhost:27017/?authSource=admi Set the MongoDB URI connection string for `pbm` CLI as an environment variable in your shell. This allows you to run `pbm` commands without specifying the `--mongodb-uri` flag each time. -=== "Replica Sets" +=== "Replica set" In a non-sharded replica set, point PBM CLI to the replica set. The following command is the example how to define the MongoDB URI connection string for a replica set with the replica set members `mongors1:27017`, `mongors2:27017` and `mongors3:27017`: @@ -138,7 +145,7 @@ Set the MongoDB URI connection string for `pbm` CLI as an environment variable i export PBM_MONGODB_URI="mongodb://pbmuser:secretpwd@mongors1:27017,mongors2:27017,mongors3:27017/?authSource=admin&replSetName=xxxx" ``` -=== "Sharded Clusters" +=== "Sharded clusters" In a sharded cluster, point PBM CLI to the config server replica set. The following command is the example how to define the MongoDB URI connection string for a sharded cluster with the config servers `mongocfg1:27017`, `mongocfg2:27017` and `mongocfg3:27017`: