DO-168: resolving discussions

siameseoriental · siameseoriental · commit 39c172eea159 · 2024-01-09T11:24:58.000+01:00
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,66 @@
+<!--
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Stub:
+## [Unreleased] - YYYY-MM-DD
+### Added
+### Changed
+### Deprecated
+### Removed
+### Fixed
+### Security
+
+-->
+
+# nd_postgres_backup
+
+## [0.3.0] - 2024-01-09
+
+### Added
+
+- Add publish decription to Docker Hub from README.md 
+- Add jq to container build 
+- Add abilty to change alias for S3 connection. Variable `S3_ALIAS`
+- Add docker-compose [example](compose-example/docker-compose.yml)
+- Add `crontab` file for managing schedule
+- Add retention functionallity 
+
+### Removed
+
+- Remove Go-Cron, replace it with standard cron
+- Cut notifications for hourly backups
+
+### Changed
+
+- Replace manual backup routine logic. To start manual backup run container without S3_BUCKET variable. See [README](README.md)
+- Change S3 storage directory from `${S3_BUCKET}/${POSTGRES_DB}` to `${S3_BUCKET}/${backup_path}`
+- Rename notification scripts. Add `.sh` extention
+
+
+### Braking changes
+- Removed:
+  - `SCHEDULE` - variable was used by Go-Cron
+  - `HEALTHCHECK_PORT` - variable was used by Go-Cron
+
+## [0.2.2] - 2023-08-25
+- Make docker-entrypoint.sh to easy manual run
+- Fix bug with `S3_OBJECT_PATH` var
+
+## [0.2.1] - 2023-08-25
+- Added var `S3_OBJECT_PATH` to define the path to the backup file in the bucket
+
+## [0.2.0] - 2023-08-25
+- refactoring and verification
+### Braking changes
+- Renamed:
+  - `S3_ACCESS_KEY_ID` -> `S3_ACCESS_KEY`
+  - `S3_SECRET_ACCESS_KEY` -> `S3_SECRET_KEY`
+
+## [0.1.0] - 2023-08-03
+- Backup Postgres DB
+- Send back up to S3
+- Notify users by telegram or private messaging system
diff --git a/Dockerfile b/Dockerfile
@@ -24,7 +24,7 @@ RUN mcli --version
 
 # PosgeSQL related variables
 ENV POSTGRES_DB="**None**" \
-    POSTGRES_HOST="db" \
+    POSTGRES_HOST="**None**" \
     POSTGRES_PORT=5432 \
     POSTGRES_USER="**None**" \
     POSTGRES_PASSWORD="**None**" \
diff --git a/README.md b/README.md
@@ -1,14 +1,22 @@
 # PostgreSQL Backup to S3 and retention container
 
+# Roadmap
+
+- [X] Add support for S3
+- [X] Add CI/CD to publish image to DockerHub
+- [X] Add retention policy settings by env vars
+- [X] Notify about backup status by HTTP-request
+- [X] Add docker-compose example
+
 ## Description
 
-Image created to automate backing up procedure of PostgreSQL databases, store them to S3 Object storage and implement retention of stored archives with `Grandfather-father-son` backup rotation [scheme](https://en.wikipedia.org/wiki/Backup_rotation_scheme).     
-It is also possible to use this container to create a single backup of specific DB. Set full S3 path (e.g `bucket_name/project_name/stage_branch/database_name.tar.gz`) as the value of variable `S3_OBJECT_PATH`  to execute single backup  
+Image created to automate backing up procedure of PostgreSQL databases, store backups to S3 Object storage and implement retention of stored archives with `Grandfather-father-son` backup rotation [scheme](https://en.wikipedia.org/wiki/Backup_rotation_scheme).     
+It is also possible to use this container to create a single backup of specific DB.  
 
 ## Usage
 
 Key idea of usage was to add this container as a service to `docker-compose.yml` manifest alongside with PostgreSQL database container. See `compose-example/docker-compose.yml`.    
-To run container as a standalone backuaper, to backup cloud SaaS or bare-metal deployed PostgreSQL, for example use following command:
+To run container as a standalone backupper, to backup cloud SaaS or bare-metal deployed PostgreSQL, for example, use following command:
 
 ```shell
 docker run -d --rm \
@@ -27,6 +35,27 @@ docker run -d --rm \
     numdes/nd_postgres_backup:v0.3.0
 ```
 
+### Manual one-time backup without schedule
+
+Set full S3 path (e.g `bucket_name/project_name/stage_branch/database_name.tar.gz`) as the value of variable `S3_OBJECT_PATH`  to execute single backup 
+
+```shell
+docker run -d --rm \
+    --env POSTGRES_HOST="DB_IP_OR_HOSTNAME" \
+    --env POSTGRES_DB="DB_NAME" \
+    --env POSTGRES_USER="DB_USERNAME" \
+    --env POSTGRES_PORT="NON_DEFAULT_PORT" \
+    --env POSTGRES_PASSWORD="DB_USERNAME_PASSWORD" \
+    --env NOTIFICATION_SERVER_URL="ONLY_SET_IF_PRIVATE_TELEGRAM_BOT_USED" \
+    --env TELEGRAM_CHAT_ID="PRIVATE_OR_TELEGRAM_BOT_ID" \
+    --env S3_ENDPOINT="S3_API_URL" \
+    --env S3_ACCESS_KEY="S3_ACCESS_KEY" \
+    --env S3_SECRET_KEY="S3_SECERT_KEY" \
+    --env S3_OBJECT_PATH="FULL_S3_PATH (e.g `bucket_name/project_name/stage_branch/database_name.tar.gz`)" \
+    --env S3_ALIAS="S3_CONFIG_SET_ALIAS" \
+    numdes/nd_postgres_backup:v0.3.0
+```
+
 ## Backup strategy
 
 By default set to make backup every hour, plus one separate backup a day, plus one separate backup a week
@@ -43,27 +72,44 @@ Maximum depth of storage for each type of backup can be tuned by changing values
 
 Schedule of retention script (`retention.sh`) execution can be edited in `crontab` file 
 
-## Variables list
-
-| Variable Name             | Default Value | Is Mandatory? | Description                                                               |
-|---------------------------|:-------------:|:-------------:|---------------------------------------------------------------------------|
-| HOURLY_BACKUP_PATH        | `hourly`      |     NO        | Path suffix to hourly-made backups storage |
-| DAILY_BACKUP_PATH         | `daily`       |     NO        | Path suffix to daily-made backups storage|
-| WEEKLY_BACKUP_PATH        | `weekly`      |     NO        | Path suffix to weekly-made backups storage|
-| WEEKLY_BACKUP_LIMIT       | `5`           |     NO        | Max number of weekly backups      |
-| DAILY_BACKUP_LIMIT        | `10`          |     NO        | Max number of daily backups       |
-| HOURLY_BACKUP_LIMIT       | `25`          |     NO        | Max number of hourly backups      |
-| S3_ACCESS_KEY             | -             |     YES       | ${S3_BUCKET} READ-WRITE S3 ACCESS KEY |
-| S3_SECRET_KEY             | -             |     YES       | ${S3_BUCKET} READ-WRITE S3 ACCESS SECRET  |
-| S3_ENDPOINT               | -             |     YES       | S3 API URL |
-| S3_BUCKET                 | -             |     YES       | Path to hourly, daily, weekly directories will be. Including bucket name        |
-| S3_OBJECT_PATH            | -             |     NO        | Optional variable to use single backup [functionality](#description) |
-| POSTGRES_DB               | -             |     YES       | PostgreSQL database name                                                            |
-| POSTGRES_HOST             | `db`          |     NO        | PostgreSQL IP or host name                                        |
+## Variables
+
+### `Gitlab Actions`
+*[variables](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository)*:
+
+| Name               | Description                 |
+|--------------------|-----------------------------|
+| DOCKERHUB_USERNAME | `Actions` Repository secret |
+| DOCKERHUB_TOKEN    | `Actions` Repository secret |
+
+### Notification environmental variables
+
+| Name                      | Description                                                           |
+|---------------------------|-----------------------------------------------------------------------|
+| NOTIFICATION_SERVER_URL   | URL of private telegram bot                                           |
+| TELEGRAM_CHAT_ID          | Custom bot ID or Telegram Bot ID when bot created using `@botfather`  |
+| TELEGRAM_BOT_TOKEN        | Created by `@botfather` bot security token                            |
+
+### Environmental variables
+
+| Variable Name             | Default Value | Is Mandatory? | Description                                                          |
+|---------------------------|:-------------:|:-------------:|----------------------------------------------------------------------|
+| HOURLY_BACKUP_PATH        | `hourly`      |     NO        | Path suffix to hourly-made backups storage                           |
+| DAILY_BACKUP_PATH         | `daily`       |     NO        | Path suffix to daily-made backups storage                            |
+| WEEKLY_BACKUP_PATH        | `weekly`      |     NO        | Path suffix to weekly-made backups storage                           |
+| HOURLY_BACKUP_LIMIT       | `25`          |     NO        | Max number of weekly backups                                         |
+| DAILY_BACKUP_LIMIT        | `10`          |     NO        | Max number of daily backups                                          |
+| WEEKLY_BACKUP_LIMIT       | `5`           |     NO        | Max number of hourly backups                                         |
+| S3_ACCESS_KEY             | -             |     YES       | ${S3_BUCKET} READ-WRITE S3 ACCESS KEY                                |
+| S3_SECRET_KEY             | -             |     YES       | ${S3_BUCKET} READ-WRITE S3 ACCESS SECRET                             |
+| S3_ENDPOINT               | -             |     YES       | S3 API URL                                                           |
+| S3_BUCKET                 | -             |     YES       | Path to hourly, daily, weekly directories. Including bucket name     |
+| S3_ALIAS                  | `backup`      |     NO        | Name of config set in `mcli` command `mcli alias set`                |
+| S3_OBJECT_PATH            | -             |     NO        | Optional variable to use single backup [functionality](#manual-backup-without-schedule) |
+| POSTGRES_DB               | -             |     YES       | PostgreSQL database name                                             |
+| POSTGRES_HOST             | -             |     YES       | PostgreSQL IP or host name                                           |
 | POSTGRES_PORT             | `5432`        |     NO        | TCP connection port                                                  |
-| POSTGRES_USER             | -             |     YES       | DB usermane                                                            |
-| POSTGRES_PASSWORD         | -             |     YES       | DB username password                                                    |
-| POSTGRES_EXTRA_OPTS       | `--blobs`     |     NO        | `pg_dump` extra options |
-| NOTIFICATION_SERVER_URL   | -             |     NO        | URL of private telegram bot                                                            |
-| TELEGRAM_CHAT_ID          | -             |     NO        | Custom bot ID or Telegram Bot ID when bot created using `@botfather`                                                   |
-| TELEGRAM_BOT_TOKEN        | -             |     NO        | Created by `@botfather` bot security token  |
+| POSTGRES_USER             | -             |     YES       | DB usermane                                                          |
+| POSTGRES_PASSWORD         | -             |     YES       | DB username password                                                 |
+| POSTGRES_EXTRA_OPTS       | `--blobs`     |     NO        | `pg_dump` extra options                                              |
+
diff --git a/backup.sh b/backup.sh
@@ -6,7 +6,7 @@
 set -euo pipefail
 IFS=$'\n\t'
 
-# Check if we have got path argument from scheduler if not set path to /
+# Check if we have gotten path argument from scheduler if not set path to /
 if [[ -z "$1" ]]; then
   backup_path=""
 elif  [[ ! "$1" == */ ]]; then
@@ -66,7 +66,8 @@ mcli cp "${ARCHIVE_FILE_NAME}" "${S3_ALIAS}"/"${relative_s3_object_path}"
 echo "Maid is here... Doing cleaning..."
 rm --force "${POSTGRES_DB}".*
 
-# # Do announce
+# Do announce
+# We are not going to spam chat every hour. Excluded hourly backups from notifications
 if [[ ! ${backup_path} =~ ^hourly.? ]]; then
   echo "Starting notification routine..."
 # Check which backup routine applied  
diff --git a/compose-example/docker-compose.yml b/compose-example/docker-compose.yml
@@ -21,9 +21,4 @@ services:
     depends_on:
       - db
 
-  redis:
-    image: redis:7
-
-  web:
-    image: ........
-
+# docker-compoe.yml continues ...
diff --git a/compose-example/variables.env b/compose-example/variables.env
@@ -1,13 +1,13 @@
 # Mandatory values
-S3_ENDPOINT=https://io.some-domain.com
+S3_ENDPOINT=https://some-s3-api.some-domain.com
 S3_ACCESS_KEY=${S3_ACCESS_KEY_DB_BACKUPS}
 S3_SECRET_KEY=${S3_SECRET_KEY_DB_BACKUPS}
 POSTGRES_DB=some_db_name
 POSTGRES_USER=some_db_username
 POSTGRES_PASSWORD=some_db_user_password
 
 # Mandatory only in case of scheduled usage
-S3_BUCKET=bucket_name/project_name/stage_branch/ # Including name of the bucket and (optional) the rest of directory structure
+S3_BUCKET=s3_path # Including name of the bucket and (optional) the rest of directory structure. E.g `bucket_name/project_name/stage_branch/`
 
 # Mandatory only in case of a single backup usage
 S3_OBJECT_PATH=some_s3_path # E.g `bucket_name/project_name/stage_branch/some_db_name.tar.gz`
diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh
@@ -12,6 +12,7 @@ if [[ ${S3_ACCESS_KEY} == "**None**" ]] ||
    [[ ${S3_ENDPOINT} == "**None**" ]] ||
    [[ ${POSTGRES_DB} == "**None**" ]] ||
    [[ ${POSTGRES_USER} == "**None**" ]] ||
+   [[ ${POSTGRES_HOST} == "**None**" ]] ||
    [[ ${POSTGRES_PASSWORD} == "**None**" ]]; then
   echo "One or more mandatory values is missing. Check your configuration..." >&2
   exit 1
diff --git a/retention.sh b/retention.sh
@@ -19,7 +19,7 @@ retention_func() {
 
   local backup_path="$1"
   local backup_limit="$2"
-# Here we are getting json-formed data from S3 and conveying to JQ
+# Here we are getting json-formed data from S3 and convey it to JQ
 # where we are sorting and selecting all backup directories except given last ones
 # How many backups should remain decides $backup_limit variable.
 # Each backup suitable for deletion conveys through `xargs` line for removal
