Skip to content

Commit 7f0319e

Browse files
hxmnhxmn
authored
Merge pull request #5 from siameseoriental/DO-168
DO-168: Add retention policy
2 parents 1699e7c + 39c172e commit 7f0319e

12 files changed

+311
-142
lines changed

‎.github/workflows/main.yaml

+7-1
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ jobs:
2626
# Login against a Docker registry
2727
# https://github.com/docker/login-action
2828
- name: Login to Docker Hub
29-
uses: docker/login-action@v2
29+
uses: docker/login-action@v3
3030
with:
3131
username: ${{ secrets.DOCKERHUB_USERNAME }}
3232
password: ${{ secrets.DOCKERHUB_TOKEN }}
@@ -50,3 +50,9 @@ jobs:
5050
cache-from: type=gha
5151
cache-to: type=gha,mode=max
5252

53+
- name: Docker Hub Description
54+
uses: peter-evans/dockerhub-description@v3
55+
with:
56+
username: ${{ secrets.DOCKERHUB_USERNAME }}
57+
password: ${{ secrets.DOCKERHUB_TOKEN }}
58+
repository: numdes/nd_postgres_backup

‎CHANGELOG.md

+28
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,34 @@ Stub:
1818

1919
# nd_postgres_backup
2020

21+
## [0.3.0] - 2024-01-09
22+
23+
### Added
24+
25+
- Add publish decription to Docker Hub from README.md
26+
- Add jq to container build
27+
- Add abilty to change alias for S3 connection. Variable `S3_ALIAS`
28+
- Add docker-compose [example](compose-example/docker-compose.yml)
29+
- Add `crontab` file for managing schedule
30+
- Add retention functionallity
31+
32+
### Removed
33+
34+
- Remove Go-Cron, replace it with standard cron
35+
- Cut notifications for hourly backups
36+
37+
### Changed
38+
39+
- Replace manual backup routine logic. To start manual backup run container without S3_BUCKET variable. See [README](README.md)
40+
- Change S3 storage directory from `${S3_BUCKET}/${POSTGRES_DB}` to `${S3_BUCKET}/${backup_path}`
41+
- Rename notification scripts. Add `.sh` extention
42+
43+
44+
### Braking changes
45+
- Removed:
46+
- `SCHEDULE` - variable was used by Go-Cron
47+
- `HEALTHCHECK_PORT` - variable was used by Go-Cron
48+
2149
## [0.2.2] - 2023-08-25
2250
- Make docker-entrypoint.sh to easy manual run
2351
- Fix bug with `S3_OBJECT_PATH` var

‎Dockerfile

+39-25
Original file line numberDiff line numberDiff line change
@@ -2,55 +2,69 @@
22

33
FROM debian:stable as base
44

5-
MAINTAINER NumDes <[email protected]>
6-
5+
LABEL org.opencontainers.image.authors="[email protected]"
76
LABEL org.opencontainers.image.vendor="Numerical Design LLC"
8-
LABEL org.opencontainers.image.description="Docker image for universal postgres backups"
9-
10-
ARG GOCRONVER=v0.0.10
7+
LABEL org.opencontainers.image.description="Docker image for complex store and retention backups on S3"
118

129
RUN apt-get update \
1310
&& apt-get install --no-install-recommends -y \
1411
curl \
1512
ca-certificates \
13+
cron \
14+
jq \
1615
postgresql-client \
16+
postgresql-common \
17+
libpq-dev \
1718
&& apt-get clean \
18-
&& rm -rf /var/lib/apt/lists/* \
19-
&& curl --fail \
20-
--retry 4 \
21-
--retry-all-errors \
22-
--output /usr/local/bin/go-cron.gz \
23-
--location https://github.com/prodrigestivill/go-cron/releases/download/$GOCRONVER/go-cron-linux-amd64.gz \
24-
&& gzip --verbose \
25-
--no-name \
26-
--decompress /usr/local/bin/go-cron.gz \
27-
&& chmod a+x /usr/local/bin/go-cron
19+
&& rm -rf /var/lib/apt/lists/*
2820

2921
RUN curl --location --output /usr/local/bin/mcli "https://dl.min.io/client/mc/release/linux-amd64/mc" && \
3022
chmod +x /usr/local/bin/mcli
3123
RUN mcli --version
3224

25+
# PosgeSQL related variables
3326
ENV POSTGRES_DB="**None**" \
3427
POSTGRES_HOST="**None**" \
3528
POSTGRES_PORT=5432 \
3629
POSTGRES_USER="**None**" \
3730
POSTGRES_PASSWORD="**None**" \
38-
POSTGRES_EXTRA_OPTS="--blobs" \
39-
SCHEDULE="**None**" \
40-
HEALTHCHECK_PORT=8080 \
31+
POSTGRES_EXTRA_OPTS="--blobs"
32+
# Paths and depth related variables
33+
ENV HOURLY_BACKUP_PATH="hourly" \
34+
DAILY_BACKUP_PATH="daily" \
35+
WEEKLY_BACKUP_PATH="weekly" \
36+
WEEKLY_BACKUP_LIMIT=5 \
37+
DAILY_BACKUP_LIMIT=10 \
38+
HOURLY_BACKUP_LIMIT=25
39+
# Notification related variables
40+
ENV NOTIFICATION_SERVER_URL="**None**" \
41+
TELEGRAM_CHAT_ID="**None**" \
42+
TELEGRAM_BOT_TOKEN="**None**"
43+
# S3 bucket related variables
44+
ENV S3_OBJECT_PATH="**None**" \
4145
S3_ACCESS_KEY="**None**" \
4246
S3_SECRET_KEY="**None**" \
43-
S3_BUCKET="**None**" \
4447
S3_ENDPOINT="**None**" \
45-
S3_OBJECT_PATH="**None**"
48+
S3_BUCKET="**None**" \
49+
S3_ALIAS="backup"
4650

47-
COPY backup.sh /
48-
COPY docker-entrypoint.sh /
51+
RUN mkdir /script
4952

53+
# Copy scripts
54+
COPY backup.sh /script/backup.sh
55+
COPY retention.sh /script/retention.sh
5056
COPY hooks /hooks
5157

52-
HEALTHCHECK --interval=5m --timeout=3s \
53-
CMD curl -f "http://localhost:$HEALTHCHECK_PORT/" || exit 1
58+
RUN chmod +x /script/backup.sh
59+
RUN chmod +x /script/retention.sh
60+
RUN chmod +x /hooks/0-send_private_notification.sh
61+
RUN chmod +x /hooks/1-send_telegram_message.sh
5462

55-
ENTRYPOINT ["/docker-entrypoint.sh"]
63+
COPY docker-entrypoint.sh /
64+
RUN chmod +x docker-entrypoint.sh
5665

66+
ADD crontab /etc/cron.d/my-cron-file
67+
RUN chmod 0644 /etc/cron.d/my-cron-file
68+
RUN crontab /etc/cron.d/my-cron-file
69+
70+
ENTRYPOINT ["/docker-entrypoint.sh"]

‎README.md

+85-84
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,76 @@
1-
# nd_postgres_backup
2-
3-
Docker image for universal postgres backups
1+
# PostgreSQL Backup to S3 and retention container
42

53
# Roadmap
64

75
- [X] Add support for S3
86
- [X] Add CI/CD to publish image to DockerHub
9-
- [ ] Add retention policy settings by env vars
7+
- [X] Add retention policy settings by env vars
108
- [X] Notify about backup status by HTTP-request
11-
- [ ] Add docker-compose example
9+
- [X] Add docker-compose example
10+
11+
## Description
12+
13+
Image created to automate backing up procedure of PostgreSQL databases, store backups to S3 Object storage and implement retention of stored archives with `Grandfather-father-son` backup rotation [scheme](https://en.wikipedia.org/wiki/Backup_rotation_scheme).
14+
It is also possible to use this container to create a single backup of specific DB.
1215

13-
# Usage
16+
## Usage
17+
18+
Key idea of usage was to add this container as a service to `docker-compose.yml` manifest alongside with PostgreSQL database container. See `compose-example/docker-compose.yml`.
19+
To run container as a standalone backupper, to backup cloud SaaS or bare-metal deployed PostgreSQL, for example, use following command:
1420

15-
## Backup manually:
16-
Use next command if you need to backup DB manually:
1721
```shell
18-
docker run --rm \
19-
--env POSTGRES_HOST="FQDN-OR-IP" \
20-
--env POSTGRES_DB="DB-NAME" \
21-
--env POSTGRES_USER="DB-USER" \
22-
--env POSTGRES_PASSWORD="PASS" \
23-
--env S3_ENDPOINT=https://YOUR-S3 \
24-
--env S3_ACCESS_KEY="access-key" \
25-
--env S3_SECRET_KEY="secret-key" \
26-
--env S3_BUCKET="BUCKET-NAME" \
27-
numdes/nd_postgres_backup:v0.2.2
22+
docker run -d --rm \
23+
--env POSTGRES_HOST="DB_IP_OR_HOSTNAME" \
24+
--env POSTGRES_DB="DB_NAME" \
25+
--env POSTGRES_USER="DB_USERNAME" \
26+
--env POSTGRES_PORT="NON_DEFAULT_PORT" \
27+
--env POSTGRES_PASSWORD="DB_USERNAME_PASSWORD" \
28+
--env NOTIFICATION_SERVER_URL="ONLY_SET_IF_PRIVATE_TELEGRAM_BOT_USED" \
29+
--env TELEGRAM_CHAT_ID="PRIVATE_OR_TELEGRAM_BOT_ID" \
30+
--env S3_ENDPOINT="S3_API_URL" \
31+
--env S3_ACCESS_KEY="S3_ACCESS_KEY" \
32+
--env S3_SECRET_KEY="S3_SECERT_KEY" \
33+
--env S3_BUCKET="S3_BUCKET_NAME(+POSSIBLE_PATH_DEEPER)" \
34+
--env S3_ALIAS="S3_CONFIG_SET_ALIAS" \
35+
numdes/nd_postgres_backup:v0.3.0
2836
```
29-
It will backup given Postgres DB and upload it to S3 bucket.
3037

31-
## Backup using `go-cron`
38+
### Manual one-time backup without schedule
39+
40+
Set full S3 path (e.g `bucket_name/project_name/stage_branch/database_name.tar.gz`) as the value of variable `S3_OBJECT_PATH` to execute single backup
3241

3342
```shell
34-
docker run --detach \
35-
--env POSTGRES_HOST="FQDN-OR-IP" \
36-
--env POSTGRES_DB="DB-NAME" \
37-
--env POSTGRES_USER="DB-USER" \
38-
--env POSTGRES_PASSWORD="PASS" \
39-
--env S3_ENDPOINT=http://YOUR-S3 \
40-
--env S3_ACCESS_KEY="KEY-ID" \
41-
--env S3_SECRET_KEY="KEY-SECRET" \
42-
--env S3_BUCKET="BUCKET-NAME" \
43-
--env PRIVATE_NOTIFICATION_URL=http://webhook \
44-
--env TELEGRAM_CHAT_ID=point_to_notify_group \
45-
--env POSTGRES_PORT=if_not_5432 \
46-
--env SCHEDULE=Chosen_schedule \
47-
numdes/nd_postgres_backup:v0.2.1
43+
docker run -d --rm \
44+
--env POSTGRES_HOST="DB_IP_OR_HOSTNAME" \
45+
--env POSTGRES_DB="DB_NAME" \
46+
--env POSTGRES_USER="DB_USERNAME" \
47+
--env POSTGRES_PORT="NON_DEFAULT_PORT" \
48+
--env POSTGRES_PASSWORD="DB_USERNAME_PASSWORD" \
49+
--env NOTIFICATION_SERVER_URL="ONLY_SET_IF_PRIVATE_TELEGRAM_BOT_USED" \
50+
--env TELEGRAM_CHAT_ID="PRIVATE_OR_TELEGRAM_BOT_ID" \
51+
--env S3_ENDPOINT="S3_API_URL" \
52+
--env S3_ACCESS_KEY="S3_ACCESS_KEY" \
53+
--env S3_SECRET_KEY="S3_SECERT_KEY" \
54+
--env S3_OBJECT_PATH="FULL_S3_PATH (e.g `bucket_name/project_name/stage_branch/database_name.tar.gz`)" \
55+
--env S3_ALIAS="S3_CONFIG_SET_ALIAS" \
56+
numdes/nd_postgres_backup:v0.3.0
4857
```
4958

50-
:wave: By default `SCHEDULE` variable is set to `@daily` in case if you need other scheduling options, please refer
51-
to `go-cron` *[Documentation](https://pkg.go.dev/github.com/robfig/cron?utm_source=godoc#hdr-Predefined_schedules)*.
59+
## Backup strategy
60+
61+
By default set to make backup every hour, plus one separate backup a day, plus one separate backup a week
62+
63+
Schedule can be tuned or changed by editing of `crontab` file
64+
65+
## Retention strategy
66+
67+
Maximum depth of storage for each type of backup can be tuned by changing values of these variables:
68+
69+
- `WEEKLY_BACKUP_LIMIT`
70+
- `DAILY_BACKUP_LIMIT`
71+
- `HOURLY_BACKUP_LIMIT`
72+
73+
Schedule of retention script (`retention.sh`) execution can be edited in `crontab` file
5274

5375
## Variables
5476

@@ -57,58 +79,37 @@ to `go-cron` *[Documentation](https://pkg.go.dev/github.com/robfig/cron?utm_sour
5779

5880
| Name | Description |
5981
|--------------------|-----------------------------|
60-
| DOCKERHUB_LOGIN | `Actions` Repository secret |
61-
| DOCKERHUB_PASSWORD | `Actions` Repository secret |
82+
| DOCKERHUB_USERNAME | `Actions` Repository secret |
83+
| DOCKERHUB_TOKEN | `Actions` Repository secret |
6284

6385
### Notification environmental variables
6486

65-
| Name | Description |
66-
|--------------------------|-----------------------------------------|
67-
| TELEGRAM_CHAT_ID | Notifying group |
68-
| PRIVATE_NOTIFICATION_URL | Private notifier URL |
69-
| TELEGRAM_BOT_TOKEN | Only used to call Telegram's public API |
87+
| Name | Description |
88+
|---------------------------|-----------------------------------------------------------------------|
89+
| NOTIFICATION_SERVER_URL | URL of private telegram bot |
90+
| TELEGRAM_CHAT_ID | Custom bot ID or Telegram Bot ID when bot created using `@botfather` |
91+
| TELEGRAM_BOT_TOKEN | Created by `@botfather` bot security token |
7092

7193
### Environmental variables
7294

73-
| Name | Default value | Is mandatory | Description |
74-
|---------------------|:--------------|:------------:|---------------------------------------------------------------------------|
75-
| POSTGRES_DB | - | YES | Database name |
76-
| POSTGRES_HOST | - | YES | PostgreSQL IP address or hostname |
77-
| POSTGRES_PORT | 5432 | - | Connection TCP port |
78-
| POSTGRES_USER | - | YES | Database user |
79-
| POSTGRES_PASSWORD | - | YES | Database user password |
80-
| POSTGRES_EXTRA_OPTS | --blobs | - | Extra options `pg_dump` run |
81-
| SCHEDULE | @daily | - | `go-cron` schedule. See [this](#backup-using-go-cron) |
82-
| HEALTHCHECK_PORT | 8080 | - | Port listening for cron-schedule health check. |
83-
| S3_ACCESS_KEY | - | YES | Key or username with RW access to bucket |
84-
| S3_SECRET_KEY | - | YES | Secret or password for `S3_ACCESS_KEY` |
85-
| S3_BUCKET | - | YES | Name of S3 bucket |
86-
| S3_ENDPOINT | - | YES | URL of S3 storage |
87-
| S3_OBJECT_PATH | - | NO | Full path to archive including bucket name and desired file name. If not present will be generated automatically |
88-
89-
### Notification selection
90-
91-
It is possible to use either private Telegram bot if you have it or Telegram public API.
92-
93-
In scenario with private bot `PRIVATE_NOTIFICATION_URL` must be set alongside with `TELEGRAM_CHAT_ID`.
94-
95-
In scenario with Telegram's public API `TELEGRAM_BOT_TOKEN` must be set as it is
96-
received (`Use this token to access the HTTP API:`) from `@BotFather` Telegram Bot. Variable `TELEGRAM_CHAT_ID` must be
97-
a proper Telegram ID of bot
98-
99-
In `docker ...` command need to replace:
100-
101-
```
102-
--env PRIVATE_NOTIFICATION_URL=http://webhook \
103-
--env TELEGRAM_CHAT_ID=point_to_notify_group \
104-
```
105-
106-
to
107-
108-
```
109-
--env TELEGRAM_BOT_TOKEN='XXXXXXX:XXXXxxxxXXXXxxx' \
110-
--env TELEGRAM_CHAT_ID=000000000 \
111-
```
95+
| Variable Name | Default Value | Is Mandatory? | Description |
96+
|---------------------------|:-------------:|:-------------:|----------------------------------------------------------------------|
97+
| HOURLY_BACKUP_PATH | `hourly` | NO | Path suffix to hourly-made backups storage |
98+
| DAILY_BACKUP_PATH | `daily` | NO | Path suffix to daily-made backups storage |
99+
| WEEKLY_BACKUP_PATH | `weekly` | NO | Path suffix to weekly-made backups storage |
100+
| HOURLY_BACKUP_LIMIT | `25` | NO | Max number of weekly backups |
101+
| DAILY_BACKUP_LIMIT | `10` | NO | Max number of daily backups |
102+
| WEEKLY_BACKUP_LIMIT | `5` | NO | Max number of hourly backups |
103+
| S3_ACCESS_KEY | - | YES | ${S3_BUCKET} READ-WRITE S3 ACCESS KEY |
104+
| S3_SECRET_KEY | - | YES | ${S3_BUCKET} READ-WRITE S3 ACCESS SECRET |
105+
| S3_ENDPOINT | - | YES | S3 API URL |
106+
| S3_BUCKET | - | YES | Path to hourly, daily, weekly directories. Including bucket name |
107+
| S3_ALIAS | `backup` | NO | Name of config set in `mcli` command `mcli alias set` |
108+
| S3_OBJECT_PATH | - | NO | Optional variable to use single backup [functionality](#manual-backup-without-schedule) |
109+
| POSTGRES_DB | - | YES | PostgreSQL database name |
110+
| POSTGRES_HOST | - | YES | PostgreSQL IP or host name |
111+
| POSTGRES_PORT | `5432` | NO | TCP connection port |
112+
| POSTGRES_USER | - | YES | DB usermane |
113+
| POSTGRES_PASSWORD | - | YES | DB username password |
114+
| POSTGRES_EXTRA_OPTS | `--blobs` | NO | `pg_dump` extra options |
112115

113-
- If `TELEGRAM_CHAT_ID` has a proper format (Only digits not less than 5 not more than 32) and `TELEGRAM_BOT_TOKEN` is
114-
set, script will try to send notification through Telegram's public API.

0 commit comments

Comments
 (0)