-
Notifications
You must be signed in to change notification settings - Fork 130
chore: setup livesync for console and terminal. #3925
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
billy-the-fish
merged 20 commits into
latest
from
3923-docs-rfc-add-console-live-sync-instructions
Apr 4, 2025
Merged
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
85eda96
chore: setup livesync for console and terminal.
billy-the-fish 933046c
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish 618da28
Update _partials/_livesync-console.md
billy-the-fish e31e8b3
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish e21acad
chore: updates on review.
billy-the-fish 630d170
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish 7bd372e
chore: updates on review.
billy-the-fish 24e0788
chore: updates on review.
billy-the-fish c06d6ca
chore: updates on review.
billy-the-fish 0838a34
chore: finish the procedure.
billy-the-fish a39e1c3
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish 0cb9edf
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish fab7b49
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish 0fb8209
chore: update on review.
billy-the-fish 6915138
chore: update on review.
billy-the-fish 1122987
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish 6b209f1
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish bd508b7
chore: update on review.
billy-the-fish 3a30c7b
chore: update on review.
billy-the-fish 514a6f2
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| <Tag variant="hollow">Early access: TimescaleDB v2.18.0</Tag> | ||
| <Tag variant="hollow">Early access</Tag> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| <Tag variant="hollow">Early access: TimescaleDB v2.18.0</Tag> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| 1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database** | ||
|
|
||
| ```sql | ||
| psql $SOURCE -c "ALTER SYSTEM SET wal_level=’logical’;" | ||
| psql $SOURCE -c "ALTER SYSTEM SET max_wal_sender=10;" | ||
| psql $SOURCE -c "ALTER SYSTEM SET wal_sender_timeout=0;" | ||
| ``` | ||
| * [ GUC “wal_level” as “logical”](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-WAL-LEVEL) | ||
| * [GUC “max_wal_senders” as 10](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-WAL-SENDERS) | ||
| * [GUC “wal_sender_timeout” as 0](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-WAL-SENDER-TIMEOUT) | ||
|
|
||
| 1. **Enable update and delete replication on the source database** | ||
|
|
||
| Replica identity assists data replication by identifying the rows being modified. | ||
| By default each table and hypertable in the source database defaults to the primary key of the table being replicated. | ||
| However, you can also have: | ||
|
|
||
| - **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns | ||
| marked as `NOT NULL`. If a `UNIQUE` index does not exists, create one to assist the migration. You can delete it after | ||
| live sync. For each table, set `REPLICA IDENTITY` to the viable unique index: | ||
|
|
||
| ```sql | ||
| psql -X -d $SOURCE -c 'ALTER TABLE <table name> REPLICA IDENTITY USING INDEX <_index_name>' | ||
| ``` | ||
|
|
||
| - **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`: | ||
|
|
||
| ```sql | ||
| psql -X -d $SOURCE -c 'ALTER TABLE <table name> REPLICA IDENTITY FULL' | ||
| ``` | ||
| For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. | ||
| This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` | ||
| operations on the table, best practice is to not use `FULL` | ||
|
|
||
| To capture only `INSERT` and ignore `UPDATE`s and `DELETE`s, use a | ||
| [publish config](https://www.postgresql.org/docs/current/sql-createpublication.html#SQL-CREATEPUBLICATION-PARAMS-WITH-PUBLISH) | ||
| while [creating the publication][lives-sync-specify-tables]. | ||
|
|
||
|
|
||
| 1. **Restart your source database** | ||
|
|
||
|
|
||
| [lives-sync-specify-tables]: /migrate/:currentVersion:/livesync/#specify-the-tables-to-synchronize |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| import PrereqCloud from "versionContent/_partials/_prereqs-cloud-only.mdx"; | ||
| import LivesyncLimitations from "versionContent/_partials/_livesync-limitations.mdx"; | ||
| import LivesyncConfigureSourceDatabase from "versionContent/_partials/_livesync-configure-source-database.mdx"; | ||
| import TuneSourceDatabaseAWSRDS from "versionContent/_partials/_migrate_live_tune_source_database_awsrds.mdx"; | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| <PrereqCloud /> | ||
|
|
||
| - Install the [PostgreSQL client tools][install-psql] on your sync machine. | ||
|
|
||
| - Ensure that the source $PG instance and the target $SERVICE_LONG have the same extensions installed. | ||
|
|
||
| LiveSync does not create extensions on the target. If the table uses column types from an extension, | ||
| first create the extension on the target $SERVICE_LONG before syncing the table. | ||
|
|
||
| ## Limitations | ||
|
|
||
| <LivesyncLimitations /> | ||
|
|
||
| ## Set your connection string | ||
|
|
||
| This variable holds the connection information for the source database. In Terminal on your migration machine, | ||
| set the following: | ||
|
|
||
| ```bash | ||
| export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>" | ||
| ``` | ||
|
|
||
| <Highlight type="important"> | ||
| Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool | ||
| requires a direct connection to the database to function properly. | ||
| </Highlight> | ||
|
|
||
|
|
||
| ## Tune your source database | ||
|
|
||
| <Tabs label="Live migration"> | ||
|
|
||
| <Tab title="From PostgreSQL"> | ||
| <Procedure> | ||
|
|
||
| <LivesyncConfigureSourceDatabase /> | ||
|
|
||
| </Procedure> | ||
|
|
||
| </Tab> | ||
| <Tab title="From AWS RDS/Aurora"> | ||
|
|
||
| <Procedure> | ||
|
|
||
| <TuneSourceDatabaseAWSRDS /> | ||
|
|
||
| </Procedure> | ||
|
|
||
| </Tab> | ||
| </Tabs> | ||
|
|
||
| ## Synchronize data to your $SERVICE_LONG | ||
|
|
||
| To sync data from your PostgreSQL database to your $SERVICE_LONG using $CONSOLE: | ||
|
|
||
| <Procedure> | ||
|
|
||
| 1. **Connect to your $SERVICE_LONG** | ||
|
|
||
| In [$CONSOLE][portal-ops-mode], select the service to sync live data to. | ||
| 1. **Start livesync** | ||
| 1. Click `Actions` > `livesync for PostgreSQL`. | ||
|
|
||
| 1. **Connect the source database and target $SERVICE_SHORT** | ||
|
|
||
|  | ||
|
|
||
| In `livesync for PostgreSQL`: | ||
| 1. Set the `Livesync Name`. | ||
| 2. Set the` PostgreSQL Connection String` to point to the source database you want to sync to Timescale. | ||
| 3. Press `Continue`. | ||
| $CONSOLE connects to the source database and retrieves the schema information. | ||
|
|
||
| 1. **Optimize the data to syncronize in hypertables** | ||
|
|
||
|  | ||
| 1. Select the table to sync, and press `+`. | ||
| $CONSOLE checks the table schema and, if possible suggests the column to use as the time dimension in a hypertable. | ||
| 1. Repeat this step for each table you want to sync. | ||
| 1. Press `Start Livesync`. | ||
|
|
||
| $CONSOLE starts livesync between the source database and the target $SERVICE_SHORT and displays the progress. | ||
|
|
||
| 1. **Monitor syncronization** | ||
| 1. To view the progress of the livesync, click the name of the livesync process: | ||
|  | ||
| 1. To pause and restart livesync, click the buttons on the right of the livesync process and select an action: | ||
|  | ||
|
|
||
| </Procedure> | ||
|
|
||
| And that is it, you are using Livesync to synchronize all the data, or specific tables, from a PostgreSQL database | ||
| instance to your $SERVICE_LONG in real-time. | ||
|
|
||
| [install-psql]: /use-timescale/:currentVersion:/integrations/psql/ | ||
| [portal-ops-mode]: https://console.cloud.timescale.com/dashboard/services | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| * Schema changes must be co-ordinated. | ||
|
|
||
| Make compatible changes to the schema in your $SERVICE_LONG first, then make | ||
| the same changes to the source PostgreSQL instance. | ||
| * Ensure that the source $PG instance and the target $SERVICE_LONG have the same extensions installed. | ||
|
|
||
| LiveSync does not create extensions on the target. If the table uses column types from an extension, | ||
| first create the extension on the target $SERVICE_LONG before syncing the table. | ||
| * There is WAL volume growth on the source PostgreSQL instance during large table copy. | ||
| * This works for PostgreSQL databases only as source. Timescaledb is not yet supported. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.