Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
85eda96
chore: setup livesync for console and terminal.
billy-the-fish Mar 13, 2025
933046c
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish Mar 13, 2025
618da28
Update _partials/_livesync-console.md
billy-the-fish Mar 14, 2025
e31e8b3
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish Mar 14, 2025
e21acad
chore: updates on review.
billy-the-fish Mar 14, 2025
630d170
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish Mar 14, 2025
7bd372e
chore: updates on review.
billy-the-fish Mar 14, 2025
24e0788
chore: updates on review.
billy-the-fish Mar 14, 2025
c06d6ca
chore: updates on review.
billy-the-fish Mar 14, 2025
0838a34
chore: finish the procedure.
billy-the-fish Mar 17, 2025
a39e1c3
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish Mar 18, 2025
0cb9edf
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish Mar 19, 2025
fab7b49
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish Mar 20, 2025
0fb8209
chore: update on review.
billy-the-fish Mar 20, 2025
6915138
chore: update on review.
billy-the-fish Mar 20, 2025
1122987
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish Mar 28, 2025
6b209f1
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish Apr 3, 2025
bd508b7
chore: update on review.
billy-the-fish Apr 3, 2025
3a30c7b
chore: update on review.
billy-the-fish Apr 3, 2025
514a6f2
Merge branch 'latest' into 3923-docs-rfc-add-console-live-sync-instru…
billy-the-fish Apr 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _partials/_cloud_self_configuration.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import EarlyAccess from "versionContent/_partials/_early_access.mdx";
import EarlyAccess from "versionContent/_partials/_early_access_2_18_0.mdx";

## Policies

Expand Down
2 changes: 1 addition & 1 deletion _partials/_early_access.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
<Tag variant="hollow">Early access: TimescaleDB v2.18.0</Tag>
<Tag variant="hollow">Early access</Tag>
1 change: 1 addition & 0 deletions _partials/_early_access_2_18_0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<Tag variant="hollow">Early access: TimescaleDB v2.18.0</Tag>
2 changes: 1 addition & 1 deletion _partials/_hypercore_manual_workflow.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import EarlyAccess from "versionContent/_partials/_early_access.mdx";
import EarlyAccess from "versionContent/_partials/_early_access_2_18_0.mdx";

1. **Stop the jobs that are automatically adding chunks to the columnstore**

Expand Down
43 changes: 43 additions & 0 deletions _partials/_livesync-configure-source-database.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database**

```sql
psql $SOURCE -c "ALTER SYSTEM SET wal_level=’logical’;"
psql $SOURCE -c "ALTER SYSTEM SET max_wal_sender=10;"
psql $SOURCE -c "ALTER SYSTEM SET wal_sender_timeout=0;"
```
* [ GUC “wal_level” as “logical”](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-WAL-LEVEL)
* [GUC “max_wal_senders” as 10](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-WAL-SENDERS)
* [GUC “wal_sender_timeout” as 0](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-WAL-SENDER-TIMEOUT)

1. **Enable update and delete replication on the source database**

Replica identity assists data replication by identifying the rows being modified.
By default each table and hypertable in the source database defaults to the primary key of the table being replicated.
However, you can also have:

- **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns
marked as `NOT NULL`. If a `UNIQUE` index does not exists, create one to assist the migration. You can delete it after
live sync. For each table, set `REPLICA IDENTITY` to the viable unique index:

```sql
psql -X -d $SOURCE -c 'ALTER TABLE <table name> REPLICA IDENTITY USING INDEX <_index_name>'
```

- **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`:

```sql
psql -X -d $SOURCE -c 'ALTER TABLE <table name> REPLICA IDENTITY FULL'
```
For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows.
This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE`
operations on the table, best practice is to not use `FULL`

To capture only `INSERT` and ignore `UPDATE`s and `DELETE`s, use a
[publish config](https://www.postgresql.org/docs/current/sql-createpublication.html#SQL-CREATEPUBLICATION-PARAMS-WITH-PUBLISH)
while [creating the publication][lives-sync-specify-tables].


1. **Restart your source database**


[lives-sync-specify-tables]: /migrate/:currentVersion:/livesync/#specify-the-tables-to-synchronize
103 changes: 103 additions & 0 deletions _partials/_livesync-console.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
import PrereqCloud from "versionContent/_partials/_prereqs-cloud-only.mdx";
import LivesyncLimitations from "versionContent/_partials/_livesync-limitations.mdx";
import LivesyncConfigureSourceDatabase from "versionContent/_partials/_livesync-configure-source-database.mdx";
import TuneSourceDatabaseAWSRDS from "versionContent/_partials/_migrate_live_tune_source_database_awsrds.mdx";

## Prerequisites

<PrereqCloud />

- Install the [PostgreSQL client tools][install-psql] on your sync machine.

- Ensure that the source $PG instance and the target $SERVICE_LONG have the same extensions installed.

LiveSync does not create extensions on the target. If the table uses column types from an extension,
first create the extension on the target $SERVICE_LONG before syncing the table.

## Limitations

<LivesyncLimitations />

## Set your connection string

This variable holds the connection information for the source database. In Terminal on your migration machine,
set the following:

```bash
export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
```

<Highlight type="important">
Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool
requires a direct connection to the database to function properly.
</Highlight>


## Tune your source database

<Tabs label="Live migration">

<Tab title="From PostgreSQL">
<Procedure>

<LivesyncConfigureSourceDatabase />

</Procedure>

</Tab>
<Tab title="From AWS RDS/Aurora">

<Procedure>

<TuneSourceDatabaseAWSRDS />

</Procedure>

</Tab>
</Tabs>

## Synchronize data to your $SERVICE_LONG

To sync data from your PostgreSQL database to your $SERVICE_LONG using $CONSOLE:

<Procedure>

1. **Connect to your $SERVICE_LONG**

In [$CONSOLE][portal-ops-mode], select the service to sync live data to.
1. **Start livesync**
1. Click `Actions` > `livesync for PostgreSQL`.

1. **Connect the source database and target $SERVICE_SHORT**

![Livesync wizard](https://assets.timescale.com/docs/images/livesync-wizard.png)

In `livesync for PostgreSQL`:
1. Set the `Livesync Name`.
2. Set the` PostgreSQL Connection String` to point to the source database you want to sync to Timescale.
3. Press `Continue`.
$CONSOLE connects to the source database and retrieves the schema information.

1. **Optimize the data to syncronize in hypertables**

![livesync start](https://assets.timescale.com/docs/images/livesync-start.png)
1. Select the table to sync, and press `+`.
$CONSOLE checks the table schema and, if possible suggests the column to use as the time dimension in a hypertable.
1. Repeat this step for each table you want to sync.
1. Press `Start Livesync`.

$CONSOLE starts livesync between the source database and the target $SERVICE_SHORT and displays the progress.

1. **Monitor syncronization**
1. To view the progress of the livesync, click the name of the livesync process:
![livesync view status](https://assets.timescale.com/docs/images/livesync-view-status.png)
1. To pause and restart livesync, click the buttons on the right of the livesync process and select an action:
![livesync start stop](https://assets.timescale.com/docs/images/livesync-start-stop.png)

</Procedure>

And that is it, you are using Livesync to synchronize all the data, or specific tables, from a PostgreSQL database
instance to your $SERVICE_LONG in real-time.

[install-psql]: /use-timescale/:currentVersion:/integrations/psql/
[portal-ops-mode]: https://console.cloud.timescale.com/dashboard/services
10 changes: 10 additions & 0 deletions _partials/_livesync-limitations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
* Schema changes must be co-ordinated.

Make compatible changes to the schema in your $SERVICE_LONG first, then make
the same changes to the source PostgreSQL instance.
* Ensure that the source $PG instance and the target $SERVICE_LONG have the same extensions installed.

LiveSync does not create extensions on the target. If the table uses column types from an extension,
first create the extension on the target $SERVICE_LONG before syncing the table.
* There is WAL volume growth on the source PostgreSQL instance during large table copy.
* This works for PostgreSQL databases only as source. Timescaledb is not yet supported.
Loading