Skip to content

Commit 377c2e7

Browse files
committed
Integrate/PostgreSQL: Add section with starter tutorial
The goal is to present concise walkthroughs without many bells and whistles, which get to the point of getting you started quickly. Use the canonical template to present another data nozzle based on the CTK Ingestr I/O subsystem.
1 parent bb54283 commit 377c2e7

File tree

4 files changed

+142
-0
lines changed

4 files changed

+142
-0
lines changed

docs/ingest/etl/index.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,10 @@ Use visual data flow and integration frameworks and platforms.
121121
MySQL and MariaDB are well-known free and open-source relational database management
122122
systems (RDBMS), available as standalone and managed variants.
123123

124+
- {ref}`postgresql`
125+
126+
PostgreSQL is the world's most advanced open source relational database.
127+
124128
- {ref}`sql-server`
125129

126130
Microsoft SQL Server Integration Services (SSIS) is a component of the Microsoft SQL
@@ -237,6 +241,7 @@ Load data from datasets and open table formats.
237241
- {ref}`n8n`
238242
- {ref}`nifi`
239243
- {ref}`node-red`
244+
- {ref}`postgresql`
240245
- {ref}`risingwave`
241246
- {ref}`sql-server`
242247
- {ref}`streamsets`

docs/integrate/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ n8n/index
5252
nifi/index
5353
node-red/index
5454
plotly/index
55+
postgresql/index
5556
Power BI <powerbi/index>
5657
prometheus/index
5758
pyviz/index

docs/integrate/postgresql/index.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
(postgresql)=
2+
# PostgreSQL
3+
4+
```{div} .float-right
5+
[![postgresql-logo](https://www.postgresql.org/media/img/about/press/elephant.png){height=60px loading=lazy}][PostgreSQL]
6+
```
7+
```{div} .clearfix
8+
```
9+
10+
:::{rubric} About
11+
:::
12+
13+
[PostgreSQL] is the world's most advanced open source relational database.
14+
15+
:::{rubric} Synopsis
16+
:::
17+
18+
```shell
19+
uvx 'cratedb-toolkit[io-ingestr]' load table \
20+
"postgresql://postgres:postgres@localhost:5432/test?table=public.demo" \
21+
--cluster-url="crate://crate:crate@localhost:4200/doc/postgresql_demo"
22+
```
23+
24+
:::{rubric} Learn
25+
:::
26+
27+
::::{grid}
28+
29+
:::{grid-item-card} Tutorial: Use CrateDB Toolkit
30+
:link: postgresql-tutorial
31+
:link-type: ref
32+
Load data from PostgreSQL into CrateDB using CrateDB Toolkit.
33+
:::
34+
35+
::::
36+
37+
38+
:::{toctree}
39+
:maxdepth: 1
40+
:hidden:
41+
Tutorial <tutorial>
42+
:::
43+
44+
45+
[PostgreSQL]: https://www.postgresql.org/
Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
(postgresql-tutorial)=
2+
# Load data from PostgreSQL into CrateDB
3+
4+
The tutorial will walk you through starting [PostgreSQL] and CrateDB,
5+
inserting a record into PostgreSQL, loading data into a CrateDB table,
6+
and validating that the data has been stored successfully.
7+
The data transfer is supported by the
8+
{ref}`CrateDB Toolkit Ingestr I/O <ctk:ingestr>` data pipeline elements.
9+
10+
## Prerequisites
11+
12+
Docker is used for running all components. This approach works consistently
13+
across Linux, macOS, and Windows. Alternatively, you can use Podman.
14+
15+
Create a shared network.
16+
```shell
17+
docker network create cratedb-demo
18+
```
19+
20+
Start CrateDB.
21+
```shell
22+
docker run --rm --name=cratedb --network=cratedb-demo \
23+
--publish=4200:4200 --publish=5432:5432 --env=CRATE_HEAP_SIZE=2g \
24+
docker.io/crate -Cdiscovery.type=single-node
25+
```
26+
27+
Start PostgreSQL.
28+
```shell
29+
docker run --rm --name=postgresql --network=cratedb-demo \
30+
--publish=5433:5432 --env "POSTGRES_HOST_AUTH_METHOD=trust" \
31+
docker.io/postgres postgres -c log_statement=all
32+
```
33+
34+
Prepare shortcuts for the CrateDB shell, CrateDB Toolkit, and the PostgreSQL client
35+
programs.
36+
37+
::::{tab-set}
38+
39+
:::{tab-item} Linux and macOS
40+
To make the settings persistent, add them to your shell profile (`~/.profile`).
41+
```shell
42+
alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash"
43+
alias ctk-ingest="docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk"
44+
alias psql="docker run --rm -i --network=cratedb-demo docker.io/postgres psql"
45+
```
46+
:::
47+
:::{tab-item} Windows PowerShell
48+
To make the settings persistent, add them to your PowerShell profile (`$PROFILE`).
49+
```powershell
50+
function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args }
51+
function ctk-ingest { docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk @args }
52+
function psql { docker run --rm -i --network=cratedb-demo docker.io/postgres psql @args }
53+
```
54+
:::
55+
:::{tab-item} Windows Command
56+
```shell
57+
doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $*
58+
doskey ctk-ingest=docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk $*
59+
doskey psql=docker run --rm -i --network=cratedb-demo docker.io/postgres psql $*
60+
```
61+
:::
62+
63+
::::
64+
65+
## Usage
66+
67+
Write a few sample records to PostgreSQL.
68+
```shell
69+
psql "postgresql://postgres:postgres@postgresql:5432/" <<SQL
70+
CREATE DATABASE test;
71+
\connect test;
72+
CREATE TABLE IF NOT EXISTS demo (id BIGINT, data JSONB);
73+
INSERT INTO demo (id, data) VALUES (1, '{"temperature": 42.84, "humidity": 83.1}');
74+
INSERT INTO demo (id, data) VALUES (2, '{"temperature": 84.84, "humidity": 56.99}');
75+
SQL
76+
```
77+
78+
Invoke the data transfer pipeline.
79+
```shell
80+
ctk-ingest load table \
81+
"postgresql://postgres:postgres@postgresql:5432/test?table=public.demo" \
82+
--cluster-url="crate://crate:crate@cratedb:4200/doc/postgresql_demo"
83+
```
84+
85+
Inspect data stored in CrateDB.
86+
```shell
87+
crash --hosts cratedb -c "SELECT * FROM doc.postgresql_demo"
88+
```
89+
90+
91+
[PostgreSQL]: https://www.postgresql.org/

0 commit comments

Comments
 (0)