Skip to content

Commit b4f3c26

Browse files
committed
prettier
1 parent 63a87d9 commit b4f3c26

File tree

1 file changed

+25
-15
lines changed

1 file changed

+25
-15
lines changed

docs/integrate/etl/iceberg-risingwave.md

Lines changed: 25 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,12 @@
11
(iceberg-risingwave)=
2+
23
# Stream processing from Iceberg tables to CrateDB using RisingWave
34

4-
[RisingWave] is a stream processing platform that allows configuring data sources, views on that data, and destinations where results are materialized.
5+
[RisingWave] is a stream processing platform that allows configuring data
6+
sources, views on that data, and destinations where results are materialized.
57

6-
This guide aims to show you an example with data coming from an Iceberg table and aggregations materialized in real-time in CrateDB.
8+
This guide aims to show you an example with data coming from an Iceberg table
9+
and aggregations materialized in real-time in CrateDB.
710

811
## Environment setup
912

@@ -18,7 +21,9 @@ podman run -d --name minio -p 9000:9000 -p 9001:9001 \
1821
quay.io/minio/minio server /data --console-address ":9001"
1922
```
2023

21-
And let's create a bucket called `warehouse`, for this point a browser to http://localhost:9001 , login with `minioadmin` / `minioadmin` , and click on "Create bucket", enter "warehouse", and click again on "Create bucket".
24+
And let's create a bucket called `warehouse`, for this point a browser to
25+
http://localhost:9001 , login with `minioadmin` / `minioadmin` , and click on
26+
"Create bucket", enter "warehouse", and click again on "Create bucket".
2227

2328
Then we will spin up an instance of RisingWave:
2429

@@ -34,16 +39,20 @@ podman run -d --name cratedb --publish=4200:4200 --publish=5432:5432 --env CRATE
3439

3540
We will need 3 consoles for this demonstration.
3641

37-
On the first console we will use [PyIceberg] and [IPython] to create an Iceberg table, and later we will add data and see how aggregations materialize in CrateDB in real-time.
42+
On the first console we will use [PyIceberg] and [IPython] to create an Iceberg
43+
table, and later we will add data and see how aggregations materialize in
44+
CrateDB in real-time.
3845

39-
On the second console we will do the RisingWave and CrateDB setups, and we will leave a Python script running for the streaming of changes.
46+
On the second console we will do the RisingWave and CrateDB setups, and we will
47+
leave a Python script running for the streaming of changes.
4048

4149
And on the 3rd console we will review how data appears in CrateDB.
4250

4351
## Creating an Iceberg table
4452

45-
Let's start on the first console.
46-
We use a Python script to create an Iceberg table on the bucket we created earlier on Minio, and as we want to keep things simple, we will use an ephemeral in-memory catalog.
53+
Let's start on the first console. We use a Python script to create an Iceberg
54+
table on the bucket we created earlier on Minio, and as we want to keep things
55+
simple, we will use an ephemeral in-memory catalog.
4756

4857
```bash
4958
pip install pyiceberg pyarrow s3fs
@@ -107,13 +116,15 @@ create_risingwave_compatible_metadata(table, "1")
107116

108117
Let's now switch to the second console.
109118

110-
To interact with both CrateDB and RisingWave we will use the `psql` command line utility, let's install it:
119+
To interact with both CrateDB and RisingWave we will use the `psql` command line
120+
utility, let's install it:
111121

112122
```bash
113123
sudo apt-get install -y postgresql-client
114124
```
115125

116-
Now let's connect first to CrateDB to create a table where we will keep the average reading for each sensor:
126+
Now let's connect first to CrateDB to create a table where we will keep the
127+
average reading for each sensor:
117128

118129
```bash
119130
psql -h localhost -U crate
@@ -137,7 +148,7 @@ psql -h localhost -p 4566 -d dev -U root
137148
```sql
138149
CREATE SOURCE sensors_readings
139150
WITH (
140-
connector = 'iceberg',
151+
connector = 'iceberg',
141152
database.name='db.db',
142153
warehouse.path='s3://warehouse/',
143154
table.name='sensors_readings',
@@ -147,7 +158,7 @@ WITH (
147158
s3.region = 'minio'
148159
);
149160
```
150-
161+
151162
And to materialize the averages:
152163

153164
```sql
@@ -246,8 +257,7 @@ table.append(data)
246257
create_risingwave_compatible_metadata(table, "2")
247258
```
248259

249-
Now let's go to the third console.
250-
Let connect to CrateDB:
260+
Now let's go to the third console. Let connect to CrateDB:
251261

252262
```bash
253263
psql -h localhost -U crate
@@ -280,8 +290,8 @@ table.append(data)
280290
create_risingwave_compatible_metadata(table, "3")
281291
```
282292

283-
If now we check again average_sensor_readings from the 3rd console we will see the average has already changed to 4.5
284-
293+
If now we check again average_sensor_readings from the 3rd console we will see
294+
the average has already changed to 4.5
285295

286296
[RisingWave]: https://github.com/risingwavelabs/risingwave
287297
[Podman]: https://github.com/containers/podman

0 commit comments

Comments
 (0)