Skip to content

Commit dd3d128

Browse files
committed
update readme
1 parent 99607a4 commit dd3d128

File tree

1 file changed

+1
-100
lines changed
  • change_sorting_key_kafka_data_source

1 file changed

+1
-100
lines changed
+1-100
Original file line numberDiff line numberDiff line change
@@ -1,102 +1,3 @@
11
# Tinybird Versions - Change Sorting Key to Kafka connected Data Source
22

3-
The solution consists of creating a Materialized View with the desired Sorting Key, between the Kafka Landing Data Source and its endpoints.
4-
5-
[Pull Request](https://github.com/tinybirdco/use-case-examples/pull/195/files)
6-
7-
## Initial status
8-
9-
We have two endpoints connected to a Kafka Data Source. We want to change the Sorting Key to the Data Source.
10-
11-
![Initial status](imgs/1.png)
12-
13-
## Create a Materialized View with the desired Sorting Key and change the Endpoint pipes
14-
15-
- Create the Materialized View ([Pipe](./pipes/mat_product_events.pipe) and [Data Source](./datasources/product_events_mv.datasource)) with the new desired Sorting Key.
16-
17-
- Change your endpoints to make use of the new Materialized View:
18-
19-
`last_month_most_selling_products.pipe`:
20-
```diff
21-
@@ -7,10 +7,10 @@
22-
SQL >
23-
SELECT
24-
product_id,
25-
COUNT(*) AS c,
26-
toMonth(now() - INTERVAL 1 MONTH) as month
27-
- FROM product_events
28-
+ FROM product_events_mv
29-
WHERE
30-
event_name = 'buy-product' AND
31-
- __timestamp >= toStartOfMonth(now()) - INTERVAL 1 MONTH
32-
+ timestamp >= toStartOfMonth(now()) - INTERVAL 1 MONTH
33-
GROUP BY
34-
product_id
35-
ORDER BY
36-
c DESC
37-
```
38-
39-
`last_month_most_visited_categories.pipe`:
40-
```diff
41-
@@ -7,10 +7,10 @@
42-
SQL >
43-
SELECT
44-
product_type,
45-
COUNT(*) AS c,
46-
toMonth(now() - INTERVAL 1 MONTH) as month
47-
- FROM product_events
48-
+ FROM product_events_mv
49-
WHERE
50-
event_name = 'page-load' AND
51-
- __timestamp >= toStartOfMonth(now()) - INTERVAL 1 MONTH
52-
+ timestamp >= toStartOfMonth(now()) - INTERVAL 1 MONTH
53-
GROUP BY
54-
product_type
55-
ORDER BY
56-
c DESC
57-
```
58-
59-
## Deployment
60-
61-
- Bump the major version in the `.tinyenv`:
62-
63-
```diff
64-
- 0.0.0
65-
+ 1.0.0
66-
```
67-
68-
Bumping the major the deployment will create a `Preview Release` with the modifications. The recently created Materialized View `product_events_mv` will start to receive production data since it's created in the `Preview release`.
69-
70-
> Previews are very convenient to check our changes after deploying them to `live`. But also required if we want to backfill the the Materialized View with the `live` data previous to their creation.
71-
72-
- Push the changes to a branch and initiate a Pull Request. If it's your case, the Continuous Integration (CI) process will validate the changes through Regression, Quality, and Fixture tests ([learn more about testing](https://versions.tinybird.co/docs/version-control/implementing-test-strategies.html)).
73-
74-
- Before merging, verify your adjustments in the temporary Tinybird Branch that is provisioned.
75-
76-
- Merge the PR to trigger the Continuous Deployment (CD) workflow will deploy your changes in a Preview Release.
77-
78-
## Backfill
79-
80-
- At this moment you can decide if you want to backfill the old data:
81-
82-
```bash
83-
tb --semver 1.0.0 pipe populate mat_product_events --node product_events_0 --sql-condition "timestamp < $BACKFILL_DATE" --wait
84-
```
85-
86-
where `$BACKFILL_DATE` is your earliest row timestamp in the recently created Materialized View `product_events_mv` in the `Preview Release`.
87-
88-
89-
## Promote from Preview to Live
90-
91-
- Once you're happy with your Preview Release you can promote it to `live` following one of the next options:
92-
93-
- The action `Tinybird - Releases Workflow` in the case you are using our workflow templates.
94-
- Promote from the UI.
95-
- Or CLI:
96-
97-
```sh
98-
tb release promote --semver 1.0.0
99-
```
100-
101-
![Final status](imgs/2.png)
102-
3+
Please refer to [this example](../change_sorting_key_landing_data_source/README.md).

0 commit comments

Comments
 (0)