|
1 | 1 | # Tinybird Versions - Change Sorting Key to Kafka connected Data Source
|
2 | 2 |
|
3 |
| -The solution consists of creating a Materialized View with the desired Sorting Key, between the Kafka Landing Data Source and its endpoints. |
4 |
| - |
5 |
| -[Pull Request](https://github.com/tinybirdco/use-case-examples/pull/195/files) |
6 |
| - |
7 |
| -## Initial status |
8 |
| - |
9 |
| -We have two endpoints connected to a Kafka Data Source. We want to change the Sorting Key to the Data Source. |
10 |
| - |
11 |
| - |
12 |
| - |
13 |
| -## Create a Materialized View with the desired Sorting Key and change the Endpoint pipes |
14 |
| - |
15 |
| -- Create the Materialized View ([Pipe](./pipes/mat_product_events.pipe) and [Data Source](./datasources/product_events_mv.datasource)) with the new desired Sorting Key. |
16 |
| - |
17 |
| -- Change your endpoints to make use of the new Materialized View: |
18 |
| - |
19 |
| -`last_month_most_selling_products.pipe`: |
20 |
| -```diff |
21 |
| -@@ -7,10 +7,10 @@ |
22 |
| -SQL > |
23 |
| - SELECT |
24 |
| - product_id, |
25 |
| - COUNT(*) AS c, |
26 |
| - toMonth(now() - INTERVAL 1 MONTH) as month |
27 |
| -- FROM product_events |
28 |
| -+ FROM product_events_mv |
29 |
| - WHERE |
30 |
| - event_name = 'buy-product' AND |
31 |
| -- __timestamp >= toStartOfMonth(now()) - INTERVAL 1 MONTH |
32 |
| -+ timestamp >= toStartOfMonth(now()) - INTERVAL 1 MONTH |
33 |
| - GROUP BY |
34 |
| - product_id |
35 |
| - ORDER BY |
36 |
| - c DESC |
37 |
| -``` |
38 |
| - |
39 |
| -`last_month_most_visited_categories.pipe`: |
40 |
| -```diff |
41 |
| -@@ -7,10 +7,10 @@ |
42 |
| -SQL > |
43 |
| - SELECT |
44 |
| - product_type, |
45 |
| - COUNT(*) AS c, |
46 |
| - toMonth(now() - INTERVAL 1 MONTH) as month |
47 |
| -- FROM product_events |
48 |
| -+ FROM product_events_mv |
49 |
| - WHERE |
50 |
| - event_name = 'page-load' AND |
51 |
| -- __timestamp >= toStartOfMonth(now()) - INTERVAL 1 MONTH |
52 |
| -+ timestamp >= toStartOfMonth(now()) - INTERVAL 1 MONTH |
53 |
| - GROUP BY |
54 |
| - product_type |
55 |
| - ORDER BY |
56 |
| - c DESC |
57 |
| -``` |
58 |
| - |
59 |
| -## Deployment |
60 |
| - |
61 |
| -- Bump the major version in the `.tinyenv`: |
62 |
| - |
63 |
| -```diff |
64 |
| -- 0.0.0 |
65 |
| -+ 1.0.0 |
66 |
| -``` |
67 |
| - |
68 |
| -Bumping the major the deployment will create a `Preview Release` with the modifications. The recently created Materialized View `product_events_mv` will start to receive production data since it's created in the `Preview release`. |
69 |
| - |
70 |
| -> Previews are very convenient to check our changes after deploying them to `live`. But also required if we want to backfill the the Materialized View with the `live` data previous to their creation. |
71 |
| -
|
72 |
| -- Push the changes to a branch and initiate a Pull Request. If it's your case, the Continuous Integration (CI) process will validate the changes through Regression, Quality, and Fixture tests ([learn more about testing](https://versions.tinybird.co/docs/version-control/implementing-test-strategies.html)). |
73 |
| - |
74 |
| -- Before merging, verify your adjustments in the temporary Tinybird Branch that is provisioned. |
75 |
| - |
76 |
| -- Merge the PR to trigger the Continuous Deployment (CD) workflow will deploy your changes in a Preview Release. |
77 |
| - |
78 |
| -## Backfill |
79 |
| - |
80 |
| -- At this moment you can decide if you want to backfill the old data: |
81 |
| - |
82 |
| - ```bash |
83 |
| - tb --semver 1.0.0 pipe populate mat_product_events --node product_events_0 --sql-condition "timestamp < $BACKFILL_DATE" --wait |
84 |
| - ``` |
85 |
| - |
86 |
| - where `$BACKFILL_DATE` is your earliest row timestamp in the recently created Materialized View `product_events_mv` in the `Preview Release`. |
87 |
| - |
88 |
| - |
89 |
| -## Promote from Preview to Live |
90 |
| - |
91 |
| -- Once you're happy with your Preview Release you can promote it to `live` following one of the next options: |
92 |
| - |
93 |
| - - The action `Tinybird - Releases Workflow` in the case you are using our workflow templates. |
94 |
| - - Promote from the UI. |
95 |
| - - Or CLI: |
96 |
| - |
97 |
| - ```sh |
98 |
| - tb release promote --semver 1.0.0 |
99 |
| - ``` |
100 |
| - |
101 |
| - |
102 |
| - |
| 3 | +Please refer to [this example](../change_sorting_key_landing_data_source/README.md). |
0 commit comments