ETL/CDC: Implement suggestions by CodeRabbit

amotl · amotl · commit 8f848b3c1c35 · 2025-08-10T20:26:54.000+02:00
diff --git a/docs/_include/links.md b/docs/_include/links.md
@@ -1,3 +1,4 @@
+<!-- markdownlint-disable MD034 -->
 <!-- markdownlint-disable MD053 -->
 
 [Amazon DynamoDB Streams]: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html
diff --git a/docs/_static/css/custom.css b/docs/_static/css/custom.css
@@ -5,6 +5,9 @@ TODO: Upstream to crate-docs-theme.
 */
 
 /* General */
+.clearfix {
+    clear: both;
+}
 /*
 .sd-card-body {
     line-height: 1.1em;
diff --git a/docs/integrate/apache-kafka/index.md b/docs/integrate/apache-kafka/index.md
@@ -5,8 +5,8 @@
 :style: "float: right; text-align: right"
 [![Apache Kafka logo](https://kafka.apache.org/logos/kafka_logo--simple.png){h=60px}][Apache Kafka]
 <br>
-<a href="https://github.com/crate/cratedb-examples/actions/workflows/framework-flink-kafka-java.yml" target="_blank">
-    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/framework-flink-kafka-java.yml?branch=main&label=Apache%20Kafka,%20Apache%20Flink" loading="lazy"></a>
+<a href="https://github.com/crate/cratedb-examples/actions/workflows/framework-flink-kafka-java.yml" target="_blank" rel="noopener noreferrer">
+    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/framework-flink-kafka-java.yml?branch=main&label=Apache%20Kafka,%20Apache%20Flink" loading="lazy" alt="CI status: Apache Kafka, Apache Flink"></a>
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/apache-superset/index.md b/docs/integrate/apache-superset/index.md
@@ -8,8 +8,8 @@
 :style: "float: right; text-align: right"
 [![Apache Superset logo](https://cratedb.com/hs-fs/hubfs/Apache-Superset-Logo-392x140@2x.png?width=604&height=216&name=Apache-Superset-Logo-392x140@2x.png){h=60px}][Apache Superset]
 <br>
-<a href="https://github.com/crate/cratedb-examples/actions/workflows/application-apache-superset.yml" target="_blank">
-    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/application-apache-superset.yml?branch=main&label=Apache Superset" loading="lazy"></a>
+<a href="https://github.com/crate/cratedb-examples/actions/workflows/application-apache-superset.yml" target="_blank" rel="noopener noreferrer">
+    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/application-apache-superset.yml?branch=main&label=Apache Superset" loading="lazy" alt="CI status: Apache Superset"></a>
 ```
 ```{div}
 :style: "clear: both"
@@ -26,7 +26,7 @@ Apache Superset.
 :::{dropdown} **Managed Superset**
 ```{div}
 :style: "float: right"
-[![](https://github.com/crate/crate-clients-tools/assets/453543/9d07da87-8aff-4569-bf2a-0a16bf89f4bc){h=60px}](https://preset.io/)
+[![Preset Cloud](https://github.com/crate/crate-clients-tools/assets/453543/9d07da87-8aff-4569-bf2a-0a16bf89f4bc){height=60px loading=lazy}][Preset Cloud]
 ```
 
 [Preset Cloud] is a fully-managed, open-source BI for the modern data stack,
diff --git a/docs/integrate/aws-dms/index.md b/docs/integrate/aws-dms/index.md
@@ -7,7 +7,7 @@
 
 ```{div}
 :style: "float: right;"
-[![AWS DMS logo](https://icon.icepanel.io/AWS/svg/Database/Database-Migration-Service.svg){h=60px}][AWS Database Migration Service (AWS DMS)]
+[![AWS DMS logo](https://icon.icepanel.io/AWS/svg/Database/Database-Migration-Service.svg){height=60px loading=lazy}][AWS Database Migration Service (AWS DMS)]
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/azure-functions/index.md b/docs/integrate/azure-functions/index.md
@@ -6,7 +6,7 @@
 
 ```{div}
 :style: "float: right;"
-[![Azure Functions logo](https://www.vectorlogo.zone/logos/azurefunctions/azurefunctions-ar21.svg){h=60px}][Azure Functions]
+[![Azure Functions logo](https://www.vectorlogo.zone/logos/azurefunctions/azurefunctions-ar21.svg){height=60px loading=lazy}][Azure Functions]
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/dbt/index.md b/docs/integrate/dbt/index.md
@@ -8,8 +8,8 @@
 :style: "float: right; text-align: right"
 [![dbt logo](https://www.getdbt.com/_next/image?url=%2Fimg%2Flogos%2Fdbt-labs-logo.svg&w=384&q=75){h=60px}][dbt]
 <br>
-<a href="https://github.com/crate/cratedb-examples/actions/workflows/framework-dbt.yml" target="_blank">
-    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/framework-dbt.yml?branch=main&label=dbt" loading="lazy"></a>
+<a href="https://github.com/crate/cratedb-examples/actions/workflows/framework-dbt.yml" target="_blank" rel="noopener noreferrer">
+    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/framework-dbt.yml?branch=main&label=dbt" loading="lazy" alt="CI status: dbt"></a>
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/django/index.md b/docs/integrate/django/index.md
@@ -3,10 +3,10 @@
 
 ```{div}
 :style: "float: right; text-align: right"
-[![Django logo](https://static.djangoproject.com/img/logos/django-logo-positive.svg){h=60px}][Django]
+[![Django logo](https://static.djangoproject.com/img/logos/django-logo-positive.svg){height=60px loading=lazy}][Django]
 <br>
-<a href="https://github.com/surister/cratedb-django/actions/workflows/tests.yml" target="_blank">
-    <img src="https://img.shields.io/github/actions/workflow/status/surister/cratedb-django/tests.yml?branch=master&label=Django" loading="lazy"></a>
+<a href="https://github.com/surister/cratedb-django/actions/workflows/tests.yml" target="_blank" rel="noopener noreferrer">
+    <img src="https://img.shields.io/github/actions/workflow/status/surister/cratedb-django/tests.yml?branch=master&label=Django" loading="lazy" alt="CI status: CrateDB Django connector"></a>
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/explo/index.md b/docs/integrate/explo/index.md
@@ -3,7 +3,7 @@
 
 ```{div}
 :style: "float: right"
-[![](https://uploads-ssl.webflow.com/62f681c18d50329187681754/62f681c18d5032d0bd681785_logo.svg){h=60px}][Explo]
+[![Explo logo](https://uploads-ssl.webflow.com/62f681c18d50329187681754/62f681c18d5032d0bd681785_logo.svg){height=60px loading=lazy}][Explo]
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/gradio/index.md b/docs/integrate/gradio/index.md
@@ -3,10 +3,10 @@
 
 ```{div}
 :style: "float: right; text-align: right"
-[![Gradio logo](https://raw.githubusercontent.com/gradio-app/gradio/main/readme_files/gradio.svg){h=60px}][Gradio]
+[![Gradio logo](https://raw.githubusercontent.com/gradio-app/gradio/main/readme_files/gradio.svg){height=60px loading=lazy}][Gradio]
 <br>
-<a href="https://github.com/crate/cratedb-examples/actions/workflows/framework-gradio.yml" target="_blank">
-    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/framework-gradio.yml?branch=main&label=Gradio" loading="lazy"></a>
+<a href="https://github.com/crate/cratedb-examples/actions/workflows/framework-gradio.yml" target="_blank" rel="noopener noreferrer">
+    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/framework-gradio.yml?branch=main&label=Gradio" loading="lazy" alt="CI status: Gradio"></a>
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/llamaindex/index.md b/docs/integrate/llamaindex/index.md
@@ -6,10 +6,10 @@
 
 ```{div}
 :style: "float: right; text-align: right"
-[![LlamaIndex logo](https://www.llamaindex.ai/llamaindex.svg){h=60px}][LlamaIndex]
+[![LlamaIndex logo](https://www.llamaindex.ai/llamaindex.svg){height=60px loading=lazy}][LlamaIndex]
 <br>
-<a href="https://github.com/crate/cratedb-examples/actions/workflows/ml-llamaindex.yml" target="_blank">
-    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/ml-llamaindex.yml?branch=main&label=LlamaIndex" loading="lazy"></a>
+<a href="https://github.com/crate/cratedb-examples/actions/workflows/ml-llamaindex.yml" target="_blank" rel="noopener noreferrer">
+    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/ml-llamaindex.yml?branch=main&label=LlamaIndex" loading="lazy" alt="CI status: LlamaIndex"></a>
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/marquez/data_ingestion.py b/docs/integrate/marquez/data_ingestion.py
@@ -0,0 +1,90 @@
+from datetime import datetime, timedelta
+
+from airflow.models.dag import DAG
+from airflow.providers.postgres.operators.postgres import SQLExecuteQueryOperator
+
+with DAG(
+    "lineage-reporting-cratedb",
+    start_date=datetime(2024, 6, 28),
+    max_active_runs=1,
+    schedule="@daily",
+    default_args={"retries": 1, "retry_delay": timedelta(minutes=1)},
+    catchup=False,
+):
+
+    ingest_customers = SQLExecuteQueryOperator(
+        task_id="ingest_customers",
+        conn_id="cratedb_default",
+        sql="""
+            INSERT INTO public.Customers (CustomerName,Country)
+			SELECT CONCAT(mountain,' Corp.')
+					,country
+			FROM sys.summits
+			LIMIT 100;
+        """,
+		inlets=[{'namespace': 'example', 'name': 'sampledata'}],
+		outlets=[{'namespace': 'example', 'name': 'customers_table'}]
+    )
+
+    ingest_invoices = SQLExecuteQueryOperator(
+        task_id="ingest_invoices",
+        conn_id="cratedb_default",
+        sql="""
+           INSERT INTO public.Invoices(date,CustomerID)
+			SELECT ('2022-01-01'::TIMESTAMP)+concat(floor(random()*1000),' DAYS')::INTERVAL
+					,(SELECT CustomerID FROM public.Customers ORDER BY random()+a.b LIMIT 1)
+			FROM GENERATE_SERIES(1,1000) a(b);
+        """,
+		inlets=[{'namespace': 'example', 'name': 'customers_table'}],
+		outlets=[{'namespace': 'example', 'name': 'invoices_table'}]
+    )
+
+    ingest_products = SQLExecuteQueryOperator(
+        task_id="ingest_products",
+        conn_id="cratedb_default",
+        sql="""
+           INSERT INTO public.Products(Description,applicable_tax_percentage)
+			SELECT CONCAT('Product ',a.b)
+					,(floor(random()*10)+15)/100.0
+			FROM GENERATE_SERIES(1,10) a(b);
+        """,
+		inlets=[{'namespace': 'example', 'name': 'more_sample_data'}],
+		outlets=[{'namespace': 'example', 'name': 'products_table'}]
+    )
+
+    ingest_invoice_items = SQLExecuteQueryOperator(
+        task_id="ingest_invoice_items",
+        conn_id="cratedb_default",
+        sql="""
+           INSERT INTO public.Invoice_items (InvoiceID,ProductID,quantity,unit_price)
+			SELECT InvoiceID,ProductID
+					,1+ceil(random()*4)
+					,random()*1000
+			FROM public.Invoices
+			INNER JOIN public.Products ON random()>0.5;
+        """,
+		inlets=[{'namespace': 'example', 'name': 'invoices_table'},{'namespace': 'example', 'name': 'products_table'}],
+		outlets=[{'namespace': 'example', 'name': 'invoice_items_table'}]
+    )
+
+    refresh_customer_table = SQLExecuteQueryOperator(
+        task_id="refresh_customer_table",
+        conn_id="cratedb_default",
+        sql="REFRESH TABLE public.Customers;",
+    )
+
+    refresh_invoices_table = SQLExecuteQueryOperator(
+        task_id="refresh_invoices_table",
+        conn_id="cratedb_default",
+        sql="REFRESH TABLE public.Invoices;",
+    )
+
+    refresh_products_table = SQLExecuteQueryOperator(
+        task_id="refresh_products_table",
+        conn_id="cratedb_default",
+        sql="REFRESH TABLE public.Products;",
+    )
+
+    ingest_customers >> refresh_customer_table >> ingest_invoices >> refresh_invoices_table
+    refresh_invoices_table >> ingest_invoice_items
+    ingest_products >> refresh_products_table >> ingest_invoice_items
diff --git a/docs/integrate/marquez/index.md b/docs/integrate/marquez/index.md
@@ -6,7 +6,7 @@
 
 ```{div}
 :style: "float: right"
-[![Marquez logo](https://marquezproject.ai/img/marquez-social-card.jpg){h=60px}][Marquez]
+[![Marquez logo](https://marquezproject.ai/img/marquez-social-card.jpg){height=60px loading=lazy}][Marquez]
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/marquez/learn.md b/docs/integrate/marquez/learn.md
@@ -18,7 +18,7 @@ sudo ./docker/up.sh
 While Marquez starts, let's open another terminal and proceed installing the Astro CLI for Airflow:
 
 ```bash
-curl -sSL install.astronomer.io | sudo bash -s
+curl -sSL https://install.astronomer.io | sudo bash -s
 ```
 
 Let's initialize a project folder:
@@ -39,7 +39,10 @@ Let's now configure Airflow to use Marquez as the lineage repository and connect
 
 ```bash
 cat <<EOF >>.env
+# Linux (Docker bridge):
 OPENLINEAGE_URL=http://172.17.0.1:5000
+# macOS/Windows (Docker Desktop):
+# OPENLINEAGE_URL=http://host.docker.internal:5000
 OPENLINEAGE_NAMESPACE=example
 EOF
 ```
@@ -68,9 +71,21 @@ sudo docker run -d --name cratedb --publish=4200:4200 --publish=5436:5432 --env
 
 Let's now use the CrateDB CLI to prepare the tables we will use for this example:
 
+Install `crash` using `pip`.
+```bash
+sudo apt install -y python3-pip
+sudo pip3 install --user crash
+crash
+```
+
+Alternatively, install `crash` using `pipx`.
+```bash
+sudo apt install -y pipx
+sudo pipx install crash
+```
+
+Connect using `crash`.
 ```bash
-sudo apt install python3-pip
-sudo pip install crash
 crash
 ```
 
@@ -124,108 +139,13 @@ Port: 5436
 
 Click the Save button.
 
-Now we will create a DAG.
-Let's go back to the `datalineageeval` folder, and run the following:
-
+Now we will create a DAG. From the `datalineageeval` folder, run:
 ```bash
-cat <<EOF >> dags/data_ingestion.py
-```
-
-```python
-from datetime import datetime, timedelta
-
-from airflow.models.dag import DAG
-from airflow.providers.postgres.operators.postgres import SQLExecuteQueryOperator
-
-with DAG(
-    "lineage-reporting-cratedb",
-    start_date=datetime(2024, 6, 28),
-    max_active_runs=1,
-    schedule="@daily",
-    default_args={"retries": 1, "retry_delay": timedelta(minutes=1)},
-    catchup=False,
-):
-
-    ingest_customers = SQLExecuteQueryOperator(
-        task_id="ingest_customers",
-        conn_id="cratedb_default",
-        sql="""
-            INSERT INTO public.Customers (CustomerName,Country)
-			SELECT CONCAT(mountain,' Corp.')
-					,country
-			FROM sys.summits
-			LIMIT 100;
-        """,
-		inlets=[{'namespace': 'example', 'name': 'sampledata'}],
-		outlets=[{'namespace': 'example', 'name': 'customers_table'}]
-    )
-
-    ingest_invoices = SQLExecuteQueryOperator(
-        task_id="ingest_invoices",
-        conn_id="cratedb_default",
-        sql="""
-           INSERT INTO public.Invoices(date,CustomerID)
-			SELECT ('2022-01-01'::TIMESTAMP)+concat(floor(random()*1000),' DAYS')::INTERVAL
-					,(SELECT CustomerID FROM public.Customers ORDER BY random()+a.b LIMIT 1)
-			FROM GENERATE_SERIES(1,1000) a(b);
-        """,
-		inlets=[{'namespace': 'example', 'name': 'customers_table'}],
-		outlets=[{'namespace': 'example', 'name': 'invoices_table'}]
-    )
-
-    ingest_products = SQLExecuteQueryOperator(
-        task_id="ingest_products",
-        conn_id="cratedb_default",
-        sql="""
-           INSERT INTO public.Products(Description,applicable_tax_percentage)
-			SELECT CONCAT('Product ',a.b)
-					,(floor(random()*10)+15)/100.0
-			FROM GENERATE_SERIES(1,10) a(b);
-        """,
-		inlets=[{'namespace': 'example', 'name': 'more_sample_data'}],
-		outlets=[{'namespace': 'example', 'name': 'products_table'}]			
-    )
-
-    ingest_invoice_items = SQLExecuteQueryOperator(
-        task_id="ingest_invoice_items",
-        conn_id="cratedb_default",
-        sql="""
-           INSERT INTO public.Invoice_items (InvoiceID,ProductID,quantity,unit_price)
-			SELECT InvoiceID,ProductID
-					,1+ceiling(random()*4)
-					,random()*1000
-			FROM public.Invoices
-			INNER JOIN public.Products ON random()>0.5;
-        """,
-		inlets=[{'namespace': 'example', 'name': 'invoices_table'},{'namespace': 'example', 'name': 'products_table'}],
-		outlets=[{'namespace': 'example', 'name': 'invoice_items_table'}]			
-    )
-
-    refresh_customer_table = SQLExecuteQueryOperator(
-        task_id="refresh_customer_table",
-        conn_id="cratedb_default",
-        sql="REFRESH TABLE public.Customers;",
-    )
-
-    refresh_invoices_table = SQLExecuteQueryOperator(
-        task_id="refresh_invoices_table",
-        conn_id="cratedb_default",
-        sql="REFRESH TABLE public.Invoices;",
-    )
-
-    refresh_products_table = SQLExecuteQueryOperator(
-        task_id="refresh_products_table",
-        conn_id="cratedb_default",
-        sql="REFRESH TABLE public.Products;",
-    )
-
-    ingest_customers >> refresh_customer_table >> ingest_invoices >> refresh_invoices_table
-    refresh_invoices_table >> ingest_invoice_items
-    ingest_products >> refresh_products_table >> ingest_invoice_items
+cp data_ingestion.py dags/
 ```
 
+Then, restart Airflow.
 ```bash
-EOF
 sudo astro dev restart
 ```
 
diff --git a/docs/integrate/metabase/index.md b/docs/integrate/metabase/index.md
@@ -3,10 +3,10 @@
 
 ```{div}
 :style: "float: right; text-align: right"
-[![Metabase logo](https://www.metabase.com/images/logo.svg){h=60px}][Metabase]
+[![Metabase logo](https://www.metabase.com/images/logo.svg){height=60px loading=lazy}][Metabase]
 <br>
-<a href="https://github.com/crate/cratedb-examples/actions/workflows/application-metabase.yml" target="_blank">
-    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/application-metabase.yml?branch=main&label=Metabase" loading="lazy"></a>
+<a href="https://github.com/crate/cratedb-examples/actions/workflows/application-metabase.yml" target="_blank" rel="noopener noreferrer">
+    <img src="https://img.shields.io/github/actions/workflow/status/crate/cratedb-examples/application-metabase.yml?branch=main&label=Metabase" loading="lazy" alt="CI status: Metabase"></a>
 ```
 ```{div}
 :style: "clear: both"
diff --git a/docs/integrate/n8n/index.md b/docs/integrate/n8n/index.md
diff --git a/docs/integrate/plotly/index.md b/docs/integrate/plotly/index.md
diff --git a/docs/integrate/powerbi/index.md b/docs/integrate/powerbi/index.md
diff --git a/docs/integrate/powerbi/service.rst b/docs/integrate/powerbi/service.rst
diff --git a/docs/integrate/rill/index.md b/docs/integrate/rill/index.md
diff --git a/docs/integrate/streamsets/index.md b/docs/integrate/streamsets/index.md

Original file line number	Diff line number	Diff line change
`@@ -1,3 +1,4 @@`
	`1`	`+<!-- markdownlint-disable MD034 -->`
`1`	`2`	`<!-- markdownlint-disable MD053 -->`
`2`	`3`
`3`	`4`	`[Amazon DynamoDB Streams]: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html`