PostgreSQL CDC

[amotl]

About

People are asking to do CDC from PostgreSQL to CrateDB. Please find a few research notes below.

Details

Dropping some notes from Carabas research and other backlogs.
See also curated information about PostgreSQL I/O per CTK-445.

Start

• https://pypi.org/project/core-cdc
• https://github.com/sshd123/pypgoutput
• https://pypi.org/project/pypg-cdc/

Progress

How to capture streams from different CDC interfaces?

• PostgreSQL Change Data Capture (CDC): The Complete Guide (2021)
• Arcion: PostgreSQL CDC: A Comprehensive Guide (2022)
• dbaStreet: Subscribing to PostgreSql logical replication using python and psycopg2 (2022)
• databricks: Production-ready data pipelines for analytics and AI (2025)
• https://github.com/dgea005/pypgoutput

Supabase

NB: common-codec's backlog also reveals pg_replicate made it into a full blown ETL layer today?

[amotl]

Status quo

Enumerating two more possibilities to read CDC streams from PostgreSQL into CrateDB.

• CrateDB Toolkit provides nozzles for Amazon Kinesis and AWS DMS.
  https://cratedb-toolkit.readthedocs.io/io/dms/
  https://github.com/crate/cratedb-toolkit/tree/main/cratedb_toolkit/io/kinesis

• The CrateDB connector for Estuary is also capable?
  https://estuary.dev/destination/cratedb/
  https://github.com/crate/cratedb-estuary

Thoughts

Otherwise, it would be so cool if CrateDB would be capable to support operations with Bruin/ingestr.

[amotl]

AWS DMS provides many obstacles, see collection of issues.

Maybe Apache Flink could save the world again?
Just use flink-cdc and the Flink Postgres CDC Connector.

-- https://cratedb-toolkit--445.org.readthedocs.build/io/postgresql/

/cc @hammerhead, @zolbatar

[amotl]

Wait, logical replication has versions?

Exploring versions of the Postgres logical replication protocol is a nice article about specific details.