Evolution of the CTK I/O subsystem

[amotl]

Hi there,

the I/O subsystem in CrateDB Toolkit is prominent to conduct all sorts of import-/export-procedures using a universal interface.

• https://cratedb-toolkit.readthedocs.io/io/
• https://github.com/crate/cratedb-toolkit/tree/main/cratedb_toolkit/io

While the universal interface tries to be stable, the subsystem is undergoing some changes under the hood, which need discussions at anyone's disposal who wants to contribute in one way or another. Feel welcome to join the conversation by adding your notes or general feedback. ¹

With kind regards,
Andreas.

Footnotes

1. If you think you've discovered a bug, please navigate to the issue tracker instead. ↩

[amotl]

This is a living document that will continue to be edited as we go.

Introduction

The module cratedb_toolkit.io was started in 2023, and has undergone a few updates since then. While the concept, architecture, and interface are roughly the same since day 1, its ingredients are changing, because we are bundling quite a bunch of other FOSS software here where a few churns are coming from.

Native connectors

DynamoDB, Kinesis, MongoDB

A few I/O nozzle components in cratedb_toolkit.io are using the commons-codec package that specifically knows how to marshal incoming data payloads from different origins (BSON, DynamoDB/Kinesis, Generic) to CrateDB SQL. While those native connectors are tailored to CrateDB, they are also custom and proprietary in the sense that nobody else is using them too much, so the situation is far away from Linus' law "given enough eyeballs, all bugs are shallow".

3rd-party connectors

Other connector adapters are standing on the shoulders of 3rd-party packages, mostly brought into CTK using little interface adapters, nothing big. In case of InfluxDB and Polyglot, it's effectively just a single invocation in cratedb_toolkit.io.router.

from cratedb_toolkit.io.influxdb import influxdb_copy
influxdb_copy(source_url, target_url, progress=True)

from cratedb_toolkit.io.omniload.api import omniload_copy
omniload_copy(source_url, target_url, progress=True)

CSV, JSON, JSONL, Parquet: CTK is just using Polars and never looked back.
Open table formats: Adapters for Delta Lake and Apache Iceberg are provided by the deltalake and pyiceberg packages.
InfluxDB: I/O nozzles for InfluxDB and the InfluxDB line protocol format are provided by the influxio package.
Polyglot: Another opportunity to bundle access to a 20+/500+ connector ensemble through the dlt ecosystem was unlocked by the dlt-cratedb package the other day ¹. Bundles of curated dlt connectors are available per ingestr and omniload.

• ingestr: CLI tool to copy data between any databases crate-clients-tools#86
• I/O: Migrate from ingestr to omniload #842
• I/O: Migrate Kafka->CrateDB recipe from ingestr to omniload cratedb-examples#1842

Footnotes

1. Of course, a native+upstream dlt implementation for CrateDB would be preferred, but we need to tackle many teething problems first. For example, we also need SQLFluff to support CrateDB. For more information, see our probing patch CrateDB destination adapter for dlt. Currently, the "interim" dlt-cratedb package is mentioned and referred to on their page Community Destinations » CrateDB. It's the best we could do at the time of conception. ↩