New python library for loading with schema evolution, with native airflow support - feedback welcome

[adrianbr]

Open source Schema evoluton & declarative loading!

hey folks, i just want to put this new development on your radar, an open source library for data loading,complete with airflow support!

Please gib star if you like what you see! github

Schema evolution and declarative loading make pipeline building as easy as can be. I have been working on creating an open source python library to achieve this, and finally now we are ready to tell the world :)

dlt was made to be a simple library that fits in your stack without adding more redundance. It's made to primarily be a pipeline building tool, but we also host some ready made pipelines for common things.

It automates structuring and normalising your data so it gets loaded typed, for example json strings that are datetime will get converted to proper date time. This is part of schema evolution which is a process that helps reduce your code and maintenance and set alerts to defend data quality. This builds towards data meshing, or democratisation of data loading with contracts between consumer and producer.

Anyway, if you would like it and consider adoption, I, my team, or our growing community would be glad to assist you with it, find us on our slack

Here's a code example of how simple this is

import dlt

# have data? dlt likes data. It will take lists of rows or generators.
data = [{'id': 1, 'name': 'John'}]

# open connection. Credentials can be passed directly or read from a toml/env.
pipe = dlt.pipeline(destination='snowflake',
                    dataset_name='raw_data')

# Upsert/merge: Update old records, insert new
job_status = pipe.run(data, 
					  write_disposition="merge",
                      primary_key="id", # declare the merge key from our json
                      table_name="users")

# DBT runner included, to not need to handle creds separately
venv = dlt.dbt.get_venv(pipe)
dbt = dlt.dbt.package(pipe,
					"https://github.com/dbt-labs/jaffle_shop.git",
					venv=venv)

models = dbt.run_all()

# Save outcomes to db for monitoring
pipe.run([job_status, models], table_name="loading_status")

Here are some docs links for more info:
Getting started (3 colab demos and some capabilities)
Ecosystem - deployments, dbt runner, existing pipelines
Join us on our Slack!