Question - using smaller DAGs as pieces of larger DAGs

[FredericoCoelhoNunes]

Hi everyone,

I'm using Airflow to orchestrate my ETLs, and I have a question on best practices.

Basically I have an ETL composed of, let's say, 5 sequential SQL queries. I am going to be triggering this ETL independently every once in a while, but I also want to include it as a part of a larger ETL, in parallel with other similar sub-ETLs.

I would like to be able to:

• create a DAG for the basic ETL and store it in one module;
• somehow import this DAG into the "big" ETL and use it as a piece in it, the same way as I would use a task, setting dependencies with previous/following tasks, etc.

I came across the SubDag Airflow module, but the usage is pretty confusing to me (having to use a DAG factory, having to set the DAG Id to ".", etc.), and it seems to be recommended for repetitive tasks, and not to compose DAGs from smaller ones.

Has anyone faced this conceptual problem before? Any tips would be much appreciated!

[turbaszek]

I think you may be interested in TaskGroups:
http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskgroup

Also, I would consider creating for each of those queries a function that returns an instance of operator. Then you can simply invoke them in any DAG and easily reuse them 🤔

Also if you are using only PythonOperator you may be interested in:
http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api

[FredericoCoelhoNunes]

Hi turbaszek, thanks a lot for your reply. The only problem I have with your suggestion is that if the small-ETL changes (for example the order of two tasks is switched), I want this change to propagate to all larger ETLs that might be using this smaller ETL, without having to manually do it.

I suppose that having a function that returns a task group might work for this case. I just call this function once in the sub-ETL module to instantiate the small DAG with ID "small_dag", and then import this function and call it with ID "big_dag" in the main ETL module. Then I can use the resulting task group as such:

big_etl_task_1 >> small_etl_task_group >> big_etl_task_3

A couple of small issues for which I can immediately think of a solution:

• in the UI, the independent small-ETL will simply be a task group (so I'll have to zoom in every time)
• not 100% sure what is the best way to define the run configuration of the small-ETL and the big-ETL separately, since the small-ETL tasks are defined in a different module.

Thanks a lot for your ideas!

[turbaszek]

ETLs that might be using this smaller ETL, without having to manually do it.

You can create two operators in a function and chain them there and then return the last one, assuming that the relation is linear.

I suppose that having a function that returns a task group might work for this case.

That was what I had in mind 👌

in the UI, the independent small-ETL will simply be a task group (so I'll have to zoom in every time)

Yes, but I'm not sure if this is a big problem, especially that you can show/hide its content. It depends how often your etl will be changing.

not 100% sure what is the best way to define the run configuration of the small-ETL and the big-ETL separately, since the small-ETL tasks are defined in a different module.

Maybe the factory function may tak some arguments and pass them to underlying task group or operators? Then you can make the etl part fully configurable 🤔

[FredericoCoelhoNunes]

Thanks for exchanging some ideas with me @turbaszek !
It's clear to me now what is the best way to proceed. Cheers!

[turbaszek]

No problem 🎉