Creating multiple DAG with similar tasks but configuration

[PaarthB]

I have a requirement to have a DAG which can be run in different ways depending on the configuration. I see I can provide configuration for the DAG run itself from UI. What I would ideally want is multiple different DAGs doing the same tasks but appearing distinctly in the UI. This is because I want to restrict access to these similar DAGs based on owner.

One way could be to repeat the DAG multiple times in the python files in dags folder, with their separate owner and config. But this can quickly become too large. Also the number of these DAGs are not fixed, and would need to be determined at runtime. For ex, at runtime, I would determine how many such "duplicate" DAGs need to be created and have the config for them, and if they already exist, I won't create them again.

What is the best way to achieve this?

[turbaszek]

Have you considered using TriggerDagRunOperator? You can create a new DAG that will trigger your DAG as many times as you want with different configs (conf argument of TriggerDagRunOperator). So basically something like that:

with DAG("main_dag", ...) as dag:
    for i, conf in generate_configs():
        TriggerDagRunOperator(task_id=f"task_id_{i}", conf=conf, trigger_dag_id="your_dag_id")

Mind that dynamic DAGs can be problematic in case of backfills or reruns - for example if they are time based.

[PaarthB]

Won't this mean that there will be one DAG which will cause all the different TriggerDagRunOperator to run together at once, when the outer DAG is triggered? If one user wants to trigger their specific TriggerDagRunOperator, then it will cause all such DAGs to trigger.

For this reason, I think I would have to separate these DAGs to be run independently of each other. This would likely require having multiple "clone" DAGs, not being triggered by one outer DAG, but have their own independent DAG. I think this is explained here, without using multiple files: https://www.astronomer.io/guides/dynamically-generating-dags.

Do you think this is still achievable using the above way you showed?

[turbaszek]

I think this is explained here, without using multiple files: https://www.astronomer.io/guides/dynamically-generating-dags.

I think this sounds feasible for you use case. However, I'm afraid that what you want to achieve is multitenancy in Airflow which is not well supported by Airflow but yet can be achieved by multiple instances of Airflow managed centrally.