Tasks within task groups are not scheduled independently #42201
Unanswered
SKisContent
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have a DAG that's defined similar to the following
The overall Airflow parallelism is set to 3 and the DAG concurrency is set to 3.
When this runs, the execution sequence is:
TG1_S1
TG2_S1
TG3_S1
TG4_S1
TG5_S1
TG1_S2
TG2_S2
TG3_S2
TG4_S2
TG5_S2
TG1_S3
TG2_S3
TG3_S3
TG4_S3
TG5_S3
After Airflow has started executing the DAG and filled the parallel slots at the beginning, I want it to prioritize finishing a task group before starting another one. So something like this:
TG1_S1
TG2_S1
TG3_S1
TG1_S2
TG2_S2
TG1_S3
TG3_S2
TG2_S3
TG3_S3
TG4_S1
TG5_S1
TG4_S2
TG4_S3
TG5_S2
TG5_S3
So the logic for scheduling the next task should be:
If there is a task group in progress with no running tasks then schedule the next task in that task group
else start executing the tasks in a new task group
What can I do to configure my DAG to run this way?
Beta Was this translation helpful? Give feedback.
All reactions