Execute setup only if task is ready to run #40855
Unanswered
bjoernpollex-sc
asked this question in
General
Replies: 1 comment
-
Bumping this, it would be really great to get some feedback on this. One possible solution would be to turn this into a feature request for Airflow - add an option to schedule setup tasks only if at least one task within the scope of that setup task is ready to run. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I'm trying to design a DAG that builds two tables. For each of the build-tasks, I have to check some prerequisites before running them. Additionally, there is a shared setup-task that needs to run. The basic DAG looks something like this:
This works fine, except for one issue. When I clear the DAG, the setup is executed immediately. However, if both of the prerequisite checks fail, then the actual build-tasks can't run, so the setup was not necessary after all. Now, to fix this, I could add a task-dependency from the prerequisite checks to the setup-task, like this:
That solves the issue described above, but introduces an unwanted side-effect, because now
build_table1
is a downstream task ofcheck_table2_prerequisites
. This means that e.g. clearing all downstream tasks ofcheck_table2_prerequesites
would trigger a build oftable1
.Is there any way to solve this in Airflow?
shared_setup
is a task that needs to run before either oftable1
ortable2
is built, ideally only after the prerequisite check is green, without introducing a transitive dependency among unrelated tasks.Beta Was this translation helpful? Give feedback.
All reactions