Multiprocessing not working in airflow DAGs #42237
Unanswered
shawnli957
asked this question in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi, my DAG takes 3 minutes to run. Inside the DAG there are some loops. I tried to use multiprocessing.Process() to accerlate the loop. I trigger the original version and multiprocessed version at the same time for comparison, but it turns out both version of DAG run slower and they run for the exact the same duration.
By the way if I just run the script defined in the DAG with no DAG definations, it takes only 50 seconds. Also multiprocessing works in the script.
So how to enable multiprocessing in airflow DAGs? Or is there any other way I can accelerate my DAG? Thanks a lot.
Beta Was this translation helpful? Give feedback.
All reactions