How parallelism works in KubernetesExecutor? #47805
Unanswered
MaksymSofer
asked this question in
Q&A
Replies: 1 comment 1 reply
-
I enabled sending of traces to tempo and now i see that numbers actually count succeded pods |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello,
I use Airflow 2.10.1 in GKE, parallelism is set to 32
i have three dags that can bring (even on theoretical level only 12 simultaneous tasks).
but i see constant issue with Scheduler going into
Executor parallelism limit reached. 0 open slots.
Checking executor code, i see that open_slots defined as:
open_slots = self.parallelism - len(self.running)
I spotted that KubernetesExecutor leave "Succeeded" pods - is it possible that they calculated into "running" amount?
Please help me.
Beta Was this translation helpful? Give feedback.
All reactions