kubernetes executor cleanup_stuck_queued_tasks optimization

[dirrao]

Problem: Airflow running the cleanup_stuck_queued_tasks function on a certain frequency. When we run the airflow on a large Kube cluster (pods more than > 5K). Internally the cleanup_stuck_queued_tasks function loops through each queued task (when they breach task queued timeout) and checks the corresponding worker pod existence in the Kube cluster. Right now, this existence check using list pods Kube API. The API is taking more than 1s. if there are 120 queued tasks, then it will take ~ 120 seconds (1s * 120). So, this leads the scheduler to spend most of its time in this function rather than scheduling the tasks. It leads to none of the jobs being scheduled or degraded scheduler performance.

Solution: Use single k8 list pods batch api call to get all the worker pod owned by scheduler. Prepare the set of searchable strings using pod labels. Use this set data structure and identify whether the task associated pod exists or not. This reduces the number kube api sever calls significantly.

set elements string format:
(dag_id=<dag_id>,task_id=<task_id>,airflow-worker=[,map_index=<map_index>],[run_id=<run_id>]

[dirrao]

@jedcunningham / @hussein-awala
Can you review it whenever you are free?

[dirrao]

@jedcunningham / @hussein-awala
Can you review it whenever you are free?

[dirrao]

@potiuk / @eladkal
Can someone review this MR?

[potiuk]

@dirrao I have very little knowiedge of those but maybe look at the history of the releavant code and ping someone who was actively implementing it before? That's the best way to find who might be good to review it rather rather than putting that on my and @eladkal shoulders?

[jedcunningham]

@dirrao can you add some details in the description? Just repeating the commit message/title isn't very useful, and having to go grok 100+ lines of change to know what the goal is isn't great for reviewing now nor next year when someone is doing git blame :)

e.g. things like what is done now, what you are doing instead, expected impact.

[dirrao]

@dirrao can you add some details in the description? Just repeating the commit message/title isn't very useful, and having to go grok 100+ lines of change to know what the goal is isn't great for reviewing now nor next year when someone is doing git blame :)

e.g. things like what is done now, what you are doing instead, expected impact.

Sorry for not putting the details around the problem. I have updated the details in description of the PR.

[bitomukesh]

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Feature Improvement - Optimize Kubernetes Executor's Cleanup Process	- kubernetes_executor.py - Refactored cleanup_stuck_queued_tasks to use batch API calls for improved performance - CHANGELOG.rst - Added warning about removal of execution_date support for pod identification - provider.yaml - Updated version number to 9.0.0 - test_kubernetes_executor.py - Updated tests to reflect changes in cleanup_stuck_queued_tasks implementation