Airflow task killed randomly with error Executor reports task instance <TaskInstance: XXXXX [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?

[MauriSanfi]

Hi everyone,

I'm using Airflow 2.4.2 with the Kubernetes executor and I'm facing a strange issue. Every now and then some pods seem to get stuck while initializing, and after some minutes I receive an alert about the task being killed externally. This is a sample of the logs that I get when this happens.

[2022-12-13T12:05:34.498+0000] {processor.py:154} INFO - Started process (PID=1069050) to work on /home/dag/airflow_dags/dags/datawarehouse/exports/disaster_report.py
[2022-12-13T12:05:34.501+0000] {processor.py:756} INFO - Processing file /home/dag/airflow_dags/dags/datawarehouse/exports/disaster_report.py for tasks to queue
[2022-12-13T12:05:34.502+0000] {logging_mixin.py:120} INFO - [2022-12-13T12:05:34.502+0000] {dagbag.py:537} INFO - Filling up the DagBag from /home/dag/airflow_dags/dags/datawarehouse/exports/disaster_report.py
[2022-12-13T12:05:35.373+0000] {logging_mixin.py:120} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/operators/bigquery.py:38 DeprecationWarning: The `airflow.operators.sql.SQLCheckOperator` class is deprecated. Please use `'airflow.providers.common.sql.operators.sql.SQLCheckOperator'`.
[2022-12-13T12:05:35.396+0000] {logging_mixin.py:120} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/operators/bigquery.py:38 DeprecationWarning: The `airflow.operators.sql.SQLIntervalCheckOperator` class is deprecated. Please use `'airflow.providers.common.sql.operators.sql.SQLIntervalCheckOperator'`.
[2022-12-13T12:05:35.397+0000] {logging_mixin.py:120} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/operators/bigquery.py:38 DeprecationWarning: The `airflow.operators.sql.SQLValueCheckOperator` class is deprecated. Please use `'airflow.providers.common.sql.operators.sql.SQLValueCheckOperator'`.
[2022-12-13T12:05:35.463+0000] {logging_mixin.py:120} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py:53 RemovedInAirflow3Warning: This module is deprecated. Please use `airflow.providers.common.sql.hooks.sql`.
[2022-12-13T12:05:35.562+0000] {logging_mixin.py:120} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py:258 RemovedInAirflow3Warning: provide_context is deprecated as of 2.0 and is no longer required
[2022-12-13T12:05:35.564+0000] {logging_mixin.py:120} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagbag.py:338 DeprecationWarning: The bigquery_conn_id parameter has been deprecated. You should pass the gcp_conn_id parameter.
[2022-12-13T12:05:35.575+0000] {processor.py:766} INFO - DAG(s) dict_keys(['disaster_report_export']) retrieved from /home/dag/airflow_dags/dags/datawarehouse/exports/disaster_report.py
[2022-12-13T12:05:35.594+0000] {logging_mixin.py:120} INFO - [2022-12-13T12:05:35.594+0000] {taskinstance.py:1853} ERROR - Executor reports task instance <TaskInstance: disaster_report_export.get_config_sheet scheduled__2022-12-13T11:45:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?

Just for clarification, the main airflow components are deployed in a different node pool than the one I use for the tasks.
This error happens at random hours, for random DAGs. Autoscaling is enabled.

The logs from the cluster are not super useful, I can't actually relate the logs with this error. But here's what I got:

Failed to create pod sandbox: rpc error: code = 2 desc = failed to start sandbox container for pod XXXXX: operation timeout: context deadline exceeded

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container

Any ideas about what to check for and where?

Thanks in advance!

[potiuk]

Yep. Reported the user to Github as spammer.

[MauriSanfi]

I have solved the issue. The question can be closed.
It had to do with a combination of a certain configuration of the Kubernetes deployment along with some airflow configuration which were causing issues with the pods.

[sabarnwal]

@MauriSanfi can you please help on the solution here?

[Sunninsky]

Hi, @MauriSanfi we also met this issue on k8s, Could you share how you solve this? and which config you change?