Container status for KubernetesPodOperator / KubernetesExecutor

[wolfier]

Description

When a container fails, the reason is not presented is a clear manner.

For example, when the base container fails due to OOM, I would like to this some version of this printed in the task logs.

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 06 Jan 2022 12:03:22 -0500
      Finished:     Thu, 06 Jan 2022 12:03:46 -0500
    Ready:          False
    Restart Count:  208

Use case/motivation

It's difficult to determine why a pod fails when the logs only says it has an event of Failed or if the worker pod received a SIGTERM.

[2022-01-25, 20:13:24 UTC] {pod_launcher.py:236} INFO - Event: optimize-lh.0d0e03a1a72a4b0b8035cb49d0bd1de3 had an event of type Failed
[2022-01-25, 20:13:24 UTC] {pod_launcher.py:236} INFO - Event: optimize-lh.0d0e03a1a72a4b0b8035cb49d0bd1de3 had an event of type Failed
[2022-01-25, 20:13:25 UTC] {taskinstance.py:1267} INFO - Marking task as FAILED. dag_id=bin_optimizer, task_id=optimize_lh, execution_date=20220125T190000, start_date=20220125T201144, end_date=20220125T201325
[2022-01-25, 20:13:25 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2022-01-25, 20:13:25 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wolfier]

I believe it should be an easy additional (at least for the KubernetesPodOperator) since the information is part of the V1Pod class, V1Pod.status.container_statuses[].state.terminated.reason.

The change would most likely be in the cleanup section.

The information is in V1ContainerStateTerminated.

[raphaelauv]

With the version 3.0.0 of the cncf provider it's perfectly clear

 'status': {'conditions': None,
            'container_statuses': None,
            'host_ip': None,
            'init_container_statuses': None,
            'message': 'Pod ephemeral local storage usage exceeds the total '
                       'limit of containers 1G. ',
            'nominated_node_name': None,
            'phase': 'Failed',
            'pod_ip': None,
            'qos_class': None,
            'reason': 'Evicted',
            'start_time': datetime.datetime(2022, 2, 5, 10, 33, 46, tzinfo=tzlocal())}}
[2022-02-05 10:54:07,601] {taskinstance.py:1511} INFO - Marking task as FAILED. dag_id=XXXXXXXXXXXX, task_id=XXXXXXXXXXXX, execution_date=20220204T000000, start_date=20220205T103341, end_date=20220205T105407
[2022-02-05 10:54:07,779] {local_task_job.py:151} INFO - Task exited with return code 1