Can't see logs while task running (apache/airflow helm chart and CeleryExecutor)

[dstandish]

Is there anything special you need to do, in order to be able to see logs while the task in progress?

update

ok so i am close...

i was able to verify logs served correctly by forwarding the port locally:

kpf airflow-worker-6668df978c-nsmkx 8793:8793
Forwarding from 127.0.0.1:8793 -> 8793
Forwarding from [::1]:8793 -> 8793
Handling connection for 8793
Handling connection for 8793

Then i can access: http://localhost:8793/log/standish_emr_test/watch_step/2021-01-09T19:47:39.894619+00:00/4.log

But still I am unable to see from UI:

*** Log file does not exist: /opt/airflow/logs/standish_emr_test/watch_step/2021-01-09T19:47:39.894619+00:00/4.log
*** Fetching from: http://airflow-worker-6668df978c-nsmkx:8793/log/standish_emr_test/watch_step/2021-01-09T19:47:39.894619+00:00/4.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-6668df978c-nsmkx', port=8793): Max retries exceeded with url: /log/standish_emr_test/watch_step/2021-01-09T19:47:39.894619+00:00/4.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff775021e50>: Failed to establish a new connection: [Errno -2] Name or service not known'))

I also see that, interestingly that the worker service has no IP... not sure if that is normal:

# k get svc
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
airflow-flower                ClusterIP   123.4.2.222     <none>        5555/TCP            113m
airflow-pgbouncer             ClusterIP   123.4.192.16    <none>        6543/TCP,9127/TCP   113m
airflow-postgresql            ClusterIP   123.4.180.195   <none>        5432/TCP            113m
airflow-postgresql-headless   ClusterIP   None             <none>        5432/TCP            113m
airflow-redis                 ClusterIP   123.4.191.44    <none>        6379/TCP            113m
airflow-statsd                ClusterIP   123.4.116.195   <none>        9125/UDP,9102/TCP   113m
airflow-webserver             ClusterIP   123.4.104.8     <none>        8080/TCP            113m
airflow-worker                ClusterIP   None             <none>        8793/TCP            113m

Also tried diff hostname:

def get_hostname():
    return socket.getfqdn() + '.airflow.pod.cluster.local'

[dstandish]

Ultimately there were two issues:

The logging path was unspecified:

ENV AIRFLOW__LOGGING__BASE_LOG_FOLDER /opt/airflow/logs

Return IP for hostname callable

For whatever reason, dns does not seem to resolve properly. Using this resolved issue:

def get_hostname():
    return socket.gethostbyname(socket.gethostname())

ENV AIRFLOW__CORE__HOSTNAME_CALLABLE airflow_extension.hostname.get_hostname