after migrating from version 2.1.4 to 2.3.0 airflow is unable to view the scheduler logs

[tronlightracer]

Apache Airflow version

main (development)

What happened

After migrating from version 2.1.4 to 2.3.0 and running airflow db upgrade I try to view the logs of tasks run in 2.1.4 in the UI and I'm given this error:

*** Log file does not exist: /usr/local/airflow/logs/dag_id=builtin_macros_most_recent/run_id=manual__2022-04-13T15:11:11.269706+00:00/task_id=test_time/attempt=1.log
*** Fetching from: http://1e317b63d5f4:8793/log/dag_id=builtin_macros_most_recent/run_id=manual__2022-04-13T15:11:11.269706+00:00/task_id=test_time/attempt=1.log
*** Failed to fetch log file from worker. [Errno 111] Connection refused

It gives this error whether you update the database with airflow db upgrade or don't update the database at all.

What you think should happen instead

I should be able to view the logs from tasks ran in a different version than what the current version is.

How to reproduce

start airflow with version 2.1.4
run dags in 2.1.4
change dockerfile to that of version 2.3.0
check logs of tasks run in version 2.1.4 and you'll receive that error

Operating System

Docker (debian:buster)

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

quay.io/astronomer/ap-airflow-dev:2.1.4-7-buster-nightly-20220411
quay.io/astronomer/ap-airflow-dev:2.3.0.dev20220411

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

This looks lie deployment issue - I think you should open support request to Astronomer.

[wbg]

I am seeing the same problem on my test instance with 2.3.0, Standard Upgrade vom 2.2.4, no errors on the DB migration.

As far as i could reverse engineer it, it seems that there is only one entry in the log_template table and this one contains the new filename format
dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log
Based on this line of code https://github.com/apache/airflow/blob/main/airflow/models/dagrun.py#L1070 i would have expected, that the original filename template {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log should also be present, but with a lower id. I tried to add it to the table, but this did not solve the problem at hand. Will try to debug this a little bit more.

[potiuk]

I think there were some consideration about changing log templates @dstandish ?

[dstandish]

we'll have to look into this cc @ashb

[ashb]

Looks like basically the deprecation handling in the config values is not 100% percent on point for the log_filename_template as starting from https://github.com/apache/airflow/blob/main/airflow/configuration.py#L276 basically log_filename_template is set to the new default when the current value matches the old default.

upgraded_values was added to cope with this https://github.com/apache/airflow/blob/main/airflow/configuration.py#L260

I wonder what I missed...

[ashb]

@wbg Can you share how to reproduce/where and what you configured log template?

[wbg]

@ashb we did not explicitly configure our log filename templates, we were using the out of the box default before a & after the upgrade to 2.3.0. first thing after upgrade was that we noticed our logs were missing. we added explicit configuration afterwards as a workaround ...

[pspeter]

I also ran into this after upgrading airflow using the default config before and after the upgrade (like @wbg). Managed to migrate to the new logs folder structure using this quick and dirty script:

https://gist.github.com/pspeter/13c18ee6cf247a5ee2b3e3b2ee8a5470

Backup your logs before running this on the system (e.g. using kubectl exec -it <webserver> python)

[thaysa]

It looks like backward compatibility bug.

[Haapalaj]

This seems not to work. Even if doing also this:
UPDATE dag_run set log_template_id = 0 WHERE log_template_id IS NULL;
When using AWS S3 external log storage. It still tryes to fetch the old logs with the new template (Having the log_template_id = 1)

[Haapalaj]

Opened issue for this: #24244

[thaysa]

@Haapalaj : did you have a chance to test the fix #24153 ?

[Haapalaj]

@Haapalaj : did you have a chance to test the fix #24153 ?

Updated from airflow 2.3.2 to 2.3.3 and after doing this in the airflow db:
"
INSERT INTO log_template (id, filename, elasticsearch_id, created_at) VALUES (0, '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log', '{dag_id}{task_id}{run_id}_{try_number}', NOW());
"
Fetching old logs that previously did not open, started now to work!
So this is now resolved.

[Haapalaj]

Updated from airflow 2.3.2 to 2.3.3 and after doing this in the airflow db:

INSERT INTO log_template (id, filename, elasticsearch_id, created_at) VALUES (0, '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log', '{dag_id}{task_id}{run_id}_{try_number}', NOW());

Fetching old logs that previously did not open, started now to work!