fix: update path handling in get_dag_directory() to return the absolute path of the symlink

[oboki]

Somtimes, I use symlinks in my local environment, like this:

❯ ls -rtlh /opt/airflow/
total 0
lrwxrwxrwx 1 oboki oboki 39 Sep 19 21:24 dags -> /home/oboki/projects/oboki-airflow/dags
lrwxrwxrwx 1 oboki oboki 42 Sep 19 21:24 plugins -> /home/oboki/projects/oboki-airflow/plugins

In this case, the get_dag_directory() function follows the symlink and returns the original path, which causes issues.

To resolve this, it should return the absolute path of the symlink itself.

Even though airflow.cfg is set with dags_folder = /opt/airflow/dags, if that path is a symlink, it returns /home/oboki/projects/oboki-airflow/dags, leading to dag_not_in_current_folder = True and causing issues. (a valid DAG is periodically deactivated.)

I changed manager.py to use the absolute path of symlink itself instead of resolved one.

[afranzi]

We are having the same issue using the gitSync in our K8s deployment 😢.
After upgrading from 2.9.3 to 2.10.1, we started having the same issue, where the dag_processor_manager adds the DAGs but the scheduler removes them in the code shared by @oboki.

Could we include this PR in the 2.10.2 please?

[bitomukesh]

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Bug Fix - Fix Symlink Path Handling in DAG Directory	- manager.py - Changed get_dag_directory() to return absolute path instead of resolved path

[ephraimbuddy]

We are having the same issue using the gitSync in our K8s deployment 😢. After upgrading from 2.9.3 to 2.10.1, we started having the same issue, where the dag_processor_manager adds the DAGs but the scheduler removes them in the code shared by @oboki.

Could we include this PR in the 2.10.2 please?

The issue you had has been fixed in 2.10.2, which will be released later today. Makes me think that this PR is also a result of the same problem.

[afranzi]

@ephraimbuddy even if the 2.10.2 is reverting the changes causing the issue, I believe this PR still adds value and robustness to airflow, so I believe it would be a good addition.

[ephraimbuddy]

@ephraimbuddy even if the 2.10.2 is reverting the changes causing the issue, I believe this PR still adds value and robustness to airflow, so I believe it would be a good addition.

There's a similar change here: https://github.com/apache/airflow/pull/42142/files. So yeah, we will look into it

[afranzi]

I believe this one would be cleaner than the other one you mentioned