Import error : need to restart dag processor

[Jeoffreybauvin]

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When someone deploy a new python function, airflow is not happy with it :

Broken DAG: [/opt/airflow/dags/live/sha/appconf/testdag/testdag_encoursEspeceTitre_dag.py]
Traceback (most recent call last):
  File "<attrs generated init core.pipeline.spark.process.ModuleLevelPipeline>", line 28, in __init__
    self.__attrs_post_init__()
  File "/opt/airflow/dags/live/sha/core/pipeline/pipeline.py", line 181, in __attrs_post_init__
    from core.utils.logging_utils import init_bdt_log_filez
ImportError: cannot import name 'init_bdt_log_filez' from 'core.utils.logging_utils' (/opt/airflow/dags/live/sha/core/utils/logging_utils.py)

I'm using a standalone Dag Processor, and I need to restart it to fix the issue. Same with a non standalone dag processor, I need to restart the scheduler as well.

In fact, I'm mostly looking for "ways" to debug this, I don't understand why I need to restart the dag processor in order to fix things... My code is correctly synced between different containers...

What you think should happen instead?

No broken dags.

How to reproduce

Just push a new function.

Operating System

Debian 11 with Docker

Versions of Apache Airflow Providers

apache-airflow==2.9.1
apache-airflow-providers-amazon==8.20.0
apache-airflow-providers-apache-cassandra==3.4.2
apache-airflow-providers-apache-hdfs==4.3.3
apache-airflow-providers-apache-spark==4.7.2
apache-airflow-providers-celery==3.6.2
apache-airflow-providers-cncf-kubernetes==7.14.0
apache-airflow-providers-common-io==1.3.1
apache-airflow-providers-common-sql==1.12.0
apache-airflow-providers-docker==3.10.0
apache-airflow-providers-elasticsearch==5.3.4
apache-airflow-providers-fab==1.0.4
apache-airflow-providers-ftp==3.8.0
apache-airflow-providers-google==10.17.0
apache-airflow-providers-grpc==3.4.1
apache-airflow-providers-hashicorp==3.6.4
apache-airflow-providers-http==4.10.1
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-microsoft-azure==10.0.0
apache-airflow-providers-microsoft-mssql==3.6.1
apache-airflow-providers-mysql==5.5.4
apache-airflow-providers-odbc==4.5.0
apache-airflow-providers-openlineage==1.7.0
apache-airflow-providers-oracle==3.9.2
apache-airflow-providers-postgres==5.10.2
apache-airflow-providers-redis==3.6.1
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sftp==4.9.1
apache-airflow-providers-slack==8.6.2
apache-airflow-providers-smtp==1.6.1
apache-airflow-providers-snowflake==5.4.0
apache-airflow-providers-sqlite==3.7.1
apache-airflow-providers-ssh==3.10.1

Deployment

Docker-Compose

Deployment details

I'm using gitsync for airflow dags, and another git-sync for plugins.

Anything else?

.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Taragolis]

I'm using a standalone Dag Processor, and I need to restart it to fix the issue. Same with a non standalone dag processor, I need to restart the scheduler as well.

In fact, I'm mostly looking for "ways" to debug this, I don't understand why I need to restart the dag processor in order to fix things... My code is correctly synced between different containers...

There is no minimal reproducible example which might help someone to understand what is the issue here.
Until then it would considered as troubleshooting, so that is why it moved into the discussion

Please provide also deployment information, configurations, and everything which might help someone to understand what is going wrong here

[praveen-rekhapalli]

@Jeoffreybauvin Did you find the answer. I am facing the same issue. Mounted an Azure file share to all the airflow instances when a new subfolder is added the dag is recognized but the helper.py in the same folder is not found and getting the import error. After restarting the dag_processor it is working or if airflow in standalone mode need to restart the scheduler. I am not seeing any errors in dag_processor instance. Using Airflow version 2.9.1 running on AKS cluster

[praveen-rekhapalli]

@Jeoffreybauvin I fixed the problem by setting the AIRFLOW__CORE__LAZY_LOAD_PLUGINS to False which complied the modules along with dags and loaded them without import error

[praveen-rekhapalli]

extraEnv: |

• name: AIRFLOW__CORE__LAZY_LOAD_PLUGINS
  value: "False" makes the helper/ util moduels to be compiled along with the dag and prevents the restart of dag_processor also if using the helper outside the plugins folder add this line import os, sys
  sys.path.insert(0, '/opt/airflow/dags/') to the dag file where the /opt/airflow/dags is the location where the dags are loaded