DAGS disappear from UI [2.5.1]

[taylorfinnell]

Apache Airflow version

Other Airflow 2 version (please specify below)

2.5.1

What happened

Hello,

Since upgrading to 2.5.1, we have noticed the sporadic disappearance of DAGs from the web UI. Sadly, we have not been able to reproduce it reliably. As such, this bug report is unfortunately lacking in details and is meant to hopefully get some ideas on how to approach debugging this.

One thing we have determined brings the DAGs back is to run airflow dags reserialize. It's worth noting that the DAG files are still on disk when missing in the UI. They also fail to show up in airflow dags list. Additionally, we believe, but are not certain, that the DAGs are still getting scheduled and executing work despite being absent in the web UI.

What you think should happen instead

DAGs should not disappear.

How to reproduce

We are looking for help or ideas on how to repro this.I'm looking for help or ideas on how to fix

Operating System

Centos 7

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

K8s

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

his bug report is unfortunately lacking in details and is meant to hopefully get some ideas on how to approach debugging this.

then it should be a discussion. converted it.

[potiuk]

I think you should describe the ways you are synchronizing your DAGs or to look what you are doing in top-level of your DAGs. Likely you have periods where DAGs are not visible for scheduler (DAG folder appears to be empty or files cannot be read or are partially visible/broken) or you have intermittent problems that parsing your DAGs are flaky and sometimes works, sometimes not.

BTW. It is a bad practice to have your DAGs reach out to external sources like DB/Network in top-level code https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code and any flakiness there might cause similar problems.

Also My comments from similar discussion in Slack https://apache-airflow.slack.com/archives/CSS36QQS1/p1680059510273789

Hard to say but maybe some database corruption
I think you should look at the logs in your database
For some kind of catastrophic events
You might want to reserialize dags (look in cli) and if it does not help likely recreating the db is a good idea - you will lose history but if your database is corrupted and you have no regular backup that you can restore it from, i think that might be the best way
For example if your synchronisation process deletes and recreates DAG files. Or when (for whatever reason) the DAG folder that scheduler sees is empty
It might be for example because it does not have permissions to see the dags
Those are things that come to my mind

[WattsInABox]

I work with Taylor.

I'm completely open to this being our fault somehow, but what made us suspect a bug is that this started happening with the 2.5.1 upgrade and we have made no other major changes. So far the database looks fine but we'll keep looking and we don't empty the DAG folder or delete DAG files through any automated processes.

Also, we are getting an absolutely ridiculous amount of "already registered for DAG" warnings (due to Airflow incorrectly assuming that the method name is the task name when performing this check) and we have over 1,200 DAGs. These two things seem to make reserialize take an inordinate amount of time. Would this be having an influence on the issue at hand?

Could there be a bug in Airflow 2.5.x? With at least one other person reporting something similar with 2.5.x, it seems suspicious.

[potiuk]

It could, of course. But without any tangible traces, logs, explaining how your dag are constructed it's impossible to say. That's including whether you do something unusual, and not following the best practices, especailly regaring the top-level code of yours and whether you for example reach out to an external DB while the DAGs are parsed - this is one of the reasons why your reserialize might be ridiculously slow.

Help us to help you by investigating it on your side and providing some tangible evidences and details that can be used by voiunteers helping poople lke you here for free trying to wildly guess what so special about your case (because obviously, it does not happen for everyone - at least so far we have no evidences of that) to diagnose it.

[potiuk]

Also, we are getting an absolutely ridiculous amount of "already registered for DAG" warnings (due to Airflow incorrectly assuming that the method name is the task name when performing this check) and we have over 1,200 DAGs. These two things seem to make reserialize take an inordinate amount of time. Would this be having an influence on the issue at hand?

TBH. I don't even know what you are talking about :). What name? What task? Remember that no-one here saw what you see what you write here is rather crypting for anyone who has no idea what your DAGs are and what your exact problems are that you are seeing.

[kobethuwis]

I join @potiuk in the fact that there is literally no way to help you if you don't share anything 😅 How are you instantiating your Operators & DAGs? Do you decorate them like @dag @task, do you simply import & use the official modules or both?

[teu]

Hi,

we have a similar issue with dynamic DAGs on airflow 2.3.3. I can see in the logs crearly that the DAG is being created and the mode is available in globals(), however for some reason some dags never make it to Airflow UI. Interestingly the dynamic DAGs are being generated by on file and some do make it and some dont.

Amazingly it does work fine in development, locally and I have printed the inspect.getmembers(dag_x) and compared them between my local log and production log and there is virtually no difference aside references and some dates.

airflow dags list does NOT show the dags in question
airflow dags report shows the dags in question

[potiuk]

You should look then at your DAG file processor environemnt and logs. The difference between airflow dags list and airflow dags report is that first shows you DAGs serialized to the DB, the second one loads the DAG file dynamically in the python interpreter you run with airflow CLI and reports from there.

The DAG File processor runs in the background and periodically stores them in the database in "serialized" form (just the structure of the DAGS) so that UI can display them - UI/Webserver never parses the DAG files - it displays whatever is serialized in the DB.

There can be several reasons why some of your DAGs are not serialized - for example your your processes of DAG file processor might be really busy and do not complete in https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dagbag-import-timeout .

But if you consistently see some DAGs missing and some DAGs available, then most likely your DAG file processor process has different environment than your local airflow dags report .

This process runs in background - airflow scheduler usually runs as non-interactive application and often it will have different environment variables, user. it will run on a different machine, possibly have different parmissions etc. This might lead to different behaviour of your Dynamic task generation - especially if this dynamic task generation uses environment variables or some files/information downloaded from external systems to generate the DAGs (because the information might depend on what and where your DAG file processor is run).

So you need to make sure your environment where DAG file processor process is run is what you expect it to be I am afraid.

BTW. DAG File processor runs usually as a child process of scheduler but you can have a "standaloine" processor instead - depending on your deployment.

[zach-nicoll-wcq]

@taylorfinnell I hope you have found a fix to this over a year later, but to anyone visiting this today:

TLDR; More RAM & threads, and check your docker image architecture

I was running into the exact same problem on my M1 Macbook, where I would see all the DAGs in the UI, and then within a minute they would start to disappear and potentially never return.

In the Scheduler logs, I could see that it was removing "serialized" tables from the database for no apparent reason (no files were being removed), and killing file processors too.

Today I realised that I was running the amd64 docker images on my arm64 computer. This was causing an extreme resource load on my machine, and it seems like the extra load of emulating amd64 was causing the Scheduler to run out of memory or CPU time or both.

Now that I am using the arm64 images, everything is way faster and more consistent.

I can still make the DAGs "disappear" by running a bunch at the same time, sapping all the resources available to the Scheduler.

So honestly it seems like you just need enough RAM and threads to keep Airflow from imploding on itself.

[teu]

To make a long story short - for large amounts of generated DAGs we had to make dag_file_processor_timeout and dagbag_import_timeout much larger than the defaults as it just failed silently after the timeout.

[potiuk]

I'd like to clarify: It's not silent. YYou should monitor the dag file processor logs and metrics for errors.

The dag file process does not fail silently. Also dag file processor emits metrics showing parsing failures if you want to monitor them. Can you please connect proper monitoring and verify (and confifrm here that it's not "silent" but "not monitored")?

[teu]

@potiuk Unfortunately, I do not work there anymore. As for failing silently, you might be correct that it emitted a metric that I did miss, such errors, it feels, would have been better propagated into the error log, not hidden away in a metric somewhere.

[potiuk]

They are also propagated to the log

[potiuk]

Specifically to DAGFileProcessor logs (Where all the stats and parsing results are also printed).