Hanging tasks in the running state using DockerOpertator and StandardTaskRunner

[M3dv3D3V]

Hello, on Airflow 2.9.2 version, tasks are hanging in the running status using DockerOpertator and StandardTaskRunner as task_runner. There is a theory that freezes occur due to deadlocks, because one of the first messages in the logs talks about possible deadlocks if you use os.fork(), which is used in StandardTaskRunner (message below).

{warnings.py:112} WARNING - /home/airflow/.local/lib/python3.12/site-packages/airflow/task/task_runner/standard_task_runner.py:61: DeprecationWarning: This process (pid=62100) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()

The status of a hung task (process) - S (sleeping) in the STAT column, was obtained using the ps -aux command. An attempt to wake up the process using the kill -CONT <PID> command was unsuccessful.

After creating a new attempt at a hung task using the Clear task button in the Airflow UI, the previous task (process) goes into the Z - zombie state and remains in the system in the future. Zombie processes disappear only after a complete reboot of the system.

I will be glad to receive any comments and help in solving this problem. 🙏

[potiuk]

Speaking of zombie state, you might have similar problem as in #40596 where badly written Docker image (badly written init script) was not reaping the zombie processes correctly. You can read it and see if it causes your problems.

When it comes the threading warning, this is an interesting one, I've not seen that one before and we should take a look at that likely. I don't recall that we start threads in Airlfow by default but it might be that there are other things at play. If you are using open-lineage. I believe some older versions of openlineage started some threads and that kind of warning could be caused by those threads starting before fork() happened. And I believe recently it was changed to multi-processing: @mobuchowski @kacpermuda -> WDYT? can you comment on that please?

[kacpermuda]

Ok, good to know, so as Jarek said, it might be some issues related to your airflow deployment or docker operator and looks like OpenLineage has nothing to do with it.

[M3dv3D3V]

We may be working on verifying our deployment right now. I would also like to point out that freezes don't always happen. There are times when tasks don't hang after a reboot.

[M3dv3D3V]

@potiuk, thanks for your advice with dumb-init. We slightly changed the call /entrypoint of the script (as indicated in this discussion) and the zombie processes disappeared. But unfortunately, the processes also continue to hang.

[M3dv3D3V]

I would also like to add that after a few hours the message below appears and the task continues to hang:
{local_task_job_runner.py:163} INFO - Task is not able to be run

[potiuk]

Impossible to help you. You need to describe more details - including what and how you are running your Docker Operator and Standard Task Runner (but this is what everyone does) - It's likely you have some extra plugins or hooks or policies that start new threads before the fork happens - because this is not something that Airflow itself does.

So you should focus on either describing your case in detail or looking somewhere in the cusomisations, special deployment or anything that you customized in your Airflow deployment - because in no circumstances Airflow would start new threads before running a task. Ideally you can look at all your logs and see what is starting the threads - but that's something only you can do, possibly with tools like pyspy you can track what's going on and who calls what to find out.

This is what my guess is with the very limited information you provided.

[kacpermuda]

It would be good to know if you are indeed using OpenLineage. When it comes to OpenLineage, we had these two PRs: Snowflake and Redshift that may have been causing deadlocks. We later switched to ProcessPoolExecutor in this PR for DAG level events and the latest PR executes whole OL extraction in separate processes.

EDIT: I did not notice you mentioned DockerOperator, so not all i wrote makes sense.