Docker: Python 3.10-based image fails to start (_mysql is not defined)

[omgMath]

Apache Airflow version

2.3.4

What happened

When using the docker image apache/airflow:2.3.4-python3.10, the container fails with

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/MySQLdb/__init__.py", line 18, in <module>
    from . import _mysql
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: cannot allocate memory in static TLS block

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/__init__.py", line 47, in <module>
    settings.initialize()
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py", line 562, in initialize
    configure_adapters()
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py", line 427, in configure_adapters
    import MySQLdb.converters
  File "/home/airflow/.local/lib/python3.10/site-packages/MySQLdb/__init__.py", line 24, in <module>
    version_info, _mysql.version_info, _mysql.__file__
NameError: name '_mysql' is not defined

However, when using apache/airflow:2.3.4-python3.9 the container starts without problems.

What you think should happen instead

The docker image should not fail to start.

How to reproduce

No response

Operating System

Docker Container

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

ARM images do not support MySQL -> look at the changelog.

[potiuk]

Ah I was too fast. I thin the problem is with some of your settings. Likely you do something else with the imaage or override LD_PRELOAD env variable.

The problem you are mentioning should be solved (and I just checked that it is still solved) by having this variable defined:

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6

This is in order to workaround mysql cluent wrongly allocating memory (this is mysql problem not Airflow). You should leep that variable set in order to keep workaround, well, working.

You can see more details here #17546

If you are somehow overriding this variable you will trigger MySQL problem so you should keep it set. I guess you override it with your own version - so please double check that you have it set. If it still does not work for you, then you need to raise it to MySQL team not to airflow, I am afraid.

Converting it into discussion.

[potiuk]

Please let me know what kind of customisations/settings led to this, I would be courious to know what your investigation brings.

[omgMath]

Cool, that worked! Thank you so much!

I guess we don't have much of customization, we are basically using the plain Docker image with the following ENV vars set:

LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libstdc++.so.6"
LOAD_EX="n"
EXECUTOR="Local"
FERNET_KEY="foo"
AIRFLOW__CORE__SQLALCHEMY_DATABASE_URI="mysql+mysqldb://..."
AIRFLOW__CORE__DEFAULT_TIMEZONE="Europe/Berlin"
AIRFLOW__CORE__EXECUTOR="LocalExecutor"
AIRFLOW__CORE__STORE_DAG_CODE="True"
AIRFLOW__CORE__STORE_SERIALIZED_DAGS="True"
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN="mysql+mysqldb://..."
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER="gs://..."
AIRFLOW__LOGGING__REMOTE_LOGGING="True"
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID="google_cloud_default"

Additionally for the scheduler:

AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG=3
AIRFLOW__SCHEDULER__PARSING_PROCESSES=5
AIRFLOW__SCHEDULER__PROCESSOR_POLL_INTERVAL=60
AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT="False"
AIRFLOW_VAR_RUN_TASKS_ON_GCE_VMS="True"
AIRFLOW_VAR_DOCKER_SKIP_PULL="False"

Additionally for the webserver:

AIRFLOW__WEBSERVER__BASE_URL="https://foo.bar"
AIRFLOW__API__AUTH_BACKENDS="airflow.api.auth.backend.basic_auth"
AIRFLOW__CORE__PARALLELIS=1
AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX="True"
AIRFLOW__WEBSERVER__EXPOSE_CONFIG="True"
AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE="Europe/Berlin"
AIRFLOW__WEBSERVER__DAG_DEFAULT_VIEW="graph"
AIRFLOW__WEBSERVER__RBAC="True"
AIRFLOW__WEBSERVER__WORKERS=1
AIRFLOW__WEBSERVER__WORKER_REFRESH_INTERVAL=300
AIRFLOW__WEBSERVER__WEB_SERVER_WORKER_TIMEOUT=300