airflow variables list takes too long to run

[nogkaha]

Apache Airflow version

2.3.2

What happened

Hi, I got into an issue after upgrading from 2.1.2 to 2.3.2

• first I noticed that the UI in the webserver failed to show the list of variables, instead it hanged for a while and eventually crushed with .. err.
• I run the equivalent from the CLI : airflow variables list and it took 4m to complete
• Querying the variables directly from the DB takes less than 1s
• Getting the value of a single variable also returns immediately
• I have 301 variables which results to a file of size 3.4M when exported.
• All the variables are encrypted and I was suspecting that the decryption takes too long. I tested it by invalidating the fernet key, and indeed this time both the UI and the CLI returns the list immediately (although without the ability to access the encrypted values)

What you think should happen instead

Since I didn't change the amount of variables nor did I experience this issue before the update, I would expect I would expect that the list of variables would be presented in a timely manner also after the update.

How to reproduce

As mentioned, the issue started after upgrading. In order to upgrade I basically use apache/airflow:2.3.2-python3.9 as a base image. Later when I realised that the update require also a DB migration I run the migration command as described in the docs. It run for a while, and eventually I got an error, about not being able to release a lock from the DB. However, the webserver was working so I ignored it.

Operating System

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==3.4.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==4.0.2
apache-airflow-providers-docker==2.7.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==7.0.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.9.0
apache-airflow-providers-microsoft-mssql==3.0.0
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-samba==4.0.0
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.4

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

I believe your database is broken. You should upgrade your DB without airflow running. The description of your problem indicates that you broke the database. by running upgrade with airflow running and some deadlocks laading to unfinished migration. Now likely your CLI commands (and possibly more than that) are broken.

Restore your DB from backup before migration and perform "proper" migration without airflow running.

[nogkaha]

Thanks for the help, I did as suggested :)