Max Recursion Depth Exceeded when running db upgrade from 1.5.3 to 4.1.1

[alexburgoon]

Bug description

When upgrading on Kubernetes from version 1.5.3 to 4.1.1, I run the superset db upgrade command in an init job and receive a ERROR [flask_migrate] Error: maximum recursion depth exceeded while calling a Python object during migration a9422eeaae74, new_dataset_models_take_2. This appears to happen when the number of datasets stored in the metadata database is around 200. It worked with just a few datasets.

Reproduction Steps:

1. Deploy Superset version 1.5.3 on Kubernetes
2. Import 200 datasets
3. Deploy upgraded Docker image version 4.1.1
4. Run superset db upgrade

Expected Result: Database migrations run successfully.

Screenshots/recordings

No response

Superset version

4.1.1

Python version

3.11

Node version

Not applicable

Browser

Not applicable

Additional context

Logs from init script:

[notice] A new release of pip is available: 25.0 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
ERROR: You must give at least one requirement to install (see "pip help install")
Upgrading DB schema...
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
2025-02-21 11:39:15,227:INFO:superset.initialization:Setting database isolation level to READ COMMITTED
/usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
INFO  [alembic.env] Starting the migration scripts.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 58df9d617f14 -> 2ed890b36b94, rm_time_range_endpoints_from_qc
INFO  [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> b0d0249074e4, deprecate time_range_endpoints v2
INFO  [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> 8b841273bec3, sql_lab_models_database_constraint_updates
INFO  [alembic.runtime.migration] Running upgrade 8b841273bec3, b0d0249074e4 -> 9d8a8d575284, merge point
INFO  [alembic.runtime.migration] Running upgrade 9d8a8d575284 -> cecc6bf46990, rm_time_range_endpoints_2
INFO  [alembic.runtime.migration] Running upgrade cecc6bf46990 -> ad07e4fdbaba, rm_time_range_endpoints_from_qc_3
INFO  [alembic.runtime.migration] Running upgrade ad07e4fdbaba -> a9422eeaae74, new_dataset_models_take_2
ERROR [flask_migrate] Error: maximum recursion depth exceeded while calling a Python object
slices updated with no time_range_endpoints: 0
>> Copy 10 physical tables to sl_tables...
>> Copy 202 SqlaTable to sl_datasets...
   Copy dataset owners...
   Link physical datasets with tables...
>> Copy 2,491 table columns to sl_columns...
   Link all columns to sl_datasets...
>> Copy 207 metrics to sl_columns...
   Link metric columns to datasets...
>> Run postprocessing on 2,698 columns
   [Column 1 to 2,698] 2,493 may be updated
   Assign table column relations...
>> Run postprocessing on 202 datasets
   Process dataset 1~202...                              
Initializing roles...
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
2025-02-21 11:39:25,436:INFO:superset.initialization:Setting database isolation level to READ COMMITTED
/usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
2025-02-21 11:39:29,676:INFO:superset.security.manager:Syncing role definition
2025-02-21 11:39:29,811:INFO:superset.security.manager:Syncing Admin perms
2025-02-21 11:39:29,821:INFO:superset.security.manager:Syncing Alpha perms
2025-02-21 11:39:29,841:INFO:superset.security.manager:Syncing Gamma perms
2025-02-21 11:39:29,878:INFO:superset.security.manager:Syncing sql_lab perms
2025-02-21 11:39:29,928:INFO:superset.security.manager:Fetching a set of all perms to lookup which ones are missing
2025-02-21 11:39:29,953:INFO:superset.security.manager:Creating missing datasource permissions.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedColumn: column tables.catalog_perm does not exist
LINE 1: ...s_perm, tables.schema_perm AS tables_schema_perm, tables.cat...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/superset/utils/decorators.py", line 259, in wrapped
    result = func(*args, **kwargs)
  File "/app/superset/cli/main.py", line 68, in init
    security_manager.sync_role_definitions()
  File "/app/superset/security/manager.py", line 1070, in sync_role_definitions
    self.create_missing_perms()
  File "/app/superset/security/manager.py", line 1018, in create_missing_perms
    datasources = SqlaTable.get_all_datasources()
  File "/app/superset/connectors/sqla/models.py", line 1955, in get_all_datasources
    return qry.all()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2773, in all
    return self._iter().all()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
    result = self.session.execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column tables.catalog_perm does not exist
LINE 1: ...s_perm, tables.schema_perm AS tables_schema_perm, tables.cat...
                                                             ^

[SQL: SELECT tables.uuid AS tables_uuid, tables.created_on AS tables_created_on, tables.changed_on AS tables_changed_on, tables.id AS tables_id, tables.description AS tables_description, tables.default_endpoint AS tables_default_endpoint, tables.is_featured AS tables_is_featured, tables.filter_select_enabled AS tables_filter_select_enabled, tables."offset" AS tables_offset, tables.cache_timeout AS tables_cache_timeout, tables.params AS tables_params, tables.perm AS tables_perm, tables.schema_perm AS tables_schema_perm, tables.catalog_perm AS tables_catalog_perm, tables.is_managed_externally AS tables_is_managed_externally, tables.external_url AS tables_external_url, tables.table_name AS tables_table_name, tables.main_dttm_col AS tables_main_dttm_col, tables.database_id AS tables_database_id, tables.fetch_values_predicate AS tables_fetch_values_predicate, tables.schema AS tables_schema, tables.catalog AS tables_catalog, tables.sql AS tables_sql, tables.is_sqllab_view AS tables_is_sqllab_view, tables.template_params AS tables_template_params, tables.extra AS tables_extra, tables.normalize_columns AS tables_normalize_columns, tables.always_filter_main_dttm AS tables_always_filter_main_dttm, tables.created_by_fk AS tables_created_by_fk, tables.changed_by_fk AS tables_changed_by_fk 
FROM tables 
WHERE tables.is_sqllab_view = false]
(Background on this error at: https://sqlalche.me/e/14/f405)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/superset", line 8, in <module>
    sys.exit(superset())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 358, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 358, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/app/superset/utils/decorators.py", line 266, in wrapped
    return on_error(ex)
  File "/app/superset/utils/decorators.py", line 234, in on_error
    raise reraise() from ex
sqlalchemy.exc.SQLAlchemyError: ()

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

Hey @alexburgoon! I'm here to help you with any bugs, questions, or contributions. Let me know how I can assist you!

A similar issue was reported in the past regarding a failure during database migration, specifically related to the new_dataset_models_take_2 migration step. The issue was titled "superset-init-db pod failed on upgrade" and has been closed. It was resolved by a pull request that addressed an error on PostgreSQL during the migration process ^[1].

Given the similarity, it might be helpful to review the changes made in that pull request to see if they can be applied or adapted to your situation. Additionally, ensure that your environment and dependencies are correctly set up, as discrepancies in configurations or versions might also contribute to such issues.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[sfirke]

As a starting point could you try migrating to 2.0.0, then 3.0.0, etc. ? Perhaps that would succeed and if not, perhaps it would narrow down the problem. Just an idea.