[Bug] static_train --full fails on mlflow ≥ 3.0 with MlflowException: filesystem tracking backend is in maintenance mode

[jxie8301-pixel]

[Bug] static_train --full fails on mlflow ≥ 3.0 with MlflowException: filesystem tracking backend is in maintenance mode

Summary

On a clean install with mlflow >= 3.0 (reproduced on 3.13.0),
python -m quantpits.scripts.static_train --full aborts during the very
first R.start(experiment_name=...) call with:

mlflow.exceptions.MlflowException: The filesystem tracking backend
(e.g., './mlruns') is in maintenance mode and will not receive further
updates. Please migrate to a database backend (e.g., 'sqlite:///mlflow.db')
... If the filesystem backend is required for your workflow, set
`MLFLOW_ALLOW_FILE_STORE=true` to opt out of this exception.

The engine hard-codes MLFLOW_TRACKING_URI=file://<workspace>/mlruns
in quantpits/utils/env.py:34-37, never sets
MLFLOW_ALLOW_FILE_STORE, and exposes no backend override. This
makes QuantPits unusable out of the box for any new user pulling
mlflow >= 3.0 (which is what pip install -U mlflow or any
modern dependency resolver will land on).

Environment

Item	Value
QuantPits version	v0.4.3-alpha (git rev-parse HEAD → 2572bc8)
Python	3.10.20 (conda env quantpits)
Qlib	0.9.7 (pulled in by pyqlib)
MLflow	3.13.0 ← root cause
OS	WSL2 / Ubuntu (reproducer is OS-agnostic)
Workspace	workspaces/my_workspace/ (forked from Demo_Workspace/)
Qlib data	~/.qlib/qlib_data/cn_data (resolved via cn_data -> qlib_bin symlink, 835 MB, 61 058 files)
mlruns/ state	0-byte .gitkeep only — no prior experiment, no run history to migrate

Reproduction

Minimal command sequence (assumes conda activate quantpits and
pip install -r requirements.txt already run; the data download is
orthogonal to this bug — the failure happens before any data is read):

cd ~/work/QuantPits
git checkout 2572bc8                       # v0.4.3-alpha
pip install -U mlflow                      # lands on 3.13.0
source workspaces/Demo_Workspace/run_env.sh
python -m quantpits.scripts.static_train --full

Observed output (excerpt)

=== Config Loaded via config_loader ===
market: csi300
benchmark: SH000300
topk: 20
n_drop: 3
...
test_end_time: 2026-06-08
anchor_date: 2026-06-08
freq: week

======================================================================
  全量训练模型列表 (1 个模型)
======================================================================
  demo_linear_Alpha158  linear  Alpha158  csi300  baseline
...

>>> Processing Model: demo_linear_Alpha158 from config/workflow_config_demo_weekly.yaml
[MainThread] WARNING qlib.workflow - No valid experiment found. Create a new experiment with name Prod_Train_WEEK.
!!! Error running demo_linear_Alpha158: ...
mlflow.exceptions.MlflowException: The filesystem tracking backend (e.g., './mlruns') is in maintenance mode ...
ValueError: No valid experiment has been found, please make sure the input experiment name is correct.
❌ 模型 demo_linear_Alpha158 训练失败: <MlflowException above>

Full traceback includes
qlib.workflow.expm.MLflowExpManager.client
→ mlflow.tracking.MlflowClient(tracking_uri=self.uri)
→ mlflow.store.tracking.file_store.FileStore.__init__
→ raise MlflowException(InvalidParameterValue).

Impact

• Severity: Blocker for new users. Any user pulling
  mlflow >= 3.0 (the current PyPI default) cannot run the daily
  pipeline (static_train / ensemble_fusion / prod_post_trade /
  order_gen / deep-analysis / rolling). First-run experience is
  completely broken.
• Affected entry points: every script that eventually calls
  from qlib.workflow import R then R.start(experiment_name=...)
  or R.get_recorder(...). Concretely:
  • quantpits/scripts/static_train.py (train_single_model,
    predict_single_model)
  • quantpits/utils/train_utils.py:725, 897, 1441, 1470
  • All quantpits/scripts/rolling/* scripts
  • quantpits/scripts/analyze_ensembles.py,
    brute_force_ensemble.py, brute_force_fast.py,
    prod_post_trade.py (via experiment lookups)
  • ui/dashboard.py, ui/rolling_dashboard.py (read-side)
• Affected tests: tests/conftest.py:5-30 defines a
  prevent_mlruns autouse fixture that redirects
  MLflowExpManager.__init__ to file://<tmp>/mock_mlruns. The
  fixture only rewrites the URI — it does not set
  MLFLOW_ALLOW_FILE_STORE. CI on mlflow >= 3.0 will hit the same
  hard block.

Root cause

mlflow 3.0 introduced the MLFLOW_ALLOW_FILE_STORE gate in
mlflow/store/tracking/file_store.py::FileStore.__init__ to nudge
users off the file backend toward SQLite. The check is unconditional
on first construction:

if not os.environ.get("MLFLOW_ALLOW_FILE_STORE", "").lower() == "true":
    raise MlflowException(
        "The filesystem tracking backend (e.g., './mlruns') is in "
        "maintenance mode ..."
    )

QuantPits sets the backend in quantpits/utils/env.py:34-37 at
module import time:

mlruns_dir = os.path.abspath(os.path.join(ROOT_DIR, 'mlruns'))
os.environ["MLFLOW_TRACKING_URI"] = f"file://{mlruns_dir}"

and never sets MLFLOW_ALLOW_FILE_STORE. set_root_dir()
(env.py:90-105) re-writes the same env var the same way. No code
path in quantpits/ calls mlflow.set_tracking_uri(...) or
imports mlflow directly — backend choice is effectively
hard-coded to the deprecated file:// scheme.

Why the obvious workaround fails (and is not a real fix)

MLFLOW_ALLOW_FILE_STORE=true unblocks training today, but:

1. Env var must be set before any script runs. The natural place
   is workspaces/<my>/run_env.sh, and that works for now.
2. It does not fix the architectural problem. env.py keeps
   hard-coding file://. The next mlflow minor that makes the
   gate stricter (or mlflow 4.x removing FileStore outright per
   the deprecation roadmap)
   will break the same code path again.
3. The long-term fix — migrating to SQLite — is not reachable from
   the workspace layer. Setting MLFLOW_TRACKING_URI=sqlite:///...
   in run_env.sh is silently overwritten by env.py on the next
   import quantpits.utils.env (the os.environ[...] = ... line
   at env.py:36 runs unconditionally at module import). The only
   way to land a sqlite backend today is to change the engine.

This is why the fix belongs in the engine, not in user workarounds.

Suggested fix(es)

Pick one — listed in increasing order of cleanliness:

Option 1 (minimal): make the engine opt out automatically

In quantpits/utils/env.py, alongside the MLFLOW_TRACKING_URI
assignment, detect the mlflow version and set the opt-out
unconditionally when the chosen backend is file://:

try:
    import mlflow as _mlflow
    _mlflow_v = tuple(int(x) for x in _mlflow.__version__.split(".")[:2])
    if _mlflow_v >= (3, 0):
        os.environ.setdefault("MLFLOW_ALLOW_FILE_STORE", "true")
except Exception:
    pass

Also mirror this in set_root_dir() for the
set_root_dir-during-runtime path. Add a matching fixture line in
tests/conftest.py::prevent_mlruns.

Pros: zero behavior change on mlflow 2.x; one-line on mlflow 3.x;
workspace-level tests pass; no user action required.
Cons: still uses the deprecated backend.

Option 2 (recommended): make the backend configurable

1. Read tracking URI from environment first, then from
   config/prod_config.json (or a new config/mlflow_config.json),
   then fall back to the current file://<workspace>/mlruns.
2. Stop unconditionally overwriting MLFLOW_TRACKING_URI in
   env.py — guard it with os.environ.setdefault(...) so a
   user-supplied value wins.
3. Same guard in set_root_dir() and in the PlayGround manager.
4. Add MLFLOW_ALLOW_FILE_STORE=auto semantics: if backend is
   file:// and mlflow is ≥ 3.0, set it to true automatically.
5. Update tests/conftest.py to either use sqlite (preferred — see
   option 3) or to set the opt-out env var.

Option 3 (best): ship a sqlite backend by default

Change the default backend to sqlite:///<workspace>/mlflow.db,
keep file:// as a fallback for users who explicitly opt in, and
ship a one-shot migration helper:

python -m quantpits.tools.migrate_mlflow_backend \
    --from "$QLIB_WORKSPACE_DIR/mlruns" \
    --to   "sqlite:///$QLIB_WORKSPACE_DIR/mlflow.db"

(Internally wraps python -m mlflow migrate-filestore.)

mlflow migrate-filestore is already shipped in mlflow 3.13.0 and
is one-way only (no db2fs reverse). Document that in
docs/70_WALKTHROUGH.md and docs/en/70_WALKTHROUGH.md.

Workaround (until engine is fixed)

For users who can't wait for a release:

# Append to workspaces/<my_workspace>/run_env.sh (Demo_Workspace is read-only)
export MLFLOW_ALLOW_FILE_STORE=true

# Then, as before:
source workspaces/my_workspace/run_env.sh
python -m quantpits.scripts.static_train --full

The set_root_dir() path in env.py is also impacted; if you
use the Playground/feedback loop, the env var must be set in the
outer shell before any sub-script runs (it inherits from the
parent process environment, so export in run_env.sh covers it).

Evidence (file:line)

• quantpits/utils/env.py:34-37 — unconditional
  os.environ['MLFLOW_TRACKING_URI'] = f'file://{mlruns_dir}'
• quantpits/utils/env.py:90-105 — set_root_dir re-writes the
  same env var the same way
• quantpits/utils/train_utils.py:725 — R.start(experiment_name=...)
  inside train_single_model
• quantpits/utils/train_utils.py:1441 — R.get_recorder(...)
  inside predict_single_model (also affected on load side)
• quantpits/scripts/static_train.py:55, 159, 266, 428 —
  experiment-name construction only; no backend override
• tests/conftest.py:5-30 — prevent_mlruns fixture; rewrites
  URI but doesn't set opt-out → CI breakage on mlflow ≥ 3.0
• tests/quantpits/utils/test_env.py:40, 110 — asserts
  'mlruns' in os.environ['MLFLOW_TRACKING_URI']; will need
  updating if backend becomes configurable

Notes for maintainers

• The git tag for the failing version is v0.4.3-alpha
  (git rev-parse HEAD → 2572bc8).
• This bug is silent on the developer's machine if their conda
  env happened to resolve mlflow<3 (which pyqlib's transitive
  pins may or may not enforce depending on resolver). It only
  surfaces when a user (or CI) resolves a fresh mlflow>=3.
• A fix should be backported to the next patch release
  (v0.4.4-alpha or whatever supersedes v0.4.3-alpha).
• Worth a CHANGELOG.md entry under ## [Unreleased] →
  ### Fixed with a one-liner referencing this issue.

[DarkLink]

Thanks for reporting this.

We have updated the default MLflow backend to SQLite (sqlite:///mlflow.db) for newly created workspaces to avoid the filesystem backend maintenance mode error in MLflow 3.0+.

• For existing workspaces containing legacy mlruns/ data, the system will automatically fall back to the file:// backend and set MLFLOW_ALLOW_FILE_STORE=true to maintain backward compatibility.
• We have also introduced a new migration tool python -m quantpits.tools.migrate_mlflow_backend --workspace to help migrate legacy runs to SQLite when you are ready.