fix: point MLflow at a writable tracking store in the worker

[knutdrand]

Problem

Model runs in the worker container fail with:

mlflow.store.db.utils: SQLAlchemy engine could not be created. The following exception is caught.
sqlite3.OperationalError: unable to open database file

MLflow defaults its tracking store to the relative URI sqlite:///mlflow.db (mlflow/store/tracking/__init__.py, DEFAULT_TRACKING_URI) and resolves it against the process working directory. The worker has WORKDIR /app and runs with read_only: true, so the file cannot be created.

This is not a dependency drift. The sqlite default landed in mlflow-skinny 3.7.0 and is present in every version our range allows, including the 3.11.1 the lockfile pins. Downgrading does not help either: 3.6.0 defaults to ./mlruns, which resolves to the same read-only /app. The variable is the working directory, not the version.

The failure is not cosmetic. create_sqlalchemy_engine_with_retry retries 10 times with exponential backoff, roughly 100 seconds, and then raises.

Only models whose MLproject declares python_env are affected. helper_functions.py maps those to MlFlowTrainPredictRunner, which calls mlflow.projects.run and therefore creates a tracking run. Models using docker_env, uv_env, renv_env, or conda_env go through CommandLineTrainPredictRunner and never touch MLflow tracking. Reproduced with https://github.com/chap-models/minimal_template_example.

Fix

When MLFLOW_TRACKING_URI is unset, default the tracking store to a sqlite file under CHAP_RUNS_DIR, which is writable in every deployment (/data/runs in the worker, already chown chap:chap in Dockerfile.worker). A tracking URI configured by the deployment is left untouched.

This lives in code rather than in compose.yml so it also covers compose.ghcr.yml and the skaffold deployment. MLflow's own _make_parent_dirs_if_sqlite creates the parent directory, so the helper has no filesystem side effects, and set_tracking_uri mirrors the value into the environment so the project subprocess inherits it.

Tests

Two tests in tests/runners/test_runners.py: the fallback lands under CHAP_RUNS_DIR, and a deployment-configured URI wins. Added a mlflow_tracking_uri_reset fixture since set_tracking_uri mutates process-global state, and applied it to the two existing mlflow runner tests so they no longer leak into the rest of the session.

tests/runners/test_runners.py: 24 passed, 1 skipped.

Full make test: 798 passed, 1 failed. The failure is test_db_endpoints.py::test_make_prediction_with_data_source_nonexistent_id (NameError: name 'MakePredictionWithDataSourceRequest' is not defined), which fails identically on feat/hpo-rest-api with this branch's changes stashed. make lint reports 2 mypy errors in the same file, also pre-existing on the base.

Not covered

_get_sqlalchemy_store falls back to DEFAULT_LOCAL_FILE_AND_ARTIFACT_PATH for the artifact root, so artifacts still target /app/mlruns. minimal_template_example logs no artifacts so this is not hit today, but a model that does will fail the same way. The only override is the private _MLFLOW_SERVER_ARTIFACT_ROOT; the real fix is moving the worker's working_dir off /app, which has a wider blast radius and is left for a separate change.