Skip to content

feat: add binary classification and proper-scoring alert metrics - #582

Draft
knutdrand wants to merge 1 commit into
feat/outbreak-model-abstractionfrom
feat/outbreak-alert-metrics
Draft

knutdrand wants to merge 1 commit into
feat/outbreak-model-abstractionfrom
feat/outbreak-alert-metrics

Conversation

@knutdrand

Copy link
Copy Markdown
Contributor

Phase 2 of the alerting track. Stacked on #581#580#539. Last one in the stack; it retargets down as each merges.

The metrics

Scoring the alert decision:

Metric Objective? Why
outbreak_precision no maximised by alerting once, and only when certain
false_alarm_rate no maximised by never alerting
outbreak_f1 maximize harmonic mean of precision and sensitivity — prices both errors
outbreak_mcc maximize uses all four confusion cells, so true negatives count

Scoring the probability behind it:

Metric Objective?
outbreak_brier minimize mean squared error of the alert probability
outbreak_log_score minimize punishes confident misses hard; clipped at 1e-6 so one costs a large finite amount
outbreak_brier_skill maximize the number to lead with

Brier skill is the headline because outbreaks are rare: a raw score flatters any model that rarely alerts, while skill prices it against the climatological base rate instead of against nothing.

optimization_direction is set only where both error types are priced. Precision and false-alarm rate join sensitivity and specificity in leaving it unset, for the same reason #580 unset those.

A correction to the plan: GlobalOnlyMetric

The design note claimed this set would fit the existing compute_detailed → mean/sum/RMS contract, and that only AUROC would not. That was wrong. F1, Matthews correlation and Brier skill are all ratios of aggregates — there is no per-cell value whose mean recovers them, exactly the property that rules AUROC out.

Deferring all three would have left Phase 2 as precision, false-alarm rate, Brier and log score: the metrics nobody should lead with, minus every one that prices both errors. So this adds GlobalOnlyMetric to metrics/base.py — ~35 lines that report one value and raise when asked for a breakdown, rather than returning a number that looks per-cell and is not. AUROC now needs no new machinery whenever we want it.

One product gap this surfaced

test_generate_metric_visualization parametrizes over every metric the plot picker offers, and three failed: every metric plot breaks a score down by horizon, location or period, which a global-only metric has no values for. That was not a test artifact — the picker was offering metrics that render an error instead of a chart.

GET /metric-plots's metric list now filters to metrics with a per-cell value, with a regression test. Global-only scores are read from the backtest's aggregate metrics instead.

Testing

  • make lint (ruff, mypy, pyright) passes
  • uv run pytest -q --ignore=.claude: 1526 passed, 122 skipped, 4 xfailed, 1 xpassed
  • 16 new tests. The balanced_alert_scenario fixture yields exactly one TP, FP, TN and FN, so every confusion-matrix metric has a hand-checkable value — precision 0.5, F1 0.5, MCC 0.0 (chance agreement), Brier 0.5, Brier skill −1.0 (worse than climatology). Plus the log-score clip, both global-only refusals, the undefined-skill case, the objective lock-in, and the picker filter.

Still deferred

AUROC and AUPRC (machinery now exists), lead time, and skill against the persistence baseline from #581 — that last one needs last_seen_period plumbed through the flat forecast representation, the same prerequisite as the leakage cutoff, so both belong to that change.

Seven metrics scoring the alert decision and the probability behind it:
precision, false alarm rate, F1 and Matthews correlation on the decision;
Brier, log score and Brier skill on the probability. Brier skill against
the climatological base rate is the number worth leading with, since a
raw score flatters any model that rarely alerts when outbreaks are rare.

Three of them are ratios of aggregates with no per-cell value, so they
needed a base class the metric system did not have. GlobalOnlyMetric
reports a single value and raises when asked for a breakdown, rather than
returning a number that looks per-cell and is not. The plan said this set
would fit the existing contract; that was wrong for F1, Matthews and
skill, and deferring all three would have left only the metrics nobody
should lead with. AUROC now needs no new machinery when we want it.

optimization_direction is set only where both error types are priced.
Precision and false alarm rate join sensitivity and specificity in
leaving it unset: precision is maximised by alerting once and only when
certain, false alarm rate by never alerting at all.

The plot-metric picker now lists only metrics with a per-cell value.
Every metric plot breaks a score down by horizon, location or period, so
offering a global-only metric rendered an error instead of a chart.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant