Conversation
Seven metrics scoring the alert decision and the probability behind it: precision, false alarm rate, F1 and Matthews correlation on the decision; Brier, log score and Brier skill on the probability. Brier skill against the climatological base rate is the number worth leading with, since a raw score flatters any model that rarely alerts when outbreaks are rare. Three of them are ratios of aggregates with no per-cell value, so they needed a base class the metric system did not have. GlobalOnlyMetric reports a single value and raises when asked for a breakdown, rather than returning a number that looks per-cell and is not. The plan said this set would fit the existing contract; that was wrong for F1, Matthews and skill, and deferring all three would have left only the metrics nobody should lead with. AUROC now needs no new machinery when we want it. optimization_direction is set only where both error types are priced. Precision and false alarm rate join sensitivity and specificity in leaving it unset: precision is maximised by alerting once and only when certain, false alarm rate by never alerting at all. The plot-metric picker now lists only metrics with a per-cell value. Every metric plot breaks a score down by horizon, location or period, so offering a global-only metric rendered an error instead of a chart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of the alerting track. Stacked on #581 → #580 → #539. Last one in the stack; it retargets down as each merges.
The metrics
Scoring the alert decision:
outbreak_precisionfalse_alarm_rateoutbreak_f1outbreak_mccScoring the probability behind it:
outbreak_brieroutbreak_log_scoreoutbreak_brier_skillBrier skill is the headline because outbreaks are rare: a raw score flatters any model that rarely alerts, while skill prices it against the climatological base rate instead of against nothing.
optimization_directionis set only where both error types are priced. Precision and false-alarm rate join sensitivity and specificity in leaving it unset, for the same reason #580 unset those.A correction to the plan:
GlobalOnlyMetricThe design note claimed this set would fit the existing
compute_detailed→ mean/sum/RMS contract, and that only AUROC would not. That was wrong. F1, Matthews correlation and Brier skill are all ratios of aggregates — there is no per-cell value whose mean recovers them, exactly the property that rules AUROC out.Deferring all three would have left Phase 2 as precision, false-alarm rate, Brier and log score: the metrics nobody should lead with, minus every one that prices both errors. So this adds
GlobalOnlyMetrictometrics/base.py— ~35 lines that report one value and raise when asked for a breakdown, rather than returning a number that looks per-cell and is not. AUROC now needs no new machinery whenever we want it.One product gap this surfaced
test_generate_metric_visualizationparametrizes over every metric the plot picker offers, and three failed: every metric plot breaks a score down by horizon, location or period, which a global-only metric has no values for. That was not a test artifact — the picker was offering metrics that render an error instead of a chart.GET /metric-plots's metric list now filters to metrics with a per-cell value, with a regression test. Global-only scores are read from the backtest's aggregate metrics instead.Testing
make lint(ruff, mypy, pyright) passesuv run pytest -q --ignore=.claude: 1526 passed, 122 skipped, 4 xfailed, 1 xpassedbalanced_alert_scenariofixture yields exactly one TP, FP, TN and FN, so every confusion-matrix metric has a hand-checkable value — precision 0.5, F1 0.5, MCC 0.0 (chance agreement), Brier 0.5, Brier skill −1.0 (worse than climatology). Plus the log-score clip, both global-only refusals, the undefined-skill case, the objective lock-in, and the picker filter.Still deferred
AUROC and AUPRC (machinery now exists), lead time, and skill against the persistence baseline from #581 — that last one needs
last_seen_periodplumbed through the flat forecast representation, the same prerequisite as the leakage cutoff, so both belong to that change.