Skip to content

fix(analysis): handle crash-first baselines and empty keep sets#469

Open
afurm wants to merge 1 commit into
karpathy:masterfrom
afurm:af/fix-analysis-baseline
Open

fix(analysis): handle crash-first baselines and empty keep sets#469
afurm wants to merge 1 commit into
karpathy:masterfrom
afurm:af/fix-analysis-baseline

Conversation

@afurm
Copy link
Copy Markdown

@afurm afurm commented Apr 2, 2026

Summary

  • make the analysis notebook use the first non-crash run as the baseline consistently
  • raise a clear error when results.tsv has no non-crash runs to analyze
  • handle cases with no KEEP rows yet so the notebook does not fail on summary/plot generation

Why

The notebook previously used two different baseline definitions:

  • plot cell: first non-crash run
  • summary cell: first row in results.tsv

That means if the first logged experiment was a crash, the plot and summary could disagree about the baseline. The summary cell could also fail outright when there were no kept runs yet because it assumed idxmin() was always valid.

What changed

  • plot cell now guards valid.empty before reading the baseline
  • plot y-axis bounds now handle the no-kept-runs case safely
  • summary cell now derives the baseline from the first non-crash run
  • summary cell now prints n/a values instead of failing when there are no kept runs yet

Scope

This is analysis-only. No training code or experiment behavior changed.

Validation

  • verified analysis.ipynb is still valid JSON
  • did not run the notebook end-to-end in this environment

@barsharajyadav-boop

This comment was marked as low quality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants