fix(am): record every experiment's trajectory, normalized once; refuse what would be wrong (#990)

[wshlavacek]

Closes #990. Closes #998. These are the am trajectory-output defects found while reviewing #988.

What changes

1. Every experiment gets its trajectory files, including conditioned experiments (edition 2) and mutants (edition 1).
   • Buffers are keyed by each experiment's real output suffix (epi2hi), from model.get_suffixes(), instead of by the action suffix.
   • Under edition 2, action/condition pairs that no experiment: line names are left out.
   • Buffers are created at the length of the first result that supplies them, so time_length no longer decides it.
   • Matching stays exact (_keys_of, from fix(am): draw output_noise_trajectory from each column's own noise model (#986) #988), so epi cannot catch epi2.
   • A run resumed from an older backup works out its experiments from the saved config.
2. Normalization no longer crashes, and is applied once. The data reaching am is already normalized by the scoring path, so output_trajectory now records exactly what the objective scores, without the second normalization that crashed. Two combinations are refused at load:
   • output_noise_trajectory on a normalized column, because a count drawn around a normalized prediction has no meaning;
   • either output on a column normalized with scale, because that factor is applied inside the objective and is not in the simulated series.
3. normalization = scale on a noise family with no additive scale is refused at load (Configuration._require_additive_scale). The check reads the family's additive_on, not its class name. Of gaussian/normal, lognormal, lnnormal, laplace, student_t and neg_bin, only neg_bin lacks one.
   • The 'FailedSimulation' object has no attribute 'out' crash is fixed. A genuine failure can reach that line, because a chain's first result is always accepted.
   • That change also fixes local_objective_eval = 1 and smoothing > 1, which crashed every am run that wrote trajectories ('Result' has no attribute 'out').
4. A misspelled output_trajectory / output_noise_trajectory name is an error at load, naming the unmatched name and the observables, functions and species it could have matched.
   • The old first-result warning is now an error, for names the load check can't settle (for example under postprocess).
   • Two experiments whose trajectories would land in one file are refused, as is an experiment whose output length changes between results.
5. An empty ChainRecord survives a pickle round trip. mh and pt can back up a slow chain before its first result. -r then raised IndexError at that chain's first result. am cannot reach this, because of its generation barrier.

Behaviour changes worth knowing

• Every job type other than am now refuses output_trajectory and output_noise_trajectory at load. They were silently ignored before.
  • The six dream/p_dream configs in examples/sampler_benchmarking set them; those lines are removed.
• The new name check caught a real typo in examples/HIVdynamics_aMCMC/pt409/pt409.conf: lnVV is now lnV.

Sibling paths

Path	Status
am, worker scoring	changed; end-to-end tested
am, master scoring (local_objective_eval = 1)	changed; crashed before; end-to-end tested
am with smoothing > 1 / parallelize_models > 1	same change; unit-tested only
edition-2 conditions, edition-1 mutants	changed; end-to-end tested
pre-equilibration experiments	same rule; not tested
bngsim BNGL ODE	end-to-end tested
BNG2.pl, RoadRunner, bngsim SBML/Antimony, network-free	same path through get_suffixes; not run
resume (-r)	old-backup tests
mh, pt, dream, p_dream, de, check	now refuse the keys; tested
shared ChainRecord	fixed for all samplers

Tests and oracles

Of the 36 new or rewritten tests, 31 fail on main. The other 5 cover noise families that were already accepted.

• Closed-form oracles:
  • (exp(-k t) + 0.1)/1.1 under floor 0.1, init, edition 2, with a conditioned experiment;
  • CumInf/max and 0.6*CumInf for an edition-1 mutant.
• Refusal tests each assert that the error names its cause.

Files run, one at a time, on the bngsim build from bngsim main:

• test_adaptive_mcmc 115
• test_basic_mcmc_class + test_bayesian_diagnostics 86
• test_config_class 112
• test_config_unused_keys 187
• test_edition 75
• test_lwf_config 82
• test_noise_model_config 67
• test_hmc 52
• test_linear_profiling 79
• test_dream_class 23
• test_seed_determinism 23
• test_sampler_integration 13
• and about a dozen smaller files

All passed. ruff and tools/changelog.py check are clean.

Not covered here (to be filed)

• Broad excepts turn configuration errors into failed simulations. This is how the scale AttributeError became penalties: pybnf/algorithms/core.py:407 and pybnf/algorithms/base.py:948-953.
• time_error ignores scale and cumulative. Found by reading; not reproduced.
• scale is not refused for Bayesian samplers, although noise_profiling and linear_profiling are.
• am writes a never-simulated all-zero trajectory as the first row of each chain's trajectory file and, under the default and shipped settings, shifts every later row one place against the parameter rows #859 is still open: the first trajectory row is all zero.

Review

Two independent review rounds, then a short check of the final commit:

• First round: found four problems, now fixed in be7edb1:
  • archived sampler runs read back in natural space through from_pybnf's broad except;
  • a legacy per-file normalization dict was falsely refused;
  • a failed first result on the master path was misreported;
  • one inaccurate doc sentence.
• Deleting find_t_length (find_t_length refuses valid BNGL simulate actions for every job type (n_output_steps, no step count), misreads n_steps+sample_times and skips commented lines, to compute a length only am reads #998) exposed an older problem. The bngsim bridge ignored n_output_steps and defaulted to 100 output steps, where BNG2.pl defaults to 1. With n_output_steps=>20, BNG2.pl writes 21 rows and the bridge wrote 101.
  • ba37953 makes the bridge follow BNG2.pl's output-grid rules (pybnf/bngsim_model/grid.py).
  • 8319864 hands actions with options the bridge does not read (argfile, max_sim_steps, output_step_interval, print_on_stop=>0) to BNG2.pl under auto, and refuses them by name under bngsim.
• Second round and final check:
  • F1–F4 are fixed.
  • The Results trees are byte-identical to 3914cc1's across nine job shapes (edition 1 and 2, conditions, scans, pre-equilibration, steady state, SBML, _Cum).
  • No output grid changes, and no model moves off the bridge, across 498 simulate actions in 573 tracked models, the BNGL library and BioNetGen's Models2.
  • The reviewer's tests are committed (tests/test_am_trajectory_review_990.py, tests/test_bngsim_output_grid_review.py).
• Remaining sample_times difference. The bridge integrates from the first sample time and ignores an earlier t_end. That is On the default bngsim backend, a BNGL simulate() whose sample_times start after t_start is integrated from the first listed time, so the initial state is reported there and every scored point is shifted #872, which predates this PR. It is stated in the docs and pinned by a strict xfail.