Exp results #70
Open
Exp results #70
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update selectors in page object for new layout and fix timing issues with dropdown selections by waiting for listbox to close.
- Add add_probes_from_experiments method to ProbeRegistry - Wire up in core.py to populate probes from experiment items - Skip default probes when --experiments or --scenarios is provided
Extract unique decider configs from experiment results and add them to the decider dropdown. Configs are deduplicated by hashing the normalized adm section (with dataset paths stripped to filenames). Priority order: CLI --deciders > experiments > built-in deciders Extract experiment's configured LLM and put it first in the available backbones list, followed by default LLM options. Consolidate experiment config loading into experiment_config_loader.py.
Move experiment-to-registry conversion logic into experiment_converters.py. Core orchestrates calling these functions and populating registries.
- Move run_models.py from app/ to adm/ for better layering - Add run conversion functions to experiment_converters.py - Add lru_cache to load_experiment_adm_config for performance - Wire up populate_cache_bulk in core.py on startup
Keep resolved_config pure - only what the decider module needs to instantiate an ADM. GUI metadata (llm_backbones, max_alignment_attributes) now stays in decider entries and is accessed via get_decider_options(). Removed model_path_keys entirely (was redundant - injection path is hardcoded).
Editing the config and pressing Choose creates a new run with a new
"edited" decider (named "{original} - edit {n}"). The edited config
bypasses Hydra loading and uses the stored resolved config directly.
- Extract shared caching logic into _execute_with_cache() helper - Remove unused create_and_execute_run() method - Fix SearchController/RunsStateAdapter circular dependency with callback - Delete orphaned prompt_logic.py, move functions to consumers - Remove unused ui.py functions (serialize_prompt, prep_for_state, etc) - Convert RunsRegistry namedtuple to Protocol for better type hints - Rename find_probe_by_base_and_scene to find_probe_by_scenario_and_scene
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
related to #60