DecisionFlow: Fine-grained prompting, bug fixes and QOL updates #257
Conversation
dmjoy
left a comment
dmjoy
left a comment
There was a problem hiding this comment.
Just a cursory look here, but I think this all makes sense and I think moving these prompts into a dedicated set of files is good. Only a few comments I had for my own understanding
| variables: | ||
| num_samples: 1 | ||
| system_prompt_template: | ||
| _target_: align_system.prompt_engineering.outlines_prompts.DefaultITMBaselineSystemPrompt |
There was a problem hiding this comment.
Not sure if this is meant to be a placeholder, but I'm confused as to why each of the system_prompt_templates here are using this?
There was a problem hiding this comment.
Great catch!
There are a couple stages which use the same medical assistant baseline prompt as a part of the overall stage prompt.
Not all stages need this. I'll work on fixing this prior to merging this PR.
| try: | ||
| prompt_content = fine_grained_math_reason_prompt( | ||
| objective=objective, | ||
| attribute=attribute, | ||
| variable=variable, | ||
| constraints=constraints, | ||
| choice=choice, | ||
| target_attributes_values=target_attributes_values | ||
| ) | ||
| except Exception: | ||
| # For error fallback, we need to reconstruct structure format | ||
| structure = { | ||
| "Objective Function": objective, | ||
| "Decision Variables": variable, | ||
| "Constraints": constraints | ||
| } | ||
| prompt_content = fine_grained_math_reason_error_prompt( | ||
| structure=structure, | ||
| choice=choice, | ||
| target_attributes_values=target_attributes_values | ||
| ) |
There was a problem hiding this comment.
Just a little confused here why we need a fallback prompt?
There was a problem hiding this comment.
So, this is how the original DecisionFlow is structured. If the math_reason prompt cannot get the required dictionary fields (Objective Function, Decision Variables, Constraints), the math_reason_error prompt will construct a Json structure that contains those fields and then pull in the information for each of these fields.
There was a problem hiding this comment.
Ah gotcha, that makes some sense then. Thanks for clarifying
@dmjoy
This PR introduces support for fine-grained value prompts (specifically for phase-2 attributes), key bug-fixes and QOL updates to the DecisionFlow ADM.
Feature Updates
AttributeandMath Reasonstages for the DecisionFlow ADM. These two stages are the key parts in the ADM that directly include the alignment target for analysis.fine_grained_promptsthat include a generic numeric target scale for theMathReasonprompt. On the other hand, for theAttributestage, the prompt added is contextualized specifically to Phase-2 attributes with proper value scale anchoring, attribute context and in-context example scenarios to go with the prompt.Bug Fixes
Decision VariablesandConstraintsfrom theExpressstage not being propagated properly to theMathReasonstage.MathReasonstage got injected with the target alignment attribute instead of theAttributeanalysis attribute-value pairs. For example:QOL updates
outlines_prompts.pyinto a separate file underalign_system/prompt_engineering/decision_flow/high_low_prompts.pyMathReasonstage to help with figuring out inter-stage data flow anomalies.