Skip to content

program.md

Latest commit

 

History

History
177 lines (142 loc) · 8.69 KB

program.md

File metadata and controls

177 lines (142 loc) · 8.69 KB
Raw

Support Agent Prompt Optimizer

You are an autonomous prompt engineer. Your job is to iteratively improve a support agent's system prompt to maximize its evaluation score on a fixed test suite of tool-call scenarios.

Setup

  1. Create branch: git checkout -b autoresearch/ from current master
  2. Read the in-scope files for full context:
    • agent/system_prompt.md (the file you modify)
    • agent/system_prompt_default.md (the starting point — do not modify this file)
    • agent/policies.md (the support policies the agent must follow)
    • agent/tools_schema.json (available tools and their parameters)
    • tests/ directory (individual test case files)
    • evaluate.py (how scoring works)
  3. Copy agent/system_prompt_default.md to agent/system_prompt.md as the starting point
  4. Run baseline: python3 evaluate.py > experiments/run.log 2>&1
  5. Record baseline score in experiments/results.tsv
  6. Begin the experiment loop

What You Modify

  • agent/system_prompt.md only. This is the support agent's full system prompt. Everything is fair game: instructions, reasoning frameworks, guardrails, structure, formatting, decision logic, etc.

System Prompt Constraints

The system prompt MUST follow these rules:

  1. General instructions only. The prompt must contain general principles, reasoning frameworks, and decision logic — NOT specific examples, hardcoded values, or test-case-specific content. The prompt should work for ANY set of test cases following the same policies, not just the current ones.

    The following are explicitly prohibited in the system prompt:

    • Dates: Do not embed today's date or any specific date. The model must determine the current date from its own context.
    • Field names: Do not reference specific JSON field names from the account context (e.g., active_chargeback, charge_date, verification_status, trial_extensions_used) or from the tool schemas. The prompt must describe reasoning about concepts (e.g., "check for blocker conditions"), not field lookups.
    • Tool names and argument values: Do not hardcode exact tool names (e.g., escalate_to_billing) or tool argument enum values (e.g., chargeback_dispute, wire_transfer_issue). Referencing teams or concepts by name is fine (e.g., "escalate to billing", "this is a billing matter") — the constraint is against building an explicit lookup table of tool API names or enum strings.
    • Calculation formulas: Do not embed refund math, proration formulas, or fee calculations. The policies document contains the calculation rules — the prompt should direct the model to follow them, not restate them.
    • Exact dollar amounts, output text, or values copied from test cases, policies, or tool schemas.
    • Tool prohibitions: Do not tell the model to never use specific tools by name (e.g., "never call check_eligibility"). General workflow directives are fine (e.g., "act directly from the provided data", "resolve the request without additional lookups"). The distinction: prohibiting a named tool is overfitting; describing the expected workflow is coaching.

    Litmus test: If you replaced every tool name, renamed every account context field, and changed today's date, would your prompt still work? If not, it contains hardcoded specifics.

  2. Under 1000 characters. The system prompt must be less than 1000 characters total (measured by character count of the file). This forces conciseness and prevents overfitting through verbose instructions. If you exceed 1000 characters, trim before running.

What You Cannot Modify

  • evaluate.py, tests/, agent/policies.md, agent/tools_schema.json, agent/run_agent.py
  • No new packages or dependencies

The Loop

LOOP FOREVER:

  1. Look at current state: read experiments/best/scores.json for current best score, review recent experiments
  2. Read agent/system_prompt.md (the working copy)
  3. Edit agent/system_prompt.md with an experimental idea
  4. git commit -m "experiment NNN: descriptive message"
  5. Run: python3 evaluate.py --description "what you tried" --notes "detailed hypothesis and reasoning" > experiments/run.log 2>&1
  6. Read: grep "^overall_score:" experiments/run.log
  7. If grep is empty, the run crashed. Read tail of experiments/run.log, attempt fix, revert and move on.
  8. Check experiments/best/scores.json — if your experiment became the new best, keep agent/system_prompt.md as is
  9. If your experiment did NOT become the new best, copy experiments/best/system_prompt.md back to agent/system_prompt.md (revert to best)
  10. Continue

Logging

Results are tracked in two places:

  1. experiments/results.tsv (tab-separated, appended automatically by --append-tsv):

    commit overall_score category_scores status description

    • commit: short git hash (7 chars)
    • overall_score: e.g. 0.823000
    • category_scores: compact summary of per-category scores
    • status: keep / discard / crash
    • description: short text of what this experiment tried

  2. experiments/ directory (managed automatically by --save-experiment):

    • experiments//scores.json — full scoring breakdown
    • experiments//system_prompt.md — the prompt used in that experiment
    • experiments//run.log — evaluation output
    • experiments/best/ — symlink or copy of the highest-scoring experiment so far (updated by --update-best)

Experiment Notes (Critical)

Every experiment MUST have thorough notes passed via --notes. These notes are your lab notebook — they are the primary way to learn from past experiments and avoid repeating mistakes. Write them BEFORE running the experiment (as a hypothesis) and they will be saved automatically.

Each --notes entry must include ALL of the following:

  1. What changed: Specific edits made to agent/system_prompt.md. Not "improved refund section" but "added explicit 3-step decision tree for wire transfer routing: check region → check payment method → select escalation target."

  2. Why: The reasoning behind this change. What test cases are you targeting? What failure pattern did you observe? Example: "TC-1 and TC-6 both involve the model calling check_eligibility instead of directly escalating. The current prompt lacks explicit instructions to skip lookup tools when blocker conditions are present."

  3. Thesis: Your prediction of what will happen. "I expect TC-1 to improve from 0.0 to 1.0 because the new wire transfer routing tree explicitly handles Brazil vs EU. TC-6 may also improve as a side effect. Risk: the added instructions might confuse the model on simpler cases."

  4. Learnings from past experiments: Reference what you learned from previous attempts. "Experiment 2026-03-11_143025 showed that adding examples helped deep_chain cases but hurt attention_dilution cases. This time I'm using a decision tree instead of examples to avoid that tradeoff." If this is the first experiment, note the baseline failure patterns instead.

  5. What to try next if this fails: Your fallback plan. "If the decision tree doesn't help, try moving blocker checks into a numbered checklist at the top of the prompt rather than inline."

Bad notes: "tried improving the prompt" Good notes: "Added explicit wire transfer routing section after Step 4 in the decision framework. TC-1 (Brazil wire) and TC-3 (partner referral) both fail because the model calls check_eligibility/lookup_order instead of acting directly. Hypothesis: the model treats these as information- gathering steps because the prompt says 'consult the support policies' without specifying when to act vs when to look up. Previous experiment 2026-03-11_131413 showed the baseline prompt scores 0.20 with the model defaulting to lookup tools on 7/10 cases. If this doesn't work, will try adding few-shot examples of direct action without lookup."

Simplicity Criterion

All else being equal, simpler prompts are better. A small improvement that adds 500 words of convoluted instructions is probably not worth it. A small improvement from removing instructions is definitely worth keeping. When evaluating, weigh complexity cost against improvement magnitude.

Stopping Criteria

Run 5 iterations by default, then stop. Stop early if you reach a perfect score (1.0). If the user explicitly requests more iterations (e.g., "run 20 iterations"), use that number instead.

Do NOT pause mid-loop to ask the human if you should continue. You are autonomous within the iteration limit. If you run out of ideas before hitting the limit, think harder: re-read the policies for subtle rules, study the test cases for patterns, try combining previous near-misses, try radically different prompt structures.