feat: agent eval mode for loop exit conditions

[dean0x]

Summary

• Agent Eval Mode: Loops can now delegate exit condition evaluation to an AI agent instead of a shell command. Pass evalMode: 'agent' (MCP) or --eval-mode agent (CLI). The agent reads iteration output and returns pass/fail (retry) or a numeric score (optimize).
• Custom Eval Prompt: evalPrompt / --eval-prompt allows customizing the agent evaluator's instructions.
• Composite Evaluator: Routes evaluation to shell or agent implementation based on loop's evalMode setting.
• Database: Migration 15 adds eval_mode, eval_prompt columns on loops table and eval_feedback on loop_iterations.
• Pre-existing issue fixes: Defensive guard for empty exitCondition in shell evaluator, JSDoc on ExitConditionEvaluator interface, stale FEATURES.md version label removed.

Key Files

Area	Files
Domain	src/core/domain.ts, src/core/interfaces.ts
Agent evaluator	src/services/agent-exit-condition-evaluator.ts
Composite router	src/services/composite-exit-condition-evaluator.ts
Shell guard	src/services/exit-condition-evaluator.ts
Loop handler	src/services/handlers/loop-handler.ts
MCP adapter	src/adapters/mcp-adapter.ts
CLI	src/cli/commands/loop.ts
DB migration	src/implementations/database.ts

Stats

• 28 files changed, +1,988 / -114 lines
• 16 commits
• 1,661 tests passing across all suites

Test plan

• npm run test:core — 359 passed
• npm run test:handlers — 168 passed
• npm run test:services — 175 passed (includes agent evaluator, composite evaluator, shell guard tests)
• npm run test:repositories — 202 passed
• npm run test:adapters — 99 passed
• npm run test:cli — 260 passed
• npm run test:implementations — 328 passed
• npm run test:integration — 70 passed
• npm run build — clean
• npx biome check src/ — 86 files, 0 issues

[greptile-apps]

Greptile Summary

This PR introduces Agent Eval Mode for loop exit conditions, allowing an AI agent (instead of a shell command) to evaluate each iteration's output and return pass/fail (retry) or a numeric score (optimize). It adds a CompositeExitConditionEvaluator dispatcher, a new AgentExitConditionEvaluator service, database migration 15 (eval_mode, eval_prompt, eval_feedback), and threads the new fields through the CLI, MCP adapter, domain, repositories, and loop handler.

Key changes:

• EvalMode enum (shell | agent) added to domain; Loop and LoopCreateRequest updated accordingly
• AgentExitConditionEvaluator spawns a TaskDelegated eval task, waits for completion via event subscriptions with a fallback timer, and parses the last output line as PASS/FAIL or a numeric score
• Stale-state guard re-fetches loop + iteration after the (potentially slow) agent eval before persisting results
• Loop cancellation during in-flight agent eval is handled by subscribing to LoopCancelled and emitting TaskCancellationRequested
• Shell evaluator gains a defensive empty-exitCondition guard
• P1 issue found: The CLI's parseAgentModeArgs rejects --minimize/--maximize flags and omits evalDirection from its output, but LoopManagerService.createLoop unconditionally requires evalDirection for optimize strategy. This makes --eval-mode agent --strategy optimize always fail at the service layer, and the README documents this exact broken combination as a valid example.
• P2 note: When a custom evalPrompt is provided, it fully replaces defaultInstructions, silently dropping the critical format directives (PASS/FAIL or numeric last line) and the git diff / beat logs reading instructions the eval agent needs.

Confidence Score: 4/5

Not safe to merge as-is: agent optimize loops are completely non-functional from the CLI due to contradictory validation logic between the parser and service layer.

One P1 defect: the CLI unconditionally rejects --minimize/--maximize in agent mode while the service layer unconditionally requires evalDirection for optimize strategy, meaning every attempt to create an agent+optimize loop from the CLI will error. The README example documents this exact broken path. The MCP path is unaffected. All other changes (migration, repository, shell guard, stale-state guard, composite dispatcher, event-driven completion) look correct and well-tested.

src/cli/commands/loop.ts (parseAgentModeArgs — missing evalDirection propagation) and README.md (documents the broken --maximize example)

Important Files Changed

Filename	Overview
src/cli/commands/loop.ts	Refactored CLI parser into shell/agent mode sub-functions; rejects --minimize/--maximize for agent mode, breaking agent optimize loops
src/services/agent-exit-condition-evaluator.ts	New agent eval service: spawns eval task, waits with event-driven completion + fallback timer, parses PASS/FAIL or numeric score; custom evalPrompt loses format requirements
src/services/composite-exit-condition-evaluator.ts	Clean routing dispatcher; exhaustive switch with never guard is correct
src/services/handlers/loop-handler.ts	Added stale-state guard for slow agent evals; evalFeedback threaded through all result paths correctly
src/services/loop-manager.ts	Validation updated for agent mode; evalDirection still required for optimize (pre-existing); evalMode/evalPrompt cross-validation correct
src/implementations/database.ts	Migration 15 adds eval_mode (NOT NULL DEFAULT 'shell'), eval_prompt, and eval_feedback columns correctly
src/adapters/mcp-adapter.ts	evalMode/evalPrompt added to both CreateLoop and ScheduleLoop schemas; exitCondition made optional; evalDirection still accepted (MCP path correct)
src/core/domain.ts	EvalMode enum added; Loop and LoopCreateRequest updated with evalMode/evalPrompt; exitCondition made optional on request, defaults to '' in createLoop

Sequence Diagram

sequenceDiagram
    participant CLI_MCP as CLI / MCP Adapter
    participant LM as LoopManagerService
    participant LH as LoopHandler
    participant CEE as CompositeEvaluator
    participant AECE as AgentEvaluator
    participant EB as EventBus
    participant Repo as LoopRepository

    CLI_MCP->>LM: createLoop({ evalMode: agent, strategy, evalPrompt })
    LM->>LM: validate(evalMode, evalDirection if optimize)
    LM->>Repo: save(loop)
    LM-->>CLI_MCP: Loop created

    Note over LH: On task completion event
    LH->>CEE: evaluate(loop, taskId)
    CEE->>AECE: evaluate(loop, taskId) [evalMode=agent]
    AECE->>Repo: findIterationByTaskId(taskId)
    AECE->>AECE: buildEvalPrompt(loop, taskId)
    AECE->>EB: subscribe TaskCompleted/Failed/Cancelled/Timeout/LoopCancelled
    AECE->>EB: emit TaskDelegated(evalTask)
    EB-->>AECE: TaskCompleted(evalTaskId)
    AECE->>AECE: outputRepo.get(evalTaskId)
    AECE->>AECE: parseEvalOutput(lines, strategy)
    AECE-->>CEE: EvalResult { passed, score?, feedback? }
    CEE-->>LH: EvalResult
    LH->>LH: stale-state guard (re-fetch loop + iteration)
    LH->>LH: handleIterationResult(freshLoop, freshIteration, evalResult)
    LH->>Repo: updateIteration(evalFeedback, score, ...)

Loading

Comments Outside Diff (1)

1. src/cli/commands/loop.ts, line 332-367 (link)

   Agent optimize mode is always broken from the CLI

   parseAgentModeArgs explicitly rejects --minimize and --maximize (line 332-334), and the shared object never populates evalDirection. However, LoopManagerService.createLoop (in loop-manager.ts) unconditionally requires evalDirection for any optimize-strategy loop:

   if (request.strategy === LoopStrategy.OPTIMIZE && !request.evalDirection) {
     return err(new AutobeatError(ErrorCode.INVALID_INPUT, 'evalDirection is required for optimize strategy', ...));
   }

   This means beat loop "..." --eval-mode agent --strategy optimize will always fail at the service layer with "evalDirection is required for optimize strategy", and there is no way for the user to fix it because the CLI rejects the --maximize/--minimize flags needed to provide the direction.

   The README contradicts the implementation by documenting this command as valid:

   beat loop "Optimize the algorithm" --eval-mode agent --strategy optimize --maximize \
     --eval-prompt "Score the solution on correctness and efficiency (0-100)"

   The fix is to allow --minimize/--maximize when --eval-mode agent --strategy optimize is used and propagate evalDirection through parseAgentModeArgs's shared object.

_{Reviews (1): Last reviewed commit: "fix: resolve pre-existing review issues ..." | Re-trigger Greptile}

[greptile-apps]

Custom evalPrompt silently drops format requirements and git diff instructions

When loop.evalPrompt is set, it completely replaces defaultInstructions, including the critical format directives the eval agent must follow:

• Retry: the LAST LINE must be exactly PASS or FAIL.
• Optimize: the LAST LINE must be a numeric score.
• Both modes: the eval agent should run git diff and beat logs <taskId> to inspect the work.

None of these requirements appear in the built prompt when a custom evalPrompt is used. The eval agent doesn't know it needs to access the git diff, read task logs, or produce a PASS/FAIL token, so it will return free-form text and the evaluation will fail with "Eval agent output did not end with PASS or FAIL (got: ...)".

Consider injecting the gitDiffInstruction, the beat logs command, and the output format requirement unconditionally, and letting evalPrompt only override the evaluation criteria section.

[greptile-apps]

README example documents a command that the code rejects

The documented example:

beat loop "Optimize the algorithm" --eval-mode agent --strategy optimize --maximize \
  --eval-prompt "Score the solution on correctness and efficiency (0-100)"

--maximize is explicitly rejected in parseAgentModeArgs with the error "--minimize/--maximize are not valid with --eval-mode agent", so this command errors before ever reaching the service. This should be corrected once the CLI bug above is fixed.