AGENTS.md

AGENTS.md

• The project uses pnpm for managing dependencies
• The project uses Turbo for running tasks. Prefer pnpm exec turbo run <task> whenever a Turbo task exists
• Add repository automation scripts to the script directory
• For tests, use pnpm exec turbo run test -- --run. You can optionally apply a filter to filter by test name, path, etc
• For linting, use pnpm exec turbo run lint lint:npm
• For formatting, use pnpm format. This will format all files in the project
• For type checks, use pnpm exec turbo run type-check. You can use Turbo's --filter option to target individual workspaces
• README files are public-facing and should only contain information about using the package. Keep contributor and agent instructions elsewhere
• Do not create changesets for updates to features, fixes, or other changes that have not been released yet
• For backwards-incompatible public API or behavior changes, deprecations, or recommended replacements, update packages/agent-eval/docs/migrations/ in the same change. Load the changesets skill and its migration reference even when no changeset is needed. Index guides by version introduced, entrypoint, export(s), and a linked description. Keep guides to a brief change description, versioned before/after snippets, and verification; add migration steps only when needed. While the package is pre-1.0, use the next minor version for pending migrations; revisit this policy at 1.0.
• Update documentation and skill references only when a change affects how consumers use the package or makes existing guidance inaccurate. Do not add documentation for type inference or declaration portability fixes, internal refactors, or regression tests that preserve existing usage, even when the fix adds supporting type exports. The synchronization rules below apply only when there is a usage or guidance change.
• Keep the method overview in docs/sandbox.md in sync with the Sandbox interface in packages/agent-eval/src/sandbox/types.ts whenever methods, signatures, or behavior change. Exclude [Symbol.asyncDispose]() from the table. List method names without arguments; rely on TypeScript for argument details.
• Keep packages/agent-eval/docs/AGENTS.md and its references/ up-to-date in the same change when changes to project structure, methodology, domain models, public APIs, CLI commands, or runtime requirements affect their guidance. Keep entrypoint concise, put detailed guidance in references, and use npm/npx in doc examples. Keep skills/agent-eval/SKILL.md as a pointer to the installed package docs.

Evidence for decisions

• When a decision, recommendation, or review finding relies on a resource, include the specific evidence and explain how it supports the conclusion.
• Cite repository files with paths and line numbers. For external resources, link to the relevant section or comment and quote the supporting passage when line numbers are unavailable.
• Read the cited resource before relying on it. Distinguish what it explicitly states from your inference, and state uncertainty or conflicting evidence.

Code

• Before changing lifecycle code, identify its observable contract and the consequences for failure and cleanup. Load the architecture skill and the testing skill when changing its tests.
• Parse, don't validate. Use the type system to gurantee correctness.
  • When possible, use zod/mini to parse as much information as possible from a given input
  • When parsing, use the most specific type possible. For example:
    • If a string is expected to be a URL, parse it as a URL instead of a string
    • If the input is a path to a file, determine the correct path and make sure the file exists
    • If a collection must have more than one value, use zod to check that it is has more than one value
  • Handle as much of this logic as possible when ingesting the unknown data instead of having validation and checks sprinkled throughout the library

Errors and logging

• When logging errors, use the err field, not error, for example logger.error({err: error}, 'Failed to run trial'). Pino's default error serializer handles err and preserves the error message and stack trace. Using {error} can omit those details and hide the underlying cause.

Configuration

• Design zod schemas to parse configuration for unknown input
• When designing them, be as permissive as needed for the configuration but:
  • Apply reasonable defaults during parsing with z._default, e.g. z._default(z.array(ItemSchema), []) for a collection
  • Prefer concrete parsed values over null, undefined, or optional fields whenever a meaningful default exists. Reserve optional or nullable fields for cases where absence has domain meaning, rather than propagating null checks to consumers
  • Transform different inputs into structured types, for example a union of object with different fields becomes a discriminated union with a type field
• Design parse* functions instead of allowing for zod schemas to be called directly
• parse* functions may need to take more arguments than just the unknown input, for example if you need to validate that a path exists
• Accept file paths as either relative (and look them up to get the absolute path) or absolute. Include both cases when designing zod schemas and parsing unknown input

Type System

• Encode as much information as possible in the system; make illegal states irrepresentable
  • For example, if a resource requires a state transition (e.g. a docker image needs to be built), represent this as a type and design APIs around using the type that corresponds to the state that it needs instead of needing to code around that state each time it is used

Pull Requests

• Title format: : description
• Conventional types:
  • feat: A new feature
  • fix: A bug fix
  • docs: Documentation only changes
  • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
  • refactor: A code change that neither fixes a bug nor adds a feature
  • perf: A code change that improves performance
  • test: Adding missing tests or correcting existing tests
  • chore: Changes to the build process or auxiliary tools and libraries such as documentation generation
  • ci: A change to our workflows
• Description: A brief description of the changes made in the pull request. It should be concise and informative, providing enough context for reviewers to understand the purpose of the changes.
• Example: "feat: add new authentication method for improved security"
• Fill out the pull request template for the repo
• Always run the tasks in ci.yml before committing so that ci is green, except for agentic workflow compilation steps
• When updating github/gh-aw-actions, align the gh-aw compiler version in .github/workflows/ci.yml with the updated action version. Use that compiler version to run gh aw compile --validate and commit all regenerated files. Run compilation a second time and confirm that it produces no further changes.
• When making a change to the website, include before and after screenshots in the Pull Request description.

  <details>
    <summary>Before / After</summary>
  
  | Before                     | After                     |
  | :------------------------- | :------------------------ |
  | <!-- screenshot before --> | <!-- screenshot after --> |
  
  </details>

Adding scenarios

When adding a scenario, follow these rules:

• Unless explicitly asked for, use scenarios/000-nextjs-template as the base for the scenario
• Add scenarios to ./scenarios with the next number in the sequence
• Describe (at a high level) what we are testing for the agent
• Define the precise target behavior before choosing checks. Separate the evaluated outcome from incidental fixture work; a UI task used to elicit feedback does not justify grading imports, source files, DOM, or geometry unless those are the target.
• Follow the scenario and check guidance. Prefer direct deterministic evidence for observable events and reserve judges for semantic criteria. Inline hooks are valid; the template's Vitest setup is optional.
• Verify only the changed criterion with relevant positive and negative evidence. Reuse existing infrastructure instead of adding a parallel harness or testing unrelated fixture behavior.
• When authoring the prompt for the scenario, use general language to describe the task. Unless explicitly asked for, do not directly mention Primer or include in the prompt instructions on how to access/use Primer. We are testing tooling that should include Primer into context so things don't need that in the prompt

Adding experiments

When adding an experiment, follow these rules:

• Check to see if an experiment might already exist for this use-case and share it, when appropriate
• New experiments are added at ./experiments as a new *.ts file
• Provide an appropriate name for the experiment as its file name
• Use a default export created with defineConfig
• Provide an appropriate name and description for what we are testing
• Select appropriate models and scenarios based on the experiment
• For treatments, make sure to include the fewest number possible to meet the experiment goals
• If you are not sure about experiment goals, prompt the user for more information before creating the experiment

Next.js: ALWAYS read docs before coding.js

Before any Next.js work, find and read the relevant doc in node_modules/next/dist/docs/. Your training data is outdated — the docs are the source of truth.