Skip to content

AGENT-ECONOMICS-V1.md

Latest commit

 

History

History
98 lines (82 loc) · 5.52 KB

AGENT-ECONOMICS-V1.md

File metadata and controls

98 lines (82 loc) · 5.52 KB
Raw

Agent context economics v1

Audience: agent and tool authors; compiler contributors.

This offline benchmark measures how much checked context a fixed question returns. It does not measure model tokens, latency, answer quality, or a whole repository.

Status: deterministic evidence for the current context contract.

semaprax context-benchmark <manifest> evaluates checked maintenance questions from a strict tab-separated semaprax.agent-context-benchmark.v1 manifest. Sources are canonical regular files beneath the manifest directory; absolute, parent, current-directory, repeated/trailing separator, backslash, control-character, case-mismatched, Windows forbidden-character, trailing-dot/trailing-space/alternate-data-stream/reserved-device, missing, and symlink aliases fail as SPX-G005. Malformed integers, unknown or duplicate filters/cases/IDs, unavailable Graph v10 facets, and evidence outside the declared relevant set also fail closed. The command performs no network access and emits canonical semaprax.agent-context-economics.v1 JSON.

Measurements

Each case freezes its question, source, root, context limits, supported filters, relevant function IDs, and required evidence IDs. Results bind the exact manifest SHA-256, repeat the exact relevant/evidence ID arrays, bind the source revision and exact context SHA-256, and record:

  • UTF-8 source, question, and context bytes;
  • emitted function nodes;
  • semaprax.lexical-token.v1 units, which count maximal ASCII alphanumeric/underscore runs and each non-whitespace punctuation or non-ASCII scalar as one unit;
  • relevance precision as relevant emitted facts divided by all emitted facts;
  • evidence recall as emitted required-evidence facts divided by declared required evidence.

Ratios are reduced exact fractions, never floating-point estimates. The lexical unit is repository-defined and explicitly carries model_tokens: false; it is not compatible with any model tokenizer and cannot support a token-cost or model-context savings claim. Relevant/evidence IDs are reviewed corpus labels, not automatically proven answer correctness.

The v1 corpus contains four maintenance questions over one checked module. Its golden aggregate is seven emitted nodes, 1/1 relevance, 2/3 evidence recall (one intentionally node-truncated case), context/source bytes 1817/616, and context/source lexical units 43/11. The context is larger than repeatedly reading this small source. That result is retained rather than described as a saving; larger representative repositories and model-specific evaluation are future gates.

Two cases also freeze the complete context CLI JSON, while the economics golden freezes every case digest, budget, score, and aggregate. Mutation, manifest rejection, independent JSON parsing, and deterministic replay are executable tests.

Quality routing

scripts/quality.sh accepts quick, changed, or full (the no-argument default remains full). quick is advisory inner-loop feedback only. changed derives the exact union of a reviewed base-to-HEAD committed diff, staged, unstaged, untracked, deleted, and both rename-side paths directly from Git. The base is the canonical ancestor commit from SEMAPRAX_QUALITY_BASE. Otherwise, the planner resolves SEMAPRAX_QUALITY_TARGET_REF or the symbolic refs/remotes/origin/HEAD, computes git merge-base --all, and requires exactly one base. It never consults the current branch upstream. Non-UTF-8 environment values, absence, abbreviation, non-ancestor, or ambiguity reject planning. Optional explicit paths must equal that complete set. Traversal, absolute, backslash, repeated/trailing separator, control-character, case-mismatched, Windows forbidden-character/trailing-dot/trailing-space/ADS/ reserved-device, symlink, and nonexistent non-deletion aliases are rejected. Only documentation and the economics files below are narrow; unknown, workflow, platform, native, router, or core semantic paths escalate to full. full is the complete local workspace baseline, not hosted OS, sanitizer, simulator, emulator, signing, or packaging evidence.

Path family Invariant Focused changed evidence
docs/**, README.md, CHANGELOG.md Documentation truth documentation, examples, Rustdoc
src/agent_economics.rs Offline economics contract agent-context, economics, documentation, examples
benchmarks/agent-context-v1/** and its snapshots/tests Corpus, exact goldens, scores agent-context, economics, documentation, examples
src/graph.rs, src/main.rs, and every other path including router changes Broad compiler/graph dispatch, unknown, wide, or self-modifying routing impact Full workspace baseline

scripts/quality-route.sh prints the versioned semaprax.quality-route.v2 path and gate plan. The executor validates its schema, complete envelope, canonical base, and the exact ordered gate sequence for the effective profile, then dispatches only those declared gates. Missing, duplicate, reordered, or wrong-profile gates reject the plan. An unmapped path never means “no tests.”

Graphify remains deferred under ADR 0001. No new pinned temporary Graphify run was available for this tranche, so no adoption or comparative token claim was made.

The separate Agent Task Economics v1 records bounded compiler-protocol and review-material observations for the integrated twelve-step signature workflow. It does not reinterpret these context results or supply the still-unobserved model, latency and human-review measurements.