Skip to content

RESULTS.md

Latest commit

 

History

History
94 lines (78 loc) · 4.5 KB

RESULTS.md

File metadata and controls

94 lines (78 loc) · 4.5 KB
Raw

veldt PLT Benchmark Results

Summary

Harness Agent Average Score Best Worst
veldt MSWEA mini-swe-agent + OCaml scaffold 25.7% jq 55.6% php 0.6%
zagent captain code-captain + generic Rust/C/Go 10.8% tree-sitter 26.5% quickjs 0.7%

Conclusion: The veldt MSWEA pipeline with task-specific OCaml scaffolding scores 2.4x higher than zagent's generic reverse-engineering approach.

Run 1: veldt MSWEA — mini-swe-agent + OCaml scaffold

  • Run ID: 1779045491
  • Date: 2026-05-17
  • Agent: mini-swe-agent (MSWEA) via veldt orchestrator
  • Model: claude-sonnet-4-20250514
  • Golden: 53c198f5-df9e-47cc-8282-3502956614c0
  • Scaffold: OCaml 5.2.1 + dune + Lexer/Pratt/Value/Env/Runtime libs
  • Eval: programbench 1.0.2

Scores

Instance Category Score Passed Total Time Notes
jqlang/jq DSL 55.6% 3,609 6,491 20m
lua/lua interpreter 46.7% 647 1,384 18m
luajit/luajit interpreter 44.0% 1,401 3,183 18m
tree-sitter/tree-sitter parser-gen 35.9% 607 1,690 26m
paradigmxyz/solar compiler 33.7% 852 2,528 27m
parcel-bundler/lightningcss compiler 27.1% 856 3,155 18m 2 branch errors
typst/typst compiler 8.0% 142 1,786 43m
tinycc/tinycc compiler 4.9% 102 2,062 22m
bellard/quickjs interpreter 0.8% 25 3,044 34m
php/php-src interpreter 0.6% 115 20,530 27m 1 branch error
AVERAGE 25.7% 8,356 45,853

Why veldt MSWEA scores well

  • Pre-built OCaml scaffold: Lexer, Pratt parser, Value, Env, Runtime, Regex, JSON, CLI
  • Task-specific prompts: DSL-parser (jq), lang-parser (lua), full-interp (generic)
  • Pre-built evaluators: jq has 120+ builtins in main.ml; lua has complete Lua_stdlib
  • Agent only writes the parser/interpreter, not the entire tool from scratch

Run 2: zagent code-captain — generic ProgramBench

  • Run ID: veldt-plt-1779142208
  • Date: 2026-05-18
  • Agent: zagent code-captain (gate-verified loop)
  • Model: claude-sonnet-4-20250514
  • Golden: 1f6440b3-79bb-4953-8a34-c7f64a993313 (OCaml + zagent)
  • Approach: Generic reverse-engineering, writes Rust/C/Go from scratch
  • Eval: programbench 1.0.2

Scores

Instance Category Score Passed Total Time Notes
tree-sitter/tree-sitter parser-gen 26.5% 448 1,690 17m
paradigmxyz/solar compiler 24.9% 629 2,528 69m
luajit/luajit interpreter 14.8% 471 3,183 13m
lua/lua interpreter 12.1% 167 1,384 15m
jqlang/jq DSL 9.6% 622 6,491 19m
parcel-bundler/lightningcss compiler 9.6% 302 3,155 23m
typst/typst compiler 4.6% 82 1,786 30m
tinycc/tinycc compiler 3.4% 70 2,062 22m
php/php-src interpreter 2.3% 462 20,530 12m 1 branch error
bellard/quickjs interpreter 0.7% 22 3,044 20m
AVERAGE 10.8% 3,275 45,853

Why zagent scores lower

  • No domain-specific scaffolding — writes everything from scratch
  • Uses Rust/C/Go instead of OCaml's algebraic types + pattern matching
  • Generic ProgramBench prompts, no awareness of language implementation patterns
  • Gate only checks compile.sh + source files exist, not test correctness
  • Captain fires 1-2 code cannons, extracting minimal stub implementations

Head-to-head comparison

Instance veldt MSWEA zagent captain Δ
jqlang/jq 55.6% 9.6% +46.0
lua/lua 46.7% 12.1% +34.7
luajit/luajit 44.0% 14.8% +29.2
tree-sitter/tree-sitter 35.9% 26.5% +9.4
paradigmxyz/solar 33.7% 24.9% +8.9
parcel-bundler/lightningcss 27.1% 9.6% +17.6
typst/typst 8.0% 4.6% +3.4
tinycc/tinycc 4.9% 3.4% +1.6
bellard/quickjs 0.8% 0.7% +0.1
php/php-src 0.6% 2.3% -1.7

veldt wins 9/10 tasks. The only zagent win is php (2.3% vs 0.6%), where both approaches struggle with the massive 20,530-test suite. The largest gains are on jq (+46), lua (+35), and luajit (+29) — exactly the tasks where the OCaml scaffold provides the most pre-built infrastructure.