Skip to content

Benchmark harness cannot validate submissions larger than ~5-6 KB (maximum recursion depth compiling the generated Benchmark/Artifact.lean) #590

Description

@sm-stack

Summary

scripts/yukon_benchmark.py prepare renders the submitted bytecode as 64-byte chunks joined by a flat
artifactChunk0 ++ artifactChunk1 ++ … ++ artifactChunkN chain. For submissions above roughly 80–100 chunks
(≈5–6 KB) the generated Challenge/<Track>/Benchmark/Artifact.lean fails to compile under the harness's own
build settings (lake build, moreLeanArgs = ["-j1"], default maxRecDepth) with

Challenge/Modexp/Benchmark/Artifact.lean:NNNN:7: error: maximum recursion depth has been reached

so Comparator can never reach the solution theorem. Every promoted submission so far is ≤ 3,924 bytes (62 chunks),
which is under the threshold, so this has not surfaced yet. A 20,035-byte Lean-verified MODEXP submission (314 chunks)
is blocked by it.

Reproduction (repo @ 6ec74c1, Lean 4.31.0, exactly what Comparator runs)

python3 scripts/yukon_benchmark.py prepare modexp Challenge/Modexp/Submission/bytecode.hex \
  benchmark-results/modexp/verified-bytecode.hex Challenge/Modexp/Benchmark/Artifact.lean Challenge/Modexp/Benchmark/Challenge.lean
lake env lean -j1 Challenge/Modexp/Benchmark/Artifact.lean -o .lake/build/lib/lean/Challenge/Modexp/Benchmark/Artifact.olean \
  -i .lake/build/lib/lean/Challenge/Modexp/Benchmark/Artifact.ilean
# -> error: maximum recursion depth has been reached   (bytecode.hex = 20,035 bytes, 314 chunks)

Synthetic artifacts placed in the package (same template, 64-byte chunks): 62 chunks OK, 80 OK, 100 FAIL, 200 FAIL, 314 FAIL.

Why this looks unintended rather than a size rule

  • No size limit is stated in README.md ("Repository contract"), Challenge/Modexp/README.md (only the operand lengths are bounded to 1024 bytes) or SUBMITTING.md, and prepare/score never check the size — bytecodeBytes is only reported as a metric; the score is the sum of execution gas.
  • The threshold (~80–100 chunks) is an emergent property of Lean's default recursion depth in code generation, not a constant anywhere in the harness, and the failure surfaces as an internal Lean error rather than a policy message.
  • The chunked rendering exists precisely to support multi-chunk artifacts; it just does not scale past ~100 chunks.
  • The largest verified submission so far is 4,838 bytes (76 chunks); the existing failed submissions are all ~1.3–1.6 KB, so nobody has hit this yet. EIP-170 (24,576 bytes) is the natural deployability bound.
    If a maximum submission size is intended, it would help to state and enforce it explicitly (e.g. in prepare) so solvers do not discover it as a compiler error.

Root cause

The failure is in code generation for the computable abbrev bytecode (a 313-deep left-nested HAppend term):
the same file compiles at default depth when bytecode is noncomputable, and it compiles unchanged with
-DmaxRecDepth=40000 (8192 is still not enough; 40000 and 100000 pass under -j1).

Proposed fixes (any one suffices)

  1. lakefile.toml: moreLeanArgs = ["-j1", "-DmaxRecDepth=40000"] (verified to compile the 314-chunk artifact under the Comparator's conditions).
  2. Render the artifact without a deep chain, e.g. ByteArray.mk #[…] as a single literal, or a balanced fold
    (List.foldl (· ++ ·) ∅ [chunk0, …] / pairwise tree), in render_artifact.
  3. Declare bytecode as noncomputable abbrev (nothing in the Lean side needs to evaluate it; the EVM scorer uses the hex).

Happy to open a PR for (1) or (2).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions