[Bug] Reference code for class-based skip-steps (13.6, 62.1) is silently truncated, failing all dependent sub-steps

[41ow1ives]

Summary

For the three hard-coded "skip" sub-steps, the gold reference in eval/data/{prob}.{step}.txt is injected as the previous code for later sub-steps via these two calls (in both runners):

func_name = extract_function_name(prob_data["sub_steps"][prev_step]["function_header"])
function_code = get_function_from_code(prev_file_content, func_name)
self.previous_llm_code[prev_step] = function_code

For the two skip-steps whose header defines a class — 13.6 (class Maxwell) and 62.1 (class Block / class EnlargedBlock) — this silently drops the class definition. As a result, every later sub-step that uses the class fails at evaluation time with NameError. 76.3 (a plain function) is unaffected.

Both entry points are affected, because they call the same two helpers with the same inputs:

• eval/scripts/gencode.py (Gencode.generate_response_with_steps, ~L71–83)
• eval/inspect_ai/scicode.py (ScicodePromptingAssistant.prepare_final_prompt_with_steps, ~L164–170)

Affected version

main @ e3158ea; root cause in src/scicode/parse/parse.py.

Reproduction (MRE)

To isolate and demonstrate the parsing flaw, here is a Minimal Reproducible Example (MRE). From a fresh clone (pip install -e .), run the following script from the repo root:

from datasets import load_dataset
from scicode.parse.parse import extract_function_name, get_function_from_code

ds = load_dataset("SciCode1/SciCode", split="test")
prob = next(p for p in ds if p["problem_id"] == "13")
header = prob["sub_steps"][5]["function_header"]          # sub-step 13.6 (a class)

name = extract_function_name(header)
print("extract_function_name ->", repr(name), "(expected 'Maxwell')")

ref = open("eval/data/13.6.txt").read()                   # provided reference for skip-step 13.6
injected = get_function_from_code(ref, name)
print("injected first line ->", repr(injected.splitlines()[0]))
print("injected has 'class Maxwell'? ->", "class Maxwell" in injected)

# what later sub-steps do: build a Maxwell instance
src = prob["required_dependencies"] + "\n" + injected + "\nMaxwell(8, 5.0)\n"
try:
    exec(src, {})
    print("exec -> OK")
except Exception as e:
    print("exec ->", type(e).__name__ + ":", e)

Output:

extract_function_name -> '__init__' (expected 'Maxwell')
injected first line -> 'def __init__(self, n_grid, x_out):'
injected has 'class Maxwell'? -> False
exec -> NameError: name 'Maxwell' is not defined

Two root causes

1. extract_function_name matches def before class.

def extract_function_name(function_header):
    pattern = r'\bdef\s+(\w+)\s*\('
    match = re.search(pattern, function_header)   # matched first
    if match:
        return match.group(1)
    else:                                          # class only as a fallback
        pattern = r'\bclass\s+(\w+)\s*\('
        ...

The headers for these steps are class skeletons that contain a method signature, e.g.:

class Maxwell:
    def __init__(self, n_grid, x_out):
        ...

so the inner def __init__ is matched first and the function returns '__init__' instead of 'Maxwell'. get_function_from_code(content, '__init__') then returns only the __init__ method, stripped of its class.

2. get_function_from_code(content, name) returns a single named node, but the reference files can define several.

eval/data/62.1.txt defines two top-level classes (class Block and class EnlargedBlock). Even if root cause #1 is fixed so extract_function_name returns 'EnlargedBlock', get_function_from_code(content, 'EnlargedBlock') returns only that one class (493 of 978 chars) and still drops class Block.

Impact

The assembled evaluation file is <dependencies> + <previous code> + <current step code> followed by the step's test. When the injected previous code lacks the class, the test — which constructs the class — raises NameError. This is independent of the model's output: the dependent sub-steps are unconditionally failed, putting a hard ceiling on the score achievable on problems 13 and 62.

• Problem 13: steps 8–15 each receive a maxwell instance (constructed in their test cases); under the bug only the Maxwell-independent steps (1–5, 7) can pass.
• Problem 62: later steps construct Block / EnlargedBlock.
• Problem 76: the skip-step is a function (generate_dna), extracted intact — unaffected.

Note on False Positives / Benchmark Validity: > Occasionally, some LLMs might still pass these steps if they hallucinate or redundantly generate the missing class Maxwell or Block inside their current step's response. However, this creates a severe fairness issue: models that strictly follow the instruction to only write the current step's code are heavily penalized with a NameError, while models that redundantly generate previous steps' code get an unfair advantage. This significantly compromises the validity of the benchmark scores for Problems 13 and 62.

Suggested fix

The reference files already contain exactly the code intended for injection, so re-extracting a single definition is both unnecessary and lossy. The current parsing logic strips out the class definition and causes a NameError in subsequent tests.

Assuming the intention was to inject the full class definition from the gold reference (as suggested by the fallback regex for class in extract_function_name), modifying it to inject prev_file_content directly resolves this issue. In the skip-step branch of both runners, inject the file content directly:

- func_name = extract_function_name(prob_data["sub_steps"][prev_step]["function_header"])
- function_code = get_function_from_code(prev_file_content, func_name)
- self.previous_llm_code[prev_step] = function_code
+ self.previous_llm_code[prev_step] = prev_file_content

This keeps both classes in 62.1 and the full class in 13.6, and fixes both runners.

(If you prefer to keep extraction elsewhere, extract_function_name can also be made to match whichever of def/class appears first — re.search(r'\b(?:def|class)\s+(\w+)', function_header) — but that alone does not fix 62.1, which needs both classes.)

If this approach looks good to you, I'd be happy to submit a PR with this fix!