Skip to content

Latest commit

 

History

History
135 lines (100 loc) · 4.22 KB

README.md

File metadata and controls

135 lines (100 loc) · 4.22 KB

VeriGen

VeriGen generates verifiable OSWorld and ScienceBoard tasks. It asks a local coding agent to write task JSON/evaluator logic, validates the result with rules and an LLM judge, repairs common failures, and can optionally check the task inside the live OSWorld Docker/VM sandbox.

Supported environments:

  • osworld
  • scienceboard

Supported generation modes:

  • task_verify: create new verifiable tasks from docs/cache sources or optional seed task JSONs.
  • traj_verify: create new verifiable tasks from agent trajectories.

Supported coding agents:

  • claude
  • codex
  • opencode

Setup

Create the generation environment:

cd /path/to/SCALE-CUA/VeriGen
conda create -n verigen python=3.10 -y
conda activate verigen
pip install -r requirements.txt
export PYTHONPATH=$PWD

Create the sandbox environment if you want live Docker/VM verification:

conda create -n osworld python=3.10 -y
conda activate osworld
pip install -r requirements.txt
pip install -r src/envs/osworld_env/desktop_env/server/requirements.txt

Prepare local assets:

  • OSWorld cache: VeriGen/osworld_env/cache
  • VM image for sandbox verification: set SANDBOX_VM_IMAGE=/path/to/Ubuntu.qcow2

Configure at least one coding-agent CLI locally:

Task-Level Generation

Use scripts/example_task.sh for docs/cache-driven generation. It does not require seed task JSONs by default.

cd /path/to/SCALE-CUA/VeriGen

ENV_NAME=scienceboard \
CODE_AGENT=claude \
GEN_MODEL=claude-opus-4-7 \
LLM_MODEL=claude-opus-4-7 \
DOCS_SOURCE_LIMIT=5 \
OUTPUT_DIR=results/example_scienceboard \
bash scripts/example_task.sh

For OSWorld, task-level generation reuses files under osworld_env/cache and chooses startup configs from the artifact type. For ScienceBoard, generation uses the distilled task-family docs in docs/skills/task_blueprints/.

Set DOCS_SOURCE_LIMIT=0 for all docs/cache sources. Use a small value such as 5 for smoke tests.

Trajectory-Level Generation

Use scripts/example_traj.sh when you already have rollout trajectories.

cd /path/to/SCALE-CUA/VeriGen

ENV_NAME=osworld \
CODE_AGENT=claude \
GEN_MODEL=claude-opus-4-7 \
LLM_MODEL=claude-opus-4-7 \
INPUT_PATH=/path/to/trajectory_runs \
TESTSET_ROOT=/path/to/original_task_jsons \
OUTPUT_DIR=results/example_traj \
bash scripts/example_traj.sh

A trajectory directory should contain:

  • record.jsonl: action/observation trace.
  • original_config.json: original task JSON. If missing, provide TESTSET_ROOT so VeriGen can preprocess trajectories and recover it.
  • trajectory_metadata.json or result.txt: success score and task metadata.
  • screenshots/: useful for trajectory inspection when available.
  • filtered_messages_last.json: optional but useful for understanding the last model actions.

For successful trajectories, VeriGen tends to create harder related tasks. For failed trajectories, it extracts the sub-skills the agent demonstrated and creates tasks that are simpler but still verifiable.

Key Variables

  • ENV_NAME: osworld or scienceboard.
  • CODE_AGENT: claude, codex, or opencode.
  • GEN_MODEL: model used by the coding agent.
  • LLM_MODEL: model used by judging/fixing.
  • NUM_VARIATIONS: generated tasks per source, default 1.
  • WORKERS: parallel generation workers, default 2.
  • TIMEOUT: per-source generation timeout in seconds.
  • DIFFICULTY: normal, harder, or easier.
  • SANDBOX_VERIFY=true: run live Docker/VM verification.
  • AUTOGEN_CONDA_ENV: generation conda env, default verigen.
  • SANDBOX_CONDA_ENV: sandbox conda env, default osworld.
  • HTTP_PROXY_URL / HTTPS_PROXY_URL: proxy passed to the selected coding agent.

Claude, Codex, and OpenCode share the same generation, validation, repair, and sandbox pipeline. Only the non-interactive coding-agent command changes.

Outputs

Generated tasks are written under OUTPUT_DIR. Prompt traces are in prompts/, temporary evaluator files are in tmp_funcs/, and sandbox traces are in sandbox_verify/ when enabled.

Local caches, generated results, VM images, and private datasets are not meant to be committed.