VeriGen generates verifiable OSWorld and ScienceBoard tasks. It asks a local coding agent to write task JSON/evaluator logic, validates the result with rules and an LLM judge, repairs common failures, and can optionally check the task inside the live OSWorld Docker/VM sandbox.
Supported environments:
osworldscienceboard
Supported generation modes:
task_verify: create new verifiable tasks from docs/cache sources or optional seed task JSONs.traj_verify: create new verifiable tasks from agent trajectories.
Supported coding agents:
claudecodexopencode
Create the generation environment:
cd /path/to/SCALE-CUA/VeriGen
conda create -n verigen python=3.10 -y
conda activate verigen
pip install -r requirements.txt
export PYTHONPATH=$PWDCreate the sandbox environment if you want live Docker/VM verification:
conda create -n osworld python=3.10 -y
conda activate osworld
pip install -r requirements.txt
pip install -r src/envs/osworld_env/desktop_env/server/requirements.txtPrepare local assets:
- OSWorld cache:
VeriGen/osworld_env/cache - VM image for sandbox verification: set
SANDBOX_VM_IMAGE=/path/to/Ubuntu.qcow2
Configure at least one coding-agent CLI locally:
- Claude Code: https://code.claude.com/docs/en/settings
- Codex CLI: https://developers.openai.com/codex/cli
- OpenCode: https://opencode.ai/docs/
Use scripts/example_task.sh for docs/cache-driven generation. It does not
require seed task JSONs by default.
cd /path/to/SCALE-CUA/VeriGen
ENV_NAME=scienceboard \
CODE_AGENT=claude \
GEN_MODEL=claude-opus-4-7 \
LLM_MODEL=claude-opus-4-7 \
DOCS_SOURCE_LIMIT=5 \
OUTPUT_DIR=results/example_scienceboard \
bash scripts/example_task.shFor OSWorld, task-level generation reuses files under osworld_env/cache and
chooses startup configs from the artifact type. For ScienceBoard, generation
uses the distilled task-family docs in docs/skills/task_blueprints/.
Set DOCS_SOURCE_LIMIT=0 for all docs/cache sources. Use a small value such as
5 for smoke tests.
Use scripts/example_traj.sh when you already have rollout trajectories.
cd /path/to/SCALE-CUA/VeriGen
ENV_NAME=osworld \
CODE_AGENT=claude \
GEN_MODEL=claude-opus-4-7 \
LLM_MODEL=claude-opus-4-7 \
INPUT_PATH=/path/to/trajectory_runs \
TESTSET_ROOT=/path/to/original_task_jsons \
OUTPUT_DIR=results/example_traj \
bash scripts/example_traj.shA trajectory directory should contain:
record.jsonl: action/observation trace.original_config.json: original task JSON. If missing, provideTESTSET_ROOTso VeriGen can preprocess trajectories and recover it.trajectory_metadata.jsonorresult.txt: success score and task metadata.screenshots/: useful for trajectory inspection when available.filtered_messages_last.json: optional but useful for understanding the last model actions.
For successful trajectories, VeriGen tends to create harder related tasks. For failed trajectories, it extracts the sub-skills the agent demonstrated and creates tasks that are simpler but still verifiable.
ENV_NAME:osworldorscienceboard.CODE_AGENT:claude,codex, oropencode.GEN_MODEL: model used by the coding agent.LLM_MODEL: model used by judging/fixing.NUM_VARIATIONS: generated tasks per source, default1.WORKERS: parallel generation workers, default2.TIMEOUT: per-source generation timeout in seconds.DIFFICULTY:normal,harder, oreasier.SANDBOX_VERIFY=true: run live Docker/VM verification.AUTOGEN_CONDA_ENV: generation conda env, defaultverigen.SANDBOX_CONDA_ENV: sandbox conda env, defaultosworld.HTTP_PROXY_URL/HTTPS_PROXY_URL: proxy passed to the selected coding agent.
Claude, Codex, and OpenCode share the same generation, validation, repair, and sandbox pipeline. Only the non-interactive coding-agent command changes.
Generated tasks are written under OUTPUT_DIR. Prompt traces are in
prompts/, temporary evaluator files are in tmp_funcs/, and sandbox traces
are in sandbox_verify/ when enabled.
Local caches, generated results, VM images, and private datasets are not meant to be committed.