This directory contains the OSWorld and ScienceBoard evaluation runtime used by ScaleCUA. The main entry points are:
run_scalecua_os.pyfor OSWorld-Std tasks.run_scalecua_sci.pyfor ScienceBoard tasks.scripts/test_osworld_std.shandscripts/test_sci.shas simple launch templates.
osworld_eval/
├── cache/ # local OSWorld task assets, if available
├── desktop_env/ # OSWorld desktop environment runtime
├── evaluation_examples/
│ ├── examples/ # OSWorld task JSONs
│ ├── scienceboard/ # ScienceBoard task JSONs
│ ├── settings/ # account/proxy settings templates
│ ├── test_all.json # OSWorld full-set meta file
│ ├── test_nogdrive.json # OSWorld meta file without Google Drive tasks
│ └── test_scienceboard.json # ScienceBoard full-set meta file
├── mm_agents/
│ ├── scalecua-osworld/ # ScaleCUAOSWorldAgent
│ └── scalecua-sci/ # ScaleCUASciAgent
├── scripts/
│ ├── test_osworld_std.sh # OSWorld launch template
│ └── test_sci.sh # ScienceBoard launch template
├── run_scalecua_os.py
└── run_scalecua_sci.py
Generated logs/, results/, and recordings/ are local runtime outputs and
should not be committed.
Create one conda environment for this evaluation package:
cd osworld_eval
conda create -n osworld python=3.10 -y
conda activate osworld
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m playwright install
python -m pip install -e .Docker must be available on the host because the default provider is docker:
docker psNone of the large runtime assets are committed to this repo. Obtain them as follows (only the ScienceBoard VM must be downloaded manually; everything else is fetched automatically at runtime).
Evaluated model checkpoints (public).
- Hugging Face: https://huggingface.co/collections/extreme1228/scalecua-6a4cad5e3c80a230622eaff3
- ModelScope: https://modelscope.cn/collections/extreme1228/ScaleCUA
Serve any checkpoint behind an OpenAI-compatible endpoint (e.g. vLLM) and point
MODEL / MODEL_BASE_URL / MODEL_API_KEY at it (see "Model Endpoint Notes").
OSWorld VM image + container (auto). If you do not pass PATH_TO_VM, the
docker provider downloads the official Ubuntu disk from
https://huggingface.co/datasets/xlangai/ubuntu_osworld and pulls the container
image happysixd/osworld-docker automatically. To reuse a local copy:
export PATH_TO_VM=/path/to/Ubuntu.qcow2.
OSWorld task file cache (auto). Per-task seed files (xlsx / sqlite / images /
fonts …) are fetched at task setup from
https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache. No manual
step is required; to pre-seed offline, download that dataset into cache/.
ScienceBoard VM image (required for ScienceBoard, manual). A custom qcow2 with the six science apps (ChimeraX / Celestia / KAlgebra / GrassGIS / TeXstudio / Lean) and the evaluation REST server preinstalled:
- ModelScope: https://modelscope.cn/datasets/extreme1228/ScaleCUA-ScienceBoard-VM
- Hugging Face: https://huggingface.co/datasets/extreme1228/ScaleCUA-ScienceBoard-VM
pip install -U modelscope
modelscope download --dataset extreme1228/ScaleCUA-ScienceBoard-VM \
Ubuntu2_scienceboard.qcow2.zip --local_dir ./vm
unzip ./vm/Ubuntu2_scienceboard.qcow2.zip -d ./vm
export PATH_TO_VM="$(pwd)/vm/Ubuntu2_scienceboard.qcow2"Credentials / settings (optional, task-subset dependent).
evaluation_examples/settings/ ships sanitized placeholders for Google, Google
Drive, Thunderbird, and residential-proxy accounts. Fill in your own only for the
tasks that need them; the default OSWorld list (test_nogdrive.json) already
skips Google Drive tasks. See SETUP_GUIDELINE.md for the exact formats.
OSWorld tasks may refer to files under local cache. This repo uses:
osworld_eval/cache
If a task fails during setup because an asset is missing, check the task JSON in
evaluation_examples/examples/ and place the required file in the expected cache
location.
For Chrome/web tasks that require external network proxy, edit the proxy setting file used by the task JSONs, for example:
osworld_eval/evaluation_examples/settings/proxy/dataimpulse.json
Only tasks whose JSON enables proxy use will read that file. Do not hard-code the external website proxy in the run scripts.
Edit scripts/test_osworld_std.sh or export these variables before launch:
export MODEL="<your-model-name>"
export MODEL_BASE_URL="<your-model-base-url>"
export MODEL_API_KEY="<your-model-api-key>"
export PATH_TO_VM="/path/to/osworld_vm.qcow2"Run the OSWorld-Std meta file used for ScaleCUA evaluation:
bash scripts/test_osworld_std.shUseful overrides:
DOMAIN=chrome bash scripts/test_osworld_std.sh
NUM_WORKERS=4 MAX_STEPS=50 bash scripts/test_osworld_std.sh
RESULT_DIR=results/osworld-my-run LOG_DIR=logs/my-run bash scripts/test_osworld_std.shThe script calls:
osworld_eval/run_scalecua_os.py
with evaluation_examples/test_nogdrive.json and evaluation_examples/examples
by default if META_PATH is not overridden. This matches the
training/evaluation protocol that excludes Google Drive tasks. To run the
original full OSWorld list, set:
META_PATH=evaluation_examples/test_all.json bash scripts/test_osworld_std.shEdit scripts/test_sci.sh or export these variables before launch:
export MODEL="<your-model-name>"
export MODEL_BASE_URL="<your-model-base-url>"
export MODEL_API_KEY="<your-model-api-key>"
export PATH_TO_VM="/path/to/scienceboard_vm.qcow2"Run the full ScienceBoard meta file:
bash scripts/test_sci.shUseful overrides:
SCI_DOMAIN=Celestia bash scripts/test_sci.sh
NUM_WORKERS=4 MAX_STEPS=50 bash scripts/test_sci.sh
RESULT_DIR=results/sci-my-run LOG_DIR=logs/my-run bash scripts/test_sci.shThe script calls:
osworld_eval/run_scalecua_sci.py
with evaluation_examples/test_scienceboard.json and
evaluation_examples/scienceboard by default.
MODEL_BASE_URL should be the OpenAI-compatible endpoint for the deployed model.
The scripts export it as OPENAI_BASE_URL; the OSWorld script also maps the same
value to Anthropic-style variables for agent compatibility.
If model API access needs a local proxy, set LAN_PROXY before launching:
LAN_PROXY="http://127.0.0.1:<port>" bash scripts/test_osworld_std.shDo not commit real model URLs, API keys, VM paths, account credentials, or private proxy credentials.