Skip to content

Latest commit

 

History

History
209 lines (157 loc) · 6.85 KB

README.md

File metadata and controls

209 lines (157 loc) · 6.85 KB

ScaleCUA OSWorld Evaluation

This directory contains the OSWorld and ScienceBoard evaluation runtime used by ScaleCUA. The main entry points are:

  • run_scalecua_os.py for OSWorld-Std tasks.
  • run_scalecua_sci.py for ScienceBoard tasks.
  • scripts/test_osworld_std.sh and scripts/test_sci.sh as simple launch templates.

Directory Layout

osworld_eval/
├── cache/                         # local OSWorld task assets, if available
├── desktop_env/                   # OSWorld desktop environment runtime
├── evaluation_examples/
│   ├── examples/                  # OSWorld task JSONs
│   ├── scienceboard/              # ScienceBoard task JSONs
│   ├── settings/                  # account/proxy settings templates
│   ├── test_all.json              # OSWorld full-set meta file
│   ├── test_nogdrive.json         # OSWorld meta file without Google Drive tasks
│   └── test_scienceboard.json     # ScienceBoard full-set meta file
├── mm_agents/
│   ├── scalecua-osworld/          # ScaleCUAOSWorldAgent
│   └── scalecua-sci/              # ScaleCUASciAgent
├── scripts/
│   ├── test_osworld_std.sh        # OSWorld launch template
│   └── test_sci.sh                # ScienceBoard launch template
├── run_scalecua_os.py
└── run_scalecua_sci.py

Generated logs/, results/, and recordings/ are local runtime outputs and should not be committed.

Conda Setup

Create one conda environment for this evaluation package:

cd osworld_eval
conda create -n osworld python=3.10 -y
conda activate osworld
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m playwright install
python -m pip install -e .

Docker must be available on the host because the default provider is docker:

docker ps

Resources And Downloads

None of the large runtime assets are committed to this repo. Obtain them as follows (only the ScienceBoard VM must be downloaded manually; everything else is fetched automatically at runtime).

Evaluated model checkpoints (public).

Serve any checkpoint behind an OpenAI-compatible endpoint (e.g. vLLM) and point MODEL / MODEL_BASE_URL / MODEL_API_KEY at it (see "Model Endpoint Notes").

OSWorld VM image + container (auto). If you do not pass PATH_TO_VM, the docker provider downloads the official Ubuntu disk from https://huggingface.co/datasets/xlangai/ubuntu_osworld and pulls the container image happysixd/osworld-docker automatically. To reuse a local copy: export PATH_TO_VM=/path/to/Ubuntu.qcow2.

OSWorld task file cache (auto). Per-task seed files (xlsx / sqlite / images / fonts …) are fetched at task setup from https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache. No manual step is required; to pre-seed offline, download that dataset into cache/.

ScienceBoard VM image (required for ScienceBoard, manual). A custom qcow2 with the six science apps (ChimeraX / Celestia / KAlgebra / GrassGIS / TeXstudio / Lean) and the evaluation REST server preinstalled:

pip install -U modelscope
modelscope download --dataset extreme1228/ScaleCUA-ScienceBoard-VM \
  Ubuntu2_scienceboard.qcow2.zip --local_dir ./vm
unzip ./vm/Ubuntu2_scienceboard.qcow2.zip -d ./vm
export PATH_TO_VM="$(pwd)/vm/Ubuntu2_scienceboard.qcow2"

Credentials / settings (optional, task-subset dependent). evaluation_examples/settings/ ships sanitized placeholders for Google, Google Drive, Thunderbird, and residential-proxy accounts. Fill in your own only for the tasks that need them; the default OSWorld list (test_nogdrive.json) already skips Google Drive tasks. See SETUP_GUIDELINE.md for the exact formats.

Task Assets And Settings

OSWorld tasks may refer to files under local cache. This repo uses:

osworld_eval/cache

If a task fails during setup because an asset is missing, check the task JSON in evaluation_examples/examples/ and place the required file in the expected cache location.

For Chrome/web tasks that require external network proxy, edit the proxy setting file used by the task JSONs, for example:

osworld_eval/evaluation_examples/settings/proxy/dataimpulse.json

Only tasks whose JSON enables proxy use will read that file. Do not hard-code the external website proxy in the run scripts.

Running OSWorld-Std

Edit scripts/test_osworld_std.sh or export these variables before launch:

export MODEL="<your-model-name>"
export MODEL_BASE_URL="<your-model-base-url>"
export MODEL_API_KEY="<your-model-api-key>"
export PATH_TO_VM="/path/to/osworld_vm.qcow2"

Run the OSWorld-Std meta file used for ScaleCUA evaluation:

bash scripts/test_osworld_std.sh

Useful overrides:

DOMAIN=chrome bash scripts/test_osworld_std.sh
NUM_WORKERS=4 MAX_STEPS=50 bash scripts/test_osworld_std.sh
RESULT_DIR=results/osworld-my-run LOG_DIR=logs/my-run bash scripts/test_osworld_std.sh

The script calls:

osworld_eval/run_scalecua_os.py

with evaluation_examples/test_nogdrive.json and evaluation_examples/examples by default if META_PATH is not overridden. This matches the training/evaluation protocol that excludes Google Drive tasks. To run the original full OSWorld list, set:

META_PATH=evaluation_examples/test_all.json bash scripts/test_osworld_std.sh

Running ScienceBoard

Edit scripts/test_sci.sh or export these variables before launch:

export MODEL="<your-model-name>"
export MODEL_BASE_URL="<your-model-base-url>"
export MODEL_API_KEY="<your-model-api-key>"
export PATH_TO_VM="/path/to/scienceboard_vm.qcow2"

Run the full ScienceBoard meta file:

bash scripts/test_sci.sh

Useful overrides:

SCI_DOMAIN=Celestia bash scripts/test_sci.sh
NUM_WORKERS=4 MAX_STEPS=50 bash scripts/test_sci.sh
RESULT_DIR=results/sci-my-run LOG_DIR=logs/my-run bash scripts/test_sci.sh

The script calls:

osworld_eval/run_scalecua_sci.py

with evaluation_examples/test_scienceboard.json and evaluation_examples/scienceboard by default.

Model Endpoint Notes

MODEL_BASE_URL should be the OpenAI-compatible endpoint for the deployed model. The scripts export it as OPENAI_BASE_URL; the OSWorld script also maps the same value to Anthropic-style variables for agent compatibility.

If model API access needs a local proxy, set LAN_PROXY before launching:

LAN_PROXY="http://127.0.0.1:<port>" bash scripts/test_osworld_std.sh

Do not commit real model URLs, API keys, VM paths, account credentials, or private proxy credentials.