Testing

Babylon Lite uses four categories of automated tests, all orchestrated by Playwright and/or Vitest. An Azure Pipelines CI pipeline runs five parallel jobs on every PR targeting master.

Quick Reference

Command	What it runs
`pnpm test`	Build bundles → parity tests (local)
`pnpm test:parity`	Parity pixel-diff tests (local Chrome)
`pnpm test:parity-cloud`	Parity tests on BrowserStack (macOS Chrome, real WebGPU)
`pnpm test:perf`	Performance regression tests (local)
`pnpm test:perf-cloud`	Performance regression on BrowserStack
`pnpm test:bundle-size`	Bundle size ceiling checks
`pnpm test:bundle-delta`	Bundle size delta vs committed baseline
`pnpm test:all`	Parity + perf tests (local)
`pnpm test:watch`	Vitest in watch mode (unit tests)
`pnpm lint`	ESLint + TypeScript type-check

1. Unit Tests

Runner: Vitest
Location: tests/lite/unit/
Config: vitest.config.ts

Standard unit tests for core logic (shader composer, shader integration, etc.).

pnpm test:watch        # interactive
pnpm exec vitest run   # single run

2. Plumbing Tests

Runner: Playwright
Location: tests/lite/plumbing/

Browser-based integration tests that exercise engine lifecycle:

dispose.spec.ts — resource cleanup
material-swap.spec.ts — hot material replacement
memory-leak.spec.ts — allocation tracking
picking.spec.ts — GPU picking

pnpm exec playwright test tests/lite/plumbing/

3. Parity Tests (Pixel Comparison)

Runner: Playwright
Location: tests/lite/parity/scenes/ (25 scene spec files)
Configs:

Local: playwright.config.ts
Cloud: config/playwright.parity-cloud.config.ts

Compares screenshots of Babylon Lite rendering against golden reference images (BJS screenshots stored in reference/lite/). Uses Mean Absolute Difference (MAD) as the error metric; thresholds are defined per-scene in scene-config.json.

How it works

Opens the Lite bundle page (bundle-scene{N}.html)
Waits for canvas[data-ready="true"]
Takes a screenshot
Compares pixel-by-pixel against the golden reference
Asserts MAD ≤ scene threshold

Running locally

pnpm build:bundle-scenes
pnpm test:parity

Running on BrowserStack

Requires BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY (set in .env.local or as environment variables). Azure Pipelines gets these from the BabylonJS-BrowserStack variable group.

The cloud parity config connects to remote Chrome directly over CDP (wss://cdp.browserstack.com/playwright) — it does not use browserstack-node-sdk. Each Playwright worker is its own BrowserStack session, so specs shard across CIWORKERS parallel cloud browsers. The local Vite dev server is exposed to the remote browser through a BrowserStack Local tunnel started by the config's globalSetup (config/browserstack-local-tunnel.ts).

pnpm build:bundle-scenes

# One session (bare invocation never over-claims capacity):
pnpm test:parity-cloud

# Shard across up to N sessions (falls back to fewer when the plan is busy):
BSTACK_SESSIONS_REQUIRED=2 bash scripts/browserstack-wait.sh pnpm test:parity-cloud

Golden References

Golden images are committed in reference/lite/ and compared against Lite renders. captureGolden() skips BJS page capture when the golden file already exists on disk, which significantly speeds up test runs.

To force recapture of all golden references (e.g., after a Babylon.js update):

RECAPTURE_GOLDEN=true pnpm test:parity

Timeouts

Canvas-ready timeouts are set per-scene based on model complexity:

Scenes	Timeout
Most scenes	60 s
Hill Valley, KTX	90 s
Sponza	120 s

These higher values account for model downloads through the BrowserStack tunnel.

4. Performance Regression Tests

Runner: Playwright
Location: tests/lite/perf/perf-regression.spec.ts
Configs:

Local: playwright.perf.config.ts
Cloud: config/playwright.perf-cloud.config.ts

Measures CPU + GPU frame time by intercepting the engine's RAF-based render loop at runtime, then compares current Lite bundles against a baseline built from the previous release.

How it works

Runtime injection via page.addInitScript() — no scene modifications needed:
- Monkey-patches requestAnimationFrame to capture the render callback
- Monkey-patches GPUQueue.prototype.submit to capture the GPU queue
- Exposes window.__perfStop() to halt the RAF loop
- Exposes window.__perfRender() to call render + await queue.onSubmittedWorkDone()
Single-page measurement — all runs happen on one page load (one model download) to eliminate network variance:
- Each run: warmup frames → measured frames
- Measured frames use performance.now() around __perfRender() for true CPU+GPU cost
- Trimmed mean (drops top/bottom 10%) per run
- Median across all runs = final result
Assertion — only the trimmed mean average is asserted (p95 is logged but not asserted, as it's too noisy at sub-ms frame times)

Environment Variables

Variable	Default	Description
`PERF_REGRESSION_PCT`	`5`	Maximum allowed regression % (trimmed mean)
`PERF_FRAMES`	`300`	Measured frames per run
`PERF_RUNS`	`5`	Number of runs per version (takes median)
`PERF_WARMUP`	`60`	Warmup frames before each measurement run
`PERF_SCENES`	all	Comma-separated scene IDs to test (e.g., `1,5,9`)

Prerequisites

pnpm build:bundle-scenes       # build current bundles
pnpm build:perf-baseline        # build baseline from last release tag

The baseline script (scripts/build-perf-baseline.ts) uses a git worktree to check out the last v* release tag (or origin/master if no tags exist), builds its bundles, and copies them to lab/public/bundle-baseline/.

Running locally

pnpm build:bundle-scenes
pnpm build:perf-baseline
pnpm test:perf

Running on BrowserStack

pnpm build:bundle-scenes
pnpm build:perf-baseline
pnpm test:perf-cloud

Tuning for stability

If tests are flaky on noisy VMs, increase warmup and frame count:

PERF_WARMUP=120 PERF_FRAMES=500 pnpm test:perf-cloud

5. Bundle Size Checks

Runner: Playwright
Location: tests/lite/parity/bundle-size.spec.ts

Each scene bundle must stay under maxRawKB defined in scene-config.json (gzip size is shown for reference but not enforced).

pnpm build:bundle-scenes
pnpm test:bundle-size

BrowserStack Configuration

Two jobs use BrowserStack differently:

Job	How it connects	Config
Parity (Cloud)	Direct CDP (no SDK), sharded across sessions	`config/playwright.parity-cloud.config.ts`
Perf Regression	`browserstack-node-sdk` (SDK-managed tunnel)	`config/browserstack.yml`

Parity (CDP)

Setting	Value
Platform	macOS Sonoma
Browser	Chrome latest
Parallel sessions	Up to `BSTACK_SESSIONS_REQUIRED` (CI default 2)
Local tunnel	`browserstack-local`, started by config `globalSetup`

scripts/browserstack-wait.sh polls the BrowserStack plan, grabs up to the requested number of sessions (falling back to fewer when busy), and exports CIWORKERS so Playwright shards specs across exactly that many cloud browsers.

Perf (SDK)

Config file: config/browserstack.yml

Setting	Value
Platform	macOS Sonoma
Browser	Chrome latest
Parallel sessions	1
Local tunnel	Enabled (tests hit `localhost:5174`)

Credentials are read from environment variables:

BROWSERSTACK_USERNAME
BROWSERSTACK_ACCESS_KEY

For local development, add these to .env.local (git-ignored).

Azure Pipelines CI

Config: azure-pipelines.yml
Trigger: PRs targeting master

Five parallel jobs:

Job	What it does
Unit Tests	Vitest unit tests + Playwright plumbing tests
Bundle Size	Ceiling checks + delta vs baseline
Perf Regression	Current vs baseline on BrowserStack (macOS Chrome)
Parity (Cloud)	Pixel-diff on BrowserStack (macOS Chrome, real WebGPU)
Lint	ESLint + TypeScript `--noEmit` type-check

Required Pipeline Variable Groups

Azure Pipelines uses BabylonJS-BrowserStack for shared BrowserStack credentials:

BROWSERSTACK_USERNAME
BROWSERSTACK_ACCESS_KEY

It uses BabylonJS-Deployment for deployment server credentials used when uploading failed Playwright HTML reports:

DEPLOYMENT_SERVER
DEPLOY_TOKEN

Required Report Upload Variables

The failed-test report upload template also expects these pipeline variables:

DEPLOY_ENDPOINT_UPLOAD
STORAGE_ACCOUNT
SERVE_DOMAIN

Optional Pipeline Variables

PERF_REGRESSION_PCT — override regression threshold
PERF_FRAMES — override measured frames per run
PERF_RUNS — override number of runs per version
PERF_WARMUP — override warmup frames per run
BUNDLE_DELTA_PCT — override bundle delta threshold

Test Reporting

Both cloud test suites (perf and parity) produce:

JUnit XML — consumed by Azure DevOps PublishTestResults@2 and displayed in the pipeline's Tests tab with pass/fail counts, durations, and error messages
HTML report — interactive Playwright report with error details, screenshots, and traces

Report locations after a run:

Suite	JUnit XML	HTML Report
Parity	`test-results/parity-junit.xml`	`test-results/parity-report/index.html`
Perf	`test-results/perf-junit.xml`	`test-results/perf-report/index.html`

To view the HTML report locally:

pnpm exec playwright show-report test-results/parity-report
pnpm exec playwright show-report test-results/perf-report

In CI, test artifacts (including the HTML report) are uploaded as pipeline artifacts on every run and can be downloaded from the build summary.

Scene Configuration

All 25 test scenes are defined in scene-config.json at the repo root. Each entry specifies:

{
    "id": 1,
    "slug": "boombox",
    "name": "BoomBox",
    "maxMad": 1.5,
    "maxRegionMad": 3.0,
    "maxRawKB": 200
}

maxMad — parity MAD threshold (whole image)
maxRegionMad — parity MAD threshold (focus region, if defined)
maxRawKB — bundle raw size ceiling (gzip is informational only)

Environment Variables Reference

Variable	Scope	Default	Description
`PERF_REGRESSION_PCT`	Perf	`5`	Max allowed regression %
`PERF_FRAMES`	Perf	`300`	Measured frames per run
`PERF_RUNS`	Perf	`5`	Runs per version (takes median)
`PERF_WARMUP`	Perf	`60`	Warmup frames before each run
`PERF_SCENES`	Perf	all	Comma-separated scene IDs
`BUNDLE_DELTA_PCT`	Bundle	—	Max allowed bundle size growth %
`RECAPTURE_GOLDEN`	Parity	—	Set to `true` to force golden recapture
`BROWSERSTACK_USERNAME`	Cloud	—	BrowserStack credentials
`BROWSERSTACK_ACCESS_KEY`	Cloud	—	BrowserStack credentials

Navigation Menu

Uh oh!

Testing

Quick Reference

1. Unit Tests

2. Plumbing Tests

3. Parity Tests (Pixel Comparison)

How it works

Running locally

Running on BrowserStack

Golden References

Timeouts

4. Performance Regression Tests

How it works

Environment Variables

Prerequisites

Running locally

Running on BrowserStack

Tuning for stability

5. Bundle Size Checks

BrowserStack Configuration

Parity (CDP)

Perf (SDK)

Azure Pipelines CI

Required Pipeline Variable Groups

Required Report Upload Variables

Optional Pipeline Variables

Test Reporting

Scene Configuration

Environment Variables Reference

Uh oh!

FilesExpand file tree

TESTING.md

Latest commit

History

TESTING.md

File metadata and controls

Testing

Quick Reference

1. Unit Tests

2. Plumbing Tests

3. Parity Tests (Pixel Comparison)

How it works

Running locally

Running on BrowserStack

Golden References

Timeouts

4. Performance Regression Tests

How it works

Environment Variables

Prerequisites

Running locally

Running on BrowserStack

Tuning for stability

5. Bundle Size Checks

BrowserStack Configuration

Parity (CDP)

Perf (SDK)

Azure Pipelines CI

Required Pipeline Variable Groups

Required Report Upload Variables

Optional Pipeline Variables

Test Reporting

Scene Configuration

Environment Variables Reference