Skip to content

TROUBLESHOOTING.md

Latest commit

 

History

History
128 lines (81 loc) · 4.38 KB

TROUBLESHOOTING.md

File metadata and controls

128 lines (81 loc) · 4.38 KB

Troubleshooting Passmark

This guide helps you diagnose and fix the common problems contributors hit when running Passmark locally or in CI. If the steps below don't help, open an issue and include logs and the commands you ran.

Quick checklist

  • Node.js >= 18
  • Install dependencies: pnpm install (or npm install)
  • Install Playwright browsers: npx playwright install
  • Redis available at REDIS_URL or via configure({ redis: { url } }) (see .env.example)
  • Required AI keys set when using direct providers: ANTHROPIC_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY
  • If using the Vercel AI Gateway, set AI_GATEWAY_API_KEY
  • If using CUA mode (configure({ ai: { mode: "cua" } })), set OPENAI_API_KEY and use gateway: "none"

Refer to .env.example for the full list of environment variables.

Common problems

1) "vitest: command not found" or tests fail immediately

Cause: dependencies (devDependencies) are not installed.

Fix:

pnpm install
pnpm test

If you don't have pnpm, install it or use npm install and npm test.

2) Missing AI API keys / cryptic runtime errors

Cause: The library attempts to call a model provider but API keys are not set.

Fix:

  • For direct provider usage set:
export ANTHROPIC_API_KEY=sk-...
export GOOGLE_GENERATIVE_AI_API_KEY=AIza...
  • Or use the Vercel AI Gateway by calling configure({ ai: { gateway: 'vercel' } }) and setting AI_GATEWAY_API_KEY:
export AI_GATEWAY_API_KEY=your_gateway_key

The code now raises clear, actionable errors if these variables are missing — follow the message to pick the right approach for your setup.

3) Playwright browsers missing or tests that use the browser fail

Fix:

npx playwright install

On CI, install browsers in your pipeline or use the --with-deps option as needed for Linux.

4) Redis connection failures

Cause: Step caching and some runtime features rely on Redis.

Fix:

# start a local Redis server (if installed)
redis-server &

# or run Redis in Docker
docker run --rm -p 6379:6379 redis

Set REDIS_URL in your environment (see .env.example), or pass it via configure({ redis: { url } }), so Passmark can connect.

If you intentionally don't want Redis, the code logs a warning and continues without caching.

5) Cached steps act stale or cause flaky runs

Cause: Cached step data can become invalid when the page changes.

Fix:

  • Run with cache bypass for debugging: pass bypassCache: true when calling runSteps.
  • Clear a specific cached step from Redis (use a Redis client) or flush the database during local troubleshooting.

6) Timeouts and flaky AI calls

Fix:

  • Increase timeouts or retry values where appropriate (see src/constants.ts).
  • Make sure your network to the AI provider is reliable and your API quota isn't exhausted.

7) CUA mode errors

mode: "cua" calls OpenAI's Responses API directly with the built-in computer tool.

  • "CUA mode requires gateway: 'none'" — CUA doesn't work through Vercel / OpenRouter / Cloudflare gateways (the Responses-API computer tool is only available on direct OpenAI access). Use configure({ ai: { mode: "cua", gateway: "none" } }).
  • "OPENAI_API_KEY isn't set" — add OPENAI_API_KEY to your environment / .env.
  • Generic 400 with param: null in the error body — your OpenAI API key likely doesn't have access to the CUA model or the built-in computer tool on the Responses API. Verify access at https://platform.openai.com/settings/organization/limits.
  • "Tool 'computer_use_preview' is not supported with gpt-5.5" — you're on an old build. In the current API, gpt-5.5 uses the new simpler tool shape { type: "computer" }, not the legacy computer_use_preview. Rebuild from main.

8) Enable debug logs

Set the PASSMARK_LOG_LEVEL environment variable to debug to get more information from the logger:

export PASSMARK_LOG_LEVEL=debug
pnpm test

When to open an issue

If you've tried the steps above and still can't get past a problem, open an issue and include:

  • What you ran (commands)
  • Relevant environment variables (mask keys)
  • Full error output and stack trace
  • A short description of your environment (OS, Node version, Playwright/Redis running or not)

Helpful files

  • .env.example — suggested env vars
  • src/models.ts — model resolution and API key handling
  • src/config.ts — how to call configure()