Skip to content

Commit bedaeb8

Browse files
marcel-rbroclaude
marcel-rbro
and
claude
authored
docs: docs-tests harness for Console UI drift detection (#2669)
Part of #2671 Adds `docs-tests/` — a docs-as-tests harness that catches when the platform docs claim something the Console UI no longer does (a renamed tab, moved route, relabeled button). **Model:** an adjustable list of docs pages → AI-extracted assertions (run locally, human-reviewed) → a committed baseline → evaluated against Console staging with Playwright. Failures point back to `source_file:line`. Includes the scheduled GitHub Action (`docs-ui-tests.yaml`, weekly + manual dispatch) that runs the evaluation against staging and files a drift issue on failure. Extraction never runs in CI, so no API key is needed there; the staging target and login come from repo secrets at run time — nothing is committed. Coverage starts with the Console section (routes + landing-page elements); widening is a follow-up. See `docs-tests/README.md` for the model and known gaps. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent c9b4b1f commit bedaeb8

24 files changed

Lines changed: 1671 additions & 0 deletions

‎.github/workflows/docs-ui-tests.yaml‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
name: Docs UI drift tests
2+
3+
# Verifies Console-UI claims extracted from the platform docs against Console
4+
# staging with Playwright. Assertions are a committed, human-reviewed baseline
5+
# (docs-tests/assertions/); this job only evaluates them — extraction runs
6+
# locally, never in CI.
7+
#
8+
# Credentials come from repo Actions secrets (CONSOLE_STAGING_*). The seeded
9+
# staging user is low-privilege and has no 2FA; login is scripted fresh each run
10+
# and the session is held in memory (no auth file).
11+
12+
on:
13+
schedule:
14+
- cron: '0 6 * * 1' # Every Monday 06:00 UTC
15+
workflow_dispatch: # Manual on-demand run
16+
17+
defaults:
18+
run:
19+
working-directory: docs-tests
20+
21+
jobs:
22+
ui-drift:
23+
runs-on: ubuntu-latest
24+
permissions:
25+
contents: read
26+
issues: write # the failure step files a drift issue for maintainers
27+
steps:
28+
- uses: actions/checkout@v7
29+
30+
- name: Use Node.js 24
31+
uses: actions/setup-node@v7
32+
with:
33+
node-version: 24
34+
35+
- name: Install pnpm and docs-tests dependencies
36+
# Shared Apify action: sets up pnpm (version from
37+
# docs-tests/package.json "packageManager"), caches the pnpm
38+
# store, and installs with --frozen-lockfile. working-directory
39+
# is docs-tests because it's a standalone project with its own
40+
# lockfile, not part of the repo-root pnpm workspace.
41+
uses: apify/actions/pnpm-install@v1.4.0
42+
with:
43+
working-directory: docs-tests
44+
45+
- name: Install Playwright browser
46+
run: pnpm exec playwright install --with-deps chromium
47+
48+
- name: Run docs UI drift tests
49+
env:
50+
CI: 'true'
51+
CONSOLE_STAGING_URL: ${{ secrets.CONSOLE_STAGING_URL }}
52+
CONSOLE_STAGING_USER_EMAIL: ${{ secrets.CONSOLE_STAGING_USER_EMAIL }}
53+
CONSOLE_STAGING_USER_PASSWORD: ${{ secrets.CONSOLE_STAGING_USER_PASSWORD }}
54+
run: pnpm test
55+
56+
- name: Upload Playwright report
57+
if: always()
58+
uses: actions/upload-artifact@v4
59+
with:
60+
name: playwright-report
61+
path: docs-tests/playwright-report/
62+
retention-days: 14
63+
64+
- name: Upload drift report (issues.json)
65+
if: always()
66+
uses: actions/upload-artifact@v4
67+
with:
68+
name: drift-report
69+
path: docs-tests/output/
70+
retention-days: 14
71+
72+
# On any failure (drift found, or the job broke before tests ran),
73+
# open a GitHub issue for the docs maintainers. To avoid a new issue
74+
# every week, reuse the open `docs-ui-drift` issue if one exists and
75+
# comment on it instead. The body is kept public-safe: it links to
76+
# the run (where the report/artifacts live) and lists only assertion
77+
# IDs + their doc source — never the staging URL or credentials.
78+
- name: File a drift issue for maintainers
79+
if: failure()
80+
env:
81+
GH_TOKEN: ${{ github.token }}
82+
REPO: ${{ github.repository }}
83+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
84+
run: |
85+
set -euo pipefail
86+
REPORT=output/issues.json
87+
BODY="$(mktemp)"
88+
89+
if [[ -f "$REPORT" ]]; then
90+
PASSED=$(jq -r '.summary.passed' "$REPORT")
91+
FAILED=$(jq -r '.summary.failed' "$REPORT")
92+
SKIPPED=$(jq -r '.summary.skipped' "$REPORT")
93+
TOTAL=$(jq -r '.summary.total' "$REPORT")
94+
{
95+
echo "The scheduled docs-UI drift check found **${FAILED} failing assertion(s)** (${PASSED}/${TOTAL} passed, ${SKIPPED} skipped)."
96+
echo ""
97+
echo "Filed automatically by the \`Docs UI drift tests\` workflow. Logs, the Playwright report, and \`issues.json\` are on the [workflow run](${RUN_URL})."
98+
echo ""
99+
echo "### Failing assertions"
100+
echo ""
101+
echo "| Assertion | Kind | Target | Doc source |"
102+
echo "| --- | --- | --- | --- |"
103+
jq -r '.issues[] | "| `\(.id)` | \(.kind) | \(.target) | `\(.source_file):\(.source_line)` |"' "$REPORT"
104+
echo ""
105+
echo "Each failure is either genuine doc drift (the Console changed, the docs did not) or a harness limitation (selector needs a click first, or a logged-out/session-gated page). Triage against the Playwright report."
106+
} > "$BODY"
107+
else
108+
{
109+
echo "The \`Docs UI drift tests\` workflow failed before producing a drift report (likely a setup or login error)."
110+
echo ""
111+
echo "See the [workflow run](${RUN_URL}) for logs."
112+
} > "$BODY"
113+
fi
114+
115+
# Ensure the dedup label exists (upsert), then reuse an open
116+
# issue if present, otherwise create a new one. No assignee;
117+
# labels are docs-ui-drift (dedup marker) + t-docs.
118+
gh label create docs-ui-drift --repo "$REPO" \
119+
--color BFD4F2 --description "Automated Console UI drift report" --force || true
120+
121+
EXISTING=$(gh issue list --repo "$REPO" --state open --label docs-ui-drift \
122+
--limit 1 --json number --jq '.[0].number // empty')
123+
124+
if [[ -n "$EXISTING" ]]; then
125+
gh issue comment "$EXISTING" --repo "$REPO" --body-file "$BODY"
126+
echo "Commented on existing drift issue #${EXISTING}"
127+
else
128+
gh issue create --repo "$REPO" \
129+
--title "Docs UI drift detected by scheduled tests" \
130+
--body-file "$BODY" \
131+
--label docs-ui-drift --label t-docs
132+
fi

‎.markdownlintignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,4 @@ CLAUDE.md
77
GEMINI.md
88
standards
99
sources/_partials/*
10+
docs-tests

‎.vale.ini‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,3 +42,10 @@ BasedOnStyles =
4242

4343
[.agents/**/*.md]
4444
BasedOnStyles =
45+
46+
[docs-tests/**/*.md]
47+
# Ignore all Vale checks in docs-tests (internal tooling, not published docs).
48+
# BasedOnStyles = clears style-based rules; Microsoft.Dashes is toggled on
49+
# explicitly above, so it must be turned off explicitly here too.
50+
BasedOnStyles =
51+
Microsoft.Dashes = NO

‎docs-tests/.env.example‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Staging Console base URL (required).
2+
CONSOLE_STAGING_URL=https://console.staging-example.apify.dev
3+
4+
# Seeded staging user (required to run the tests). Used by tests/auth.fixture.ts
5+
# to log in fresh each run. Locally these live here; in CI they come from GitHub
6+
# Secrets, never from a committed file. The seeded account has no 2FA.
7+
CONSOLE_STAGING_USER_EMAIL=
8+
CONSOLE_STAGING_USER_PASSWORD=

‎docs-tests/.gitignore‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Never commit a logged-in session or local secrets to this public repo.
2+
node_modules/
3+
.env
4+
auth.json
5+
playwright-report/
6+
test-results/
7+
.DS_Store
8+
9+
# Runtime outputs. The stored assertion baseline (assertions/) IS committed;
10+
# only the per-run report lives here.
11+
output/

‎docs-tests/README.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# docs-tests
2+
3+
Docs-as-tests for the Apify Console. Every UI claim in the platform docs — a
4+
route resolves, a tab is named X, a button exists on page Y — is a testable
5+
assertion. This package extracts those assertions from the docs with an LLM,
6+
stores them as a reviewed baseline, and verifies them against Console staging
7+
with Playwright, so documentation drift is caught automatically.
8+
9+
```
10+
pages.json ──extract──▶ assertions/*.json ──Playwright──▶ output/issues.json
11+
(page list) (claude -p) (committed baseline) (vs staging) (drift report)
12+
```
13+
14+
## Model
15+
16+
1. **`pages.json`** is an adjustable list of documentation pages (real source
17+
files under `sources/platform/…`) to cover.
18+
2. **`scripts/extract.sh`** feeds one page to `claude -p` with a strict JSON
19+
schema and writes the result to `assertions/<slug>.json`.
20+
`scripts/extract-all.sh` does the whole manifest.
21+
3. **`assertions/`** is the *stored, reviewed baseline* — committed to the repo.
22+
Regenerate it with the LLM whenever docs change, review the diff, commit.
23+
The assertion set is owned by humans even though a model drafts it.
24+
4. **`tests/from-doc.spec.ts`** reads every stored assertion and emits one
25+
Playwright `test()` per assertion, run against `$CONSOLE_STAGING_URL`.
26+
5. Failures point back to `source_file:line` so the offending prose is one click
27+
away, and land in `output/issues.json` for downstream triage.
28+
29+
The Notion plan *"AI-based testing for docs"* (its Part 1 routes + Part 2
30+
elements) is the inspiration for which pages and claims to cover — not a fixed
31+
transcription. The authoritative set is whatever the manifest + extractor
32+
produce and a human commits.
33+
34+
## Assertion kinds
35+
36+
| Kind | Checks |
37+
| ---------------- | ------------------------------------------------------------------ |
38+
| `route` | Documented path is reachable (HTTP < 400) |
39+
| `element_tab` | Documented tab label exists on the page named in `at` |
40+
| `element_button` | Documented button label exists on the page named in `at` |
41+
| `element_text` | Documented heading/label/field is visible on the page named in `at` |
42+
43+
## One-time setup
44+
45+
```bash
46+
pnpm install
47+
pnpm exec playwright install chromium
48+
cp .env.example .env # fill in CONSOLE_STAGING_URL + seeded-user email/password
49+
```
50+
51+
## Generate / refresh the assertion baseline
52+
53+
```bash
54+
# Every page in pages.json:
55+
pnpm extract:all
56+
57+
# Or a single page:
58+
pnpm extract sources/platform/console/settings.md
59+
```
60+
61+
Review the diff in `assertions/`, then commit. This is the step a human owns.
62+
63+
## Run the tests
64+
65+
```bash
66+
pnpm test # evaluate all stored assertions against staging
67+
pnpm issues # machine-readable, action-oriented failures
68+
pnpm report # HTML report (failures include screenshots, video, trace)
69+
```
70+
71+
Authentication is automatic: a worker-scoped fixture (`tests/auth.fixture.ts`)
72+
logs in once per run with `CONSOLE_STAGING_USER_EMAIL` / `_PASSWORD` and keeps
73+
the session in memory. **No `auth.json` is written or read** — nothing has to
74+
pre-exist, so it behaves identically locally and in CI (where the credentials
75+
come from GitHub Secrets). The seeded staging user has no 2FA.
76+
77+
`pnpm test` always writes `output/issues.json` — a summary plus one entry per
78+
failing assertion, sorted by `source_line`, each carrying `source_file:line`,
79+
the offending `source_quote`, and a one-line error. For `element_*` failures it
80+
also captures the live page's same-kind labels (`observed_candidates`) and, when
81+
unambiguous, a `suggested_target`, so a downstream LLM can propose a doc fix
82+
without re-running the browser.
83+
84+
## Adjusting coverage
85+
86+
Edit `pages.json` and re-run `pnpm extract:all`. Add a page → it gets an
87+
assertion set; remove one → delete its `assertions/<slug>.json`.
88+
89+
## CI
90+
91+
`.github/workflows/docs-ui-tests.yaml` runs the evaluation on a weekly schedule
92+
and on manual dispatch: it installs Playwright, logs in with the
93+
`CONSOLE_STAGING_*` repo secrets, evaluates the committed baseline against
94+
staging, uploads the report, and files a `docs-ui-drift` issue when an assertion
95+
fails. Extraction never runs in CI — the reviewed baseline is the only input.
96+
97+
## Known gaps (deferred)
98+
99+
- **Coverage is a starting slice.** `pages.json` covers the Console section
100+
(index, settings, billing, store) — routes and landing-page elements. Widening
101+
to more pages is a follow-up: add to `pages.json`, re-extract, review, commit.
102+
- **Detail-page fixtures.** Assertions about Actor-detail, Schedule-detail, etc.
103+
need a known fixture to navigate to. The runner currently *skips* element
104+
assertions with no `at` route — surfacing the gap without false negatives.
105+
Requires the seeded-user fixtures (1 Actor, 1 task, 1 schedule, named storages,
106+
1 webhook, 1 completed run) from the Notion plan.
107+
- **Left-nav group check.** The documented global nav items (Dashboard/Store/
108+
Actors/…) are a Console-wide check, not a per-page claim — not modeled yet.
109+
- **Multi-step flows.** The schema only supports atomic claims (one
110+
navigate-then-check). "Click X, then Y, then Z" sequences are not modeled.
111+
- **Surface mismatch.** Some docs describe the public marketing site (e.g.
112+
`apify.com/store`) while the harness tests the Console (`/store`); the two can
113+
use different labels, so those element claims may not map. Check the surface a
114+
page actually documents before adding element assertions for it.
115+
116+
## Files
117+
118+
```
119+
docs-tests/
120+
├── pages.json # adjustable list of docs pages to cover
121+
├── assertions/ # committed baseline, one JSON per page (generated)
122+
├── prompts/
123+
│ ├── extract-system.md # system prompt + known-routes table
124+
│ └── assertion-schema.json # JSON Schema for the extractor output
125+
├── scripts/
126+
│ ├── extract.sh # one page → assertions/<slug>.json
127+
│ └── extract-all.sh # whole manifest
128+
├── reporters/issues-reporter.ts # custom Playwright reporter → output/issues.json
129+
├── tests/
130+
│ ├── auth.fixture.ts # worker-scoped login from env creds (in-memory session)
131+
│ ├── from-doc.spec.ts # reads assertions/*.json, emits tests
132+
│ └── similarity.ts # suggest-replacement helper for failures
133+
├── playwright.config.ts
134+
└── .env # CONSOLE_STAGING_URL (gitignored)
135+
```

0 commit comments

Comments
 (0)