Skip to content

BUILD-AND-DEPLOY.md

Latest commit

 

History

History
1298 lines (938 loc) · 63.3 KB

BUILD-AND-DEPLOY.md

File metadata and controls

1298 lines (938 loc) · 63.3 KB
Raw

BUILD-AND-DEPLOY.md — pulumi/registry

This document describes the build, test, and deployment system for the pulumi/registry repository.

Table of Contents

  1. Overview
  2. Architecture Overview
  3. Local Development
  4. Build System
  5. GitHub Actions Workflows
  6. Deployment Infrastructure
  7. Registry Publication (push-registry.py)
  8. Testing Strategy
  9. Environment & Secret Management
  10. Scheduled & Maintenance Tasks
  11. Troubleshooting
  12. Pulumi Cloud Service Integration

Overview

Quick Reference: Common Commands

Task Command
Install all dependencies make ensure
Build theme assets (CSS/JS) make build-assets
Local development server make serve
Full local dev (Hugo + asset watch) make serve-all
Generate API docs for one provider make api-docs/<package>
Run Go tests make test
Run all linters make lint
Run Markdown linter make lint-markdown
Run Go linter make lint-go
Run provider API docs tests make test_provider_api_docs
Run browser tests make run-browser-tests
Check links make check_links
Full CI push (production) make ci_push
Full CI pull request make ci-pull-request
Bucket cleanup make ci_bucket_cleanup
Clean build artifacts make clean

Key Concepts

  • Hugo: Static site generator that produces the final HTML from templates and content files.
  • Pulumi ESC (Environments, Secrets, and Configuration): Used for OIDC-based secret exchange; no long-lived secrets are stored directly in GitHub.
  • resourcedocsgen: A Go tool (in tools/resourcedocsgen/) that generates provider API documentation from Pulumi provider schemas.
  • S3 origin bucket model: Each build produces its own uniquely-named S3 bucket. A Pulumi IaC program reads a metadata file to determine which bucket to point CloudFront at. This means each PR commit gets its own preview URL, and production deploys atomically swap the CloudFront origin.
  • mktutorial: A Go tool (in tools/mktutorial/) that generates how-to guide content from the pulumi/examples repository.

Prerequisites

The canonical tool versions are tracked in mise.toml:

Tool Version
Go 1.26
Node.js 20
Yarn 1.22.22
Hugo 0.157 (extended)
golangci-lint 2.1.6
yq latest

Install and manage all tools via mise:

mise trust && mise install

Architecture Overview

                   ┌─────────────────────────────────────┐
                   │        pulumi/registry repo          │
                   │                                      │
                   │  themes/default/data/registry/       │
                   │    packages/*.yaml  ─────────────┐   │
                   │                                  ▼   │
                   │  tools/resourcedocsgen/ ──► generates API docs
                   │                                  │   │
                   │  themes/default/content/         │   │
                   │    registry/packages/ ◄──────────┤   │
                   │                                  │   │
                   │  llm-docs-out/registry/packages/ │   │
                   │    (LLM-friendly JSON) ◄─────────┘   │
                   │                                      │
                   │  Hugo build ◄── (reads content/)     │
                   │       │                              │
                   │       ▼                              │
                   │  public/  (built site)               │
                   └─────────────────────────────────────┘
                              │
                              │  scripts/ci/sync.sh
                              ▼
              ┌─────────────────────────────┐
              │  AWS S3 Origin Bucket        │
              │  registry-{env}-origin-      │
              │  {build-id}                  │
              └──────────────┬──────────────┘
                             │
                             │  Pulumi IaC (infrastructure/)
                             │  reads origin-bucket-metadata.json
                             ▼
              ┌─────────────────────────────┐
              │  AWS CloudFront Distribution │
              │  (per environment)           │
              └──────────────┬──────────────┘
                             │
                             ▼
                   pulumi.com/registry

Two Environments

Environment AWS Account ID Pulumi Stack Purpose
testing 571684982431 pulumi/registry/testing PR previews and test deployments
production 388588623842 pulumi/registry/production Live site at pulumi.com/registry

Component Roles

Component Role
Hugo 0.157 (extended) Static site generation from templates and content
resourcedocsgen Generates provider API reference docs from Pulumi schemas
mktutorial Generates how-to guides from pulumi/examples
Pulumi IaC Manages AWS resources; reads metadata file to update CloudFront origin
Algolia Search index; updated as part of production deploys via scripts/search/main.js
AWS S3 Hosts built site content as a static website origin
AWS CloudFront CDN serving the site; origin pointed at S3 bucket by Pulumi IaC
AWS SSM Parameter Store Maps commit SHAs to their corresponding S3 origin buckets

Multi-Repo Touchpoints

  • pulumi/pulumi-* provider repos: Trigger publish-provider-update.yml via repository_dispatch when a new provider version is released.
  • pulumi/examples: Source of how-to guide content; pulled nightly by update-tutorials.yml.

Local Development

Tool Installation

# Trust mise config and install all declared tools
mise trust && mise install

First-Time Setup

# 1. Install Node.js dependencies (yarn packages, etc.)
make ensure

# 2. Build theme CSS and JavaScript assets
make build-assets

# 3. Start local dev server
make serve

make serve runs:

hugo serve --buildDrafts --buildFuture --ignoreVendorPaths="github.com/pulumi/registry/**/*"

REF_NOT_FOUND Hugo warnings are suppressed automatically.

Hot-Reload Development

To watch both Hugo content changes and theme asset changes simultaneously:

make serve-all

This uses concurrently to run the Hugo server and the Yarn asset watcher (yarn --cwd ./themes/default/theme run start) in parallel. If either process exits, both are killed.

To watch only theme assets (without serving Hugo):

make serve-assets

Generating API Docs for One Package

make api-docs/<package>

For example, to generate docs for the aws provider:

make api-docs/aws

This runs bin/resourcedocsgen docs registry for the named package, reading its YAML from themes/default/data/registry/packages/aws.yaml and writing generated content to content/registry/packages/aws/ (at the repository root — note this differs from the path used by scripts/ci/build.sh in CI, which writes to themes/default/content/registry/packages/).

The Makefile uses .SECONDEXPANSION and a .make/ sentinel directory so that make api-docs/aws is a no-op if neither the YAML file nor the existing content has changed since the last run.

Building a Single Provider Locally vs. Full Build

For a single provider:

make api-docs/<package>

For a full local build (all providers, all content):

make build

make build runs make build-assets, then scripts/apply-fixes.js, then Hugo.

Memory note: A full build requires significant RAM. The CI build sets NODE_OPTIONS=--max_old_space_size=8192 to give Node 8 GB of heap. For full local builds, consider setting this in your shell:

export NODE_OPTIONS="--max_old_space_size=8192"

32 GB+ of RAM is recommended for a complete build.

Build System

4.1 Makefile Targets

All targets are defined in the repository root Makefile.

Target Command / Description
clean Runs scripts/clean.sh
ensure Runs scripts/ensure.sh — installs Node/Go/Hugo deps
build build-assets → apply-fixes.js → hugo
build-assets ensure → yarn --cwd ./themes/default/theme run build
serve hugo serve --buildDrafts --buildFuture (suppress REF_NOT_FOUND)
serve-assets yarn --cwd ./themes/default/theme run start (asset watch)
serve-all Concurrent Hugo serve + asset watch
api-docs/<pkg> Build API docs for a single package via resourcedocsgen
bin/resourcedocsgen Compile resourcedocsgen from tools/resourcedocsgen/
bin/mktutorial Compile mktutorial from tools/mktutorial/
lint lint-go + lint-markdown + yarn run lint
lint-go lint-resourcedocsgen + lint-mktutorial
lint-resourcedocsgen golangci-lint run in tools/resourcedocsgen/
lint-mktutorial golangci-lint run in tools/mktutorial/
lint-markdown scripts/lint/lint-markdown.js
test go test ./... in tools/resourcedocsgen/
test_provider_api_docs ensure + build-assets + bin/resourcedocsgen → scripts/ci/run-provider-tests.sh
run-browser-tests ensure → scripts/run-api-docs-tests.sh
check_links ensure → yarn run check-links
ci-pull-request ensure → scripts/ci/pull-request.sh
ci-pull-request-closed scripts/ci/pull-request-closed.sh
ci-scheduled scripts/ci/scheduled.sh
ci_push ensure → scripts/ci/push.sh
ci_bucket_cleanup scripts/ci/bucket-cleanup.sh

4.2 Hugo Build

Hugo reads from the following key locations:

Path Role
config/ Hugo configuration (base URL, params, etc.)
themes/default/ The default Hugo theme
themes/default/content/ Content files (Markdown)
themes/default/data/ Data files (YAML)
themes/default/static/ Static assets
public/ Hugo output directory

Build order for a full CI build:

  1. make build-assets — compiles theme JS and CSS via yarn --cwd ./themes/default/theme run build
  2. resourcedocsgen docs registry — generates provider API docs into themes/default/content/registry/packages/
  3. node scripts/apply-fixes.js — post-processes Hugo content (see script for details)
  4. hugo --minify --buildFuture --templateMetrics -e <environment> — generates public/
  5. yarn run minify-css (i.e., node scripts/minify-css.js) — purges and minifies CSS in public/

The Hugo base URL differs between build modes:

  • preview: http://<origin-bucket-prefix>-<build-id>.s3-website.us-west-2.amazonaws.com
  • update (production): uses production base URL; Hugo env is set to production

4.3 resourcedocsgen Tool

Location: tools/resourcedocsgen/

Building:

# Via Makefile (recommended — uses helpmakego for dependency tracking)
make bin/resourcedocsgen

# Manual
go build -C tools/resourcedocsgen -o ../../bin ./...

The compiled binary is placed at bin/resourcedocsgen.

Subcommands:

Subcommand Purpose
docs registry Generate API docs for all (or one) provider packages
pkgversion Check the latest published version of a community provider on GitHub
metadata from-github Generate package YAML metadata from a GitHub-hosted provider repo
metadata from-urls Generate package YAML metadata from explicit schema/index URLs

Key flags for docs registry:

bin/resourcedocsgen docs registry \
    --baseDocsOutDir themes/default/content/registry/packages \
    --basePackageTreeJSONOutDir themes/default/static/registry/packages/navs \
    --baseSchemasOutDir themes/default/static/registry/packages \
    --baseLLMDocsOutDir ./llm-docs-out/registry/packages \
    [<package-name>]

Note: The paths above reflect scripts/ci/build.sh (CI runs). The Makefile's make api-docs/<pkg> target uses ./content/registry/packages (repo root, without the themes/default/ prefix) for --baseDocsOutDir, and similarly for the other --base* flags.

When <package-name> is omitted, all packages listed in themes/default/data/registry/packages/ are processed.

Input: YAML files at themes/default/data/registry/packages/*.yaml — each file describes one provider (name, version, repo URL, schema file path, publisher, etc.)

Output:

  • Generated docs at themes/default/content/registry/packages/<pkg>/api-docs/
  • Package navigation JSON at themes/default/static/registry/packages/navs/<pkg>.json
  • Schema JSON at themes/default/static/registry/packages/<pkg>.json
  • LLM docs JSON at llm-docs-out/registry/packages/<pkg>/api-docs/llm-docs.json (only when --baseLLMDocsOutDir is set)

The format of llm-docs.json is specified in docs/llm-markdown-spec.md.

4.4 mktutorial Tool

Location: tools/mktutorial/

Building:

make bin/mktutorial

Purpose: Generates how-to guide (tutorial) content from the pulumi/examples repository for the following clouds: aws-apigateway, aws, classic-azure (mapped to azure), azure (mapped to azure-native), gcp, and kubernetes.

Output: Content written to themes/default/content/registry/packages/<cloud>/how-to-guides/.

Versioned packages (aws-v6, azure-native-v2) are also cleaned of stale tutorials.

Used only in CI via scripts/ci/mktutorial.sh. Not used in local builds.

4.5 CI Build Script (scripts/ci/build.sh)

This is the master build script for CI runs. It accepts one argument: preview or update.

Steps executed by build.sh:

  1. Calls make build-assets (theme CSS/JS compilation).
  2. Computes a build_identifier (for preview: pr-<number>-<sha8>; for push: push-<sha8>).
  3. Sets asset bundle paths:
    • CSS_BUNDLE=static/css/styles.<id>.css
    • JS_BUNDLE=static/js/bundle.min.<id>.js
  4. Restores cached API docs from .cache/api-docs/ into the Hugo content/static trees (see section 4.7 for details).
  5. Runs make api-docs, which compiles resourcedocsgen and generates all provider API docs. The tool skips unchanged packages using sentinel files.
  6. Saves API docs output (including sentinel files) back to .cache/api-docs/ for the next run. Versioned packages (@-suffixed) are excluded — they have their own cache. LLM docs from llm-docs-out/ are also cached to .cache/api-docs/llm-docs/ and restored on the next run.
  7. Runs node ./scripts/apply-fixes.js.
  8. Runs Hugo with --minify --buildFuture --templateMetrics:
    • preview mode: sets HUGO_BASEURL to the S3 website URL and uses -e preview
    • update mode: uses -e production
  9. Runs yarn run minify-css to purge and minify CSS.

4.6 Versioned documentation

Location: scripts/generate-versioned-docs.sh

Blessed packages (first-party Pulumi providers listed in ci-mgmt) get versioned API docs for the last 3 major versions. URLs use the pkg@X.x format:

  • /registry/packages/aws/ - latest version (e.g., v7.x)
  • /registry/packages/aws@6.x/ - previous major version
  • /registry/packages/aws@5.x/ - older major version

How it works:

  1. make api-docs/<pkg> runs resourcedocsgen for the latest version, then calls generate-versioned-docs.sh
  2. The script fetches the blessed packages list from ci-mgmt
  3. For blessed packages, it discovers the last 3 major versions using registry-mirror-discover
  4. For each older major version:
    • Downloads and caches the schema (schemas are 50MB+)
    • Fetches version-specific _index.md from GitHub or falls back to latest
    • Generates API docs with resourcedocsgen
    • Creates versioned nav JSON (e.g., aws@6.x.json)
    • Generates metadata YAML in package_versions/ directory

Generated output:

Output Location
Versioned docs themes/default/content/registry/packages/<pkg>@<X>.x/api-docs/
Versioned nav JSON themes/default/static/registry/packages/navs/<pkg>@<X>.x.json
Version metadata themes/default/data/registry/package_versions/<pkg>@<X>.x.yaml

Dependencies:

  • registry-mirror-discover: built via go install from github.com/pulumi/registry-mirror-tools at a pinned commit. In CI, the binary is cached by GitHub Actions to avoid rebuilding on every run (see section 4.7).

Hugo template handling:

Templates parse versioned package names to extract the base name and version slug:

{{ $rawPackageName := index $directories 2 }}
{{ $isVersioned := strings.Contains $rawPackageName "@" }}
{{ $basePackageName := cond $isVersioned (index (split $rawPackageName "@") 0) $rawPackageName }}

The version selector dropdown appears on package pages when multiple versions exist, reading from $.Site.Data.registry.package_versions.

4.7 Build Caching

CI builds use multiple cache layers to avoid redundant work. All caches are stored in GitHub Actions cache and restored at the start of each build.

GitHub Actions–level caches

Cache Key Paths What it stores
Node/Yarn node-cache-Linux-x64-yarn-<yarn.lock hash> ~/.cache/yarn/v6 Yarn package cache
Go (resourcedocsgen) setup-go-...-<tools/resourcedocsgen/go.sum hash> GOMODCACHE, GOCACHE Go module and build cache
Go (mktutorial) setup-go-...-<tools/mktutorial/go.sum hash> GOMODCACHE, GOCACHE Go module and build cache
registry-mirror-tools binaries registry-mirror-tools-bins-<os>-<commit hash> bin/registry-mirror-discover, bin/registry-mirror-publish Pre-built binaries for test-ci-scripts.yml
Docs + schemas docs-cache-<run_id> (restore key: docs-cache-) .cache/schemas, .cache/versioned-docs, .cache/api-docs API docs output, versioned docs, provider schemas, LLM docs JSON
registry-mirror-discover registry-mirror-discover-<commit hash> bin/registry-mirror-discover Pre-built binary for versioned docs discovery

The docs cache uses restore-keys: docs-cache- so it falls back to the most recent previous run's cache when an exact match isn't found (the key includes run_id, so it's always unique).

Each Go job keys on the go.sum of the module it compiles, via cache-dependency-path. Keep it that way: one key per module, never one key listing both. setup-go restores on an exact primary-key match and exposes no restore-keys, so a single shared key means the first job to finish decides what every later job restores — and if that is a mktutorial check, the jobs building resourcedocsgen stay cold. Jobs that compile neither module set cache: false rather than falling back to the repo-root go.mod, which describes the Hugo theme module and never changes.

Incremental API docs generation

The resourcedocsgen tool skips unchanged packages using sentinel files. Each generated package directory contains a .generated file recording a cache key composed of:

  • SHA-256 of the package YAML metadata — changes when the package version or config is updated.
  • Go toolchain version — changes on Go upgrades.
  • Source hash — a SHA-256 of all .go, .tmpl, and go.sum files in tools/resourcedocsgen/, injected at build time via -ldflags. Changes when the doc generation logic or templates change.

On each run, resourcedocsgen compares the computed cache key against the sentinel. If they match and the expected output files (api-docs, nav JSON, schema JSON) all exist, the package is skipped. Otherwise it regenerates.

The scripts/ci/build.sh script manages the cache lifecycle:

  1. Restore: copies cached content from .cache/api-docs/ into the Hugo content/static trees before running resourcedocsgen.
  2. Generate: make api-docs runs resourcedocsgen, which skips fresh packages and regenerates stale ones.
  3. Save: copies the generated output (including updated sentinel files) back to .cache/api-docs/ for the next run.

LLM docs follow the same lifecycle: on restore, .cache/api-docs/llm-docs/<pkg>/api-docs/ is copied to llm-docs-out/registry/packages/<pkg>/api-docs/; on save, the reverse copy is performed. Only schema.json (not the entire directory tree) is cached per package in the schema layer, to avoid persisting stale LLM doc files from older builds. LLM docs are stored uncompressed in the cache; sync.sh gzip-compresses them in place immediately before uploading to S3 (with Content-Encoding: gzip).

Versioned docs cache

Versioned docs (older major versions of blessed packages) are cached separately in .cache/versioned-docs/, which has three subdirectories: content/, navs/, and metadata/. The generate-versioned-docs.sh script restores from this cache before processing and saves back to it afterward.

Each versioned package directory has a .generated sentinel file, but unlike the API docs cache, it stores only the schema URL (not a full composite key). If the schema URL in the sentinel matches the current version's schema URL, generation is skipped. This means versioned docs only regenerate when the schema URL changes (i.e., when a new version is published for that major version line).

Provider schemas themselves are cached separately in .cache/schemas/ (keyed by <package>-v<version>.json) since individual schemas can be 50MB+.

Cache invalidation triggers

Trigger What invalidates
Package YAML file changes That specific package's API docs regenerate
Go source in tools/resourcedocsgen/ changes All API docs regenerate (source hash changes)
Go toolchain upgrade All API docs regenerate
Schema URL changes for a versioned package That specific versioned package regenerates
registry-mirror-discover commit hash changes Binary is rebuilt and re-cached
yarn.lock changes Yarn cache miss, full yarn install

GitHub Actions Workflows

All workflow files live in .github/workflows/.

Workflow Summary

File Name Trigger
pull-request.yml Pull request PR to master
push.yml Build and deploy Push to master
testing-deploy.yml Build and deploy - test environment workflow_dispatch
pull-request-closed.yml Close pull request PR closed
check-go.yml Check Go workflow_call (reusable)
check-links.yml Scheduled jobs: Check links Every Monday 3:00 PM UTC
run-browser-tests.yml Scheduled jobs: Run browser tests Daily 2:00 PM UTC
generate-package-metadata.yml Check for Community Package Updates Daily 5:30 AM + 5:30 PM UTC + push to master touching package-list.json
community-package-check.yml Community package check PR touching community-packages/package-list.json
community-package-report.yml Community package report workflow_run after the check completes
community-package-check-command.yml Community package /check command /check comment on a package PR
community-package-preview-command.yml Community package /preview command /preview comment on a package PR
community-package-policy.yml Community package pipeline policy PR touching the pipeline sources
publish-provider-update.yml provider docs build repository_dispatch
bucket-cleanup.yml Scheduled jobs: Bucket cleanup Daily 3:00 PM UTC
update-tutorials.yml Scheduled jobs: Update How To Guides Daily 3:00 PM UTC
priority-digest.yml Scheduled jobs: Priority digest Daily 3:00 PM UTC
export-repo-secrets.yml Export secrets to ESC workflow_dispatch
add-triage-label.yml Add triage label to new issues Issue opened / reopened
add-to-project.yml Add issues to project Issue opened / reopened

5.1 pull-request.yml — PR Validation + Preview Deploy

Trigger: Pull request to master

ESC environment: github-secrets/pulumi-registry (OIDC, no long-lived secrets exported automatically)

Flow:

PR opened / committed
        │
        ├── resourcedocsgen (calls check-go.yml)
        │       ├── lint (golangci-lint)
        │       └── test (go test ./...)
        │
        ├── mktutorial (calls check-go.yml)
        │       ├── lint
        │       └── test
        │
        ├── lint-markdown
        │       └── yarn install → make lint-markdown
        │
        ├── lint-scripts
        │       └── yarn install → yarn run lint
        │
        ├── lint-dark-logos
        │       └── make lint-dark-logos
        │
        ├── test-live-publish
        │       └── uv run push-registry.py --dry-run
        │
        ├── test-provider-api-docs
        │       └── make ensure build-assets → make test_provider_api_docs
        │
        ├── preview  (skipped for fork PRs; skipped for automation/tfgen-provider-docs label)
        │       ├── Fetch ESC secrets
        │       ├── Install Node 22, Go 1.26, Hugo 0.157
        │       ├── Validate community-packages/package-list.json
        │       ├── Configure AWS credentials → assume testing account role
        │       ├── Install s5cmd v2.3.0
        │       └── make ci-pull-request
        │               ├── scripts/ci/validate-packages.sh
        │               ├── scripts/ci/build.sh preview
        │               └── scripts/ci/sync.sh preview
        │                       ├── Create / reuse S3 bucket
        │                       ├── s5cmd sync public/ → bucket
        │                       ├── gzip -9 llm-docs-out/ (parallel pre-compress)
        │                       ├── s5cmd sync llm-docs-out/ → bucket (Content-Encoding: gzip)
        │                       ├── Run browser tests (Cypress smoke test)
        │                       ├── Write origin-bucket-metadata.json
        │                       └── Update the pinned PR comment (preview URL + changed pages)
        │
        └── sentinel (depends on all jobs above)
                └── Writes "Sentinel" GitHub status check = success

Skipping preview for fork PRs: Fork PRs are excluded at the workflow level. The preview job has an if: condition that only allows it to run when github.event.pull_request.head.repo.full_name == github.repository, so the job is never scheduled for PRs from forks. pull-request.sh also contains a defensive credential check (for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and PULUMI_ACCESS_TOKEN) as a fallback, but this code path is not expected to be reached in practice.

The sentinel job: The sentinel job creates a GitHub status check called "Sentinel" only after all required jobs pass. This single required check simplifies branch protection rules. The sentinel job runs for non-fork PRs and for repository_dispatch events (condition: github.event_name == 'repository_dispatch' || github.event.pull_request.head.repo.full_name == github.repository).

Key environment variables in preview job:

Variable Source
PULUMI_ACCESS_TOKEN ESC output
GITHUB_TOKEN GitHub Actions default
PULUMI_STACK_NAME GitHub Actions variable
NODE_OPTIONS Hardcoded: --max_old_space_size=8192
ALGOLIA_APP_ID GitHub Actions variable
ALGOLIA_APP_SEARCH_KEY GitHub Actions variable

Runner: pulumi-service-ubuntu-24.04-16core (large runner required for the full build)

5.2 push.yml — Production Build + Deploy

Trigger: Push to master

ESC environment: github-secrets/pulumi-registry (OIDC via id-token: write)

Flow:

Push to master
        │
        └── build job
                ├── Fetch ESC secrets (OIDC)
                ├── Install Node 22, Go 1.26, Hugo 0.157
                ├── Checkout (using PULUMI_BOT_TOKEN for private module access)
                ├── Configure AWS credentials
                │       └── Assume arn:aws:iam::388588623842:role/ContinuousDelivery
                ├── Install s5cmd v2.3.0
                ├── Install Pulumi CLI
                ├── make ci_push
                │       ├── scripts/ci/login.sh
                │       │       └── pulumi login && pulumi -C infrastructure stack select $PULUMI_STACK_NAME
                │       ├── scripts/ci/build.sh update
                │       ├── scripts/ci/sync.sh update
                │       │       ├── Create / reuse S3 bucket
                │       │       ├── s5cmd sync public/ → bucket
                │       │       ├── gzip -9 llm-docs-out/ (parallel pre-compress)
                │       │       ├── s5cmd sync llm-docs-out/ → bucket (Content-Encoding: gzip)
                │       │       ├── Run browser tests (Cypress smoke test)
                │       │       └── Write origin-bucket-metadata.json
                │       ├── scripts/generate-search-index.sh
                │       │       └── Upload search-index.json to S3
                │       ├── node scripts/await-in-progress.js
                │       ├── scripts/ci/run-pulumi.sh update
                │       │       └── pulumi -C infrastructure update --yes
                │       │           (reads origin-bucket-metadata.json, updates CloudFront)
                │       └── scripts/ci/make-s3-redirects.sh
                │               └── Apply 301 redirects from scripts/redirects/
                ├── Archive origin-bucket-metadata.json as artifact
                └── uv run push-registry.py
                        └── Publish new provider versions to registry service

Runner: pulumi-service-ubuntu-24.04-16core

Key environment variables:

Variable Source
PULUMI_ACCESS_TOKEN ESC output
GITHUB_TOKEN GitHub Actions default
ALGOLIA_APP_ID GitHub Actions variable
ALGOLIA_APP_SEARCH_KEY GitHub Actions variable
ALGOLIA_APP_ADMIN_KEY ESC output
PULUMI_STACK_NAME GitHub Actions variable
PULUMI_DOCS_STACK_NAME GitHub Actions variable
DEPLOYMENT_ENVIRONMENT GitHub Actions variable
NODE_OPTIONS Hardcoded: --max_old_space_size=8192

5.3 Other Workflows

testing-deploy.yml — Manual Test Environment Deploy

Identical to push.yml but triggered manually via workflow_dispatch and deploys to the testing environment (account 571684982431, role arn:aws:iam::571684982431:role/ContinuousDelivery). Uses Go 1.21.x (note: older than production).

pull-request-closed.yml — PR Preview Cleanup

Trigger: PR closed (any PR from this repo, not forks)

Runs make ci-pull-request-closed which calls scripts/ci/pull-request-closed.sh:

  1. Fetches all commits associated with the closed PR from GitHub API.
  2. For each commit, looks up the associated S3 bucket in SSM Parameter Store.
  3. If a bucket is found and accessible, deletes it (aws s3 rb ... --force).
  4. Posts a PR comment noting that previews have been removed.

Requires: ESC secrets + AWS credentials (testing account role via AWS_CI_ROLE_ARN from ESC).

check-go.yml — Reusable Go Lint + Test

A reusable workflow_call workflow. Called by pull-request.yml for both tools/resourcedocsgen/ and tools/mktutorial/.

Jobs:

  • lint: Sparse checkout → golangci-lint v2.1.6 with --config .golangci.yml
  • test: Sparse checkout → go test ./... -v

check-links.yml — Link Validation

Trigger: Every Monday at 3:00 PM UTC (cron: 0 15 * * MON); also workflow_dispatch

Runs make check_links which calls yarn run check-links, which runs node scripts/link-checker/check-links.js "https://www.pulumi.com/registry" 2 (2 retries on failure). Broken links are reported to the #registry-ops Slack channel.

Node version: 22.x; Hugo 0.157.0 installed but not explicitly used.

run-browser-tests.yml — Scheduled Browser Tests

Trigger: Daily at 2:00 PM UTC; also workflow_dispatch

Runs make run-browser-tests on a pulumi-ubuntu-8core runner. Assumes the production AWS role (388588623842:role/ContinuousDelivery) to be able to reach the live site.

Node version: 22.x; Hugo 0.157.0 installed.

generate-package-metadata.yml — Nightly Community Package Check

Trigger: Daily at 5:30 AM UTC and 5:30 PM UTC; also workflow_dispatch

Flow:

  1. generate-packages-list job: Runs python generate_package_list.py in community-packages/ to build a matrix of community provider repos to check.
  2. check-for-package-update job (matrix, max-parallel: 1): For each provider, runs resourcedocsgen pkgversion to check if a new version is available. If so, runs resourcedocsgen metadata from-github to generate updated metadata and opens a PR via .github/actions/new-provider-version-pr.
  3. PRs are skipped if an open PR already exists for that provider (deduplication check via list_pull_requests in scripts/common.sh).

community-package-*.yml — Community Package Verified Check

The check pipeline gives a contributor who adds one entry to community-packages/package-list.json an automated, security-reviewed fact-sheet before a maintainer approves. It runs in two planes that never share a job: a secret-free plane that touches contributor input, and a privileged plane that never runs contributor code. community-package-policy.yml fails CI if any workflow mixes the two (SecretCodeSeparationTests).

  • community-package-check.yml (secret-free, runs on forks): for each added entry, reads the package's schema and docs at its latest GitHub release, then probes without executing the package's code — installs the plugin (blocking), resolves the npm/PyPI/Go SDKs and lints the docs (advisory). Writes a fact-sheet artifact. The plugin install is the only blocking check, alongside successful docs generation and a present docs/_index.md.
  • community-package-report.yml (workflow_run, write token, no secrets, no contributor code): downloads the fact-sheet artifact and posts it as a sticky PR comment, keyed to the PR number recorded by the check.
  • community-package-check-command.yml (issue_comment): re-runs the check when the author or a maintainer comments /check, authorized and rate-limited.
  • community-package-preview-command.yml (issue_comment): builds an on-demand site preview when a maintainer comments /preview. A fork's own pull_request build gets no secrets, so this maintainer-triggered run stands in for it: it materializes the fork's entry as data and reuses the build-and-deploy-preview action, never running the fork's code.
  • community-package-policy.yml: runs the toolchain's unit tests and mypy --strict, including the plane-separation test, as a required check.

After merge, generate-package-metadata.yml (above) generates and publishes the package's docs metadata.

publish-provider-update.yml — Provider Doc Update via Repository Dispatch

Trigger: repository_dispatch with event types resource-provider or push-provider-update

Used by first-party Pulumi provider repos to trigger documentation regeneration when a new provider version is released.

Event type Use case Required inputs
resource-provider GitHub-hosted provider (Pulumi repo) project-shortname, ref (version tag)
push-provider-update Opaque provider (no assumed GitHub structure) project-shortname, schema-url, index-url

For resource-provider: Calls resourcedocsgen metadata from-github → creates a PR. For push-provider-update: Downloads schema from schema-url, extracts version from schema, calls resourcedocsgen metadata from-urls → creates a PR.

bucket-cleanup.yml — Remove Stale S3 Preview Buckets

Trigger: Daily at 3:00 PM UTC; also workflow_dispatch

Runs make ci_bucket_cleanup which calls scripts/ci/bucket-cleanup.sh, which in turn calls scripts/ci/remove-buckets.sh push and scripts/ci/remove-buckets.sh pr.

For each deletable bucket (associated with a closed PR):

  1. Applies a lifecycle policy: all objects expire after 1 day.
  2. Adds a CleanupStarted tag with a timestamp.
  3. If the bucket has been in cleanup state for 48+ hours, attempts aws s3 rb.
  4. Gives up (with an error) if cleanup has been stalled for 7+ days.

Runs in the production environment (388588623842:role/ContinuousDelivery). Node 18.x / Go 1.20.x (older versions pinned in this workflow).

update-tutorials.yml — Regenerate Tutorials from Examples

Trigger: Daily at 3:00 PM UTC; also workflow_dispatch

  1. Checks out both pulumi/registry and pulumi/examples (into examples/).
  2. Runs scripts/ci/mktutorial.sh $GITHUB_WORKSPACE/examples.
  3. Opens or updates a PR on branch tutorials/refresh via peter-evans/create-pull-request@v7.

Auto-merge is currently disabled (commented out in the workflow).

priority-digest.yml — Post Open P0 and P1 Issues to Slack

Trigger: Daily at 3:00 PM UTC; also workflow_dispatch with a dry-run input

Runs scripts/ci/priority_digest.py, which searches GitHub for open issues labelled p0 or p1 across pulumi/registry and pulumi/terraform-to-pulumi-registry-pipeline, then posts them to #team-iac-cloud, oldest first, with each issue's age and assignee.

Replaces a Metabase subscription that posted the same query as a screenshot. Uses PULUMI_BOT_TOKEN and SLACK_WEBHOOK_URL from ESC; no other configuration. --dry-run prints the message to the job log instead of posting.

export-repo-secrets.yml — Sync GitHub Secrets → ESC

Trigger: workflow_dispatch only

Exports all GitHub repository secrets (except EXPORT_SECRETS_PRIVATE_KEY) to the github-secrets/pulumi-registry ESC environment using the pulumi/esc-export-secrets-action.

add-triage-label.yml — Auto-label New Issues

Trigger: Issue opened or reopened

Adds the needs-triage label to all new issues automatically.

add-to-project.yml — Add Issues to GitHub Project

Trigger: Issue opened or reopened

Adds new issues to the Pulumi Docs GitHub project (project #79) using PULUMI_BOT_GHA_MARKETING token from ESC.

Deployment Infrastructure

6.1 Pulumi IaC

Location: infrastructure/

Runtime: Node.js (runtime: nodejs, per infrastructure/Pulumi.yaml)

Purpose: Manages AWS resources. After a successful build and sync, the CI pipeline runs pulumi -C infrastructure update --yes, which reads origin-bucket-metadata.json to determine the newly-built S3 bucket and updates the CloudFront origin accordingly.

Stacks:

Stack file Stack name AWS account Purpose
Pulumi.yaml (base config) — Project definition
Pulumi.testing.yaml testing 571684982431 Test environment
Pulumi.production.yaml production 388588623842 Production environment

Key config values (both stacks):

Key Purpose
registry:pathToOriginBucketMetadata ../origin-bucket-metadata.json — where Pulumi reads the newly built bucket
registry:websiteLogsBucketName S3 bucket for CloudFront access logs
registry:e2eTestsBucketName S3 bucket for Cypress test results

6.2 AWS Resources

IAM roles: Both accounts have an arn:aws:iam::<account>:role/ContinuousDelivery role that CI assumes via OIDC or static key → role assumption.

S3 origin buckets: Public static website buckets named by the origin_bucket_prefix() + build_identifier() functions in scripts/ci/common.sh:

  • Prefix: registry-<deployment-env>-origin (e.g., registry-testing-origin)
  • Identifier for PRs: pr-<number>-<sha8> (e.g., registry-testing-origin-pr-42-a1b2c3d4)
  • Identifier for pushes: push-<sha8>

Each bucket is created as an S3 static website with index.html / 404.html and public-read ACL.

AWS SSM Parameter Store: Each commit's bucket name is stored at:

/registry/commits/<full-sha>/bucket

This mapping is used by pull-request-closed.sh to find and delete preview buckets when a PR closes.

CloudFront distributions: Managed by the Pulumi IaC program. After each production push, Pulumi reads origin-bucket-metadata.json and updates the CloudFront origin to point to the new S3 bucket.

6.3 S3 Preview Bucket Lifecycle

PR commit pushed
       │
       ▼
scripts/ci/sync.sh preview
  1. aws s3 mb registry-testing-origin-pr-<N>-<sha8>
  2. Enable static website hosting
  3. s5cmd sync public/ → bucket (--delete)
  3a. gzip -9 llm-docs.json files in llm-docs-out/ (parallel)
  3b. s5cmd sync llm-docs-out/ → bucket (Content-Encoding: gzip)
  4. Run Cypress smoke tests
  5. Write origin-bucket-metadata.json
  6. aws ssm put-parameter /registry/commits/<sha>/bucket = <bucket-name>
  7. Create or update the pinned PR comment with the preview URL and changed pages

PR merged or closed
       │
       ▼
scripts/ci/pull-request-closed.sh
  1. List all commits for the PR (GitHub API)
  2. For each commit: aws ssm get-parameter /registry/commits/<sha>/bucket
  3. If bucket exists: aws s3 rb s3://<bucket> --force
  4. aws ssm delete-parameter /registry/commits/<sha>/bucket
  5. Post PR comment: "Site previews have been removed."

Daily bucket-cleanup.yml (3:00 PM UTC)
  - Catches any buckets not cleaned up by pull-request-closed.sh
  - Applies 1-day lifecycle expiration policy
  - Deletes bucket after 48+ hours in cleanup state

6.4 The Pinned Preview Comment

Each preview build maintains a single comment on the PR rather than adding one per commit. The comment is written by post_preview_comment in scripts/ci/sync.sh, and carries:

  1. The preview URL for the current commit (<bucket-website>/registry/).
  2. A Changed pages list — direct links to the pages the PR changed, so a reviewer lands on them instead of navigating the preview by hand.

How it stays pinned: the body opens with the HTML marker <!-- registry-preview-link -->. upsert_github_pr_comment (scripts/ci/common.sh) pages through the PR's comments looking for that marker on a comment authored by github-actions[bot] or pulumi-bot, then PATCHes that comment; it only POSTs a new one when no match exists. Matching on the author as well as the marker means a contributor can't redirect the pinned comment by quoting the marker. The comment list is paginated deliberately — GitHub returns 30 comments per page by default, and an unpaginated search would miss the marker on a long PR and post a duplicate on every build.

How changed pages are resolved: changed_pages_section (scripts/ci/common.sh) reads the changed-file list from the GitHub API (/pulls/<n>/files), not a local git diff, so it works identically for the pull_request build and the maintainer-triggered /preview command. It collects every API page before mapping — changed_paths_to_urls de-duplicates only within a single invocation, so mapping page by page would double-list a package whose YAML and landing page straddle the 100-file page boundary. Each path is mapped under two rules:

Changed path URL
themes/default/content/**/*.md Hugo's own rules (content_path_to_url)
themes/default/data/registry/packages/<pkg>.yaml /registry/packages/<pkg>/

The YAML rule is the one that matters most here: the generated api-docs/ content is gitignored and never appears in a PR diff, so without it the list would be empty on most registry PRs. Results are de-duplicated, then filtered to URLs that actually rendered (public/<url>index.html exists), which drops removed files and url:/alias overrides rather than linking them as dead URLs. The list is capped at 50 entries with an "…and N more" line.

The whole block is reporting, not deployment: it is invoked as post_preview_comment || log … so a GitHub API hiccup can never fail an otherwise-good build. Conversely, because sync.sh runs under set -o errexit after the Cypress smoke test, a failed build posts nothing.

make test-preview-comment (scripts/ci/test-preview-comment.sh) covers the mapping, the de-duplication, and the existence gate offline; it runs in the Lint Scripts PR job.

6.5 Custom Redirects

Source files: scripts/redirects/ — pipe-delimited text files with key | location entries.

Applied by: scripts/ci/make-s3-redirects.sh as part of every production push (make ci_push).

Mechanism: For each redirect entry, creates an S3 object key with a WebsiteRedirectLocation header, causing S3 to return a proper 301 HTTP redirect rather than an HTML meta-refresh. This improves SEO and supports URL anchors.

Registry Publication (push-registry.py)

Location: scripts/ci/push-registry.py

Runtime: Python 3 (via uv run --with requests,pyyaml)

Invoked:

  • On every push to master (in push.yml, after the build+deploy completes)
  • In dry-run mode on every PR (in pull-request.yml, test-live-publish job)

What it does:

  1. Reads all YAML files from themes/default/data/registry/packages/*.yaml.
  2. For each package:
    • Skips packages where publisher == "DEPRECATED".
    • Skips packages whose name matches azure-native-v* (except azure-native itself) — these are aliases.
    • Skips packages whose name matches aws-v<N> — these are legacy versioned packages.
    • Calls the Pulumi registry API (https://api.pulumi.com/api/registry/packages/{source}/{publisher}/{name}/versions/{version}) to check if this version already exists.
    • If it does not exist (404): downloads the schema from the provider repo or schema_file_url, corrects the version field if needed, and calls pulumi package publish.
    • If --installation-configuration exists (_installation-configuration.md), passes it to pulumi package publish.
  3. In --dry-run mode: prints the pulumi package publish command instead of running it.

Required environment variable: PULUMI_ACCESS_TOKEN

Optional: PULUMI_BACKEND_URL (defaults to https://api.pulumi.com/api)

Package source: Determined by schema_url:

  • Contains registry.opentofu.org → source = "opentofu"
  • Otherwise → source = "pulumi"

Publisher lookup: tools/resourcedocsgen/pkg/publishers/publisher-names.json maps display names (as in YAML) to canonical publisher IDs.

Testing Strategy

8.1 Go Unit Tests

make test
# Runs: cd tools/resourcedocsgen && go test ./...

Also run in CI via check-go.yml for both tools/resourcedocsgen/ and tools/mktutorial/.

8.2 Go Linting

make lint-go
# Runs golangci-lint in both:
#   tools/resourcedocsgen/
#   tools/mktutorial/

Config: .golangci.yml (repo root).

golangci-lint version: v2.1.6 (specified in both mise.toml and check-go.yml).

8.3 Markdown Linting

make lint-markdown
# Runs: scripts/lint/lint-markdown.js

Node version used in CI: 23.x (in pull-request.yml lint-markdown job).

8.4 Script Linting

yarn run lint
# Runs: eslint scripts && prettier scripts --check

Lints all files under scripts/ with ESLint and Prettier. Config defined in the root ESLint and Prettier config files.

Node version used in CI: 23.x.

To auto-fix formatting:

yarn run format
# Runs: prettier scripts --write

8.5 Dark Logo Variants

make lint-dark-logos
# Runs: python3 scripts/generate-dark-logos.py --check

The dark-mode package marks under themes/default/assets/fingerprinted/logos/pkg/ (<name>-on-dark.svg) are generated from their light siblings, so adding or replacing a local logo leaves them stale. The check is deterministic, offline and stdlib-only, and runs in PR CI as the lint-dark-logos job. Regenerate with:

python3 scripts/generate-dark-logos.py

Its sibling, scripts/classify-external-logos.py, decides which packages with a third-party logo_url need a light chip in dark mode and writes themes/default/data/registry/external_logo_treatment.yaml. It downloads every external logo (and shells out to macOS sips for non-PNG rasters), so it is not wired into CI — run it by hand after adding a package with a logo_url, or when a vendor changes their logo. Its --check mode exits 2, rather than claiming the file is stale, if any logo could not be measured.

8.6 Provider API Docs Tests

make test_provider_api_docs
# Runs: scripts/ci/run-provider-tests.sh

Requires: ensure, build-assets, and bin/resourcedocsgen to have been built. Runs in PR CI as the test-provider-api-docs job.

Node version in CI: 23.x. Go version: stable (latest).

8.7 Browser Tests (Cypress)

make run-browser-tests
# Runs: scripts/run-api-docs-tests.sh

Config: cypress.config.js — default base URL is http://localhost:1313.

Test location: cypress/ directory.

Reporters: cypress-multi-reporters (config in reporter-config.json).

Browser tests are run:

  1. As a smoke test inside scripts/ci/sync.sh after each S3 deploy (both preview and production), using the deployed S3 website URL.
  2. Daily at 2:00 PM UTC via run-browser-tests.yml against the live production site.

8.8 Link Checking

make check_links
# Runs: yarn run check-links
# Which runs: node scripts/link-checker/check-links.js "https://www.pulumi.com/registry" 2
  • Fetches the sitemap from https://www.pulumi.com/registry/sitemap.xml.
  • Excludes API docs pages, SDK reference pages, and install/versions pages.
  • Uses broken-link-checker with filterLevel: 1 and GET requests.
  • Up to 2 retries if broken links are found.
  • Reports broken links to the #registry-ops Slack channel via SLACK_ACCESS_TOKEN.
  • Many known-flaky domains are excluded (LinkedIn, YouTube, Twitter, etc.).

Runs every Monday at 3:00 PM UTC in CI.

Environment & Secret Management

9.1 Pulumi ESC

Organization: pulumi Environment: github-secrets/pulumi-registry

All workflows use OIDC token exchange to authenticate with Pulumi ESC — no long-lived secrets are stored in GitHub Actions secrets directly (except for AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY used in the testing environment for PR preview deploys).

The ESC action is configured via workflow-level environment variables:

ESC_ACTION_OIDC_AUTH: true
ESC_ACTION_OIDC_ORGANIZATION: pulumi
ESC_ACTION_OIDC_REQUESTED_TOKEN_TYPE: urn:pulumi:token-type:access_token:organization
ESC_ACTION_ENVIRONMENT: github-secrets/pulumi-registry
ESC_ACTION_EXPORT_ENVIRONMENT_VARIABLES: false  # (or a specific mapping)

The export-repo-secrets.yml workflow provides a manual escape hatch to sync GitHub repository secrets into ESC.

9.2 Environment Variables Reference

Variable Source Used in Purpose
PULUMI_ACCESS_TOKEN ESC build, preview, cleanup Authenticate with Pulumi Cloud
AWS_ACCESS_KEY_ID GitHub secret preview (testing) Initial AWS auth for testing environment
AWS_SECRET_ACCESS_KEY GitHub secret preview (testing) Initial AWS auth for testing environment
AWS_CI_ROLE_ARN ESC preview, cleanup IAM role to assume in testing account
GITHUB_TOKEN GitHub Actions build, preview, cleanup, metadata GitHub API access
PULUMI_BOT_TOKEN ESC push checkout, tutorials, metadata PRs Bot token for PR creation
ALGOLIA_APP_ID GitHub var build, preview Algolia application ID
ALGOLIA_APP_SEARCH_KEY GitHub var build, preview Algolia public search key
ALGOLIA_APP_ADMIN_KEY ESC production build Algolia admin key (index writes)
PULUMI_STACK_NAME GitHub var build, cleanup Pulumi stack to select (e.g., pulumi/registry/production)
PULUMI_DOCS_STACK_NAME GitHub var build Pulumi docs stack reference
DEPLOYMENT_ENVIRONMENT GitHub var build, cleanup e.g., testing or production
NODE_OPTIONS Hardcoded build, browser tests --max_old_space_size=8192
SLACK_ACCESS_TOKEN ESC link check, cleanup Slack Web API token for posting messages
SLACK_WEBHOOK_URL ESC notify jobs, priority digest Slack incoming webhook for failure alerts and the daily digest
ASSET_BUNDLE_ID build.sh (computed) Hugo templates Unique suffix for CSS/JS cache-busting
CSS_BUNDLE / JS_BUNDLE build.sh (computed) Hugo templates Paths to versioned asset bundles

9.3 Tool Version Matrix

Tool mise.toml pull-request.yml (preview) push.yml (production) testing-deploy.yml
Node.js 20 22.x 22.x 22.x
Go 1.26 tools/resourcedocsgen/go.mod tools/resourcedocsgen/go.mod tools/resourcedocsgen/go.mod
Hugo 0.157 0.157.0 0.157.0 0.157.0
golangci-lint 2.1.6 v2.1.6 (check-go.yml) — —
s5cmd — v2.3.0 v2.3.0 v2.3.0

Note: mise.toml specifies Node 20 for local development, while CI workflows use Node 22. The lint-markdown and lint-scripts jobs in pull-request.yml use Node 23.x. The bucket-cleanup.yml workflow uses Node 18.x.

Scheduled & Maintenance Tasks

Task Schedule (UTC) Workflow Key Command
Community package metadata check 5:30 AM + 5:30 PM daily generate-package-metadata.yml python generate_package_list.py → resourcedocsgen pkgversion / metadata from-github
Tutorial refresh from examples 3:00 PM daily update-tutorials.yml scripts/ci/mktutorial.sh → PR on branch tutorials/refresh
Link check 3:00 PM every Monday check-links.yml make check_links
Browser tests (scheduled) 2:00 PM daily run-browser-tests.yml make run-browser-tests
Stale bucket cleanup 3:00 PM daily bucket-cleanup.yml make ci_bucket_cleanup
Open P0 and P1 issue digest 3:00 PM daily priority-digest.yml scripts/ci/priority_digest.py

Troubleshooting

Hugo version mismatch

Symptom: Build fails with template errors or unexpected output.

Fix: Ensure you are using Hugo 0.157 extended. Check with hugo version. If using mise: mise install will install the correct version. Ensure you have the extended variant (required for SCSS processing).

Node out-of-memory error

Symptom: Node process killed during build with FATAL ERROR: Reached heap limit.

Fix: Set the Node memory limit before running the build:

export NODE_OPTIONS="--max_old_space_size=8192"
make build

This is set automatically in CI. If local builds still fail, consider increasing the value or ensuring adequate system RAM (32 GB+ recommended for full builds).

PR preview not posting

Symptom: PR has no preview comment; the preview job shows "Missing secret tokens, possibly due to a forked PR."

Cause: PRs from forks do not have access to repository secrets. scripts/ci/pull-request.sh detects the absence of AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, or PULUMI_ACCESS_TOKEN and skips the S3 sync. The build still runs; only the deployment is skipped.

Fix: This is expected behavior for fork PRs. If you need a full preview, merge the fork into a branch on the upstream repo.

resourcedocsgen schema fetch failure

Symptom: resourcedocsgen docs registry exits with a non-zero status; error mentions a specific provider.

Cause: The provider's schema URL (in its YAML file under themes/default/data/registry/packages/) may be unreachable, or the version field may reference a tag that does not exist in the provider's GitHub repo.

Fix: Check the YAML file for the failing provider. Verify that the version tag exists on the provider's GitHub repo and that the schema file path is correct.

S3 bucket cleanup failures

Symptom: bucket-cleanup.yml fails or leaves stale buckets.

Cause: The lifecycle policy approach means buckets are not immediately deleted. The 2-step process (apply lifecycle → wait 48h → delete) is intentional. Buckets that have been in cleanup state for 7+ days will cause the cleanup script to exit non-zero.

Fix: Manually inspect and delete the affected bucket via AWS console or CLI. Remove the associated SSM parameter.

Pulumi stack selection failure

Symptom: scripts/ci/login.sh fails with "stack not found" or similar.

Cause: PULUMI_STACK_NAME environment variable is not set or does not match an available stack.

Fix: Ensure PULUMI_STACK_NAME is set correctly in the GitHub Actions environment variables for the relevant environment (testing or production). The stack must exist in the Pulumi Cloud org. Run pulumi -C infrastructure stack ls locally (with correct credentials) to confirm available stacks.

push-registry.py publisher not found

Symptom: push-registry.py raises Exception: Missing publisher entry for "<publisher-name>".

Cause: A package YAML file references a publisher display name that is not listed in tools/resourcedocsgen/pkg/publishers/publisher-names.json.

Fix: Add the publisher to publisher-names.json with the correct canonical identifier, or update the package YAML to use an already-registered publisher name.

Pulumi Cloud Service Integration

12.1 Two Parallel Systems

The Pulumi Registry is actually two separate but tightly coupled systems that must remain in sync:

System URL Primary Consumer Data Source
Static Hugo site pulumi.com/registry Humans (browser) YAML files in themes/default/data/registry/packages/
Pulumi Cloud Registry API api.pulumi.com/api/registry Pulumi CLI (pulumi up, pulumi package add) Pulumi Cloud database (populated via pulumi package publish)

These two systems are not the same thing and do not share a data store. The static site is rebuilt from YAML files on every push to master; it does not query the Pulumi Cloud API at runtime. The Pulumi Cloud API is a live service that stores package metadata independently.

The bridge between them is scripts/ci/push-registry.py, which runs after every production build and publishes any new package versions to the Pulumi Cloud API.

YAML files in repo                            Pulumi Cloud Registry API
(source of truth for                          (source of truth for CLI
 the static Hugo site)                         package resolution)
        │                                               │
        │  scripts/ci/push-registry.py                  │
        │  (runs on every production push)              │
        └──────────────────────────────────────────────►│
                  pulumi package publish                │

Consequence: If push-registry.py fails silently on a particular package, the Hugo site will show the package correctly but the Pulumi CLI will not be able to resolve it. The two systems can drift.

Consequence: The static site does not support version browsing (no "select a version" dropdown) because Hugo generates a fixed set of pages from a fixed set of YAML files. Versioned snapshots (e.g., aws-v6) are implemented as entirely separate YAML files, separate Hugo pages, and separate API publication entries — not as a first-class versioned concept.

12.2 How push-registry.py Bridges Them

scripts/ci/push-registry.py is the synchronization mechanism. On every push to master, after the Hugo site is built and deployed, this script:

  1. Reads every YAML file from themes/default/data/registry/packages/.
  2. For each package, queries GET /api/registry/packages/{source}/{publisher}/{name}/versions/{version} to check whether this exact version already exists in the API.
  3. If it does not exist (404):
    • Downloads the provider schema JSON from the URL in the YAML file.
    • Corrects the version field in the schema if it is absent or inconsistent with the YAML.
    • Runs pulumi package publish <schema.json> --readme <_index.md> --source <source> --publisher <publisher> to register the version.
  4. If it does exist (200): no-op.
  5. Skips deprecated packages, azure-native-v* aliases, and aws-v* legacy versioned packages.

Important: push-registry.py is also run in dry-run mode on every PR (--dry-run flag) as the test-live-publish CI job. This validates that all YAML files are parseable, all publishers are known, and the pulumi package publish invocation would be valid — without actually touching the production API.

Required credential: PULUMI_ACCESS_TOKEN must be set. This is sourced from Pulumi ESC in CI.

12.3 Pulumi Cloud Roles Summary

Pulumi Cloud plays four distinct roles in this system:

Role Mechanism Purpose
Registry API api.pulumi.com/api/registry Stores published package versions; queried by the Pulumi CLI for package and plugin resolution
Secrets / ESC github-secrets/pulumi-registry environment Provides CI secrets (tokens, keys) via OIDC; eliminates long-lived secrets in GitHub
IaC State Backend pulumi/registry/testing and pulumi/registry/production stacks Stores Terraform-like state for the AWS infrastructure (CloudFront, S3, policies) managed by infrastructure/index.ts
Stack References pulumi/dwh-workflows-* stacks Exposes runtime service URLs and IAM role ARNs as stack outputs, consumed by the build and deploy pipeline