Correct pre-built policy pack coverage, and enforce the allowlist against the registry - #21055
Correct pre-built policy pack coverage, and enforce the allowlist against the registry#21055CamSoper wants to merge 4 commits into
Conversation
The pre-built packs table and the pages that point at it were written when ISO 27001, NIST, and PCI DSS shipped for fewer clouds than they do today, and were never refreshed as coverage grew. The generated reference pages under /docs/reference/pre-built-policy-packs/ (driven by data/policy_packs.yaml) have been correct all along, so the summary surfaces were the only thing telling readers we ship less than we do. Reconciled against data/policy_packs.yaml and data/policy_pack_policies/: - ISO/IEC 27001:2022 — listed as AWS-only; we ship AWS, Azure, and Google Cloud (238 + 158 + 153 = 549 policies). - PCI DSS v4.0.1 — listed as AWS-only; we ship all three clouds. - NIST SP 800-53 — listed as AWS and Google Cloud; we also ship Azure. - CMMC 2.0 (AWS) — missing from the table entirely. - HITRUST CSF — missing its AWS Native (aws-native provider) pack. Also added ISO 27001 and CMMC to the framework lists on the Policies overview, the policy packs overview, the Insights landing page, and the onboarding guide, and filled in the missing packs on the per-cloud AWS, Azure, and Google Cloud integration pages. Every link added was checked against the allowlist that generates those pages. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
|
Your site preview for commit cd00d9a is ready! 🎉 http://www-testing-pulumi-docs-origin-pr-21055-cd00d9a2.s3-website.us-west-2.amazonaws.com Changed pages: |
github-actions
Bot
added
review:triaging
![unblocked[bot]](https://avatars.githubusercontent.com/in/166219?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ No issues found
About Unblocked
Unblocked has been set up to automatically review your team's pull requests to identify genuine bugs and issues.
📖 Documentation — Learn more in our docs.
💬 Ask questions — Mention @unblocked to request a review or summary, or ask follow-up questions.
👍 Give feedback — React to comments with 👍 or 👎 to help us improve.
⚙️ Customize — Adjust settings in your preferences.
github-actions
Bot
added
the
review:in-progress
Pre-merge Review — Last updated 2026-08-24T21:44:57ZTip Summary: A docs accuracy PR that widens the advertised coverage of Pulumi's pre-built compliance policy packs, now carrying a second, infra-shaped change: Review confidence:
Investigation log
🔍 Verification trail25 claims extracted · 17 verified · 2 unverifiable · 5 contradicted
🚨 Outstanding in this PRThese must be resolved or refuted before merging.
|
github-actions
Bot
added
review:no-blockers
| - [NIST 800-53 for AWS](/docs/reference/pre-built-policy-packs/nist/aws/) | ||
| - [PCI DSS for AWS](/docs/reference/pre-built-policy-packs/pci-dss/aws/) | ||
| - [HITRUST CSF for AWS](/docs/reference/pre-built-policy-packs/hitrust/aws/) | ||
| - [HITRUST CSF for AWS Native](/docs/reference/pre-built-policy-packs/hitrust/aws-native/) — for the AWS Cloud Control (`aws-native`) provider. |
There was a problem hiding this comment.
This isn't actually published. It exists in our repo privately. I noticed it's appearing in references, too. Would you mind removing that in this PR?
I think some docs automation might have picked it up @jkodroff
There was a problem hiding this comment.
Good catch (sorry for the Claude-speak). I'll hit you up on Slack to figure out the source of truth on what should actually be published.
There was a problem hiding this comment.
Done in 93d5904 — removed from the table, this page, and data/policy_packs.yaml, which is what was generating the reference page. Also dropped the fetched policy data and its lastmod entry, and added an S3 redirect from /docs/reference/pre-built-policy-packs/hitrust/aws-native/ to the HITRUST AWS page, since the old URL is live today.
On how it got there, since it's relevant to the source-of-truth question: data/policy_packs.yaml is a hand-maintained allowlist, and its rule read "only document packs that are built from the policy-packs-internal repo, i.e. the ones we ship to customers" — treating those two as one condition. hitrust-awsnative satisfies the first and not the second, and nothing available to the docs side distinguishes them: it's built there, it published cleanly in the 2026-07-09 batch, and the Pulumi Cloud API returns it like any other pack, which is how the nightly fetch picked up 114 policies for it. So the automation wasn't working from a bad signal — it was working from the only signal it has, which doesn't carry the private/public bit at all.
I've reworded the rule as two independent conditions and recorded the exclusion at the bottom of that file, so an audit that diffs the org listing against the repo doesn't re-add it. But that's a comment telling a human to go ask, not a check — if there's a durable way to mark a pack private in policy-packs-internal (a private: true in its package.json, a naming convention, anything the fetch script could read), that'd be a much better fix than trusting the next person to read a comment. Happy to wire that up on the docs side once you and @jkodroff land on what the source of truth should be.
Worth a look while you're in there: the same batch had four packs fail to publish and never get retried, one of them aws-organizations-tag-policies v2.0.1 — pulumi/policy-packs-internal#211. Found by an agent, unverified by a human, so treat it accordingly.
Generated by Claude Code
Upstream restructured content/docs/administration/, which moved setting-up-for-success.md from onboarding-guide/ to get-started/. Git followed the rename and carried this branch's edit with it; no conflicts to resolve by hand. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
@danbiwer flagged in review that the HITRUST AWS Native pack is private to policy-packs-internal and is not published to customers, and that docs automation had generated a public reference page for it at /docs/reference/pre-built-policy-packs/hitrust/aws-native/. Removes it from the pre-built packs table, the AWS integrations page, and data/policy_packs.yaml, which is what generated the reference page. Also drops the fetched policy data and its lastmod entry so the data set stays consistent with the allowlist. The reference page is live today, so removing it changes a public URL: adds an S3 redirect to the HITRUST AWS page, the nearest real equivalent. The allowlist rule is what let this through. It said to document packs "built from the policy-packs-internal repo, i.e. the ones we ship to customers", treating those as the same set. They aren't: this pack is built there, publishes cleanly, and is returned by the API, and none of that makes it customer-facing. Restates the rule as two independent conditions, notes that neither the repo nor the API can distinguish them, and records the exclusion at the bottom of the file so a future audit against the org doesn't re-add it. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
github-actions
Bot
added
review:stale
|
@claude #update-review Two pushes since the last review:
The removal spans the table, the AWS integrations page, Not disputing anything else. The out-of-scope note about Generated by Claude Code |
github-actions
Bot
added
review:in-progress
|
🤖 Review updated on @CamSoper's request. |
github-actions
Bot
added
review:no-blockers
| # guide, not a policy listing. Stays hand-maintained. | ||
| # cis-kubernetes-google-cloud A stale earlier publish, superseded by | ||
| # cis-kubernetes-gcp. See the cis-kubernetes section. | ||
| # hitrust-awsnative Built here and resolvable through the API, but NOT a |
There was a problem hiding this comment.
It's resolvable through the API b/c someone published it specifically to the pulumi org. We either need an allowlist on the docs end, or a "clean" org for reading things from API endpoints.
There was a problem hiding this comment.
Built the allowlist option, in cd00d9a2, using the source field you and @danbiwer landed on in Slack.
data/policy_packs.yaml was already an allowlist — it's how we document 22 of the packs that repo carries rather than all of them. What it didn't have was any check on its own entries, so a human adding a wrong row was enough. fetch-policy-packs.js now requires every entry to come back from /api/registry/policypacks?orgLogin=<org> with source: "pulumi", and fails naming the offenders and their actual source.
Verified against the two records Dan pasted, including both failure modes from this incident — a source: "private" pack, and a pack missing from the listing entirely, which is hitrust-awsnative's state now that you've deleted it.
Two things from your thread are captured as comments in the code, because both are easy to get wrong later:
- Not
publisher. It's"pulumi"for everything in our org, so filtering on it admits the private packs too. It's the more obvious-looking field, so there's a comment saying don't switch to it. - Your spoofing question is what made a hard failure defensible. Dan's answer —
sourceis admin-set, not set by whoever publishes — is why the check throws rather than warns. If a passing engineer could set it, gating the build on it would be theater. That reasoning is in the comment too, so it survives.
The check fails closed: an unreadable or changed response stops the job rather than publishing unverified pages.
I couldn't exercise the live call — both endpoints need a token this environment doesn't have — so the parsing is written against Dan's records and unit-tested, and the envelope key is accepted loosely since only the record shape was confirmed. First real run will settle it, and it's on the nightly, so that's tonight.
Which brings up something time-sensitive: deleting the pack from the org means master is now broken for that job. fetch-policy-packs.js throws when an allowlisted pack isn't in the org, and master still carries the entry — so the 07:30 UTC run fails and pings #docs-ops. This PR is the fix.
One more, tangential: Dan's aws-organizations-tag-policies record reads version: 1.0.0, while the repo is at 2.0.1 and there's a GitHub release for 2.0.0. If that was a live listing rather than an illustrative paste, the published version is lagging further than pulumi/policy-packs-internal#211 estimates and I should correct that issue. If it was just a paste, ignore me.
Generated by Claude Code
jkodroff
left a comment
jkodroff
left a comment
There was a problem hiding this comment.
Approving, since this fixes the most important mistake, but it's not a full solution.
|
@jkodroff I'm rolling in the allowlist idea |
Turns the rule in data/policy_packs.yaml from prose into a check. `/api/orgs/<org>/ policypacks`, which this script already reads, returns everything the org holds -- products and privately published packs alike, indistinguishably. That is how the private hitrust-awsnative pack got a public reference page (#21055). The registry listing carries the distinction in `source`: "pulumi" for a published product, "private" otherwise. Every allowlisted pack must now come back as a product or the job fails naming the offenders and their actual source. Verified against real records from @danbiwer, including the two failure modes behind this incident: a privately published pack, and a pack absent from the listing entirely. Deliberately not `publisher`: that is "pulumi" for everything in our org, so it would admit private packs too. `source` is set by an admin command rather than by whoever publishes, so it cannot be set by an engineer testing something -- which is what makes it safe to fail the build on. Fails closed. An unreadable or changed response stops the job rather than falling back to publishing unverified pages. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
github-actions
Bot
added
review:stale
CamSoper
changed the title
|
@claude #update-review One new commit since the last refresh request, plus scope this PR didn't have before — it now touches
Three things worth pointing your infra checks at:
Also note this reverses your earlier triage of the Not disputing anything outstanding. The two pre-existing findings in Generated by Claude Code |
github-actions
Bot
added
review:in-progress
|
🤖 Review updated on @CamSoper's request. |
github-actions
Bot
added
review:outstanding-issues
5 participants
Proposed changes
Two things: the pre-built packs table understated what we ship, and the allowlist that decides what gets documented had no way to tell a product from a pack someone published privately to the org. The second is what let the first get "fixed" wrongly in an earlier revision of this PR.
1. Coverage corrections
/docs/insights/policy/policy-packs/pre-built-packs/ was written when several frameworks were AWS-only and never refreshed as coverage grew. The generated reference pages under
/docs/reference/pre-built-policy-packs/were right all along; the hand-maintained summary surfaces were the only thing telling readers we support less than we do. Two accounts independently concluded we don't support ISO 27001 and went elsewhere, while we ship 549 ISO 27001 policies across all three clouds.Also added ISO 27001 and CMMC to the framework lists on the Policies overview (which omitted ISO 27001 entirely), the policy packs overview, the Insights landing page, and the onboarding guide; and filled in the missing packs on the per-cloud AWS, Azure, and Google Cloud integration pages, where Azure and Google Cloud were each missing NIST, PCI DSS, and ISO 27001.
2. Removing
hitrust-awsnative@danbiwer flagged in review that the HITRUST AWS Native pack is private and not shipped to customers, and that docs automation had generated a public reference page for it. An earlier revision of this PR added it to the table on the strength of that page existing — that was wrong. 93d5904 removes it from the table, the AWS integrations page,
data/policy_packs.yaml(which generated the page), the fetched policy data, and the lastmod ledger, and adds an S3 redirect for the now-removed reference URL.@jkodroff has since deleted the pack from the
pulumiorg. That makes this PR time-sensitive:fetch-policy-packs.jsthrows when an allowlisted pack isn't in the org, so onmaster— which still carries the entry — tonight's 07:30 UTC run fails and pings #docs-ops. Merging this clears it.3. Enforcing the allowlist (
cd00d9a2)The removal above fixes one bad row. This stops the next one.
data/policy_packs.yamlwas already an allowlist, but its rule was prose: "packs built from policy-packs-internal, i.e. the ones we ship to customers", treating one condition as two.hitrust-awsnativesatisfied the first and not the second, and/api/orgs/{org}/policypacks— the endpoint this script already reads — returns products and privately published packs indistinguishably, so nothing caught it.Per @danbiwer, the registry listing does carry the distinction, in
source:{ "source": "private", "publisher": "pulumi", "name": "approved-component-versions-bad", ... } { "source": "pulumi", "publisher": "pulumi", "name": "aws-organizations-tag-policies", ... }fetch-policy-packs.jsnow requires every allowlisted pack to come back from/api/registry/policypacks?orgLogin={org}withsource: "pulumi", and fails naming any that don't along with their actual source. Verified against those real records, including both failure modes behind this incident: a privately published pack, and a pack absent from the listing entirely (which ishitrust-awsnative's state now).Two notes for whoever touches this next, both recorded in comments:
publisher. It's"pulumi"for everything in our org, so filtering on it would admit the private packs too — it just looks like the right field.sourceis set by an admin command, not by whoever publishes, so an engineer testing something can't set it by accident. That's what makes it safe to fail a build on, per @jkodroff's question in Slack.The check fails closed: an unreadable or changed response stops the job rather than falling back to publishing unverified pages.
Allowlist audit
data/policy_packs.yamlnow documents 22 packs. Everything elsepolicy-packs-internalcarries stays out, and the new check enforces the last row rather than trusting a comment:cmmc-azureindex.tsis ~135 lines of comments and no policies; publish fails[400] Policy Pack must have at least one policy. Never published.cmmc-google-cloudcis-kubernetes-self-hostedexport const selfHostedK8sPolicies = []under a TODO list. Never published.aws-configworkspacesarray, so changesets never versions it and the publish workflow can't see it.package.jsonstill says"name": "aws-typescript", v0.0.1, no CHANGELOG. A WIP port of the AWS Config ruleset per its ownCLAUDE.md.hitrust-awsnativeThe first four were visible from the repo. The fifth wasn't, which is the whole point of §3.
Verification
node scripts/lint/lint-markdown.jsandprettier --checkpass. Every added reference link resolves against the allowlist, and the allowlist,data/policy_pack_policies/, anddata/policy_pack_lastmod.jsonagree on all 22 packs with no orphans either way. The registry check's parsing and gate logic were exercised against @danbiwer's real records; the live call itself is unrun here, since both endpoints need a token this environment doesn't have. Hugo isn't installed either, somake buildwasn't run — the content changes are prose and links only.Known follow-up, not in this PR
Several pages still carry the old framework list and will read as incomplete once this lands: the "Policy as code" row across
content/docs/iac/comparisons/*.md, andcontent/what-is/what-is-policy-as-code.md(L199). Left out to keep this diff reviewable.Separately, pulumi/policy-packs-internal#211 records four packs that failed to publish in the 2026-07-09 batch and were never retried, one being the shipped
aws-organizations-tag-policies. Unrelated to this diff. Worth noting the record above shows that pack atversion: 1.0.0while the repo is at 2.0.1 — if that listing is live, the published version lags further than #211 estimates, and that issue needs correcting.