Evaluate inferred modules for SDK compile performance - #1653
Draft
corymhall wants to merge 3 commits into
Draft
Conversation
Summary: - Infer modules from Terraform resource and data source names. - Retain every v6.19 resource and function token with hard clones. - Record the immutable v6.19 compatibility surface in a manifest. - Avoid data source token conflicts and C# namespace conflicts. Rationale: - The single Go index package is too large to compile reliably. - Hard compatibility resources keep existing programs source compatible while new programs can import smaller module packages. Tests: - cd provider && go test . ./pkg/...
Summary: - Regenerate the provider schema and bridge metadata with inferred modules. - Add tests for the retained v6.19 resources, functions, types, and aliases. - Update the provider overview example to use the modular DNS resource. Rationale: - Keep the generated schema review separate from provider implementation and language SDK output. - Verify that modular tokens are additive to the complete v6.19 surface. Tests: - cd provider && go test . ./pkg/... - schema-tools compare v6.19.0 with the local schema: no breaking changes
Summary: - Regenerate the Go, Node.js, Python, .NET, and Java SDKs. - Add packages for all inferred Cloudflare resource families. - Retain the complete v6.19 root SDK surface for compatibility. Rationale: - Keep the large generated SDK change isolated from provider and schema logic. - Let new Go programs import small packages instead of the large index package. Tests: - cd sdk && go test ./go/cloudflare/... - yarn install --frozen-lockfile && yarn run tsc - dotnet build - gradle --console=plain compileJava - python3 -m compileall -q sdk/python/pulumi_cloudflare
|
Tested this directly against the PR commit ( mkdir cf1653test && cd cf1653test
printf 'module cf1653test\ngo 1.25\n' > go.mod
GOFLAGS=-mod=mod go get github.com/pulumi/pulumi-cloudflare/sdk/v6@e2460a2014b59cbaa9c71aeb10a18453a2b06575
GOCACHE=$(mktemp -d) time GOFLAGS=-mod=mod go build \
github.com/pulumi/pulumi-cloudflare/sdk/v6/go/cloudflare/dns \
github.com/pulumi/pulumi-cloudflare/sdk/v6/go/cloudflare/zoneSame machine/method as the measurements in #1652:
This fully resolves our case — not just "isolated cleanly" as hoped, but 3.6× lower peak memory than v5, since we're now compiling ~646 KB of actual surface (8 + 19 files) instead of the full 27 MB / 666-file provider. On a 7 GB CI runner this moves from "reliably OOMs" to a non-issue. Happy to be a real-world data point if it's useful context for the merge decision — glad to re-test against later iterations too. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This draft evaluates inferred modules as an option for the Go SDK compile-time and memory problem in #1652.
New resources and functions use inferred product-family modules, while hard compatibility clones retain the complete v6.19
indexsurface. Existing programs remain source compatible; migrated Go programs can import smaller packages.schema-toolsreports 238 new resources, 405 new functions, 2,624 new types, and no breaking changes against v6.19.0.Important tradeoffs:
worker/workersandcustomPage/customPages.Pulumi.Cloudflare.Modules.*because retained root classes conflict with inferred namespace names. New C# resources also keep descriptive class names to avoid class/member name conflicts.The commits intentionally separate provider logic, schema output, and generated SDKs so each part can be reviewed independently.
Validation: provider tests; all Go SDK tests; Node.js type checking; .NET and Java builds; Python compilation; schema comparison against v6.19.0.
This is a design evaluation, not a request to merge the approach as-is.
Re #1652