Drop the pulumi/pulumi submodule - #1065
Draft
iwahbe wants to merge 12 commits into
Draft
Conversation
iwahbe
added a commit
that referenced
this pull request
Jul 3, 2026
iwahbe
marked this pull request as draft
July 3, 2026 14:20
Generate ResourceHook declarations for named hook blocks (running the hook's command via System.Diagnostics.Process and honoring the onDryRun and ignoreErrors attributes), ErrorHook declarations for hook blocks of kind error (which request a retry if and only if the command exits successfully), and the Hooks resource option binding hooks to their lifecycle events. This makes the l2-resource-option-hooks, l2-resource-hook-after-failure, and l2-resource-hook-ignore-errors conformance tests pass; remove them from the expected failures and commit their snapshots.
The test was added in pulumi/pulumi#23839, which is not in any release yet. The snapshot was generated and verified by pinning github.com/pulumi/pulumi/{pkg,sdk}/v3 to that commit (8a79286b30228227ab34b6471298d29005f0fcfa) and running PULUMI_ACCEPT=1 go test -run 'TestLanguage/l2-resource-hook-on-error$' . followed by a clean run without PULUMI_ACCEPT. At the currently pinned v3.253.0 the test does not exist, so the snapshot is inert until the next pulumi/pulumi upgrade picks it up.
The l2-resource-hook-on-error conformance test covers the same round trip: an error hook attached to a resource whose first create fails retryably runs in-process, requests a retry, and the resource is created on the second attempt. pulumi/pulumi#23840 also removed the FlakyCreate resource from the upstream testprovider that this test loaded from the submodule, so the test would break on the next pulumi/pulumi upgrade regardless.
changie kinds are matched by label (Improvements), not by a lowercased key; only Bug Fixes defines a key (bug-fixes).
…test
Pin github.com/pulumi/pulumi/{pkg,sdk}/v3 and the pulumi submodule to
master commit cd588358a1979fe390708412b970a2f6ab96fa88, which contains
the l2-resource-hook-on-error conformance test (pulumi/pulumi#23839).
No release includes it yet; the next regular dependency upgrade moves
the pin back to a release tag.
Fallout from moving past v3.253.0:
- Regenerate sdk/Pulumi/Pulumi.xml for new proto messages.
- Regenerate the l2-enum and l2-external-enum snapshots: upstream added
a NumberEnum to the shared enum test provider.
- Remove the simple-enum-schema codegen goldens; the upstream test case
was deleted in pulumi/pulumi#23952.
- Replace the now-deprecated maputil.SortedKeys with
slices.Sorted(maps.Keys(...)) to satisfy staticcheck.
The l2-enum regeneration also produced Deluxe.cs, Inputs/HolderArgs.cs, and Outputs/Holder.cs for the resources upstream added to the enum test provider; they were left untracked in the previous commit, which made snapshot validation fail in CI while passing locally.
The submodule provided three things; each is replaced by an alternative pinned to the pulumi/pulumi version in pulumi-language-dotnet/go.mod: - Protobuf definitions: the .proto files the C# SDK compiles via Grpc.Tools are now vendored under proto/ at the repo root, together with the Automation API CLI-spec overrides file. They cannot be sourced from the Go module graph because the pkg and sdk modules do not ship the .proto sources, so `make sync_protos` copies them from a sparse clone of pulumi/pulumi at the pinned version, and the format CI job fails if the vendored files drift from that version. - generate_cli_spec: the pulumi CLI is now built through a Go `tool` directive in pulumi-language-dotnet/go.mod instead of the submodule's pkg directory. `go run` of the dependency's main package does not work because module graph pruning leaves the CLI's dependency closure out of go.sum, and `go run pkg@version` is rejected because pkg/v3's go.mod contains replace directives; the tool directive records the full closure and stays in lockstep with the pkg/v3 requirement. - tests/testprovider: the Go test provider used by TestDotnetErrorHooks is copied to integration_tests/testprovider_go. It runs shimless via its PulumiPlugin.yaml, so it needs no separate build step. The C# testprovider cannot replace it because only the Go provider returns the ErrorResourceInitFailed detail that marks a create failure as retryable. CI no longer checks out submodules.
The codegen tests read the shared test corpus (schemas, PCL programs, types.json) from tests/testdata/codegen in pulumi/pulumi, previously reached through the submodule. At 11MB across ~2,250 files the corpus is too large to vendor, so `make test_codegen` now fetches it into the gitignored pulumi-language-dotnet/codegen/testdata/upstream directory via a sparse clone of the version pinned in go.mod. A version stamp skips the download when the corpus is already current. The script uses cp instead of rsync so it also runs under Git Bash on Windows.
The branch now stacks on a pin to an unreleased pulumi/pulumi master commit, which sync_protos.sh and get_codegen_testdata.sh could not handle: a pseudo-version has no tag to clone. Factor the sparse checkout into scripts/checkout_pulumi.sh, which resolves a pseudo-version to its commit through the Go module proxy and fetches it by hash. Re-sync the vendored protos and automation-overrides.json at the pinned commit (cd588358a1).
Its only consumer was TestDotnetErrorHooks, which was removed in favor of the l2-resource-hook-on-error conformance test.
|
|
iwahbe
force-pushed
the
iwahbe/drop-submodule
branch
from
July 17, 2026 12:14
8be0567 to
2c7f3c8
Compare
Frassle
reviewed
Jul 17, 2026
Frassle
left a comment
Frassle
left a comment
Member
There was a problem hiding this comment.
I'm really not convinced this is better than just the submodule.
The testdata changes are fine, if we want to pull that via go instead of the submodule that makes sense. But for the proto files the submodule seems way simpler.
Member
Author
|
I'm not happy with a bunch of stuff in this PR, and I haven't read through carefully. Package gen tests are a real drag here. |
iwahbe
force-pushed
the
iwahbe/error-hook-conformance
branch
from
July 29, 2026 09:17
2d01193 to
fb28bf6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #1076: this branch is rebased onto
iwahbe/error-hook-conformanceand targets it as the PR base. It inherits that PR's pin to an unreleased pulumi/pulumi master commit (cd588358a1); the sync scripts below handle pseudo-version pins for that reason.Summary
Removes the
pulumigit submodule. Everything it provided is replaced by an alternative pinned to the same source of truth: thegithub.com/pulumi/pulumi/{pkg,sdk}/v3version inpulumi-language-dotnet/go.mod.The
.protofiles cannot be sourced from the Go module graph — thepkgandsdkGo modules ship only the generated Go code (sdk/v3/proto/go), not the.protosources, which live at the repo root of pulumi/pulumi outside both module boundaries. They are vendored instead.Changes
sdk/Pulumi/Pulumi.csproj): vendored underproto/at the repo root, along with the Automation API CLI-spec overrides file (tools/automation/automation-overrides.json→sdk/Pulumi.Automation.Codegen/).make sync_protosre-syncs both from a sparse clone of pulumi/pulumi at the pinned version. TheformatCI job re-runs the sync and fails on drift, so ago.modbump without a proto sync cannot merge.make generate_cli_spec: builds the pulumi CLI through a Gotooldirective inpulumi-language-dotnet/go.mod(go tool pulumi generate-cli-spec).go runof the dependency's main package fails under module graph pruning (the CLI's dep closure isn't ingo.sum), andgo run pkg@versionis rejected becausepkg/v3's go.mod containsreplacedirectives. The tool directive records the full closure and moves in lockstep with thepkg/v3requirement.types.json) fromtests/testdata/codegen. At 11MB across ~2,250 files it is too large to vendor, somake test_codegenfetches it into the gitignoredpulumi-language-dotnet/codegen/testdata/upstream/via a sparse clone of the pinned version, with a version stamp to skip re-downloads.scripts/checkout_pulumi.sh): the sparse checkout used bysync_protos.shandget_codegen_testdata.shis factored into a shared helper. A pseudo-version pin (like the one inherited from Generate resource hooks in dotnet programs; adopt the error hook conformance test #1076) has no tag to clone, so the helper resolves it to its commit through the Go module proxy and fetches by hash..gitmodules+ gitlink) and allsubmodules: recursivecheckouts from CI.An earlier revision vendored
pulumi/tests/testproviderasintegration_tests/testprovider_goforTestDotnetErrorHooks; that test was replaced by thel2-resource-hook-on-errorconformance test in #1076, so the vendored provider is gone again.Testing
make build_sdkcompiles against the vendored protos (noPulumi.xmldrift).make test_codegen: 235 tests pass, 18 skipped, with the corpus fetched at the pseudo-version.make test_automation_codegen: 57 tests pass (spec generated viago tool pulumi).make test_conformance TEST_FILTER=TestLanguage/l2-resource-hook-on-errorpasses.make lint_language_host,make lint_integration_testsclean.scripts/sync_protos.shround-trips with no diff at the pinned commit.