Skip pulumi install when there is nothing to resolve - #1298
Open
bpalermo wants to merge 1 commit into
Open
Conversation
Contributor
|
PR is now waiting for a maintainer to run the acceptance tests. This PR will only perform build and linting. |
The workspace controller called the agent's Install RPC unconditionally, with no check for whether the project had anything to resolve. Two cases where it does not, both of which turn wasted work into a stalled workspace. A project using runtime.options.binary runs an already-compiled program. Nothing needs resolving, but the workspace still requires a complete language toolchain, so a missing writable build cache, an unwritable module cache, or a compiler version that wants to download a toolchain each stops every update rather than degrading anything. It also forces the image to carry a toolchain the program does not need at runtime. Note install runs more often than it appears: it is gated by the pod's auto.pulumi.com/initialized annotation, so once per pod, but the source revision is part of the StatefulSet, so every new commit or Flux digest rolls a new pod and installs again. A project-info workspace is worse. It is bootstrapped from a Pulumi.yaml synthesized with only a name and runtime -- no program at all -- so that a Stack can be destroyed from backend state after its source is gone. Installing there fails for every runtime except yaml, and the failure stalls the workspace, so the Stack cannot destroy: it retains its finalizer and strands its cloud resources, which is what the source-free destroy path exists to avoid. The stalled-during-deletion finalizer release does not help, because it requires status.lastUpdate to be nil while reaching this path requires a prior successful update. - The agent's Install RPC now loads the project settings and returns without doing anything when the runtime declares a binary. The agent already read Pulumi.yaml at startup but kept only the runtime name, discarding the options map that carries it. The same rule is applied on the agent's own startup path, so it does not depend on how the agent is driven. This also skips plugin acquisition, which install would otherwise perform. The engine acquires missing plugins during an update by default, so this is normally invisible, but a deployment that disables automatic acquisition has to pre-seed them. - The operator skips the RPC entirely for a project-info workspace. It can decide that locally, and must: the synthesized project file is written with nil runtime options, so it drops runtime.options.binary even for a project that declared it, and no project-file-based detection can fire there. - New Workspace.spec.skipInstall disables the step outright, for a workspace whose image already has what the program needs. Because StackSpec.WorkspaceTemplate is a Workspace apply configuration, spec.workspaceTemplate.spec.skipInstall works from a Stack with no further plumbing. - InstallResult gains skipped/reason so an agent-side skip is visible. Without it the operator would log "Running pulumi install" and report Ready with no sign anything was skipped, trading one confusing log for another. Every skip, from either side, is logged with its reason and emits an InstallationSkipped event. Note that --skip-install, which the operator passes to every agent, was never the control it looks like: it only suppresses the agent's startup install so that installing happens through the RPC where the controller can report status. It is now commented as such, and spec.skipInstall is the user-facing setting. Fixes pulumi#1297 Fixes pulumi#1299 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bpalermo
force-pushed
the
fix/1297-skip-install-prebuilt-binary
branch
from
August 8, 2026 12:09
5578aef to
e163854
Compare
Contributor
|
PR is now waiting for a maintainer to run the acceptance tests. This PR will only perform build and linting. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
The workspace controller called the agent's
InstallRPC unconditionally, with no check for whether the project had anything to resolve. There are two such cases, and in both the wasted work turns into a stalled workspace.A prebuilt program (#1297)
A project using
runtime.options.binaryruns an already-compiled program, so there is nothing to resolve — but the workspace still needs a complete language toolchain. A missing writable build cache, an unwritable module cache, or a compiler version that wants to download a toolchain each stops every update rather than degrading anything, and none of it is visible from the Stack, which keeps displaying its previouslastUpdatemessage. It also forces the image to carry a toolchain the program does not need at runtime, ruling out a CLI-only base.One clarification on frequency, since it isn't obvious from the code: install is gated by the pod's
auto.pulumi.com/initializedannotation, so it runs once per pod — but the source revision is part of the StatefulSet, so every new commit or Flux digest rolls a new pod and installs again.A project-info workspace (#1299)
Worse, and the reason this PR covers two issues. The source-free destroy path from #1222 bootstraps a workspace from a
Pulumi.yamlsynthesized with only a name and runtime — deliberately no program — so a Stack can be destroyed from backend state after its source is gone. Install ran there too. Confirmed against Pulumi v3.256.0, one temp dir per runtime containing only that file:pulumi installgoerror: installing dependencies: `go mod tidy` failed(go.mod file not found)nodejsexit status 254dotnetunable to find program: dotnetyamlSo it fails for everything but YAML, the workspace stalls, and the Stack retains its finalizer and never destroys — the resource-orphaning outcome #1222 and #1233 exist to prevent. The stalled-during-deletion finalizer release at
stack_controller.go:997does not help: it requiresstatus.lastUpdate == nil, while reaching this path requiresstatus.projectInfo, which onlymarkStackSucceededsets.How each is decided
The agent handles the prebuilt binary.
Server.Installnow loads the project settings and returns without doing anything when the runtime declares abinary. The agent already readPulumi.yamlat startup but kept onlyRuntime.Name(), discarding the options map that carries it — the information was on hand and thrown away. The same rule is applied on the agent's own startup path, so behaviour does not depend on how the agent is driven.This had to live in the agent: the operator learns a project's runtime only from
ProjectInfoon an update result, i.e. after a successful operation, so at install time it knows nothing about the project. (For the same reason, #1297's first suggested option — havingpulumi installitself no-op — belongs inpulumi/pulumi's CLI;LocalWorkspace.Installis runtime-agnostic and just shells out. Worth filing upstream; this doesn't block it.)The operator handles the project-info workspace, because it can decide that locally — and because it must.
writeProjectFilepassesNewProjectRuntimeInfo(runtime, nil), so the synthesized project dropsruntime.options.binaryeven for a project that declared it; no project-file-based detection can ever fire on that path.Behaviour change worth calling out: skipping install also skips provider plugin acquisition, so this leans on the engine's automatic acquisition during an update. That is the default, so normally invisible — but a deployment that sets
PULUMI_DISABLE_AUTOMATIC_PLUGIN_ACQUISITIONmust pre-seed plugins in its image.An explicit opt-out
New
Workspace.spec.skipInstalldisables the step outright, for a workspace whose image already has what the program needs (vendored dependencies, or dependencies baked in). BecauseStackSpec.WorkspaceTemplateis a Workspace apply configuration, it works from a Stack with no further plumbing:Making skips visible
InstallResultgainsskipped/reason(both messages were previously empty, so this is additive and wire-compatible). Without it the operator would logRunning pulumi installand reportReadywith no sign that anything was skipped — trading one confusing log for another. Every skip, from either side, is logged with its reason and emits a NormalInstallationSkippedevent.On
--skip-install#1297 is right that this reads as intent and doesn't behave like it. It was never the control it looks like: the operator passes it to every agent so the startup install is suppressed and installing happens through the RPC instead, where the controller can report progress and failures on the Workspace. That is now stated in a comment at the call site, with
spec.skipInstallas the user-facing setting.Testing
agent/pkg/server: newtestdata/prebuiltfixture — runtimego, nogo.mod, plusoptions.binary. Deliberately the same shape as the existinguninstallablefixture, so the pair is the assertion: installing would fail here, and succeeding proves it was skipped rather than attempted. Plus a table test for the detection helper covering absent options, an emptybinary, and a non-string value (the options map ismap[string]any).operator/internal/controller/auto:mockAutomationServerpreviously ignored its request and recorded nothing, so it could not express "was not called" — it now countsInstallinvocations.TestWorkspaceSkipInstallcovers install-by-default,skipInstall: true,skipInstall: false, a project-info workspace (RPC never made, workspace still Ready, event emitted), and an agent-reported skip being surfaced.go build ./..., the full operator + agent suites, andgolangci-lintare clean, with no codegen drift. Generated protobuf was produced with protoc 29.3 to match the checked-in files, so the diff is limited to the new fields andagent_grpc.pb.gois untouched.make test-e2ehas not been run, and neither failure has been reproduced end-to-end against a live cluster.Related issues
Fixes #1297
Fixes #1299
🤖 Generated with Claude Code