pulumi install runs on every update even when the program is a prebuilt binary

[bpalermo]

Version: v2.9.0

Summary

The workspace controller calls the agent's Install RPC unconditionally on every Workspace reconcile, with no check for whether the project actually needs dependency resolution. For a project using runtime.options.binary — a prebuilt program — there is nothing to install, yet a failure there marks the Workspace InstallationFailed and no update ever runs.

The --skip-install flag does not help: it gates only the agent's serve-time path, not the RPC.

// operator/internal/controller/auto/workspace_controller.go:324
l.Info("Running pulumi install")
ready.Reason = "Installing"
_, err = wc.Install(ctx, &agentpb.InstallRequest{})
if err != nil {
    l.Error(err, "unable to install; marking workspace as stalled")
    ready.Reason = "InstallationFailed"

// agent/cmd/serve.go:154 — the flag the operator passes at :533
if !_skipInstall {
    ... auto.InstallOptions ...
}

Both lines appear in a single workspace's log, which is what makes the behaviour confusing in practice:

INFO  cmd.serve  installation skipped   {"project":"...","runtime":"go"}   ← startup, --skip-install honoured
INFO  server     installing the project dependencies                        ← per reconcile, via the RPC
ERROR server     install completed with an error

Why it matters beyond wasted work

We moved a Go project to a prebuilt binary (runtime.options.binary) specifically to stop compiling inside the workspace — the compile peaked near 1 GiB RSS, was OOM-killed, and died holding the S3 backend lock, wedging most of a ~290-Stack fleet.

Because install still runs, the workspace still needs a complete Go environment, and each missing piece is a total outage rather than a degradation. Three separate failures, each surfacing only after the previous was fixed:

	failure	why
1	determining go version: exit status 1	go.mod required a newer toolchain than the image shipped, so Go tried to download one; the workspace has no egress to the module proxy
2	failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied	the operator forces runAsUser: 1000; the image had no home for that uid, so HOME=/ and GOCACHE=/.cache/go-build
3	go: could not create module cache: mkdir /go/pkg: permission denied	GOPATH=/go is set in the image, so GOMODCACHE=/go/pkg/mod, also unwritable by uid 1000

None of these would exist if install were skipped for a prebuilt program. And none is visible from the Stack: the Workspace goes Stalled/InstallationFailed while the Stack keeps displaying its previous lastUpdate message, so the fleet showed 287 Stacks with unchanged status while none progressed.

It also constrains the image. The point of a prebuilt binary is that the workspace only has to run the program, which would allow a CLI-only base (pulumi/pulumi-base, ~200 MB). Because install runs, the image must carry a full Go toolchain and writable caches, so the smaller base is not usable.

Expected

Either of:

• pulumi install no-ops when runtime.options.binary is set — there is nothing to resolve for an already-compiled program.
• The operator skips the Install RPC for such projects, or honours a Workspace-level opt-out, rather than marking the Workspace InstallationFailed.

A third, weaker option: keep running it but treat failure as non-fatal when the program is prebuilt, so a broken install cannot stop updates that do not depend on it.

Note on --skip-install

The operator already passes --skip-install to the agent (workspace_controller.go:533), which reads as intent to skip installation — but it only affects serve.go's startup path while the RPC the operator itself calls is unguarded. Whatever the resolution, having one flag cover both would remove a real footgun.