Go SDK v6 compile is GC-bound: ~13x slower than v5, OOMs 7GB CI runners

[peter-ambiorix]

Summary

Compiling sdk/v6/go/cloudflare is GC-bound, not compute-bound. The package
is a single Go compilation unit with ~22k types and ~62k methods, so the
compiler holds a multi-GB pointer-dense live heap and the GC rescans it
repeatedly. On identical hardware and flags, sdk/v5 compiles the equivalent
package in 26 s; sdk/v6 takes 326 s — ~13× slower. This reliably
OOMs / gets SIGTERM-killed on standard 2-core / 7 GB CI runners.

Related to #1383 (closed as no-repro) — full repro, profile, and measurements
in that comment.
Opening as a separate issue because the root cause is now pinned and actionable,
which a closed thread won't surface in triage.

Minimal repro

No Pulumi program needed; building the SDK package alone reproduces it.

mkdir cfrepro && cd cfrepro
printf 'module cfrepro\ngo 1.24\n' > go.mod
GOFLAGS=-mod=mod go get github.com/pulumi/pulumi-cloudflare/sdk/v6@latest
GOCACHE=$(mktemp -d) time GOFLAGS=-mod=mod go build github.com/pulumi/pulumi-cloudflare/sdk/v6/go/cloudflare

Swap v6 for v5 for the comparison. A cold, isolated GOCACHE is required —
a warm cache hit takes no time and hides the problem.

Measurements

Same machine, cold build cache, warm module cache, GOOS=linux GOARCH=amd64,
Apple M3 Pro / 18 GB RAM, go1.25.11, machine otherwise idle (verified no swap
pressure before/after each run).

SDK	.go files	Package size	Peak RSS	Wall	CPU (user+sys)
sdk/v5 v5.49.1	206	7.1 MB	3.32 GB	26 s	98 s
sdk/v6 v6.19.0	666	27 MB	4.53 GB	326 s	769 s

Source grew ~3.9×, compile time grew ~13× — superlinear.

	v5	v6	ratio
type declarations	5,797	22,386	3.9×
methods	15,934	62,397	3.9×
ApplyT / ApplyTWithContext call sites	5,813	23,645	4.1×
RegisterOutputType / RegisterInputType	3,028	11,562	3.8×
total LOC	148,206	552,189	3.7×

Root cause: the compile is GC-bound

Profiling the compiler while it builds only this package (deps pre-warmed):

go build -gcflags="github.com/pulumi/pulumi-cloudflare/sdk/v6/go/cloudflare=-cpuprofile=cf6.prof" \
  github.com/pulumi/pulumi-cloudflare/sdk/v6/go/cloudflare
go tool pprof -top cf6.prof

      flat  flat%   sum%        cum   cum%
   396.13s 50.10% 50.10%    396.17s 50.10%  runtime.(*mspan).heapBitsSmallForAddr
    47.13s  5.96% 56.06%     47.13s  5.96%  runtime.memclrNoHeapPointers
    22.29s  2.82% 58.88%    418.50s 52.93%  runtime.(*mspan).typePointersOfUnchecked
    20.03s  2.53% 66.65%    470.46s 59.50%  runtime.scanobject
    16.76s  2.12% 68.77%    103.24s 13.06%  cmd/internal/obj.WriteObjFile

runtime.scanobject is 59.5% cumulative; small-object pointer scanning
(heapBitsSmallForAddr) is 50% flat. The largest genuine compiler cost,
WriteObjFile, is 13%. GC cost scales with live-heap size × cycle count, which
is consistent with a 3.9× code-size increase producing a 13× wall-time increase.

GC tuning does not remove the problem

Because a maintainer's first instinct will reasonably be "tune GOGC" — already
measured, so it can be ruled out up front:

Setting	Wall	Peak RSS
default (GOGC=100)	326 s	4.53 GB
GOGC=off	539 s	4.42 GB

GOGC=off does confirm the diagnosis — CPU (user+sys) drops from 769 s to
418 s, a 1.8× reduction — but wall time gets worse. Turning the GC off doesn't
shrink the live heap; the compiler just keeps allocating into it unchecked, and
absent periodic collection the allocator's own bookkeeping cost grows instead.
The multi-GB live heap is the thing that has to shrink; it isn't a tunable
from the consumer side.

Impact

Reliably kills builds on standard 2-core / 7 GB CI runners: the job is
terminated mid-compile at a varying point (observed via SIGTERM, exit 143), so
it presents as a flaky/hanging step rather than an obvious OOM. Because the
build never completes, actions/setup-go's cache-save step never runs, so
every subsequent run recompiles cold and fails the same way — a fresh
checkout can never get past this step on a small runner.

Suggested direction

The v5→v6 regression is structural: v6 puts the whole SDK in one package
(sdk/v6/go/cloudflare, 666 files). Go compiles a package as a single
indivisible unit, so the whole type graph must be live at once and the work
can't be split across cores (confirmed: -p 1 vs default parallelism barely
moves peak RSS, since the cost is one compilation unit, not concurrent ones).

pulumi-aws's Go SDK splits into per-service subpackages (aws/s3, aws/ecs,
aws/iam, …), so no single compilation unit gets pathologically large, and a
consumer only pays for what it imports. Splitting the Cloudflare Go SDK into
subpackages by resource family (Zone, DNS, Access, Workers, …) would cap
per-package heap size, restore build parallelism, and let most consumers —
who use a small fraction of the surface — compile only that slice.

If splitting the package isn't feasible short-term, reducing the generated
surface per logical type (Output/PtrOutput/ArrayOutput/MapOutput +
their Input counterparts — roughly 6 wrapper types per type today) would cut
the live heap roughly proportionally.

Happy to test a patched SDK against this repro, or share the full CPU/heap
profiles.

[corymhall]

@peter-ambiorix thanks for reporting this! I'm working on a POC where we duplicate everything into separate smaller modules. This would maintain backwards compatibility while allowing users a patch to migrate to the new module structure. Would that type of solution work for you?

[peter-ambiorix]

Thanks for jumping on this. Directionally that's exactly the fix — splitting the compilation unit is what caps the heap.

One question that determines how much it helps our case: would DNS + Zone resources land in their own tight module, or get bundled together with other resource families in the split? Our consumer only needs 5 symbols (NewDnsRecord, LookupZone, LookupZoneArgs, GetZoneFilter, DnsRecordArgs) — if those get isolated cleanly, that's close to a v5-sized module for us. If the split is coarser, the win might be partial.

Happy to test against a POC/branch once you have one, using the same repro from the issue above — real numbers rather than guessing. Also curious if there's a rough timeline, since we're actively deciding on an interim workaround on our end and that affects how much we lean on waiting for this.