__pulumi_raw_state_delta for deeply-nested resources exceeds the language SDKs' protobuf recursion limit — every update after the first up fails with "Protocol message had too many levels of nesting"

[marthursson]

What happened?

For resources with deeply-nested property paths, the raw-state delta introduced by the raw state
preservation rollout (#3003) is returned inside RegisterResourceResponse outputs as a nested
object tree deep enough to exceed the default protobuf recursion limit (100) in the language
SDK's protobuf runtime. The create succeeds — there is no prior state, so no delta — and every
subsequent pulumi up or pulumi preview touching the resource fails while the SDK parses
the response:

io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte sequence
  (underlying: Protocol message had too many levels of nesting.  May be malicious.
   Use CodedInputStream.setRecursionLimit() to increase the depth limit.)

We hit this from the Java SDK; pulumi/pulumi-dotnet#628 shows the identical exception from the
.NET SDK's protobuf runtime, so this is not language-specific — every SDK enforces the same
default limit and none exposes the knob.

Once the delta has been written to the statefile, pulumi preview fails too. The only recovery we
found is pulumi stack export, stripping __pulumi_raw_state_delta from the affected resource's
outputs, and pulumi stack import — after which the next up writes the delta again.

Why the depth explodes

Reading pkg/tfbridge/rawstate.go:

• RawStateDelta.Marshal() JSON-marshals the delta but then rebuilds it as a nested
  PropertyValue tree via resource.NewPropertyValueRepl (rawstate.go:304-328), which lands in
  the outputs map (rawstate.go:481) and is therefore Struct-encoded on the wire.
• Each plain object level in the source state costs ~3 JSON levels in the delta
  (delta node → obj → ps → child delta).
• Each Terraform MaxItems=1 flattened block costs ~5 (a plu → i pluralize wrapper around the
  object expansion).
• Leaf replace deltas are wrapped in secrets (rawstate.go:319-322), adding another object level
  each.
• On the wire, every JSON object level costs ~3 protobuf message frames
  (Struct → FieldsEntry → Value), and the SDK-side parser counts all of them against its
  100-frame default.

So a resource whose deepest written path is ~8 logical levels — where several levels are
MaxItems=1 blocks — lands at roughly 90–105 protobuf frames and crosses the limit.

Reproduction

Any bridged resource with a deep enough property path triggers it; the reliable public repro we
hit (twice, on independent stacks, months apart) is a GCP Cloud Run v2 Job with a secret env
var — the path
template.template.containers[*].envs[*].valueSource.secretKeyRef.{secret,version} stacks four
MaxItems=1 blocks on top of two arrays:

1. Java Pulumi program: gcp.cloudrunv2.Job with one container env using
   valueSource.secretKeyRef.
2. pulumi up → succeeds (create, no delta yet).
3. Change anything on the Job (e.g. add a plain env var); pulumi up → fails as above while
   parsing RegisterResourceResponse.

Supporting evidence that it's purely a depth cliff: a Cloud Run v2 Service with the identical
secretKeyRef shape — one template level shallower — is unaffected.

Expected behavior

Updates succeed regardless of resource nesting depth. Possible directions:

• Encode the delta as a single JSON string property instead of a nested Struct tree —
  UnmarshalRawStateDelta already round-trips through json.Marshal of the mappable value
  (rawstate.go:330-340), so a string encoding looks nearly drop-in and removes the depth
  amplification entirely.
• Alternatively, cap or flatten the delta encoding depth, or have the SDKs raise
  setRecursionLimit (none currently exposes it).

Environment

• com.pulumi:pulumi (Java SDK) 1.35.0, com.pulumi:gcp 9.33.0
• Pulumi CLI v3.256.0
• macOS / Linux CI, JVM 25

[corymhall]

@marthursson thanks for reporting this! It sounds like the best approach to take might be to just encode it as a JSON string. I'll try to get a PR up soon.

Edit: the other alternative which may be easier is to bump the limit on the SDK side.