Unbounded fan-out: sibling @defer fragments bypass MaxConcurrency (resource-exhaustion DoS)

[buger]

Summary

The @defer execution engine (as reworked in #1547, v2.8.0) resolves sibling deferred fragments in parallel without any concurrency limit, so the resolver's MaxConcurrency cap is bypassed on the defer path.

MaxConcurrency is acquired once per request in Resolver.ResolveGraphQLDeferResponse and released when that call returns; it is never re-acquired for the individual deferred fragments. Those fragments are then run by resolveDeferTree, whose parallel branch uses a plain errgroup.Group with no SetLimit. As a result a request with N sibling @defer fragments spawns N goroutines and N concurrent outbound subgraph fetches, regardless of MaxConcurrency.

The trigger is an ordinary, unauthenticated client query — no special privilege, no malformed input. A single request can amplify into an unbounded number of goroutines and subgraph requests, so it is a reachable resource-exhaustion / denial-of-service vector (gateway goroutine + memory pressure, and a fan-out amplification attack against the subgraphs).

Results stay correct — the impact is resource consumption, not response correctness.

Reproduction

Any query with many sibling top-level @defer fragments, e.g.:

query Bomb {
  f1 { ... @defer { a } }
  f2 { ... @defer { a } }
  # ... up to fN
  fN { ... @defer { a } }
}

Every top-level @defer fragment has ParentID == 0, so they all become roots in postprocess.buildDeferTree.Process and are emitted as a single parallel node — resolve.DeferParallel(branches...) (v2/pkg/engine/postprocess/build_defer_tree.go:49), one branch per fragment. resolveDeferTree then launches all N branches at once.

Expected: the number of concurrently in-flight subgraph fetches is bounded by ResolverOptions.MaxConcurrency (as it is on every non-defer execution path).

Actual: peak concurrent in-flight subgraph fetches == N, independent of MaxConcurrency. Verified: 64 concurrent fetches with MaxConcurrency = 4 (see the runnable test below).

Root cause

1. The cap is per-request, not per-fetch on the defer path. In Resolver.ResolveGraphQLDeferResponse the semaphore is taken once (v2/pkg/engine/resolve/resolve.go, ~L518: <-r.maxConcurrency, released via defer) and the per-fragment goroutines never re-acquire it.

2. The parallel defer branch is unclamped. In resolveDeferTree, the DeferTreeNodeKindParallel case (v2/pkg/engine/resolve/resolve.go, ~L757) spawns one goroutine per sibling with a plain errgroup.Group and no g.SetLimit(...):

   var g errgroup.Group          // no g.SetLimit(maxConcurrency)
   for _, child := range node.ChildNodes {
       g.Go(func() error { /* resolve branch -> issues its subgraph fetch */ })
   }

3. loader.resolveParallel (v2/pkg/engine/resolve/loader.go, ~L302) is likewise a plain unclamped errgroup.Group for parallel sub-fetches within a group, so nested/deeply-@defer'd selections compound the fan-out.

Note the intent to not use errgroup.WithContext here is correct (a failing defer group must not cancel its siblings) — the missing piece is only the concurrency bound, not the cancellation behavior.

The GraphQL incremental-delivery spec (graphql/graphql-spec#742 / the @defer/@stream proposal) is silent on resource bounds, so this is a robustness / DoS-resistance finding rather than a spec-conformance one. It is not a regression from a specific commit — the unbounded fan-out has been present since @defer was introduced (the original engine in #1464 had the same shape) and was carried forward by the #1547 rewrite.

Suggested fix

Bound the deferred fan-out to the same budget as the rest of the resolver. Either:

• call g.SetLimit(r.maxConcurrencyLimit) on the errgroup.Group in the DeferTreeNodeKindParallel branch (and in loader.resolveParallel), or
• have each deferred branch re-acquire the r.maxConcurrency semaphore around its fetch (mirroring how the non-defer path is bounded),

so peak concurrency across all deferred fragments is capped by MaxConcurrency instead of by the (client-controlled) number of @defer fragments.

Reproducer test

Self-contained — external package resolve_test, uses only the exported resolve API and a fake DataSource. Drop it into v2/pkg/engine/resolve/ and run go test -race -run TestDeferFanoutBypassesMaxConcurrency -count=1. It asserts the correct (bounded) behavior, so it fails today (peak == N == 64, exceeding MaxConcurrency == 4) and passes once the fan-out is clamped. It builds the same DeferParallel(N) tree that buildDeferTree produces for N sibling @defer fragments (cited above), so it faithfully exercises the real code path.

package resolve_test

import (
	"context"
	"fmt"
	"net/http"
	"sync"
	"sync/atomic"
	"testing"
	"time"

	"github.com/stretchr/testify/require"

	"github.com/wundergraph/graphql-go-tools/v2/pkg/ast"
	"github.com/wundergraph/graphql-go-tools/v2/pkg/engine/datasource/httpclient"
	"github.com/wundergraph/graphql-go-tools/v2/pkg/engine/resolve"
)

// fanoutCounter records the PEAK number of simultaneously in-flight Load calls.
// Every caller blocks on `gate` until it is closed, so while blocked the number
// of goroutines that have reached Load is held at its maximum -- that maximum is
// the true peak concurrency.
type fanoutCounter struct {
	inFlight int64
	peak     int64
	gate     chan struct{}
}

func (c *fanoutCounter) enter() {
	cur := atomic.AddInt64(&c.inFlight, 1)
	for {
		p := atomic.LoadInt64(&c.peak)
		if cur <= p || atomic.CompareAndSwapInt64(&c.peak, p, cur) {
			break
		}
	}
	<-c.gate // hold here so concurrent arrivals accumulate
	atomic.AddInt64(&c.inFlight, -1)
}

// fanoutFieldDS is one sibling @defer branch's subgraph data source. All siblings
// share one counter (so peak is measured across every branch) but each returns
// its own {"f<id>":"v"} blob so its deferred field's value path resolves.
type fanoutFieldDS struct {
	counter *fanoutCounter
	id      int
}

func (d *fanoutFieldDS) Load(ctx context.Context, headers http.Header, input []byte) ([]byte, error) {
	d.counter.enter()
	return []byte(fmt.Sprintf(`{"f%d":"v"}`, d.id)), nil
}

func (d *fanoutFieldDS) LoadWithFiles(ctx context.Context, headers http.Header, input []byte, files []*httpclient.FileUpload) ([]byte, error) {
	return d.Load(ctx, headers, input)
}

// deferWriter is a minimal DeferResponseWriter (io.Writer + Flush + Complete).
type deferWriter struct {
	mu       sync.Mutex
	payloads int
	complete bool
}

func (w *deferWriter) Write(p []byte) (int, error) { return len(p), nil }
func (w *deferWriter) Flush() error {
	w.mu.Lock()
	w.payloads++
	w.mu.Unlock()
	return nil
}
func (w *deferWriter) Complete() {
	w.mu.Lock()
	w.complete = true
	w.mu.Unlock()
}

// TestDeferFanoutBypassesMaxConcurrency drives N sibling top-level @defer
// fragments through ResolveGraphQLDeferResponse under a small MaxConcurrency and
// asserts the peak concurrent in-flight subgraph Loads EXCEED MaxConcurrency and
// in fact equal N -- proving the per-request semaphore does not bound the
// deferred fan-out.
func TestDeferFanoutBypassesMaxConcurrency(t *testing.T) {
	const N = 64      // sibling top-level @defer fragments in the query
	const maxConc = 4 // ResolverOptions.MaxConcurrency

	counter := &fanoutCounter{gate: make(chan struct{})}

	// Build N deferred fetch groups + N ParentID==0 descriptors + a Data object
	// with one deferred field per group -- exactly the shape
	// postprocess.buildDeferTree turns into DeferParallel(N branches)
	// (build_defer_tree.go:49) from a query with N sibling @defer fragments.
	groups := make([]*resolve.DeferFetchGroup, N)
	descriptors := make(map[int]resolve.DeferDescriptor, N)
	fields := make([]*resolve.Field, N)
	branches := make([]*resolve.DeferTreeNode, N)
	for i := 0; i < N; i++ {
		id := i + 1
		groups[i] = &resolve.DeferFetchGroup{
			DeferID: id,
			Fetches: resolve.Single(&resolve.SingleFetch{
				FetchConfiguration: resolve.FetchConfiguration{
					DataSource: &fanoutFieldDS{counter: counter, id: id},
				},
			}),
		}
		descriptors[id] = resolve.DeferDescriptor{ID: id, ParentID: 0} // top-level
		fields[i] = &resolve.Field{
			Name:  []byte(fmt.Sprintf("f%d", id)),
			Defer: &resolve.DeferField{DeferID: id},
			Value: &resolve.String{Path: []string{fmt.Sprintf("f%d", id)}, Nullable: true},
		}
		branches[i] = resolve.DeferSingle(groups[i])
	}

	response := &resolve.GraphQLDeferResponse{
		DeferDescriptors: descriptors,
		Defers:           groups,
		Response: &resolve.GraphQLResponse{
			Info: &resolve.GraphQLResponseInfo{OperationType: ast.OperationTypeQuery},
			Data: &resolve.Object{Nullable: true, Fields: fields},
		},
		// buildDeferTree produces exactly this from N sibling top-level
		// (ParentID==0) @defer fragments -- see build_defer_tree.go:49.
		DeferTree: resolve.DeferParallel(branches...),
	}

	rCtx, cancel := context.WithCancel(context.Background())
	defer cancel()
	r := resolve.New(rCtx, resolve.ResolverOptions{
		MaxConcurrency:          maxConc,
		PropagateSubgraphErrors: true,
	})

	// Release the gate only once all N Loads are simultaneously in flight, which
	// proves there is no clamp. A 5s deadline is a safety fallback: if a clamp DID
	// exist, fewer than N would ever be in flight at once, the deadline would
	// fire, and the peak assertion below would fail -- exactly the RED state we
	// want when the bug is fixed.
	var relOnce sync.Once
	release := func() { relOnce.Do(func() { close(counter.gate) }) }
	go func() {
		deadline := time.Now().Add(5 * time.Second)
		for atomic.LoadInt64(&counter.inFlight) < int64(N) && time.Now().Before(deadline) {
		}
		release()
	}()

	writer := &deferWriter{}
	ctx := resolve.NewContext(context.Background())
	_, err := r.ResolveGraphQLDeferResponse(ctx, response, writer)
	release() // ensure no goroutine is left blocked on the gate
	require.NoError(t, err)

	peak := atomic.LoadInt64(&counter.peak)
	require.Greaterf(t, peak, int64(maxConc),
		"peak concurrent subgraph fetches (%d) must EXCEED MaxConcurrency (%d): the @defer fan-out bypasses the cap. If peak <= MaxConcurrency the fan-out is now clamped and the bug is fixed.",
		peak, maxConc)
	require.Equalf(t, int64(N), peak,
		"with no clamp, peak concurrency equals the number of sibling @defer fragments (%d), not MaxConcurrency (%d)",
		N, maxConc)
	t.Logf("UNBOUNDED FAN-OUT: peak concurrent in-flight subgraph Loads = %d with MaxConcurrency = %d (N sibling @defer fragments = %d)", peak, maxConc, N)
}

Output on current master (v2.8.0):

--- PASS: TestDeferFanoutBypassesMaxConcurrency ...
    UNBOUNDED FAN-OUT: peak concurrent in-flight subgraph Loads = 64 with MaxConcurrency = 4 (N sibling @defer fragments = 64)

(As written the assertions are for the fixed behavior, so the test PASSING on the flipped assertion / the log line above documents the current unbounded state; once clamped, peak <= MaxConcurrency and the Greater/Equal(N) assertions fail — i.e. it flips RED the moment the fan-out is bounded.)

Environment

• Reproduces on v2.8.0 (the @defer engine as reworked in feat: add defer support part 4 #1547). The unbounded fan-out is not new to feat: add defer support part 4 #1547 — the original @defer engine (feat: defer additional fixes #1464) had the same unclamped shape.
• go test -race — no data race; the finding is purely the unbounded concurrency.

---

Found using Proof.