Skip to content

perf: Reduce allocations on the response-body streaming path - #2252

Open
albertoperdomo2 wants to merge 1 commit into
llm-d:mainfrom
albertoperdomo2:perf/reduce-response-streaming-allocs
Open

perf: Reduce allocations on the response-body streaming path#2252
albertoperdomo2 wants to merge 1 commit into
llm-d:mainfrom
albertoperdomo2:perf/reduce-response-streaming-allocs

Conversation

@albertoperdomo2

@albertoperdomo2 albertoperdomo2 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

The EPP response-body hot path allocates on every streaming chunk for operations that are either unnecessary at that phase or can be simply avoided. Under sustained load (~40 req/s, streaming, gpt-oss-120b) GC dominates the CPU profile (lfstack.pop, gcDrainN, and getempty together consume over 50% of total CPU time).

Which issue(s) this PR fixes:

This PR targets the three cheapest wins on that path:

  1. Per-phase metadata extraction instead of every-request: ExtractMetadataValues was called unconditionally before the phase switch, allocating a map on every message including intermediate response-body chunks that never use metadata. Moved the call into the three cases that consume it (request headers, end-of-stream body, response headers) so intermediate chunks pay nothing.

  2. Guarded TRACE logging with .Enabled() checks: HandleResponseBody and runResponseBodyPlugins are called per chunk. Unconditional logger.V(TRACE).Info(...) calls allocate zap encoder clones even when TRACE is disabled. Added .Enabled() guards on the per-chunk path.

  3. Cached plugin.TypedName() into a local variable: Each run*Plugins loop called plugin.TypedName() 2-4 times per plugin per request (for logging and metrics). TypedName() is an interface method returning a two-string struct; caching it in a local avoids repeated virtual dispatch and copies.

Test plan: All existing a new added tests pass.

Profiles (30s window, ~1,240 requests, openai/gpt-oss-120b) collected on the same node under the same load (~40 req/s steady-state). PRE uses the upstream v0.9.0 image; POST uses the perf/reduce-response-streaming-allocs build.

Runtime metrics (go_memstats, 30s delta):

Metric PRE (v0.9.0) POST (perf/reduce-response-streaming-allocs) Delta
Bytes allocated 8.02 GB 7.89 GB -1.6%
Malloc count 91.5M 90.0M -1.7%
GC cycles 43 35 -18.6%
GC wall time 43.8 ms 40.6 ms -7.3%
Total CPU (all cores) 243.1 s 182.4 s -25.0%

Response-body path allocations (pprof alloc_objects, focused on HandleResponseBody + related):

Function PRE (v0.9.0) POST (perf/reduce-response-streaming-allocs) Delta
ExtractMetadataValues (flat) 1,518,320 0 -100%
Director.HandleResponseBody (cum) 17,110,843 16,404,916 -4.1%
HandleResponseBody full path (cum) 26,797,938 24,228,775 -9.6%
zapLogger.WithValues (flat) 4,718,807 4,423,882 -6.3%

CPU breakdown: GC + memory management (lfstack.pop, gcDrainN, getempty, gcDrain):

PRE (v0.9.0) POST (perf/reduce-response-streaming-allocs) Delta
GC-related CPU 125.9 s 57.8 s -54.1%

The top-line allocation reduction is modest (-1.6% bytes) because the optimized functions are small allocators relative to the total, but the GC impact is outsized: the response-body path fires per chunk (high frequency), so small per-call savings compound across thousands of chunks per window. The net effect is a 54% reduction in GC CPU and 18.6% fewer GC cycles under comparable load.

Release note (write NONE if no user-facing change):
NONE

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
@albertoperdomo2
albertoperdomo2 requested review from a team and shmuelk as code owners August 1, 2026 12:50
@github-actions github-actions Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. and removed kind/bug Categorizes issue or PR as related to a bug. labels Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant