[Feature] Decoder Append-Prefill for Agentic Scenario Prefiller Offloading by Ydr-pku · Pull Request #19884 · vllm-project/vllm-ascend

Ydr-pku · 2026-10-27T01:28:19Z

What this PR does / why we need it?

Agentic workloads often have long inputs—tens of thousands of tokens from user and system prompts—but short outputs, such as tool-call instructions of around a hundred tokens. In P/D-disaggregated serving, this imbalance concentrates pressure on Prefiller instances. Even when multiple Prefillers serve one Decoder (nP1D), a fixed P/D ratio cannot dynamically adjust prefill capacity to fluctuating request rates, leaving Prefillers vulnerable to sharp load increases and queueing during request bursts.

Successive turns in an agentic session often extend the same conversation history, so the Decoder that served an earlier turn may still hold a reusable prefix in its KV cache. This creates an opportunity to attempt prefill for a follow-up request on that Decoder, reusing the cached prefix and computing the uncached suffix to reduce Prefiller load. This PR introduces Decoder Append-Prefill to support this approach, with a fallback to the Prefiller when no usable local prefix remains.

Add decoder_append_prefill_proxy_server.py to route subsequent requests from the same session to its owner Decoder instance and owner DP rank.
Check actual local KV-cache hits in the Scheduler before admitting Decoder-local prefill. If no usable local prefix remains, reject the local attempt before generating output so the proxy can retry through the Prefiller and the normal P/D path. Otherwise, reuse the prefix and perform append-prefill on the Decoder.
Enable append-prefill admission through scheduler_config.decoder_append_prefill_enable in each Decoder's --additional-config. The platform automatically selects the synchronous/asynchronous class and its combined Recompute variant when configured. The feature defaults to disabled, preserving existing native/custom Scheduler handling. Starting the proxy does not configure remote backends.
Support synchronous/asynchronous scheduling and Recompute/offload restoration, including recovery of the same preempted request and no replay after output. Recovery-state admission is contained in the feature Scheduler, which queries existing connector hooks through the public MultiConnector.sub_connectors interface.
Add a dedicated Decoder Append-Prefill guide with setup, session routing, fallback behavior and optional Recompute/offload usage.

The design and benchmark benefits for the target deployment were discussed and recognized in the weekly community meeting recorded in #10928 (2026.10.23).

Does this PR introduce any user-facing change?

Yes. The standalone proxy adds opt-in --enable-decoder-append-prefill routing. To use Decoder Append-Prefill, clients must send the same session ID across turns, using either the X-Session-ID header or a nonempty conversation_id string in the JSON body. The header takes precedence when both are provided. Requests without a session ID use normal P/D routing and do not attempt Decoder-local append-prefill. Clients must still send the complete conversation history with every request.

Deployment configuration must guarantee that every Decoder uses kv_consumer, prefix caching and scheduler_config.decoder_append_prefill_enable=true in --additional-config. The platform selects the matching synchronous/asynchronous Scheduler and, when recompute_scheduler_enable=true is also set, the combined Recompute class; no --scheduler-cls is required. This includes newly added or restarted Decoders. The proxy trusts that deployment configuration and performs no capability probing; this PR adds no capability endpoint or API-router patch. Per-request KV residency is still checked by the Scheduler, and an admission cache miss still falls back through P/D.

Request examples

The following examples show a follow-up turn after an initial request has completed with the same session ID, conversation-001. Enable the proxy and Decoder options above, replace the example proxy address and model name, and supply the original system prompt, full user input and actual assistant reply from the preceding turn.

curl with the session ID in the HTTP header

curl -sS http://127.0.0.1:9000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'X-Session-ID: conversation-001' \
  -d '{
    "model": "your-model",
    "messages": [
      {"role": "system", "content": "<original system prompt>"},
      {"role": "user", "content": "Analyze this report: <full report>"},
      {"role": "assistant", "content": "<actual reply from the preceding turn>"},
      {"role": "user", "content": "Suggest three improvements."}
    ]
  }'

JSON body with the session ID in conversation_id

Alternatively, put the same session ID in the JSON body. Save the following as request.json; the X-Session-ID header can be omitted.

{
  "model": "your-model",
  "conversation_id": "conversation-001",
  "messages": [
    {"role": "system", "content": "<original system prompt>"},
    {"role": "user", "content": "Analyze this report: <full report>"},
    {"role": "assistant", "content": "<actual reply from the preceding turn>"},
    {"role": "user", "content": "Suggest three improvements."}
  ]
}

Send this JSON as the HTTP request body:

curl -sS http://127.0.0.1:9000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @request.json

The first turn uses normal P/D. Later turns with the same session ID attempt Decoder-local append-prefill and fall back through P/D if no usable local KV prefix remains.

How was this patch tested?

Decoder Append-Prefill supports both Model Runner V1 (MRV1) and Model Runner V2 (MRV2). The performance comparison below reports MRV2 results.

Qwen3 performance with MRV2

Model. Qwen3-30B-A3B-Instruct-2507.
Hardware. Three NPUs.
Deployment. P TP1 and D TP1/DP2/EP2.
Execution settings. BF16 weights, eager execution, asynchronous scheduling and chunked prefill.
Runtime versions. vLLM 544f182d9f0c9bfe8bd7645525951d5e3692a542 and vLLM-Ascend 1d848d299e8765b980085fe4346c6a7a19fb542f.
Model runner. MRV2 enabled in both arms, verified through actual initialized worker objects and source identities.
Workload. The same frozen workload of 90 sessions with three turns each for both arms.
Arrival window. New sessions arrived over 120 seconds.
Inter-turn think time. Fixed at 10 seconds.
Initial prompt length. 12,254–12,288 tokens.
Output budget. At most 64 tokens per request.
Conversation history. Later turns used actual completed assistant responses.
Baseline. Ordinary P/D for every turn.
Feature arm. Local append attempts on the owner Decoder for follow-ups, with a P/D fallback on an initial local admission miss.
Execution order and cache isolation. The baseline completed before the feature arm started, with separate cache namespaces between arms.

Metric	Baseline, ordinary P/D	Decoder Append-Prefill	Gain (+) / loss (-)
Protocol-valid / answer-correct requests	270 / 269	270 / 269	Unchanged
Local append hits / follow-ups	N/A	113 / 180	N/A
Safe P/D fallbacks / follow-ups	N/A	67 / 180	N/A
Output throughput, tokens/s	8.156	13.179	+61.58%
Request throughput, requests/s	0.9145	1.4777	+61.58%
Total completion time, s	295.233	182.717	+38.11%
TTFT p50, s	45.936	4.521	+90.16%
TTFT p95, s	68.007	30.243	+55.53%
TPOT p50, ms	209.5	229.6	-9.60%
TPOT p95, ms	216.0	345.5	-59.98%
Prefiller queue peak in the arrival window	60	27	+55.00%

Gains use unrounded values. Higher throughput and lower elapsed time, latency or queue length count as positive gains. Output throughput is the total number of output tokens divided by the total completion time, including output from the single answer-scoring failure in each arm. Request throughput counts protocol-valid completions over that interval. Total completion time runs from the workload start until the last request completes, including think time and the completion tail, with model loading excluded. TTFT starts at turn eligibility and includes client and queueing time.

The MRV2 run exercised 113 real local append operations and 67 verified zero-output misses followed by one P/D fallback each. Both DP ranks were covered by warm-hit and intentional-miss preflight probes. Local-hit records were joined to native Scheduler admission and checked against same-backend P/D READ logs.

The feature reduced Prefiller queueing and improved throughput and TTFT in this workload, while TPOT increased. Additional prefill work extends the duration of mixed prefill/decode steps on the Decoder, increasing TPOT, and we are designing an approach to reduce this impact.

Compatibility with PreemptOffload

PreemptOffload is supported with Decoder Append-Prefill in the tested MRV2 configuration.

Decoder Append-Prefill targets P-heavy workloads, shifting follow-up prefill work to a Decoder with available capacity. PreemptOffload targets D-heavy workloads, saving and restoring active-request KV when Decoder memory pressure causes preemption. In their intended operating regimes, the two features typically do not trigger together.

We adapted the combined Scheduler and validated the overlap with the following setup.

Model and runner. Qwen3-30B-A3B-Instruct-2507 with MRV2.
Feature configuration. Decoder Append-Prefill and PreemptOffload both enabled.
Workload. A separate KV-pressure workload constructed to force both paths within the same request.

With both features enabled, 72/72 formal requests passed protocol and answer validation. Native traces confirmed one complete formal same-request chain from local append through preemption, device-to-host KV save, CPU state ready, host-to-device restore and continued generation to 512 output tokens. This verified functional compatibility when the two features were deliberately triggered together.

vLLM main: vllm-project/vllm@24420b7

github-actions · 2026-10-27T01:29:22Z

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

A PR should do only one thing, smaller PRs enable faster reviews.
Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

Tip

💡 Consider Linking a Related Issue or RFC

Your PR title contains the [Feature] tag, indicating a bug fix or new feature.

Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:

Fixes #<issue_number>
Closes #<issue_number>
Resolves #<issue_number>
Refs #<rfc_or_issue_number> (for RFCs)

🙏 Thanks for helping us keep the project well-organized!

gemini-code-assist · 2026-10-27T01:30:23Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces 'Decoder Append-Prefill' to optimize agentic workloads by reusing KV-cache resident on Decoder instances. By routing subsequent session turns to the same Decoder, the system reduces redundant prefill work. The changes include a new proxy server for session-affine routing, enhanced scheduler admission logic to handle cache misses gracefully, and platform-level configuration validation to ensure feature compatibility.

Highlights

Decoder Append-Prefill Proxy: Introduced a standalone proxy server that enables session-affine routing to Decoder instances, allowing incremental prefill work to be offloaded from the Prefiller.
Scheduler Admission Control: Implemented admission logic in the Scheduler to reject local prefill attempts if the required KV-cache prefix is missing, triggering a fallback to the standard Prefiller/Decoder path.
Platform Integration: Added automated capability discovery and policy validation to ensure the Decoder-local append-prefill feature is only enabled with compatible configurations.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist · 2026-10-27T01:34:11Z

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

github-actions · 2026-10-27T02:15:55Z