[Tjh-UKN]

关联 RFC： msprobe #680

What this PR does / why we need it?

Add opt-in Runner V2 KV address-plan checks using standard SchedulerOutput bindings. Each engine keeps an independent request/block mirror and checks final attention metadata once per cache group. No Scheduler, allocator, V1 Runner or attention-operator monkeypatch is added.

The review base remains the upstream-main snapshot 28e9e97ba254b076a88275e8a275556cb9a11664. The feature is adapted from #17833.

Enable with VLLM_USE_V2_MODEL_RUNNER=1 and --additional-config '{"kv_guard_config":{"enabled":true,"action":"raise"}}'.

The adapters distinguish connector-admitted local bindings, speculative scheduler budgets and independent worker frontiers, decoder self/cross caches, and DCP/PCP rank-local address plans. Remote KV bytes, speculative acceptance policy and model numerical correctness are outside the address-plan contract. DBO is no longer rejected by configuration name, but this backend revision itself disables DBO; actual DBO execution is not claimed.

This follow-up also fixes the observed native PCP decode crash: an empty rank completes peer KV gathering/cache writes, then clears its output inside the existing transfer window and skips local attention. Active graph capture and DCP overrides retain their existing paths. This is a native backend fix, in a separate commit, rather than operator instrumentation.

The audit constructs large prefill slot plans with block runs, bypasses unnecessary flattening for flat slots, and reuses pinned storage while preserving the direct-copy fast path for repeated shapes. Copies must still complete before reuse, and violations still block output in raise mode.

How was this patch tested?

• Local: 222 debug contracts and seven native PCP contracts passed; mypy passed for 18 debug files. Pinned Ruff 0.14.0, Markdownlint, release YAML and whitespace checks passed. bash format.sh ci was attempted but failed on missing /bin/bash, python3 and shellcheck; this is not a full CI pass.
• Frozen deployment: 4,700 Ascend tracked blobs, 8,656 pinned-core files and 983 native resources verified. Core commit: 24420b7a341af37a9bc3186e6fc6062512e0aa58. The Catlass gitlink is recorded as a pointer; its checkout contents are not validated.
• PCP2 on 179/Qwen3-0.6B: eager and FULL_DECODE_ONLY (capture sizes 1/2), OFF/ON, single257→8, batch2→8 each, and single257→16. All 16 logical requests completed; 160 output token IDs matched within each OFF/ON comparison. Both ranks enabled the guard without coverage gaps. FULL capture completed on both ranks in both graph-mode arms; per-step graph replay was not directly traced. PCP tested d7b467de972e; all 906 runtime files and core/native maps are byte-identical at final f417980e0d42, whose only added file is the C8 test. The complete PCP bundle contains 90 hashed files.
• Final-head native C8 regression: passed in eager and FULL: 48 requests, 12,288 actual output token IDs, OFF repeat and ON plans/IDs equal. The test enqueues all eight inputs before stepping the engine, first requires two stable OFF waves, then compares ON using complete generated token IDs and normalized execution plans. It retains physical request-slot layouts separately. This is functional validation, not a throughput measurement.
• Original C8 HTTP differences were investigated independently: with the guard OFF throughout, changing only admission order reproduces all eight complete texts of the two old OFF branches, together with the changed per-prompt prefill splits. This establishes that the guard is not necessary for that observed divergence; no particular numerical operator is assigned as its root cause.
• Final-head real NPU D2H/event/output-gate preflight: five cases passed, including device mutation, delayed readers, buffer reuse, sticky raise, log mode and direct output. Intentional bad-plan cases are separate from healthy service runs.
• Isolated same-card metadata micro, old 0a79437ac059 versus final f417980e0d42, 40 hot samples per scenario: C1 prefill CPU audit median 1,501.77→364.59 µs; mixed C8 1,563.83→410.91 µs. C8 decode snapshot 260.28→257.43 µs. Alternating prefill/decode shapes no longer allocate pinned buffers on each switch. Copy counts/bytes match. These are metadata-only timings, not end-to-end TTFT or throughput results.

Final-head performance on 179: one Ascend 910B4, Llama-3.1-8B-Instruct BF16, FULL, 2,048 input/256 output tokens, block128, prefix cache off, chunked prefill on. OFF→ON→ON→OFF at C1/C8: eight trials, 220 requests; each trial exceeds 60 seconds. Both ON servers must report deferred audit. Server CPU0–15/client16–23; external resource sampling, no model hooks or profiler.

Concurrency	OFF → ON output tokens/s	Throughput drop	OFF → ON mean TTFT	TTFT increase
C1	17.3420 → 17.3371	0.028%	216.38 → 227.16 ms	+10.78 ms / +4.98%
C8	117.3379 → 117.0921	0.210%	854.20 → 874.94 ms	+20.74 ms / +2.43%

C1 paired throughput-drop range: +0.015% to +0.042%.

C8 paired throughput-drop range: +0.044% to +0.375%.

Each switch/concurrency has two repetitions; these observed ranges are not confidence intervals. Positive drop means ON is slower. The host is shared; RSS can double-count shared pages, and HBM snapshots are not peak memory.

Raw HTTP complete-text differences against the first OFF arm (ON1, ON2, OFF2): C1 [0, 0, 0]; C8 [8, 6, 5]. This HTTP benchmark does not fix admission/prefill plans and does not establish complete-text equivalence for this workload. The separate native regression fixes those plans and compares actual token IDs.

All C8 complete-text differences are confined to the initial eight requests; the remaining 40 responses match across all four arms.

Strict source/array verification passed with zero errors; 124 performance evidence files were hashed, including source bytes before/after, input/output lengths, all actual SSE intervals, affinity/configuration, warmup outputs and NPU preflight receipts.

All four server logs retain an EngineDeadError stack after the last successful HTTP response and API shutdown/SIGTERM. These teardown errors are preserved separately; the logs are not described as entirely error-free. No business-stage traceback, guard violation or coverage gap was observed.

Earlier adapter checks at 0a79437ac059 covered ordinary eager, asynchronous NGram (21 drafts/18 accepted/3 rejected per arm), genuine Mooncake transfer (257 external tokens/three remote blocks/read completion/ACK), random small Whisper offline engines, and a valid random MQA DCP2 service. These are earlier-head functional evidence, not new-head performance or production-model accuracy claims. The DCP2 run did not request output token IDs. Production encoder/decoder and MLA models, additional speculative proposers, DBO execution and unrecorded combinations remain unvalidated.

• vLLM version: v0.32.0
• vLLM main: vllm-project/vllm@24420b7