[Feature][Debug] Check V2 KV plans across service execution layouts

docs/source/user_guide/configuration/additional_config.md

@@ -38,6 +38,7 @@ The following table lists additional configuration options available in vLLM Asc
 |-------------------------------------|------|---------|-----------------------------------------------------------------------------------------------------------|
 | `xlite_graph_config`                | dict | `{}`    | Configuration options for Xlite graph mode                                                                |
 | `finegrained_tp_config`             | dict | `{}`    | Configuration options for fine-grained tensor parallelism. See [Fine-Grained Tensor Parallelism](../feature_guide/Fine_grained_TP.md). |
+| `kv_guard_config`                  | dict | `{}`    | Per-engine KV cache address diagnostics for Model Runner V2. `enabled` defaults to `false`; `action` accepts `"log"` (default) or `"raise"`. See [KV cache address guard](../feature_guide/kv_cache_guard.md). |
 | `ascend_compilation_config`         | dict | `{}`    | Configuration options for ascend compilation                                                              |
 | `ascend_warmup_config`              | dict | `{}`    | Configuration options for startup warmup that overlaps weight loading                                     |
 | `eplb_config`                       | dict | `{}`    | Runner-specific EPLB extensions. See [Expert Parallelism Load Balancer](../feature_guide/expert_parallelism_load_balancer.md). |

docs/source/user_guide/feature_guide/kv_cache_guard.md

@@ -0,0 +1,118 @@
+# KV cache address guard
+
+KV Guard in Model Runner V2 compares the scheduler's declared token destinations with the runner's
+final attention metadata, once per KV cache group. Eager and FULL graph modes
+use the same metadata-plan checks. Attention implementations are unchanged;
+operator arguments and per-layer execution order are not recorded.
+
+## Enable the guard
+
+The guard is disabled by default and requires Model Runner V2. Enable it for an engine with:
+
+```bash
+VLLM_USE_V2_MODEL_RUNNER=1 vllm serve MODEL --additional-config '{"kv_guard_config":{"enabled":true,"action":"raise"}}'
+```
+
+| Option | Type | Default | Meaning |
+| --- | --- | --- | --- |
+| `enabled` | bool | `false` | Enable worker metadata-plan checks. |
+| `action` | `"log"` or `"raise"` | `"log"` | Report failures or stop returning worker results. |
+
+The Python API accepts the same `additional_config` dictionary. Unknown fields,
+non-boolean `enabled` values and unknown actions are rejected. Each runner owns
+its guard; the configuration object does not hold mutable checking state.
+Enabling the guard with Model Runner V1 raises a configuration error.
+
+`raise` latches the first violation or incomplete audit. Catching an exception
+does not reset the guard; restart the worker before reuse. There is no collective
+failure coordination across workers.
+
+## Independent execution plan
+
+The worker consumes the unmodified `SchedulerOutput` before its runner updates:
+
+- New requests supply complete block tables and computed-token counts.
+- Cached requests supply block-table increments, or replacements for resumed
+  requests, and current computed-token counts.
+- Scheduled token counts define the token positions authorized for this step.
+- Finished and preempted request IDs retire the corresponding bindings.
+
+The guard keeps its own request-to-block mirror. It derives expected slots,
+read block tables and sequence lengths from these scheduler messages and the
+actual batch order. It does not use the runner's address metadata to construct
+its expected plan.
+
+For each KV cache group, the guard selects the first non-null metadata item
+whose layer belongs to that group, matching the current group-level audit.
+Other layers in that group are not independently audited. A missing group,
+incomplete plan or incorrect batch binding reports a coverage gap.
+
+Checks compare exact token slots, logical read block order and declared read
+lengths. For example, swapping A's and B's slots is an error even if both target
+blocks exist. Negative slot padding and zero-length padded request rows are
+allowed; padding must not authorize extra writes or reads.
+
+FULL replay uses this execution's final metadata after graph parameter updates.
+Captured or previous-step plans cannot substitute for the current plan. V2
+uses the actual batch request order, not persistent request-state indices.
+
+Runner and model-state implementations call the guard explicitly. No Scheduler
+or allocator patch is installed, and runner, connector and asynchronous output
+methods are not replaced at runtime.
+
+## Coverage boundaries
+
+Dense PA and MLA append-only cache paths in Runner V2 are adapted. The guard
+checks consistency between scheduler messages and selected metadata. It does
+not prove that a particular layer executed, that operator arguments matched
+the metadata, or that a kernel used those addresses.
+
+Allocator generations, initialized KV contents, historical prefix writes and
+transport integrity are outside this revision's scope. A computed prefix is
+a framework input, not proof that its bytes were initialized correctly. Prefix
+reuse retains address-plan checks without validating the reused contents.
+
+KV connectors, CP, DBO, speculative decoding, encoder/decoder execution,
+sliding/circular caches, compressed/state caches and unadapted attention or
+model-state implementations are rejected. There is no remote-load submission,
+completion, cancellation or failed-load recovery guarantee.
+
+Startup graph capture does not read device metadata on the CPU. Recapture with
+live request state is rejected. Dummy writes can be checked against request
+blocks currently known to the mirror; this is not an allocator-liveness proof.
+
+The guard cannot prevent or undo writes already submitted to the device, verify
+KV numerical values, or establish model-output correctness.
+
+## Failure reporting and performance
+
+Logs use `[KV_GUARD]`. Plan mismatches use `guard.violation` with
+`evidence=execution_plan` and a cache group when available.
+`guard.coverage_gap` means required evidence is missing, not a successful check.
+Reports are deduplicated with bounded state.
+
+Audits complete synchronously before guarded execution returns. Copying address
+metadata to CPU can synchronize the NPU. No deferred audit is attached to an
+asynchronous output. Performance results for earlier deferred implementations
+do not apply; measure this revision on the intended workload.
+
+## Validation
+
+Run focused tests with:
+
+```bash
+pytest -q --confcutdir=tests/ut/debug tests/ut/debug
+```
+
+The service regression compares disabled/enabled greedy outputs for Runner V2,
+eager/FULL, repeated prompts and concurrent requests:
+
+```bash
+pytest -q tests/e2e/pull_request/one_card/test_kv_cache_guard.py
+```
+
+CPU tests establish checker behavior, not NPU service correctness. Record
+exact-head NPU service runs, current replay metadata and controlled plan faults
+before merge. Standalone NPU metadata or synthetic graph checks do not replace
+a full model-service run. Allocator-generation or transport guarantees require
+an upstream vLLM lifecycle contract rather than Scheduler patches in Ascend.

mkdocs.yml

@@ -296,6 +296,7 @@ nav:
         - user_guide/feature_guide/quantization.md
         - user_guide/feature_guide/sleep_mode.md
         - user_guide/feature_guide/runtime_guard.md
+        - user_guide/feature_guide/kv_cache_guard.md
         - user_guide/feature_guide/structured_output.md
         - user_guide/feature_guide/lora.md
         - user_guide/feature_guide/expert_parallelism_load_balancer.md

release_notes/fragments/17833.yaml

@@ -0,0 +1,13 @@
+pr: 17833
+entries:
+  - category: feature
+    text: >-
+      Added opt-in KV cache metadata-plan diagnostics for dense PA and MLA paths
+      in Runner V2. Each cache group is checked once using final attention
+      metadata in both eager and FULL modes; attention implementations are
+      unchanged and operator/per-layer accesses are not recorded. Standard
+      scheduler messages provide the independent expected address plan, without
+      Scheduler/allocator patches or runtime method replacement. Per-engine
+      additional_config.kv_guard_config supports log or sticky raise actions.
+      Audits complete synchronously. Allocator generations, initialized-prefix
+      validity and KV connectors are outside this feature's scope.

tests/e2e/pull_request/one_card/test_kv_cache_guard.py

@@ -0,0 +1,96 @@
+import json
+from concurrent.futures import ThreadPoolExecutor
+
+import pytest
+from vllm.utils.network_utils import get_open_port
+
+from tests.e2e.conftest import RemoteOpenAIServer, wait_until_npu_memory_free
+
+
+@pytest.mark.e2e_model("Qwen/Qwen3-0.6B")
+@pytest.mark.parametrize(
+    "graph_mode",
+    [
+        pytest.param(
+            "NONE",
+            marks=pytest.mark.e2e_coverage(
+                arch="dense",
+                feature="prefix_caching,chunked_prefill",
+                parallel="TP",
+                deploy="pd_mix",
+                hardware="A2",
+                quantization="BF16",
+                graph_mode="eager",
+            ),
+        ),
+        pytest.param(
+            "FULL",
+            marks=pytest.mark.e2e_coverage(
+                arch="dense",
+                feature="prefix_caching,chunked_prefill",
+                parallel="TP",
+                deploy="pd_mix",
+                hardware="A2",
+                quantization="BF16",
+                graph_mode="full_graph",
+            ),
+        ),
+    ],
+)
+@wait_until_npu_memory_free()
+def test_guard_preserves_sequential_and_concurrent_generation(graph_mode):
+    model = "Qwen/Qwen3-0.6B"
+    prompt = "Explain how a key value cache works. " * 40
+    arguments = [
+        "--max-model-len",
+        "1024",
+        "--max-num-seqs",
+        "4",
+        "--max-num-batched-tokens",
+        "256",
+        "--block-size",
+        "128",
+        "--dtype",
+        "bfloat16",
+        "--gpu-memory-utilization",
+        "0.3",
+        "--enable-prefix-caching",
+        "--enable-chunked-prefill",
+        "--compilation-config",
+        json.dumps({"cudagraph_mode": graph_mode, "cudagraph_capture_sizes": [1, 2, 4]}),
+    ]
+    if graph_mode == "NONE":
+        arguments.append("--enforce-eager")
+    results = []
+    for enabled in (False, True):
+        port = get_open_port()
+        environment = {"VLLM_USE_V2_MODEL_RUNNER": "1"}
+        with (
+            RemoteOpenAIServer(
+                model,
+                [
+                    *arguments,
+                    "--additional-config",
+                    json.dumps({"kv_guard_config": {"enabled": enabled, "action": "raise"}}),
+                    "--port",
+                    str(port),
+                ],
+                env_dict=environment,
+                seed=1234,
+                server_host="127.0.0.1",
+                server_port=port,
+                auto_port=False,
+            ) as server,
+            server.get_client() as client,
+        ):
+
+            def generate(request_index):
+                response = client.completions.create(model=model, prompt=prompt, temperature=0, max_tokens=8, seed=1234)
+                assert response.choices[0].text
+                return response.choices[0].text
+
+            sequential = [generate(request_index) for request_index in range(2)]
+            with ThreadPoolExecutor(max_workers=4) as executor:
+                concurrent = list(executor.map(generate, range(4)))
+            results.append(sequential + concurrent)
+    assert results[0] == results[1]

tests/ut/debug/__init__.py

@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

tests/ut/debug/conftest.py

@@ -0,0 +1,67 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Load the device-independent guard without importing the NPU plugin."""
+
+import importlib.util
+import logging
+import sys
+from pathlib import Path
+from types import ModuleType
+from types import SimpleNamespace as NS
+from typing import Any, cast
+
+import pytest
+
+ROOT = Path(__file__).parents[3]
+
+
+@pytest.fixture
+def subject(monkeypatch):
+    for name in ("vllm_ascend", "vllm_ascend.debug", "vllm"):
+        module = ModuleType(name)
+        module.__path__ = []
+        monkeypatch.setitem(sys.modules, name, module)
+    logger = ModuleType("vllm.logger")
+    cast(Any, logger).logger = logging.getLogger("kv-guard-test")
+    monkeypatch.setitem(sys.modules, "vllm.logger", logger)
+    for name, path in (
+        ("vllm_ascend.config_utils", "config_utils.py"),
+        ("vllm_ascend.debug.kv_guard_config", "debug/kv_guard_config.py"),
+        ("vllm_ascend.debug.kv_guard", "debug/kv_guard.py"),
+        ("vllm_ascend.debug.kv_guard_runtime", "debug/kv_guard_runtime.py"),
+    ):
+        spec = importlib.util.spec_from_file_location(name, ROOT / "vllm_ascend" / path)
+        assert spec is not None and spec.loader is not None
+        module = importlib.util.module_from_spec(spec)
+        monkeypatch.setitem(sys.modules, name, module)
+        spec.loader.exec_module(module)
+    return module
+
+
+@pytest.fixture
+def config():
+    return NS(
+        additional_config={"kv_guard_config": {"enabled": True}},
+        model_config=NS(is_encoder_decoder=False),
+        parallel_config=NS(prefill_context_parallel_size=1, decode_context_parallel_size=1, enable_dbo=False),
+        compilation_config=NS(static_forward_context={}),
+        speculative_config=None,
+        kv_transfer_config=None,
+    )
+
+
+@pytest.fixture
+def runtime(subject, config):
+    guard = subject.create_guard(config)
+    guard.events = []
+    report = guard.report
+
+    def recorded(event, **fields):
+        guard.events.append({"event": event, **fields})
+        report(event, **fields)
+
+    guard.report = recorded
+    guard.state.emit = recorded
+    guard.state.configure({0: 4})
+    guard.layer_groups = {"layer0": 0}
+    return guard

tests/ut/debug/helpers.py

@@ -0,0 +1,40 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+from types import SimpleNamespace as NS
+
+import torch
+
+
+def message(*, new=(), cached=(), resumed=(), counts=None, finished=(), preempted=()):
+    return NS(
+        scheduled_new_reqs=[
+            NS(req_id=req, block_ids=groups, num_computed_tokens=computed) for req, groups, computed in new
+        ],
+        scheduled_cached_reqs=NS(
+            req_ids=[req for req, _, _ in cached],
+            new_block_ids=[groups for _, groups, _ in cached],
+            num_computed_tokens=[computed for _, _, computed in cached],
+            resumed_req_ids=set(resumed),
+        ),
+        num_scheduled_tokens=counts or {},
+        finished_req_ids=set(finished),
+        preempted_req_ids=set(preempted),
+    )
+
+
+def metadata(slots, tables, lengths):
+    return NS(
+        slot_mapping=torch.tensor(slots),
+        block_tables=torch.tensor(tables),
+        seq_lens=torch.tensor(lengths),
+        num_actual_tokens=len(slots),
+    )
+
+
+def observe(guard, request_ids, slots, tables, lengths, *, full_graph=False):
+    guard.observe_metadata({"layer0": metadata(slots, tables, lengths)}, request_ids, full_graph)
+
+
+def reasons(guard):
+    return [event.get("reason") for event in guard.events]