[Feature][Worker] Add MRV2 KV Trace

docs/source/user_guide/configuration/additional_config.md

@@ -38,6 +38,7 @@ The following table lists additional configuration options available in vLLM Asc
 |-------------------------------------|------|---------|-----------------------------------------------------------------------------------------------------------|
 | `xlite_graph_config`                | dict | `{}`    | Configuration options for Xlite graph mode                                                                |
 | `finegrained_tp_config`             | dict | `{}`    | Configuration options for fine-grained tensor parallelism. See [Fine-Grained Tensor Parallelism](../feature_guide/Fine_grained_TP.md). |
+| `kv_trace`                  | dict | `{}`    | Per-engine KV cache address diagnostics for Model Runner V2 (no Scheduler patch). `enabled` defaults to `false`; `action` accepts `"log"` (default) or `"raise"`. See [KV Trace monitoring](../feature_guide/kv_trace.md). |
 | `ascend_compilation_config`         | dict | `{}`    | Configuration options for ascend compilation                                                              |
 | `ascend_warmup_config`              | dict | `{}`    | Configuration options for startup warmup that overlaps weight loading                                     |
 | `eplb_config`                       | dict | `{}`    | Runner-specific EPLB extensions. See [Expert Parallelism Load Balancer](../feature_guide/expert_parallelism_load_balancer.md). |

docs/source/user_guide/feature_guide/kv_trace.md

@@ -0,0 +1,136 @@
+# KV Trace monitoring
+
+KV Trace compares native scheduler declarations with the final address metadata
+used by Ascend Model Runner V2, once per KV cache group. It records bindings,
+write/read plans and public connector information. It is disabled by default.
+Model Runner V1 and attention operators are unchanged.
+
+## Enable KV Trace
+
+```bash
+export VLLM_USE_V2_MODEL_RUNNER=1
+vllm serve MODEL --additional-config '{"kv_trace":{"enabled":true,"action":"log"}}'
+```
+
+| `kv_trace` option | Type | Default | Meaning |
+| --- | --- | --- | --- |
+| `enabled` | bool | `false` | Enable the MRV2 observer for this engine. |
+| `action` | `"log"` or `"raise"` | `"log"` | Log violations or stop returning model results. |
+
+Omitting `kv_trace` or setting `enabled=false` disables the observer. Invalid
+keys, non-boolean `enabled` and unknown actions are rejected. Environment
+variables and `dump_config` do not enable this feature.
+
+```python
+from vllm import LLM
+
+llm = LLM(model="MODEL", additional_config={"kv_trace": {"enabled": True, "action": "raise"}})
+```
+
+The observer installs in MRV2. It does not patch Scheduler, KVCacheManager or
+BlockPool, and it does not add private attributes to SchedulerOutput.
+
+## What is checked
+
+The MRV2 execution entry consumes the original SchedulerOutput before the
+runner updates its own request tables. New and resumed requests replace their
+worker-local binding; cached requests append native block deltas. Finished and
+preempted IDs remove local request references. This is a request map, not an
+allocator state machine or proof of physical release.
+
+Each step retains its own immutable binding snapshot and computed/scheduled
+token ranges. The checker derives expected addresses from this declaration,
+then compares the final metadata in the runner's actual batch order.
+
+| Constraint | Example |
+| --- | --- |
+| Exact token destination | Cross-request slot swaps, shifted writes or non-PAD write tails. |
+| Logical block order | The input read table differs from the declared request binding. |
+| Declared length | The final input length disagrees with the scheduled token range. |
+| Capacity and reserved storage | A write is outside cache capacity or targets a forbidden null block. |
+| Dummy writes | An idle dummy has a non-PAD write target outside the explicitly allowed native null-block range. |
+
+Warmup/capture scratch is not treated as a real scheduled request. Ordinary
+dummy writes must use PAD `-1`, except when the native block table explicitly
+enables null-block redirection. No per-layer access hooks are installed.
+
+There is no allocation epoch, owners index or `valid_end`. A worker cannot
+reliably distinguish prefix-cache reuse from physical reallocation of the same
+block ID. Therefore this feature does not prove initialized prefixes, detect
+every stale reference, establish actual allocation/free history, or diagnose
+KV numerical corruption when the visible declaration and input plan agree.
+
+## Execution paths and supported configurations
+
+Adapted dense PA/MLA MRV2 paths use one final metadata view per cache group in
+eager and FULL modes. Allocator and kernel block coordinates must match.
+CP, DBO, speculative decoding, encoder/decoder and non-append cache layouts
+remain unadapted in this implementation; the checker does not silently apply
+the ordinary address formula to those modes.
+
+The final metadata is frozen after the native execution returns. Ordinary
+async TP1/PP1 generation retains one pending D2H snapshot and completes it at
+the original result getter or before reusing the copy buffers. Other result
+paths retain synchronous checking. Each plan carries its own binding snapshot,
+so a later message cannot change an earlier pending audit.
+
+Zero/copy fields are recorded when the original `update_requests` returns;
+they do not establish content validity. Connector public input fields and
+native finished/failed/invalid result sets are recorded without wrapping
+transport methods or polling completion queues. A finished receive can also
+be failed; the records preserve both fields. There is no submission-identity
+or remote-content validity check.
+
+## Read the logs
+
+Records go through the vLLM logger to the process log:
+
+- `kv_trace.enabled`: the configured failure action.
+- `kv_trace.bind`: reconstructed request bindings, token ranges and native finish/preempt notices.
+- `kv_trace.write` / `kv_trace.read`: per-group final execution plans, not kernel memory accesses.
+- `kv_trace.boundary`: native update or connector public input/result fields.
+- `kv_trace.violation`: an address-plan constraint violation and its expected/actual fields.
+
+`log` records violations and continues. `raise` latches the first violation
+and prevents normal result delivery; the exception can terminate EngineCore.
+This does not guarantee detection before a kernel runs or undo device writes.
+Missing observations are debug-level skipped checks, not violations, and do
+not by themselves latch failure. No `coverage_gap` error is emitted.
+
+## Validation and performance
+
+Run the focused host checks:
+
+```bash
+pytest -q --confcutdir=tests/ut/kv_trace tests/ut/kv_trace
+```
+
+The suite covers native binding deltas without private scheduler attributes,
+prefix sharing, actual batch order, slot/table faults, current FULL metadata,
+step-local async snapshots, log/raise, dummy arguments and connector results.
+These checks do not establish NPU correctness.
+
+The MRV2 eager/FULL service regression is available for an NPU environment:
+
+```bash
+pytest -q tests/e2e/pull_request/one_card/test_kv_trace.py
+```
+
+The worker-only change has not been revalidated on NPU or benchmarked. Earlier
+Scheduler-instrumented results do not validate its reduced contract. Hardware
+fault injection and performance measurements must use the changed source,
+fixed models/settings and normal trace logging enabled.
+
+## Implementation guide
+
+| File | Responsibility |
+| --- | --- |
+| `kv_trace_config.py` | Per-engine `additional_config.kv_trace` options. |
+| `kv_trace_state.py` | Worker-local immutable request block tables from native messages. |
+| `kv_trace_plan.py` | Step-local bindings, metadata snapshots and address comparisons. |
+| `kv_trace_runtime.py` | Local message consumption, pending audits and output failure handling. |
+| `kv_trace_runner.py` | MRV2-only installation, batch order, metadata and capture boundaries. |
+| `kv_trace_connector.py` | Public connector plan and native result projection, with no lifecycle hooks. |
+
+The feature integration is limited to MRV2 initialization and KV-cache
+configuration. No scheduler patch or MRV1 integration is required.

mkdocs.yml

@@ -307,6 +307,7 @@ nav:
         - user_guide/feature_guide/quantization.md
         - user_guide/feature_guide/sleep_mode.md
         - user_guide/feature_guide/runtime_guard.md
+        - user_guide/feature_guide/kv_trace.md
         - user_guide/feature_guide/structured_output.md
         - user_guide/feature_guide/lora.md
         - user_guide/feature_guide/expert_parallelism_load_balancer.md

release_notes/fragments/17833.yaml

@@ -0,0 +1,15 @@
+pr: 17833
+entries:
+  - category: feature
+    text: >-
+      Added disabled-by-default KV Trace monitoring through additional_config.kv_trace
+      (enabled and action) for adapted dense PA and MLA paths in Model Runner V2.
+      Native SchedulerOutput binding deltas are consumed inside MRV2 without
+      patching Scheduler, KVCacheManager or BlockPool. Model Runner V1 is unchanged.
+      Per-group execution plans and public connector inputs/results are recorded;
+      allocation generations, physical release and KV content validity are not inferred.
+  - category: performance
+    text: >-
+      Ordinary async single-worker generation retains one metadata snapshot and
+      immutable binding plan per step, completing its audit at the original result
+      getter or before buffer reuse. It does not copy KV values or add per-layer hooks.

tests/e2e/pull_request/one_card/test_kv_trace.py

@@ -0,0 +1,101 @@
+import json
+from concurrent.futures import ThreadPoolExecutor
+
+import pytest
+from vllm.utils.network_utils import get_open_port
+
+from tests.e2e.conftest import RemoteOpenAIServer, wait_until_npu_memory_free
+
+
+@pytest.mark.e2e_model("Qwen/Qwen3-0.6B")
+@pytest.mark.parametrize(
+    "graph_mode",
+    [
+        pytest.param(
+            "NONE",
+            marks=pytest.mark.e2e_coverage(
+                arch="dense",
+                feature="prefix_caching,chunked_prefill",
+                parallel="TP",
+                deploy="pd_mix",
+                hardware="A2",
+                quantization="BF16",
+                graph_mode="eager",
+            ),
+        ),
+        pytest.param(
+            "FULL",
+            marks=pytest.mark.e2e_coverage(
+                arch="dense",
+                feature="prefix_caching,chunked_prefill",
+                parallel="TP",
+                deploy="pd_mix",
+                hardware="A2",
+                quantization="BF16",
+                graph_mode="full_graph",
+            ),
+        ),
+    ],
+)
+@wait_until_npu_memory_free()
+def test_guard_preserves_sequential_and_concurrent_generation(graph_mode):
+    model = "Qwen/Qwen3-0.6B"
+    # Keep a cacheable prefix and give greedy decoding a definite task. The
+    # repeated-text fixture has multiple continuations even with tracing off.
+    prompt = (
+        "A key value cache stores previous attention keys and values. " * 40
+        + "\nContinue the integer sequence, using the same format:\n1, 2, 3, 4, 5,"
+    )
+    arguments = [
+        "--max-model-len",
+        "1024",
+        "--max-num-seqs",
+        "4",
+        "--max-num-batched-tokens",
+        "256",
+        "--block-size",
+        "128",
+        "--dtype",
+        "bfloat16",
+        "--gpu-memory-utilization",
+        "0.3",
+        "--enable-prefix-caching",
+        "--enable-chunked-prefill",
+        "--compilation-config",
+        json.dumps({"cudagraph_mode": graph_mode, "cudagraph_capture_sizes": [1, 2, 4]}),
+    ]
+    if graph_mode == "NONE":
+        arguments.append("--enforce-eager")
+    results = []
+    for enabled in (False, True):
+        port = get_open_port()
+        environment = {"VLLM_USE_V2_MODEL_RUNNER": "1"}
+        with (
+            RemoteOpenAIServer(
+                model,
+                [
+                    *arguments,
+                    "--additional-config",
+                    json.dumps({"kv_trace": {"enabled": enabled, "action": "raise"}}),
+                    "--port",
+                    str(port),
+                ],
+                env_dict=environment,
+                seed=1234,
+                server_host="127.0.0.1",
+                server_port=port,
+                auto_port=False,
+            ) as server,
+            server.get_client() as client,
+        ):
+
+            def generate(request_index):
+                response = client.completions.create(model=model, prompt=prompt, temperature=0, max_tokens=8, seed=1234)
+                assert response.choices[0].text
+                return response.choices[0].text
+
+            sequential = [generate(request_index) for request_index in range(2)]
+            with ThreadPoolExecutor(max_workers=4) as executor:
+                concurrent = list(executor.map(generate, range(4)))
+            results.append(sequential + concurrent)
+    assert results[0] == results[1]

tests/ut/distributed/kv_transfer/kv_p2p/test_mooncake_connector.py

@@ -5431,6 +5431,45 @@ def test_unused_remote_ports_are_notified_once_by_primary_receiver(primary, has_
     assert sent == ([("r", "idle-peer", 7000, ports)] if primary and has_ports else [])
 
 
+@pytest.mark.parametrize("awaiting_kvs", [False, True])
+@pytest.mark.parametrize("transfer_failed", [False, True])
+def test_receive_completion_contract_is_preserved(awaiting_kvs, transfer_failed):
+    receiver = KVCacheRecvingThread.__new__(KVCacheRecvingThread)
+    receiver._is_failed_recv_request = MagicMock(side_effect=[False, transfer_failed])
+    receiver._transfer_kv_cache_all_groups = MagicMock()
+    receiver._mark_request_task_done = MagicMock(return_value=True)
+    receiver.pending_reformat_lock = threading.Lock()
+    receiver.pending_reformat = {"r": object()}
+    receiver.proc_not_transfer_request_lock = threading.Lock()
+    receiver.proc_not_transfer_request = {}
+    receiver._reformat_pending_kv_caches = MagicMock()
+    receiver._clear_failed_recv_request = MagicMock()
+    receiver.task_tracker = MagicMock()
+    receiver.request_queue = MagicMock()
+    receiver._send_done_signal_to_free_remote_port = MagicMock()
+    receiver._send_done_recv_signal = MagicMock()
+    meta = {
+        "request_id": "r",
+        "remote_request_id": "remote-r",
+        "remote_host": "host",
+        "remote_handshake_port": 1,
+        "remote_port_send_num": {},
+        "all_task_done": True,
+        "awaiting_kvs": awaiting_kvs,
+    }
+    receiver._handle_request(meta)
+    if transfer_failed:
+        receiver._reformat_pending_kv_caches.assert_not_called()
+    else:
+        receiver._reformat_pending_kv_caches.assert_called_once_with("r")
+    if awaiting_kvs:
+        receiver.task_tracker.update_done_task_count.assert_called_once_with("r")
+    else:
+        receiver.task_tracker.update_done_task_count.assert_called_once_with("r", report_completion=False)
+    receiver.request_queue.task_done.assert_called_once()
+    receiver._send_done_recv_signal.assert_called_once()
+
+
 @pytest.mark.parametrize("pulls", [1, 2, 4])
 def test_head_reformat_preserves_token_head_order_at_scatter_boundary(monkeypatch, pulls):
     """Validate PD's CPU reshape arithmetic, not the external NPU scatter op."""

tests/ut/kv_trace/__init__.py

@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project