Track production validation for topology KV candidate schedules

[teerthsharma]

Context

PR #1160 adds an opt-in benchmark for topology-derived KV candidate schedules. The benchmark builds request-local candidate rows from sink, local, and high-drift KV block metadata, expands the final candidate row into token indices, then compares dense PyTorch SDPA decode attention against SDPA over gathered candidate K/V tokens.

This issue tracks the remaining production validation gate. The current PR is intentionally limited to a screening benchmark and does not change RTP-LLM serving behavior.

Current benchmark signal

Validated in WSL2 on an NVIDIA GeForce RTX 4060 Laptop GPU with PyTorch 2.5.1+cu121:

python benchmark/topology_kv_candidate_schedule.py \
  --seq-len 16384 \
  --selected-tokens 128 256 512 1024 \
  --heads 16 \
  --head-dim 64 \
  --rounds 60 \
  --warmup 20 \
  --device cuda

seq_len	selected_tokens	dense_sdpa_ms	sparse_selected_ms	speedup
16384	128	0.2827	0.0324	8.72x
16384	256	0.2845	0.0578	4.92x
16384	512	0.2826	0.0745	3.79x
16384	1024	0.2834	0.1320	2.15x

Validation already done for PR #1160

python -m pytest benchmark/test_topology_kv_candidate_schedule.py -q
9 passed

python -m py_compile benchmark/topology_kv_candidate_schedule.py benchmark/test_topology_kv_candidate_schedule.py
passed

python -m flake8 benchmark/topology_kv_candidate_schedule.py benchmark/test_topology_kv_candidate_schedule.py
passed

Remaining production gate

The benchmark is only a screening signal. Before using this in a production sparse attention path, validate model quality and runtime behavior after wiring candidate rows into the existing sparse MLA/indexer path.

Acceptance criteria

• Candidate schedules can be consumed by the runtime sparse MLA/indexer path behind an opt-in flag or experiment config.
• Default serving behavior remains unchanged when the opt-in path is disabled.
• End-to-end RTP-LLM runtime benchmarks show a useful latency or memory tradeoff against the current baseline on representative hardware.
• Model-quality validation passes for representative long-context workloads.
• Documentation clearly separates microbenchmark speedup from serving speedup.

Related draft PR: #1160

[teerthsharma]

Tracking update from draft PR #1162: #1162

I added the first implementation step after the #1160 topology candidate benchmark. This is still draft-only and stacked on #1160, but the branch now contains code/test/build changes only.

Technical pieces in #1162:

• Adds rtp_llm/models_py/modules/hybrid/topology_kv_policy.py for topology-aware KV candidate rewriting.
• Adds opt-in policies:
  • disabled: passthrough/default behavior.
  • topology_sparse_merge: merge topology sink/local/witness candidates with learned top-k candidates.
  • topology_compress_sparse: sparse merge plus stable scaffold/output-contract compression counters.
  • topology_only: topology candidates only.
• Gates the runtime hook in Indexer._compute_topk behind RTP_LLM_TOPOLOGY_KV_POLICY=disabled by default, so existing serving behavior stays unchanged unless explicitly enabled.
• Adds stable scaffold/output-contract fingerprinting for repeated agentic workflows.
• Emits counters for raw selected tokens, compressed tokens represented, raw KV tokens avoided, compression hits, and scheduling time.
• Adds benchmark/test_topology_kv_policy.py, including CUDA policy-output-to-sparse-decode E2E coverage.
• Keeps the new benchmark policy target manual so it should not enter wildcard GPU CI unexpectedly.

Validation already run locally for #1162:

• Windows RTX 4060 path:
  • python -m unittest benchmark.test_topology_kv_policy -v -> 7 passed.
  • python -m unittest benchmark.test_topology_kv_candidate_schedule -v -> 22 passed.
  • python -m py_compile ... -> 5 files compiled.
• Docker Linux GPU path using rtp-llm:sm8x-opensource with --gpus all:
  • benchmark.test_topology_kv_policy -> 7 passed.
  • benchmark.test_topology_kv_candidate_schedule -> 22 passed.
  • py_compile for the changed Python files -> passed.
• Local Gemma LiteRT CPU checkpoint smoke:
  • generation prompt returned Ready!.
  • benchmark: prefill 120.72 tokens/s, decode 0.90 tokens/s, init 12.2206 s, TTFT 2.1682 s.

Remaining gate before this can become ready-for-review:

• Run the CUDA MLA sparse op / same-budget ablation / tail-latency validation on the intended H20 or SM90 CUDA 12.9 environment. I cannot run that hardware gate locally.