[Bugfix][NIXL] Fix best_of_n KV leak and early-notification race in P/D

[crazyguitar]

Summary

Fixes two NIXL P/D bugs that hit under parallel sampling (n > 1):

1. FIX(best_of fan-out) — the decode pulls KV for only one of n siblings (others served by its prefix cache), so the producer frees only that one; the other n - 1 strand in _reqs_to_send until the 480s expiry → prefill KV pins ~100% → engine hangs.
2. FIX(early-notif race) — a pull-complete notification arriving before start_load_kv() registers the request was dropped as "unrecognized"; the request only freed at the 480s expiry. Under heterogeneous TP (cpp > 1) it stalls entirely.

Validated end-to-end on three NIXL P/D topologies (1p3d TP=1, 1p1d_tp2 hetero-TP, 1p1d_2tp homogeneous TP=2): no expiries, no engine hangs, KV drains between batches.

Why this is not duplicating an existing PR

Searched open / merged PRs for NIXL best_of, NIXL parallel sampling, NIXL KV leak, _get_new_notifs, start_load_kv stranded, _reqs_to_send 480s. Closest matches:

• [P/D][Feature] Support kv_transfer_params for parallel sampling (n>1) #38900 same n > 1 scenario at the API layer (outputs.py / parallel_sampling.py); this PR fixes the connector release path (nixl/worker.py). Complementary.
• [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks #41269 same "free stranded KV" theme at the admission boundary; this PR is the analog at the completion boundary.
• [Bugfix] Fix MultiConnector state corruption in P/D disaggregated con… #42841 different scope (MultiConnector dispatch).
• fix(mooncake): resolve HBM leak from stuck WAITING_FOR_REMOTE_KVS requests #36014 same leak pattern, Mooncake connector.

No existing PR covers the NIXL producer _get_new_notifs / start_load_kv release path.

Purpose

Two related bug fixes to the NIXL P/D producer KV-release path that together resolve an engine hang under parallel sampling (n > 1 / best_of_n > 1) and a notification-vs-registration race that strands requests until the 480s expiry under heterogeneous TP.

1. FIX(best_of fan-out) — n > 1 KV leak

When n > 1 (parallel sampling), vLLM splits one prompt into n sibling requests f"{i}_{parent}" that share one prompt KV on the prefill. The decode pulls that KV for only one sibling and serves the rest from its prefix cache, so the producer gets only one pull-complete notification per parent and frees only one of n siblings. The other n - 1 strand until VLLM_NIXL_ABORT_REQUEST_TIMEOUT (default 480s), pinning prefill GPU KV cache at ~100% under load → engine hangs.

Before:

  PREFILL                                   DECODE (n=8)
  ┌──────────────────────────┐              ┌──────────────────────────┐
  │ register 8 siblings:     │              │ pull KV for sibling      │
  │   0_cmpl-fb4bc0e0        │ ───KV────►   │   7_cmpl-fb4bc0e0 only   │
  │   1_cmpl-fb4bc0e0        │              │ (others = prefix cache)  │
  │   ...                    │ ◄──notif───  │ send "7_cmpl-fb4bc0e0:1" │
  │   7_cmpl-fb4bc0e0        │              │                          │
  │ → 8 KV blocks held       │              └──────────────────────────┘
  │                          │
  │ free 7_cmpl-fb4bc0e0     │
  │ 0..6_cmpl-fb4bc0e0 stay  │
  │ → 480s expiry, KV pins   │
  │   ~100% → engine hangs   │
  └──────────────────────────┘

After:

  PREFILL                                   DECODE (n=8)
  ┌──────────────────────────┐              ┌──────────────────────────┐
  │ register 8 siblings:     │              │ pull KV for sibling      │
  │   0_cmpl-fb4bc0e0        │ ───KV────►   │   7_cmpl-fb4bc0e0 only   │
  │   1_cmpl-fb4bc0e0        │              │ (others = prefix cache)  │
  │   ...                    │ ◄──notif───  │ send "7_cmpl-fb4bc0e0:1" │
  │   7_cmpl-fb4bc0e0        │              │                          │
  │ → 8 KV blocks held       │              └──────────────────────────┘
  │                          │
  │ parse parent             │
  │   "cmpl-fb4bc0e0"        │
  │ free 7_cmpl-fb4bc0e0     │
  │  + all 0..6_cmpl-fb4bc0e0│
  │ → all 8 freed, no leak   │
  └──────────────────────────┘

2. FIX(early-notif race) — notification arrives before producer registration

Under async scheduling, the consumer's pull-complete notification can arrive at the producer before start_load_kv() registers the request. Stock _get_new_notifs() dropped any notification whose req_id was not yet in _reqs_to_process (logging Potentially invalid KV blocks ...). The request then only freed at the 480s expiry. Under heterogeneous TP (consumers_per_producer > 1), more than one notification can race the registration and the request stalls entirely.

Before:

  PREFILL                                   DECODE
  ┌──────────────────────────┐              ┌──────────────────────────┐
  │ (req cmpl-35dba481 NOT   │              │ pull KV for              │
  │  yet in _reqs_to_process)│ ◄──notif───  │   cmpl-35dba481          │
  │                          │              │ send "cmpl-35dba481:N"   │
  │ _get_new_notifs:         │              │  (N = decode_TP)         │
  │   not registered →       │              └──────────────────────────┘
  │   DROP + ERROR log       │
  │                          │
  │ start_load_kv() later:   │
  │   register the req       │
  │                          │
  │ TP=1  (cpp=1):           │
  │   one notif was dropped  │
  │   → 480s expiry frees it │
  │                          │
  │ TP>1  (cpp=N>1):         │
  │   if all N notifs raced  │
  │   the registration,      │
  │   none counted →         │
  │   request stalls entirely│
  └──────────────────────────┘

After:

  PREFILL                                   DECODE
  ┌──────────────────────────┐              ┌──────────────────────────┐
  │ (req cmpl-35dba481 NOT   │              │ pull KV for              │
  │  yet in _reqs_to_process)│ ◄──notif───  │   cmpl-35dba481          │
  │                          │              │ send "cmpl-35dba481:N"   │
  │ _get_new_notifs:         │              │  (N = decode_TP)         │
  │   stage N in             │              └──────────────────────────┘
  │   _notif_n_consumers[X]  │
  │   counter[X] += 1        │
  │                          │
  │ start_load_kv() later:   │
  │   register the req       │
  │   settle if counter ≥ cpp│
  │                          │
  │ TP=1  (cpp=1):           │
  │   counter=1 ≥ 1 → release│
  │                          │
  │ TP>1  (cpp=N):           │
  │   counter accumulates    │
  │   per orphan notif;      │
  │   release once counter   │
  │   reaches cpp            │
  └──────────────────────────┘

Changes

• worker.py: track _pulled_bases + _fanout_released to release best_of siblings together; fold into get_finished().
• worker.py: stage early notifications in _notif_n_consumers; settle on registration into _late_released.
• worker.py: add _best_of_parent() helper that parses the f"{i}_{parent}" sibling naming convention.
• test_nixl_connector.py: add _QueueingFakeNixlWrapper + 3 BDD tests covering both fixes.

Test Plan

Unit tests

3 new tests in tests/v1/kv_connector/unit/test_nixl_connector.py using the existing FakeNixlConnectorWorker / FakeNixlWrapper:

• test_one_sibling_pull_releases_all_registered_siblings
• test_sibling_registering_after_parent_pulled_is_freed_at_registration
• test_notification_arriving_before_registration_settles_on_registration

pytest tests/v1/kv_connector/unit/test_nixl_connector.py -v

End-to-end

Environment: vllm/vllm-openai:v0.21.0, 4× A100-80GB, Qwen2.5-7B fp8, best_of_n=8, 2048 input / 80 output tokens, NIXL/UCX over cuda_ipc.

• 1p3d (TP=1) — 1 prefill + 3 decode; exercises FIX(best_of fan-out).
• 1p1d_tp2 — prefill TP=1 + decode TP=2; exercises both fixes (cpp > 1 regime).

Bring up the topology with the attached run.sh, then drive load with profiles.py:

# terminal 1
./run.sh   # 1p3d or 1p1d_tp2

# terminal 2
python3 profiles.py \
    --host localhost:10001 \
    --best_of_n 8 \
    --input_tokens 2048 \
    --output_tokens 80 \
    --max_latency 5

Test Result

With both fixes applied on the 1p3d and 1p1d_tp2 setups (Qwen2.5-7B fp8, best_of_n=8, NIXL/UCX, A100-80GB):

• Prefill GPU KV cache drains between batches (peaks well under 5%); zero Releasing expired KV blocks for request log lines (no 480s timeouts firing).
• Zero Potentially invalid KV blocks for unrecognized request ERROR lines (these fire without FIX(early-notif race)).
• All requests succeed; no errors or tracebacks on prefill or decode.

Checks

Local pre-commit run on the changed files:

SKIP=pip-compile,pip-compile-rocm,pip-compile-xpu,pip-compile-docs,\
update-dockerfile-graph,validate-docker-versions,format-torch-nightly-test \
  pre-commit run --files \
    vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py \
    tests/v1/kv_connector/unit/test_nixl_connector.py

All pass: ruff-check, ruff-format, typos, SPDX headers, root lazy imports, forbidden imports, torch.cuda call check, mypy-3.10, attention backend docs, boolean-ops-in-with-statements. DCO signoff present on both commits.

Repro scripts (1p3d/run.sh)

#!/usr/bin/env bash
set -o pipefail

cd "$(dirname "${BASH_SOURCE[0]}")"
mkdir -p logs && rm -f logs/*.log
touch logs/proxy.log logs/prefill_0.log logs/decode_{0,1,2}.log

PIDS=()
trap 'echo; echo "[run] stopping..."; kill "${PIDS[@]:-}" 2>/dev/null; pkill -P $$ 2>/dev/null; wait 2>/dev/null; exit' INT TERM

HERE="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
PROXY="$HERE/toy_proxy_server.py"

export NCCL_DEBUG=WARN
export UCX_RNDV_THRESH=8192
export UCX_TLS=tcp,cuda_ipc,cuda_copy,shm,cma,self
export VLLM_NIXL_SIDE_CHANNEL_HOST=127.0.0.1

COMMON=(
    --tensor-parallel-size 1
    --gpu-memory-utilization 0.92
    --max-model-len 10240
    --max-num-batched-tokens 32768
    --max-num-seqs 128
    --trust-remote-code
    --kv-cache-dtype fp8
)

kv() { printf '{"kv_connector":"NixlConnector","kv_role":"%s"}' "$1"; }

python3 "$PROXY" --host 0.0.0.0 --port 10001 \
    --prefiller-hosts 127.0.0.1 --prefiller-ports 8100 \
    --decoder-hosts   127.0.0.1 127.0.0.1 127.0.0.1 --decoder-ports 8200 8201 8202 \
    > logs/proxy.log 2>&1 & PIDS+=($!)

CUDA_VISIBLE_DEVICES=0 VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
    vllm serve Qwen/Qwen2.5-7B-Instruct "${COMMON[@]}" \
        --host 0.0.0.0 --port 8100 --kv-transfer-config "$(kv kv_producer)" \
    > logs/prefill_0.log 2>&1 & PIDS+=($!)

for i in 0 1 2; do
    CUDA_VISIBLE_DEVICES=$((i+1)) VLLM_NIXL_SIDE_CHANNEL_PORT=$((5601+i)) \
        vllm serve Qwen/Qwen2.5-7B-Instruct "${COMMON[@]}" \
            --host 0.0.0.0 --port $((8200+i)) --kv-transfer-config "$(kv kv_consumer)" \
        > logs/decode_${i}.log 2>&1 & PIDS+=($!)
done

echo "[run] launched ${#PIDS[@]} services — tailing logs (Ctrl-C to stop)"
tail -F logs/*.log

Repro scripts (1p1d_tp2/run.sh, hetero-TP)

#!/usr/bin/env bash
set -o pipefail

cd "$(dirname "${BASH_SOURCE[0]}")"
mkdir -p logs && rm -f logs/*.log
touch logs/proxy.log logs/prefill_0.log logs/decode_0.log

PIDS=()
trap 'echo; echo "[run] stopping..."; kill "${PIDS[@]:-}" 2>/dev/null; pkill -P $$ 2>/dev/null; wait 2>/dev/null; exit' INT TERM

HERE="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
PROXY="$HERE/toy_proxy_server.py"

export NCCL_DEBUG=WARN
export UCX_RNDV_THRESH=8192
export UCX_TLS=tcp,cuda_ipc,cuda_copy,shm,cma,self
export VLLM_NIXL_SIDE_CHANNEL_HOST=127.0.0.1

kv() { printf '{"kv_connector":"NixlConnector","kv_role":"%s"}' "$1"; }

python3 "$PROXY" --host 0.0.0.0 --port 10001 \
    --prefiller-hosts 127.0.0.1 --prefiller-ports 8100 \
    --decoder-hosts   127.0.0.1 --decoder-ports   8200 \
    > logs/proxy.log 2>&1 & PIDS+=($!)

CUDA_VISIBLE_DEVICES=0 VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
    vllm serve Qwen/Qwen2.5-7B-Instruct \
        --tensor-parallel-size 1 \
        --gpu-memory-utilization 0.92 \
        --max-model-len 10240 --max-num-batched-tokens 32768 --max-num-seqs 128 \
        --trust-remote-code --kv-cache-dtype fp8 \
        --host 0.0.0.0 --port 8100 --kv-transfer-config "$(kv kv_producer)" \
    > logs/prefill_0.log 2>&1 & PIDS+=($!)

CUDA_VISIBLE_DEVICES=1,2 VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
    vllm serve Qwen/Qwen2.5-7B-Instruct \
        --tensor-parallel-size 2 \
        --gpu-memory-utilization 0.92 \
        --max-model-len 10240 --max-num-batched-tokens 32768 --max-num-seqs 128 \
        --trust-remote-code --kv-cache-dtype fp8 \
        --host 0.0.0.0 --port 8200 --kv-transfer-config "$(kv kv_consumer)" \
    > logs/decode_0.log 2>&1 & PIDS+=($!)

echo "[run] launched ${#PIDS[@]} services — tailing logs (Ctrl-C to stop)"
tail -F logs/*.log

Repro scripts (1p1d_2tp/run.sh, prefill TP2 and decode TP2)

#!/usr/bin/env bash
set -o pipefail

cd "$(dirname "${BASH_SOURCE[0]}")"
mkdir -p logs && rm -f logs/*.log
touch logs/proxy.log logs/prefill_0.log logs/decode_0.log

PIDS=()
trap 'echo; echo "[run] stopping..."; kill "${PIDS[@]:-}" 2>/dev/null; pkill -P $$ 2>/dev/null; wait 2>/dev/null; exit' INT TERM

echo "[run] starting 1p1d_2tp (prefill TP=2 GPUs 0,1 + decode TP=2 GPUs 2,3)"

HERE="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
# Patches applied manually; see ../../pd-nixl/apply_nixl_{best_of_fanout,early_notif}.py.

PROXY="$HERE/../../pd-nixl/toy_proxy_server.py"

export NCCL_DEBUG=WARN
export UCX_RNDV_THRESH=8192
export UCX_TLS=tcp,cuda_ipc,cuda_copy,shm,cma,self
export VLLM_NIXL_SIDE_CHANNEL_HOST=127.0.0.1

kv() {  # $1 = kv_role
    printf '{"kv_connector":"NixlConnector","kv_role":"%s"}' "$1"
}

python3 "$PROXY" --host 0.0.0.0 --port 10001 \
    --prefiller-hosts 127.0.0.1 --prefiller-ports 8100 \
    --decoder-hosts   127.0.0.1 --decoder-ports   8200 \
    > logs/proxy.log 2>&1 & PIDS+=($!)

# Prefill: 2 GPUs, TP=2.
CUDA_VISIBLE_DEVICES=0,1 VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
    vllm serve Qwen/Qwen2.5-7B-Instruct \
        --tensor-parallel-size 2 \
        --gpu-memory-utilization 0.92 \
        --max-model-len 10240 \
        --max-num-batched-tokens 32768 \
        --max-num-seqs 128 \
        --trust-remote-code \
        --kv-cache-dtype fp8 \
        --host 0.0.0.0 --port 8100 --kv-transfer-config "$(kv kv_producer)" \
    > logs/prefill_0.log 2>&1 & PIDS+=($!)

# Decode: 2 GPUs, TP=2 (homogeneous w/ prefill TP).
CUDA_VISIBLE_DEVICES=2,3 VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
    vllm serve Qwen/Qwen2.5-7B-Instruct \
        --tensor-parallel-size 2 \
        --gpu-memory-utilization 0.92 \
        --max-model-len 10240 \
        --max-num-batched-tokens 32768 \
        --max-num-seqs 128 \
        --trust-remote-code \
        --kv-cache-dtype fp8 \
        --host 0.0.0.0 --port 8200 --kv-transfer-config "$(kv kv_consumer)" \
    > logs/decode_0.log 2>&1 & PIDS+=($!)

echo "[run] launched ${#PIDS[@]} services — tailing logs (Ctrl-C to stop)"
tail -F logs/*.log

Benchmark client (profiles.py)

import argparse
import concurrent.futures
import json
import pathlib
import random
import statistics
import time

import requests


DEFAULT_PROMPTS = str(pathlib.Path(__file__).resolve().parent / "model_outputs.json")
DEFAULT_MODEL = "Qwen/Qwen2.5-7B-Instruct"


def percentile(xs, q):
    s = sorted(xs)
    return s[max(0, min(int(len(s) * q / 100), len(s) - 1))]


def load_prompts(path, n):
    with open(path) as f:
        rows = [json.loads(line) for line in f if line.strip()]
    return [r.get("model_input", "") for r in rows[:n]]


def build_body(text, model, input_tokens, output_tokens, best_of_n, shuffle):
    if shuffle:
        words = text.split(" ")
        random.shuffle(words)
        text = " ".join(words)
    body = {"model": model, "prompt": text}
    if output_tokens:
        body["max_tokens"] = output_tokens + random.randint(-10, 10)
    if input_tokens:
        body["truncate_prompt_tokens"] = input_tokens + random.randint(-100, 100)
    if best_of_n:
        body["n"] = best_of_n
    return body


def send_one(host, model, prompt, input_tokens, output_tokens, best_of_n, shuffle):
    body = build_body(prompt, model, input_tokens, output_tokens, best_of_n, shuffle)
    start = time.time()
    try:
        r = requests.post(f"{host}/v1/completions", json=body, timeout=600)
        r.raise_for_status()
        usage = r.json().get("usage") or {}
        cached = (usage.get("prompt_tokens_details") or {}).get("cached_tokens", 0)
        return {
            "latency": time.time() - start,
            "error": False,
            "in_tokens": usage.get("prompt_tokens"),
            "out_tokens": usage.get("completion_tokens", 0),
            "cached": cached,
        }
    except Exception:
        return {"latency": time.time() - start, "error": True}


def run_batch(host, model, prompts, batch_size, **kw):
    start = time.time()
    with concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as ex:
        futures = [ex.submit(send_one, host, model, p, **kw) for p in prompts]
        results = [f.result() for f in concurrent.futures.as_completed(futures)]
    return time.time() - start, results


def summarise(bs, duration, results):
    times = [r["latency"] for r in results]
    errors = sum(1 for r in results if r["error"])
    ok = [r for r in results if not r["error"]]
    print(f"\n### batch_size={bs} ###")
    print(f"  requests:           {len(results)}")
    print(f"  duration (s):       {duration:.2f}")
    print(f"  errors:             {errors}")
    print(f"  throughput (req/s): {len(results) / duration:.2f}")
    print(f"  avg latency (s):    {statistics.fmean(times):.2f}")
    for q in (50, 75, 90, 95):
        print(f"  p{q} latency (s):     {percentile(times, q):.2f}")
    if not ok:
        return
    ins = [r["in_tokens"] for r in ok if r["in_tokens"]]
    outs = [r["out_tokens"] for r in ok]
    if ins:
        print(f"  mean input tokens:  {statistics.fmean(ins):.1f}")
        print(f"  mean output tokens: {statistics.fmean(outs):.1f}")
        total_in = sum(ins)
        total_cached = sum(r["cached"] for r in ok)
        print(f"  cache hit rate:     {100 * total_cached / total_in:.1f}%")


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--host", default="localhost:10001")
    p.add_argument("--prompts", default=DEFAULT_PROMPTS)
    p.add_argument("--model", default=DEFAULT_MODEL)
    p.add_argument("--samples", type=int, default=200)
    p.add_argument("--batches", default=",".join(map(str, range(3, 100))))
    p.add_argument("--input_tokens", type=int)
    p.add_argument("--output_tokens", type=int)
    p.add_argument("--best_of_n", type=int)
    p.add_argument("--max_latency", type=float, default=5.0)
    p.add_argument("--no_shuffle_input", dest="shuffle_input", action="store_false")
    args = p.parse_args()

    host = args.host if args.host.startswith("http") else f"http://{args.host}"
    prompts = load_prompts(args.prompts, args.samples)
    kw = dict(
        input_tokens=args.input_tokens,
        output_tokens=args.output_tokens,
        best_of_n=args.best_of_n,
        shuffle=args.shuffle_input,
    )
    for bs in [int(b) for b in args.batches.split(",")]:
        duration, results = run_batch(host, args.model, prompts, bs, **kw)
        summarise(bs, duration, results)
        times = [r["latency"] for r in results]
        if (
            statistics.fmean(times) > args.max_latency
            and percentile(times, 50) > args.max_latency
        ):
            print(f"break: latency exceeds max ({args.max_latency}s)")
            break


if __name__ == "__main__":
    main()

# terminal 1 — bring up the topology (pick one)
./run_1p3d.sh            # 1 prefill + 3 decode (TP=1)
# or
./run_1p1d_tp2.sh     # 1 prefill TP=1 + 1 decode TP=2 (hetero-TP)
# or
./run_1p1d_2tp.sh.   # 1 prefill TP=2 + 1 decode TP=2

# terminal 2 — drive load
python3 profiles.py --host localhost:10001 --best_of_n 8 --input_tokens 2048 --output_tokens 80 --max_latency 5

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[gemini-code-assist]

Code Review

This pull request addresses race conditions and resource stranding in the NIXL connector by implementing logic to handle early pull-complete notifications and 'best_of' sibling fan-out. It introduces tracking for staged notifications and pulled parent IDs, ensuring that requests are correctly released even when notifications arrive out of order or when only a subset of siblings is explicitly pulled. A performance optimization was suggested for the start_load_kv method to reduce the complexity of settling early notifications from O(M) to O(B) by integrating the check into the request registration loop.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74828c1117

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[NUABO]

hi @crazyguitar Please take a look at the error in this screenshot. Can it be fixed by your commit? Thank you very much!

[crazyguitar]

hi @crazyguitar Please take a look at the error in this screenshot. Can it be fixed by your commit? Thank you very much!

@NUABO Yes, this looks like the issue this PR is intended to fix. However, the screenshot still shows the old failure signatures, so I suspect the run didn't apply this PR, right? If it is ok, could u provide commands how did u reproduce the issue so that I can verify on my environment. Thank you.

[NUABO]

hi @crazyguitar Please take a look at the error in this screenshot. Can it be fixed by your commit? Thank you very much!

@NUABO Yes, this looks like the issue this PR is intended to fix. However, the screenshot still shows the old failure signatures, so I suspect the run didn't apply this PR, right? If it is ok, could u provide commands how did u reproduce the issue so that I can verify on my environment. Thank you.

yes, I haven't merged this patch yet. The configuration I'm using is below, you can give it a try.

decode
--kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_consumer"},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}]}}' 

prefill
--kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_producer","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_producer"},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}]}}'

The issue I encountered before is documented here: #42831

[crazyguitar]

Hi @NUABO

I was unable to use MooncakeStoreConnector due to MooncakeStoreConnector segmentation fault.

Fortunately, If I utilized multi-connector with NixlConnector only, everything looked good. This is my test script (1p3d). You can use profiles.py in Checks section to test. Let me know if u encounter any issue. thx

#!/usr/bin/env bash
set -o pipefail

cd "$(dirname "${BASH_SOURCE[0]}")"
mkdir -p logs && rm -f logs/*.log
touch logs/proxy.log logs/prefill_0.log logs/decode_{0,1,2}.log

PIDS=()
trap 'echo; echo "[run] stopping..."; kill "${PIDS[@]:-}" 2>/dev/null; pkill -P $$ 2>/dev/null; wait 2>/dev/null; exit' INT TERM

echo "[run] starting 1p3d (NVIDIA MultiConnector[Nixl]: 1 toy proxy + 1 prefill + 3 decode)"

HERE="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
PROXY="$HERE/../../pd-nixl/toy_proxy_server.py"

export NCCL_DEBUG=WARN
export UCX_RNDV_THRESH=8192
export UCX_TLS=tcp,cuda_ipc,cuda_copy,shm,cma,self
export VLLM_NIXL_SIDE_CHANNEL_HOST=127.0.0.1

COMMON=(
    --tensor-parallel-size 1
    --gpu-memory-utilization 0.92
    --max-model-len 10240
    --max-num-batched-tokens 32768
    --max-num-seqs 128
    --trust-remote-code
    --kv-cache-dtype fp8
)

kv() {  # $1 = kv_role — MultiConnector wrapping a single NixlConnector child
    printf '{"kv_connector":"MultiConnector","kv_role":"%s","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"%s"}]}}' "$1" "$1"
}

python3 "$PROXY" --host 0.0.0.0 --port 10001 \
    --prefiller-hosts 127.0.0.1 --prefiller-ports 8100 \
    --decoder-hosts   127.0.0.1 127.0.0.1 127.0.0.1 --decoder-ports 8200 8201 8202 \
    > logs/proxy.log 2>&1 & PIDS+=($!)

CUDA_VISIBLE_DEVICES=0 VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
    vllm serve Qwen/Qwen2.5-7B-Instruct "${COMMON[@]}" \
        --host 0.0.0.0 --port 8100 --kv-transfer-config "$(kv kv_producer)" \
    > logs/prefill_0.log 2>&1 & PIDS+=($!)

CUDA_VISIBLE_DEVICES=1 VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
    vllm serve Qwen/Qwen2.5-7B-Instruct "${COMMON[@]}" \
        --host 0.0.0.0 --port 8200 --kv-transfer-config "$(kv kv_consumer)" \
    > logs/decode_0.log 2>&1 & PIDS+=($!)

CUDA_VISIBLE_DEVICES=2 VLLM_NIXL_SIDE_CHANNEL_PORT=5602 \
    vllm serve Qwen/Qwen2.5-7B-Instruct "${COMMON[@]}" \
        --host 0.0.0.0 --port 8201 --kv-transfer-config "$(kv kv_consumer)" \
    > logs/decode_1.log 2>&1 & PIDS+=($!)

CUDA_VISIBLE_DEVICES=3 VLLM_NIXL_SIDE_CHANNEL_PORT=5603 \
    vllm serve Qwen/Qwen2.5-7B-Instruct "${COMMON[@]}" \
        --host 0.0.0.0 --port 8202 --kv-transfer-config "$(kv kv_consumer)" \
    > logs/decode_2.log 2>&1 & PIDS+=($!)

echo "[run] launched ${#PIDS[@]} services — tailing logs (Ctrl-C to stop)"
tail -F logs/*.log