Multi-Turn Chat & Session Affinity Performance Evaluation Report

This report evaluates the performance, latency, and resource utilization of the Endpoint Picker (EPP) router when handling multi-turn conversational chat workloads. It compares three distinct routing architectures—session-affinity-filter, random-passthrough-multiturn, and optimized-baseline-multiturn—across the exact same 10 QPS multi-turn chat benchmark.

1. Inference-Perf Multi-Turn Chat Job Explained

A. Why `enable_multi_turn_chat: true` is Required for Session Affinity

Testing session affinity requires stateful request sessions that persist across conversational turns:

Stateless Single-Turn Mode (enable_multi_turn_chat: false): By default, shared_prefix generates independent prompts with session_id = None. Without a session identifier, inference-perf never captures or re-sends session token headers.
Stateful Multi-Turn Mode (enable_multi_turn_chat: true): Enabling multi-turn chat instructs shared_prefix to generate stateful LocalUserSession objects (user_session_id = "user_session_0", "user_session_1", etc.).
Token Capture and Replay (PR 596):
1. Turn 1 (Session Initiation): inference-perf sends a request without a session token header. EPP selects an endpoint and returns x-session-token: <encoded-pod> in the response HTTP headers.
2. Turns 2+ (Affinity Pinning): inference-perf stores the received token and automatically injects x-session-token into every subsequent request belonging to that session_id, testing EPP's session-affinity-filter routing path.

B. Why Prompt Sizing was Adjusted (`2k / 500 / 200`)

In multi-turn chat mode, inference-perf appends each turn's user question and assistant response to the session's ongoing conversation history (context):

Why large prompt sizes fail: When configured with 95,000 system prompt tokens + 5,000 question tokens + 1,000 output tokens, a session's input prompt reaches 100,000 tokens on Turn 1 and grows by 6,000 tokens per turn. By Turn 6, requests exceed the vLLM simulator pod's maximum context length (max_model_len = 131072), causing HTTP 400 Bad Request errors and multi-megabyte JSON body timeouts ("body size exceeds the given limit").
Why 2,000 / 500 / 200 sizing is optimal:
- Configured in shared_prefix_100k-1k-10qps_session-affinity.yaml with system_prompt_len: 2000, question_len: 500, and output_len: 200, each turn adds 700 tokens to the conversation.
- Across a 300-second benchmark run (~60 turns per session at 10 QPS across 50 sessions), a session's total history reaches ~44,000 tokens—safely below the 131,072 context limit.
- This cleanly isolates control-plane session affinity overhead (header inspection, token decoding, and pod pinning) without network I/O saturation.

2. Multi-Turn Routing Performance Comparison Table

All tests were executed on GKE (e2 machine family) with 10 simulated vLLM backend replicas (--sim-replicas 10) at 10 QPS for 300 seconds after a 15-second warmup stage.

Metric / Attribute	🔗 `session-affinity-filter`	🎲 `random-passthrough`	⚙️ `optimized-baseline`	Key Architectural Takeaway
P50 Latency (ms)	0.39 ms	0.69 ms	1.20 ms	`session-affinity-filter` is ~3x faster at P50 than baseline scoring
P95 Tail Latency (ms)	0.96 ms	1.22 ms	1.96 ms	Sub-millisecond tail latency by bypassing scoring & JSON unmarshalling
EPP Peak CPU (m)	1,227m (1.23 cores)	1,266m (1.27 cores)	1,626m (1.63 cores)	~32% higher EPP CPU in `optimized-baseline` due to JSON & hash compute
Envoy Peak CPU (m)	669m (0.67 cores)	679m (0.68 cores)	735m (0.74 cores)	Consistent sidecar proxy CPU overhead across workloads
Total System CPU (m)	1,896m (1.90 cores)	1,937m (1.94 cores)	2,317m (2.32 cores)	~22% lower total CPU for session affinity vs. baseline scoring
EPP Peak Heap Memory	104 MiB	183 MiB	310 MiB	~3x lower EPP heap memory without prefix/KV cache tracking
Total System Memory	164 MiB	244 MiB	365 MiB	~2.2x lower total system memory footprint
Router Configuration	session-affinity-filter.yaml	random-passthrough-parser.yaml	optimized-baseline.yaml	Configured plugins & request parsers

3. Key Architectural Insights: Why `session-affinity-filter` Outperforms

🟢 Elimination of JSON Parsing & Hashing Overhead vs. optimized-baseline:
- In optimized-baseline-multiturn, parsing the JSON body and computing approximate prefix hashes across conversation turns adds ~0.81 ms of median latency (0.39 ms vs 1.20 ms) and increases EPP heap memory from 104 MiB to 310 MiB (a ~3x memory footprint increase).
- Bypassing scoring evaluation on pinned requests avoids executing queue-scorer, kv-cache-utilization-scorer, and prefix-cache-scorer.
🟢 Lowest Resource Footprint:
- session-affinity-filter achieves the lowest overall CPU utilization (1.90 total cores saturated at 10 QPS) and the smallest memory footprint (104 MiB EPP heap).

Multi-Turn Chat & Session Affinity Performance Evaluation Report

1. Inference-Perf Multi-Turn Chat Job Explained

A. Why enable_multi_turn_chat: true is Required for Session Affinity

B. Why Prompt Sizing was Adjusted (2k / 500 / 200)

2. Multi-Turn Routing Performance Comparison Table

3. Key Architectural Insights: Why session-affinity-filter Outperforms

A. Why `enable_multi_turn_chat: true` is Required for Session Affinity

B. Why Prompt Sizing was Adjusted (`2k / 500 / 200`)

3. Key Architectural Insights: Why `session-affinity-filter` Outperforms

Uh oh!

[Scalability] Test session affinity + load aware HA configuration #1347

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions