[EPD] Allow to deallocate ec cache

[knlnguyen1802]

Purpose

We can enable a flag "deallocate_cache" in start script to allow deallocate ec cache after the PD read it

Click to expand code example

###############################################################################
# Encoder worker
###############################################################################
CUDA_VISIBLE_DEVICES="$GPU_E" vllm serve "$MODEL" \
    --gpu-memory-utilization 0.01 \
    --port "$ENCODE_PORT" \
    --enforce-eager \
    --enable-request-id-headers \
    --no-enable-prefix-caching \
    --max-num-batched-tokens 114688 \
    --max-num-seqs 128 \
    --allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
    --ec-transfer-config '{
        "ec_connector": "ECExampleConnector",
        "ec_role": "ec_producer",
        "ec_connector_extra_config": {
            "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'",
            "deallocate_cache": true
        }
    }' \
    >"${ENC_LOG}" 2>&1 &

PIDS+=($!)

###############################################################################
# Prefill+Decode worker
###############################################################################
CUDA_VISIBLE_DEVICES="$GPU_PD" vllm serve "$MODEL" \
    --gpu-memory-utilization 0.7 \
    --port "$PREFILL_DECODE_PORT" \
    --enforce-eager \
    --enable-request-id-headers \
    --max-num-seqs 128 \
    --allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
    --ec-transfer-config '{
        "ec_connector": "ECExampleConnector",
        "ec_role": "ec_consumer",
        "ec_connector_extra_config": {
            "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'",
            "deallocate_cache": true
        }
    }' \
    >"${PD_LOG}" 2>&1 &

Design

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[knlnguyen1802]

cc: @fake0fan @khuonglmhw @herotai214 for review

[Copilot]

Pull request overview

This PR introduces encoder cache deallocation capabilities for the EC (Encoder Cache) transfer system. The main purpose is to enable automatic cleanup of remote encoder cache files when they are no longer needed, based on read/write count tracking.

Key changes include:

• Added metadata tracking (read_count, write_count) to monitor cache usage and trigger deallocation when appropriate
• Extended the scheduler and connector interfaces to pass cache hit information (local_hit, remote_hit) through the allocation flow
• Implemented file-based locking mechanism to prevent race conditions during concurrent metadata operations

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
vllm/v1/worker/ec_connector_model_runner_mixin.py	Removed debug log and added call to update remote cache state
vllm/v1/core/sched/scheduler.py	Updated scheduler to track and pass local/remote cache hit information through allocation flow
vllm/distributed/ec_transfer/ec_connector/example_connector.py	Implemented deallocation logic with metadata tracking, file locking, and cache cleanup
vllm/distributed/ec_transfer/ec_connector/base.py	Added abstract methods for cache state updates with hit information parameters

Comments suppressed due to low confidence (2)

vllm/distributed/ec_transfer/ec_connector/example_connector.py:1

• The variable name 'external_update_encoder_input' is ambiguous. Consider renaming it to 'encoder_cache_hit_tracking' or 'encoder_input_hit_states' to better reflect that it tracks cache hit status (local_hit, remote_hit) rather than just updates.

# SPDX-License-Identifier: Apache-2.0

vllm/distributed/ec_transfer/ec_connector/example_connector.py:1

• Corrected 'Store to option update' to 'Store to optionally update' for grammatical correctness.

# SPDX-License-Identifier: Apache-2.0

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The _mm_datas_need_saves dictionary is accessed without acquiring the _mm_state_lock, but it's modified later under the lock (line 318). This creates a potential race condition. The lock should be acquired before creating the snapshot with list() to ensure thread-safe iteration.

Suggested change

	# Iterate over a snapshot so we can modify the original safely.
	for mm_hash, num_token in list(self._mm_datas_need_saves.items()):
	# Take a snapshot under lock to ensure thread safety.
	with self._mm_state_lock:
	mm_datas_need_saves_snapshot = list(self._mm_datas_need_saves.items())
	for mm_hash, num_token in mm_datas_need_saves_snapshot:

[Copilot]

The _mm_datas_need_update_meta list is accessed without acquiring the _mm_state_lock, but it's modified later under the lock (line 320). This creates a potential race condition. The lock should be acquired before creating the snapshot with list() to ensure thread-safe iteration.

Suggested change

	for mm_hash in list(self._mm_datas_need_update_meta):
	with self._mm_state_lock:
	mm_datas_need_update_meta_snapshot = list(self._mm_datas_need_update_meta)
	for mm_hash in mm_datas_need_update_meta_snapshot:

[fake0fan]

I've left a few suggestions for your consideration.

Please also consider what if such an atomic count needs to be maintained by an EC connector.

Would this be easily scalable for other external storage?

[fake0fan]

Therefore, we need a file lock for each mm-encoded embedding?

[knlnguyen1802]

Right

[fake0fan]

I remember there was a previous PR that needed to avoid repeatedly checking all the encode caches every time it was accessed?

[knlnguyen1802]

It here: knlnguyen1802#7

[fake0fan]

Why is there still an update here?

[knlnguyen1802]

To make consistent between Encoder and PD, every access to local encoder cache also need to update remote cache