ChatCompletionCache omits tool_choice from the cache key, so 'must call a tool' and 'must not' share one entry

[vasilisnasopoulos]

Summary

ChatCompletionCache accepts tool_choice, forwards it to the underlying client, and never includes it in the cache key. Requests that differ only in tool-selection policy — "you must call a tool", "you must not call any tool", "you must call this specific tool" — map to one cache entry, and the second and third callers are served the first one's answer.

The same omission is present on the streaming path.

Where

python/packages/autogen-ext/src/autogen_ext/models/cache/_chat_completion_cache.py:

def _check_cache(
    self,
    messages: Sequence[LLMMessage],
    tools: Sequence[Tool | ToolSchema],
    json_output: Optional[bool | type[BaseModel]],
    extra_create_args: Mapping[str, Any],
) -> tuple[...]:
    ...
    data = {
        "messages": [message.model_dump() for message in messages],
        "tools": [(tool.schema if isinstance(tool, Tool) else tool) for tool in tools],
        "json_output": json_output_data,
        "extra_create_args": extra_create_args,
    }
    cache_key = hashlib.sha256(json.dumps(data, sort_keys=True).encode()).hexdigest()

create() accepts tool_choice: Tool | Literal["auto","required","none"] and passes it to the real client, then calls _check_cache(messages, tools, json_output, extra_create_args) — without it. create_stream() repeats this verbatim.

Since tool_choice accepts a Tool object to force one specific tool, the cache also cannot distinguish "force get_weather" from "force send_email": a single key covers every tool-selection policy over the same prompt.

Reproduction

Offline, against a fake ChatCompletionClient whose answer depends on tool_choice, as a real one's would:

tool_choice='required' -> 'ANSWER-FOR-tool_choice=required'  cached=True
tool_choice='none'     -> 'ANSWER-FOR-tool_choice=required'  cached=True
tool_choice=send_email -> 'ANSWER-FOR-tool_choice=required'  cached=True
underlying client was actually called 1 time(s) (expected 3)
WRONG PAIRING: True

Same on the streaming path:

stream tool_choice='none' -> 'ANSWER-FOR-tool_choice=required' cached=True (client calls=1, expected 2)

Negative controls

Both pass, which is what makes this a gap rather than a cache that ignores context wholesale:

# differing system message DOES split the key
underlying client called 2 time(s) (expected 2) -> CORRECT, no collision

# differing tool list DOES split the key
underlying client called 2 time(s) (expected 2) -> CORRECT, no collision

The silence

No exception. warnings.catch_warnings(record=True) with simplefilter("always") around all three calls captured zero warnings — notable because the module already imports warnings and uses it elsewhere, so the mechanism for a warning exists and is not used here.

The only observable signal is CreateResult.cached == True, which reports "this came from the cache", not "this came from a different request than the one you made". A caller has no way to detect the mismatch.

What the docs say

python/docs/src/user-guide/core-user-guide/components/model-clients.ipynb:

Note that the caching is sensitive to the exact arguments provided to cached_client.create or cached_client.create_stream, so changing tools or json_output arguments might lead to a cache miss.

tool_choice is one of the exact arguments provided to create. The sentence names tools and json_output as examples rather than as an exhaustive list, so this is not a flat contradiction of an enumeration — but the general claim it makes is false for tool_choice, and a reader has no cue that the guarantee stops short of it.

The model-client protocol itself documents tool_choice as behaviour-changing (autogen_core/models/_model_client.py): "a single Tool object to force the model to use, 'auto' … 'required' … or 'none' to disable tool usage". A parameter the protocol defines as changing the answer is not in the key for the answer.

Scope

AssistantAgent._reflect_on_tool_use_flow passes tool_choice="none" explicitly, so the parameter is live in the framework's own path. In that specific call the reflection request also omits tools, so the message list and tool list differ too and no collision occurs there today. The exposure is to user code and future call sites that vary tool_choice while holding messages and tools fixed — precisely the "first let the model choose, then force a tool" retry pattern.

The fix is one line: pass tool_choice into _check_cache and add it to data, normalising a Tool to its name or schema.

An artefact of mine, not a finding

In my script r1.cached prints True even though the first call was a genuine miss. That is my own doing: InMemoryStore stores the CreateResult by reference, and the later cache hit mutates cached = True on that shared object, retroactively changing the first caller's result object. It is a real if minor aliasing wart in the same file, but it is separate from the report above and I am not claiming it here.

Not tested

Real model providers — everything ran against fakes, and I took the protocol documentation as the specification of what tool_choice means; if some provider ignored it the collision would be harmless for that provider, and the cache is provider-agnostic so this does not narrow the finding. DiskCacheStore and RedisStore were not exercised; the key is computed before the store is consulted, so the finding is store-independent, but the serialisation branches went untested. The underlying client's model name is also absent from the key, so two caches over different models sharing one store would collide — I did not build a reproduction because it requires deliberately sharing a store, and the docstring partly disclaims store lifecycle. Flagged as an untested lead, not a finding.

I have not checked whether this was raised before; a pointer to an existing issue is welcome and I will close this in favour of it.

Version

autogen-core / autogen-ext 0.7.5, source at 027ecf0a379bcc1d09956d46d12d44a3ad9cee14, Python 3.12, offline.

[MMaazesen]

I reproduced this offline on the current main commit (027ecf0) and would like to work on it. I have a focused local patch ready that includes tool_choice in the cache-key material, normalizing a concrete Tool through its schema.

The regression coverage exercises literal required/none choices, different concrete tool selections, and create_stream. All three tests fail on unpatched main; with the change, the focused module reports 28 passed, 1 skipped, and Ruff and Pyright are clean.

This changes the cache-key namespace, so existing persistent entries will become cold misses rather than risk being reused under a different tool-selection policy. I will keep the separate cached-result aliasing observation out of this PR.

I used AI assistance for repository investigation and implementation, and I personally reviewed the diff and ran the verification above.

[ZIFeIYUuuuuuu]

I opened #8213 with the cache-key fix and regression coverage for both create and create_stream: #8213