When will the agent use the contents of the memory?

[zhangjy859]

Hi,

I am trying to create a agent with memory, I use the demo code from document

# ...
# Use a temporary directory for ChromaDB persistence
with tempfile.TemporaryDirectory() as tmpdir:
    chroma_user_memory = ChromaDBVectorMemory(
        config=PersistentChromaDBVectorMemoryConfig(
            collection_name="preferences",
            persistence_path=tmpdir,  # Use the temp directory here
            k=2,  # Return top k results
            score_threshold=0.4,  # Minimum similarity score
            embedding_function_config=SentenceTransformerEmbeddingFunctionConfig(
                model_name="all-MiniLM-L6-v2"  # Use default model for testing
            ),
        )
    )
    # Add user preferences to memory
    asyncio.run(chroma_user_memory.add(
        MemoryContent(
            content="The weather should be in metric units",
            mime_type=MemoryMimeType.TEXT,
            metadata={"category": "preferences", "type": "units"},
        )
    ))

    asyncio.run(chroma_user_memory.add(
        MemoryContent(
            content="Meal recipe must be vegan",
            mime_type=MemoryMimeType.TEXT,
            metadata={"category": "preferences", "type": "dietary"},
        )
    ))


    async def get_weather(city: str, units: str = "imperial") -> str:
        if units == "imperial":
            return f"The weather in {city} is 73 °F and Sunny."
        elif units == "metric":
            return f"The weather in {city} is 23 °C and Sunny."
        else:
            return f"Sorry, I don't know the weather in {city}."

    # Create assistant agent with ChromaDB memory
    assistant_agent = AssistantAgent(
        name="assistant_agent",
        model_client=custom_model_client,
        tools=[get_weather],
        memory=[chroma_user_memory],
    )

    stream = assistant_agent.run_stream(task="What's the weather in New York? ")
    asyncio.run(Console(stream))

    asyncio.run(custom_model_client.close())
    asyncio.run(chroma_user_memory.close())

Its ok, and output

---------- TextMessage (user) ----------
What's the weather in New York? 
---------- MemoryQueryEvent (assistant_agent) ----------
[MemoryContent(content='The weather should be in metric units', mime_type='MemoryMimeType.TEXT', metadata={'mime_type': 'MemoryMimeType.TEXT', 'category': 'preferences', 'type': 'units', 'score': 0.41220688819885254, 'id': '61a4be3c-b2c6-4a03-8e6e-9e31689bbbbe'})]
---------- ToolCallRequestEvent (assistant_agent) ----------
[FunctionCall(id='call_98806457', arguments='{"city":"New York","units":"metric"}', name='get_weather')]
---------- ToolCallExecutionEvent (assistant_agent) ----------
[FunctionExecutionResult(content='The weather in New York is 23 °C and Sunny.', name='get_weather', call_id='call_98806457', is_error=False)]
---------- ToolCallSummaryMessage (assistant_agent) ----------
The weather in New York is 23 °C and Sunny.

But, if we change the task promopt

stream = assistant_agent.run_stream(task="How are you?")

output:

---------- TextMessage (user) ----------
How are you?
---------- ThoughtEvent (assistant_agent) ----------
...
---------- TextMessage (assistant_agent) ----------
I'm doing well, thank you! How can I assist you today?

I noticed that the agent is no longer generating MemoryQueryEvent. Does this mean that the agent will not further use the content in the Memory? But if it doesn't even Query the content in the memory, how can it determine whether to perform a MemoryQuery or not?

[Christopher Krah (chrikrah)]

zhangjy859 the agent never decides. _assistant_agent.py:940 calls _update_model_context_with_memory on every turn, under no condition. ChromaDBVectorMemory.update_context then runs the similarity search, your conversational prompt included. Only the event is conditional. The guard at _assistant_agent.py:1047 wraps the MemoryQueryEvent construction and nothing else.

The memory also stays live. update_context appends a SystemMessage to the context. It stays there for the rest of the conversation. Both of your scripts call run_stream once, so each paste is a fresh first turn with nothing earlier to carry.

Two turns against ListMemory, one stored item, and the message types the model client got:

$ python -c "import autogen_agentchat; print(autogen_agentchat.__file__, autogen_agentchat.__version__)"
.../autogen-agentchat/src/autogen_agentchat/__init__.py 0.7.5     # main @ 027ecf0a, py 3.12.3

$ python memory_probe.py   # AssistantAgent + ListMemory(1 item), ChatCompletionClient stubbed to record its messages
turn 1: ['SystemMessage', 'UserMessage', 'SystemMessage']
    'You are a helpful AI assistant. ...'
    '\nRelevant memory content (in chronological order):\n1. The weather should be in metric units\n'
turn 2: ['SystemMessage', 'UserMessage', 'SystemMessage', 'AssistantMessage', 'UserMessage', 'SystemMessage']
    'You are a helpful AI assistant. ...'
    '\nRelevant memory content (in chronological order):\n1. The weather should be in metric units\n'
    '\nRelevant memory content (in chronological order):\n1. The weather should be in metric units\n'
    events: ['TextMessage', 'MemoryQueryEvent', 'TextMessage']
# that header is ListMemory's; ChromaDBVectorMemory writes '\nRelevant memory content:\n' (_chromadb.py:328)
# not run: ChromaDBVectorMemory itself, which needs chromadb and the embedding model

score_threshold=0.4 does all of the work. k caps how many rows come back (n_results, _chromadb.py:372), and two stored items never reach that cap. The bar also sits higher than the sample suggests. The class docstring at _chromadb.py:162 retrieves the Celsius preference for a temperature question, a dead-on match, at 0.313. Your own hit cleared 0.4 by 0.012.

That 0.4 is the number worth re-testing. I can post a Chroma-side run by Wednesday that sweeps the threshold across both prompts.