Task-Centric Memory #5227

rickyloynd-microsoft · 2025-01-28T01:47:21Z

(EXPERIMENTAL, RESEARCH IN PROGRESS)

In 2023 AutoGen introduced Teachable Agents that users could teach new facts, preferences and skills. But teachable agents were limited in several ways: They could only be ConversableAgent subclasses, they couldn't learn a new skill unless the user stated (in a single turn) both the task and how to solve it, and they couldn't learn on their own. Task-Centric Memory overcomes these limitations, allowing users to teach arbitrary agents (or teams) more flexibly and reliably, and enabling agents to learn from their own trial-and-error experiences.

This PR is large and complex. All of the files are new, and most of the added components depend on the others to run at all. But the review process can be accelerated if approached in the following order.

Start with the Task-Centric Memory README.
1. Install the memory extension locally, since it won't be in pypi until it's merged. In the agentic_memory branch, and the python/packages directory:
  - pip install -e autogen-agentchat
  - pip install -e autogen-ext[openai]
  - pip install -e autogen-ext[task-centric-memory]
2. Run the Quickstart sample code, then immediately open the ./pagelogs/quick/0 Call Tree.html file in a browser to view the work in progress.
3. Click through the web page links to see the details.
Continue through the rest of the main README to get a high-level overview of the architecture.
Read through the code samples README, running each of the 4 code samples while viewing their page logs.
Skim through the 4 code samples, along with their corresponding yaml config files:
1. chat_with_teachable_agent.py
2. eval_retrieval.py
3. eval_teachability.py
4. eval_learning_from_demonstration.py
5. eval_self_teaching.py
Read task_centric_memory_controller.py, referring back to the previously generated page logs as needed. This is the most important and complex file in the PR.
Read the remaining core files.
1. _task_centric_memory_bank.py
2. _string_similarity_map.py
3. _prompter.py
Read the supporting files in the utils dir.
1. teachability.py
2. apprentice.py
3. grader.py
4. page_logger.py
5. _functions.py

Make memory optional. Filter out insights with negative scores.

Refactor memory paths. Enrich page logging.

Seed messages with random int for variability.

Save sessions as yaml for readability.

Eval simplifications.

…ions.

…ified in settings.

python/.gitignore

python/packages/autogen-ext/imgs/task_centric_memory.png

husseinmozannar · 2025-02-22T01:42:09Z

...n/packages/autogen-ext/src/autogen_ext/task_centric_memory/task_centric_memory_controller.py

+            self.logger.info(task)
+
+            # Get a list of topics from the generalized task.
+            generalized_task = await self.prompter.generalize_task(task)


as discussed, there is an option to combine these two lines in a single API call, it would sacrifice accuracy potentially, but go from minimum 2 LLM calls to 1 LLM call

husseinmozannar · 2025-02-22T01:43:35Z

python/packages/autogen-ext/src/autogen_ext/task_centric_memory/_prompter.py

+        )
+        user_message = [
+            "Now put yourself in the mind of the students. What misconception led them to their incorrect answer?"
+        ]


there is potential to combine these three LLM calls into a single call, it depends on the LLM, but GPT-4o should be able to do this in a single shot with a COT prompt, other less capable models might need extra reflection

jackgerrits · 2025-02-26T17:45:49Z

A good way to convey the development status of this would be to put the module in autogen_ext.experimental.task_centric_memory. This will set clear expectations to users and give us the space to evolve

gagb · 2025-02-26T18:45:36Z

...n/packages/autogen-ext/src/autogen_ext/task_centric_memory/task_centric_memory_controller.py

use shorter name, e.g., "memory_controller.py"

gagb · 2025-02-26T18:46:14Z

python/packages/autogen-ext/src/autogen_ext/task_centric_memory/_task_centric_memory_bank.py

use shorter name

gagb · 2025-02-26T18:56:58Z

python/packages/autogen-ext/src/autogen_ext/task_centric_memory/utils/teachability.py

+        model_context: ChatCompletionContext,
+    ) -> UpdateContextResult:
+        """
+        Extracts any advice from the last user turn to be stored in memory,


Clarify how advice is extracted in the docs.
[advanced] expose parameters controlling this process in the API of the memory controller or in the teachability obj.

gagb · 2025-02-26T19:01:45Z

python/samples/task_centric_memory/README.md

+Execute the corresponding commands from this (autogen_ext/task_centric_memory) directory.
+
+
+### Making AssistantAgent teachable


Suggested change

### Making AssistantAgent teachable

### Making AssistantAgent Teachable

gagb · 2025-02-26T19:03:00Z

python/samples/task_centric_memory/README.md

+Each sample is contained in a separate python script, using data and configs stored in yaml files for easy modification.
+Note that since agent behavior is non-deterministic, results will vary between runs.
+
+To watch operations live in a browser and see how task-centric memory works,


How do I turn this on/off? Would prefer if the code didn't make changes outside the cwd.

gagb · 2025-02-26T19:04:08Z

python/samples/task_centric_memory/README.md

+starting with an empty memory bank.
+
+```bash
+rm -r memory_bank


unclear why this line is there. memory_bank dir was never mentioned before.

gagb · 2025-02-26T19:50:20Z

python/packages/autogen-ext/src/autogen_ext/task_centric_memory/utils/teachability.py

+
+class Teachability(Memory):
+    """
+    Gives an AssistantAgent the ability to learn quickly from user teachings, hints, and advice.


Expand the documentation, please.

gagb · 2025-02-26T19:51:41Z

python/packages/autogen-ext/src/autogen_ext/task_centric_memory/__init__.py

@@ -0,0 +1,3 @@
+from .task_centric_memory_controller import TaskCentricMemoryController
+
+__all__ = ["TaskCentricMemoryController"]


Should this expose Teachbility?

rickyloynd-microsoft · 2025-02-27T01:28:57Z

This PR is simply too large. Please propose your changes progressively and iteratively using a separate sequence of PRs. We cannot effectively review these changes as is.

@jackgerrits, I've moved the code under autogen_ext/experimentaltask_centric_memory to convey the status and manage expectations as you suggested. And reviewer feedback has led to many improvements. At this point, do you recommend a series of PRs to review the content in a more iterative fashion?

ekzhu

As a next step #5542.

ekzhu · 2025-02-27T17:11:04Z

python/packages/autogen-core/docs/src/reference/index.md

@@ -52,6 +52,8 @@ python/autogen_ext.models.openai
 python/autogen_ext.models.replay
 python/autogen_ext.models.azure
 python/autogen_ext.models.semantic_kernel
+python/autogen_ext.experimental.task_centric_memory


Can we move this below auth so we are not breaking up the models.

victordibia · 2025-02-27T21:40:21Z

The teachability example is pretty cool, shows how the MemoryController can be used with the an AssistantAgent via the Memory interface.

rickyloynd-microsoft · 2025-02-27T22:55:20Z

The teachability example is pretty cool, shows how the MemoryController can be used with the an AssistantAgent via the Memory interface.

And with very little code!
agent = AssistantAgent(model_client=client, memory=[Teachability(MemoryController(False, client))])

rickyloynd-microsoft added 30 commits November 29, 2024 12:31

initial checkin

442a9d8

support for extensive evaluations

f8584cd

Enhance retrieval with task generalization and insight validation

607e7ff

Support TRAPI client.

b045636

Make memory optional. Filter out insights with negative scores.

Restoring earlier results, and general cleanup.

63b28d7

Merge branch 'refs/heads/main' into agentic_memory

b921d83

Modify imports after merge from main.

9dfb074

Log model and token counts.

93a5ca4

Only instantiate the client once.

2cb9344

Fix bug that was duplicating insights across trials.

878f458

Add the Grader class.

21562f1

Refactor memory paths. Enrich page logging.

Adjustments for comparison tests.

3a40b30

Test generalization over multiple tasks.

8622c5e

Add teachability and a test for it.

20b26c1

Learning from demonstration, in-progress.

9d47227

In memory retrieval, validate insights separately rather than together.

52d4e00

Finish learning from demonstration.

6b15777

Seed messages with random int for variability.

Added RecordableChatCompletionClient as a guardrail during refactoring.

a18674c

Ran 3 evals with session recording and replay.

52e213e

Add results to recorded sessions, including session length.

a440b0a

Save sessions as yaml for readability.

Use yaml file for eval settings.

cab51f1

Simplify paths and other settings.

d91e58c

Renamed the memory classes.

f1d7a2f

Apprentice.

17d4c42

Eval simplifications.

Moved test into the evaluator, and removed eval.py's other util funct…

19654e8

…ions.

renaming

7aa20c1

Rerouted calls to AgenticMemoryController through FastLearner.

83a7ddc

Replace task_assignment_callback with AgentWrapper.

3047c1c

Segregate files into subfolders, eval framework vs. implementation, etc.

1f20b79

Rename FastLearner subclass to Apprentice, and import it only as spec…

de4c12b

…ified in settings.

restore previous uv.lock

64dc3c0

gagb reviewed Feb 20, 2025

View reviewed changes

python/.gitignore Outdated Show resolved Hide resolved

gagb reviewed Feb 20, 2025

View reviewed changes

python/packages/autogen-ext/imgs/task_centric_memory.png Outdated Show resolved Hide resolved

rickyloynd-microsoft added 2 commits February 20, 2025 18:26

changes for webby

e15d0eb

experimental

af362f6

husseinmozannar reviewed Feb 22, 2025

View reviewed changes

rickyloynd-microsoft added 2 commits February 23, 2025 13:14

docs

4d6c9f4

Add Teachability(Memory)

261fe6f

gagb reviewed Feb 26, 2025

View reviewed changes

gagb self-requested a review February 26, 2025 20:31

gagb approved these changes Feb 26, 2025

View reviewed changes

addressing reviewer feedback

7cf1d47

rickyloynd-microsoft added 3 commits February 26, 2025 17:53

doc-build fixes

d12a2b4

doc-build fixes

f482b23

doc-build fixes

7719026

ekzhu approved these changes Feb 27, 2025

View reviewed changes

rickyloynd-microsoft added 2 commits February 27, 2025 10:09

Move to end

bf22657

Merge branch 'main' into agentic_memory

b897b15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task-Centric Memory #5227

Task-Centric Memory #5227

rickyloynd-microsoft commented Jan 28, 2025 •

edited

Loading

husseinmozannar Feb 22, 2025

husseinmozannar Feb 22, 2025

jackgerrits commented Feb 26, 2025

gagb Feb 26, 2025

gagb Feb 26, 2025

gagb Feb 26, 2025

gagb Feb 26, 2025

gagb Feb 26, 2025

gagb Feb 26, 2025

gagb Feb 26, 2025

gagb Feb 26, 2025

rickyloynd-microsoft commented Feb 27, 2025

ekzhu left a comment

ekzhu Feb 27, 2025

victordibia commented Feb 27, 2025

rickyloynd-microsoft commented Feb 27, 2025

		Execute the corresponding commands from this (autogen_ext/task_centric_memory) directory.


		### Making AssistantAgent teachable

		@@ -0,0 +1,3 @@
		from .task_centric_memory_controller import TaskCentricMemoryController

		__all__ = ["TaskCentricMemoryController"]

Task-Centric Memory #5227

Are you sure you want to change the base?

Task-Centric Memory #5227

Conversation

rickyloynd-microsoft commented Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackgerrits commented Feb 26, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rickyloynd-microsoft commented Feb 27, 2025

ekzhu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

victordibia commented Feb 27, 2025

rickyloynd-microsoft commented Feb 27, 2025

rickyloynd-microsoft commented Jan 28, 2025 •

edited

Loading