Persisting Files Locally for Sequential Agent Conversations

[anu43]

I'm working on a project that simulates a data science team. Different agents are responsible for distinct tasks, such as data reading, preprocessing, feature engineering, etc. The goal is to pass an updated dataset from one agent to the next during sequential interactions.

I've experimented with various approaches to have agents write to a specific folder in our Docker-based code execution environment. The idea was that if agents can read files, they should also be able to save them. However, despite specifying file-saving instructions in the system prompt and task descriptions, the agents have not persisted in any local files.

Has anyone successfully implemented local file persistence between sequential agent chats? Any insights or suggestions would be greatly appreciated.

[ekzhu]

@anu43, if you post your code, package version number and environment, it will make it easier for others to help and identify specific improvement.

Generally speaking, if you have a specific workflow and a deterministic sequence of steps in mind, it is better to not use Agents, but simply use workflows to chain your steps together.

[anu43]

you're right. let me share the code.

from pathlib import Path

from autogen import ConversableAgent, GroupChat, GroupChatManager, UserProxyAgent
from autogen.coding import DockerCommandLineCodeExecutor
from config import LLM_CONFIG

# Temperature configuration
LLM_CONFIG["temperature"] = 0.3

# Work directory
WORK_DIR = Path("data")
WORK_DIR.mkdir(exist_ok=True)

# Docker Command Line Code Executor
docker_exec = DockerCommandLineCodeExecutor(
    image="python:3.10-slim", container_name="simple-python-executor", work_dir=WORK_DIR
)


# Create agents
admin = UserProxyAgent(
    name="admin",
    human_input_mode="NEVER",
    code_execution_config=False,
    llm_config=False,
)

code_executor = UserProxyAgent(
    name="code-executor",
    is_termination_msg=lambda msg: msg.get("content") is not None
    and "TERMINATE" in msg["content"],
    human_input_mode="NEVER",
    code_execution_config={"executor": docker_exec},
    llm_config=LLM_CONFIG,
    system_message="You have only one job: code execution. Nothing else. If you encounter any execution error, you reflect that.",
)

data_collector = ConversableAgent(
    name="data-collector",
    system_message="You are only responsible for providing Python script to read data and bash code if any package is missing.",
    human_input_mode="NEVER",
    llm_config=LLM_CONFIG,
)


principal_ds = ConversableAgent(
    name="principal-data-scientist",
    system_message="You are a principal data scientist. You share your expertise with the data scientist team. You only share insights and methods to conduct. You don't tend to write code. That is the data scientists' job.",
    human_input_mode="NEVER",
    llm_config=LLM_CONFIG,
)

data_scientist = ConversableAgent(
    name="data-scientist",
    system_message="You are a data scientist specializing in Python development. You translate the principal data scientist’s conceptual ideas into well-structured, efficient Python scripts. While following their direction, you also apply your own expertise to optimize logic, enhance performance, and ensure best practices. Provide a complete, well-documented Python script that accurately reflects the given idea.",
)

data_visualizer = ConversableAgent(
    name="data-visualizer",
    system_message="You are a seasoned data visualization expert skilled in Python. Given a structured dataset, you will write efficient Python code to generate insightful charts that reveal trends, distributions, and correlations. Use libraries such as Matplotlib, Seaborn, or Plotly to create clear, well-labeled visualizations. Ensure the code is optimized and includes brief explanations of the insights derived from the charts.",
)

group_chat_1 = GroupChat(
    agents=[code_executor, data_collector],
    messages=[],
    max_round=10,
    admin_name="admin",
    speaker_selection_method="round_robin",
)

# Group Chat Manager (to act as recipient)
group_chat_manager_1 = GroupChatManager(
    groupchat=group_chat_1,
    llm_config=LLM_CONFIG,
)

group_chat_2 = GroupChat(
    agents=[code_executor, principal_ds, data_scientist],
    messages=[],
    max_round=10,
    admin_name="admin",
    speaker_selection_method="auto",
    allow_repeat_speaker=False,
)

# Group Chat Manager (to act as recipient)
group_chat_manager_2 = GroupChatManager(
    groupchat=group_chat_2,
    llm_config=LLM_CONFIG,
)

# Start a Sequential Chat (individual processing -> collaborative discussion)
sequential_result = admin.initiate_chats(
    [
        {
            "recipient": group_chat_manager_1,
            "message": "Read 100 samples from the titanic.csv. Save it as titanic-sample.csv. Print the sample.",
            "summary_method": "last_msg",
        },
        {
            "recipient": group_chat_manager_2,
            "message": "Read the data titanic-sample.csv. Write a Python script to preprocess this dataset before Exploratory Data Analysis (EDA). Focus only on preprocessing, including handling missing values, encoding categorical variables, and standardizing numerical features where necessary. Apply those to the titanic.csv. The final cleaned dataset should be saved as 'titanic-preprocessed.csv' in CSV format. Do not perform any visualization or analysis—only preprocessing..",
            "summary_method": "last_msg",
        },
    ]
)

package version: autogen-agentchat==0.2.40
ProductName: macOS
ProductVersion: 15.3

Generally speaking, if you have a specific workflow and a deterministic sequence of steps in mind, it is better to not use Agents, but simply use workflows to chain your steps together.

I am also aware of it, thanks. The intention is to adapt dynamically to any dataset.