notebook/agentchat_teachable_oai_assistants.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making OpenAI Assistants Teachable\n",
    "\n",
    "Conversational assistants based on LLMs can remember the current chat with the user, and can even demonstrate in-context learning of things that the user teaches the assistant during the chat. But these memories and learnings are lost once the chat is over, or when a single chat grows too long for the LLM to handle effectively. In subsequent chats, the user is forced to repeat any necessary instructions over and over.\n",
    "\n",
    "The optional agent capability called `Teachability` addresses these limitations by persisting user teachings across chat boundaries in long-term memory (a vector database). Memories (called memos) are created and saved to disk throughout a conversation, then loaded from disk later. Instead of copying all the memos into the context window, which would eat up valuable space, individual memos are retrieved into context only as needed. This allows the user to teach many facts, preferences and skills to the teachable agent just once, and have it remember them in later chats.\n",
    "\n",
    "In making decisions about memo storage and retrieval, `Teachability` calls an instance of `TextAnalyzerAgent` to analyze pieces of text in several different ways. This adds extra LLM calls involving a relatively small number of tokens. These calls can add a few seconds to the time a user waits for a response.\n",
    "\n",
    "This notebook demonstrates how `Teachability` can be added to instances of `GPTAssistantAgent`\n",
    "so that they can learn facts, preferences, and skills from users. As explained [here](https://docs.ag2.ai/docs/topics/openai-assistant/gpt_assistant_agent), each instance of `GPTAssistantAgent` wraps an OpenAI Assistant that can be given a set of tools including functions, code interpreter, and retrieval. Assistants with these tools are demonstrated in separate standalone sections below, which can be run independently.\n",
    "\n",
    "## Requirements\n",
    "\n",
    "AG2 requires `Python>=3.9`. To run this notebook example, please install the [teachable] option.\n",
    "```bash\n",
    "pip install \"autogen[teachable]\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "# %pip install \"autogen[teachable]\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set your API Endpoint\n",
    "\n",
    "The [`config_list_from_json`](https://docs.ag2.ai/docs/api-reference/autogen/config_list_from_json#config-list-from-json) function loads a list of configurations from an environment variable or a json file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "\n",
    "import requests\n",
    "\n",
    "import autogen\n",
    "from autogen import UserProxyAgent, config_list_from_json\n",
    "\n",
    "# from autogen.agentchat import UserProxyAgent\n",
    "from autogen.agentchat.contrib.capabilities.teachability import Teachability\n",
    "from autogen.agentchat.contrib.gpt_assistant_agent import GPTAssistantAgent\n",
    "\n",
    "config_list = autogen.config_list_from_json(\n",
    "    env_or_file=\"OAI_CONFIG_LIST\",\n",
    "    file_location=\".\",\n",
    "    filter_dict={\n",
    "        \"model\": [\"gpt-4\", \"gpt-4-1106-preview\", \"gpt4\", \"gpt-4-32k\"],\n",
    "    },\n",
    ")\n",
    "\n",
    "print(config_list[0][\"model\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\". It filters the configs by models (you can filter by other keys as well). After application of the filter shown above, only the gpt-4 models are considered.\n",
    "\n",
    "The config list may look like the following:\n",
    "```python\n",
    "config_list = [\n",
    "    {\n",
    "        'model': 'gpt-4-1106-preview',\n",
    "        'api_key': '<your OpenAI API key here>',\n",
    "    },\n",
    "    {\n",
    "        'model': 'gpt-4',\n",
    "        'api_key': '<your OpenAI API key here>',\n",
    "    },\n",
    "    {\n",
    "        'model': 'gpt-4',\n",
    "        'api_key': '<your Azure OpenAI API key here>',\n",
    "        'base_url': '<your Azure OpenAI API base here>',\n",
    "        'api_type': 'azure',\n",
    "        'api_version': '2024-02-01',\n",
    "    },\n",
    "    {\n",
    "        'model': 'gpt-4-32k',\n",
    "        'api_key': '<your Azure OpenAI API key here>',\n",
    "        'base_url': '<your Azure OpenAI API base here>',\n",
    "        'api_type': 'azure',\n",
    "        'api_version': '2024-02-01',\n",
    "    },\n",
    "]\n",
    "```\n",
    "\n",
    "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
    "\n",
    "You can set the value of config_list in other ways if you prefer, e.g., loading from a YAML file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Teachable OpenAI Assistant with Functions\n",
    "This section is based on [agentchat_oai_assistant_function_call.ipynb](https://github.com/ag2ai/ag2/blob/main/notebook/agentchat_oai_assistant_function_call.ipynb), which demonstrates an assistant accomplishing a task through **function calling**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function schema and implementation\n",
    "This example leverages OSS Insight (Open Source Software Insight) for advanced GitHub data analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logger = logging.getLogger(__name__)\n",
    "logger.setLevel(logging.WARNING)\n",
    "\n",
    "ossinsight_api_schema = {\n",
    "    \"name\": \"ossinsight_data_api\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\n",
    "            \"question\": {\n",
    "                \"type\": \"string\",\n",
    "                \"description\": (\n",
    "                    \"Enter your GitHub data question in the form of a clear and specific question to ensure the returned data is accurate and valuable. \"\n",
    "                    \"For optimal results, specify the desired format for the data table in your request.\"\n",
    "                ),\n",
    "            }\n",
    "        },\n",
    "        \"required\": [\"question\"],\n",
    "    },\n",
    "    \"description\": \"This is an API endpoint allowing users (analysts) to input question about GitHub in text format to retrieve the related and structured data.\",\n",
    "}\n",
    "\n",
    "\n",
    "def get_ossinsight(question):\n",
    "    \"\"\"Retrieve the top 10 developers with the most followers on GitHub.\"\"\"\n",
    "    url = \"https://api.ossinsight.io/explorer/answer\"\n",
    "    headers = {\"Content-Type\": \"application/json\"}\n",
    "    data = {\"question\": question, \"ignoreCache\": True}\n",
    "\n",
    "    response = requests.post(url, headers=headers, json=data)\n",
    "    if response.status_code == 200:\n",
    "        answer = response.json()\n",
    "    else:\n",
    "        return f\"Request to {url} failed with status code: {response.status_code}\"\n",
    "\n",
    "    report_components = []\n",
    "    report_components.append(f\"Question: {answer['question']['title']}\")\n",
    "    if answer[\"query\"][\"sql\"] != \"\":\n",
    "        report_components.append(f\"querySQL: {answer['query']['sql']}\")\n",
    "\n",
    "    if answer.get(\"result\", None) is None or len(answer[\"result\"][\"rows\"]) == 0:\n",
    "        result = \"Result: N/A\"\n",
    "    else:\n",
    "        result = \"Result:\\n  \" + \"\\n  \".join([str(row) for row in answer[\"result\"][\"rows\"]])\n",
    "    report_components.append(result)\n",
    "\n",
    "    if answer.get(\"error\", None) is not None:\n",
    "        report_components.append(f\"Error: {answer['error']}\")\n",
    "    return \"\\n\\n\".join(report_components) + \"\\n\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the OpenAI Assistant with function calling as a tool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assistant_id = os.environ.get(\"ASSISTANT_ID\", None)\n",
    "config_list = config_list_from_json(\"OAI_CONFIG_LIST\")\n",
    "llm_config = {\n",
    "    \"config_list\": config_list,\n",
    "    \"assistant_id\": assistant_id,\n",
    "    \"tools\": [\n",
    "        {\n",
    "            \"type\": \"function\",\n",
    "            \"function\": ossinsight_api_schema,\n",
    "        }\n",
    "    ],\n",
    "}\n",
    "\n",
    "oss_analyst = GPTAssistantAgent(\n",
    "    name=\"OSS_Analyst\",\n",
    "    instructions=(\n",
    "        \"Hello, Open Source Project Analyst. You'll conduct comprehensive evaluations of open source projects or organizations on the GitHub platform, \"\n",
    "        \"analyzing project trajectories, contributor engagements, open source trends, and other vital parameters. \"\n",
    "        \"Please carefully read the context of the conversation to identify the current analysis question or problem that needs addressing.\"\n",
    "    ),\n",
    "    llm_config=llm_config,\n",
    "    verbose=True,\n",
    ")\n",
    "oss_analyst.register_function(\n",
    "    function_map={\n",
    "        \"ossinsight_data_api\": get_ossinsight,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Give the assistant a task involving function calls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_proxy = UserProxyAgent(\n",
    "    name=\"user_proxy\",\n",
    "    code_execution_config={\n",
    "        \"work_dir\": \"coding\",\n",
    "        \"use_docker\": False,\n",
    "    },  # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.\n",
    "    is_termination_msg=lambda msg: \"TERMINATE\" in msg[\"content\"],\n",
    "    human_input_mode=\"NEVER\",\n",
    "    max_consecutive_auto_reply=0,\n",
    ")\n",
    "\n",
    "user_proxy.initiate_chat(oss_analyst, message=\"Top 10 developers with the most followers\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make the assistant teachable\n",
    "We can make any `ConversableAgent` teachable by adding a `Teachability` object to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "teachability = Teachability(reset_db=True, llm_config={\"config_list\": config_list})\n",
    "teachability.add_to_agent(oss_analyst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the teachable assistant\n",
    "This time let's teach the assistant a more specific way of formatting lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_proxy.initiate_chat(\n",
    "    oss_analyst,\n",
    "    message=\"List the top 10 developers with the most followers. When listing things, please put them in a table.\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's clear the chat history to see whether the assistant can remember to use our preferred formatting in a new chat without having to be told again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_proxy.initiate_chat(oss_analyst, message=\"List the top 10 developers with the most followers.\", clear_history=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the teachable assistant\n",
    "All OpenAI Assistants can be created, viewed and deleted manually through OpenAI's [website](https://platform.openai.com/assistants). They can also be deleted through the `GPTAssistantAgent` instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oss_analyst.delete_assistant()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Teachable OpenAI Assistant with Code Interpreter\n",
    "This section is based on [agentchat_oai_code_interpreter.ipynb](https://github.com/ag2ai/ag2/blob/main/notebook/agentchat_oai_code_interpreter.ipynb), which demonstrates an assistant accomplishing a task by **writing code and executing it** in a sandbox."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the OpenAI Assistant with code interpreter as a tool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initiate an agent equipped with code interpreter\n",
    "coder_assistant = GPTAssistantAgent(\n",
    "    name=\"Coder_Assistant\",\n",
    "    llm_config={\n",
    "        \"tools\": [{\"type\": \"code_interpreter\"}],\n",
    "        \"config_list\": config_list,\n",
    "    },\n",
    "    instructions=\"You are an expert at solving math questions. Write code and run it to solve math problems. Reply TERMINATE when the task is solved and there is no problem.\",\n",
    ")\n",
    "\n",
    "user_proxy = UserProxyAgent(\n",
    "    name=\"user_proxy\",\n",
    "    is_termination_msg=lambda msg: \"TERMINATE\" in msg[\"content\"],\n",
    "    code_execution_config={\n",
    "        \"work_dir\": \"coding\",\n",
    "        \"use_docker\": False,  # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.\n",
    "    },\n",
    "    human_input_mode=\"NEVER\",\n",
    "    max_consecutive_auto_reply=0,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Give the assistant a task involving code execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_proxy.initiate_chat(\n",
    "    coder_assistant, message=\"If $725x + 727y = 1500$ and $729x+ 731y = 1508$, what is the value of $x - y$ ?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make the assistant teachable\n",
    "We can make any `ConversableAgent` teachable by adding a `Teachability` object to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "teachability = Teachability(reset_db=True, llm_config={\"config_list\": config_list})\n",
    "teachability.add_to_agent(coder_assistant)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the teachable assistant\n",
    "This time let's teach the assistant to show its work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_proxy.initiate_chat(\n",
    "    coder_assistant,\n",
    "    message=\"If $725x + 727y = 1500$ and $729x+ 731y = 1508$, what is the value of $x - y$ ? After finding the values of variables, always explain how to find the solution.\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's clear the chat history to see whether the assistant can remember to show its work in a new chat without having to be told again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_proxy.initiate_chat(\n",
    "    coder_assistant,\n",
    "    message=\"If $725x + 727y = 1500$ and $729x+ 731y = 1508$, what is the value of $x - y$ ?\",\n",
    "    clear_history=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the teachable assistant\n",
    "All OpenAI Assistants can be created, viewed and deleted manually through OpenAI's [website](https://platform.openai.com/assistants). They can also be deleted through the `GPTAssistantAgent` instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coder_assistant.delete_assistant()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Teachable OpenAI Assistant with Retrieval\n",
    "This section is based on [agentchat_oai_assistant_retrieval.ipynb](https://github.com/ag2ai/ag2/blob/main/notebook/agentchat_oai_assistant_retrieval.ipynb), which demonstrates an assistant accomplishing a task through **retrieval augmented generation (RAG)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the OpenAI Assistant with retrieval as a tool\n",
    "For this example, first upload the [conversable_agent.py](https://github.com/ag2ai/ag2/blob/main/autogen/agentchat/conversable_agent.py) file to your OpenAI API account. This can be done manually through the [website](https://platform.openai.com/assistants). Then find the uploaded File ID on the [Files page](https://platform.openai.com/files), and paste that ID into the `file_ids` list in the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logger = logging.getLogger(__name__)\n",
    "logger.setLevel(logging.WARNING)\n",
    "\n",
    "assistant_id = os.environ.get(\"ASSISTANT_ID\", None)\n",
    "\n",
    "config_list = config_list_from_json(\"OAI_CONFIG_LIST\")\n",
    "llm_config = {\n",
    "    \"config_list\": config_list,\n",
    "    \"assistant_id\": assistant_id,\n",
    "    \"tools\": [{\"type\": \"retrieval\"}],\n",
    "    \"file_ids\": [\"file-HPDOsp8k4dz95QRx9bnfNJHp\"],\n",
    "}\n",
    "\n",
    "rag_assistant = GPTAssistantAgent(\n",
    "    name=\"RAG_Assistant\", instructions=\"You are adapt at question answering\", llm_config=llm_config\n",
    ")\n",
    "\n",
    "user_proxy = UserProxyAgent(\n",
    "    name=\"user_proxy\",\n",
    "    code_execution_config=False,\n",
    "    is_termination_msg=lambda msg: \"TERMINATE\" in msg[\"content\"],\n",
    "    human_input_mode=\"ALWAYS\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Give the assistant a task involving file retrieval\n",
    "When prompted, type \"Does the file contain any bugs?\". On the next prompt, type \"exit\" to end the chat."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_proxy.initiate_chat(rag_assistant, message=\"What is the name of the class of agents in the file I gave you?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make the assistant teachable\n",
    "We can make any `ConversableAgent` teachable by adding a `Teachability` object to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "teachability = Teachability(reset_db=True, llm_config={\"config_list\": config_list})\n",
    "teachability.add_to_agent(rag_assistant)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the teachable assistant\n",
    "This time let's teach the assistant to report its confidence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_proxy.initiate_chat(\n",
    "    rag_assistant,\n",
    "    message=\"What is the name of the class of agents in the file I gave you? When you answer a question based on a file, always report your confidence in the answer as a percentage.\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's clear the chat history to see whether the assistant can remember to report its confidence in a new chat without having to be told again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_proxy.initiate_chat(\n",
    "    rag_assistant, message=\"What is the name of the class of agents in the file I gave you?\", clear_history=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the teachable assistant\n",
    "All OpenAI Assistants can be created, viewed and deleted manually through OpenAI's [website](https://platform.openai.com/assistants). They can also be deleted through the `GPTAssistantAgent` instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rag_assistant.delete_assistant()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "front_matter": {
   "description": "Teach OpenAI assistants.",
   "tags": [
    "teachability",
    "capability",
    "learning",
    "RAG",
    "OpenAI Assistant"
   ]
  },
  "kernelspec": {
   "display_name": "flaml",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}