notebook/agentchat_RetrieveChat.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using RetrieveChat for Retrieve Augmented Code Generation and Question Answering\n",
    "\n",
    "AG2 offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation.\n",
    "Please find documentation about this feature [here](https://docs.ag2.ai/docs/Use-Cases/agent_chat).\n",
    "\n",
    "RetrieveChat is a conversational system for retrieval-augmented code generation and question answering. In this notebook, we demonstrate how to utilize RetrieveChat to generate code and answer questions based on customized documentations that are not present in the LLM's training dataset. RetrieveChat uses the `AssistantAgent` and `RetrieveUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/ag2ai/ag2/blob/main/notebook/agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `RetrieveUserProxyAgent` implement a different auto-reply mechanism corresponding to the RetrieveChat prompts.\n",
    "\n",
    "## Table of Contents\n",
    "We'll demonstrate six examples of using RetrieveChat for code generation and question answering:\n",
    "\n",
    "- [Example 1: Generate code based off docstrings w/o human feedback](#example-1)\n",
    "- [Example 2: Answer a question based off docstrings w/o human feedback](#example-2)\n",
    "- [Example 3: Generate code based off docstrings w/ human feedback](#example-3)\n",
    "- [Example 4: Answer a question based off docstrings w/ human feedback](#example-4)\n",
    "- [Example 5: Solve comprehensive QA problems with RetrieveChat's unique feature `Update Context`](#example-5)\n",
    "- [Example 6: Solve comprehensive QA problems with customized prompt and few-shot learning](#example-6)\n",
    "\n",
    "\n",
    "````{=mdx}\n",
    ":::info Requirements\n",
    "Some extra dependencies are needed for this notebook, which can be installed via pip:\n",
    "\n",
    "```bash\n",
    "pip install autogen[retrievechat] flaml[automl]\n",
    "```\n",
    "\n",
    "For more information, please refer to the [installation guide](/docs/user-guide/basic-concepts/installing-ag2).\n",
    ":::\n",
    "````"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set your API Endpoint\n",
    "\n",
    "The [`config_list_from_json`](https://docs.ag2.ai/docs/api-reference/autogen/config_list_from_json#config-list-from-json) function loads a list of configurations from an environment variable or a json file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "import chromadb\n",
    "\n",
    "import autogen\n",
    "from autogen import AssistantAgent\n",
    "from autogen.agentchat.contrib.retrieve_user_proxy_agent import RetrieveUserProxyAgent\n",
    "\n",
    "# Accepted file formats for that can be stored in\n",
    "# a vector database instance\n",
    "from autogen.retrieve_utils import TEXT_FORMATS\n",
    "\n",
    "config_list = autogen.config_list_from_json(\"OAI_CONFIG_LIST\")\n",
    "\n",
    "assert len(config_list) > 0\n",
    "print(\"models to use: \", [config_list[i][\"model\"] for i in range(len(config_list))])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "````{=mdx}\n",
    ":::tip\n",
    "Learn more about configuring LLMs for agents [here](/docs/topics/llm_configuration).\n",
    ":::\n",
    "````\n",
    "\n",
    "## Construct agents for RetrieveChat\n",
    "\n",
    "We start by initializing the `AssistantAgent` and `RetrieveUserProxyAgent`. The system message needs to be set to \"You are a helpful assistant.\" for AssistantAgent. The detailed instructions are given in the user message. Later we will use the `RetrieveUserProxyAgent.message_generator` to combine the instructions and a retrieval augmented generation task for an initial prompt to be sent to the LLM assistant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Accepted file formats for `docs_path`:\")\n",
    "print(TEXT_FORMATS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. create an AssistantAgent instance named \"assistant\"\n",
    "assistant = AssistantAgent(\n",
    "    name=\"assistant\",\n",
    "    system_message=\"You are a helpful assistant.\",\n",
    "    llm_config={\n",
    "        \"timeout\": 600,\n",
    "        \"cache_seed\": 42,\n",
    "        \"config_list\": config_list,\n",
    "    },\n",
    ")\n",
    "\n",
    "# 2. create the RetrieveUserProxyAgent instance named \"ragproxyagent\"\n",
    "# Refer to https://docs.ag2.ai/docs/reference/agentchat/contrib/retrieve_user_proxy_agent\n",
    "# and https://docs.ag2.ai/docs/reference/agentchat/contrib/vectordb/chromadb\n",
    "# for more information on the RetrieveUserProxyAgent and ChromaVectorDB\n",
    "ragproxyagent = RetrieveUserProxyAgent(\n",
    "    name=\"ragproxyagent\",\n",
    "    human_input_mode=\"NEVER\",\n",
    "    max_consecutive_auto_reply=3,\n",
    "    retrieve_config={\n",
    "        \"task\": \"code\",\n",
    "        \"docs_path\": [\n",
    "            \"https://raw.githubusercontent.com/microsoft/FLAML/main/website/docs/Examples/Integrate%20-%20Spark.md\",\n",
    "            \"https://raw.githubusercontent.com/microsoft/FLAML/main/website/docs/Research.md\",\n",
    "        ],\n",
    "        \"chunk_token_size\": 2000,\n",
    "        \"model\": config_list[0][\"model\"],\n",
    "        \"vector_db\": \"chroma\",\n",
    "        \"overwrite\": False,  # set to True if you want to overwrite an existing collection\n",
    "        \"get_or_create\": True,  # set to False if don't want to reuse an existing collection\n",
    "    },\n",
    "    code_execution_config=False,  # set to False if you don't want to execute the code\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1\n",
    "\n",
    "[Back to top](#table-of-contents)\n",
    "\n",
    "Use RetrieveChat to help generate sample code and automatically run the code and fix errors if there is any.\n",
    "\n",
    "Problem: Which API should I use if I want to use FLAML for a classification task and I want to train the model in 30 seconds. Use spark to parallel the training. Force cancel jobs if time limit is reached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reset the assistant. Always reset the assistant before starting a new conversation.\n",
    "assistant.reset()\n",
    "\n",
    "# given a problem, we use the ragproxyagent to generate a prompt to be sent to the assistant as the initial message.\n",
    "# the assistant receives the message and generates a response. The response will be sent back to the ragproxyagent for processing.\n",
    "# The conversation continues until the termination condition is met, in RetrieveChat, the termination condition when no human-in-loop is no code block detected.\n",
    "# With human-in-loop, the conversation will continue until the user says \"exit\".\n",
    "code_problem = \"How can I use FLAML to perform a classification task and use spark to do parallel training. Train 30 seconds and force cancel jobs if time limit is reached.\"\n",
    "chat_result = ragproxyagent.initiate_chat(\n",
    "    assistant, message=ragproxyagent.message_generator, problem=code_problem, search_string=\"spark\"\n",
    ")  # search_string is used as an extra filter for the embeddings search, in this case, we only want to search documents that contain \"spark\"."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2\n",
    "\n",
    "[Back to top](#table-of-contents)\n",
    "\n",
    "Use RetrieveChat to answer a question that is not related to code generation.\n",
    "\n",
    "Problem: Who is the author of FLAML?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reset the assistant. Always reset the assistant before starting a new conversation.\n",
    "assistant.reset()\n",
    "\n",
    "qa_problem = \"Who is the author of FLAML?\"\n",
    "chat_result = ragproxyagent.initiate_chat(assistant, message=ragproxyagent.message_generator, problem=qa_problem)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3\n",
    "\n",
    "[Back to top](#table-of-contents)\n",
    "\n",
    "Use RetrieveChat to help generate sample code and ask for human-in-loop feedbacks.\n",
    "\n",
    "Problem: how to build a time series forecasting model for stock price using FLAML?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reset the assistant. Always reset the assistant before starting a new conversation.\n",
    "assistant.reset()\n",
    "\n",
    "# set `human_input_mode` to be `ALWAYS`, so the agent will ask for human input at every step.\n",
    "ragproxyagent.human_input_mode = \"ALWAYS\"\n",
    "code_problem = \"how to build a time series forecasting model for stock price using FLAML?\"\n",
    "chat_result = ragproxyagent.initiate_chat(assistant, message=ragproxyagent.message_generator, problem=code_problem)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 4\n",
    "\n",
    "[Back to top](#table-of-contents)\n",
    "\n",
    "Use RetrieveChat to answer a question and ask for human-in-loop feedbacks.\n",
    "\n",
    "Problem: Is there a function named `tune_automl` in FLAML?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reset the assistant. Always reset the assistant before starting a new conversation.\n",
    "assistant.reset()\n",
    "\n",
    "# set `human_input_mode` to be `ALWAYS`, so the agent will ask for human input at every step.\n",
    "ragproxyagent.human_input_mode = \"ALWAYS\"\n",
    "qa_problem = \"Is there a function named `tune_automl` in FLAML?\"\n",
    "chat_result = ragproxyagent.initiate_chat(\n",
    "    assistant, message=ragproxyagent.message_generator, problem=qa_problem\n",
    ")  # type \"exit\" to exit the conversation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 5\n",
    "\n",
    "[Back to top](#table-of-contents)\n",
    "\n",
    "Use RetrieveChat to answer questions for [NaturalQuestion](https://ai.google.com/research/NaturalQuestions) dataset.\n",
    "\n",
    "First, we will create a new document collection which includes all the contextual corpus. Then, we will choose some questions and utilize RetrieveChat to answer them. For this particular example, we will be using the `gpt-3.5-turbo` model, and we will demonstrate RetrieveChat's feature of automatically updating context in case the documents retrieved do not contain sufficient information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config_list[0][\"model\"] = \"gpt-35-turbo\"  # change model to gpt-35-turbo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_file = \"https://huggingface.co/datasets/thinkall/NaturalQuestionsQA/resolve/main/corpus.txt\"\n",
    "\n",
    "# Create a new collection for NaturalQuestions dataset\n",
    "# `task` indicates the kind of task we're working on. In this example, it's a `qa` task.\n",
    "ragproxyagent = RetrieveUserProxyAgent(\n",
    "    name=\"ragproxyagent\",\n",
    "    human_input_mode=\"NEVER\",\n",
    "    max_consecutive_auto_reply=10,\n",
    "    retrieve_config={\n",
    "        \"task\": \"qa\",\n",
    "        \"docs_path\": corpus_file,\n",
    "        \"chunk_token_size\": 2000,\n",
    "        \"model\": config_list[0][\"model\"],\n",
    "        \"client\": chromadb.PersistentClient(path=\"/tmp/chromadb\"),\n",
    "        \"collection_name\": \"natural-questions\",\n",
    "        \"chunk_mode\": \"one_line\",\n",
    "        \"embedding_model\": \"all-MiniLM-L6-v2\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# queries_file = \"https://huggingface.co/datasets/thinkall/NaturalQuestionsQA/resolve/main/queries.jsonl\"\n",
    "queries = \"\"\"{\"_id\": \"ce2342e1feb4e119cb273c05356b33309d38fa132a1cbeac2368a337e38419b8\", \"text\": \"what is non controlling interest on balance sheet\", \"metadata\": {\"answer\": [\"the portion of a subsidiary corporation 's stock that is not owned by the parent corporation\"]}}\n",
    "{\"_id\": \"3a10ff0e520530c0aa33b2c7e8d989d78a8cd5d699201fc4b13d3845010994ee\", \"text\": \"how many episodes are in chicago fire season 4\", \"metadata\": {\"answer\": [\"23\"]}}\n",
    "{\"_id\": \"fcdb6b11969d5d3b900806f52e3d435e615c333405a1ff8247183e8db6246040\", \"text\": \"what are bulls used for on a farm\", \"metadata\": {\"answer\": [\"breeding\", \"as work oxen\", \"slaughtered for meat\"]}}\n",
    "{\"_id\": \"26c3b53ec44533bbdeeccffa32e094cfea0cc2a78c9f6a6c7a008ada1ad0792e\", \"text\": \"has been honoured with the wisden leading cricketer in the world award for 2016\", \"metadata\": {\"answer\": [\"Virat Kohli\"]}}\n",
    "{\"_id\": \"0868d0964c719a52cbcfb116971b0152123dad908ac4e0a01bc138f16a907ab3\", \"text\": \"who carried the usa flag in opening ceremony\", \"metadata\": {\"answer\": [\"Erin Hamlin\"]}}\n",
    "\"\"\"\n",
    "queries = [json.loads(line) for line in queries.split(\"\\n\") if line]\n",
    "questions = [q[\"text\"] for q in queries]\n",
    "answers = [q[\"metadata\"][\"answer\"] for q in queries]\n",
    "print(questions)\n",
    "print(answers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(questions)):\n",
    "    print(f\"\\n\\n>>>>>>>>>>>>  Below are outputs of Case {i + 1}  <<<<<<<<<<<<\\n\\n\")\n",
    "\n",
    "    # reset the assistant. Always reset the assistant before starting a new conversation.\n",
    "    assistant.reset()\n",
    "\n",
    "    qa_problem = questions[i]\n",
    "    chat_result = ragproxyagent.initiate_chat(\n",
    "        assistant, message=ragproxyagent.message_generator, problem=qa_problem, n_results=30\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, questions were directly selected from the dataset. RetrieveChat was able to answer the questions correctly in the first attempt as the retrieved context contained the necessary information in the first two cases. However, in the last three cases, the context with the highest similarity to the question embedding did not contain the required information to answer the question. As a result, the LLM model responded with `UPDATE CONTEXT`. With the unique and innovative ability to update context in RetrieveChat, the agent automatically updated the context and sent it to the LLM model again. After several rounds of this process, the agent was able to generate the correct answer to the questions."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 6\n",
    "\n",
    "[Back to top](#table-of-contents)\n",
    "\n",
    "Use RetrieveChat to answer multi-hop questions for [2WikiMultihopQA](https://github.com/Alab-NII/2wikimultihop) dataset with customized prompt and few-shot learning.\n",
    "\n",
    "First, we will create a new document collection which includes all the contextual corpus. Then, we will choose some questions and utilize RetrieveChat to answer them. For this particular example, we will be using the `gpt-3.5-turbo` model, and we will demonstrate RetrieveChat's feature of automatically updating context in case the documents retrieved do not contain sufficient information. Moreover, we'll demonstrate how to use customized prompt and few-shot learning to address tasks that are not pre-defined in RetrieveChat."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROMPT_MULTIHOP = \"\"\"You're a retrieve augmented chatbot. You answer user's questions based on your own knowledge and the context provided by the user. You must think step-by-step.\n",
    "First, please learn the following examples of context and question pairs and their corresponding answers.\n",
    "\n",
    "Context:\n",
    "Kurram Garhi: Kurram Garhi is a small village located near the city of Bannu, which is the part of Khyber Pakhtunkhwa province of Pakistan. Its population is approximately 35000.\n",
    "Trojkrsti: Trojkrsti is a village in Municipality of Prilep, Republic of Macedonia.\n",
    "Q: Are both Kurram Garhi and Trojkrsti located in the same country?\n",
    "A: Kurram Garhi is located in the country of Pakistan. Trojkrsti is located in the country of Republic of Macedonia. Thus, they are not in the same country. So the answer is: no.\n",
    "\n",
    "\n",
    "Context:\n",
    "Early Side of Later: Early Side of Later is the third studio album by English singer- songwriter Matt Goss. It was released on 21 June 2004 by Concept Music and reached No. 78 on the UK Albums Chart.\n",
    "What's Inside: What's Inside is the fourteenth studio album by British singer- songwriter Joan Armatrading.\n",
    "Q: Which album was released earlier, What'S Inside or Cassandra'S Dream (Album)?\n",
    "A: What's Inside was released in the year 1995. Cassandra's Dream (album) was released in the year 2008. Thus, of the two, the album to release earlier is What's Inside. So the answer is: What's Inside.\n",
    "\n",
    "\n",
    "Context:\n",
    "Maria Alexandrovna (Marie of Hesse): Maria Alexandrovna , born Princess Marie of Hesse and by Rhine (8 August 1824 – 3 June 1880) was Empress of Russia as the first wife of Emperor Alexander II.\n",
    "Grand Duke Alexei Alexandrovich of Russia: Grand Duke Alexei Alexandrovich of Russia,(Russian: Алексей Александрович; 14 January 1850 (2 January O.S.) in St. Petersburg – 14 November 1908 in Paris) was the fifth child and the fourth son of Alexander II of Russia and his first wife Maria Alexandrovna (Marie of Hesse).\n",
    "Q: What is the cause of death of Grand Duke Alexei Alexandrovich Of Russia's mother?\n",
    "A: The mother of Grand Duke Alexei Alexandrovich of Russia is Maria Alexandrovna. Maria Alexandrovna died from tuberculosis. So the answer is: tuberculosis.\n",
    "\n",
    "\n",
    "Context:\n",
    "Laughter in Hell: Laughter in Hell is a 1933 American Pre-Code drama film directed by Edward L. Cahn and starring Pat O'Brien. The film's title was typical of the sensationalistic titles of many Pre-Code films.\n",
    "Edward L. Cahn: Edward L. Cahn (February 12, 1899 – August 25, 1963) was an American film director.\n",
    "Q: When did the director of film Laughter In Hell die?\n",
    "A: The film Laughter In Hell was directed by Edward L. Cahn. Edward L. Cahn died on August 25, 1963. So the answer is: August 25, 1963.\n",
    "\n",
    "Second, please complete the answer by thinking step-by-step.\n",
    "\n",
    "Context:\n",
    "{input_context}\n",
    "Q: {input_question}\n",
    "A:\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the RetrieveUserProxyAgent instance named \"ragproxyagent\"\n",
    "corpus_file = \"https://huggingface.co/datasets/thinkall/2WikiMultihopQA/resolve/main/corpus.txt\"\n",
    "\n",
    "# Create a new collection for NaturalQuestions dataset\n",
    "ragproxyagent = RetrieveUserProxyAgent(\n",
    "    name=\"ragproxyagent\",\n",
    "    human_input_mode=\"NEVER\",\n",
    "    max_consecutive_auto_reply=3,\n",
    "    retrieve_config={\n",
    "        \"task\": \"qa\",\n",
    "        \"docs_path\": corpus_file,\n",
    "        \"chunk_token_size\": 2000,\n",
    "        \"model\": config_list[0][\"model\"],\n",
    "        \"client\": chromadb.PersistentClient(path=\"/tmp/chromadb\"),\n",
    "        \"collection_name\": \"2wikimultihopqa\",\n",
    "        \"chunk_mode\": \"one_line\",\n",
    "        \"embedding_model\": \"all-MiniLM-L6-v2\",\n",
    "        \"customized_prompt\": PROMPT_MULTIHOP,\n",
    "        \"customized_answer_prefix\": \"the answer is\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# queries_file = \"https://huggingface.co/datasets/thinkall/2WikiMultihopQA/resolve/main/queries.jsonl\"\n",
    "queries = \"\"\"{\"_id\": \"61a46987092f11ebbdaeac1f6bf848b6\", \"text\": \"Which film came out first, Blind Shaft or The Mask Of Fu Manchu?\", \"metadata\": {\"answer\": [\"The Mask Of Fu Manchu\"]}}\n",
    "{\"_id\": \"a7b9672009c311ebbdb0ac1f6bf848b6\", \"text\": \"Are North Marion High School (Oregon) and Seoul High School both located in the same country?\", \"metadata\": {\"answer\": [\"no\"]}}\n",
    "\"\"\"\n",
    "queries = [json.loads(line) for line in queries.split(\"\\n\") if line]\n",
    "questions = [q[\"text\"] for q in queries]\n",
    "answers = [q[\"metadata\"][\"answer\"] for q in queries]\n",
    "print(questions)\n",
    "print(answers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(questions)):\n",
    "    print(f\"\\n\\n>>>>>>>>>>>>  Below are outputs of Case {i + 1}  <<<<<<<<<<<<\\n\\n\")\n",
    "\n",
    "    # reset the assistant. Always reset the assistant before starting a new conversation.\n",
    "    assistant.reset()\n",
    "\n",
    "    qa_problem = questions[i]\n",
    "    chat_result = ragproxyagent.initiate_chat(\n",
    "        assistant, message=ragproxyagent.message_generator, problem=qa_problem, n_results=10\n",
    "    )"
   ]
  }
 ],
 "metadata": {
  "front_matter": {
   "description": "Explore the use of AG2's RetrieveChat for tasks like code generation from docstrings, answering complex questions with human feedback, and exploiting features like Update Context, custom prompts, and few-shot learning.",
   "tags": [
    "RAG"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  },
  "skip_test": "Requires interactive usage"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}