nemo/data-flywheel/tool-calling/1_data_preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d9a65fb3-f1a5-4de8-b54d-67346fa35c8a",
   "metadata": {},
   "source": [
    " # Part I: Preparing Datasets for Fine-tuning and Evaluation\n",
    "\n",
    "This notebook covers the following:\n",
    "\n",
    "1. [Download xLAM Dataset](#step-1)\n",
    "2. [Prepare Data for Customization](#step-2)\n",
    "3. [Prepare Data for Evaluation](#step-3)\n",
    "   \n",
    "This notebook showcases transforming a dataset for finetuning and evaluating an LLM for tool calling with NeMo Microservices."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1905e4a5",
   "metadata": {},
   "source": [
    "The following code cell imports necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6193d10c-583c-4a78-8459-66e7ec2bbab1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import random\n",
    "from pprint import pprint\n",
    "from typing import Any, Dict, List, Union\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99fc25e7",
   "metadata": {},
   "source": [
    "The following code cell sets a random seed for reproducibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3fc7314b-0f86-4ed4-bdc1-3d2598092cf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "SEED = 1234\n",
    "\n",
    "# Limits to at most N tool properties\n",
    "LIMIT_TOOL_PROPERTIES = 8\n",
    "\n",
    "torch.manual_seed(SEED)\n",
    "torch.cuda.manual_seed_all(SEED)\n",
    "np.random.seed(SEED)\n",
    "random.seed(SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfab9962",
   "metadata": {},
   "source": [
    "The following code cell defines the data root directory and creates necessary directories for storing processed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b6434b0c-9ad0-403e-9cf4-d9ab22306769",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Processed data will be stored here\n",
    "DATA_ROOT = os.path.join(os.getcwd(), \"data\")\n",
    "CUSTOMIZATION_DATA_ROOT = os.path.join(DATA_ROOT, \"customization\")\n",
    "VALIDATION_DATA_ROOT = os.path.join(DATA_ROOT, \"validation\")\n",
    "EVALUATION_DATA_ROOT = os.path.join(DATA_ROOT, \"evaluation\")\n",
    "\n",
    "os.makedirs(DATA_ROOT, exist_ok=True)\n",
    "os.makedirs(CUSTOMIZATION_DATA_ROOT, exist_ok=True)\n",
    "os.makedirs(VALIDATION_DATA_ROOT, exist_ok=True)\n",
    "os.makedirs(EVALUATION_DATA_ROOT, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2556c45-8c14-4ea2-a2c2-7937c9902b2d",
   "metadata": {},
   "source": [
    "---\n",
    "<a id=\"step-1\"></a>\n",
    "## Step 1: Download xLAM Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1db1f5a8-c741-4657-b6c9-56afd9dccc94",
   "metadata": {},
   "source": [
    "This step loads the xLAM dataset from Hugging Face.\n",
    "\n",
    "Ensure that you have followed the prerequisites mentioned in the associated README, obtained a Hugging Face access token, and configured it in [config.py](./config.py). In addition to getting an access token, you need to apply for access to the xLAM dataset on its [page](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k), which will be approved instantly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4f729c88-e633-4914-b3b3-09311ad79a76",
   "metadata": {},
   "outputs": [],
   "source": [
    "from config import HF_TOKEN\n",
    "\n",
    "os.environ[\"HF_TOKEN\"] = HF_TOKEN\n",
    "os.environ[\"HF_ENDPOINT\"] = \"https://huggingface.co\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "04d19368-0c33-4595-bea8-24ef645eaaa3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'answers': '[{\"name\": \"live_giveaways_by_type\", \"arguments\": {\"type\": '\n",
      "            '\"beta\"}}, {\"name\": \"live_giveaways_by_type\", \"arguments\": '\n",
      "            '{\"type\": \"game\"}}]',\n",
      " 'id': 0,\n",
      " 'query': 'Where can I find live giveaways for beta access and games?',\n",
      " 'tools': '[{\"name\": \"live_giveaways_by_type\", \"description\": \"Retrieve live '\n",
      "          'giveaways from the GamerPower API based on the specified type.\", '\n",
      "          '\"parameters\": {\"type\": {\"description\": \"The type of giveaways to '\n",
      "          'retrieve (e.g., game, loot, beta).\", \"type\": \"str\", \"default\": '\n",
      "          '\"game\"}}}]'}\n"
     ]
    }
   ],
   "source": [
    "# Download from Hugging Face\n",
    "dataset = load_dataset(\"Salesforce/xlam-function-calling-60k\")\n",
    "\n",
    "# Inspect a sample\n",
    "example = dataset['train'][0]\n",
    "pprint(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6caa142c-b051-42ad-ad24-aabdd05ea391",
   "metadata": {},
   "source": [
    "For more details on the structure of this data, refer to the [data structure of the xLAM dataset](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k#structure) in the Hugging Face documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00f41f50-2142-4de5-80ee-22e10a868559",
   "metadata": {},
   "source": [
    "---\n",
    "<a id=\"step-2\"></a>\n",
    "## Step 2: Prepare Data for Customization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc98b800-5b23-48a4-983b-dc4797e7d496",
   "metadata": {},
   "source": [
    "For Customization, the NeMo Microservices platform leverages the OpenAI data format, comprised of `messages` and `tools`:\n",
    "\n",
    "* `messages` include the `user` query, as well as the ground truth `assistant` response to the query. This response contains the function name(s) and associated argument(s) in a \"tool_calls\" dict.\n",
    "* `tools` include a list of functions and parameters available to the LLM to choose from, as well as their descriptions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24f81d2a-ece1-4be2-abe3-e6404e473665",
   "metadata": {},
   "source": [
    "The following is an example of the data format:\n",
    "```\n",
    "{\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Where can I find live giveaways for beta access and games?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"tool_calls\": [\n",
    "                {\n",
    "                    \"id\": \"call_beta\",\n",
    "                    \"type\": \"function\",\n",
    "                    \"function\": {\n",
    "                        \"name\": \"live_giveaways_by_type\",\n",
    "                        \"arguments\": {\"type\": \"beta\"}\n",
    "                    }\n",
    "                },\n",
    "                {\n",
    "                    \"id\": \"call_game\",\n",
    "                    \"type\": \"function\",\n",
    "                    \"function\": {\n",
    "                        \"name\": \"live_giveaways_by_type\",\n",
    "                        \"arguments\": {\"type\": \"game\"}\n",
    "                    }\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    \"tools\": [\n",
    "        {\n",
    "            \"type\": \"function\",\n",
    "            \"function\": {\n",
    "                \"name\": \"live_giveaways_by_type\",\n",
    "                \"description\": \"Retrieve live giveaways from the GamerPower API based on the specified type.\",\n",
    "                \"parameters\": {\n",
    "                    \"type\": \"object\",\n",
    "                    \"properties\": {\n",
    "                        \"type\": {\n",
    "                            \"type\": \"string\",\n",
    "                            \"description\": \"The type of giveaways to retrieve (e.g., game, loot, beta).\",\n",
    "                            \"default\": \"game\"\n",
    "                        }\n",
    "                    },\n",
    "                    \"required\": []\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6fb67d5-f95b-47f4-99bc-7a2ad4117f7a",
   "metadata": {},
   "source": [
    "The following helper functions convert a single xLAM JSON data point into OpenAI format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "47fc5c3e-0f4b-4423-845e-5c32865283e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize_type(param_type: str) -> str:\n",
    "    \"\"\"\n",
    "    Normalize Python type hints and parameter definitions to OpenAI function spec types.\n",
    "\n",
    "    Args:\n",
    "        param_type: Type string that could include default values or complex types\n",
    "\n",
    "    Returns:\n",
    "        Normalized type string according to OpenAI function spec\n",
    "    \"\"\"\n",
    "    # Remove whitespace\n",
    "    param_type = param_type.strip()\n",
    "\n",
    "    # Handle types with default values (e.g. \"str, default='London'\")\n",
    "    if \",\" in param_type and \"default\" in param_type:\n",
    "        param_type = param_type.split(\",\")[0].strip()\n",
    "\n",
    "    # Handle types with just default values (e.g. \"default='London'\")\n",
    "    if param_type.startswith(\"default=\"):\n",
    "        return \"string\"  # Default to string if only default value is given\n",
    "\n",
    "    # Remove \", optional\" suffix if present\n",
    "    param_type = param_type.replace(\", optional\", \"\").strip()\n",
    "\n",
    "    # Handle complex types\n",
    "    if param_type.startswith(\"Callable\"):\n",
    "        return \"string\"  # Represent callable as string in JSON schema\n",
    "    if param_type.startswith(\"Tuple\"):\n",
    "        return \"array\"  # Represent tuple as array in JSON schema\n",
    "    if param_type.startswith(\"List[\"):\n",
    "        return \"array\"\n",
    "    if param_type.startswith(\"Set\") or param_type == \"set\":\n",
    "        return \"array\"  # Represent set as array in JSON schema\n",
    "\n",
    "    # Map common type variations to OpenAI spec types\n",
    "    type_mapping: Dict[str, str] = {\n",
    "        \"str\": \"string\",\n",
    "        \"int\": \"integer\",\n",
    "        \"float\": \"number\",\n",
    "        \"bool\": \"boolean\",\n",
    "        \"list\": \"array\",\n",
    "        \"dict\": \"object\",\n",
    "        \"List\": \"array\",\n",
    "        \"Dict\": \"object\",\n",
    "        \"set\": \"array\",\n",
    "        \"Set\": \"array\"\n",
    "    }\n",
    "\n",
    "    if param_type in type_mapping:\n",
    "        return type_mapping[param_type]\n",
    "    else:\n",
    "        print(f\"Unknown type: {param_type}\")\n",
    "        return \"string\"  # Default to string for unknown types\n",
    "\n",
    "\n",
    "def convert_tools_to_openai_spec(tools: Union[str, List[Dict[str, Any]]]) -> List[Dict[str, Any]]:\n",
    "    # If tools is a string, try to parse it as JSON\n",
    "    if isinstance(tools, str):\n",
    "        try:\n",
    "            tools = json.loads(tools)\n",
    "        except json.JSONDecodeError as e:\n",
    "            print(f\"Failed to parse tools string as JSON: {e}\")\n",
    "            return []\n",
    "\n",
    "    # Ensure tools is a list\n",
    "    if not isinstance(tools, list):\n",
    "        print(f\"Expected tools to be a list, but got {type(tools)}\")\n",
    "        return []\n",
    "\n",
    "    openai_tools: List[Dict[str, Any]] = []\n",
    "    for tool in tools:\n",
    "        # Check if tool is a dictionary\n",
    "        if not isinstance(tool, dict):\n",
    "            print(f\"Expected tool to be a dictionary, but got {type(tool)}\")\n",
    "            continue\n",
    "\n",
    "        # Check if 'parameters' is a dictionary\n",
    "        if not isinstance(tool.get(\"parameters\"), dict):\n",
    "            print(f\"Expected 'parameters' to be a dictionary, but got {type(tool.get('parameters'))} for tool: {tool}\")\n",
    "            continue\n",
    "\n",
    "    \n",
    "\n",
    "        normalized_parameters: Dict[str, Dict[str, Any]] = {}\n",
    "        for param_name, param_info in tool[\"parameters\"].items():\n",
    "            if not isinstance(param_info, dict):\n",
    "                print(\n",
    "                    f\"Expected parameter info to be a dictionary, but got {type(param_info)} for parameter: {param_name}\"\n",
    "                )\n",
    "                continue\n",
    "\n",
    "            # Create parameter info without default first\n",
    "            param_dict = {\n",
    "                \"description\": param_info.get(\"description\", \"\"),\n",
    "                \"type\": normalize_type(param_info.get(\"type\", \"\")),\n",
    "            }\n",
    "\n",
    "            # Only add default if it exists, is not None, and is not an empty string\n",
    "            default_value = param_info.get(\"default\")\n",
    "            if default_value is not None and default_value != \"\":\n",
    "                param_dict[\"default\"] = default_value\n",
    "\n",
    "            normalized_parameters[param_name] = param_dict\n",
    "\n",
    "        openai_tool = {\n",
    "            \"type\": \"function\",\n",
    "            \"function\": {\n",
    "                \"name\": tool[\"name\"],\n",
    "                \"description\": tool[\"description\"],\n",
    "                \"parameters\": {\"type\": \"object\", \"properties\": normalized_parameters},\n",
    "            },\n",
    "        }\n",
    "        openai_tools.append(openai_tool)\n",
    "    return openai_tools\n",
    "\n",
    "\n",
    "def save_jsonl(filename, data):\n",
    "    \"\"\"Write a list of json objects to a .jsonl file\"\"\"\n",
    "    with open(filename, \"w\") as f:\n",
    "        for entry in data:\n",
    "            f.write(json.dumps(entry) + \"\\n\")\n",
    "\n",
    "\n",
    "def convert_tool_calls(xlam_tools):\n",
    "    \"\"\"Convert XLAM tool format to OpenAI's tool schema.\"\"\"\n",
    "    tools = []\n",
    "    for tool in json.loads(xlam_tools):\n",
    "        tools.append({\"type\": \"function\", \"function\": {\"name\": tool[\"name\"], \"arguments\": tool.get(\"arguments\", {})}})\n",
    "    return tools\n",
    "\n",
    "\n",
    "def convert_example(example, dataset_type='single'):\n",
    "    \"\"\"Convert an XLAM dataset example to OpenAI format.\"\"\"\n",
    "    obj = {\"messages\": []}\n",
    "\n",
    "    # User message\n",
    "    obj[\"messages\"].append({\"role\": \"user\", \"content\": example[\"query\"]})\n",
    "\n",
    "    # Tools\n",
    "    if example.get(\"tools\"):\n",
    "        obj[\"tools\"] = convert_tools_to_openai_spec(example[\"tools\"])\n",
    "\n",
    "    # Assistant message\n",
    "    assistant_message = {\"role\": \"assistant\", \"content\": \"\"}\n",
    "    if example.get(\"answers\"):\n",
    "        tool_calls = convert_tool_calls(example[\"answers\"])\n",
    "        \n",
    "        if dataset_type == \"single\":\n",
    "            # Only include examples with a single tool call\n",
    "            if len(tool_calls) == 1:\n",
    "                assistant_message[\"tool_calls\"] = tool_calls\n",
    "            else:\n",
    "                return None\n",
    "        else:\n",
    "            # For other dataset types, include all tool calls\n",
    "            assistant_message[\"tool_calls\"] = tool_calls\n",
    "                \n",
    "    obj[\"messages\"].append(assistant_message)\n",
    "\n",
    "    return obj"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b78b514a",
   "metadata": {},
   "source": [
    "The following code cell converts the example data to the OpenAI format required by NeMo Customizer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3366c584-bf5f-47f2-b0ea-aa421c3d083b",
   "metadata": {},
   "outputs": [],
   "source": [
    "convert_example(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62515958-e98d-475b-9fda-18890d353813",
   "metadata": {},
   "source": [
    "**NOTE**: The `convert_example` function by default only retains data points that have exactly one `tool_call` in the output.\n",
    "The `llama-3.2-1b-instruct` model does not support parallel tool calls.\n",
    "For more information, refer to the [supported models](https://docs.nvidia.com/nim/large-language-models/latest/function-calling.html#supported-models) in the NeMo documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab76d74a-5df2-443f-aff6-0feff0e06285",
   "metadata": {},
   "source": [
    "### Process Entire Dataset\n",
    "Convert each example by looping through the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "706c81c0-3900-4157-9366-afa10a142c5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_examples = []\n",
    "with open(os.path.join(DATA_ROOT, \"xlam_openai_format.jsonl\"), \"w\") as f:\n",
    "    for example in dataset[\"train\"]:\n",
    "        converted = convert_example(example)\n",
    "        if converted is not None:\n",
    "            all_examples.append(converted)\n",
    "            f.write(json.dumps(converted) + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37b7d931-6c33-4d8f-8a57-e188aacc7616",
   "metadata": {},
   "source": [
    "### Split Dataset\n",
    "This step splits the dataset into a train, validation, and test set.\n",
    "For demonstration, we use a smaller subset of all the examples.\n",
    "You may choose to modify `NUM_EXAMPLES` to leverage a larger subset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f8be515e-4255-4682-8499-a30c71f129e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure to change the size of dataset to use\n",
    "NUM_EXAMPLES = 5000\n",
    "\n",
    "assert NUM_EXAMPLES <= len(all_examples), f\"{NUM_EXAMPLES} exceeds the total number of available ({len(all_examples)}) data points\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5cdc6072-0b85-45be-b3b1-fe9a1c26d641",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Randomly choose a subset\n",
    "sampled_examples = random.sample(all_examples, NUM_EXAMPLES)\n",
    "\n",
    "# Split into 70% training, 15% validation, 15% testing\n",
    "train_size = int(0.7 * len(sampled_examples))\n",
    "val_size = int(0.15 * len(sampled_examples))\n",
    "\n",
    "train_data = sampled_examples[:train_size]\n",
    "val_data = sampled_examples[train_size : train_size + val_size]\n",
    "test_data = sampled_examples[train_size + val_size :]\n",
    "\n",
    "# Save the training and validation splits. We will use test split in the next section\n",
    "save_jsonl(os.path.join(CUSTOMIZATION_DATA_ROOT, \"training.jsonl\"), train_data)\n",
    "save_jsonl(os.path.join(VALIDATION_DATA_ROOT,\"validation.jsonl\"), val_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74afdfed-4af3-4deb-90df-081f00300714",
   "metadata": {},
   "source": [
    "---\n",
    "<a id=\"step-3\"></a>\n",
    "## Step 3: Prepare Data for Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a69f2094-2afb-458d-8c9d-526f1270c017",
   "metadata": {},
   "source": [
    "For evaluation, the NeMo Microservices platform uses a format with a minor modification to the OpenAI format. This requires `tools_calls` to be brought out of `messages` to create a distinct parallel field.\n",
    "\n",
    "* `messages` includes the `user` query\n",
    "* `tools` includes a list of functions and parameters available to the LLM to choose from, as well as their descriptions.\n",
    "* `tool_calls` is the ground truth response to the user query. This response contains the function name(s) and associated argument(s) in a \"tool_calls\" dict.\n",
    "\n",
    "Here is an example -"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1afc1448-b09f-426d-81eb-52122a961d80",
   "metadata": {},
   "source": [
    "```\n",
    "{\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Where can I find live giveaways for beta access?\"\n",
    "        },\n",
    "    ],\n",
    "    \"tools\": [\n",
    "        {\n",
    "            \"type\": \"function\",\n",
    "            \"function\": {\n",
    "                \"name\": \"live_giveaways_by_type\",\n",
    "                \"description\": \"Retrieve live giveaways from the GamerPower API based on the specified type.\",\n",
    "                \"parameters\": {\n",
    "                    \"type\": \"object\",\n",
    "                    \"properties\": {\n",
    "                        \"type\": {\n",
    "                            \"type\": \"string\",\n",
    "                            \"description\": \"The type of giveaways to retrieve (e.g., game, loot, beta).\",\n",
    "                            \"default\": \"game\"\n",
    "                        }\n",
    "                    },\n",
    "                    \"required\": []\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    ],\n",
    "    \"tool_calls\": [\n",
    "        {\n",
    "            \"id\": \"call_beta\",\n",
    "            \"type\": \"function\",\n",
    "            \"function\": {\n",
    "                \"name\": \"live_giveaways_by_type\",\n",
    "                \"arguments\": {\"type\": \"beta\"}\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6424507e-4097-49bb-b750-96a66f9abd77",
   "metadata": {},
   "source": [
    "The following steps transform the test dataset into a format compatible with the NeMo Evaluator microservice.\n",
    "This dataset is for measuring accuracy metrics before and after customization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d6487d80-36a0-4314-8c21-93744043ce05",
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_example_eval(entry):\n",
    "    \"\"\"Convert a single entry in the dataset to the evaluator format\"\"\"\n",
    "\n",
    "    # Note: This is a WAR for a known bug with tool calling in NIM\n",
    "    for tool in entry[\"tools\"]:\n",
    "        if len(tool[\"function\"][\"parameters\"][\"properties\"]) > LIMIT_TOOL_PROPERTIES:\n",
    "            return None\n",
    "    \n",
    "    new_entry = {\n",
    "        \"messages\": [],\n",
    "        \"tools\": entry[\"tools\"],\n",
    "        \"tool_calls\": []\n",
    "    }\n",
    "    \n",
    "    for msg in entry[\"messages\"]:\n",
    "        if msg[\"role\"] == \"assistant\" and \"tool_calls\" in msg:\n",
    "            new_entry[\"tool_calls\"] = msg[\"tool_calls\"]\n",
    "        else:\n",
    "            new_entry[\"messages\"].append(msg)\n",
    "    \n",
    "    return new_entry\n",
    "\n",
    "def convert_dataset_eval(data):\n",
    "    \"\"\"Convert the entire dataset for evaluation by restructuring the data format.\"\"\"\n",
    "    return [result for entry in data if (result := convert_example_eval(entry)) is not None]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0873c030-56b8-4366-9984-24c310327d26",
   "metadata": {},
   "source": [
    "`NOTE:` We have implemented a workaround for a known bug where tool calls freeze the NIM if a tool description includes a function with a larger number of parameters. As such, we have limited the dataset to use examples with available tools having at most 8 parameters. This will be resolved in the next NIM release."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6028d6c5-7a67-4b63-a1ef-e93e0177893a",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data_eval = convert_dataset_eval(test_data)\n",
    "save_jsonl(os.path.join(EVALUATION_DATA_ROOT, \"xlam-test-single.jsonl\"), test_data_eval)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}