labs/model-routing/model-routing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# APIM ❤️ OpenAI\n",
    "\n",
    "## Model routing lab\n",
    "![flow](../../images/model-routing.gif)\n",
    "\n",
    "Playground to try routing to a backend based on Azure OpenAI model and version.\n",
    "\n",
    "### TOC\n",
    "- [0️⃣ Initialize notebook variables](#0)\n",
    "- [1️⃣ Create the Azure Resource Group](#1)\n",
    "- [2️⃣ Create deployment using 🦾 Bicep](#2)\n",
    "- [3️⃣ Get the deployment outputs](#3)\n",
    "- [🧪 Test the API using a direct HTTP call](#requests)\n",
    "- [🧪 Test the API using the Azure OpenAI Python SDK](#sdk)\n",
    "- [🗑️ Clean up resources](#clean)\n",
    "\n",
    "### Backlog\n",
    "- Improve the notebook\n",
    "\n",
    "### Prerequisites\n",
    "- [Python 3.12 or later version](https://www.python.org/) installed\n",
    "- [Pandas Library](https://pandas.pydata.org/) installed\n",
    "- [VS Code](https://code.visualstudio.com/) installed with the [Jupyter notebook extension](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter) enabled\n",
    "- [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) installed\n",
    "- [An Azure Subscription](https://azure.microsoft.com/free/) with Contributor permissions\n",
    "- [Access granted to Azure OpenAI](https://aka.ms/oai/access)\n",
    "- [Sign in to Azure with Azure CLI](https://learn.microsoft.com/cli/azure/authenticate-azure-cli-interactively)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='0'></a>\n",
    "### 0️⃣ Initialize notebook variables\n",
    "\n",
    "- Resources will be suffixed by a unique string based on your subscription id.\n",
    "- Adjust the location parameters according your preferences and on the [product availability by Azure region.](https://azure.microsoft.com/explore/global-infrastructure/products-by-region/?cdn=disable&products=cognitive-services,api-management) \n",
    "- Adjust the OpenAI model and version according the [availability by region.](https://learn.microsoft.com/azure/ai-services/openai/concepts/models) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import datetime\n",
    "import requests\n",
    "\n",
    "deployment_name = os.path.basename(os.path.dirname(globals()['__vsc_ipynb_file__']))\n",
    "index = ''  # Set a matching value here and in clean-up-resources.ipynb if you want separate instances of the lab. This is helpful when tearing down resources and mitigating API Management's soft delete.\n",
    "resource_group_name = f\"lab-{deployment_name}{index}\" # change the name to match your naming style\n",
    "resource_group_location = \"westeurope\"\n",
    "\n",
    "# Define three OpenAI model and version combinations\n",
    "# https://learn.microsoft.com/azure/ai-services/openai/concepts/models#gpt-35\n",
    "# Please note that availability of models and versions is variable and that you may need to adjust the model and version names to match the available models and versions in your Azure subscription.\n",
    "# For this lab, we are using the following combinations based on PayGo availability on June 21, 2024:\n",
    "#\n",
    "#   1) GPT-3.5 Turbo 1106: France Central, Sweden Central\n",
    "#   2) GPT-3.5 Turbo 0125: North Central US, South Central US\n",
    "#   3) GPT-4o 2024-05-13: East US, West US\n",
    "\n",
    "openai_model_gpt_35_turbo_0125 = { \"name\": \"gpt-35-turbo\", \"version\": \"0125\" }\n",
    "openai_model_gpt_35_turbo_1106 = { \"name\": \"gpt-35-turbo\", \"version\": \"1106\" }\n",
    "openai_model_gpt_4o_20240513 = { \"name\": \"gpt-4o\", \"version\": \"2024-05-13\" }\n",
    "\n",
    "openai_model_1_name = \"gpt-35-turbo\"\n",
    "openai_model_1_version = \"1106\"\n",
    "openai_deployment_1_name = f\"{openai_model_1_name}-{openai_model_1_version}\"\n",
    "openai_resources_1 = [ {\"name\": \"oai-francecentral\", \"location\": \"francecentral\"}, {\"name\": \"oai-swedencentral\", \"location\": \"swedencentral\"} ]\n",
    "\n",
    "openai_model_2_name = \"gpt-35-turbo\"\n",
    "openai_model_2_version = \"0125\"\n",
    "openai_deployment_2_name = f\"{openai_model_2_name}-{openai_model_2_version}\"\n",
    "openai_resources_2 = [ {\"name\": \"oai-northcentralus\", \"location\": \"northcentralus\"}, {\"name\": \"oai-southcentralus\", \"location\": \"southcentralus\"} ]\n",
    "\n",
    "openai_model_3_name = \"gpt-4o\"\n",
    "openai_model_3_version = \"2024-05-13\"\n",
    "openai_deployment_3_name = f\"{openai_model_3_name}-{openai_model_3_version}\"\n",
    "openai_resources_3 = [ {\"name\": \"oai-eastus\", \"location\": \"eastus\"}, {\"name\": \"oai-westus\", \"location\": \"westus\"} ]\n",
    "\n",
    "# Define Azure OpenAI resources\n",
    "openai_resources_sku = \"S0\"\n",
    "openai_api_version = \"2024-02-01\"\n",
    "openai_specification_url='https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/stable/' + openai_api_version + '/inference.json'\n",
    "\n",
    "# Define Azure API Management\n",
    "apim_resource_name = \"apim\"\n",
    "apim_resource_location = \"westeurope\"\n",
    "apim_resource_sku = \"Basicv2\"\n",
    "\n",
    "# Define the Azure OpenAI backends and backend pools per Azure OpenAI model and version\n",
    "openai_backend_pool_1 = f\"oai-backend-pool-{openai_deployment_1_name}\"\n",
    "openai_backend_pool_2 = f\"oai-backend-pool-{openai_deployment_2_name}\"\n",
    "openai_backend_pool_3 = f\"oai-backend-pool-{openai_deployment_3_name}\"\n",
    "\n",
    "log_analytics_name = \"workspace\"\n",
    "app_insights_name = 'insights'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='1'></a>\n",
    "### 1️⃣ Create the Azure Resource Group\n",
    "All resources deployed in this lab will be created in the specified resource group. Skip this step if you want to use an existing resource group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# type: ignore\n",
    "resource_group_stdout = ! az group create --name {resource_group_name} --location {resource_group_location}\n",
    "\n",
    "if resource_group_stdout.n.startswith(\"ERROR\"):\n",
    "    print(resource_group_stdout)\n",
    "else:\n",
    "    print(\"✅ Azure Resource Group \", resource_group_name, \" created ⌚ \", datetime.datetime.now().time())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2'></a>\n",
    "### 2️⃣ Create deployment using 🦾 Bicep\n",
    "\n",
    "This lab uses [Bicep](https://learn.microsoft.com/azure/azure-resource-manager/bicep/overview?tabs=bicep) to declarative define all the resources that will be deployed. Change the parameters or the [main.bicep](main.bicep) directly to try different configurations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(openai_resources_1) > 0:\n",
    "    backend_id_1 = openai_backend_pool_1 if len(openai_resources_1) > 1 else openai_resources_1[0].get(\"name\")\n",
    "if len(openai_resources_2) > 0:\n",
    "    backend_id_2 = openai_backend_pool_2 if len(openai_resources_2) > 1 else openai_resources_2[0].get(\"name\")\n",
    "if len(openai_resources_3) > 0:\n",
    "    backend_id_3 = openai_backend_pool_3 if len(openai_resources_3) > 1 else openai_resources_3[0].get(\"name\")\n",
    "\n",
    "with open(\"policy.xml\", 'r') as policy_xml_file:\n",
    "    policy_template_xml = policy_xml_file.read()\n",
    "    policy_xml = policy_template_xml.replace(\"{backend-id-1}\", backend_id_1).replace(\"{backend-id-2}\", backend_id_2).replace(\"{backend-id-3}\", backend_id_3)\n",
    "    policy_xml_file.close()\n",
    "open(\"policy.xml\", 'w').write(policy_xml)\n",
    "\n",
    "bicep_parameters = {\n",
    "  \"$schema\": \"https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#\",\n",
    "  \"contentVersion\": \"1.0.0.0\",\n",
    "  \"parameters\": {\n",
    "    \"openAIBackendPoolName_1\": { \"value\": openai_backend_pool_1 },\n",
    "    \"openAIBackendPoolName_2\": { \"value\": openai_backend_pool_2 },\n",
    "    \"openAIBackendPoolName_3\": { \"value\": openai_backend_pool_3 },\n",
    "    \"openAIConfig_1\": { \"value\": openai_resources_1 },\n",
    "    \"openAIConfig_2\": { \"value\": openai_resources_2 },\n",
    "    \"openAIConfig_3\": { \"value\": openai_resources_3 },\n",
    "    \"openAIDeploymentName_1\": { \"value\": openai_deployment_1_name },\n",
    "    \"openAIDeploymentName_2\": { \"value\": openai_deployment_2_name },\n",
    "    \"openAIDeploymentName_3\": { \"value\": openai_deployment_3_name },\n",
    "    \"openAISku\": { \"value\": openai_resources_sku },\n",
    "    \"openAIModelName_1\": { \"value\": openai_model_1_name },\n",
    "    \"openAIModelName_2\": { \"value\": openai_model_2_name },\n",
    "    \"openAIModelName_3\": { \"value\": openai_model_3_name },\n",
    "    \"openAIModelVersion_1\": { \"value\": openai_model_1_version },\n",
    "    \"openAIModelVersion_2\": { \"value\": openai_model_2_version },\n",
    "    \"openAIModelVersion_3\": { \"value\": openai_model_3_version },\n",
    "    \"openAIAPISpecURL\": { \"value\": openai_specification_url },\n",
    "    \"apimResourceName\": { \"value\": apim_resource_name},\n",
    "    \"apimResourceLocation\": { \"value\": apim_resource_location},\n",
    "    \"apimSku\": { \"value\": apim_resource_sku},\n",
    "    \"logAnalyticsName\": { \"value\": log_analytics_name },\n",
    "    \"applicationInsightsName\": { \"value\": app_insights_name },\n",
    "    \"index\": { \"value\": index}\n",
    "  }\n",
    "}\n",
    "with open('params.json', 'w') as bicep_parameters_file:\n",
    "    bicep_parameters_file.write(json.dumps(bicep_parameters))\n",
    "\n",
    "! az deployment group create --name {deployment_name} --resource-group {resource_group_name} --template-file \"main.bicep\" --parameters \"params.json\"\n",
    "\n",
    "open(\"policy.xml\", 'w').write(policy_template_xml)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3'></a>\n",
    "### 3️⃣ Get the deployment outputs\n",
    "\n",
    "We are now at the stage where we only need to retrieve the gateway URL and the subscription before we are ready for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# type: ignore\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.apimSubscriptionKey.value -o tsv\n",
    "apim_subscription_key = deployment_stdout.n\n",
    "\n",
    "# type: ignore\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.apimResourceGatewayURL.value -o tsv\n",
    "apim_resource_gateway_url = deployment_stdout.n\n",
    "print(\"👉🏻 API Gateway URL: \", apim_resource_gateway_url)\n",
    "\n",
    "# type: ignore\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.logAnalyticsWorkspaceId.value -o tsv\n",
    "workspace_id = deployment_stdout.n\n",
    "print(\"👉🏻 Workspace ID: \", workspace_id)\n",
    "\n",
    "# type: ignore\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.applicationInsightsAppId.value -o tsv\n",
    "app_id = deployment_stdout.n\n",
    "print(\"👉🏻 App ID: \", app_id)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='requests'></a>\n",
    "### 🧪 Test the API using a direct HTTP call\n",
    "Requests is an elegant and simple HTTP library for Python that will be used here to make raw API requests and inspect the responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "runs = 2\n",
    "sleep_time_ms = 1000\n",
    "\n",
    "# Initialize a session for connection pooling\n",
    "session = requests.Session()\n",
    "\n",
    "def make_api_request(deployment_name, openai_resources, apim_subscription_key, apim_resource_gateway_url, openai_api_version, sleep_time_ms):\n",
    "    if len(openai_resources) > 0:\n",
    "        messages = {\n",
    "            \"messages\": [\n",
    "                {\"role\": \"system\", \"content\": \"You are a sarcastic, unhelpful assistant.\"},\n",
    "                {\"role\": \"user\", \"content\": \"Can you tell me the time, please?\"}\n",
    "            ]\n",
    "        }\n",
    "        url = f\"{apim_resource_gateway_url}/openai/deployments/{deployment_name}/chat/completions?api-version={openai_api_version}\"\n",
    "        response = session.post(url, headers={'api-key': apim_subscription_key}, json=messages)\n",
    "        print(\"url: \", url)\n",
    "        print(\"status code: \", response.status_code)\n",
    "        print(\"headers \", response.headers)\n",
    "        print(\"x-ms-region: \", response.headers.get(\"x-ms-region\"))  # Useful to determine the region of the backend\n",
    "        if response.status_code == 200:\n",
    "            data = response.json()\n",
    "            print(\"\\nresponse: \", data.get(\"choices\")[0].get(\"message\").get(\"content\"))\n",
    "        else:\n",
    "            print(response.text)\n",
    "        time.sleep(sleep_time_ms / 1000)\n",
    "\n",
    "# Define your deployments and resources\n",
    "deployments = [\n",
    "    {\"name\": openai_deployment_1_name, \"resources\": openai_resources_1},    # GPT-3.5 Turbo 1106\n",
    "    {\"name\": openai_deployment_2_name, \"resources\": openai_resources_2},    # GPT-3.5 Turbo 0125\n",
    "    {\"name\": openai_deployment_3_name, \"resources\": openai_resources_3},    # GPT-4o 2024-05-13\n",
    "]\n",
    "\n",
    "# Loop through each deployment and make the API request\n",
    "for deployment in deployments:\n",
    "    for i in range(runs):\n",
    "        print(f\"\\n▶️ Run: {i+1} for deployment {deployment['name']}\")\n",
    "        make_api_request(deployment['name'], deployment['resources'], apim_subscription_key, apim_resource_gateway_url, openai_api_version, sleep_time_ms)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='sdk'></a>\n",
    "### 🧪 Test the API using the Azure OpenAI Python SDK\n",
    "OpenAPI provides a widely used [Python library](https://github.com/openai/openai-python). The library includes type definitions for all request params and response fields. The goal of this test is to assert that APIM can seamlessly proxy requests to OpenAI without disrupting its functionality.\n",
    "- Note: run ```pip install openai``` in a terminal before executing this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "runs = 2\n",
    "sleep_time_ms = 1000\n",
    "\n",
    "def make_openaisdk_request(deployment_name, openai_resources, apim_subscription_key, apim_resource_gateway_url, openai_api_version, sleep_time_ms):\n",
    "    from openai import AzureOpenAI\n",
    "    if len(openai_resources) > 0:\n",
    "        messages = {\n",
    "            \"messages\": [\n",
    "                {\"role\": \"system\", \"content\": \"You are a sarcastic, unhelpful assistant.\"},\n",
    "                {\"role\": \"user\", \"content\": \"Can you tell me the time, please?\"}\n",
    "            ]\n",
    "        }\n",
    "        client = AzureOpenAI(\n",
    "            azure_endpoint=apim_resource_gateway_url,\n",
    "            api_key=apim_subscription_key,\n",
    "            api_version=openai_api_version\n",
    "        )\n",
    "        response = client.chat.completions.create(model=openai_deployment_1_name, messages=messages)\n",
    "        print(response.choices[0].message.content)\n",
    "        time.sleep(sleep_time_ms/1000)\n",
    "\n",
    "# Define your deployments and resources\n",
    "deployments = [\n",
    "    {\"name\": openai_deployment_1_name, \"resources\": openai_resources_1},    # GPT-3.5 Turbo 1106\n",
    "    {\"name\": openai_deployment_2_name, \"resources\": openai_resources_2},    # GPT-3.5 Turbo 0125\n",
    "    {\"name\": openai_deployment_3_name, \"resources\": openai_resources_3},    # GPT-4o 2024-05-13\n",
    "]\n",
    "\n",
    "# Loop through each deployment and make the API request\n",
    "for deployment in deployments:\n",
    "    for i in range(runs):\n",
    "        print(f\"\\n▶️ Run: {i+1} for deployment {deployment['name']}\")\n",
    "        make_api_request(deployment['name'], deployment['resources'], apim_subscription_key, apim_resource_gateway_url, openai_api_version, sleep_time_ms)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='portal'></a>\n",
    "### 🔍 Open the workbook in the Azure Portal\n",
    "\n",
    "Open the workbook resource and review the usage analysis to confirm the model routing and other metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='clean'></a>\n",
    "### 🗑️ Clean up resources\n",
    "\n",
    "When you're finished with the lab, you should remove all your deployed resources from Azure to avoid extra charges and keep your Azure subscription uncluttered.\n",
    "Use the [clean-up-resources notebook](clean-up-resources.ipynb) for that."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}