custom_model_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building, Testing, and Deploying a Custom Model\n",
    "\n",
    "This notebook walks through the general workflow for building, testing, and deploying a custom inference model on a custom environment. \n",
    "\n",
    "## Note\n",
    "To finish this tutorial, you must have access to either Cloud DataRobot or On-Site Deploy of DataRobot.\n",
    "\n",
    "This tutorial is using Cloud DataRobot (app.datarobot.com).\n",
    "\n",
    "## Agenda\n",
    "In this tutorial, we'll learn:\n",
    "1. How to use the client to create an environment\n",
    "2. How to check the status of an environment build\n",
    "3. How to create a custom model\n",
    "4. How to iteratively test and debug a custom model on a custom environment\n",
    "5. How to deploy and run predictions on a tested custom model.\n",
    "\n",
    "## Setup and Requirements\n",
    "This tutorial assumes a few things about your filepath and prior work. \n",
    "\n",
    "**Firstly, you need a feature flag enabled:**\n",
    "- Enable MLOps\n",
    "\n",
    "Secondly, you should have a folder at the path `~/datarobot-user-models/`. If you put the folder in a different location, make sure you update the `TESTING_PATH` variable. This folder should contain 4 things:\n",
    "1. A folder containing your properly configured custom environment.     \n",
    "    In this example, it's named `public_dropin_environments/python3_pytorch/`\n",
    "    \n",
    "    \n",
    "2. A folder containing your properly-configured custom model.     \n",
    "    In this example, it's named `model_templates/python3_pytorch/`\n",
    "    \n",
    "    \n",
    "3. The current version of the DataRobot Python Client.\n",
    "    - Installation instructions for the client can be found here: [DataRobot Python Client Docs](https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.20.0/setup/getting_started.html#installation)\n",
    "    - Full documentation for the client can be found here: [DataRobot Python Client Docs](https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.20.0/index.html)\n",
    "\n",
    "\n",
    "4. A test dataset that you can use to test predictions from your custom model.     \n",
    "    In this example, it's stored at `tests/testdata/juniors_3_year_stats_regression.csv`\n",
    "\n",
    "It also assumes that you have access to app.datarobot.com.\n",
    "If you use another version of DataRobot - use appropriate credentials and URL.\n",
    "\n",
    "\n",
    "## Configuring Models and Environments\n",
    "For more information on how to properly configure custom models and environments, read the README of our [DataRobot User Models repository](https://github.com/datarobot/datarobot-user-models).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "First, we need to make the proper imports. Make sure the `TESTING_PATH` is correct and pointing to the right folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import sys\n",
    "import os\n",
    "import requests\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is where you save the `TESTING_PATH` that contains the relevant folders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the path to the custom model testing folder, and add it to the PYTHONPATH so we can import the client\n",
    "TESTING_PATH = os.getcwd() + '/'\n",
    "sys.path.append(TESTING_PATH)\n",
    "\n",
    "import datarobot as dr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Configuring User Credentials\n",
    "Make sure to fill in your username and API token from app.datarobot.com.\n",
    "\n",
    "Also ensure that all the paths are correct!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Save user credentials ##\n",
    "TOKEN = ''\n",
    "USERNAME = ''\n",
    "DATAROBOT_KEY = ''  # required to make predictions against deployments\n",
    "\n",
    "## Save path to environment ##\n",
    "environment_folder = TESTING_PATH + 'public_dropin_environments/python3_pytorch/'\n",
    "\n",
    "## Save path to custom model ##\n",
    "custom_model_folder = TESTING_PATH + 'model_templates/python3_pytorch/'\n",
    "\n",
    "## Save test dataset path ##\n",
    "test_dataset = TESTING_PATH + 'tests/testdata/juniors_3_year_stats_regression.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the API client\n",
    "This command initializes the API client. **You shouldn't need to change anything in this block if you configured your credentials properly!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure client\n",
    "client = dr.Client(\n",
    "    endpoint='https://app.datarobot.com/api/v2',\n",
    "    token=TOKEN,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Custom Environment\n",
    "This command creates a custom environment! When you run the command, it uploads your Docker context and we attempt to build the Docker Image (the container that your model will eventually run in). \n",
    " \n",
    "Depending on the environment and the libraries you want to download, this process can take a while (10-30 minutes)! This command sets the wait time to 1 hour, but if it fails with a AsyncTimeoutError, it's possible that the environment is still processing and could still succeed.\n",
    "\n",
    "### Custom Environment Templates\n",
    "Custom environment templates can be found here: [environment templates](https://github.com/datarobot/datarobot-user-models/tree/master/public_dropin_environments)\n",
    "\n",
    "You'll find templates for Python 3, Java and R environments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## Create the environment, which will eventually contain versions  ##\n",
    "execution_environment = dr.ExecutionEnvironment.create(\n",
    "    name=\"Python3 PyTorch Environment\",\n",
    "    description=\"This environment contains Python3 pytorch library.\",\n",
    ")\n",
    "\n",
    "## Create the environment version ##\n",
    "environment_version = dr.ExecutionEnvironmentVersion.create(\n",
    "    execution_environment.id,\n",
    "    environment_folder,\n",
    "    max_wait=3600,  # 1 hour timeout\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Custom Model\n",
    "Once the Custom Environment is successfully built, now it's time to build the Custom Model. You will need to define details about your custom model in this command, depending on the type of model.\n",
    "\n",
    "### Required fields:\n",
    "`model_path` : string containing the path to the model folder\n",
    "\n",
    "`name` : string that defines the name of the model\n",
    "\n",
    "`target_name` : string that defines the name of the target column that the model was trained on\n",
    "\n",
    "`target_type` : boolean that describes the target type. Supported target types are \"Binary\" (`datarobot.TARGET_TYPE.BINARY`) and \"Regression\" (`datarobot.TARGET_TYPE.REGRESSION`).\n",
    "\n",
    "`positive_class_label` : string that defines the \"positive class\". Only required for Binary Classification models\n",
    "\n",
    "`negative_class_label` : string that defines the \"negative class\". Only required for Binary Classification models\n",
    "\n",
    "### Optional Fields:\n",
    "`prediction_threshold` : a float that defines the prediction threshold for binary classification. This value is used for features and charts in MMM.\n",
    "\n",
    "`description` : a string that describe the model. User can input whatever they want for the description.\n",
    "\n",
    "`language` : a string that details the language the model uses. User can input whatever they want for the language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create the custom model ##\n",
    "custom_model = dr.CustomInferenceModel.create(\n",
    "    name='Python 3 PyTorch Custom Model',\n",
    "    target_type=dr.TARGET_TYPE.REGRESSION,\n",
    "    target_name='Grade 2014',\n",
    "    description='This is a Python3-based custom model. It has a simple PyTorch model built on juniors_3_year_stats_regression dataset',\n",
    "    language='python'\n",
    ")\n",
    "\n",
    "## Create the custom model version ##\n",
    "model_version = dr.CustomModelVersion.create_clean(\n",
    "    custom_model_id=custom_model.id,\n",
    "    folder_path=custom_model_folder,\n",
    "    base_environment_id=execution_environment.id\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Model Testing Workflow\n",
    "Just because you created an environment and a model doesn't mean that it will actually work in production! There are all sorts of things that can go wrong, whether on the engineering side or the data science side. Bad code, an environment with the wrong versions of libraries, or even a model that can't handle missing values in the inference data can all lead to a model that will break in production.\n",
    "\n",
    "With this in mind, we created an easy way to ensure that a custom inference model will work in production: You can actually test your model with a specific environment using sample inference data before deploying the model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Run the Test\n",
    "To run a custom model test, you upload and save a test dataset from the sample inference data. Then, you simply select the appropriate model and environment (as well as version) IDs, and test it on that dataset.\n",
    "\n",
    "Depending on the k8s cluster and the model itself, it may take a few minutes to test the model. Once the test is finished, it will have a status property to let you know whether the test passed. If it failed, it will contain an `error` property that contains the relevant error!\n",
    "\n",
    "An important note: As of right now, the only available test is an error check, where we simply ensure the model can return predictions. In the future, we will add more tests to that suite: prediction consistency, missing value handling, and more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dr.Dataset.create_from_file(file_path=test_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Perform custom model test\n",
    "custom_model_test = dr.CustomModelTest.create(\n",
    "    custom_model_id=custom_model.id, \n",
    "    custom_model_version_id=model_version.id,\n",
    "    dataset_id=dataset.id,\n",
    "    max_wait=3600,  # 1 hour timeout\n",
    ")\n",
    "\n",
    "print(\"Overall testing status: {}\".format(custom_model_test.overall_status))\n",
    "\n",
    "if any(test['status'] == 'failed' for test in custom_model_test.detailed_status.values()):\n",
    "    print('Test log:\\n')\n",
    "    print(custom_model_test.get_log())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Iterate\n",
    "If the test passed, then congratulations! You can skip this test; your model is ready to be deployed. If it failed the test however, it's easy to iterate. \n",
    "\n",
    "First, check the error from the custom model test. Then, fix any errors in the code that you uploaded. Finally, upload a new version of the model using the updated code, and test it again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add new version of custom model. Repeat these last two blocks until the model passes testing!\n",
    "model_version = dr.CustomModelVersion.create_clean(\n",
    "    custom_model_id=custom_model.id,\n",
    "    folder_path=custom_model_folder,\n",
    "    base_environment_id=execution_environment.id\n",
    ")\n",
    "model_version.update(description='Fixing errors from testing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform custom model test... again\n",
    "custom_model_test = dr.CustomModelTest.create(\n",
    "    custom_model_id=custom_model.id, \n",
    "    custom_model_version_id=model_version.id,\n",
    "    dataset_id=dataset.id,\n",
    "    max_wait=3600,  # 1 hour timeout\n",
    ")\n",
    "\n",
    "print(\"Overall testing status: {}\".format(custom_model_test.overall_status))\n",
    "\n",
    "if any(test['status'] == 'failed' for test in custom_model_test.detailed_status.values()):\n",
    "    print('Test log:\\n')\n",
    "    print(custom_model_test.get_log())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This command shows all tests that have been run on the model\n",
    "model_tests = dr.CustomModelTest.list(custom_model_id=custom_model.id)\n",
    "print(model_tests)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying the model\n",
    "To deploy an inference model, you create something called a `custom_model_image`, which saves the custom model code with a _specific_ environment. This will make it easy to see which custom models have been tested or deployed on specific environments.\n",
    "\n",
    "Once you have the desired custom model image, simply call the `dr.Deployment.create_from_custom_model_image()` method, inputting the model image's id, the prediction server's `default_prediction_server_id`, and the desired deployment label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure that the client is using the correct prediction server to deploy the model. \n",
    "# This uses the prediction server for testing on Cloud DataRobot.\n",
    "\n",
    "available_prediction_server_urls = [\n",
    "    \"https://datarobot-predictions.orm.datarobot.com\",\n",
    "]\n",
    "\n",
    "prediction_server = None\n",
    "\n",
    "for pred_server in dr.PredictionServer.list():\n",
    "    if pred_server.url in available_prediction_server_urls:\n",
    "        prediction_server = pred_server\n",
    "        break\n",
    "else:\n",
    "    raise Exception(\"no suitable prediction server found\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "deployment = dr.Deployment.create_from_custom_model_version(\n",
    "    model_version.id,\n",
    "    label='Test client deployment',\n",
    "    # instance id is only required for Cloud DataRobot App\n",
    "    # ignore for on-premises Platform installations.\n",
    "    default_prediction_server_id=prediction_server.id,\n",
    "    max_wait=3600,  # 1 hour timeout\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making predictions on a deployed custom inference model\n",
    "Predictions look exactly the same for a custom inference model and a native DR model. If training data was assigned to the model, then we can also provide predictions explanations and all MMM features, deeply integrated with the custom model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions on the custom model deployment\n",
    "url = '{}/predApi/v1.0/deployments/{}/predictions'.format(prediction_server.url, deployment.id)\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "predictions_dataset = pd.read_csv(test_dataset)\n",
    "predictions_data = predictions_dataset.to_json(orient='records')\n",
    "\n",
    "headers = dr.client.get_client().headers\n",
    "headers['datarobot-key'] = DATAROBOT_KEY\n",
    "headers['Content-Type'] = 'application/json'\n",
    "\n",
    "response = requests.post(url, headers=headers, data=predictions_data)\n",
    "\n",
    "predictions = response.json()\n",
    "pprint(predictions)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env",
   "language": "python",
   "name": "env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}