03-triton-vertex-inference-example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jirdTjhETQW0"
   },
   "source": [
    "# **redis-feast-gcp**: 03 - Triton + Vertex AI Prediction Inference Example\n",
    "\n",
    "In this notebook, we will test the deployed Triton model on the Vertex AI Prediction endpoint.\n",
    "\n",
    "**This notebook assumes that you've already set up your Feature Store, model repo in GCP, and deployed your model in Vertex AI with NVIDIA Triton**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![architecture](img/redis-feast-gcp-architecture.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unpacking the Triton Ensemble\n",
    "\n",
    "Before we test the inference endpoint to forecast Covid vaccinations for the state of Virginia, we will unpack the Triton Ensemble used to create the DAG of operations.\n",
    "\n",
    "### What is an Ensemble???\n",
    "An [Ensemble model](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/architecture.md#ensemble-models) represents a pipeline of one or many operations (models) and connects inputs to outputs of each stage. These are useful for inference workflows that involve several stages like data preprocessing, postprocessing, and other transformations or business logic.\n",
    "\n",
    "\n",
    "![ensemble](./img/RedisFeastTriton.png)\n",
    "\n",
    "**Checkout the structure of the Triton Model repository below.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ensemble  fetch-vaccine-features  predict-vaccine-counts\n"
     ]
    }
   ],
   "source": [
    "!ls ./docker/triton/models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a model for:\n",
    "- `fetch-vaccine-features` - Fetch vaccine count features from Redis at low-latency.\n",
    "- `predict-vaccine-counts` - Use XGBoost model (with [Triton FIL](https://developer.nvidia.com/blog/real-time-serving-for-xgboost-scikit-learn-randomforest-lightgbm-and-more/) backend) to forecast the counts.\n",
    "- `ensemble` - Wraps the other two - creating the pipeline.\n",
    "\n",
    "Each model here has a `config.pbtxt`. Let's look at the ensemble model config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: \"ensemble\"\n",
      "platform: \"ensemble\"\n",
      "max_batch_size: 256\n",
      "input [\n",
      "  {\n",
      "    name: \"state\"\n",
      "    data_type: TYPE_STRING\n",
      "    dims: 1\n",
      "  }\n",
      "]\n",
      "output [\n",
      "  {\n",
      "    name: \"prediction\"\n",
      "    data_type: TYPE_FP32\n",
      "    dims: 1\n",
      "  }\n",
      "]\n",
      "ensemble_scheduling {\n",
      "  step [\n",
      "    {\n",
      "      model_name: \"fetch-vaccine-features\"\n",
      "      model_version: -1\n",
      "      input_map {\n",
      "        key: \"state\"\n",
      "        value: \"state\"\n",
      "      }\n",
      "      output_map {\n",
      "        key: \"feature_values\"\n",
      "        value: \"feature_values\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      model_name: \"predict-vaccine-counts\"\n",
      "      model_version: -1\n",
      "      input_map {\n",
      "        key: \"input__0\"\n",
      "        value: \"feature_values\"\n",
      "      }\n",
      "      output_map {\n",
      "        key: \"output__0\"\n",
      "        value: \"prediction\"\n",
      "      }\n",
      "    }\n",
      "  ]\n",
      "}"
     ]
    }
   ],
   "source": [
    "!cat ./docker/triton/models/ensemble/config.pbtxt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Inference Instances\n",
    "\n",
    "Before we can test the Vertex AI Prediction endpoint, we need to construct a JSON body that represents an inference request. See the example below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# Create inference instance\n",
    "payload = {\n",
    "    \"id\": \"1\",\n",
    "    \"inputs\": [\n",
    "        {\n",
    "            \"name\": \"state\",        ## Triton model input name\n",
    "            \"shape\": [1, 1],        ## Triton model input shape\n",
    "            \"datatype\": \"BYTES\",    ## Triton model input datatype\n",
    "            \"data\": [[\"Virginia\"]]  ## Triton model input data\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "# Save to file\n",
    "with open(\"instances.json\", \"w\") as f:\n",
    "    json.dump(payload, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6cJFAJiuGxM3"
   },
   "source": [
    "## Test Endpoint\n",
    "\n",
    "You can test the Vertex AI Prediction `rawPredict` endpoint using any HTTP tool or library, including `curl`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Activated service account credentials for: [tyler-redis@nifty-time-353221.iam.gserviceaccount.com]\n",
      "\n",
      "\n",
      "Updates are available for some Google Cloud CLI components.  To install them,\n",
      "please run:\n",
      "  $ gcloud components update\n",
      "\n",
      "\n",
      "\n",
      "To take a quick anonymous survey, run:\n",
      "  $ gcloud survey\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Log in to GCloud using the CLI and your service account\n",
    "!gcloud auth activate-service-account $SERVICE_ACCOUNT_EMAIL \\\n",
    "    --key-file=$GOOGLE_APPLICATION_CREDENTIALS \\\n",
    "    --project=$PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using endpoint [https://us-east1-aiplatform.googleapis.com/]\n",
      "4273405872979312640\n"
     ]
    }
   ],
   "source": [
    "# List out the Vertex AI endpoints\n",
    "!echo $(gcloud ai endpoints list \\\n",
    "  --region=$GCP_REGION \\\n",
    "  --filter=display_name=vaccine-predictor-endpoint \\\n",
    "  --format=\"value(name)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":\"1\",\"model_name\":\"ensemble\",\"model_version\":\"1\",\"parameters\":{\"sequence_id\":0,\"sequence_start\":false,\"sequence_end\":false},\"outputs\":[{\"name\":\"prediction\",\"datatype\":\"FP32\",\"shape\":[1],\"data\":[40990.265625]}]}"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using endpoint [https://us-east1-aiplatform.googleapis.com/]\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100   319    0   215  100   104    181     87  0:00:01  0:00:01 --:--:--   268\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# Fetch Token\n",
    "TOKEN=$(gcloud auth print-access-token)\n",
    "\n",
    "# Fetch the Endpoint ID\n",
    "ENDPOINT_ID=$(gcloud ai endpoints list \\\n",
    "  --region=$GCP_REGION \\\n",
    "  --filter=display_name=vaccine-predictor-endpoint \\\n",
    "  --format=\"value(name)\")\n",
    "\n",
    "# POST to the endpoint to get a response from the Triton ensemble model\n",
    "curl \\\n",
    "  -X POST \\\n",
    "  -H \"Authorization: Bearer ${TOKEN}\" \\\n",
    "  -H \"Content-Type: application/json\" \\\n",
    "  https://${GCP_REGION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-east1/endpoints/${ENDPOINT_ID}:rawPredict \\\n",
    "  -d \"@instances.json\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "We have just built an end-to-end ML system using Feast, Redis Enterprise, and NVIDIA Triton -- all in GCP.\n",
    "\n",
    "This system generates realtime predictions using up to date feature values and allows us to manage \"point-in-time correct\" datasets from our offline datasource for training and model exploration.\n",
    "\n",
    "Next steps that you can take after completing this tutorial include:\n",
    "\n",
    "- Pull this repo and collaboration with your team.\n",
    "- Use this tutorial to bootstrap a model for your use case by editing features / model.\n",
    "- Incorporate the code in this tutorial into your company's batch pipelines by creating stages that perform feature creation and materialization.\n",
    "\n",
    "**Redis and Triton are a perfect match: bringing the data layer (optimized for fast data access) close to the computing infrastructure (optimized for fast data processing).**"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Fraud Detection Tutorial",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "e1c5a7c9cc0d58080444e081b74a0823c09a12f0209aca730c38726ea6940124"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}