nlp-llm-llmops-pipeline-dev-stag-prod.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":32267,"sourceType":"datasetVersion","datasetId":24984}],"dockerImageVersionId":30627,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"source":"<a href=\"https://www.kaggle.com/code/yannicksteph/nlp-llm-llmops-pipeline-dev-stag-prod?scriptVersionId=158424344\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","source":"# | NLP | LLM | LLMOps | Pipeline Dev Stag Prod |\n\n## Natural Language Processing (NLP) and Large Language Models (LLM) with LLMOps and make a Pipeline Dev Stag Prod.\n\n\n![Learning](https://t3.ftcdn.net/jpg/06/14/01/52/360_F_614015247_EWZHvC6AAOsaIOepakhyJvMqUu5tpLfY.jpg)\n\n\n# <b>1 <span style='color:#78D118'>|</span> Overview</b>\n\n\nIn this example, we will walk through some key steps for taking an LLM-based pipeline to production.  Our pipeline will be: summarization of news articles using a pre-trained model from Hugging Face.  \n\nBut in this walkthrough, we will be more rigorous about LLMOps.\n\n**Develop an LLM pipeline**\n\nOur LLMOps goals during development are (a) to track what we do carefully for later auditing and reproducibility and (b) to package models or pipelines in a format which will make future deployment easier.  Step-by-step, we will:\n* Load data.\n* Build an LLM pipeline.\n* Test applying the pipeline to data, and log queries and results to MLflow Tracking.\n* Log the pipeline to the MLflow Tracking server as an MLflow Model.\n\n**Test the LLM pipeline**\n\nOur LLMOps goals during testing (in the staging or QA stage) are (a) to track the LLM's progress through testing and towards production and (b) to do so programmatically to demonstrate the APIs needed for future CI/CD automation.  Step-by-step, we will:\n* Register the pipeline to the MLflow Model Registry.\n* Test the pipeline on sample data.\n* Promote the registered model (pipeline) to production.\n\n**Create a production workflow for batch inference**\n\nOur LLMOps goals during production are (a) to write scale-out code which can meet scaling demands in the future and (b) to simplify deployment by using MLflow to write model-agnostic deployment code.  Step-by-step, we will:\n* Load the latest production LLM pipeline from the Model Registry.\n* Apply the pipeline to an Apache Spark DataFrame.\n* Append the results to a Delta Lake table.\n\n### Notes about this workflow\n\n**This notebook vs. modular scripts**: Since this demo is in a single notebook, we will divide the workflow from development to production via notebook sections.  In a more realistic LLM Ops setup, you would likely have the sections split into separate notebooks or scripts.\n\n**Promoting models vs. code**: We track the path from development to production via the MLflow Model Registry.  That is, we are *promoting models* towards production, rather than promoting code.  For more discussion of these two paradigms, see [\"The Big Book of MLOps\"](https://www.databricks.com/resources/ebook/the-big-book-of-mlops).\n\n### Learning Objectives\n1. Walk through a simple but realistic workflow to take an LLM pipeline from development to production.\n2. Make use of MLflow Tracking and the Model Registry to package and manage the pipeline.\n3. Scale out batch inference using Apache Spark and Delta Lake.\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"### Setup\n","metadata":{}},{"cell_type":"code","source":"%%capture\n\n!pip install sparkmagic\n!pip install pyspark\n!pip install mlflow\n!pip install pandas --upgrade","metadata":{"execution":{"iopub.status.busy":"2024-01-10T09:59:21.86856Z","iopub.execute_input":"2024-01-10T09:59:21.868919Z","iopub.status.idle":"2024-01-10T10:04:05.973724Z","shell.execute_reply.started":"2024-01-10T09:59:21.86889Z","shell.execute_reply":"2024-01-10T10:04:05.972511Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"code","source":"cache_dir = \"./cache\"\nuser_path = \"user\"\nworking_dir= \"./working_dir\"","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:04:05.97623Z","iopub.execute_input":"2024-01-10T10:04:05.976714Z","iopub.status.idle":"2024-01-10T10:04:05.981826Z","shell.execute_reply.started":"2024-01-10T10:04:05.976679Z","shell.execute_reply":"2024-01-10T10:04:05.980832Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\npd.set_option('display.max_column', None)\npd.set_option('display.max_rows', None)\npd.set_option('display.max_seq_items', None)\npd.set_option('display.max_colwidth', 500)\npd.set_option('expand_frame_repr', True)","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:04:05.982991Z","iopub.execute_input":"2024-01-10T10:04:05.983322Z","iopub.status.idle":"2024-01-10T10:04:06.404149Z","shell.execute_reply.started":"2024-01-10T10:04:05.983288Z","shell.execute_reply":"2024-01-10T10:04:06.403389Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"# <b>2 <span style='color:#78D118'>|</span> Prepare data</b>\n\nFor this notebook we'll use the <a href=\"https://huggingface.co/datasets/xsum\" target=\"_blank\">Extreme Summarization (XSum) Dataset</a>  with the <a href=\"https://huggingface.co/t5-small\" target=\"_blank\">T5 Text-To-Text Transfer Transformer</a> from Hugging Face.\n","metadata":{"execution":{"iopub.status.busy":"2023-12-29T10:22:36.998626Z","iopub.execute_input":"2023-12-29T10:22:36.99952Z","iopub.status.idle":"2023-12-29T10:22:37.004082Z","shell.execute_reply.started":"2023-12-29T10:22:36.999482Z","shell.execute_reply":"2023-12-29T10:22:37.002916Z"}}},{"cell_type":"code","source":"from datasets import load_dataset\nfrom transformers import pipeline\n\nxsum_dataset = load_dataset(\n    \"xsum\", \n    version=\"1.2.0\",\n    cache_dir=cache_dir\n)  # Note: We specify cache_dir to use pre-cached data.\nxsum_sample = xsum_dataset[\"train\"].select(range(10))\ndisplay(xsum_sample.to_pandas().head())","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:04:06.406696Z","iopub.execute_input":"2024-01-10T10:04:06.407452Z","iopub.status.idle":"2024-01-10T10:06:26.41298Z","shell.execute_reply.started":"2024-01-10T10:04:06.407415Z","shell.execute_reply":"2024-01-10T10:06:26.412052Z"},"trusted":true},"execution_count":4,"outputs":[{"name":"stderr","text":"/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3\n  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"Downloading builder script:   0%|          | 0.00/2.05k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"d5639024e74a49b7abce9f38a9d60ab9"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading metadata:   0%|          | 0.00/954 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"9a89a76ce1494a76b821d88e61899391"}},"metadata":{}},{"name":"stdout","text":"Downloading and preparing dataset xsum/default (download: 245.38 MiB, generated: 507.60 MiB, post-processed: Unknown size, total: 752.98 MiB) to ./cache/xsum/default/1.2.0/32c23220eadddb1149b16ed2e9430a05293768cfffbdfd151058697d4c11f934...\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"d40df1e268ad4b25b9e2302740ad1851"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading data:   0%|          | 0.00/255M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"ed2fbf13d51440cc895955f80d5f1206"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading data:   0%|          | 0.00/1.00M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"1b9d9ce85b2d45c1a28e13464ea1ae67"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Generating train split:   0%|          | 0/204045 [00:00<?, ? examples/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Generating validation split:   0%|          | 0/11332 [00:00<?, ? examples/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Generating test split:   0%|          | 0/11334 [00:00<?, ? examples/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"name":"stdout","text":"Dataset xsum downloaded and prepared to ./cache/xsum/default/1.2.0/32c23220eadddb1149b16ed2e9430a05293768cfffbdfd151058697d4c11f934. Subsequent calls will reuse this data.\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"  0%|          | 0/3 [00:00<?, ?it/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"1827e3c36a474b71b2b6ffaad64f3348"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              document  \\\n0  The full cost of damage in Newton Stewart, one of the areas worst affected, is still being assessed.\\nRepair work is ongoing in Hawick and many roads in Peeblesshire remain badly affected by standing water.\\nTrains on the west coast mainline face disruption due to damage at the Lamington Viaduct.\\nMany businesses and householders were affected by flooding in Newton Stewart after the River Cree overflowed into the town.\\nFirst Minister Nicola Sturgeon visited the area to inspect the damage.\\n...   \n1  A fire alarm went off at the Holiday Inn in Hope Street at about 04:20 BST on Saturday and guests were asked to leave the hotel.\\nAs they gathered outside they saw the two buses, parked side-by-side in the car park, engulfed by flames.\\nOne of the tour groups is from Germany, the other from China and Taiwan. It was their first night in Northern Ireland.\\nThe driver of one of the buses said many of the passengers had left personal belongings on board and these had been destroyed.\\nBoth groups...   \n2  Ferrari appeared in a position to challenge until the final laps, when the Mercedes stretched their legs to go half a second clear of the red cars.\\nSebastian Vettel will start third ahead of team-mate Kimi Raikkonen.\\nThe world champion subsequently escaped punishment for reversing in the pit lane, which could have seen him stripped of pole.\\nBut stewards only handed Hamilton a reprimand, after governing body the FIA said \"no clear instruction was given on where he should park\".\\nBelgian St...   \n3  John Edward Bates, formerly of Spalding, Lincolnshire, but now living in London, faces a total of 22 charges, including two counts of indecency with a child.\\nThe 67-year-old is accused of committing the offences between March 1972 and October 1989.\\nMr Bates denies all the charges.\\nGrace Hale, prosecuting, told the jury that the allegations of sexual abuse were made by made by four male complainants and related to when Mr Bates was a scout leader in South Lincolnshire and Cambridgeshire.\\n...   \n4  Patients and staff were evacuated from Cerahpasa hospital on Wednesday after a man receiving treatment at the clinic threatened to shoot himself and others.\\nOfficers were deployed to negotiate with the man, a young police officer.\\nEarlier reports that the armed man had taken several people hostage proved incorrect.\\nThe chief consultant of Cerahpasa hospital, Zekayi Kutlubay, who was evacuated from the facility, said that there had been \"no hostage crises\", adding that the man was \"alone i...   \n\n                                                                                                                                         summary  \\\n0                 Clean-up operations are continuing across the Scottish Borders and Dumfries and Galloway after flooding caused by Storm Frank.   \n1                                              Two tourist buses have been destroyed by fire in a suspected arson attack in Belfast city centre.   \n2                                    Lewis Hamilton stormed to pole position at the Bahrain Grand Prix ahead of Mercedes team-mate Nico Rosberg.   \n3                      A former Lincolnshire Police officer carried out a series of sex attacks on boys, a jury at Lincoln Crown Court was told.   \n4  An armed man who locked himself into a room at a psychiatric hospital in Istanbul has ended his threat to kill himself, Turkish media report.   \n\n         id  \n0  35232142  \n1  40143035  \n2  35951548  \n3  36266422  \n4  38826984  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>document</th>\n      <th>summary</th>\n      <th>id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>The full cost of damage in Newton Stewart, one of the areas worst affected, is still being assessed.\\nRepair work is ongoing in Hawick and many roads in Peeblesshire remain badly affected by standing water.\\nTrains on the west coast mainline face disruption due to damage at the Lamington Viaduct.\\nMany businesses and householders were affected by flooding in Newton Stewart after the River Cree overflowed into the town.\\nFirst Minister Nicola Sturgeon visited the area to inspect the damage.\\n...</td>\n      <td>Clean-up operations are continuing across the Scottish Borders and Dumfries and Galloway after flooding caused by Storm Frank.</td>\n      <td>35232142</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A fire alarm went off at the Holiday Inn in Hope Street at about 04:20 BST on Saturday and guests were asked to leave the hotel.\\nAs they gathered outside they saw the two buses, parked side-by-side in the car park, engulfed by flames.\\nOne of the tour groups is from Germany, the other from China and Taiwan. It was their first night in Northern Ireland.\\nThe driver of one of the buses said many of the passengers had left personal belongings on board and these had been destroyed.\\nBoth groups...</td>\n      <td>Two tourist buses have been destroyed by fire in a suspected arson attack in Belfast city centre.</td>\n      <td>40143035</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Ferrari appeared in a position to challenge until the final laps, when the Mercedes stretched their legs to go half a second clear of the red cars.\\nSebastian Vettel will start third ahead of team-mate Kimi Raikkonen.\\nThe world champion subsequently escaped punishment for reversing in the pit lane, which could have seen him stripped of pole.\\nBut stewards only handed Hamilton a reprimand, after governing body the FIA said \"no clear instruction was given on where he should park\".\\nBelgian St...</td>\n      <td>Lewis Hamilton stormed to pole position at the Bahrain Grand Prix ahead of Mercedes team-mate Nico Rosberg.</td>\n      <td>35951548</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>John Edward Bates, formerly of Spalding, Lincolnshire, but now living in London, faces a total of 22 charges, including two counts of indecency with a child.\\nThe 67-year-old is accused of committing the offences between March 1972 and October 1989.\\nMr Bates denies all the charges.\\nGrace Hale, prosecuting, told the jury that the allegations of sexual abuse were made by made by four male complainants and related to when Mr Bates was a scout leader in South Lincolnshire and Cambridgeshire.\\n...</td>\n      <td>A former Lincolnshire Police officer carried out a series of sex attacks on boys, a jury at Lincoln Crown Court was told.</td>\n      <td>36266422</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Patients and staff were evacuated from Cerahpasa hospital on Wednesday after a man receiving treatment at the clinic threatened to shoot himself and others.\\nOfficers were deployed to negotiate with the man, a young police officer.\\nEarlier reports that the armed man had taken several people hostage proved incorrect.\\nThe chief consultant of Cerahpasa hospital, Zekayi Kutlubay, who was evacuated from the facility, said that there had been \"no hostage crises\", adding that the man was \"alone i...</td>\n      <td>An armed man who locked himself into a room at a psychiatric hospital in Istanbul has ended his threat to kill himself, Turkish media report.</td>\n      <td>38826984</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"cell_type":"code","source":"xsum_dataset","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:06:26.414221Z","iopub.execute_input":"2024-01-10T10:06:26.415913Z","iopub.status.idle":"2024-01-10T10:06:26.421479Z","shell.execute_reply.started":"2024-01-10T10:06:26.415874Z","shell.execute_reply":"2024-01-10T10:06:26.420601Z"},"trusted":true},"execution_count":5,"outputs":[{"execution_count":5,"output_type":"execute_result","data":{"text/plain":"DatasetDict({\n    train: Dataset({\n        features: ['document', 'summary', 'id'],\n        num_rows: 204045\n    })\n    validation: Dataset({\n        features: ['document', 'summary', 'id'],\n        num_rows: 11332\n    })\n    test: Dataset({\n        features: ['document', 'summary', 'id'],\n        num_rows: 11334\n    })\n})"},"metadata":{}}]},{"cell_type":"markdown","source":"Later on, when we show Production inference, we will want a dataset saved for it.  See the production section below for more information about Delta, the format we use to save the data here.","metadata":{}},{"cell_type":"markdown","source":"#### Library pre-requisites\nIMPORTANT! Since we will be interacting with Spark by writing a Spark dataframe, we need a Spark Connector.\n\nYou need to attach a Spark-Pinecone connector **s3://pinecone-jars/0.2.1/spark-pinecone-uberjar.jar** in the cluster you are using. Refer to this documentation if you need more information.","metadata":{}},{"cell_type":"code","source":"##################\n#### For Demo ####\n##################\nfrom pyspark.sql import SparkSession\n# Spark in local mode else using spark from your Spark Connector \n# NOTE: For demo\nspark = SparkSession.builder.master(\"local[*]\").getOrCreate()\n##################","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:06:26.422684Z","iopub.execute_input":"2024-01-10T10:06:26.422971Z","iopub.status.idle":"2024-01-10T10:06:31.401692Z","shell.execute_reply.started":"2024-01-10T10:06:26.422947Z","shell.execute_reply":"2024-01-10T10:06:31.400335Z"},"trusted":true},"execution_count":6,"outputs":[{"name":"stderr","text":"Setting default log level to \"WARN\".\nTo adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n24/01/10 10:06:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n","output_type":"stream"}]},{"cell_type":"code","source":"prod_data_path = f\"{cache_dir}/prod_data\"\ntest_spark_dataset = spark.createDataFrame(xsum_dataset[\"test\"].to_pandas())\n\n##################\n#### For Demo ####\n##################\ntest_spark_dataset.write.mode(\"overwrite\").save(prod_data_path)\n##################\n\n# Spark Connector \n#test_spark_dataset.write.format(\"delta\").mode(\"overwrite\").save(prod_data_path)","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:06:31.403304Z","iopub.execute_input":"2024-01-10T10:06:31.403694Z","iopub.status.idle":"2024-01-10T10:06:39.232441Z","shell.execute_reply.started":"2024-01-10T10:06:31.403655Z","shell.execute_reply":"2024-01-10T10:06:39.229846Z"},"trusted":true},"execution_count":7,"outputs":[{"name":"stderr","text":"24/01/10 10:06:36 WARN TaskSetManager: Stage 0 contains a task of very large size (7188 KiB). The maximum recommended task size is 1000 KiB.\n                                                                                \r","output_type":"stream"}]},{"cell_type":"markdown","source":"# <b>2 <span style='color:#78D118'>|</span> Develop an LLM pipeline</b>","metadata":{}},{"cell_type":"markdown","source":"Create a Hugging Face pipeline","metadata":{}},{"cell_type":"code","source":"from transformers import pipeline\n\n# Later, we plan to log all of these parameters to MLflow.\n# Storing them as variables here will help with that.\nhf_model_name = \"t5-small\"\nmin_length = 20\nmax_length = 40\ntruncation = True\ndo_sample = True\n\nsummarizer = pipeline(\n    task=\"summarization\",\n    model=hf_model_name,\n    min_length=min_length,\n    max_length=max_length,\n    truncation=truncation,\n    do_sample=do_sample,\n    model_kwargs={\"cache_dir\": cache_dir},\n)  # Note: We specify cache_dir to use pre-cached models.\n","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:06:39.23364Z","iopub.execute_input":"2024-01-10T10:06:39.234021Z","iopub.status.idle":"2024-01-10T10:06:48.611103Z","shell.execute_reply.started":"2024-01-10T10:06:39.233983Z","shell.execute_reply":"2024-01-10T10:06:48.610081Z"},"trusted":true},"execution_count":8,"outputs":[{"output_type":"display_data","data":{"text/plain":"config.json:   0%|          | 0.00/1.21k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"f9870f6a290646b3a7d146e46ec69cdc"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"config.json:   0%|          | 0.00/1.21k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"99c32c92ff45442aa12b5022422ad0ea"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"model.safetensors:   0%|          | 0.00/242M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"e594b971d2c4486bbae1062fac1d7845"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"generation_config.json:   0%|          | 0.00/147 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"15eb29fd1bae44a4b05ecf17168f28d4"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"tokenizer_config.json:   0%|          | 0.00/2.32k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"00749b6c752049b3a65a1d69c5ef68dd"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"spiece.model:   0%|          | 0.00/792k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"172a91a8ed094f5280faa4153bccaf9e"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"tokenizer.json:   0%|          | 0.00/1.39M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"28084e654154407b9130a75045e4b1f0"}},"metadata":{}}]},{"cell_type":"markdown","source":"We can now examine the `summarizer` pipeline summarizing a document from the `xsum` dataset.","metadata":{}},{"cell_type":"code","source":"doc0 = xsum_sample[\"document\"][0]\nprint(f\"Summary: {summarizer(doc0)[0]['summary_text']}\")\nprint(\"===============================================\")\nprint(f\"Original Document: {doc0}\")","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:06:48.612258Z","iopub.execute_input":"2024-01-10T10:06:48.612553Z","iopub.status.idle":"2024-01-10T10:06:50.700969Z","shell.execute_reply.started":"2024-01-10T10:06:48.612529Z","shell.execute_reply":"2024-01-10T10:06:50.699525Z"},"trusted":true},"execution_count":9,"outputs":[{"name":"stdout","text":"Summary: the full cost of damage in Newton Stewart is still being assessed . the water breached a retaining wall, flooding many commercial properties . a flood alert remains in place across\n===============================================\nOriginal Document: The full cost of damage in Newton Stewart, one of the areas worst affected, is still being assessed.\nRepair work is ongoing in Hawick and many roads in Peeblesshire remain badly affected by standing water.\nTrains on the west coast mainline face disruption due to damage at the Lamington Viaduct.\nMany businesses and householders were affected by flooding in Newton Stewart after the River Cree overflowed into the town.\nFirst Minister Nicola Sturgeon visited the area to inspect the damage.\nThe waters breached a retaining wall, flooding many commercial properties on Victoria Street - the main shopping thoroughfare.\nJeanette Tate, who owns the Cinnamon Cafe which was badly affected, said she could not fault the multi-agency response once the flood hit.\nHowever, she said more preventative work could have been carried out to ensure the retaining wall did not fail.\n\"It is difficult but I do think there is so much publicity for Dumfries and the Nith - and I totally appreciate that - but it is almost like we're neglected or forgotten,\" she said.\n\"That may not be true but it is perhaps my perspective over the last few days.\n\"Why were you not ready to help us a bit more when the warning and the alarm alerts had gone out?\"\nMeanwhile, a flood alert remains in place across the Borders because of the constant rain.\nPeebles was badly hit by problems, sparking calls to introduce more defences in the area.\nScottish Borders Council has put a list on its website of the roads worst affected and drivers have been urged not to ignore closure signs.\nThe Labour Party's deputy Scottish leader Alex Rowley was in Hawick on Monday to see the situation first hand.\nHe said it was important to get the flood protection plan right but backed calls to speed up the process.\n\"I was quite taken aback by the amount of damage that has been done,\" he said.\n\"Obviously it is heart-breaking for people who have been forced out of their homes and the impact on businesses.\"\nHe said it was important that \"immediate steps\" were taken to protect the areas most vulnerable and a clear timetable put in place for flood prevention plans.\nHave you been affected by flooding in Dumfries and Galloway or the Borders? Tell us about your experience of the situation and how it was handled. Email us on selkirk.news@bbc.co.uk or dumfries@bbc.co.uk.\n","output_type":"stream"}]},{"cell_type":"markdown","source":"# <b>3 <span style='color:#78D118'>|</span> Track LLM development with MLflow</b>","metadata":{}},{"cell_type":"markdown","source":"[MLflow](https://mlflow.org/) has a Tracking component that helps you to track exactly how models or pipelines are produced during development.  Although we are not fitting (tuning or training) a model here, we can still make use of tracking to:\n\n* Track example queries and responses to the LLM pipeline, for later review or analysis\n* Store the model as an [MLflow Model flavor](https://mlflow.org/docs/latest/models.html#built-in-model-flavors), thus packaging it for simpler deployment\n","metadata":{}},{"cell_type":"markdown","source":"Apply to a batch of articles","metadata":{}},{"cell_type":"code","source":"# Apply to a batch of articles\nimport pandas as pd\n\nresults = summarizer(xsum_sample[\"document\"])\ndisplay(pd.DataFrame(results, columns=[\"summary_text\"]).head())","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:06:50.706393Z","iopub.execute_input":"2024-01-10T10:06:50.707136Z","iopub.status.idle":"2024-01-10T10:07:06.70315Z","shell.execute_reply.started":"2024-01-10T10:06:50.707093Z","shell.execute_reply":"2024-01-10T10:07:06.702091Z"},"trusted":true},"execution_count":10,"outputs":[{"output_type":"display_data","data":{"text/plain":"                                                                                                                                                                                    summary_text\n0             the full cost of damage in Newton Stewart is still being assessed . many roads in peeblesshire remain badly affected by standing water . a flood alert remains in place across the\n1       a fire alarm went off at the Holiday Inn in Hope Street on Saturday . guests were asked to leave the hotel as they gathered outside . both groups have organised replacement coaches and\n2                                                     Ferrari will start third ahead of team-mate Kimi Raikkonen . stewards only handed Hamilton a reprimand after governing body said \"no clear\n3                                       the 67-year-old is accused of committing the offences between March 1972 and October 1989 . he denies all the charges, including two counts of indecency\n4  a man receiving treatment at the clinic threatened to shoot himself and others . he was evacuated from the hospital . the incident comes amid tension in Istanbul following several attacks .","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>summary_text</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>the full cost of damage in Newton Stewart is still being assessed . many roads in peeblesshire remain badly affected by standing water . a flood alert remains in place across the</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>a fire alarm went off at the Holiday Inn in Hope Street on Saturday . guests were asked to leave the hotel as they gathered outside . both groups have organised replacement coaches and</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Ferrari will start third ahead of team-mate Kimi Raikkonen . stewards only handed Hamilton a reprimand after governing body said \"no clear</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>the 67-year-old is accused of committing the offences between March 1972 and October 1989 . he denies all the charges, including two counts of indecency</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>a man receiving treatment at the clinic threatened to shoot himself and others . he was evacuated from the hospital . the incident comes amid tension in Istanbul following several attacks .</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"[MLflow Tracking](https://mlflow.org/docs/latest/tracking.html) is organized hierarchically as follows:\n* **An [experiment](https://mlflow.org/docs/latest/tracking.html#organizing-runs-in-experiments)** generally corresponds to the creation of 1 primary model or pipeline.  In our case, this is our LLM pipeline.  It contains some number of *runs*.\n\n    * **A [run](https://mlflow.org/docs/latest/tracking.html#organizing-runs-in-experiments)** generally corresponds to the creation of 1 sub-model, such as 1 trial during hyperparameter tuning in traditional ML.  In our case, executing this notebook once will only create 1 run, but a second execution of the notebook will create a second run.  This version tracking can be useful during iterative development.  Each run contains some number of logged parameters, metrics, tags, models, artifacts, and other metadata.\n        * **A [parameter](https://mlflow.org/docs/latest/tracking.html#concepts)** is an input to the model or pipeline, such as a regularization parameter in traditional ML or `max_length` for our LLM pipeline.\n        * **A [metric](https://mlflow.org/docs/latest/tracking.html#concepts)** is an output of evaluation, such as accuracy or loss.\n        * **An [artifact](https://mlflow.org/docs/latest/tracking.html#concepts)** is an arbitrary file stored alongside a run's metadata, such as the serialized model itself.\n        * **A [flavor](https://mlflow.org/docs/latest/models.html#storage-format)** is an MLflow format for serializing models.  This format uses the underlying ML library's format (such as PyTorch, TensorFlow, Hugging Face, or your custom format), plus metadata.\n\n\nMLflow has an API for tracking queries and predictions [`mlflow.llm.log_predictions()`](https://mlflow.org/docs/latest/python_api/mlflow.llm.html), which we will use below.  Note that, as of MLflow 2.3.1 (Apr 28, 2023), this API is Experimental, so it may change in later releases.  See the [LLM Tracking page](https://mlflow.org/docs/latest/llm-tracking.html) for more information.\n\n***Tip***: We wrap our model development workflow with a call to `with mlflow.start_run():`.  This context manager syntax starts and ends the MLflow run explicitly, which is a best practice for code which may be moved to production.  See the [API doc](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.start_run) for more information.\n","metadata":{}},{"cell_type":"code","source":"import mlflow\n\n# Tell MLflow Tracking to use this explicit experiment path,\n# which is located on the left hand sidebar under Machine Learning -> Experiments \nmlflow.set_experiment(f\"/Users/{user_path}/experiment\")\n\nwith mlflow.start_run():\n    # LOG PARAMS\n    mlflow.log_params(\n        {\n            \"hf_model_name\": hf_model_name,\n            \"min_length\": min_length,\n            \"max_length\": max_length,\n            \"truncation\": truncation,\n            \"do_sample\": do_sample,\n        }\n    )\n\n    # --------------------------------\n    # LOG INPUTS (QUERIES) AND OUTPUTS\n    # Logged `inputs` are expected to be a list of str, or a list of str->str dicts.\n    results_list = [r[\"summary_text\"] for r in results]\n\n    # Our LLM pipeline does not have prompts separate from inputs, so we do not log any prompts.\n    mlflow.llm.log_predictions(\n        inputs=xsum_sample[\"document\"],\n        outputs=results_list,\n        prompts=[\"\" for _ in results_list],\n    )\n\n    # ---------\n    # LOG MODEL\n    # We next log our LLM pipeline as an MLflow model.\n    # This packages the model with useful metadata, such as the library versions used to create it.\n    # This metadata makes it much easier to deploy the model downstream.\n    # Under the hood, the model format is simply the ML library's native format (Hugging Face for us), plus metadata.\n\n    # It is valuable to log a \"signature\" with the model telling MLflow the input and output schema for the model.\n    signature = mlflow.models.infer_signature(\n        xsum_sample[\"document\"][0],\n        mlflow.transformers.generate_signature_output(\n            summarizer, xsum_sample[\"document\"][0]\n        ),\n    )\n    print(f\"Signature:\\n{signature}\\n\")\n\n    # For mlflow.transformers, if there are inference-time configurations,\n    # those need to be saved specially in the log_model call (below).\n    # This ensures that the pipeline will use these same configurations when re-loaded.\n    inference_config = {\n        \"min_length\": min_length,\n        \"max_length\": max_length,\n        \"truncation\": truncation,\n        \"do_sample\": do_sample,\n    }\n\n    # Logging a model returns a handle `model_info` to the model metadata in the tracking server.\n    # This `model_info` will be useful later in the notebook to retrieve the logged model.\n    model_info = mlflow.transformers.log_model(\n        transformers_model=summarizer,\n        artifact_path=\"summarizer\",\n        task=\"summarization\",\n        inference_config=inference_config,\n        signature=signature,\n        input_example=\"This is an example of a long news article which this pipeline can summarize for you.\",\n    )","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:06.704811Z","iopub.execute_input":"2024-01-10T10:07:06.70544Z","iopub.status.idle":"2024-01-10T10:07:21.06796Z","shell.execute_reply.started":"2024-01-10T10:07:06.705393Z","shell.execute_reply":"2024-01-10T10:07:21.067085Z"},"trusted":true},"execution_count":11,"outputs":[{"name":"stderr","text":"2024/01/10 10:07:07 INFO mlflow.tracking.fluent: Experiment with name '/Users/user/experiment' does not exist. Creating a new experiment.\n/tmp/ipykernel_42/3393809541.py:25: FutureWarning: ``mlflow.tracking.llm_utils.log_predictions`` is deprecated since 2.8.1. This method will be removed in MLflow 2.9.0. Use ``mlflow.log_table`` instead.\n  mlflow.llm.log_predictions(\n2024/01/10 10:07:07 INFO mlflow.tracking.llm_utils: Creating a new llm_predictions.csv for run 2ef13404ff6946bca51c7d43177dc984.\n","output_type":"stream"},{"name":"stdout","text":"Signature:\ninputs: \n  [string]\noutputs: \n  [string]\nparams: \n  None\n\n\n","output_type":"stream"},{"name":"stderr","text":"/tmp/ipykernel_42/3393809541.py:59: FutureWarning: The 'transformers' MLflow Models integration is known to be compatible with the following package version ranges: ``4.25.1`` -  ``4.35.2``. MLflow Models integrations with transformers may not succeed when used with package versions outside of this range.\n  model_info = mlflow.transformers.log_model(\n/opt/conda/lib/python3.10/site-packages/mlflow/models/model.py:619: FutureWarning: The 'transformers' MLflow Models integration is known to be compatible with the following package version ranges: ``4.25.1`` -  ``4.35.2``. MLflow Models integrations with transformers may not succeed when used with package versions outside of this range.\n  flavor.save_model(path=local_path, mlflow_model=mlflow_model, **kwargs)\n2024/01/10 10:07:09 WARNING mlflow.transformers: Indicating `inference_config` is deprecated and will be removed in a future version of MLflow. Use `model_config` instead.\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"README.md:   0%|          | 0.00/8.47k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"e34042d0e67b4fa48b1f39af0da0d484"}},"metadata":{}},{"name":"stderr","text":"2024/01/10 10:07:20 WARNING mlflow.utils.environment: Encountered an unexpected error while inferring pip requirements (model URI: /tmp/tmpj_v13b90/model, flavor: transformers), fall back to return ['transformers==4.36.0', 'torch==2.0.0', 'torchvision==0.15.1', 'accelerate==0.25.0']. Set logging level to DEBUG to see the full traceback.\n/opt/conda/lib/python3.10/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.\n  warnings.warn(\"Setuptools is replacing distutils.\")\n","output_type":"stream"}]},{"cell_type":"markdown","source":"# <b>4 <span style='color:#78D118'>|</span> Query the MLflow Tracking server</b>\n\n**MLflow Tracking API**: We briefly show how to query the logged model and metadata in the MLflow Tracking server, by loading the logged model.  See the [MLflow API](https://mlflow.org/docs/latest/python_api/mlflow.html) for more information about programmatic access.\n\n**MLflow Tracking UI**: You can also use the UI.  In the right-hand sidebar, click the beaker icon to access the MLflow experiments run list, and then click through to access the Tracking server UI.  There, you can see the logged metadata and model.  Note in particular that our LLM inputs and outputs have been logged as a CSV file under model artifacts.\n\nGIF of MLflow UI:\n![GIF of MLflow UI](https://files.training.databricks.com/images/llm/llmops.gif)\n","metadata":{}},{"cell_type":"markdown","source":"Now, we can load the pipeline back from MLflow as a [pyfunc](https://mlflow.org/docs/latest/python_api/mlflow.pyfunc.html) and use the `.predict()` method to summarize an example document.","metadata":{}},{"cell_type":"code","source":"loaded_summarizer = mlflow.pyfunc.load_model(model_uri=model_info.model_uri)\nloaded_summarizer.predict(xsum_sample[\"document\"][0])\n","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:21.069203Z","iopub.execute_input":"2024-01-10T10:07:21.069525Z","iopub.status.idle":"2024-01-10T10:07:22.552907Z","shell.execute_reply.started":"2024-01-10T10:07:21.069497Z","shell.execute_reply":"2024-01-10T10:07:22.551922Z"},"trusted":true},"execution_count":12,"outputs":[{"name":"stderr","text":"2024/01/10 10:07:21 WARNING mlflow.transformers: Inference config stored in file ``inference_config.txt`` is deprecated. New logged models will store the model configuration in the ``pyfunc`` flavor configuration.\n","output_type":"stream"},{"execution_count":12,"output_type":"execute_result","data":{"text/plain":"['many businesses and householders were affected by flooding in Newton Stewart . the water breached a retaining wall, flooding many commercial properties . a flood alert remains in place across']"},"metadata":{}}]},{"cell_type":"markdown","source":"The `.predict()` method can handle more than one document at a time, below we pass in all the data from `xsum_sample`.","metadata":{}},{"cell_type":"code","source":"results = loaded_summarizer.predict(xsum_sample.to_pandas()[\"document\"])\ndisplay(pd.DataFrame(results, columns=[\"generated_summary\"]).head())","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:22.554323Z","iopub.execute_input":"2024-01-10T10:07:22.554997Z","iopub.status.idle":"2024-01-10T10:07:26.771322Z","shell.execute_reply.started":"2024-01-10T10:07:22.55496Z","shell.execute_reply":"2024-01-10T10:07:26.770257Z"},"trusted":true},"execution_count":13,"outputs":[{"output_type":"display_data","data":{"text/plain":"                                                                                                                                                                                 generated_summary\n0  many businesses and householders were affected by flooding in Newton Stewart . the water breached a retaining wall, flooding many commercial properties . a flood alert remains in place across\n1                          fire alarm went off at the Holiday Inn in Hope Street on Saturday . guests were asked to leave the hotel . one of the tour groups is from germany, the other from china\n2                                                stewards only handed reprimand after governing body says \"no clear instruction was given on where he should park\" Stoffel Vandoorne out-qualified\n3                                         the 67-year-old is accused of committing the offences between March 1972 and October 1989 . he denies all the charges, including two counts of indecency\n4                         a man receiving psychiatric treatment at the clinic threatened to shoot himself and others . he was evacuated from the hospital . earlier reports that the armed man had","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>generated_summary</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>many businesses and householders were affected by flooding in Newton Stewart . the water breached a retaining wall, flooding many commercial properties . a flood alert remains in place across</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>fire alarm went off at the Holiday Inn in Hope Street on Saturday . guests were asked to leave the hotel . one of the tour groups is from germany, the other from china</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>stewards only handed reprimand after governing body says \"no clear instruction was given on where he should park\" Stoffel Vandoorne out-qualified</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>the 67-year-old is accused of committing the offences between March 1972 and October 1989 . he denies all the charges, including two counts of indecency</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>a man receiving psychiatric treatment at the clinic threatened to shoot himself and others . he was evacuated from the hospital . earlier reports that the armed man had</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"We are now ready to move to the staging step of deployment.  To get started, we will register the model in the MLflow Model Registry (more info below).","metadata":{}},{"cell_type":"code","source":"# Define the name for the model in the Model Registry.\n# We filter out some special characters which cannot be used in model names.\nmodel_name = f\"summarizer - {user_path}\"\nmodel_name = model_name.replace(\"/\", \"_\").replace(\".\", \"_\").replace(\":\", \"_\")\nprint(model_name)","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:26.772486Z","iopub.execute_input":"2024-01-10T10:07:26.772809Z","iopub.status.idle":"2024-01-10T10:07:26.778255Z","shell.execute_reply.started":"2024-01-10T10:07:26.772771Z","shell.execute_reply":"2024-01-10T10:07:26.777324Z"},"trusted":true},"execution_count":14,"outputs":[{"name":"stdout","text":"summarizer - user\n","output_type":"stream"}]},{"cell_type":"code","source":"# Register a new model under the given name, or a new model version if the name exists already.\nmlflow.register_model(model_uri=model_info.model_uri, name=model_name)","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:26.779505Z","iopub.execute_input":"2024-01-10T10:07:26.779857Z","iopub.status.idle":"2024-01-10T10:07:26.863273Z","shell.execute_reply.started":"2024-01-10T10:07:26.779825Z","shell.execute_reply":"2024-01-10T10:07:26.862407Z"},"trusted":true},"execution_count":15,"outputs":[{"name":"stderr","text":"Successfully registered model 'summarizer - user'.\nCreated version '1' of model 'summarizer - user'.\n","output_type":"stream"},{"execution_count":15,"output_type":"execute_result","data":{"text/plain":"<ModelVersion: aliases=[], creation_timestamp=1704881246855, current_stage='None', description=None, last_updated_timestamp=1704881246855, name='summarizer - user', run_id='2ef13404ff6946bca51c7d43177dc984', run_link=None, source='file:///kaggle/working/mlruns/548139300933260725/2ef13404ff6946bca51c7d43177dc984/artifacts/summarizer', status='READY', status_message=None, tags={}, user_id=None, version=1>"},"metadata":{}}]},{"cell_type":"markdown","source":"# <b>5 <span style='color:#78D118'>|</span> Test the LLM pipeline</b>\n\n","metadata":{}},{"cell_type":"markdown","source":"During the Staging step of development, our goal is to move code and/or models from Development to Production.  In order to do so, we must test the code and/or models to make sure they are ready for Production.\n\nWe track our progress here using the [MLflow Model Registry](https://mlflow.org/docs/latest/model-registry.html).  This metadata and model store organizes models as follows:\n* **A registered model** is a named model in the registry, in our case corresponding to our summarization model.  It may have multiple *versions*.\n    * **A model version** is an instance of a given model.  As you update your model, you will create new versions.  Each version is designated as being in a particular *stage* of deployment.\n    * **A stage** is a stage of deployment: `None` (development), `Staging`, `Production`, or `Archived`.\n\nThe model we registered above starts with 1 version in stage `None` (development).\n\nIn the workflow below, we will programmatically transition the model from development to staging to production.  For more information on the Model Registry API, see the [Model Registry docs](https://mlflow.org/docs/latest/model-registry.html).  Alternatively, you can edit the registry and make model stage transitions via the UI.  To access the UI, click the Experiments menu option in the left-hand sidebar, and search for your model name.\n","metadata":{}},{"cell_type":"code","source":"from mlflow import MlflowClient\n\nclient = MlflowClient()\n\nclient.search_registered_models(filter_string=f\"name = '{model_name}'\")\n","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:26.864407Z","iopub.execute_input":"2024-01-10T10:07:26.86468Z","iopub.status.idle":"2024-01-10T10:07:26.878606Z","shell.execute_reply.started":"2024-01-10T10:07:26.864655Z","shell.execute_reply":"2024-01-10T10:07:26.87789Z"},"trusted":true},"execution_count":16,"outputs":[{"execution_count":16,"output_type":"execute_result","data":{"text/plain":"[<RegisteredModel: aliases={}, creation_timestamp=1704881246852, description=None, last_updated_timestamp=1704881246855, latest_versions=[<ModelVersion: aliases=[], creation_timestamp=1704881246855, current_stage='None', description=None, last_updated_timestamp=1704881246855, name='summarizer - user', run_id='2ef13404ff6946bca51c7d43177dc984', run_link=None, source='file:///kaggle/working/mlruns/548139300933260725/2ef13404ff6946bca51c7d43177dc984/artifacts/summarizer', status='READY', status_message=None, tags={}, user_id=None, version=1>], name='summarizer - user', tags={}>]"},"metadata":{}}]},{"cell_type":"markdown","source":"In the metadata above, you can see that the model is currently in stage `None` (development).  In this workflow, we will run manual tests, but it would be reasonable to run both automated evaluation and human evaluation in practice.  Once tests pass, we will promote the model to stage `Production` to mark it ready for user-facing applications.\n\n*Model URIs*: Below, we use model URIs to tell MLflow which model and version we are referring to.  Two common URI patterns for the MLflow Model Registry are:\n* `f\"models:/{model_name}/{model_version}\"` to refer to a specific model version by number\n* `f\"models:/{model_name}/{model_stage}\"` to refer to the latest model version in a given stage\n","metadata":{}},{"cell_type":"code","source":"model_version = 1\ndev_model = mlflow.pyfunc.load_model(model_uri=f\"models:/{model_name}/{model_version}\")\ndev_model","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:26.879645Z","iopub.execute_input":"2024-01-10T10:07:26.879932Z","iopub.status.idle":"2024-01-10T10:07:27.577418Z","shell.execute_reply.started":"2024-01-10T10:07:26.879907Z","shell.execute_reply":"2024-01-10T10:07:27.576397Z"},"trusted":true},"execution_count":17,"outputs":[{"name":"stderr","text":"2024/01/10 10:07:26 WARNING mlflow.transformers: Inference config stored in file ``inference_config.txt`` is deprecated. New logged models will store the model configuration in the ``pyfunc`` flavor configuration.\n","output_type":"stream"},{"execution_count":17,"output_type":"execute_result","data":{"text/plain":"mlflow.pyfunc.loaded_model:\n  artifact_path: summarizer\n  flavor: mlflow.transformers\n  run_id: 2ef13404ff6946bca51c7d43177dc984"},"metadata":{}}]},{"cell_type":"markdown","source":"*Note about model dependencies*:\nWhen you load the model via MLflow above, you may see warnings about the Python environment.  It is very important to ensure that the environments for development, staging, and production match.\n* For this demo notebook, everything is done within the same notebook environment, so we do not need to worry about libraries and versions.  However, in the Production section below, we demonstrate how to pass the `env_manager` argument to the method for loading the saved MLflow model, which tells MLflow what tooling to use to recreate the environment.\n* To create a genuine production job, make sure to install the needed libraries.  MLflow saves these libraries and versions alongside the logged model; see the [MLflow docs on model storage](https://mlflow.org/docs/latest/models.html#storage-format) for more information.  While using Databricks for this course, you can also generate an example inference notebook which includes code for setting up the environment; see [the model inference docs](https://docs.databricks.com/machine-learning/manage-model-lifecycle/index.html#use-model-for-inference) for batch or streaming inference for more information.\n","metadata":{}},{"cell_type":"markdown","source":"# <b>6 <span style='color:#78D118'>|</span> Transition to Staging</b>\n\nWe will move the model to stage `Staging` to indicate that we are actively testing it.","metadata":{}},{"cell_type":"code","source":"client.transition_model_version_stage(model_name, model_version, \"staging\")","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:27.578689Z","iopub.execute_input":"2024-01-10T10:07:27.58013Z","iopub.status.idle":"2024-01-10T10:07:27.593741Z","shell.execute_reply.started":"2024-01-10T10:07:27.5801Z","shell.execute_reply":"2024-01-10T10:07:27.592903Z"},"trusted":true},"execution_count":18,"outputs":[{"name":"stderr","text":"/tmp/ipykernel_42/4144300577.py:1: FutureWarning: ``mlflow.tracking.client.MlflowClient.transition_model_version_stage`` is deprecated since 2.9.0. Model registry stages will be removed in a future major release. To learn more about the deprecation of model registry stages, see our migration guide here: https://mlflow.org/docs/2.9.2/model-registry.html#migrating-from-stages\n  client.transition_model_version_stage(model_name, model_version, \"staging\")\n","output_type":"stream"},{"execution_count":18,"output_type":"execute_result","data":{"text/plain":"<ModelVersion: aliases=[], creation_timestamp=1704881246855, current_stage='Staging', description=None, last_updated_timestamp=1704881247581, name='summarizer - user', run_id='2ef13404ff6946bca51c7d43177dc984', run_link=None, source='file:///kaggle/working/mlruns/548139300933260725/2ef13404ff6946bca51c7d43177dc984/artifacts/summarizer', status='READY', status_message=None, tags={}, user_id=None, version=1>"},"metadata":{}}]},{"cell_type":"markdown","source":"See the current_stage move to Staging","metadata":{}},{"cell_type":"code","source":"##################\n#### For Demo ####\n##################\nstaging_model = dev_model\n##################\n\n# An actual CI/CD workflow might load the `staging_model` programmatically.  For example:\n#   mlflow.pyfunc.load_model(model_uri=f\"models:/{model_name}/{Staging}\")\n# or\n#   mlflow.pyfunc.load_model(model_uri=f\"models:/{model_name}/{model_version}\")\n","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:27.595136Z","iopub.execute_input":"2024-01-10T10:07:27.595449Z","iopub.status.idle":"2024-01-10T10:07:27.600662Z","shell.execute_reply.started":"2024-01-10T10:07:27.595425Z","shell.execute_reply":"2024-01-10T10:07:27.599571Z"},"trusted":true},"execution_count":19,"outputs":[]},{"cell_type":"markdown","source":"We now \"test\" the model manually on sample data. Here, we simply print out results and compare them with the original data.  In a more realistic setting, we might use a set of human evaluators to decide whether the model outperformed the previous model or system.","metadata":{}},{"cell_type":"code","source":"results = staging_model.predict(xsum_sample.to_pandas()[\"document\"])\ndisplay(pd.DataFrame(results, columns=[\"generated_summary\"]).head())\n","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:27.602073Z","iopub.execute_input":"2024-01-10T10:07:27.602747Z","iopub.status.idle":"2024-01-10T10:07:31.600449Z","shell.execute_reply.started":"2024-01-10T10:07:27.602714Z","shell.execute_reply":"2024-01-10T10:07:31.599423Z"},"trusted":true},"execution_count":20,"outputs":[{"output_type":"display_data","data":{"text/plain":"                                                                                                                                                                                   generated_summary\n0                 the full cost of damage in Newton Stewart is still being assessed . many roads in peeblesshire remain badly affected by standing water . a flood alert remains in place across the\n1  fire alarm went off at the Holiday Inn in Hope Street on Saturday . guests were asked to leave the hotel as they gathered outside . both groups have organised replacement coaches and will begin\n2                                                           Sebastian Vettel will start third ahead of team-mate Kimi Raikkonen . stewards handed Hamilton a reprimand after governing body said \"no\n3                                           the 67-year-old is accused of committing the offences between March 1972 and October 1989 . he denies all the charges, including two counts of indecency\n4                                          a man receiving treatment at the clinic threatened to shoot himself and others . the incident comes amid tensions in Istanbul following several attacks .","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>generated_summary</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>the full cost of damage in Newton Stewart is still being assessed . many roads in peeblesshire remain badly affected by standing water . a flood alert remains in place across the</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>fire alarm went off at the Holiday Inn in Hope Street on Saturday . guests were asked to leave the hotel as they gathered outside . both groups have organised replacement coaches and will begin</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Sebastian Vettel will start third ahead of team-mate Kimi Raikkonen . stewards handed Hamilton a reprimand after governing body said \"no</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>the 67-year-old is accused of committing the offences between March 1972 and October 1989 . he denies all the charges, including two counts of indecency</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>a man receiving treatment at the clinic threatened to shoot himself and others . the incident comes amid tensions in Istanbul following several attacks .</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"# <b>7 <span style='color:#78D118'>|</span> Transition to Production</b>\n","metadata":{}},{"cell_type":"markdown","source":"Let's transition the model to Production.","metadata":{}},{"cell_type":"code","source":"client.transition_model_version_stage(model_name, model_version, \"production\")","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:31.601907Z","iopub.execute_input":"2024-01-10T10:07:31.602761Z","iopub.status.idle":"2024-01-10T10:07:31.616926Z","shell.execute_reply.started":"2024-01-10T10:07:31.602724Z","shell.execute_reply":"2024-01-10T10:07:31.615968Z"},"trusted":true},"execution_count":21,"outputs":[{"name":"stderr","text":"/tmp/ipykernel_42/3157488612.py:1: FutureWarning: ``mlflow.tracking.client.MlflowClient.transition_model_version_stage`` is deprecated since 2.9.0. Model registry stages will be removed in a future major release. To learn more about the deprecation of model registry stages, see our migration guide here: https://mlflow.org/docs/2.9.2/model-registry.html#migrating-from-stages\n  client.transition_model_version_stage(model_name, model_version, \"production\")\n","output_type":"stream"},{"execution_count":21,"output_type":"execute_result","data":{"text/plain":"<ModelVersion: aliases=[], creation_timestamp=1704881246855, current_stage='Production', description=None, last_updated_timestamp=1704881251604, name='summarizer - user', run_id='2ef13404ff6946bca51c7d43177dc984', run_link=None, source='file:///kaggle/working/mlruns/548139300933260725/2ef13404ff6946bca51c7d43177dc984/artifacts/summarizer', status='READY', status_message=None, tags={}, user_id=None, version=1>"},"metadata":{}}]},{"cell_type":"markdown","source":"See the current_stage move to Production","metadata":{}},{"cell_type":"markdown","source":"### Create a production workflow for batch inference\n\nOnce the LLM pipeline is in Production, it may be used by one or more production jobs or serving endpoints.  Common deployment locations are:\n* Batch or streaming inference jobs\n* Model serving endpoints\n* Edge devices\n\nHere, we will show batch inference using Apache Spark DataFrames, with Delta Lake format.  Spark allows simple scale-out inference for high-throughput, low-cost jobs, and Delta allows us to append to and modify inference result tables with ACID transactions.  See the [Apache Spark page](https://spark.apache.org/) and the [Delta Lake page](https://delta.io/) more more information on these technologies.\n","metadata":{}},{"cell_type":"code","source":"# Load our data as a Spark DataFrame.\n# Recall that we saved this as Delta at the start of the notebook.\n# Also note that it has a ground-truth summary column.\n\n##################\n#### For Demo ####\n##################\nprod_data = spark.read.load(prod_data_path).limit(10)\n##################\n\n# Spark Connector \n# prod_data = spark.read.format(\"delta\").load(prod_data_path).limit(10)\n\ndisplay(prod_data)\n","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:31.618125Z","iopub.execute_input":"2024-01-10T10:07:31.61844Z","iopub.status.idle":"2024-01-10T10:07:31.951517Z","shell.execute_reply.started":"2024-01-10T10:07:31.618414Z","shell.execute_reply":"2024-01-10T10:07:31.950574Z"},"trusted":true},"execution_count":22,"outputs":[{"output_type":"display_data","data":{"text/plain":"DataFrame[document: string, summary: string, id: string]"},"metadata":{}}]},{"cell_type":"markdown","source":"Below, we load the model using `mlflow.pyfunc.spark_udf`.  This returns the model as a Spark User Defined Function which can be applied efficiently to big data.  *Note that the deployment code is library-agnostic: it never references that the model is a Hugging Face pipeline.*  This simplified deployment is possible because MLflow logs environment metadata and \"knows\" how to load the model and run it.","metadata":{}},{"cell_type":"code","source":"# MLflow lets you grab the latest model version in a given stage.  Here, we grab the latest Production version.\nprod_model_udf = mlflow.pyfunc.spark_udf(\n    spark,\n    model_uri=f\"models:/{model_name}/Production\",\n    env_manager=\"local\",\n    result_type=\"string\",\n)","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:31.952536Z","iopub.execute_input":"2024-01-10T10:07:31.952896Z","iopub.status.idle":"2024-01-10T10:07:32.856297Z","shell.execute_reply.started":"2024-01-10T10:07:31.952856Z","shell.execute_reply":"2024-01-10T10:07:32.855111Z"},"trusted":true},"execution_count":23,"outputs":[{"name":"stderr","text":"/opt/conda/lib/python3.10/site-packages/mlflow/store/artifact/utils/models.py:32: FutureWarning: ``mlflow.tracking.client.MlflowClient.get_latest_versions`` is deprecated since 2.9.0. Model registry stages will be removed in a future major release. To learn more about the deprecation of model registry stages, see our migration guide here: https://mlflow.org/docs/2.9.2/model-registry.html#migrating-from-stages\n  latest = client.get_latest_versions(name, None if stage is None else [stage])\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"Downloading artifacts:   0%|          | 0/16 [00:00<?, ?it/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"ff952cbcfde44405b8df599a681567ec"}},"metadata":{}},{"name":"stderr","text":"2024/01/10 10:07:32 INFO mlflow.store.artifact.artifact_repo: The progress bar can be disabled by setting the environment variable MLFLOW_ENABLE_ARTIFACTS_PROGRESS_BAR to false\n2024/01/10 10:07:32 WARNING mlflow.pyfunc: Calling `spark_udf()` with `env_manager=\"local\"` does not recreate the same environment that was used during training, which may lead to errors or inaccurate predictions. We recommend specifying `env_manager=\"conda\"`, which automatically recreates the environment that was used to train the model and performs inference in the recreated environment.\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"Downloading artifacts:   0%|          | 0/1 [00:00<?, ?it/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"57833b9dacd0447f94f3a2900214e979"}},"metadata":{}},{"name":"stderr","text":"2024/01/10 10:07:32 INFO mlflow.models.flavor_backend_registry: Selected backend for flavor 'python_function'\n","output_type":"stream"}]},{"cell_type":"code","source":"# Run inference by appending a new column to the DataFrame\n\nbatch_inference_results = prod_data.withColumn(\n    \"generated_summary\", prod_model_udf(\"document\")\n)\ndisplay(batch_inference_results)","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:32.860997Z","iopub.execute_input":"2024-01-10T10:07:32.861423Z","iopub.status.idle":"2024-01-10T10:07:33.040364Z","shell.execute_reply.started":"2024-01-10T10:07:32.861383Z","shell.execute_reply":"2024-01-10T10:07:33.039263Z"},"trusted":true},"execution_count":24,"outputs":[{"output_type":"display_data","data":{"text/plain":"DataFrame[document: string, summary: string, id: string, generated_summary: string]"},"metadata":{}}]},{"cell_type":"markdown","source":"We can now write out our inference results to another Delta table.  Here, we append the results to an existing table (and create the table if it does not exist).","metadata":{}},{"cell_type":"code","source":"inference_results_path = f\"{working_dir}/m6-inference-results\".replace(\n    \"/dbfs\", \"dbfs:\"\n)\n##################\n#### For Demo ####\n##################\nbatch_inference_results.write.mode(\"append\").save(\n    inference_results_path\n)\n##################\n\n# Spark Connector \n#batch_inference_results.write.format(\"delta\").mode(\"append\").save(\n#    inference_results_path\n#)","metadata":{"execution":{"iopub.status.busy":"2024-01-10T10:07:33.042184Z","iopub.execute_input":"2024-01-10T10:07:33.042913Z","iopub.status.idle":"2024-01-10T10:08:01.822198Z","shell.execute_reply.started":"2024-01-10T10:07:33.042876Z","shell.execute_reply":"2024-01-10T10:08:01.819492Z"},"trusted":true},"execution_count":25,"outputs":[{"name":"stderr","text":"/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3\n  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3\n  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3\n  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3\n  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n2024/01/10 10:07:41 WARNING mlflow.transformers: Inference config stored in file ``inference_config.txt`` is deprecated. New logged models will store the model configuration in the ``pyfunc`` flavor configuration.\n2024/01/10 10:07:42 WARNING mlflow.transformers: Inference config stored in file ``inference_config.txt`` is deprecated. New logged models will store the model configuration in the ``pyfunc`` flavor configuration.\n2024/01/10 10:07:42 WARNING mlflow.transformers: Inference config stored in file ``inference_config.txt`` is deprecated. New logged models will store the model configuration in the ``pyfunc`` flavor configuration.\n2024/01/10 10:07:42 WARNING mlflow.transformers: Inference config stored in file ``inference_config.txt`` is deprecated. New logged models will store the model configuration in the ``pyfunc`` flavor configuration.\nYour max_length is set to 40, but your input_length is only 39. Since this is a summarization task, where outputs shorter than the input are typically wanted, you might consider decreasing max_length manually, e.g. summarizer('...', max_length=19)\n/opt/conda/lib/python3.10/site-packages/mlflow/pyfunc/__init__.py:1481: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n  result = result.applymap(str)\n/opt/conda/lib/python3.10/site-packages/mlflow/pyfunc/__init__.py:1481: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n  result = result.applymap(str)\n/opt/conda/lib/python3.10/site-packages/mlflow/pyfunc/__init__.py:1481: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n  result = result.applymap(str)\n/opt/conda/lib/python3.10/site-packages/mlflow/pyfunc/__init__.py:1481: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n  result = result.applymap(str)\n                                                                                \r","output_type":"stream"}]},{"cell_type":"markdown","source":"To create a production job, we could for example take the new lines of code above, put them in a new notebook, and schedule it as an automated workflow.  MLflow can be integrated with essentially any deployment system, but for more information specific to this Databricks workspace, see the \"Use model for inference\" documentation for [AWS](https://docs.databricks.com/machine-learning/manage-model-lifecycle/index.html#use-model-for-inference), [Azure](https://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/#--use-model-for-inference), or [GCP](https://docs.gcp.databricks.com/machine-learning/manage-model-lifecycle/index.html#use-model-for-inference).\n\nWe did not cover model serving for real-time inference, but MLflow models can be deployed to any cloud or on-prem serving systems.  For more information, see the [open-source MLflow Model Registry docs](https://mlflow.org/docs/latest/model-registry.html) or the [Databricks Model Serving docs](https://docs.databricks.com/machine-learning/model-serving/index.html).\n\nFor other topics not covered, see [\"The Big Book of MLOps.\"](https://www.databricks.com/resources/ebook/the-big-book-of-mlops)\n\n\n# <b>8 <span style='color:#78D118'>|</span> Summary</b>\n\n\nWe have now walked through a full example of going from development to production.  Our LLM pipeline was very simple, but LLM Ops for a more complex workflow (such as fine-tuning a custom model) would be very similar.  You still follow the basic Ops steps of:\n* Development: Creating the pipeline or model, tracking the process in the MLflow Tracking server and saving the final pipeline or model.\n* Staging: Registering a new model or version in the MLflow Model Registry, testing it, and promoting it through Staging to Production.\n* Production: Creating an inference job, or creating a model serving endpoint.\n","metadata":{}}]}