openai-fine-tuning/OpenAI_fine-tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-Tune a GPT-4o-Mini Model to Answer Questions about Hurricanes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Curation and Preparation\n",
    "\n",
    "To demonstrate the fine-tuning process, we'll be working with a real-world dataset—recent updates to hurricane data from Wikipedia. The steps are as follows:\n",
    "\n",
    "1. Collect Data: Gather the latest revisions from selected Wikipedia pages.\n",
    "2. Generate Q&A Pairs: Turn the raw hurricane data into a useful set of question-and-answer pairs.\n",
    "3. Create Fine-Tuning Dataset: Format the dataset to fit OpenAI's requirements for fine-tuning.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pull Wikipedia Revisions for Hurricane Topics (after a certain date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of relevant topics\n",
    "topics = [\n",
    "    \"List_of_United_States_hurricanes\",\n",
    "    \"2024_Atlantic_hurricane_season\",\n",
    "    \"Hurricane_Milton\",\n",
    "    \"Hurricane_Beryl\",\n",
    "    \"Hurricane_Francine\",\n",
    "    \"Hurricane_Helene\",\n",
    "    \"Hurricane_Isaac\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching revisions for List_of_United_States_hurricanes starting from 2023-09-01T00:00:00Z...\n",
      "Fetching revisions for 2024_Atlantic_hurricane_season starting from 2023-09-01T00:00:00Z...\n",
      "Fetching revisions for Hurricane_Milton starting from 2023-09-01T00:00:00Z...\n",
      "Fetching revisions for Hurricane_Beryl starting from 2023-09-01T00:00:00Z...\n",
      "Fetching revisions for Hurricane_Francine starting from 2023-09-01T00:00:00Z...\n",
      "Fetching revisions for Hurricane_Helene starting from 2023-09-01T00:00:00Z...\n",
      "Fetching revisions for Hurricane_Isaac starting from 2023-09-01T00:00:00Z...\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import difflib\n",
    "import re\n",
    "\n",
    "def get_wikipedia_revisions(article_title, start_date):\n",
    "    \"\"\"Fetches Wikipedia revisions after a certain date for a given article and collects the data into a list of dictionaries.\"\"\"\n",
    "    url = \"https://en.wikipedia.org/w/api.php\"\n",
    "    params = {\n",
    "        \"action\": \"query\",\n",
    "        \"prop\": \"revisions\",\n",
    "        \"titles\": article_title,\n",
    "        \"rvstart\": start_date,\n",
    "        \"rvdir\": \"newer\",  # Fetch revisions newer than start_date\n",
    "        \"rvlimit\": \"500\",\n",
    "        \"rvprop\": \"timestamp|user|comment|content\",\n",
    "        \"format\": \"json\"\n",
    "    }\n",
    "    \n",
    "    dataset = []  # List to hold the dictionaries\n",
    "    continue_token = True  # To handle pagination\n",
    "    \n",
    "    while continue_token:\n",
    "        response = requests.get(url, params=params)\n",
    "        data = response.json()\n",
    "\n",
    "        if 'query' in data and 'pages' in data['query']:\n",
    "            pages = data['query']['pages']\n",
    "            for page_id in pages:\n",
    "                revisions = pages[page_id].get('revisions', [])\n",
    "                if revisions:\n",
    "                    previous_content = \"\"  # Initialize variable to hold previous revision content\n",
    "                    for rev in revisions:\n",
    "                        current_content = rev.get('*', '')  # Get the full revision content\n",
    "                        timestamp = rev.get('timestamp', 'No timestamp')\n",
    "                        user = rev.get('user', 'Anonymous')\n",
    "                        comment = rev.get('comment', 'No comment')\n",
    "\n",
    "                        # If there is a previous revision, calculate the difference (new content added)\n",
    "                        if previous_content:\n",
    "                            diff = list(difflib.unified_diff(previous_content.splitlines(), current_content.splitlines()))\n",
    "                            new_content = \"\\n\".join(line[1:] for line in diff if line.startswith('+') and not line.startswith('+++'))\n",
    "\n",
    "                            # Extract citation links (URLs and <ref> tags)\n",
    "                            citations = re.findall(r'<ref.*?>.*?</ref>|https?://\\S+', new_content)\n",
    "\n",
    "                            # Create a dictionary to hold the relevant information\n",
    "                            revision_data = {\n",
    "                                \"article_title\": article_title,\n",
    "                                \"timestamp\": timestamp,\n",
    "                                \"user\": user,\n",
    "                                \"comment\": comment,\n",
    "                                \"new_content\": new_content,\n",
    "                                # \"citations\": citations\n",
    "                            }\n",
    "\n",
    "                            # Append the dictionary to the dataset list\n",
    "                            dataset.append(revision_data)\n",
    "\n",
    "                        # Update previous content\n",
    "                        previous_content = current_content\n",
    "                else:\n",
    "                    print(f\"No revisions found for {article_title} after {start_date}.\")\n",
    "\n",
    "        else:\n",
    "            print(f\"Error fetching data for {article_title}.\")\n",
    "\n",
    "        # Check if pagination is needed (continue parameter)\n",
    "        if 'continue' in data:\n",
    "            continue_token = data['continue'].get('rvcontinue', False)\n",
    "            params['rvcontinue'] = continue_token\n",
    "        else:\n",
    "            continue_token = False  # End loop if no more pages\n",
    "    \n",
    "    return dataset  # Return the dataset\n",
    "\n",
    "\n",
    "def fetch_revisions_for_topics(topics, start_date):\n",
    "    \"\"\"Fetches revisions for all topics after a certain date and returns a combined dataset.\"\"\"\n",
    "    full_dataset = []  # List to hold data for all topics\n",
    "    for topic in topics:\n",
    "        try:\n",
    "            print(f\"Fetching revisions for {topic} starting from {start_date}...\")\n",
    "            topic_data = get_wikipedia_revisions(topic, start_date)\n",
    "            full_dataset.extend(topic_data)  # Append the data for each topic to the full dataset\n",
    "        except Exception as e:\n",
    "            print(f\"Error fetching revisions for {topic}: {str(e)}\")\n",
    "    \n",
    "    return full_dataset  # Return the full dataset\n",
    "\n",
    "\n",
    "# Specify the start date (ISO 8601 format)\n",
    "start_date = \"2023-09-01T00:00:00Z\"\n",
    "\n",
    "# Fetch the latest revisions for all topics and store them in a dataset\n",
    "dataset = fetch_revisions_for_topics(topics, start_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of examples in dataset:  9489\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of examples in dataset: \", len(dataset))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-09-07T18:42:52Z', 'user': '2603:6011:B200:AB2:4053:9549:C77A:8B0F', 'comment': '/* Louisiana */', 'new_content': '| bgcolor=#{{storm colour|3}}|{{center|Category&nbsp;5 hurricane}}'}\n",
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-09-07T18:43:59Z', 'user': '2603:6011:B200:AB2:4053:9549:C77A:8B0F', 'comment': '/* Louisiana */', 'new_content': '| bgcolor=#{{storm colour|5}}|{{center|Category&nbsp;5 hurricane}}'}\n",
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-09-07T23:47:34Z', 'user': 'Jay Jor', 'comment': '/* Louisiana */ While Katrina was Category 5 in the Gulf of Mexico, within the state of Louisiana it was Category 3.', 'new_content': '| [[Hurricane Katrina|Katrina]]<ref>{{cite web |title=Hurricane Katrina, New Orleans, Louisiana, USA |url=https://eros.usgs.gov/media-gallery/earthshot/hurricane-katrina-new-orleans-louisiana-usa |website=USGS |publisher=U.S. Dept. of the Interior |access-date=7 September 2023}}</ref>\\n| bgcolor=#{{storm colour|3}}|{{center|Category&nbsp;3 hurricane}}'}\n",
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-09-13T01:34:29Z', 'user': 'HurricaneKappa', 'comment': '/* Virginia */ it just said \"Hurricane ian\" at the bottom of this section, it\\'s fixed but that was kinda funny to me lmao', 'new_content': ''}\n",
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-09-24T22:38:24Z', 'user': 'RCraig09', 'comment': '/* Climatological statistics */ add cites for 2022+ data', 'new_content': \"|image1= 1980- Atlantic region category 4 and 5 hurricanes - NYTimes and NOAA.svg | caption1=The 20-year average of the number of annual Category 4 and 5 hurricanes in the Atlantic region has approximately doubled since the year 2000.<ref name=NYTimes_20220929>{{cite news |last1=Leonhardt |first1=David |last2=Moses |first2=Claire |last3=Philbrick |first3=Ian Prasad |title=Ian Moves North / Category 4 and 5 Atlantic hurricanes since 1980 |url=https://www.nytimes.com/2022/09/29/briefing/hurricane-ian-storm-climate-change.html |newspaper=The New York Times |date=29 September 2022 |archive-url=https://web.archive.org/web/20220930003545/https://www.nytimes.com/2022/09/29/briefing/hurricane-ian-storm-climate-change.html |archive-date=30 September 2022 |quote=Source: NOAA – Graphic by Ashley Wu, ''The New York Times'' |url-status=live }} ([https://commons.wikimedia.org/wiki/File:1980-_Atlantic_region_category_4_and_5_hurricanes_-_NYTimes_and_NOAA.svg cites for 2022— data])</ref>\"}\n",
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-10-15T21:30:50Z', 'user': 'LiamMcGu', 'comment': '/* Southwestern United States */Added information', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in Arizona, [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages, along with heavy winds. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-10-15T21:32:36Z', 'user': 'LiamMcGu', 'comment': '/* Southwestern United States */Added information', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in Arizona, [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages, in [[California]], along with heavy winds. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-10-15T21:33:08Z', 'user': 'LiamMcGu', 'comment': '/* Southwestern United States */Fixed typo', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in Arizona, [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages in [[California]], along with heavy winds. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-10-15T21:36:56Z', 'user': 'LiamMcGu', 'comment': '/* Southwestern United States */Fixed typo', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in [[Arizona]], [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages in California, along with heavy winds. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'article_title': 'List_of_United_States_hurricanes', 'timestamp': '2023-10-15T21:39:59Z', 'user': 'LiamMcGu', 'comment': '/* Southwestern United States */Added information', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in [[Arizona]], [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages in California, along with heavy winds. It also affected places such as [[Idaho]] as a post-tropical remnant. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n"
     ]
    }
   ],
   "source": [
    "# Example: Print the first 10 examples\n",
    "for data in dataset[:10]:\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Q&A Pairs\n",
    "\n",
    "Now that we have our dataset, we need to generate a set of question-and-answer pairs. We'll use OpenAI's API to analyze the new content and generate a set of question-answer pairs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "from pydantic import BaseModel\n",
    "from typing import List, Literal\n",
    "import json\n",
    "\n",
    "# Define the Pydantic model for the output format\n",
    "class QAItem(BaseModel):\n",
    "    prompt: str\n",
    "    completion: str\n",
    "\n",
    "class QADataset(BaseModel):\n",
    "    dataset: List[QAItem]\n",
    "\n",
    "\n",
    "def generate_qa_pairs_from_changes(new_content, article_title):\n",
    "    \"\"\"\n",
    "    Query OpenAI to analyze the new content and generate a set of question-answer pairs.\n",
    "    If substantial information changes are detected (such as new sections, significant updates, or meaningful additions of facts),\n",
    "    the function returns a list of question-answer pairs in the specified JSON format.\n",
    "    \"\"\"\n",
    "\n",
    "    client = OpenAI()\n",
    "\n",
    "    # Create a query prompt to ask OpenAI to generate question-answer pairs based on the content\n",
    "    prompt = f\"\"\"\n",
    "    The following is newly added content to the Wikipedia article titled '{article_title}'.\n",
    "    Analyze the content and generate a set of specific question-answer pairs based on the new facts, updates, or meaningful changes.\n",
    "    Focus on creating general questions that a person might ask and answered them comprehensively with the content provided.\n",
    "    Do not ask questions that directly reference the date of the revision or the specific article title. \n",
    "    If a hurricane is mentioned, it should be referred to by its full name.\n",
    "    Ignore trivial changes such as typos or formatting.\n",
    "\n",
    "    Example questions: \n",
    "    - List the hurricanes that hit the US in 2024.\n",
    "    - What was the most recent hurricane to hit the US?\n",
    "    - What was the name of the hurricane that hit Florida in 2024?\n",
    "    - What was the category of hurricane Beryl?\n",
    "    - What was the path of hurricane Milton?\n",
    "\n",
    "    New Content:\n",
    "    \\\"\\\"\\\"{new_content[:3000]}\\\"\\\"\\\"\n",
    "\n",
    "    Please return a set of question-answer pairs in the form of a JSON array where each item is an object containing \n",
    "    'prompt' as the question and 'completion' as the direct answer from the content.\n",
    "    \"\"\"\n",
    "\n",
    "    try:\n",
    "        response = client.beta.chat.completions.parse(\n",
    "            model=\"gpt-4o-2024-08-06\",  # or \"gpt-4o\" if available\n",
    "            messages=[\n",
    "                {\"role\": \"system\", \"content\": \"You are a helpful assistant that generates a set of specific question-answer pairs based on the new facts, updates, or meaningful changes Wikipedia articles.\"},\n",
    "                {\"role\": \"user\", \"content\": prompt}\n",
    "            ],\n",
    "            max_tokens=1000,\n",
    "            temperature=0.7,\n",
    "            response_format=QADataset\n",
    "        )\n",
    "\n",
    "        # Parse the response from a JSON string to a Python dictionary\n",
    "        response_json = json.loads(response.choices[0].message.content)['dataset'][0]\n",
    "\n",
    "        # Create a new dictionary with the parsed content and additional fields\n",
    "        result = {\n",
    "            \"prompt\": response_json['prompt'],\n",
    "            \"completion\": response_json['completion'],\n",
    "            \"article_title\": article_title,\n",
    "            \"new_content\": new_content\n",
    "        }\n",
    "\n",
    "        return result\n",
    "\n",
    "    except Exception as e:\n",
    "        return f\"Error in generating QA pairs: {str(e)}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'prompt': 'What does NOAA predict for the 2024 Atlantic hurricane season?', 'completion': 'NOAA predicts an above-normal 2024 Atlantic hurricane season.', 'article_title': '2024_Atlantic_hurricane_season', 'new_content': '|align=\"left\"|NOAA\\n|align=\"left\"|May 23, 2024\\n|17–25\\n|8–13\\n|4–7\\n|<ref name=\"NOAA season forecast\">{{cite web|date=2024-05-23|title=NOAA predicts above-normal 2024 Atlantic hurricane season|url=https://www.noaa.gov/news-release/noaa-predicts-above-normal-2024-atlantic-hurricane-season|access-date=2024-05-23|website=National Oceanic and Atmospheric Administration|language=en-US}}</reF>\\n|-'}\n"
     ]
    }
   ],
   "source": [
    "# Test the Q&A pair generation\n",
    "a = generate_qa_pairs_from_changes(dataset[256]['new_content'], dataset[256]['article_title'])\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll now run this function over our entire dataset and collect the results. Depending on the length of the dataset, this may take a while. For a very large dataset, it can be more cost effective to run this in batch mode. For more information, see the [OpenAI Batch API](https://platform.openai.com/docs/guides/batch/overview).\n",
    "\n",
    "Note: We have decided to only process revisions longer than 1000 characters, as shorter revisions are less likely to contain significant changes or new information, and it reduces the size of the dataset we can create.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Processing dataset: 100%|██████████| 100/100 [01:59<00:00,  1.19s/it]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "MAX_DATASET_SIZE = 100\n",
    "MIN_CONTENT_LENGTH = 1000\n",
    "\n",
    "# Master list to collect all Q&A pairs\n",
    "master_qa_list = []\n",
    "\n",
    "for data in tqdm(dataset[:MAX_DATASET_SIZE], desc=\"Processing dataset\"):\n",
    "    # Skip if the content is too short\n",
    "    if len(data['new_content']) < MIN_CONTENT_LENGTH:\n",
    "        continue\n",
    "\n",
    "    # Call the function to generate QA pairs\n",
    "    try:\n",
    "        qa_response = generate_qa_pairs_from_changes(data['new_content'], data['article_title'])\n",
    "        if isinstance(qa_response, dict) and 'prompt' in qa_response and 'completion' in qa_response:\n",
    "            master_qa_list.append(qa_response)\n",
    "        else:\n",
    "            print(f\"Skipping invalid response for article: {data['article_title']}\")\n",
    "    except Exception as e:\n",
    "        print(f\"Exception encountered for article {data['article_title']}: {str(e)}\")\n",
    "        # Continue processing instead of breaking\n",
    "        continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of examples in master_qa_list:  23\n",
      "{'prompt': 'Which Pacific hurricane is known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in Arizona, [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages, along with heavy winds. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'prompt': 'What is the only Pacific tropical cyclone known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in Arizona, [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages, in [[California]], along with heavy winds. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'prompt': 'What is the only Pacific tropical cyclone known to have produced hurricane-force winds in California?', 'completion': 'The only Pacific tropical cyclone known to have produced hurricane-force winds in California is the 1858 San Diego hurricane.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in Arizona, [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages in [[California]], along with heavy winds. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'prompt': 'What is the only Pacific tropical cyclone known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in [[Arizona]], [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages in California, along with heavy winds. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'prompt': 'What is the only Pacific hurricane known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific hurricane known to have produced hurricane-force winds in California.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in [[Arizona]], [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona, and [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages in California, along with heavy winds. It also affected places such as [[Idaho]] as a post-tropical remnant. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'prompt': 'What is the only Pacific tropical cyclone known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in [[Arizona]], [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, and [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona. Many tropical cyclones have affected these areas in the 21st century, with a notable one being [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages in California, along with heavy winds. It also affected places such as [[Idaho]] as a post-tropical remnant. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'prompt': 'Which Pacific hurricane is known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': 'Additionally, three [[Pacific hurricane]]s struck [[Hawaii]], and one Pacific tropical cyclone brought hurricane-force winds to [[California]]. The tables list hurricanes by category on the [[Saffir–Simpson scale]], based on winds that occurred in each state.\\nThe [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in Arizona, [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, and [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona.<ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'prompt': 'What was the category of Hurricane Dora in 2023?', 'completion': 'Hurricane Dora in 2023 was a Category 4 hurricane.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': '|-\\n|-\\n| [[Hurricane Dora (2023)|Dora]]\\n| bgcolor=#{{storm colour|4}}|{{center|Category&nbsp;4 hurricane}}\\n| {{center|August 8}}\\n| 2023\\n|colspan=\"9\"|<small>\\n\\'\\'\\'Sources:\\'\\'\\' <ref>{{cite web|author=Central Pacific Hurricane Season|title=1959 Central Pacific Hurricane Season|access-date=2008-01-26|url=http://www.prh.noaa.gov/cphc/summaries/1959.php#Dot}}</ref><ref>{{cite web|author=Central Pacific Hurricane Season|title=1982 Central Pacific Hurricane Season|access-date=2008-01-26|url=http://www.prh.noaa.gov/cphc/summaries/1982.php#Iniki}}</ref><ref>{{cite web|author=Central Pacific Hurricane Season|title=1992 Central Pacific Hurricane Season|access-date=2008-01-26|url=http://www.prh.noaa.gov/cphc/summaries/1992.php#Iniki}}</ref><ref>{{Cite report|last=Jelsema|first=Jon|date=August 8, 2023|title=Hurricane Dora Discussion Number 33|url=https://www.nhc.noaa.gov/archive/2023/ep05/ep052023.discus.033.shtml?|publisher=Central Pacific Hurricane Center|location=Honolulu, Hawaii|language=en-US|access-date=August 9, 2023}}</ref></small>'}\n",
      "{'prompt': 'Which Pacific tropical cyclone is known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': 'The [[1858 San Diego hurricane]] is the only [[Pacific hurricane|Pacific]] tropical cyclone known to have produced hurricane-force winds in [[California]]; it affected [[San Diego, California|San Diego]] on October 2, 1858, though its center remained just offshore. In the 20th&nbsp;century, only four tropical cyclones produced tropical storm force winds in the southwestern United States: a [[1939 California tropical storm|tropical storm]] in September 1939 in California, [[Hurricane Joanne (1972)|Hurricane Joanne]] in October 1972 in Arizona, [[Hurricane Kathleen (1976)|Hurricane Kathleen]] in September 1976 in Arizona and California, and [[Hurricane Nora (1997)|Hurricane Nora]] in September 1997 in Arizona. In the 21st century, the only tropical cyclone to achieve this was [[Hurricane Hilary]] in 2023, which caused large amounts of flooding and power outages, along with heavy winds. <ref>{{cite web|author1=Michael Chenoweth  |author2=Chris Landsea |name-list-style=amp|year=2004|title=The San Diego Hurricane of October 2, 1858|publisher=American Meteorological Society|access-date=2008-01-26|url=http://www.aoml.noaa.gov/hrd/Landsea/chenowethlandsea.pdf}}</ref>'}\n",
      "{'prompt': 'Did Hurricane Dora (2023) make landfall in Hawaii?', 'completion': 'No, Hurricane Dora (2023) did not make landfall in Hawaii. It passed about 500 miles south of the Hawaiian islands.', 'article_title': 'List_of_United_States_hurricanes', 'new_content': '[[2023 Pacific hurricane season#Hurricane Dora|Hurricane Dora (2023)]] affected [[2023 Hawaii wildfires|wildfires in Hawaii]], \"although there is some debate about how much influence the storm had as it passed about 500 miles south of the Hawaiian islands.\"<ref>{{cite web |author1=Dan Stillman |title=Unrelenting Hurricane Dora makes history by becoming a typhoon |url=https://www.washingtonpost.com/weather/2023/08/11/hurricane-dora-typhoon-hawaii-record/ |publisher=The Washington Post |access-date=18 October 2023 |date=11 August 2023}}</ref> However, it is not included in this list since this is a list of storms producing hurricane force winds within the states and territories listed, not a list of hurricanes passing within 500 miles of Hawaii that made \"history without making landfall.\"<ref>{{cite web |author1=Jennifer Robbins |title=Hurricane Dora makes history without making landfall |url=https://www.hawaiinewsnow.com/2023/08/12/hurricane-dora-makes-history-without-making-landfall/ |publisher=Hawaii News Now |access-date=18 October 2023 |date=12 August 2023}}</ref><br/><br/>\\n\\'\\'Note: This section only includes tropical cyclones that occurred [[Territory of Hawaii|during or after 1900]], when [[Hawaii]] was acquired by the United States\\'\\'<br/>\\'\\'This section does not include tropical cyclones that affected the [[Northwestern Hawaiian Islands]]\\'\\''}\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of examples in master_qa_list: \", len(master_qa_list))\n",
    "# Example: Print the first 10 examples\n",
    "for data in master_qa_list[:10]:\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review the QA pairs in Label Studio\n",
    "We can now review the QA pairs in Label Studio to ensure they are of high quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_qa_list_for_label_studio(qa_list):\n",
    "    \"\"\"\n",
    "    Format the Q&A list into the desired structure for Label Studio.\n",
    "    \n",
    "    :param qa_list: List of dictionaries containing 'prompt' and 'completion' keys\n",
    "    :return: Formatted list of Q&A pairs\n",
    "    \"\"\"\n",
    "    formatted_list = []\n",
    "    for qa_pair in qa_list:\n",
    "        formatted_qa = {\n",
    "            \"data\": {\n",
    "                \"question\": qa_pair['prompt'],\n",
    "                \"article_title\": qa_pair['article_title'],\n",
    "                \"context\": qa_pair['new_content']\n",
    "            },\n",
    "            \"predictions\": [\n",
    "                {\n",
    "                    \"model_version\": \"v1\",\n",
    "                    \"result\": [\n",
    "                        {\n",
    "                            \"from_name\": \"completion\",\n",
    "                            \"to_name\": \"question\",\n",
    "                            \"type\": \"textarea\",\n",
    "                            \"value\": {\n",
    "                                \"text\": [qa_pair['completion']]\n",
    "                            }\n",
    "                        }\n",
    "                    ]\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "        formatted_list.append(formatted_qa)\n",
    "    return formatted_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the formatted version of master_qa_list\n",
    "formatted_qa_list = format_qa_list_for_label_studio(master_qa_list)\n",
    "with open('formatted_qa_list.json', 'w') as f:\n",
    "    json.dump(formatted_qa_list, f, indent=2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the URL where Label Studio is accessible and the API key for your user account\n",
    "LABEL_STUDIO_URL = 'http://localhost:8080'  # Your Label Studio instance URL\n",
    "API_KEY = '<your-api-key>'  # Your Label Studio API key\n",
    "\n",
    "# Import the SDK and the client module\n",
    "from label_studio_sdk.client import LabelStudio\n",
    "\n",
    "# Connect to the Label Studio API and check the connection\n",
    "ls = LabelStudio(base_url=LABEL_STUDIO_URL, api_key=API_KEY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_config = \"\"\"\n",
    "<View>\n",
    "  <Header value=\"Please read the question, article title, and context. Modify the completion if needed.\" />\n",
    "  \n",
    "  <Header value=\"Article Title\" />\n",
    "  <Text name=\"article_title\" value=\"$article_title\" />\n",
    "\n",
    "  <Header value=\"Context\" />\n",
    "  <Text name=\"context\" value=\"$context\" />\n",
    "\n",
    "  <Header value=\"Question\" />\n",
    "  <Text name=\"question\" value=\"$question\" />\n",
    "\n",
    "  <Header value=\"Completion (Prediction)\" />\n",
    "  <TextArea name=\"completion\" toName=\"question\"\n",
    "            editable=\"true\" required=\"true\" />\n",
    "</View>\n",
    "\"\"\"\n",
    "\n",
    "# Create a new project\n",
    "project = ls.projects.create(\n",
    "    title='Hurricane Data Project',\n",
    "    description='Label questions and completions about hurricanes with their respective contexts and titles.',\n",
    "    label_config=label_config\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task created with ID: 56601\n",
      "Prediction added for task ID: 56601\n",
      "Task created with ID: 56602\n",
      "Prediction added for task ID: 56602\n",
      "Task created with ID: 56603\n",
      "Prediction added for task ID: 56603\n",
      "Task created with ID: 56604\n",
      "Prediction added for task ID: 56604\n",
      "Task created with ID: 56605\n",
      "Prediction added for task ID: 56605\n",
      "Task created with ID: 56606\n",
      "Prediction added for task ID: 56606\n",
      "Task created with ID: 56607\n",
      "Prediction added for task ID: 56607\n",
      "Task created with ID: 56608\n",
      "Prediction added for task ID: 56608\n",
      "Task created with ID: 56609\n",
      "Prediction added for task ID: 56609\n",
      "Task created with ID: 56610\n",
      "Prediction added for task ID: 56610\n",
      "Task created with ID: 56611\n",
      "Prediction added for task ID: 56611\n",
      "Task created with ID: 56612\n",
      "Prediction added for task ID: 56612\n",
      "Task created with ID: 56613\n",
      "Prediction added for task ID: 56613\n",
      "Task created with ID: 56614\n",
      "Prediction added for task ID: 56614\n",
      "Task created with ID: 56615\n",
      "Prediction added for task ID: 56615\n",
      "Task created with ID: 56616\n",
      "Prediction added for task ID: 56616\n",
      "Task created with ID: 56617\n",
      "Prediction added for task ID: 56617\n",
      "Task created with ID: 56618\n",
      "Prediction added for task ID: 56618\n",
      "Task created with ID: 56619\n",
      "Prediction added for task ID: 56619\n",
      "Task created with ID: 56620\n",
      "Prediction added for task ID: 56620\n",
      "Task created with ID: 56621\n",
      "Prediction added for task ID: 56621\n",
      "Task created with ID: 56622\n",
      "Prediction added for task ID: 56622\n",
      "Task created with ID: 56623\n",
      "Prediction added for task ID: 56623\n"
     ]
    }
   ],
   "source": [
    "from label_studio_sdk.label_interface.objects import PredictionValue\n",
    "\n",
    "for qa_pair in master_qa_list:\n",
    "    # Create task data\n",
    "    task_data = {\n",
    "        \"data\": {\n",
    "            \"question\": qa_pair['prompt'],\n",
    "            \"article_title\": qa_pair['article_title'],\n",
    "            \"context\": qa_pair['new_content']\n",
    "        }\n",
    "    }\n",
    "\n",
    "    # Create the task in Label Studio\n",
    "    task = ls.tasks.create(project=project.id, **task_data)\n",
    "    task_id = task.id\n",
    "    print(f\"Task created with ID: {task_id}\")\n",
    "\n",
    "    # Create prediction data\n",
    "    prediction = PredictionValue(\n",
    "        model_version=\"v1\",\n",
    "        result=[\n",
    "            {\n",
    "                \"from_name\": \"completion\",\n",
    "                \"to_name\": \"question\",\n",
    "                \"type\": \"textarea\",\n",
    "                \"value\": {\n",
    "                    \"text\": [qa_pair['completion']]\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    # Insert prediction into the task\n",
    "    ls.predictions.create(task=task_id, **prediction.model_dump())\n",
    "    print(f\"Prediction added for task ID: {task_id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Annotate the Dataset and Download the Results\n",
    "Once we have annotated the dataset, we can download the results and convert them into the format required for OpenAI fine-tuning.\n",
    "\n",
    "Note: If we just want to simply create annotations from the predictions, we can select all the Tasks in Label Studio and click the \"Create Annotations From Predictions\" button."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Label Studio, Views are customizable tabs in the Data Manager where you can configure filters and specify how tasks and data appear within the project. Each view is associated with a project and can include various filtering criteria, ordering options, and other display preferences.\n",
    "\n",
    "We can create a View to filter the Tasks that are annotated, so we only export the annotated Tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "from label_studio_sdk.data_manager import Filters, Column, Type, Operator\n",
    "\n",
    "filters = Filters.create(Filters.AND, [\n",
    "    Filters.item(\n",
    "        Column.completed_at,\n",
    "        Operator.EMPTY,\n",
    "        Type.Boolean,\n",
    "        Filters.value(False)\n",
    "    )\n",
    "])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "view = ls.views.create(\n",
    "    project=project.id,\n",
    "    data={\n",
    "        'title': 'Annotated Tasks',\n",
    "        'filters': filters\n",
    "    }\n",
    ")\n",
    "tab = ls.views.get(id=view.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can extract the annotated QA pairs from the annotated tasks and format them into a simple prompt-completion pair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_task_data(task):\n",
    "    \"\"\"\n",
    "    Format a single task's data into {'prompt': prompt, 'completion': completion} structure.\n",
    "    \"\"\"\n",
    "    formatted_data = []\n",
    "    \n",
    "    # Assuming the question is in task.data['question']\n",
    "    prompt = task.data.get('question', '')\n",
    "    \n",
    "    # Check annotations first, then predictions\n",
    "    if task.annotations:\n",
    "        for annotation in task.annotations:\n",
    "            for result in annotation.get('result', []):\n",
    "                if result.get('from_name') == 'completion' and result.get('to_name') == 'question':\n",
    "                    completion = result.get('value', {}).get('text', [''])[0]\n",
    "                    formatted_data.append({'prompt': prompt, 'completion': completion})\n",
    "    elif task.predictions:\n",
    "        for prediction in task.predictions:\n",
    "            for result in prediction.get('result', []):\n",
    "                if result.get('from_name') == 'completion' and result.get('to_name') == 'question':\n",
    "                    completion = result.get('value', {}).get('text', [''])[0]\n",
    "                    formatted_data.append({'prompt': prompt, 'completion': completion})\n",
    "    \n",
    "    return formatted_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List to store all formatted data\n",
    "all_formatted_data = []\n",
    "\n",
    "# Fetch annotated tasks\n",
    "annotated_tasks = ls.tasks.list(view=tab.id, fields='all')\n",
    "\n",
    "# Process each task\n",
    "for task in annotated_tasks:\n",
    "    formatted_task_data = format_task_data(task)\n",
    "    all_formatted_data.extend(formatted_task_data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of examples in master_qa_list:  23\n",
      "{'prompt': 'Which Pacific hurricane is known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.'}\n",
      "{'prompt': 'What is the only Pacific tropical cyclone known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.'}\n",
      "{'prompt': 'What is the only Pacific tropical cyclone known to have produced hurricane-force winds in California?', 'completion': 'The only Pacific tropical cyclone known to have produced hurricane-force winds in California is the 1858 San Diego hurricane.'}\n",
      "{'prompt': 'What is the only Pacific tropical cyclone known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.'}\n",
      "{'prompt': 'What is the only Pacific hurricane known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific hurricane known to have produced hurricane-force winds in California.'}\n",
      "{'prompt': 'What is the only Pacific tropical cyclone known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.'}\n",
      "{'prompt': 'Which Pacific hurricane is known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.'}\n",
      "{'prompt': 'What was the category of Hurricane Dora in 2023?', 'completion': 'Hurricane Dora in 2023 was a Category 4 hurricane.'}\n",
      "{'prompt': 'Which Pacific tropical cyclone is known to have produced hurricane-force winds in California?', 'completion': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.'}\n",
      "{'prompt': 'Did Hurricane Dora (2023) make landfall in Hawaii?', 'completion': 'No, Hurricane Dora (2023) did not make landfall in Hawaii. It passed about 500 miles south of the Hawaiian islands.'}\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of examples in master_qa_list: \", len(all_formatted_data))\n",
    "# Example: Print the first 10 examples\n",
    "for data in all_formatted_data[:10]:\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to a JSON file\n",
    "import json\n",
    "with open('annotated_qa_pairs.json', 'w') as f:\n",
    "    json.dump(all_formatted_data, f, indent=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the Fine-Tuning Dataset\n",
    "We will now convert the list of QA pairs into the format required for OpenAI fine-tuning and write it to a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# System message for all entries\n",
    "system_message = {\"role\": \"system\", \"content\": \"You are a helpful assistant that answers questions about current events in hurricanes. Provide detailed answers.\"}\n",
    "\n",
    "# List to store the converted dataset\n",
    "new_format_dataset = []\n",
    "\n",
    "# Convert each prompt-completion pair to the new format\n",
    "for entry in all_formatted_data:\n",
    "    new_entry = {\n",
    "        \"messages\": [\n",
    "            system_message,\n",
    "            {\"role\": \"user\", \"content\": entry['prompt']},\n",
    "            {\"role\": \"assistant\", \"content\": entry['completion']}\n",
    "        ]\n",
    "    }\n",
    "    new_format_dataset.append(new_entry)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'messages': [{'role': 'system',\n",
       "   'content': 'You are a helpful assistant that answers questions about current events in hurricanes. Provide detailed answers.'},\n",
       "  {'role': 'user',\n",
       "   'content': 'Which Pacific hurricane is known to have produced hurricane-force winds in California?'},\n",
       "  {'role': 'assistant',\n",
       "   'content': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.'}]}"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_format_dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_to_jsonl(qa_list, filename):\n",
    "    with open(filename, 'w') as f:\n",
    "        for qa_item in qa_list:\n",
    "            # Convert each dictionary to JSON format and write to file\n",
    "            f.write(json.dumps(qa_item) + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "write_to_jsonl(new_format_dataset, 'qa_pairs_openai_wiki_hurricane_dataset_format.jsonl')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset Analysis and Cost Estimates\n",
    "\n",
    "We will use this file to analyze the data and create a new file with the correct format. The analysis functions are pulled from [Data preperation and analysis for chat model fine-tuning](https://cookbook.openai.com/examples/chat_finetuning_data_prep)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import tiktoken # for token counting\n",
    "import numpy as np\n",
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num examples: 23\n",
      "First example:\n",
      "{'role': 'system', 'content': 'You are a helpful assistant that answers questions about current events in hurricanes. Provide detailed answers.'}\n",
      "{'role': 'user', 'content': 'Which Pacific hurricane is known to have produced hurricane-force winds in California?'}\n",
      "{'role': 'assistant', 'content': 'The 1858 San Diego hurricane is the only Pacific tropical cyclone known to have produced hurricane-force winds in California.'}\n"
     ]
    }
   ],
   "source": [
    "data_path = \"qa_pairs_openai_wiki_hurricane_dataset_format.jsonl\"\n",
    "\n",
    "# Load the dataset\n",
    "with open(data_path, 'r', encoding='utf-8') as f:\n",
    "    dataset = [json.loads(line) for line in f]\n",
    "\n",
    "# Initial dataset stats\n",
    "print(\"Num examples:\", len(dataset))\n",
    "print(\"First example:\")\n",
    "for message in dataset[0][\"messages\"]:\n",
    "    print(message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No errors found\n"
     ]
    }
   ],
   "source": [
    "# Format error checks\n",
    "format_errors = defaultdict(int)\n",
    "\n",
    "for ex in dataset:\n",
    "    if not isinstance(ex, dict):\n",
    "        format_errors[\"data_type\"] += 1\n",
    "        continue\n",
    "        \n",
    "    messages = ex.get(\"messages\", None)\n",
    "    if not messages:\n",
    "        format_errors[\"missing_messages_list\"] += 1\n",
    "        continue\n",
    "        \n",
    "    for message in messages:\n",
    "        if \"role\" not in message or \"content\" not in message:\n",
    "            format_errors[\"message_missing_key\"] += 1\n",
    "        \n",
    "        if any(k not in (\"role\", \"content\", \"name\", \"function_call\", \"weight\") for k in message):\n",
    "            format_errors[\"message_unrecognized_key\"] += 1\n",
    "        \n",
    "        if message.get(\"role\", None) not in (\"system\", \"user\", \"assistant\", \"function\"):\n",
    "            format_errors[\"unrecognized_role\"] += 1\n",
    "            \n",
    "        content = message.get(\"content\", None)\n",
    "        function_call = message.get(\"function_call\", None)\n",
    "        \n",
    "        if (not content and not function_call) or not isinstance(content, str):\n",
    "            format_errors[\"missing_content\"] += 1\n",
    "    \n",
    "    if not any(message.get(\"role\", None) == \"assistant\" for message in messages):\n",
    "        format_errors[\"example_missing_assistant_message\"] += 1\n",
    "\n",
    "if format_errors:\n",
    "    print(\"Found errors:\")\n",
    "    for k, v in format_errors.items():\n",
    "        print(f\"{k}: {v}\")\n",
    "else:\n",
    "    print(\"No errors found\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoding = tiktoken.get_encoding(\"cl100k_base\")\n",
    "\n",
    "# not exact!\n",
    "# simplified from https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb\n",
    "def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1):\n",
    "    num_tokens = 0\n",
    "    for message in messages:\n",
    "        num_tokens += tokens_per_message\n",
    "        for key, value in message.items():\n",
    "            num_tokens += len(encoding.encode(value))\n",
    "            if key == \"name\":\n",
    "                num_tokens += tokens_per_name\n",
    "    num_tokens += 3\n",
    "    return num_tokens\n",
    "\n",
    "def num_assistant_tokens_from_messages(messages):\n",
    "    num_tokens = 0\n",
    "    for message in messages:\n",
    "        if message[\"role\"] == \"assistant\":\n",
    "            num_tokens += len(encoding.encode(message[\"content\"]))\n",
    "    return num_tokens\n",
    "\n",
    "def print_distribution(values, name):\n",
    "    print(f\"\\n#### Distribution of {name}:\")\n",
    "    print(f\"min / max: {min(values)}, {max(values)}\")\n",
    "    print(f\"mean / median: {np.mean(values)}, {np.median(values)}\")\n",
    "    print(f\"p5 / p95: {np.quantile(values, 0.1)}, {np.quantile(values, 0.9)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num examples missing system message: 0\n",
      "Num examples missing user message: 0\n",
      "\n",
      "#### Distribution of num_messages_per_example:\n",
      "min / max: 3, 3\n",
      "mean / median: 3.0, 3.0\n",
      "p5 / p95: 3.0, 3.0\n",
      "\n",
      "#### Distribution of num_total_tokens_per_example:\n",
      "min / max: 61, 83\n",
      "mean / median: 74.17391304347827, 75.0\n",
      "p5 / p95: 71.0, 76.0\n",
      "\n",
      "#### Distribution of num_assistant_tokens_per_example:\n",
      "min / max: 15, 29\n",
      "mean / median: 25.47826086956522, 24.0\n",
      "p5 / p95: 22.2, 29.0\n",
      "\n",
      "0 examples may be over the 16,385 token limit, they will be truncated during fine-tuning\n"
     ]
    }
   ],
   "source": [
    "# Warnings and tokens counts\n",
    "n_missing_system = 0\n",
    "n_missing_user = 0\n",
    "n_messages = []\n",
    "convo_lens = []\n",
    "assistant_message_lens = []\n",
    "\n",
    "for ex in dataset:\n",
    "    messages = ex[\"messages\"]\n",
    "    if not any(message[\"role\"] == \"system\" for message in messages):\n",
    "        n_missing_system += 1\n",
    "    if not any(message[\"role\"] == \"user\" for message in messages):\n",
    "        n_missing_user += 1\n",
    "    n_messages.append(len(messages))\n",
    "    convo_lens.append(num_tokens_from_messages(messages))\n",
    "    assistant_message_lens.append(num_assistant_tokens_from_messages(messages))\n",
    "    \n",
    "print(\"Num examples missing system message:\", n_missing_system)\n",
    "print(\"Num examples missing user message:\", n_missing_user)\n",
    "print_distribution(n_messages, \"num_messages_per_example\")\n",
    "print_distribution(convo_lens, \"num_total_tokens_per_example\")\n",
    "print_distribution(assistant_message_lens, \"num_assistant_tokens_per_example\")\n",
    "n_too_long = sum(l > 16385 for l in convo_lens)\n",
    "print(f\"\\n{n_too_long} examples may be over the 16,385 token limit, they will be truncated during fine-tuning\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset has ~1706 tokens that will be charged for during training\n",
      "By default, you'll train for 4 epochs on this dataset\n",
      "By default, you'll be charged for ~6824 tokens\n"
     ]
    }
   ],
   "source": [
    "# Pricing and default n_epochs estimate\n",
    "MAX_TOKENS_PER_EXAMPLE = 16385\n",
    "\n",
    "TARGET_EPOCHS = 3\n",
    "MIN_TARGET_EXAMPLES = 100\n",
    "MAX_TARGET_EXAMPLES = 25000\n",
    "MIN_DEFAULT_EPOCHS = 1\n",
    "MAX_DEFAULT_EPOCHS = 25\n",
    "\n",
    "n_epochs = TARGET_EPOCHS\n",
    "n_train_examples = len(dataset)\n",
    "if n_train_examples * TARGET_EPOCHS < MIN_TARGET_EXAMPLES:\n",
    "    n_epochs = min(MAX_DEFAULT_EPOCHS, MIN_TARGET_EXAMPLES // n_train_examples)\n",
    "elif n_train_examples * TARGET_EPOCHS > MAX_TARGET_EXAMPLES:\n",
    "    n_epochs = max(MIN_DEFAULT_EPOCHS, MAX_TARGET_EXAMPLES // n_train_examples)\n",
    "\n",
    "n_billing_tokens_in_dataset = sum(min(MAX_TOKENS_PER_EXAMPLE, length) for length in convo_lens)\n",
    "print(f\"Dataset has ~{n_billing_tokens_in_dataset} tokens that will be charged for during training\")\n",
    "print(f\"By default, you'll train for {n_epochs} epochs on this dataset\")\n",
    "print(f\"By default, you'll be charged for ~{n_epochs * n_billing_tokens_in_dataset} tokens\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload the Dataset to OpenAI\n",
    "Once we are happy with the cost and details of our dataset, we can upload it to OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FileObject(id='file-qBVnzhGrEHvEPvZg6ZwvTfpK', bytes=4301148, created_at=1728916096, filename='qa_pairs_openai_wiki_hurricane_dataset_format.jsonl', object='file', purpose='fine-tune', status='processed', status_details=None)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = OpenAI()\n",
    "\n",
    "client.files.create(\n",
    "  file=open(\"qa_pairs_openai_wiki_hurricane_dataset_format.jsonl\", \"rb\"),\n",
    "  purpose=\"fine-tune\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Fine-Tuning Job\n",
    "\n",
    "We can now create a fine-tuning job programmatically or via the [fine-tuning UI](https://platform.openai.com/finetune), using the file ID for the file we just uploaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FineTuningJob(id='ftjob-YSQajs0bsLVi0IQfSv6b5jbz', created_at=1728916171, error=Error(code=None, message=None, param=None), fine_tuned_model=None, finished_at=None, hyperparameters=Hyperparameters(n_epochs='auto', batch_size='auto', learning_rate_multiplier='auto'), model='gpt-4o-mini-2024-07-18', object='fine_tuning.job', organization_id='org-19ZNwom3AVpfo00n76H0UeLL', result_files=[], seed=433405727, status='validating_files', trained_tokens=None, training_file='file-qBVnzhGrEHvEPvZg6ZwvTfpK', validation_file=None, estimated_finish=None, integrations=[], user_provided_suffix='wiki-hurricane-2024')"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.fine_tuning.jobs.create(\n",
    "  training_file=\"file-qBVnzhGrEHvEPvZg6ZwvTfpK\", \n",
    "  model=\"gpt-4o-mini-2024-07-18\",\n",
    "  suffix=\"wiki-hurricane-2024\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the status of our fine-tuning job using the job ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ft_job.fine_tuned_model:  ft:gpt-4o-mini-2024-07-18:personal:wiki-hurricane-2024:AIGr7s2N\n"
     ]
    }
   ],
   "source": [
    "ft_job = client.fine_tuning.jobs.retrieve(\"ftjob-YSQajs0bsLVi0IQfSv6b5jbz\")\n",
    "print(\"ft_job.fine_tuned_model: \", ft_job.fine_tuned_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use our fine-tuned model\n",
    "\n",
    "Once our fine-tuning job is complete, we can use the new model to generate answers to questions about hurricanes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ParsedChatCompletionMessage[NoneType](content='The most recent hurricane to hit the US in 2024 is Hurricane Milton, which struck Florida on October 9.', refusal=None, role='assistant', function_call=None, tool_calls=[], parsed=None)\n"
     ]
    }
   ],
   "source": [
    "from openai import OpenAI\n",
    "client = OpenAI()\n",
    "\n",
    "completion = client.beta.chat.completions.parse(\n",
    "  model=\"ft:gpt-4o-mini-2024-07-18:personal:wiki-hurricane-2024:AIGr7s2N\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant that answers questions about current events in hurricanes. Provide detailed answers.\"},\n",
    "    {\"role\": \"user\", \"content\": \"What's the most recent hurricane to hit the US in 2024?\"}\n",
    "  ]\n",
    ")\n",
    "print(completion.choices[0].message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}