12-feature-selection/find_signature.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Starter Code for the Lesson 12: Feature Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "import numpy\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "numpy.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The words (features) and authors (labels), already largely processed.\n",
    "\n",
    "These files should have been created from the previous (Lesson 11) mini-project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words_file = \"../11-text-learning/your_word_data.pkl\" \n",
    "authors_file = \"../11-text-learning/your_email_authors.pkl\"\n",
    "\n",
    "with open(words_file, \"r\") as f:\n",
    "    word_data = pickle.load(f)\n",
    "       \n",
    "with open(authors_file, \"r\") as f:\n",
    "    authors = pickle.load(f) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`test_size` is the percentage of events assigned to the test set (the remainder go into training) feature matrices changed to dense representations for compatibility with classifier functions in versions 0.15.2 and earlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features_train, features_test, labels_train, labels_test = train_test_split(word_data, authors, test_size=0.1, random_state=42)\n",
    "vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5, stop_words='english')\n",
    "\n",
    "features_train = vectorizer.fit_transform(features_train)\n",
    "features_test  = vectorizer.transform(features_test).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A classic way to overfit is to use a small number of data points and a large number of features;\n",
    "\n",
    "train on only 150 events to put ourselves in this regime:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features_train = features_train[:150].toarray()\n",
    "labels_train = labels_train[:150]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ↓↓↓ Your Code Goes Here ↓↓↓"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}