expt/TrainingVAEsparse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VAEs on sparse data\n",
    "* The following notebook provides an example of how to load a dataset, setup parameters for it, create the model\n",
    "and train it for a few epochs.\n",
    "* In the notebook, we will use with the RCV1 dataset (assuming it has been setup previously). For details on how to set it up, run `python rcv2.py` in the [optvaedatasets](../optvaedatasets) folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys,os,glob\n",
    "from collections import OrderedDict\n",
    "import numpy as np\n",
    "from utils.misc import readPickle, createIfAbsent\n",
    "sys.path.append('../')\n",
    "from optvaedatasets.load import loadDataset as loadDataset_OVAE\n",
    "from sklearn.feature_extraction.text import TfidfTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Parameters\n",
    "* The model parameters have been saved here, we'll load them and look at them\n",
    "* These are what the model will be built based on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read  1  objects\n",
      "( q_dim_hidden 400 ) ( grad_noise 0.0 ) ( opt_type none ) ( dataset binarized_mnist ) ( epochs 500 ) ( seed 1 ) ( n_steps 200 ) ( q_layers 2 ) ( init_weight 0.1 ) ( reg_spec _ ) ( reg_value 0.01 ) ( input_type normalize ) ( p_dim_hidden 400 ) ( reloadFile ./NOSUCHFILE ) ( dim_stochastic 100 ) ( lr 0.0008 ) ( p_layers 2 ) ( init_scheme uniform ) ( input_dropout 0.0001 ) ( reg_type l2 ) ( optimizer adam ) ( batch_size 500 ) ( opt_method adam ) ( savedir ./chkpt ) ( param_lr 0.01 ) ( likelihood mult ) ( savefreq 5 ) ( paramFile ./NOSUCHFILE ) ( emission_type mlp ) ( nonlinearity relu ) ( anneal_rate 0 ) ( unique_id VAE_lr-8_0e-04-ph-400-qh-400-ds-100-pl-2-ql-2-nl-relu-bs-500-ep-500-plr-1_0e-02-ar-0-otype-none-ns-200-etype-mlp-ll-mult-itype-normalizel20_01_-uid ) ( leaky_param 0.0 )\n"
     ]
    }
   ],
   "source": [
    "default_params = readPickle('../optvaeutils/default_settings.pkl')[0]\n",
    "for k in default_params:\n",
    "    print '(',k,default_params[k],')',\n",
    "print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the moment, we will leave everything as is. Some worthwhile parameters to note: \n",
    "* `n_steps`: Number of steps of optimizing $\\psi(x)$, the local variational parameters as output by the inference network. We'll set this to 10 below for the moment.\n",
    "* `dim_stochastic`: Number of latent dimensions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "default_params['opt_type'] = 'finopt' #set to finopt to optimize var. params, none otherwise\n",
    "default_params['n_steps'] = 5\n",
    "#temporary directory where checkpoints are saved\n",
    "default_params['savedir'] = './tmp'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load dataset\n",
    "* Lets load the RCV1(v2) dataset and visualize how the dataset `<dict>` is structured\n",
    "* We'll need to append some parameters from the dataset into the default parameters dict that we will use to create the model\n",
    "* Also, compute the idf vectors for the entire dataset (the term frequencies will be multiplied dynamically) inside the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vocabulary <type 'list'>\n",
      "data_type <type 'str'> bow\n",
      "dim_observations <type 'int'> 10000\n",
      "train <class 'scipy.sparse.csr.csr_matrix'> (789414, 10000)\n",
      "test <class 'scipy.sparse.csr.csr_matrix'> (10000, 10000)\n",
      "valid <class 'scipy.sparse.csr.csr_matrix'> (5000, 10000)\n"
     ]
    }
   ],
   "source": [
    "dset = loadDataset_OVAE('rcv2')\n",
    "\n",
    "#Visualize structure of dataset dict\n",
    "for k in dset:\n",
    "    print k, type(dset[k]),\n",
    "    if hasattr(dset[k],'shape'):\n",
    "        print dset[k].shape\n",
    "    elif type(dset[k]) is not list:\n",
    "        print dset[k]\n",
    "    else:\n",
    "        print\n",
    "\n",
    "#Add parameters to default_params\n",
    "for k in ['dim_observations','data_type']:\n",
    "    default_params[k] = dset[k]\n",
    "default_params['max_word_count'] =dset['train'].max()\n",
    "\n",
    "\n",
    "#Create IDF\n",
    "additional_attrs        = {}\n",
    "tfidf                   = TfidfTransformer(norm=None) \n",
    "tfidf.fit(dset['train'])\n",
    "additional_attrs['idf'] = tfidf.idf_ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optvaemodels.vae import VAE as Model\n",
    "import optvaemodels.vae_learn as Learn\n",
    "import optvaemodels.vae_evaluate as Evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup \n",
    "* Create directory for configuration files. The configuration file for a single experiment is in the pickle file.\n",
    "* We will use this directory to save checkpoint files as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model from scratch. Parameters in:  ./tmp-rcv2-finopt/VAE_lr-8_0e-04-ph-400-qh-400-ds-100-pl-2-ql-2-nl-relu-bs-500-ep-500-plr-1_0e-02-ar-0-otype-none-ns-200-etype-mlp-ll-mult-itype-normalizel20_01_-uid-config.pkl\n",
      "\t<<Nparameters: 8451800>>\n",
      "\t<<Setting idf as theano shared variable>>\n",
      "\t<<WARNING: iter_ctr will not differentiated with respect to>>\n",
      "\t<<WARNING: anneal will not differentiated with respect to>>\n",
      "\t<<WARNING: lr will not differentiated with respect to>>\n",
      "\t<<Building Functions for Evaluation>>\n",
      "\t<<Inference with dropout :0.0000>>\n",
      "\t<<Optimizing variational parameters w/ ADAM>>\n",
      "\t<<Evaluation: Setting opt_method: ADAM, 100 steps w/ 8e-3 lr>>\n",
      "\t<<Inference with dropout :0.0000>>\n",
      "\t<<Optimizing variational parameters w/ ADAM>>\n",
      "\t<<Building Functions for Training>>\n",
      "\t<<Inference with dropout :0.0000>>\n",
      "\t<<Optimizing variational parameters w/ ADAM>>\n",
      "\t<<Modifying : [p_0_W,p_0_b,p_1_W,p_1_b,p_mean_W,p_mean_b]>>\n",
      "\t<<# additional updates: 0>>\n",
      "\t<<Modifying : [q_0_W,q_0_b,q_1_W,q_1_b,q_mu_W,q_logcov_W,q_mu_b,q_logcov_b]>>\n",
      "\t<<Inference with dropout :0.0000>>\n",
      "\t<<Done creating functions for training>>\n",
      "\t<<_buildModel took : 36.2146 seconds>>\n",
      "\t<<Modifying : [p_0_W,p_0_b,p_1_W,p_1_b,p_mean_W,p_mean_b]>>\n"
     ]
    }
   ],
   "source": [
    "default_params['savedir']+='-rcv2-'+default_params['opt_type']\n",
    "createIfAbsent(default_params['savedir'])\n",
    "pfile= default_params['savedir']+'/'+default_params['unique_id']+'-config.pkl'\n",
    "print 'Training model from scratch. Parameters in: ',pfile\n",
    "model = Model(default_params, paramFile = pfile, additional_attrs = additional_attrs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the model\n",
    "* We can now train the model we created\n",
    "* This is the overall setup for the file [`train.py`](train.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "savef      = os.path.join(default_params['savedir'],default_params['unique_id']) #Prefix for saving in checkpoint directory\n",
    "savedata   = Learn.learn( model, \n",
    "                                dataset     = dset['train'],\n",
    "                                epoch_start = 0 , \n",
    "                                epoch_end   = 3,  #epochs -- set w/ default_params['epochs'] \n",
    "                                batch_size  = default_params['batch_size'], #batch size \n",
    "                                savefreq    = default_params['savefreq'], #frequency of saving\n",
    "                                savefile    = savef,\n",
    "                                dataset_eval= dset['valid']\n",
    "                                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for k in savedata:\n",
    "    print k, type(savedata[k]), savedata[k].shape"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}