Action_CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a ConvNet PyTorch\n",
    "\n",
    "In this notebook, you'll learn how to use the powerful PyTorch framework to specify a conv net architecture and train it on the human action recognition dataset. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.autograd import Variable\n",
    "from torch.utils.data import DataLoader,sampler,Dataset\n",
    "import torchvision.datasets as dset\n",
    "import torchvision.transforms as T\n",
    "import timeit\n",
    "from PIL import Image\n",
    "import os\n",
    "import numpy as np\n",
    "import scipy.io\n",
    "import torchvision.models.inception as inception"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's this PyTorch business?\n",
    "\n",
    "* When using a framework like PyTorch or TensorFlow you can harness the power of the GPU for your own custom neural network architectures without having to write CUDA code directly.\n",
    "* this notebook will walk you through much of what you need to do to train models using pytorch. if you want to learn more or need further clarification on topics that aren't fully explained here, here are 2 good Pytorch tutorials. 1): http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html 2)http://pytorch.org/tutorials/beginner/pytorch_with_examples.html\n",
    "* It's not necessary to have a GPU for this homework, using a GPU can make your code run faster.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Datasets\n",
    "\n",
    "In this part, we will load the action recognition dataset for the neural network. In order to load data from our custom dataset, we need to write a custom Dataloader. If you put q3_2_data.mat, /valClips,/trainClips,/testClips under the folder of ./data/ , you do not need to change anything in this part."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, load the labels of the dataset, you should write your path of the q3_2_data.mat file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7770\n",
      "2230\n"
     ]
    }
   ],
   "source": [
    "label_mat=scipy.io.loadmat('./data/q3_2_data.mat')\n",
    "label_train=label_mat['trLb']\n",
    "print(len(label_train))\n",
    "label_val=label_mat['valLb']\n",
    "print(len(label_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset class\n",
    "\n",
    "torch.utils.data.Dataset is an abstract class representing a dataset. The custom dataset should inherit Dataset and override the following methods:\n",
    "\n",
    "    __len__ so that len(dataset) returns the size of the dataset.\n",
    "    __getitem__ to support the indexing such that dataset[i] can be used to get ith sample\n",
    "\n",
    "Let’s create a dataset class for our action recognition dataset. We will read images in __getitem__. This is memory efficient because all the images are not stored in the memory at once but read as required.\n",
    "\n",
    "Sample of our dataset will be a dict {'image':image,'img_path':img_path,'Label':Label}. Our datset will take an optional argument transform so that any required processing can be applied on the sample. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "class ActionDataset(Dataset):\n",
    "    \"\"\"Action dataset.\"\"\"\n",
    "\n",
    "    def __init__(self,  root_dir,labels=[], transform=None):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            root_dir (string): Directory with all the images.\n",
    "            labels(list): labels if images.\n",
    "            transform (callable, optional): Optional transform to be applied on a sample.\n",
    "        \"\"\"\n",
    "        self.root_dir = root_dir\n",
    "        self.transform = transform\n",
    "        self.length=len(os.listdir(self.root_dir))\n",
    "        self.labels=labels\n",
    "    def __len__(self):\n",
    "        return self.length*3\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        \n",
    "        folder=idx/3+1\n",
    "        imidx=idx%3+1\n",
    "        folder=format(folder,'05d')\n",
    "        imgname=str(imidx)+'.jpg'\n",
    "        img_path = os.path.join(self.root_dir,\n",
    "                                folder,imgname)\n",
    "        image = Image.open(img_path)\n",
    "        if len(self.labels)!=0:\n",
    "            Label=self.labels[idx/3][0]-1\n",
    "        if self.transform:\n",
    "            image = self.transform(image)\n",
    "        if len(self.labels)!=0:\n",
    "            sample={'image':image,'img_path':img_path,'Label':Label}\n",
    "        else:\n",
    "            sample={'image':image,'img_path':img_path}\n",
    "        return sample\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Iterating over the dataset by a for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00001/1.jpg\n",
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00001/2.jpg\n",
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00001/3.jpg\n",
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00002/1.jpg\n",
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00002/2.jpg\n",
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00002/3.jpg\n",
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00003/1.jpg\n",
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00003/2.jpg\n",
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00003/3.jpg\n",
      "torch.Size([3, 64, 64])\n",
      "0.0\n",
      "./data/trainClips/00004/1.jpg\n"
     ]
    }
   ],
   "source": [
    "image_dataset=ActionDataset(root_dir='./data/trainClips/',\\\n",
    "                            labels=label_train,transform=T.ToTensor())\n",
    "\n",
    "#iterating though the dataset\n",
    "for i in range(10):\n",
    "    sample=image_dataset[i]\n",
    "    print(sample['image'].shape)\n",
    "    print(sample['Label'])\n",
    "    print(sample['img_path'])\n",
    "     \n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can iterate over the created dataset with a 'for' loop as before. However, we are losing a lot of features by using a simple for loop to iterate over the data. In particular, we are missing out on:\n",
    "\n",
    "* Batching the data\n",
    "* Shuffling the data\n",
    "* Load the data in parallel using multiprocessing workers.\n",
    "\n",
    "torch.utils.data.DataLoader is an iterator which provides all these features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, torch.Size([4, 3, 64, 64]), ['./data/trainClips/05346/3.jpg', './data/trainClips/13846/1.jpg', './data/trainClips/06021/2.jpg', './data/trainClips/11875/2.jpg'], \n",
      " 6\n",
      " 7\n",
      " 7\n",
      " 4\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(1, torch.Size([4, 3, 64, 64]), ['./data/trainClips/00760/3.jpg', './data/trainClips/03196/3.jpg', './data/trainClips/06171/1.jpg', './data/trainClips/03803/3.jpg'], \n",
      " 0\n",
      " 3\n",
      " 7\n",
      " 4\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(2, torch.Size([4, 3, 64, 64]), ['./data/trainClips/13575/1.jpg', './data/trainClips/08375/2.jpg', './data/trainClips/12116/2.jpg', './data/trainClips/07553/2.jpg'], \n",
      " 7\n",
      " 0\n",
      " 5\n",
      " 9\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(3, torch.Size([4, 3, 64, 64]), ['./data/trainClips/15641/1.jpg', './data/trainClips/08955/2.jpg', './data/trainClips/13966/1.jpg', './data/trainClips/15627/3.jpg'], \n",
      " 0\n",
      " 1\n",
      " 7\n",
      " 0\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(4, torch.Size([4, 3, 64, 64]), ['./data/trainClips/16112/2.jpg', './data/trainClips/03895/1.jpg', './data/trainClips/15631/1.jpg', './data/trainClips/06503/1.jpg'], \n",
      " 2\n",
      " 4\n",
      " 0\n",
      " 8\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(5, torch.Size([4, 3, 64, 64]), ['./data/trainClips/08583/2.jpg', './data/trainClips/02591/2.jpg', './data/trainClips/03922/3.jpg', './data/trainClips/04438/3.jpg'], \n",
      " 0\n",
      " 2\n",
      " 4\n",
      " 5\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(6, torch.Size([4, 3, 64, 64]), ['./data/trainClips/07515/1.jpg', './data/trainClips/03073/2.jpg', './data/trainClips/07976/2.jpg', './data/trainClips/01640/3.jpg'], \n",
      " 9\n",
      " 3\n",
      " 0\n",
      " 1\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(7, torch.Size([4, 3, 64, 64]), ['./data/trainClips/05921/3.jpg', './data/trainClips/03903/3.jpg', './data/trainClips/08216/2.jpg', './data/trainClips/12206/3.jpg'], \n",
      " 7\n",
      " 4\n",
      " 0\n",
      " 5\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(8, torch.Size([4, 3, 64, 64]), ['./data/trainClips/11321/3.jpg', './data/trainClips/10832/1.jpg', './data/trainClips/07909/1.jpg', './data/trainClips/05874/2.jpg'], \n",
      " 4\n",
      " 3\n",
      " 0\n",
      " 7\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(9, torch.Size([4, 3, 64, 64]), ['./data/trainClips/16941/3.jpg', './data/trainClips/07516/1.jpg', './data/trainClips/11525/2.jpg', './data/trainClips/03773/1.jpg'], \n",
      " 5\n",
      " 9\n",
      " 4\n",
      " 4\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(10, torch.Size([4, 3, 64, 64]), ['./data/trainClips/00150/3.jpg', './data/trainClips/13194/1.jpg', './data/trainClips/12647/3.jpg', './data/trainClips/12887/2.jpg'], \n",
      " 0\n",
      " 6\n",
      " 5\n",
      " 6\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(11, torch.Size([4, 3, 64, 64]), ['./data/trainClips/02204/2.jpg', './data/trainClips/03897/1.jpg', './data/trainClips/01167/1.jpg', './data/trainClips/03110/1.jpg'], \n",
      " 2\n",
      " 4\n",
      " 1\n",
      " 3\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(12, torch.Size([4, 3, 64, 64]), ['./data/trainClips/02534/1.jpg', './data/trainClips/10641/2.jpg', './data/trainClips/04910/1.jpg', './data/trainClips/09308/3.jpg'], \n",
      " 2\n",
      " 3\n",
      " 5\n",
      " 1\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(13, torch.Size([4, 3, 64, 64]), ['./data/trainClips/15418/2.jpg', './data/trainClips/11762/1.jpg', './data/trainClips/00839/3.jpg', './data/trainClips/03833/2.jpg'], \n",
      " 9\n",
      " 4\n",
      " 0\n",
      " 4\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(14, torch.Size([4, 3, 64, 64]), ['./data/trainClips/11201/1.jpg', './data/trainClips/04135/3.jpg', './data/trainClips/12701/1.jpg', './data/trainClips/09206/3.jpg'], \n",
      " 3\n",
      " 4\n",
      " 5\n",
      " 1\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(15, torch.Size([4, 3, 64, 64]), ['./data/trainClips/16852/3.jpg', './data/trainClips/16635/3.jpg', './data/trainClips/12938/1.jpg', './data/trainClips/07243/1.jpg'], \n",
      " 5\n",
      " 4\n",
      " 6\n",
      " 9\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(16, torch.Size([4, 3, 64, 64]), ['./data/trainClips/06900/1.jpg', './data/trainClips/00971/2.jpg', './data/trainClips/14132/1.jpg', './data/trainClips/17620/3.jpg'], \n",
      " 8\n",
      " 1\n",
      " 7\n",
      " 9\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(17, torch.Size([4, 3, 64, 64]), ['./data/trainClips/08289/1.jpg', './data/trainClips/16871/3.jpg', './data/trainClips/02716/2.jpg', './data/trainClips/07680/2.jpg'], \n",
      " 0\n",
      " 5\n",
      " 3\n",
      " 9\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(18, torch.Size([4, 3, 64, 64]), ['./data/trainClips/10754/3.jpg', './data/trainClips/05368/2.jpg', './data/trainClips/05868/3.jpg', './data/trainClips/16399/2.jpg'], \n",
      " 3\n",
      " 6\n",
      " 7\n",
      " 3\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(19, torch.Size([4, 3, 64, 64]), ['./data/trainClips/14084/1.jpg', './data/trainClips/02405/3.jpg', './data/trainClips/09737/1.jpg', './data/trainClips/10512/1.jpg'], \n",
      " 7\n",
      " 2\n",
      " 2\n",
      " 3\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(20, torch.Size([4, 3, 64, 64]), ['./data/trainClips/02891/3.jpg', './data/trainClips/00681/3.jpg', './data/trainClips/04176/3.jpg', './data/trainClips/13530/1.jpg'], \n",
      " 3\n",
      " 0\n",
      " 4\n",
      " 7\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(21, torch.Size([4, 3, 64, 64]), ['./data/trainClips/00662/1.jpg', './data/trainClips/13266/3.jpg', './data/trainClips/00778/2.jpg', './data/trainClips/04094/1.jpg'], \n",
      " 0\n",
      " 6\n",
      " 0\n",
      " 4\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "image_dataloader = DataLoader(image_dataset, batch_size=4,\n",
    "                        shuffle=True, num_workers=4)\n",
    "\n",
    "\n",
    "for i,sample in enumerate(image_dataloader):\n",
    "    sample['image']=sample['image']\n",
    "    print(i,sample['image'].shape,sample['img_path'],sample['Label'])\n",
    "    if i>20: \n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataloaders for the training, validationg and testing set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "image_dataset_train=ActionDataset(root_dir='./data/trainClips/',labels=label_train,transform=T.ToTensor())\n",
    "\n",
    "image_dataloader_train = DataLoader(image_dataset_train, batch_size=32,\n",
    "                        shuffle=True, num_workers=4)\n",
    "image_dataset_val=ActionDataset(root_dir='./data/valClips/',labels=label_val,transform=T.ToTensor())\n",
    "\n",
    "image_dataloader_val = DataLoader(image_dataset_val, batch_size=32,\n",
    "                        shuffle=False, num_workers=4)\n",
    "image_dataset_test=ActionDataset(root_dir='./data/testClips/',labels=[],transform=T.ToTensor())\n",
    "\n",
    "image_dataloader_test = DataLoader(image_dataset_test, batch_size=32,\n",
    "                        shuffle=False, num_workers=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dtype = torch.FloatTensor # the CPU datatype\n",
    "# Constant to control how frequently we print train loss\n",
    "print_every = 100\n",
    "# This is a little utility that we'll use to reset the model\n",
    "# if we want to re-initialize all our parameters\n",
    "def reset(m):\n",
    "    if hasattr(m, 'reset_parameters'):\n",
    "        m.reset_parameters()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Model\n",
    "\n",
    "### Some assorted tidbits\n",
    "\n",
    "Let's start by looking at a simple model. First, note that PyTorch operates on Tensors, which are n-dimensional arrays functionally analogous to numpy's ndarrays, with the additional feature that they can be used for computations on GPUs.\n",
    "\n",
    "We'll provide you with a Flatten function, which we explain here. Remember that our image data (and more relevantly, our intermediate feature maps) are initially N x C x H x W, where:\n",
    "* N is the number of datapoints\n",
    "* C is the number of image channels. \n",
    "* H is the height of the intermediate feature map in pixels\n",
    "* W is the height of the intermediate feature map in pixels\n",
    "\n",
    "This is the right way to represent the data when we are doing something like a 2D convolution, that needs spatial understanding of where the intermediate features are relative to each other. When we input  data into fully connected affine layers, however, we want each datapoint to be represented by a single vector -- it's no longer useful to segregate the different channels, rows, and columns of the data. So, we use a \"Flatten\" operation to collapse the C x H x W values per representation into a single long vector. The Flatten function below first reads in the N, C, H, and W values from a given batch of data, and then returns a \"view\" of that data. \"View\" is analogous to numpy's \"reshape\" method: it reshapes x's dimensions to be N x ??, where ?? is allowed to be anything (in this case, it will be C x H x W, but we don't need to specify that explicitly). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Flatten(nn.Module):\n",
    "    def forward(self, x):\n",
    "        N, C, H, W = x.size() # read in N, C, H, W\n",
    "        return x.view(N, -1)  # \"flatten\" the C * H * W values into a single vector per image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The example model itself\n",
    "\n",
    "The first step to training your own model is defining its architecture.\n",
    "\n",
    "Here's an example of a convolutional neural network defined in PyTorch -- try to understand what each line is doing, remembering that each layer is composed upon the previous layer. We haven't trained anything yet - that'll come next - for now, we want you to understand how everything gets set up.  nn.Sequential is a container which applies each layer\n",
    "one after the other.\n",
    "\n",
    "In this example, you see 2D convolutional layers (Conv2d), ReLU activations, and fully-connected layers (Linear). You also see the Cross-Entropy loss function, and the Adam optimizer being used. \n",
    "\n",
    "Make sure you understand why the parameters of the Linear layer are 10092 and 10.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Here's where we define the architecture of the model... \n",
    "simple_model = nn.Sequential(\n",
    "                nn.Conv2d(3, 32, kernel_size=7, stride=2),\n",
    "                nn.ReLU(inplace=True),\n",
    "                Flatten(), # see above for explanation\n",
    "                nn.Linear(10092, 10), # affine layer\n",
    "              )\n",
    "\n",
    "# Set the type of all data in this model to be FloatTensor \n",
    "simple_model.type(dtype)\n",
    "\n",
    "loss_fn = nn.CrossEntropyLoss().type(dtype)\n",
    "optimizer = optim.Adam(simple_model.parameters(), lr=1e-2) # lr sets the learning rate of the optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch supports many other layer types, loss functions, and optimizers - you will experiment with these next. Here's the official API documentation for these (if any of the parameters used above were unclear, this resource will also be helpful). \n",
    "\n",
    "* Layers: http://pytorch.org/docs/nn.html\n",
    "* Activations: http://pytorch.org/docs/nn.html#non-linear-activations\n",
    "* Loss functions: http://pytorch.org/docs/nn.html#loss-functions\n",
    "* Optimizers: http://pytorch.org/docs/optim.html#algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a specific model\n",
    "\n",
    "In this section, we're going to specify a model for you to construct. The goal here isn't to get good performance (that'll be next), but instead to get comfortable with understanding the PyTorch documentation and configuring your own model. \n",
    "\n",
    "Using the code provided above as guidance, and using the following PyTorch documentation, specify a model with the following architecture:\n",
    "\n",
    "* 7x7 Convolutional Layer with 8 filters and stride of 1\n",
    "* ReLU Activation Layer\n",
    "* 2x2 Max Pooling layer with a stride of 2\n",
    "* 7x7 Convolutional Layer with 16 filters and stride of 1\n",
    "* ReLU Activation Layer\n",
    "* 2x2 Max Pooling layer with a stride of 2\n",
    "* Flatten the feature map\n",
    "* ReLU Activation Layer\n",
    "* Affine layer to map input units to 10 outputs, you need to figure out the input size here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fixed_model_base = nn.Sequential( \n",
    "    #########1st To Do  (10 points)###################\n",
    "                nn.Conv2d(3, 8, kernel_size=7, stride=1),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.MaxPool2d(2, stride = 2),\n",
    "                nn.Conv2d(8, 16, kernel_size=7, stride=1),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.MaxPool2d(2, stride = 2),\n",
    "                Flatten(),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.Linear(1936, 10)\n",
    "    ####################################\n",
    "            )\n",
    "fixed_model = fixed_model_base.type(dtype)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make sure you're doing the right thing, use the following tool to check the dimensionality of your output (it should be 32 x 10, since our batches have size 32 and the output of the final affine layer should be 10, corresponding to our 10 classes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[32 10]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Now we're going to feed a random batch into the model you defined and make sure the output is the right size\n",
    "x = torch.randn(32, 3, 64, 64).type(dtype)\n",
    "x_var = Variable(x.type(dtype)) # Construct a PyTorch Variable out of your input data\n",
    "ans = fixed_model(x_var)        # Feed it through the model! \n",
    "\n",
    "# Check to make sure what comes out of your model\n",
    "# is the right dimensionality... this should be True\n",
    "# if you've done everything correctly\n",
    "print(np.array(ans.size()))\n",
    "np.array_equal(np.array(ans.size()), np.array([32, 10]))   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model.\n",
    "\n",
    "Now that you've seen how to define a model and do a single forward pass of some data through it, let's  walk through how you'd actually train one whole epoch over your training data (using the fixed_model_base we provided above).\n",
    "\n",
    "Make sure you understand how each PyTorch function used below corresponds to what you implemented in your custom neural network implementation.\n",
    "\n",
    "Note that because we are not resetting the weights anywhere below, if you run the cell multiple times, you are effectively training multiple epochs (so your performance should improve).\n",
    "\n",
    "First, set up an RMSprop optimizer (using a 1e-4 learning rate) and a cross-entropy loss function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "################ 2nd To Do  (5 points)##################\n",
    "optimizer = torch.optim.RMSprop(fixed_model_base.parameters(), lr = 0.0001)\n",
    "#optimizer = torch.optim.Adadelta(fixed_model_base.parameters(), lr = 0.001)\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "#loss_fn = nn.MultiMarginLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t = 100, loss = 1.7533\n",
      "t = 200, loss = 1.7724\n",
      "t = 300, loss = 1.6815\n",
      "t = 400, loss = 1.3511\n",
      "t = 500, loss = 1.3011\n",
      "t = 600, loss = 1.2171\n",
      "t = 700, loss = 1.1594\n"
     ]
    }
   ],
   "source": [
    "# This sets the model in \"training\" mode. \n",
    "# This is relevant for some layers that may have different behavior\n",
    "# in training mode vs testing mode, such as Dropout and BatchNorm. \n",
    "fixed_model.train()\n",
    "\n",
    "# Load one batch at a time.\n",
    "for t, sample in enumerate(image_dataloader_train):\n",
    "    x_var = Variable(sample['image'])\n",
    "    #print(type(x_var.data))\n",
    "    #print(x_var.shape)\n",
    "    y_var = Variable(sample['Label']).long()\n",
    "\n",
    "    # This is the forward pass: predict the scores for each class, for each x in the batch.\n",
    "    scores = fixed_model(x_var)\n",
    "    \n",
    "    # Use the correct y values and the predicted y values to compute the loss.\n",
    "    loss = loss_fn(scores, y_var)\n",
    "    \n",
    "    if (t + 1) % print_every == 0:\n",
    "        print('t = %d, loss = %.4f' % (t + 1, loss.data[0]))\n",
    "\n",
    "    # Zero out all of the gradients for the variables which the optimizer will update.\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # This is the backwards pass: compute the gradient of the loss with respect to each \n",
    "    # parameter of the model.\n",
    "    loss.backward()\n",
    "    \n",
    "    # Actually update the parameters of the model using the gradients computed by the backwards pass.\n",
    "    optimizer.step()\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you've seen how the training process works in PyTorch. To save you writing boilerplate code, we're providing the following helper functions to help you train for multiple epochs and check the accuracy of your model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(model, loss_fn, optimizer, dataloader, num_epochs = 1):\n",
    "    for epoch in range(num_epochs):\n",
    "        print('Starting epoch %d / %d' % (epoch + 1, num_epochs))\n",
    "        check_accuracy(fixed_model, image_dataloader_val)# check accuracy on the training set\n",
    "\n",
    "        model.train()\n",
    "        for t, sample in enumerate(dataloader):\n",
    "            x_var = Variable(sample['image'])\n",
    "            y_var = Variable(sample['Label'].long())\n",
    "\n",
    "            scores = model(x_var)\n",
    "            \n",
    "            loss = loss_fn(scores, y_var)\n",
    "            if (t + 1) % print_every == 0:\n",
    "                print('t = %d, loss = %.4f' % (t + 1, loss.data[0]))\n",
    "\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "def check_accuracy(model, loader):\n",
    "    '''\n",
    "    if loader.dataset.train:\n",
    "        print('Checking accuracy on validation set')\n",
    "    else:\n",
    "        print('Checking accuracy on test set')  \n",
    "    '''\n",
    "    num_correct = 0\n",
    "    num_samples = 0\n",
    "    model.eval() # Put the model in test mode (the opposite of model.train(), essentially)\n",
    "    for t, sample in enumerate(loader):\n",
    "        x_var = Variable(sample['image'])\n",
    "        y_var = sample['Label']\n",
    "        #y_var=y_var.cpu()\n",
    "        scores = model(x_var)\n",
    "        _, preds = scores.data.max(1)#scores.data.cpu().max(1)\n",
    "        #print(preds)\n",
    "        #print(y_var)\n",
    "        num_correct += (preds.numpy() == y_var.numpy()).sum()\n",
    "        num_samples += preds.size(0)\n",
    "    acc = float(num_correct) / num_samples\n",
    "    print('Got %d / %d correct (%.2f)' % (num_correct, num_samples, 100 * acc))\n",
    "    \n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the accuracy of the model.\n",
    "\n",
    "Let's see the train and check_accuracy code in action -- feel free to use these methods when evaluating the models you develop below.\n",
    "\n",
    "You should get a training loss of around 1.0-1.2, and a validation accuracy of around 50-60%. As mentioned above, if you re-run the cells, you'll be training more epochs, so your performance will improve past these numbers.\n",
    "\n",
    "But don't worry about getting these numbers better -- this was just practice before you tackle designing your own model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 1 / 4\n",
      "t = 100, loss = 2.2741\n",
      "t = 200, loss = 2.1686\n",
      "t = 300, loss = 2.1330\n",
      "t = 400, loss = 1.9836\n",
      "t = 500, loss = 1.8171\n",
      "t = 600, loss = 1.8728\n",
      "t = 700, loss = 1.7021\n",
      "Starting epoch 2 / 4\n",
      "t = 100, loss = 1.6225\n",
      "t = 200, loss = 1.2808\n",
      "t = 300, loss = 1.1693\n",
      "t = 400, loss = 1.5054\n",
      "t = 500, loss = 0.9549\n",
      "t = 600, loss = 1.3470\n",
      "t = 700, loss = 1.2962\n",
      "Starting epoch 3 / 4\n",
      "t = 100, loss = 1.3634\n",
      "t = 200, loss = 1.1321\n",
      "t = 300, loss = 0.9616\n",
      "t = 400, loss = 1.0537\n",
      "t = 500, loss = 0.5659\n",
      "t = 600, loss = 1.3193\n",
      "t = 700, loss = 0.7227\n",
      "Starting epoch 4 / 4\n",
      "t = 100, loss = 1.0016\n",
      "t = 200, loss = 0.9088\n",
      "t = 300, loss = 0.7913\n",
      "t = 400, loss = 1.1037\n",
      "t = 500, loss = 0.9311\n",
      "t = 600, loss = 0.6950\n",
      "t = 700, loss = 0.8325\n",
      "Got 16979 / 23310 correct (72.84)\n"
     ]
    }
   ],
   "source": [
    "torch.random.manual_seed(54321)\n",
    "fixed_model.cpu()\n",
    "fixed_model.apply(reset) \n",
    "fixed_model.train() \n",
    "train(fixed_model, loss_fn, optimizer,image_dataloader_train, num_epochs=4) \n",
    "check_accuracy(fixed_model, image_dataloader_train)# check accuracy on the training set\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Don't forget the validation set!\n",
    "\n",
    "And note that you can use the check_accuracy function to evaluate on the validation set, by passing **image_dataloader_val** as the second argument to check_accuracy. The accuracy on validation set is arround 40-50%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got 3356 / 6690 correct (50.16)\n"
     ]
    }
   ],
   "source": [
    "check_accuracy(fixed_model, image_dataloader_val)#check accuracy on the validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Train a better  model for action recognition!\n",
    "\n",
    "Now it's your job to experiment with architectures, hyperparameters, loss functions, and optimizers to train a model that achieves better accuracy on the action recognition **validation** set. You can use the check_accuracy and train functions from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "###########3rd To Do (16 points, must submit the results to Kaggle) ##############\n",
    "# Train your model here, and make sure the output of this cell is the accuracy of your best model on the \n",
    "# train, val, and test sets. Here's some code to get you started. The output of this cell should be the training\n",
    "# and validation accuracy on your best model (measured by validation accuracy).\n",
    "fixed_model_base = nn.Sequential( \n",
    "                nn.Conv2d(3, 200, kernel_size=10, stride=3),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.MaxPool2d(3, stride = 1),\n",
    "                nn.BatchNorm2d(200),\n",
    "                nn.Dropout2d(0.1),\n",
    "                nn.Conv2d(200, 100, kernel_size=5, stride=2),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.MaxPool2d(3, stride = 1),\n",
    "                nn.BatchNorm2d(100),\n",
    "                nn.Dropout2d(0.2),\n",
    "                nn.Conv2d(100, 50, kernel_size=3, stride=1),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.MaxPool2d(2, stride=1),\n",
    "                nn.BatchNorm2d(50),\n",
    "                Flatten(),\n",
    "                nn.Linear(200, 100),\n",
    "                nn.Linear(100, 50),\n",
    "                nn.Linear(50, 10),\n",
    "                nn.LogSoftmax()\n",
    "    ####################################\n",
    "            )\n",
    "\n",
    "\n",
    "fixed_model = fixed_model_base.type(dtype)\n",
    "\n",
    "optimizer = torch.optim.RMSprop(fixed_model_base.parameters(), lr = 0.0001)\n",
    "#optimizer = torch.optim.Adadelta(fixed_model_base.parameters(), lr = 0.001)\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Describe what you did \n",
    "\n",
    "In the cell below you should write an explanation of what you did, any additional features that you implemented, and any visualizations or graphs that you make in the process of training and evaluating your network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tell us here!\n",
    "########### 4th To Do (4 points) ##############\n",
    "* 10X10 Convolution layer with 200 filters with stride 3\n",
    "* ReLU layer\n",
    "* Max Pool layer with window size 3X3 with stride 1\n",
    "* Batch Norm layer with input size 200\n",
    "* Dropout layer with penalty 0.1\n",
    "* 5X5 Convolution layer with 100 filters with stride 2\n",
    "* ReLU layer\n",
    "* Max Pool layer with window size 3X3 with stride 1\n",
    "* Batch Norm layer with input size 100\n",
    "* Dropout layer with penalty 0.2\n",
    "* 3X3 Convolution layer with 50 filters and stride 1\n",
    "* ReLU layer\n",
    "* Max Pool layer with window size 2 and stride 1\n",
    "* Batch Norm layer with input size 50\n",
    "* Flatten\n",
    "* affine layer to reduce inputs from 200 to 100\n",
    "* affine layer to reduce inputs from 100 to 50\n",
    "* affine layer to reduce inputs from 50 to 10\n",
    "* logsoftmaxing layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the model and submit on Kaggle\n",
    "Testing the model on the testing set and save the results as a .csv file. \n",
    "Please submitted the results.csv file generated by predict_on_test() to Kaggle(https://www.kaggle.com/c/cse512springhw3) to see how well your network performs on the test set. \n",
    "#######5th To Do (submit the result to Kaggle,the highest 3 entries get extra 10 points )###############\n",
    "\n",
    "* Rank: 10\n",
    "* Score: 70.34658"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[32 10]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Now we're going to feed a random batch into the model you defined and make sure the output is the right size\n",
    "x = torch.randn(32, 3, 64, 64).type(dtype)\n",
    "x_var = Variable(x.type(dtype)) # Construct a PyTorch Variable out of your input data\n",
    "ans = fixed_model(x_var)        # Feed it through the model! \n",
    "\n",
    "# Check to make sure what comes out of your model\n",
    "# is the right dimensionality... this should be True\n",
    "# if you've done everything correctly\n",
    "print(np.array(ans.size()))\n",
    "np.array_equal(np.array(ans.size()), np.array([32, 10]))   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 1 / 12\n",
      "Got 840 / 6690 correct (12.56)\n",
      "t = 100, loss = 1.9508\n",
      "t = 200, loss = 1.9221\n",
      "t = 300, loss = 1.8645\n",
      "t = 400, loss = 1.6400\n",
      "t = 500, loss = 1.5899\n",
      "t = 600, loss = 1.6505\n",
      "t = 700, loss = 1.5328\n",
      "Starting epoch 2 / 12\n",
      "Got 3147 / 6690 correct (47.04)\n",
      "t = 100, loss = 1.5677\n",
      "t = 200, loss = 1.4366\n",
      "t = 300, loss = 1.5529\n",
      "t = 400, loss = 1.3599\n",
      "t = 500, loss = 1.4940\n",
      "t = 600, loss = 1.4985\n",
      "t = 700, loss = 1.4644\n",
      "Starting epoch 3 / 12\n",
      "Got 3410 / 6690 correct (50.97)\n",
      "t = 100, loss = 1.3165\n",
      "t = 200, loss = 1.3851\n",
      "t = 300, loss = 1.3073\n",
      "t = 400, loss = 1.2537\n",
      "t = 500, loss = 1.2312\n",
      "t = 600, loss = 1.2057\n",
      "t = 700, loss = 1.2748\n",
      "Starting epoch 4 / 12\n",
      "Got 3935 / 6690 correct (58.82)\n",
      "t = 100, loss = 1.1150\n",
      "t = 200, loss = 1.1912\n",
      "t = 300, loss = 1.1928\n",
      "t = 400, loss = 1.2118\n",
      "t = 500, loss = 1.1382\n",
      "t = 600, loss = 1.1113\n",
      "t = 700, loss = 1.1510\n",
      "Starting epoch 5 / 12\n",
      "Got 3989 / 6690 correct (59.63)\n",
      "t = 100, loss = 1.2359\n",
      "t = 200, loss = 1.2185\n",
      "t = 300, loss = 1.3121\n",
      "t = 400, loss = 1.1823\n",
      "t = 500, loss = 1.1608\n",
      "t = 600, loss = 1.2905\n",
      "t = 700, loss = 1.1246\n",
      "Starting epoch 6 / 12\n",
      "Got 4168 / 6690 correct (62.30)\n",
      "t = 100, loss = 1.1691\n",
      "t = 200, loss = 1.1006\n",
      "t = 300, loss = 1.0614\n",
      "t = 400, loss = 1.1612\n",
      "t = 500, loss = 1.1206\n",
      "t = 600, loss = 1.0532\n",
      "t = 700, loss = 1.0436\n",
      "Starting epoch 7 / 12\n",
      "Got 4074 / 6690 correct (60.90)\n",
      "t = 100, loss = 1.0467\n",
      "t = 200, loss = 1.0700\n",
      "t = 300, loss = 1.0069\n",
      "t = 400, loss = 1.1824\n",
      "t = 500, loss = 1.2059\n",
      "t = 600, loss = 1.0909\n",
      "t = 700, loss = 1.1460\n",
      "Starting epoch 8 / 12\n",
      "Got 4442 / 6690 correct (66.40)\n",
      "t = 100, loss = 1.1749\n",
      "t = 200, loss = 1.0559\n",
      "t = 300, loss = 0.9741\n",
      "t = 400, loss = 1.0383\n",
      "t = 500, loss = 1.0966\n",
      "t = 600, loss = 1.0737\n",
      "t = 700, loss = 1.0539\n",
      "Starting epoch 9 / 12\n",
      "Got 4107 / 6690 correct (61.39)\n",
      "t = 100, loss = 1.1125\n",
      "t = 200, loss = 1.0154\n",
      "t = 300, loss = 1.0778\n",
      "t = 400, loss = 0.9919\n",
      "t = 500, loss = 1.1474\n",
      "t = 600, loss = 1.0799\n",
      "t = 700, loss = 0.9521\n",
      "Starting epoch 10 / 12\n",
      "Got 4108 / 6690 correct (61.41)\n",
      "t = 100, loss = 1.0542\n",
      "t = 200, loss = 0.9333\n",
      "t = 300, loss = 0.9560\n",
      "t = 400, loss = 0.9869\n",
      "t = 500, loss = 0.9713\n",
      "t = 600, loss = 0.9696\n",
      "t = 700, loss = 0.9646\n",
      "Starting epoch 11 / 12\n",
      "Got 4424 / 6690 correct (66.13)\n",
      "t = 100, loss = 1.0716\n",
      "t = 200, loss = 1.0256\n",
      "t = 300, loss = 1.0024\n",
      "t = 400, loss = 1.0990\n",
      "t = 500, loss = 0.9405\n",
      "t = 600, loss = 1.0760\n",
      "t = 700, loss = 0.9582\n",
      "Starting epoch 12 / 12\n",
      "Got 4183 / 6690 correct (62.53)\n",
      "t = 100, loss = 0.9074\n",
      "t = 200, loss = 1.0102\n",
      "t = 300, loss = 1.0446\n",
      "t = 400, loss = 1.0196\n",
      "t = 500, loss = 0.9676\n",
      "t = 600, loss = 0.9586\n",
      "t = 700, loss = 1.0493\n",
      "Got 22453 / 23310 correct (96.32)\n"
     ]
    }
   ],
   "source": [
    "torch.random.manual_seed(54321)\n",
    "fixed_model.cpu()\n",
    "fixed_model.apply(reset) \n",
    "fixed_model.train() \n",
    "train(fixed_model, loss_fn, optimizer,image_dataloader_train, num_epochs=12) \n",
    "check_accuracy(fixed_model, image_dataloader_train)# check accuracy on the training set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got 4292 / 6690 correct (64.16)\n"
     ]
    }
   ],
   "source": [
    "check_accuracy(fixed_model, image_dataloader_val)# check accuracy on the training set\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Things you should try:\n",
    "- **Filter size**: Above we used 7x7; this makes pretty pictures but smaller filters may be more efficient\n",
    "- **Number of filters**: Do more or fewer do better?\n",
    "- **Pooling vs Strided Convolution**: Do you use max pooling or just stride convolutions?\n",
    "- **Batch normalization**: Try adding spatial batch normalization after convolution layers and vanilla batch normalization after affine layers. Do your networks train faster?\n",
    "- **Network architecture**: The network above has two layers of trainable parameters. Can you do better with a deep network? Good architectures to try include:\n",
    "    - [conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]\n",
    "    - [conv-relu-conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]\n",
    "    - [batchnorm-relu-conv]xN -> [affine]xM -> [softmax or SVM]\n",
    "- **Global Average Pooling**: Instead of flattening and then having multiple affine layers, perform convolutions until your image gets small (7x7 or so) and then perform an average pooling operation to get to a 1x1 image picture (1, 1 , Filter#), which is then reshaped into a (Filter#) vector. This is used in [Google's Inception Network](https://arxiv.org/abs/1512.00567) (See Table 1 for their architecture).\n",
    "- **Regularization**: Add l2 weight regularization, or perhaps use Dropout.\n",
    "\n",
    "### Tips for training\n",
    "For each network architecture that you try, you should tune the learning rate and regularization strength. When doing this there are a couple important things to keep in mind:\n",
    "\n",
    "- If the parameters are working well, you should see improvement within a few hundred iterations\n",
    "- Remember the coarse-to-fine approach for hyperparameter tuning: start by testing a large range of hyperparameters for just a few training iterations to find the combinations of parameters that are working at all.\n",
    "- Once you have found some sets of parameters that seem to work, search more finely around these parameters. You may need to train for more epochs.\n",
    "- You should use the validation set for hyperparameter search, and save your test set for evaluating your architecture on the best parameters as selected by the validation set.\n",
    "\n",
    "### Going above and beyond\n",
    "If you are feeling adventurous there are many other features you can implement to try and improve your performance. You are **not required** to implement any of these; however they would be good things to try.\n",
    "\n",
    "- Alternative update steps: For the assignment we implemented SGD+momentum, RMSprop, and Adam; you could try alternatives like AdaGrad or AdaDelta.\n",
    "- Alternative activation functions such as leaky ReLU, parametric ReLU, ELU, or MaxOut.\n",
    "- Model ensembles\n",
    "- Data augmentation\n",
    "- New Architectures\n",
    "  - [ResNets](https://arxiv.org/abs/1512.03385) where the input from the previous layer is added to the output.\n",
    "  - [DenseNets](https://arxiv.org/abs/1608.06993) where inputs into previous layers are concatenated together.\n",
    "  - [This blog has an in-depth overview](https://chatbotslife.com/resnets-highwaynets-and-densenets-oh-my-9bb15918ee32)\n",
    "\n",
    "If you do decide to implement something extra, clearly describe it in the \"Extra Credit Description\" cell below.\n",
    "\n",
    "### What we expect\n",
    "At the very least, you should be able to train a ConvNet that gets at least 55% accuracy on the validation set. This is just a lower bound - if you are careful it should be possible to get accuracies much higher than that! Extra credit points will be awarded for particularly high-scoring models or unique approaches.\n",
    "\n",
    "You should use the space below to experiment and train your network. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train(fixed_model_base, loss_fn, optimizer,image_dataloader_train, num_epochs=4) \n",
    "check_accuracy(fixed_model, image_dataloader_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPU! (This part is optional, 0 points)\n",
    "\n",
    "If you have access to GPU, you can make the code run on GPU, it would be much faster. \n",
    "\n",
    "Now, we're going to switch the dtype of the model and our data to the GPU-friendly tensors, and see what happens... everything is the same, except we are casting our model and input tensors as this new dtype instead of the old one.\n",
    "\n",
    "If this returns false, or otherwise fails in a not-graceful way (i.e., with some error message), you may not have an NVIDIA GPU available on your machine. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verify that CUDA is properly configured and you have a GPU available\n",
    "\n",
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import copy\n",
    "gpu_dtype = torch.cuda.FloatTensor\n",
    "\n",
    "fixed_model_gpu = copy.deepcopy(fixed_model_base)#.type(gpu_dtype)\n",
    "fixed_model_gpu.cuda()\n",
    "x_gpu = torch.randn(4, 3, 64, 64).cuda()#.type(gpu_dtype)\n",
    "x_var_gpu = Variable(x_gpu)#type(gpu_dtype)) # Construct a PyTorch Variable out of your input data\n",
    "ans = fixed_model_gpu(x_var_gpu)        # Feed it through the model! \n",
    "\n",
    "# Check to make sure what comes out of your model\n",
    "# is the right dimensionality... this should be True\n",
    "# if you've done everything correctly\n",
    "np.array_equal(np.array(ans.size()), np.array([4, 10]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to evaluate the performance of the forward pass running on the CPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 3: 667 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit \n",
    "ans = fixed_model(x_var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and now the GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 loops, best of 3: 4.32 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit \n",
    "torch.cuda.synchronize() # Make sure there are no pending GPU computations\n",
    "ans = fixed_model_gpu(x_var_gpu)        # Feed it through the model! \n",
    "torch.cuda.synchronize() # Make sure there are no pending GPU computations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should observe that even a simple forward pass like this is significantly faster on the GPU. So for the rest of the assignment (and when you go train your models in assignment 3 and your project!), you should use the GPU datatype for your model and your tensors: as a reminder that is *torch.cuda.FloatTensor* (in our notebook here as *gpu_dtype*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make the loss function and training variables to GPU friendly format by '.cuda()'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss_fn = nn.CrossEntropyLoss().cuda()\n",
    "optimizer = optim.RMSprop(fixed_model_gpu.parameters(), lr=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(model, loss_fn, optimizer, dataloader, num_epochs = 1):\n",
    "    for epoch in range(num_epochs):\n",
    "        print('Starting epoch %d / %d' % (epoch + 1, num_epochs))\n",
    "        model.train()\n",
    "        check_accuracy(fixed_model_gpu, image_dataloader_val)# check accuracy on the training set\n",
    "        for t, sample in enumerate(dataloader):\n",
    "            x_var = Variable(sample['image'].cuda())\n",
    "            y_var = Variable(sample['Label'].cuda().long())\n",
    "\n",
    "            scores = model(x_var)\n",
    "            \n",
    "            loss = loss_fn(scores, y_var)\n",
    "            if (t + 1) % print_every == 0:\n",
    "                print('t = %d, loss = %.4f' % (t + 1, loss.data[0]))\n",
    "\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "def check_accuracy(model, loader):\n",
    "    '''\n",
    "    if loader.dataset.train:\n",
    "        print('Checking accuracy on validation set')\n",
    "    else:\n",
    "        print('Checking accuracy on test set')  \n",
    "    '''\n",
    "    num_correct = 0\n",
    "    num_samples = 0\n",
    "    model.eval() # Put the model in test mode (the opposite of model.train(), essentially)\n",
    "    for t, sample in enumerate(loader):\n",
    "        x_var = Variable(sample['image'].cuda())\n",
    "        y_var = sample['Label'].cuda()\n",
    "        y_var=y_var.cpu()\n",
    "        scores = model(x_var)\n",
    "        _, preds = scores.data.cpu().max(1)\n",
    "        #print(preds)\n",
    "        #print(y_var)\n",
    "        num_correct += (preds.numpy() == y_var.numpy()).sum()\n",
    "        num_samples += preds.size(0)\n",
    "    acc = float(num_correct) / num_samples\n",
    "    print('Got %d / %d correct (%.2f)' % (num_correct, num_samples, 100 * acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run on GPU!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 1 / 4\n",
      "Got 840 / 6690 correct (12.56)\n",
      "t = 100, loss = 1.3573\n",
      "t = 200, loss = 1.2604\n",
      "t = 300, loss = 1.1004\n",
      "t = 400, loss = 0.6845\n",
      "t = 500, loss = 0.7816\n",
      "t = 600, loss = 0.1433\n",
      "t = 700, loss = 0.3777\n",
      "Starting epoch 2 / 4\n",
      "Got 3688 / 6690 correct (55.13)\n",
      "t = 100, loss = 0.2311\n",
      "t = 200, loss = 0.3680\n",
      "t = 300, loss = 0.3695\n",
      "t = 400, loss = 0.1617\n",
      "t = 500, loss = 0.1303\n",
      "t = 600, loss = 0.2555\n",
      "t = 700, loss = 0.0288\n",
      "Starting epoch 3 / 4\n",
      "Got 3297 / 6690 correct (49.28)\n",
      "t = 100, loss = 0.0734\n",
      "t = 200, loss = 0.1286\n",
      "t = 300, loss = 0.0804\n",
      "t = 400, loss = 0.0252\n",
      "t = 500, loss = 0.2002\n",
      "t = 600, loss = 0.0100\n",
      "t = 700, loss = 0.0381\n",
      "Starting epoch 4 / 4\n",
      "Got 3737 / 6690 correct (55.86)\n",
      "t = 100, loss = 0.0790\n",
      "t = 200, loss = 0.1109\n",
      "t = 300, loss = 0.0735\n",
      "t = 400, loss = 0.0238\n",
      "t = 500, loss = 0.0115\n",
      "t = 600, loss = 0.0069\n",
      "t = 700, loss = 0.0017\n",
      "Got 22885 / 23310 correct (98.18)\n"
     ]
    }
   ],
   "source": [
    "torch.cuda.random.manual_seed(873271)\n",
    "\n",
    "fixed_model_gpu.apply(reset) \n",
    "fixed_model_gpu.train() \n",
    "train(fixed_model_gpu, loss_fn, optimizer,image_dataloader_train, num_epochs=4) \n",
    "check_accuracy(fixed_model_gpu, image_dataloader_train)# check accuracy on the training set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got 3957 / 6690 correct (59.15)\n"
     ]
    }
   ],
   "source": [
    "check_accuracy(fixed_model_gpu, image_dataloader_val)# check accuracy on the training set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9810\n"
     ]
    }
   ],
   "source": [
    "def predict_on_test(model, loader):\n",
    "    '''\n",
    "    if loader.dataset.train:\n",
    "        print('Checking accuracy on validation set')\n",
    "    else:\n",
    "        print('Checking accuracy on test set')  \n",
    "    '''\n",
    "    num_correct = 0\n",
    "    num_samples = 0\n",
    "    model.eval() # Put the model in test mode (the opposite of model.train(), essentially)\n",
    "    results=open('results.csv','w')\n",
    "    count=0\n",
    "    results.write('Id'+','+'Class'+'\\n')\n",
    "    for t, sample in enumerate(loader):\n",
    "        x_var = Variable(sample['image'])\n",
    "        scores = model(x_var)\n",
    "        _, preds = scores.data.max(1)\n",
    "        for i in range(len(preds)):\n",
    "            results.write(str(count)+','+str(preds[i])+'\\n')\n",
    "            count+=1\n",
    "    results.close()\n",
    "    return count\n",
    "    \n",
    "count=predict_on_test(fixed_model, image_dataloader_test)\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3D Convolution on video clips (25 points+10 extra points)\n",
    "3D convolution is for videos, it has one more dimension than 2d convolution. You can find the document for 3D convolution here http://pytorch.org/docs/master/nn.html#torch.nn.Conv3dIn. In our dataset, each clip is a video of 3 frames. Lets classify the each clip rather than each image using 3D convolution.\n",
    "We offer the data loader, the train_3d and check_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00001\n",
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00002\n",
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00003\n",
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00004\n",
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00005\n",
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00006\n",
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00007\n",
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00008\n",
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00009\n",
      "torch.Size([3, 3, 64, 64])\n",
      "0.0\n",
      "00010\n"
     ]
    }
   ],
   "source": [
    "class ActionClipDataset(Dataset):\n",
    "    \"\"\"Action Landmarks dataset.\"\"\"\n",
    "\n",
    "    def __init__(self,  root_dir,labels=[], transform=None):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            csv_file (string): Path to the csv file with annotations.\n",
    "            root_dir (string): Directory with all the images.\n",
    "            transform (callable, optional): Optional transform to be applied\n",
    "                on a sample.\n",
    "        \"\"\"\n",
    "        \n",
    "        self.root_dir = root_dir\n",
    "        self.transform = transform\n",
    "        self.length=len(os.listdir(self.root_dir))\n",
    "        self.labels=labels\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.length\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        \n",
    "        folder=idx+1\n",
    "        folder=format(folder,'05d')\n",
    "        clip=[]\n",
    "        if len(self.labels)!=0:\n",
    "            Label=self.labels[idx][0]-1\n",
    "        for i in range(3):\n",
    "            imidx=i+1\n",
    "            imgname=str(imidx)+'.jpg'\n",
    "            img_path = os.path.join(self.root_dir,\n",
    "                                    folder,imgname)\n",
    "            image = Image.open(img_path)\n",
    "            image=np.array(image)\n",
    "            clip.append(image)\n",
    "        if self.transform:\n",
    "            clip=np.asarray(clip)\n",
    "            clip=np.transpose(clip, (0,3,1,2))\n",
    "            clip = torch.from_numpy(np.asarray(clip))\n",
    "        if len(self.labels)!=0:\n",
    "            sample={'clip':clip,'Label':Label,'folder':folder}\n",
    "        else:\n",
    "            sample={'clip':clip,'folder':folder}\n",
    "        return sample\n",
    "\n",
    "clip_dataset=ActionClipDataset(root_dir='./data/trainClips/',\\\n",
    "                               labels=label_train,transform=T.ToTensor())#/home/tqvinh/Study/CSE512/cse512-s18/hw2data/trainClips/\n",
    "for i in range(10):\n",
    "    sample=clip_dataset[i]\n",
    "    print(sample['clip'].shape)\n",
    "    print(sample['Label'])\n",
    "    print(sample['folder'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, torch.Size([4, 3, 3, 64, 64]), ['03235', '05470', '03868', '05685'], \n",
      " 3\n",
      " 6\n",
      " 4\n",
      " 7\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(1, torch.Size([4, 3, 3, 64, 64]), ['02698', '07158', '01084', '01569'], \n",
      " 3\n",
      " 9\n",
      " 1\n",
      " 1\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(2, torch.Size([4, 3, 3, 64, 64]), ['03901', '05320', '03313', '06915'], \n",
      " 4\n",
      " 6\n",
      " 3\n",
      " 8\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(3, torch.Size([4, 3, 3, 64, 64]), ['04355', '06344', '05925', '05250'], \n",
      " 5\n",
      " 7\n",
      " 7\n",
      " 6\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(4, torch.Size([4, 3, 3, 64, 64]), ['06595', '03737', '07300', '03475'], \n",
      " 8\n",
      " 4\n",
      " 9\n",
      " 3\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(5, torch.Size([4, 3, 3, 64, 64]), ['03213', '02172', '03751', '03010'], \n",
      " 3\n",
      " 2\n",
      " 4\n",
      " 3\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(6, torch.Size([4, 3, 3, 64, 64]), ['05076', '00565', '06546', '05880'], \n",
      " 6\n",
      " 0\n",
      " 8\n",
      " 7\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(7, torch.Size([4, 3, 3, 64, 64]), ['06666', '03632', '01269', '01160'], \n",
      " 8\n",
      " 4\n",
      " 1\n",
      " 1\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(8, torch.Size([4, 3, 3, 64, 64]), ['00282', '00221', '03597', '03103'], \n",
      " 0\n",
      " 0\n",
      " 4\n",
      " 3\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(9, torch.Size([4, 3, 3, 64, 64]), ['01589', '01557', '04384', '07519'], \n",
      " 1\n",
      " 1\n",
      " 5\n",
      " 9\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(10, torch.Size([4, 3, 3, 64, 64]), ['00068', '03450', '05545', '01511'], \n",
      " 0\n",
      " 3\n",
      " 6\n",
      " 1\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(11, torch.Size([4, 3, 3, 64, 64]), ['04882', '04138', '05031', '02033'], \n",
      " 5\n",
      " 4\n",
      " 6\n",
      " 2\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(12, torch.Size([4, 3, 3, 64, 64]), ['05983', '07330', '01013', '00879'], \n",
      " 7\n",
      " 9\n",
      " 1\n",
      " 0\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(13, torch.Size([4, 3, 3, 64, 64]), ['01141', '07084', '00133', '01475'], \n",
      " 1\n",
      " 8\n",
      " 0\n",
      " 1\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(14, torch.Size([4, 3, 3, 64, 64]), ['00526', '02953', '00091', '06346'], \n",
      " 0\n",
      " 3\n",
      " 0\n",
      " 7\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(15, torch.Size([4, 3, 3, 64, 64]), ['02010', '03289', '04859', '01188'], \n",
      " 2\n",
      " 3\n",
      " 5\n",
      " 1\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(16, torch.Size([4, 3, 3, 64, 64]), ['00110', '06602', '03389', '04565'], \n",
      " 0\n",
      " 8\n",
      " 3\n",
      " 5\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(17, torch.Size([4, 3, 3, 64, 64]), ['02794', '00412', '05322', '05751'], \n",
      " 3\n",
      " 0\n",
      " 6\n",
      " 7\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(18, torch.Size([4, 3, 3, 64, 64]), ['05043', '00267', '02612', '01703'], \n",
      " 6\n",
      " 0\n",
      " 2\n",
      " 1\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(19, torch.Size([4, 3, 3, 64, 64]), ['05842', '01014', '07196', '06772'], \n",
      " 7\n",
      " 1\n",
      " 9\n",
      " 8\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(20, torch.Size([4, 3, 3, 64, 64]), ['01050', '01815', '05526', '02789'], \n",
      " 1\n",
      " 1\n",
      " 6\n",
      " 3\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n",
      "(21, torch.Size([4, 3, 3, 64, 64]), ['05750', '01722', '07217', '03292'], \n",
      " 7\n",
      " 1\n",
      " 9\n",
      " 3\n",
      "[torch.DoubleTensor of size 4]\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "clip_dataloader = DataLoader(clip_dataset, batch_size=4,\n",
    "                        shuffle=True, num_workers=4)\n",
    "\n",
    "\n",
    "for i,sample in enumerate(clip_dataloader):\n",
    "    print(i,sample['clip'].shape,sample['folder'],sample['Label'])\n",
    "    if i>20: \n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clip_dataset_train=ActionClipDataset(root_dir='./data/trainClips/',labels=label_train,transform=T.ToTensor())\n",
    "\n",
    "clip_dataloader_train = DataLoader(clip_dataset_train, batch_size=16,\n",
    "                        shuffle=True, num_workers=4)\n",
    "clip_dataset_val=ActionClipDataset(root_dir='./data/valClips/',labels=label_val,transform=T.ToTensor())\n",
    "\n",
    "clip_dataloader_val = DataLoader(clip_dataset_val, batch_size=16,\n",
    "                        shuffle=True, num_workers=4)\n",
    "clip_dataset_test=ActionClipDataset(root_dir='./data/testClips/',labels=[],transform=T.ToTensor())\n",
    "\n",
    "clip_dataloader_test = DataLoader(clip_dataset_test, batch_size=16,\n",
    "                        shuffle=False, num_workers=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write the Flatten for 3d covolution feature maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Flatten3d(nn.Module):\n",
    "    def forward(self, x):\n",
    "        ###############6th To Do (5 points)###################\n",
    "        N, C, D, H, W = x.size() # read in N, C, D, H, W\n",
    "        return x.view(N, -1)  # \"flatten\" the C * D * H * W values into a single vector per image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Design a network using 3D convolution on videos for video classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fixed_model_3d = nn.Sequential( # You fill this in!\n",
    "    ###############7th To Do (16 points)#########################\n",
    "    nn.Conv3d(in_channels = 3, out_channels = 50, kernel_size = 2, stride = 1),\n",
    "    nn.ReLU(inplace=True),\n",
    "    nn.MaxPool3d((1, 2, 2), stride = 2),\n",
    "    nn.Conv3d(in_channels = 50, out_channels = 100, kernel_size = (1, 3, 3), stride = 1),\n",
    "    nn.ReLU(inplace = True),\n",
    "    nn.MaxPool3d((1, 3, 3), stride = 2),\n",
    "    nn.Dropout3d(0.1),\n",
    "    Flatten3d(),\n",
    "    nn.ReLU(inplace=True),\n",
    "    nn.Linear(19600, 10),\n",
    "    nn.LogSoftmax()\n",
    "    ###############################   \n",
    ")\n",
    "\n",
    "fixed_model_3d = fixed_model_3d.type(dtype)\n",
    "x = torch.randn(32,3, 3, 64, 64).type(dtype)\n",
    "x_var = Variable(x).type(dtype) # Construct a PyTorch Variable out of your input data\n",
    "ans = fixed_model_3d(x_var) \n",
    "np.array_equal(np.array(ans.size()), np.array([32, 10]))\n",
    "#Accuracy 62 iterations 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Describe what you did (4 points)\n",
    "\n",
    "In the cell below you should write an explanation of what you did, any additional features that you implemented, and any visualizations or graphs that you make in the process of training and evaluating your network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8th To Do\n",
    "Tell us here: \n",
    "* 2X2X2 Convolution layer with 50 filters\n",
    "* ReLU layer inplace True\n",
    "* Max Pooling layer with window size (1, 2, 2) stride = 2\n",
    "* 1X3X3 Convolution layer with 100 filters\n",
    "* ReLU layer with inplace True \n",
    "* Max Pooling layer with window size (1, 3, 3) stride = 2\n",
    "* dropout layer with penalty 0.1\n",
    "* flattening\n",
    "* ReLU layer with inplace True\n",
    "* Affine layer\n",
    "* LogSoftmax Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss_fn = nn.CrossEntropyLoss().type(dtype)\n",
    "optimizer = optim.RMSprop(fixed_model_3d.parameters(), lr=1e-4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_3d(model, loss_fn, optimizer,dataloader,num_epochs = 1):\n",
    "    for epoch in range(num_epochs):\n",
    "        print('Starting epoch %d / %d' % (epoch + 1, num_epochs))\n",
    "        check_accuracy_3d(fixed_model_3d, clip_dataloader_val)\n",
    "        model.train()\n",
    "        for t, sample in enumerate(dataloader):\n",
    "            x_var = Variable(sample['clip'].type(dtype))\n",
    "            y_var = Variable(sample['Label'].type(dtype).long())\n",
    "\n",
    "            scores = model(x_var)\n",
    "            \n",
    "            loss = loss_fn(scores, y_var)\n",
    "            if (t + 1) % print_every == 0:\n",
    "                print('t = %d, loss = %.4f' % (t + 1, loss.data[0]))\n",
    "\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "def check_accuracy_3d(model, loader):\n",
    "    '''\n",
    "    if loader.dataset.train:\n",
    "        print('Checking accuracy on validation set')\n",
    "    else:\n",
    "        print('Checking accuracy on test set')  \n",
    "    '''\n",
    "    num_correct = 0\n",
    "    num_samples = 0\n",
    "    model.eval() # Put the model in test mode (the opposite of model.train(), essentially)\n",
    "    for t, sample in enumerate(loader):\n",
    "        x_var = Variable(sample['clip'].type(dtype))\n",
    "        y_var = sample['Label'].type(dtype)\n",
    "        y_var=y_var.cpu()\n",
    "        scores = model(x_var)\n",
    "        _, preds = scores.data.cpu().max(1)\n",
    "        #print(preds)\n",
    "        #print(y_var)\n",
    "        num_correct += (preds.numpy() == y_var.numpy()).sum()\n",
    "        num_samples += preds.size(0)\n",
    "    acc = float(num_correct) / num_samples\n",
    "    print('Got %d / %d correct (%.2f)' % (num_correct, num_samples, 100 * acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 1 / 5\n",
      "Got 273 / 2230 correct (12.24)\n",
      "t = 100, loss = 1.7179\n",
      "t = 200, loss = 1.6346\n",
      "t = 300, loss = 0.8054\n",
      "t = 400, loss = 0.9482\n",
      "Starting epoch 2 / 5\n",
      "Got 896 / 2230 correct (40.18)\n",
      "t = 100, loss = 0.1385\n",
      "t = 200, loss = 0.8375\n",
      "t = 300, loss = 0.9638\n",
      "t = 400, loss = 0.3625\n",
      "Starting epoch 3 / 5\n",
      "Got 1250 / 2230 correct (56.05)\n",
      "t = 100, loss = 0.1886\n",
      "t = 200, loss = 0.4034\n",
      "t = 300, loss = 0.0648\n",
      "t = 400, loss = 0.3983\n",
      "Starting epoch 4 / 5\n",
      "Got 1369 / 2230 correct (61.39)\n",
      "t = 100, loss = 0.1659\n",
      "t = 200, loss = 0.7913\n",
      "t = 300, loss = 0.0354\n",
      "t = 400, loss = 0.0297\n",
      "Starting epoch 5 / 5\n",
      "Got 1348 / 2230 correct (60.45)\n",
      "t = 100, loss = 0.0625\n",
      "t = 200, loss = 0.0724\n",
      "t = 300, loss = 0.8474\n",
      "t = 400, loss = 0.2689\n",
      "Got 7526 / 7770 correct (96.86)\n",
      "Got 1340 / 2230 correct (60.09)\n"
     ]
    }
   ],
   "source": [
    "torch.cuda.random.manual_seed(782374)\n",
    "fixed_model_3d.apply(reset) \n",
    "fixed_model_3d.train() \n",
    "train_3d(fixed_model_3d, loss_fn, optimizer,clip_dataloader_train, num_epochs=5) \n",
    "fixed_model_3d.eval() \n",
    "check_accuracy_3d(fixed_model_3d, clip_dataloader_train)\n",
    "check_accuracy_3d(fixed_model_3d, clip_dataloader_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Test your 3d convolution model on the validation set. You don't need to submit the result of this part to kaggle.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Test your model on the test set, predict_on_test_3d() will generate a file named 'results_3d.csv'. Please submit the csv file to kaggle https://www.kaggle.com/c/cse512springhw3video\n",
    "The highest 3 entries get extra 10 points.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3270\n"
     ]
    }
   ],
   "source": [
    "def predict_on_test_3d(model, loader):\n",
    "    '''\n",
    "    if loader.dataset.train:\n",
    "        print('Checking accuracy on validation set')\n",
    "    else:\n",
    "        print('Checking accuracy on test set')  \n",
    "    '''\n",
    "    num_correct = 0\n",
    "    num_samples = 0\n",
    "    model.eval() # Put the model in test mode (the opposite of model.train(), essentially)\n",
    "    results=open('results_3d.csv','w')\n",
    "    count=0\n",
    "    results.write('Id'+','+'Class'+'\\n')\n",
    "    for t, sample in enumerate(loader):\n",
    "        x_var = Variable(sample['clip'].type(dtype))\n",
    "        scores = model(x_var)\n",
    "        _, preds = scores.data.max(1)\n",
    "        for i in range(len(preds)):\n",
    "            results.write(str(count)+','+str(preds[i])+'\\n')\n",
    "            count+=1\n",
    "    results.close()\n",
    "    return count\n",
    "    \n",
    "count=predict_on_test_3d(fixed_model_3d, clip_dataloader_test)\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "* Rank on kaggle: 27\n",
    "* Score: 61.80428"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}