lec7-cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS231n Winter 2016: Lecture 7\n",
    "## Topics\n",
    "- CNN\n",
    "**TODO:** add topics. hm maybe there is some plugin to do this quick?\n",
    "\n",
    "## Sources\n",
    "- video: https://www.youtube.com/watch?v=LxfUGhug-iQ\n",
    "- original notes by Andrej Karpathy: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/LxfUGhug-iQ?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "video_id='LxfUGhug-iQ'\n",
    "HTML(f'<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/{video_id}?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolution filters\n",
    "- filter == kernal\n",
    "- examples:\n",
    "Roberts filter(2x2), Sobel filter(3x3), Prewitt filter(3x3), Laplacian filter(3x3), Gauss-Laplacian filter(5x5), Krisch filter(3x3)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolution layer\n",
    "- convolute (slide) over all spatical locations\n",
    "- result - activation map\n",
    "- apply ReLU\n",
    "- params: `width, height, depth, stride, pad`\n",
    "```python\n",
    "N = input.width\n",
    "F = filter.width\n",
    "P = padding\n",
    "S = stride\n",
    "output_width = (N - F + 2*P) / S + 1\n",
    "```\n",
    "- weights\n",
    "```python\n",
    "one_filter = N*N*input_depth + 1 # +1 bies\n",
    "one_layer_filters = one_filter*num_of_filters_in_one_layer\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization of layers\n",
    "- [Visualizing and Understanding Convolutional Networks by Matthew D Zeiler, Rob Fergus](https://arxiv.org/abs/1311.2901)\n",
    "- **TODO:** try to visualize\n",
    "  - filters\n",
    "  - activation maps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Padding\n",
    "- prevent shrinking of activation map \n",
    "- **TODO**: doesn't padding introduce borders? We could have almost white image and `0` padding will add border which could activate filter which detects edges and in this way all our images would have encoded borders.\n",
    "_Actually `0` doesn't contribute to the product of filter to input values and actually `0` is not black it is 'grey'. So it seem that edge filter will get the same result as just squre filter, or line filter. So padded number won't contribute to the activation map at all._\n",
    "- **TODO**: btw doesn't it mean that edges on grey background (input `0`) won't be seen by edge filter?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1x1 convolution layer\n",
    "decreases number activation map depth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pooling layer\n",
    "- downsampeling\n",
    "- maxpooling\n",
    "- avgpooling\n",
    "- params: filter_size (F), stride (S). Usually: `F=2, S=2` and `F=3, S=2`\n",
    "- trend toward getting rid of POOL/FC layers (just CONV with strides)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimation (ILSVRC)\n",
    "- top 5 error - because a lot of categories 1000 we give 5 chances to guess what is the single image is"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: LeNet-5 (1998)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Just Before AlexNet\n",
    "- `28.2% (ILSVRC'10) -> 25.8% (ILSVRC'11) (top 5 error)`\n",
    "- layers: shallow \n",
    "- in test time could be run on phone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: AlexNet (2012)\n",
    "- 18.2% -> 15.4%\n",
    "- `16.4% (top 5 error)`\n",
    "- layers: 8\n",
    "- 2 separated streams - because they have 2 GPUs - which is not problem right now and we don't need 2 streams\n",
    "- normalization layer (NORM) - which we don't use any more\n",
    "- 1st use of ReLU\n",
    "- heavy data augmentation\n",
    "- dropout 0.5\n",
    "- batch size 128\n",
    "- SGD Momentum 0.9\n",
    "- learning rate 1e-2, reduced by 10 manually when val accuracy plateaus\n",
    "- L2 weight decay 5e-4\n",
    "- 7 CNN ensemble: 18.2% -> 15.4%\n",
    "- FC6, FC7, FC9 - fully connected layers\n",
    "  - sometimes people refer to `FC7` - as the last layer before classifer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: ZFNet (2013)\n",
    "- 15.4% -> 14.8%\n",
    "- `11.7% (top 5 error)`\n",
    "- layers: 8\n",
    "- CONV1: 11x11 stride 4 -> 7x7 stride 2\n",
    "- CONV3,4,5: depth 384, 384, 256 -> 512, 1024, 512"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: VGGNet (2014)\n",
    "- `7.3% (top 5 error)`\n",
    "- layers: 19\n",
    "- footprint of the callculations: 24M * 4bytes ~ 93Mb (weights from forward propagation)\n",
    "- parameters: 138M\n",
    "- only 3x3 CONV stride 1, poll 1\n",
    "- and 2x2 MAX POOL stride 2\n",
    "- TODO: could get actual architecture [here](https://youtu.be/LxfUGhug-iQ?list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC&t=3535)\n",
    "\n",
    "problem:\n",
    "- a lot of parameters on 1st FC but the most data on the 1st few CNN layers. Because it connected to the previous activation map\n",
    "  - solution: use POOLAVG and shrink 7x7x512 -> 1x1x512"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: Google Net (2014)\n",
    "- `6.7%` (top 5 error)\n",
    "- layers: 22\n",
    "- inception module (many parallel CONV\n",
    "- only 5M parameters (remove FC layers) - 12x less parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: ResNet (2015)\n",
    "- `3.57% (top 5 error)`\n",
    "- layers: 152\n",
    "- Microsoft\n",
    "- 2-3 weeks on 8 GPU\n",
    "- faster than VGG (?)\n",
    "- decrease spatial resolution very early (by MAX POLL) - what decrease number of params and gives posibility to increase number of layers\n",
    "- Batch Normalization for every CONV layer\n",
    "- Xavier/2 initialization\n",
    "- SGD + Momentum (0.9)\n",
    "- Learning rate: 0.1, divided by 10 when validation error plateaus\n",
    "- Mini-batch size 256\n",
    "- Weight decay of 1e-5\n",
    "- No dropout used (because they told that with batch norm you don't need it)\n",
    "- additional subtleties https://youtu.be/LxfUGhug-iQ?list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC&t=4480\n",
    "  - all 3x3\n",
    "  - bottleneck (for ResNET-50/101/152)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: ? (2016)\n",
    "TODO: research"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: ? (2017)\n",
    "TODO: research"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: ? (2018)\n",
    "TODO: research"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: CIFAR-10 experiments\n",
    "- CIFAR-10 plain net\n",
    "- CIFAR-10 ResNets\n",
    "- Residual Net - have skip connections\n",
    "  - add identity before last ReLU\n",
    "  - 2 skipped layers compute delta of your representation (classifer gradient) instead of new representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Study: Deep Mind's (AlphaGo)\n",
    "- Input: [19x19x48], because 48 feature planes of each position\n",
    "- CONV1: 192 5x5 filters, stride 1, pad 2 => 19x19x192\n",
    "- CONV2..12: 192 3x3 filters, stride 1, pad => 19x19x192\n",
    "- CONV: 1x1 filter, stride 1, pad 0 (with different bies for each position), softmax => [19x19] (probability map of promising moves)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "- typical architecture: \n",
    "`[(CONV-RELU)*N-POLL?]*M-(FC-RELU)*K,SOFTMAX`\n",
    "  - N ~ 5\n",
    "  - M is large\n",
    "  - K <= 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}