lec8-localization-detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS231n Winter 2016: Lecture 8\n",
    "## Topics\n",
    "**TODO:** add topics. hm maybe there is some plugin to do this quick?\n",
    "\n",
    "## Sources\n",
    "- video: https://www.youtube.com/watch?v=GxZrEKZfW2o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/GxZrEKZfW2o?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "video_id='GxZrEKZfW2o'\n",
    "HTML(f'<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/{video_id}?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Increase hardness\n",
    "- Classification \n",
    "- Localization (one of some objects)\n",
    "- Object Detection (variable number of objects)\n",
    "- Instance Segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Localization as Regrassion\n",
    "box (x,y,width,height)\n",
    "### Regresssion head\n",
    "- get pretrained model\n",
    "- replace the last fully connected layer (\"classification head\")\n",
    "- set \"regression head\" instead with few fully connected layers which output coors of box\n",
    "- in test time we use both \"heads\" (classification and regression)\n",
    "\n",
    "### Types of head\n",
    "- class specific regressor or class agnostic regressor\n",
    "\n",
    "### Design choices\n",
    "- where to attach regression head\n",
    "  - after conv layers - Overfeat, VGG \n",
    "  - after the last FC layer - DeepPose, R-CNN\n",
    "  \n",
    "### Application: Human Pose Estimation\n",
    "- [DeepPose: Human Pose Estimation via Deep Neural Networks Alexander Toshev, Christian Szegedy 2014](https://arxiv.org/abs/1312.4659) uses R-CNN\n",
    "\n",
    "### Sliding Window\n",
    "- run on multiple possion and aggregate result (use heuristics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Application: AlexNet\n",
    "- localization method was done but wasn't published\n",
    "\n",
    "### Application: Overfeat \n",
    "- alex net class architect\n",
    "  - input 3x221x221\n",
    "- FC 4096 -> FC 1024 -> FC Boxes(1000x4)\n",
    "- problem - expensive\n",
    "- we could consider each FC layer as Conv layer\n",
    "  - once we increase size of input matrix output conv layer also increases its size\n",
    "  \n",
    "### Application: VGG\n",
    "- uses overfeat but fewer scales and localization, simpler method, but deper network\n",
    "\n",
    "### Application: ResNet\n",
    "- uses RPN and deeper features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Object Detection\n",
    "- slide different windows and try to classify (use image pyramid) \n",
    "\n",
    "### History\n",
    "- 2005 HOG (Histogram oriented gradients)\n",
    "- 2010 DPM (Deformable Parts Model)\n",
    "- 2015 realize that DPM is just simple CNN [Deformable Part Models are Convolutional Neural Networks, Ross Girshick, Forrest Iandola, Trevor Darrell, Jitendra Malik](https://arxiv.org/abs/1409.5403)\n",
    "\n",
    "### Region Proposals\n",
    "- problem - a lot of tests, solution - use fast classifier but CNN is complex one, so we should decrease number of locations\n",
    "- class-agnostic object detection\n",
    "- it's looking for blob like stracture at an image\n",
    "\n",
    "#### Selective Search\n",
    "- [Selective Search for Object Recognition J. R. R. Uijlings](https://link.springer.com/article/10.1007/s11263-013-0620-5)\n",
    "- start from pixels and merge them like regions\n",
    "- put boxes around regions\n",
    "- other: [What makes for effective detection proposals? Jan Hosang, Rodrigo Benenson, Piotr Dollár, Bernt Schiele](https://arxiv.org/abs/1502.05082) - Lectors fav is EdgeBoxes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### R-CNN\n",
    "[Rich feature hierarchies for accurate object detection and semantic segmentation Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik](https://arxiv.org/abs/1311.2524)\n",
    "- apply selective search and get about 2k boxes\n",
    "- preprocess: crop and warp regions\n",
    "- use CNN (with regression head and classification head) for 2k boxes\n",
    "- train on possitive and neg regions of image\n",
    "- use multiple binarry classifiers to be able to extract multiple classes for the single region\n",
    "- Quality\n",
    "    - mAP (performance0) 66.0\n",
    "    \n",
    "#### R-CNN problems:\n",
    "- slow at test-time\n",
    "- use classier and regression head post-hoc - CNN features are not updated to them\n",
    "- complex multistage training pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fast R-CNN\n",
    "[Fast R-CNN Ross Girshick](https://arxiv.org/abs/1504.08083)\n",
    "- remove `selective search`\n",
    "- apply CNN directly to the image\n",
    "- get Regions of interests (RoI) - conv5 feature map of image\n",
    "- add `RoI-Polling` (single-level SPP) layer - which plays role of Selective Search\n",
    "- after apply FC layer\n",
    "- in result 2 head: classifier (Softmax) and Boundig box-regressors (Linear)\n",
    "\n",
    "#### Solves R-CNN problems\n",
    "- faster - use single CNN for all regions (we have single region right now - the whole image)\n",
    "- train all at once - sum of 2 head loss functions (Smooth L1 Loss + Log loss) - multi-task loss\n",
    "- Quality\n",
    "    - mAP (performance0) 66.9\n",
    "    - 8.8x faster training (9.5 hours for Pascal)\n",
    "    - 146x faster testing (0.32 seconds) - doesn't include region proposals\n",
    "    - 25x faster testing (2 seconds) - include region proposals\n",
    "\n",
    "### RoI-Polling\n",
    "- get reg proposal\n",
    "- apply `reg proposal` to Hi-res conv features: `C x H x W`\n",
    "- RoI conv features: max-pool within each grid cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Faster R-CNN\n",
    "[Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun](https://arxiv.org/abs/1506.01497)\n",
    "- attach region proposal network (CNN) which creates feature map of proposals\n",
    "- then (class-agnostic detector):\n",
    "    - slide a small window on the convolutional feature map\n",
    "    - small neural network \n",
    "      - classifying objects or non-object \n",
    "      - regressing bbox locations\n",
    "    - n anchors boxes\n",
    "    - classification gives probability of each anchor box\n",
    "    - and coors corrections (offset)\n",
    "- one network, 4 losses\n",
    "  - RPN classification (anchor good/bad)\n",
    "  - RPN regression (anchor -> proposal)\n",
    "  - Fast R-CNN classification (over classes)\n",
    "  - Fast R-CNN regression (proposal -> box)\n",
    "- Quality\n",
    "  - 0.2 sec, 250x \n",
    "  - mAP (performance) 66.9\n",
    "- [Deep Residual Learning for Image Recognition Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun](https://arxiv.org/abs/1512.03385)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## YOLO (You Only Look Once)\n",
    "[You Only Look Once: Unified, Real-Time Object Detection - Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi](https://arxiv.org/abs/1506.02640)\n",
    "\n",
    "Detection as Regression\n",
    "- dive in a spatial grid (7x7)\n",
    "- in each cell predict bbox and class\n",
    "\n",
    "- Real-time detector \n",
    "  - YOLO - 45 fps, mAP=63.4\n",
    "  - Fast YOLO - 155 fps, mAP=52.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other\n",
    "- [Spatial Transformer Networks Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu](https://arxiv.org/abs/1506.02025) - uses bilinear interpolation and could be useful for rotation\n",
    "  - another problem - we don't have that ground truth dataset \n",
    "    - but maybe we could make it if we know for sure that bound box alight correctly we just can rotate an image\n",
    "- 100Hz DPM or 30Hz DPM also are real-time detectors but with mAP = 16 and mAP = 26.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Datasets for Object Detection\n",
    "#### Pascal (VOC 2010)\n",
    "- number of classes: **20**\n",
    "- number of images (train + val): **~20k**\n",
    "- mean objects per image: **2.4**\n",
    "\n",
    "#### ImageNet Detection (ILSVRC 2014)\n",
    "- number of classes: **200**\n",
    "- number of images (train + val): **~470k**\n",
    "- mean objects per image: **1.1**\n",
    "\n",
    "#### MS-COCO (2014)\n",
    "- number of classes: **80**\n",
    "- number of images (train + val): **~120k**\n",
    "- mean objects per image: **7.2**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}