4. Image Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Image Processing\n",
    "\n",
    "\n",
    "* [Changing Color-spaces](#Changing-Color-spaces)\n",
    "* [Object-Tracking](#Object-Tracking)\n",
    "* [Image-Thresholding](#Image-Thresholding)\n",
    "* [Smoothing-Images](#Smoothing-Images)\n",
    "* [Smoothing-Images](#Smoothing-Images)\n",
    "* [Image-Blurring](#Image-Blurring)\n",
    "* [Morphological-Transformations](#Morphological-Transformations)\n",
    "* [Image-Gradients](#Image-Gradients)\n",
    "* [Canny-Edge-Detection](#Canny-Edge-Detection)\n",
    "* [Image-Blending-using-Pyramids](#Image-Blending-using-Pyramids)\n",
    "* [Contours](#Contours)\n",
    "* [Histograms](#Histograms)\n",
    "* [Matching-Shapes](#Matching-Shapes)\n",
    "* [Template-Matching](#Template-Matching)\n",
    "* [Hough-Transforms](#Hough-Transforms)\n",
    "* [Image-Segmentation](#Image-Segmentation)\n",
    "* [Foreground-Extraction](#Foreground-Extraction)\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___\n",
    "## Changing Color-spaces\n",
    "In OpenCV, the color-coding format is BGR ie Blue-Green-Red instead of the original RGB like in case of display in computers. Here we will see how to convert images from BGR format to grayscale or BGR to HSV (Hue-Saturation-Value)and vice-versa. For BGR→Gray conversion we use the flags `cv2.COLOR_BGR2GRAY`. Similarly for BGR→HSV, we use the flag `cv2.COLOR_BGR2HSV`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['COLOR_BAYER_BG2BGR', 'COLOR_BAYER_BG2BGR_EA', 'COLOR_BAYER_BG2BGR_VNG', 'COLOR_BAYER_BG2GRAY', 'COLOR_BAYER_BG2RGB']\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "flags = [i for i in dir(cv2) if i.startswith('COLOR_')]\n",
    "print flags[:5] \n",
    "# I printed only the first 5 flags just for example purposes\n",
    "# If you want to check all of them just use:\n",
    "# print flags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___\n",
    "## Object Tracking\n",
    "Now we know how to convert BGR image to HSV, we can use this to extract a colored object. In HSV, it is more easier\n",
    "to represent a color than RGB color-space. In our application, we will try to extract a blue colored object. So here is\n",
    "the method:\n",
    "1. Take each frame of the video\n",
    "2. Convert from BGR to HSV color-space\n",
    "3. We threshold the HSV image for a range of blue color\n",
    "4. Now extract the blue object alone, we can do whatever on that image we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "cap = cv2.VideoCapture(0)\n",
    "while(1):\n",
    "    # 1. Take each frame\n",
    "    ret, frame = cap.read()\n",
    "    \n",
    "    # 2. Convert BGR to HSV\n",
    "    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)\n",
    "    \n",
    "    # define range of blue color in HSV\n",
    "    lower_blue = np.array([110,50,50])\n",
    "    upper_blue = np.array([130,255,255])\n",
    "    \n",
    "    # Threshold the HSV image to get only blue colors\n",
    "    mask = cv2.inRange(hsv, lower_blue, upper_blue)\n",
    "    \n",
    "    # Bitwise-AND mask and original image\n",
    "    res = cv2.bitwise_and(frame,frame, mask= mask)\n",
    "    cv2.imshow('frame',frame)\n",
    "    cv2.imshow('mask',mask)\n",
    "    cv2.imshow('res',res)\n",
    "    k = cv2.waitKey(5) & 0xFF\n",
    "    if k == 27:\n",
    "        break\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The resulting output would be something as follows:\n",
    "\n",
    "**The Original Webcam Capture:**\n",
    "<img src=\"captures/original.png\">\n",
    "**The Mask image:**\n",
    "<img src=\"captures/mask.png\">\n",
    "**The final result image:**\n",
    "<img src=\"captures/res.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### How to find HSV values to track?\n",
    "This question arises everytime you'll perform object tracking. What you can do is use the same function,\n",
    "_cv2.cvtColor()_ and instead of passing an image, you just pass the BGR values you want. For example, to find the\n",
    "HSV value of Blue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[120 255 255]]]\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "blue = np.uint8([[[255,0,0 ]]])\n",
    "hsv_blue = cv2.cvtColor(blue,cv2.COLOR_BGR2HSV)\n",
    "print hsv_blue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now you take [H-10, 100,100] and [H+10, 255, 255] as lower bound and upper bound respectively. Apart from this\n",
    "method, you can use any image editing tools like GIMP or any online converters to find these values, but then again you have\n",
    "to adjust the HSV ranges.\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Image Thresholding\n",
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "#img = cv2.imread('images/th.png')\n",
    "img = cv2.imread('images/th4.png')\n",
    "# grayscale\n",
    "grayscaled = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)\n",
    "\n",
    "# simple threshold\n",
    "#If pixel value is greater than a threshold value, it is assigned one value (may be white 255),\n",
    "# else it is assigned another value (may be black 0)\n",
    "retval, threshold = cv2.threshold(img, 12, 255, cv2.THRESH_BINARY)\n",
    "\"\"\"\n",
    "The first parameter here is the image.\n",
    "The next parameter is the threshold, we are choosing 12.\n",
    "(We are choosing 12, because this is a low-light picture, If everything is very bright it can be 220 or so)\n",
    "The next is the maximum value, which we're choosing as 255.\n",
    "Next and finally we have the type of threshold, which we've chosen as THRESH_BINARY.\n",
    "Normally, a threshold of 10 would be somewhat poor of a choice.\n",
    "\"\"\"\n",
    "\n",
    "# threshold with grayscale\n",
    "retval, threshold2 = cv2.threshold(grayscaled, 12, 255, cv2.THRESH_BINARY)\n",
    "\n",
    "\"\"\"\n",
    "Above we used a global value as threshold value.\n",
    "But it may not be good in all the conditions where image has different lighting conditions in different areas.\n",
    "In that case, we go for adaptive thresholding.\n",
    "In this, the algorithm calculates the threshold for a small regions of the image.\n",
    "So we get different thresholds for different regions of the same image\n",
    "and it gives us better results for images with varying illumination\n",
    "\"\"\"\n",
    "\n",
    "# adaptive mean threshold\n",
    "# threshold value is the mean of neighbourhood area.\n",
    "amt = cv2.adaptiveThreshold(grayscaled,255,cv2.ADAPTIVE_THRESH_MEAN_C,cv2.THRESH_BINARY,11,2)\n",
    "\n",
    "# guassian adaptive threshold\n",
    "# threshold value is the weighted sum of neighbourhood values where weights are a gaussian window\n",
    "gat = cv2.adaptiveThreshold(grayscaled, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 115, 1)\n",
    "\n",
    "# otsu threshold\n",
    "retval2,otsuth = cv2.threshold(grayscaled,125,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)\n",
    "\n",
    "cv2.imshow('Original Image',img)\n",
    "cv2.imshow('Simple Thresholding',threshold)\n",
    "cv2.imshow('Thresholding with Grayscale',threshold2)\n",
    "cv2.imshow('Adaptive Mean Thresholding',amt)\n",
    "cv2.imshow('Adaptive Gaussian Thresholding',gat)\n",
    "cv2.imshow('Otsu thresholding',otsuth)\n",
    "cv2.waitKey(0)\n",
    "cv2.destroyAllWindows()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The output would look like: \n",
    "    <img src=\"captures/thres1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The original image was bright on one part, and dark on the other. Due to variations in lighting conditions we have such varied results of thresholding. Feel free to try different images with different lighting surroundings and analyze the results.\n",
    "Most of the times one of these methods of thresholding would give you appropriate expected results as in this case Adaptive Mean Thresholding did the job for us.\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Smoothing Images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As in one-dimensional signals, images also can be filtered with various low-pass filters(LPF), high-pass filters(HPF) etc. LPF helps in removing noises, blurring the images etc. HPF filters helps in finding edges in the images.\n",
    "OpenCV provides a function <a href=\"http://docs.opencv.org/3.0.0/d4/d86/group__imgproc__filter.html#ga27c049795ce870216ddfb366086b5a04\">cv2.filter2D()</a> to convolve a kernel with an image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "img = cv2.imread('images/opencv_logo.png')\n",
    "kernel = np.ones((5,5),np.float32)/25\n",
    "dst = cv2.filter2D(img,-1,kernel)\n",
    "plt.subplot(121),plt.imshow(img),plt.title('Original')\n",
    "plt.xticks([]), plt.yticks([])\n",
    "plt.subplot(122),plt.imshow(dst),plt.title('Averaging')\n",
    "plt.xticks([]), plt.yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"images/filter.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Image Blurring\n",
    "Image blurring is achieved by convolving the image with a low-pass filter kernel. It is useful for removing noise. It\n",
    "actually removes high frequency content (e.g: noise, edges) from the image resulting in edges being blurred when this\n",
    "is filter is applied.\n",
    "\n",
    "The following code captures video using webcam and applies the following types of blurring techniques:\n",
    "* Gaussian Blurring\n",
    "* Median Blurring\n",
    "* Bilateral Blurring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "cap = cv2.VideoCapture(0)\n",
    "\n",
    "while(1):\n",
    "    _, frame = cap.read()\n",
    "    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)\n",
    "    \n",
    "    lower_red = np.array([30,150,50])\n",
    "    upper_red = np.array([255,255,180])\n",
    "    \n",
    "    mask = cv2.inRange(hsv, lower_red, upper_red)\n",
    "    res = cv2.bitwise_and(frame,frame, mask= mask)\n",
    "\n",
    "    # Blurring\n",
    "    \n",
    "    # Gaussian Blurring\n",
    "    blur = cv2.GaussianBlur(res,(15,15),0)\n",
    "    cv2.imshow('Gaussian Blurring',blur)\n",
    "\n",
    "    # Median Blurring\n",
    "    median = cv2.medianBlur(res,15)\n",
    "    cv2.imshow('Median Blur',median)\n",
    "\n",
    "    # Bilateral Blurring\n",
    "    bilateral = cv2.bilateralFilter(res,15,75,75)\n",
    "    cv2.imshow('bilateral Blur',bilateral)\n",
    "\n",
    "    #cv2.imshow('frame',frame)\n",
    "    #cv2.imshow('mask',mask)\n",
    "    #cv2.imshow('res',res)\n",
    "    \n",
    "    k = cv2.waitKey(5) & 0xFF\n",
    "    if k == 27:\n",
    "        break\n",
    "\n",
    "cv2.destroyAllWindows()\n",
    "cap.release()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Morphological Transformations\n",
    "Morphological transformations are simple operations based on the image shape. It is usually performed on\n",
    "binary images. It needs two inputs, one is our original image, second one is called structuring element or kernel\n",
    "which decides the nature of operation. Two basic morphological operators are Erosion and Dilation. Then its variant\n",
    "forms like Opening, Closing, Gradient etc also comes into play.\n",
    "\n",
    "You can find detailed explanations about these operations at: \n",
    "<a href=\"http://homepages.inf.ed.ac.uk/rbf/HIPR2/morops.htm\">Morphological Operations at HIPR2</a>\n",
    "\n",
    "The following code captures video using webcam and applies the above listed morphological operations on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "cap = cv2.VideoCapture(0)\n",
    "\n",
    "while(1):\n",
    "\n",
    "    _, frame = cap.read()\n",
    "    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)\n",
    "    \n",
    "    lower_red = np.array([30,150,50])\n",
    "    upper_red = np.array([255,255,180])\n",
    "    \n",
    "    mask = cv2.inRange(hsv, lower_red, upper_red)\n",
    "    res = cv2.bitwise_and(frame,frame, mask= mask)\n",
    "\n",
    "    # Morphological Traansformation\n",
    "    kernel = np.ones((5,5),np.uint8)\n",
    "    erosion = cv2.erode(mask,kernel,iterations = 1)\n",
    "    dilation = cv2.dilate(mask,kernel,iterations = 1)\n",
    "    opening = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)  # remove background noise \n",
    "    closing = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)  # remove black pixels within the object\n",
    "\n",
    "    cv2.imshow('Original',frame)\n",
    "    cv2.imshow('Mask',mask)\n",
    "\n",
    "    cv2.imshow('Erosion',erosion)\n",
    "    cv2.imshow('Dilation',dilation)\n",
    "\n",
    "    cv2.imshow('Opening',opening)\n",
    "    cv2.imshow('Closing',closing)\n",
    "\n",
    "\n",
    "    k = cv2.waitKey(5) & 0xFF\n",
    "    if k == 27:\n",
    "        break\n",
    "\n",
    "cv2.destroyAllWindows()\n",
    "cap.release()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Image Gradients\n",
    "OpenCV provides three types of gradient filters or High-pass filters, Sobel, Scharr and Laplacian.\n",
    "\n",
    "Sobel operators is a joint Gausssian smoothing plus differentiation operation, so it is more resistant to noise. You\n",
    "can specify the direction of derivatives to be taken, vertical or horizontal (by the arguments, yorder and xorder\n",
    "respectively). You can also specify the size of kernel by the argument ksize. If ksize = -1, a 3x3 Scharr filter is used\n",
    "which gives better results than 3x3 Sobel filter. Laplacian Derivatives calculates the Laplacian of the image.\n",
    "\n",
    "Below code shows all operators in a single diagram. All kernels are of 5x5 size. Depth of output image is passed -1 to\n",
    "get the result in np.uint8 type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "img = cv2.imread('dave.jpg',0)\n",
    "laplacian = cv2.Laplacian(img,cv2.CV_64F)\n",
    "sobelx = cv2.Sobel(img,cv2.CV_64F,1,0,ksize=5)\n",
    "sobely = cv2.Sobel(img,cv2.CV_64F,0,1,ksize=5)\n",
    "\n",
    "plt.subplot(2,2,1),plt.imshow(img,cmap = 'gray')\n",
    "plt.title('Original'), plt.xticks([]), plt.yticks([])\n",
    "plt.subplot(2,2,2),plt.imshow(laplacian,cmap = 'gray')\n",
    "plt.title('Laplacian'), plt.xticks([]), plt.yticks([])\n",
    "plt.subplot(2,2,3),plt.imshow(sobelx,cmap = 'gray')\n",
    "plt.title('Sobel X'), plt.xticks([]), plt.yticks([])\n",
    "plt.subplot(2,2,4),plt.imshow(sobely,cmap = 'gray')\n",
    "plt.title('Sobel Y'), plt.xticks([]), plt.yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"images/grad.png\">\n",
    "\n",
    "In our last example, output datatype is cv2.CV_8U or np.uint8. But there is a slight problem with that. Black-to-White\n",
    "transition is taken as Positive slope (it has a positive value) while White-to-Black transition is taken as a Negative slope\n",
    "(It has negative value). So when you convert data to np.uint8, all negative slopes are made zero. In simple words, you\n",
    "miss that edge.\n",
    "\n",
    "If you want to detect both edges, better option is to keep the output datatype to some higher forms, like cv2.CV_16S,\n",
    "cv2.CV_64F etc, take its absolute value and then convert back to cv2.CV_8U. Below code demonstrates this procedure\n",
    "for a horizontal Sobel filter and difference in results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "img = cv2.imread('box.png',0)\n",
    "\n",
    "# Output dtype = cv2.CV_8U\n",
    "sobelx8u = cv2.Sobel(img,cv2.CV_8U,1,0,ksize=5)\n",
    "\n",
    "# Output dtype = cv2.CV_64F. Then take its absolute and convert to cv2.CV_8U\n",
    "sobelx64f = cv2.Sobel(img,cv2.CV_64F,1,0,ksize=5)\n",
    "abs_sobel64f = np.absolute(sobelx64f)\n",
    "sobel_8u = np.uint8(abs_sobel64f)\n",
    "\n",
    "plt.subplot(1,3,1),plt.imshow(img,cmap = 'gray')\n",
    "plt.title('Original'), plt.xticks([]), plt.yticks([])\n",
    "plt.subplot(1,3,2),plt.imshow(sobelx8u,cmap = 'gray')\n",
    "plt.title('Sobel CV_8U'), plt.xticks([]), plt.yticks([])\n",
    "plt.subplot(1,3,3),plt.imshow(sobel_8u,cmap = 'gray')\n",
    "plt.title('Sobel abs(CV_64F)'), plt.xticks([]), plt.yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The result would be look like:\n",
    "<img src=\"images/grad2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Canny Edge Detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Canny Edge Detection is one of the most commonly used techniques to find edges in images. In OpenCV _cv2.Canny()_ is used to carry out Canny Edge Detection. First argument to be passed in this function is our input\n",
    "image. Second and third arguments are our minVal and maxVal respectively. Third argument is aperture_size. It is the\n",
    "size of Sobel kernel used for find image gradients. By default it is 3. Last argument is L2gradient which specifies the\n",
    "equation for finding gradient magnitude."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "img = cv2.imread('images/cube.png',0)\n",
    "edges = cv2.Canny(img,100,200)\n",
    "plt.subplot(121),plt.imshow(img,cmap = 'gray')\n",
    "plt.title('Original Image'), plt.xticks([]), plt.yticks([])\n",
    "plt.subplot(122),plt.imshow(edges,cmap = 'gray')\n",
    "plt.title('Edge Image'), plt.xticks([]), plt.yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The resulting output would look like:\n",
    "    <img src=\"images/cubeedge.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "___\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Image Blending using Pyramids\n",
    "___\n",
    "Normally, we used to work with an image of constant size. But in some occassions, we need to work with images of different resolution of the same image. For example, while searching for something in an image, like face, we are not sure at what size the object will be present in the image. In that case, we will need to create a set of images with different resolution and search for object in all the images. These set of images with different resolution are called Image Pyramids (because when they are kept in a stack with biggest image at bottom and smallest image at top look like a pyramid). Basically an “image pyramid” is a multi-scale representation of an image.\n",
    "\n",
    "There are two kinds of Image Pyramids. 1) Gaussian Pyramid and 2) Laplacian Pyramids\n",
    "We can find Gaussian pyramids using <a href=\"http://docs.opencv.org/3.0.0/d4/d86/group__imgproc__filter.html#gaf9bba239dfca11654cb7f50f889fc2ff\">cv2.pyrDown()</a> and <a href=\"http://docs.opencv.org/3.0.0/d4/d86/group__imgproc__filter.html#gada75b59bdaaca411ed6fee10085eb784\">cv2.pyrUp()</a> functions.\n",
    "Laplacian Pyramids are formed from the Gaussian Pyramids. There is no exclusive function for that. Laplacian\n",
    "pyramid images are like edge images only. Most of its elements are zeros. They are used in image compression. A\n",
    "level in Laplacian Pyramid is formed by the difference between that level in Gaussian Pyramid and expanded version\n",
    "of its upper level in Gaussian Pyramid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "img = cv2.imread('images/googlelogo.jpg')\n",
    "lower_reso = cv2.pyrDown(higher_reso)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Now you can go down the image pyramid with cv2.pyrUp() function.\n",
    "higher_reso2 = cv2.pyrUp(lower_reso)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The following code results in forming the image pyramid of a 500x500 image of Google Logo. First we import the imutils package which contains a handful of image processing convenience functions that are commonly used such as resizing, rotating, translating, etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# import the necessary packages\n",
    "import imutils\n",
    "\n",
    "def pyramid(image, scale=1.5, minSize=(30, 30)):\n",
    "    # yield the original image\n",
    "    yield image\n",
    "\n",
    "    # keep looping over the pyramid\n",
    "    while True:\n",
    "        # compute the new dimensions of the image and resize it\n",
    "        w = int(image.shape[1] / scale)\n",
    "        image = imutils.resize(image, width=w)\n",
    "\n",
    "        if image.shape[0] < minSize[1] or image.shape[1] < minSize[0]:\n",
    "            break\n",
    "\n",
    "        # yield the next image in the pyramid\n",
    "        yield image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This pyramid function takes two arguments.\n",
    "1. Scale : By how much the image is resized at each layer.\n",
    "2. minSize :The minimum required width and height of the layer. If an image in the pyramid falls below this minSize , we stop constructing the image pyramid.\n",
    "\n",
    "Initially we yield the original image. Then we loop over to form our image pyramid.\n",
    "\n",
    "Over each iteration, scaling of the image in the next layer of the pyramid (while preserving the aspect ratio) is performed. This scale is controlled by the scale factor.\n",
    "\n",
    "Until our image meets the minSize  requirements, we loop. Then finally we have different sized versions of our image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import argparse\n",
    "import cv2\n",
    "\n",
    "\n",
    "# load the 500 x 500 image of Google Logo\n",
    "image = cv2.imread(\"images/googlelogo.jpg\")\n",
    "\n",
    "# loop over the image pyramid\n",
    "for (i, resized) in enumerate(pyramid(image, scale=2)):\n",
    "    # show the resized image\n",
    "    cv2.imshow(\"Layer {}\".format(i + 1), resized)\n",
    "    # saving all the resized images\n",
    "    cv2.imwrite(\"ImagePyramid/Layer {}.jpg\".format(i + 1),resized)\n",
    "    cv2.waitKey(0)\n",
    "\n",
    "# close all windows\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The resulting resized images would be saved in a folder named ImagePyramid in your project directory.\n",
    "It would look something like:\n",
    "<img src=\"captures/imagepyramid.png\">\n",
    "    \n",
    "If all the images are compared in terms of size, it would look like:\n",
    "<img src=\"captures/resizedip.png\">\n",
    "\n",
    "In this code, I used scale = 2. Hence each next image was half the size of its previous image.\n",
    "Feel free to use different scales and compare the results.\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "One application of Pyramids is Image Blending. For example, in image stitching, you will need to stack two images together, but it may not look good due to discontinuities between images. In that case, image blending with Pyramids gives you seamless blending without leaving much data in the images. \n",
    "\n",
    "Given below is an example of how to make one of the most iconic pictures in comic book history- **Batman and Joker** with the technique of Image Blending.\n",
    "\n",
    "You can find elaborate explanations with diagrams at <a href=\"http://pages.cs.wisc.edu/~csverma/CS766_09/ImageMosaic/imagemosaic.html\">Image Blending</a>.\n",
    "\n",
    "Following are the steps to be performed:\n",
    "1. Load the two images of Batman and Joker\n",
    "2. Find the Gaussian Pyramids for Batman and Joker (in this particular example, number of layers is 6)\n",
    "3. From Gaussian Pyramids, find their Laplacian Pyramids\n",
    "4. Now join the left half of Batman's face and right half of Joker's ever similing face in each layers of Laplacian Pyramids\n",
    "5. Finally from this joint image pyramids, reconstruct the original image.\n",
    "\n",
    "Below is the full code. (For sake of simplicity, each step is done separately which may take more memory. You can\n",
    "optimize it if you want so).\n",
    "We would be using these two images of size 700 x 315 as input (However you can use any two images if you want.):\n",
    "\n",
    "\n",
    "<img  style=\"float:left;padding-left:20px;padding-right:20px;\" src=\"images/batman.png\" height = \"180\" width=\"400\" /> \n",
    "\n",
    "<img  style=\"float:left;padding-left:20px;padding-right:20px;\" src=\"images/joker.png\" height = \"180\" width=\"400\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np,sys\n",
    "\n",
    "A = cv2.imread('captures/batman1.png')\n",
    "B = cv2.imread('captures/joker2.png')\n",
    "\n",
    "# generate Gaussian pyramid for A\n",
    "G = A.copy()\n",
    "gpA = [G]\n",
    "for i in xrange(6):\n",
    "    G = cv2.pyrDown(gpA[i])\n",
    "    gpA.append(G)\n",
    "\n",
    "# generate Gaussian pyramid for B\n",
    "G = B.copy()\n",
    "gpB = [G]\n",
    "for i in xrange(6):\n",
    "    G = cv2.pyrDown(gpB[i])\n",
    "    gpB.append(G)\n",
    "\n",
    "# generate Laplacian Pyramid for A\n",
    "lpA = [gpA[5]]\n",
    "for i in xrange(5,0,-1):\n",
    "    size = (gpA[i-1].shape[1], gpA[i-1].shape[0])\n",
    "    GE = cv2.pyrUp(gpA[i], dstsize = size)\n",
    "    L = cv2.subtract(gpA[i-1],GE)\n",
    "    lpA.append(L)\n",
    "\n",
    "# generate Laplacian Pyramid for B\n",
    "lpB = [gpB[5]]\n",
    "for i in xrange(5,0,-1):\n",
    "    size = (gpB[i-1].shape[1], gpB[i-1].shape[0])\n",
    "    GE = cv2.pyrUp(gpB[i], dstsize = size)\n",
    "    L = cv2.subtract(gpB[i-1],GE)\n",
    "    lpB.append(L)\n",
    "    \n",
    "# Now add left and right halves of images in each level\n",
    "LS = []\n",
    "for la,lb in zip(lpA,lpB):\n",
    "    rows,cols,dpt = la.shape\n",
    "    ls = np.hstack((la[:,0:cols/2], lb[:,cols/2:]))\n",
    "    LS.append(ls)\n",
    "    \n",
    "# now reconstruct\n",
    "ls_ = LS[0]\n",
    "for i in xrange(1,6):\n",
    "    size = (LS[i].shape[1], LS[i].shape[0])\n",
    "    ls_ = cv2.pyrUp(ls_, dstsize = size)\n",
    "    ls_ = cv2.add(ls_, LS[i])\n",
    "\n",
    "    \n",
    "# image with direct connecting each half\n",
    "real = np.hstack((A[:,:cols/2],B[:,cols/2:]))\n",
    "cv2.imwrite('Pyramid blending.jpg',ls_)\n",
    "cv2.imwrite('Direct blending.jpg',real)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The resulting images after blending would look like: \n",
    "    \n",
    "    Direct Blending :\n",
    "    \n",
    "<img src=\"results/Directblending.jpg\">\n",
    "    \n",
    "    and Pyramid Blending : \n",
    "        \n",
    "<img src=\"results/Pyramidblending.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___\n",
    "_**Pretty cool isn't it! :D**_\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Contours\n",
    "Contours is simply a curve joining all the continuous points along the boundary, having same color or intensity. The contours can be useful for shape analysis and object detection and recognition.\n",
    "\n",
    "To find contours with a better accuracy, binary images are used. So most of the times before finding contours, threshold or canny edge detection is applied.\n",
    "_findContours()_ function modifies the source image. So you have to save the image in some temporary variable before applying it, if you want source image.\n",
    "\n",
    "In OpenCV, finding contours is like finding white object from black background. Here's how it can be done:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "im = cv2.imread('images/testimage.jpg')\n",
    "imgray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)\n",
    "ret,thresh = cv2.threshold(imgray,127,255,0)\n",
    "image, contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "There are three arguments in cv2.findContours() function\n",
    "1. Source image\n",
    "2. Contour retrieval mode,\n",
    "3. Contour approximation method.\n",
    "\n",
    "If you pass cv2.CHAIN_APPROX_NONE as the third argument, all the boundary points are stored i.e. if we found contour of a straight line, we'll get all the points on the line.\n",
    "\n",
    "Whereas if we need just the two end points of the line, then cv2.CHAIN_APPROX_SIMPLE should be passed. It removes all redundant points and compresses the contour, thereby saving memory.\n",
    "\n",
    "Below image of a rectangle demonstrate this technique. Just draw a circle on all the coordinates in the contour array (drawn in blue color). First image shows points I got with cv2.CHAIN_APPROX_NONE (734 points) and second image shows the one with cv2.CHAIN_APPROX_SIMPLE (only 4 points). See, how much memory it saves!\n",
    "\n",
    "<img src=\"images/contour.jpg\">\n",
    "\n",
    "The output of this function is the contours and hierarchy. _contours_ is a Python list of all the contours in the image. Each individual contour is a Numpy array of (x,y) coordinates of boundary points of the object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Drawing the contours:\n",
    "To draw the contours, _cv2.drawContours()_ function is used. It can also be used to draw any shape provided you\n",
    "have its boundary points. Its arguments are\n",
    "1. Source image\n",
    "2. Contours which should be passed as a Python list\n",
    "3. Index of contours (useful when drawing individual contour. To draw all contours, pass -1)\n",
    "4. Color, Thickness etc.\n",
    "\n",
    "To draw all the contours in an image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "img = cv2.drawContours(img, contours, -1, (0,255,0), 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To draw an individual contour, say 4th contour:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "img = cv2.drawContours(img, contours, 3, (0,255,0), 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Another more useful method is :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "cnt = contours[4]\n",
    "cv2.drawContours(img, [cnt], 0, (0,255,0), 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Finding the Contour Features:\n",
    "\n",
    "### Image Moments\n",
    "<a href=\"http://en.wikipedia.org/wiki/Image_moment\">Image moments</a> can help you to calculate some features like center of mass of the object, area of the object etc\n",
    "The function cv2.moments() gives a dictionary of all moment values calculated. See below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'mu02': 4293924291.143133, 'mu03': 63142297875.93164, 'm11': 10223790320.25, 'nu02': 0.1488429929630718, 'm12': 3525902955684.9165, 'mu21': 21803315369.182495, 'mu20': 4410354955.029182, 'nu20': 0.15287890214787567, 'm30': 5991419770513.0, 'nu21': 0.0018338552415319222, 'mu11': 4699166.697471619, 'mu12': -3947014808.411865, 'nu11': 0.00016289016486079318, 'nu12': -0.00033197950276135505, 'm02': 14056865696.0, 'm03': 5492215401421.25, 'm00': 169849.0, 'm01': 40721319.166666664, 'mu30': -13260752252.398438, 'nu30': -0.0011153487262349736, 'nu03': 0.005310836079810775, 'm10': 42623924.0, 'm20': 15106908347.5, 'm21': 3646045025879.5835}\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "# I am using a basic star image\n",
    "img = cv2.imread('images/star.png',0)\n",
    "ret,thresh = cv2.threshold(img,127,255,0)\n",
    "_,contours,hierarchy, = cv2.findContours(thresh, 1, 2)\n",
    "cnt = contours[0]\n",
    "M = cv2.moments(cnt)\n",
    "print M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# From this moments, you can extract useful data like:\n",
    "# Centroid\n",
    "cx = int(M['m10']/M['m00'])\n",
    "cy = int(M['m01']/M['m00'])\n",
    "print \"Cx =\",cx,\" Cy =\",cy\n",
    "\n",
    "# Contour Area\n",
    "# Contour area is given by the function cv2.contourArea() or from moments, M[’m00’].\n",
    "area = cv2.contourArea(cnt)\n",
    "print \"Contour Area: \", area\n",
    "\n",
    "# Contour Perimeter\n",
    "# It is also called arc length. It can be found out using cv2.arcLength() function.\n",
    "# Second argument specify whether shape is a closed contour (if passed True), or just a curve.\n",
    "perimeter = cv2.arcLength(cnt,True)\n",
    "print \"Contour Perimeter: \", perimeter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Contour Approximation\n",
    "It approximates a contour shape to another shape with less number of vertices depending upon the precision we specify.\n",
    "It is an implementation of <a href=\"http://en.wikipedia.org/wiki/Ramer-Douglas-Peucker_algorithm\">Douglas-Peucker algorithm</a>.\n",
    "To understand this, suppose you are trying to find a square in an image, but due to some problems in the image, you\n",
    "didn’t get a perfect square, but a “bad shape” (As shown in first image below). Now you can use this function to\n",
    "approximate the shape. In this, second argument is called epsilon, which is maximum distance from contour to\n",
    "approximated contour. It is an accuracy parameter. A wise selection of epsilon is needed to get the correct output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "epsilon = 0.1*cv2.arcLength(cnt,True)\n",
    "approx = cv2.approxPolyDP(cnt,epsilon,True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Below, in second image, green line shows the approximated curve for epsilon = 10% of arc length. Third\n",
    "image shows the same for epsilon = 1% of the arc length. Third argument specifies whether curve is\n",
    "closed or not\n",
    "\n",
    "<img src=\"images/contourapprox.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Checking Convexity\n",
    "_cv2.isContourConvex()_ is used to check if a curve is convex or not. It just return whether True or False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n"
     ]
    }
   ],
   "source": [
    "k = cv2.isContourConvex(cnt)\n",
    "print k "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Thus stating the obvious that a star (the image I have taken) is not convex."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Bounding Rectangle\n",
    "There are two types of bounding rectangles.\n",
    "\n",
    "**1. Straight Bounding Rectangle:**\n",
    "\n",
    "It is a straight rectangle, it doesn’t consider the rotation of the object. So area of\n",
    "the bounding rectangle won’t be minimum. It is found by the function cv2.boundingRect().\n",
    "Let (x,y) be the top-left coordinate of the rectangle and (w,h) be its width and height."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x,y,w,h = cv2.boundingRect(cnt)\n",
    "img = cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,0),2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**2. Rotated Rectangle:**\n",
    "\n",
    "Here, bounding rectangle is drawn with minimum area, so it considers the rotation also.\n",
    "The function used is cv2.minAreaRect(). It returns a Box2D structure which contains following detals - ( top-left\n",
    "corner(x,y), (width, height), angle of rotation ). But to draw this rectangle, we need 4 corners of the rectangle. It is\n",
    "obtained by the function cv2.boxPoints()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rect = cv2.minAreaRect(cnt)\n",
    "box = cv2.boxPoints(rect)\n",
    "box = np.int0(box)\n",
    "im = cv2.drawContours(im,[box],0,(0,0,255),2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Both the rectangles are shown in a single image. Green rectangle shows the normal bounding rect. Red rectangle is\n",
    "the rotated rect.\n",
    "\n",
    "<img src=\"images/boundingrect.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Minimum Enclosing Circle\n",
    "Next we find the circumcircle of an object using the function cv2.minEnclosingCircle(). It is a circle which completely\n",
    "covers the object with minimum area."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "(x,y),radius = cv2.minEnclosingCircle(cnt)\n",
    "center = (int(x),int(y))\n",
    "radius = int(radius)\n",
    "img = cv2.circle(img,center,radius,(0,255,0),2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"images/circumcircle.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Fitting an Ellipse\n",
    "Next one is to fit an ellipse to an object. It returns the rotated rectangle in which the ellipse is inscribed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "ellipse = cv2.fitEllipse(cnt)\n",
    "cv2.ellipse(img,ellipse,(0,255,0),2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"images/fitellipse.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Fitting a Line\n",
    "Similarly we can fit a line to a set of points. Below image contains a set of white points. We can approximate a straight\n",
    "line to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rows,cols = img.shape[:2]\n",
    "[vx,vy,x,y] = cv2.fitLine(cnt, cv2.DIST_L2,0,0.01,0.01)\n",
    "lefty = int((-x*vy/vx) + y)\n",
    "righty = int(((cols-x)*vy/vx)+y)\n",
    "img = cv2.line(img,(cols-1,righty),(0,lefty),(0,255,0),2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"images/fitline.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___\n",
    "## Histograms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This is a sample for histogram plotting for RGB images and grayscale images for better understanding of colour distribution.\n",
    "\n",
    "Using this we can learn how to draw histogram of images and also get familier with cv2.calcHist, cv2.equalizeHist,cv2.normalize and some drawing functions\n",
    "\n",
    "We use the following functions :\n",
    "1) hist_curve : returns histogram of an image drawn as curves\n",
    "2) hist_lines : return histogram of an image drawn as bins ( only for grayscale images )\n",
    "\n",
    "#### Usage :\n",
    "```python hist.py <image_file>```\n",
    "\n",
    "Credits : Abid Rahman 3/14/12 debug Gary Bradski"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### hist.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#!/usr/bin/env python\n",
    "# Python 2/3 compatibility\n",
    "from __future__ import print_function\n",
    "\n",
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "bins = np.arange(256).reshape(256,1)\n",
    "\n",
    "def hist_curve(im):\n",
    "    h = np.zeros((300,256,3))\n",
    "    if len(im.shape) == 2:\n",
    "        color = [(255,255,255)]\n",
    "    elif im.shape[2] == 3:\n",
    "        color = [ (255,0,0),(0,255,0),(0,0,255) ]\n",
    "    for ch, col in enumerate(color):\n",
    "        hist_item = cv2.calcHist([im],[ch],None,[256],[0,256])\n",
    "        cv2.normalize(hist_item,hist_item,0,255,cv2.NORM_MINMAX)\n",
    "        hist=np.int32(np.around(hist_item))\n",
    "        pts = np.int32(np.column_stack((bins,hist)))\n",
    "        cv2.polylines(h,[pts],False,col)\n",
    "    y=np.flipud(h)\n",
    "    return y\n",
    "\n",
    "def hist_lines(im):\n",
    "    h = np.zeros((300,256,3))\n",
    "    if len(im.shape)!=2:\n",
    "        print(\"hist_lines applicable only for grayscale images\")\n",
    "        #print(\"so converting image to grayscale for representation\"\n",
    "        im = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)\n",
    "    hist_item = cv2.calcHist([im],[0],None,[256],[0,256])\n",
    "    cv2.normalize(hist_item,hist_item,0,255,cv2.NORM_MINMAX)\n",
    "    hist=np.int32(np.around(hist_item))\n",
    "    for x,y in enumerate(hist):\n",
    "        cv2.line(h,(x,0),(x,y),(255,255,255))\n",
    "    y = np.flipud(h)\n",
    "    return y\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "\n",
    "    import sys\n",
    "\n",
    "    if len(sys.argv)>1:\n",
    "        fname = sys.argv[1]\n",
    "    else :\n",
    "        fname = 'images/9ball.jpg'\n",
    "        print(\"usage : python hist.py <image_file>\")\n",
    "\n",
    "    im = cv2.imread(fname)\n",
    "\n",
    "    if im is None:\n",
    "        print('Failed to load image file:', fname)\n",
    "        sys.exit(1)\n",
    "\n",
    "    gray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)\n",
    "\n",
    "\n",
    "    print(''' Histogram plotting \\n\n",
    "    Keymap :\\n\n",
    "    a - show histogram for color image in curve mode \\n\n",
    "    b - show histogram in bin mode \\n\n",
    "    c - show equalized histogram (always in bin mode) \\n\n",
    "    d - show histogram for color image in curve mode \\n\n",
    "    e - show histogram for a normalized image in curve mode \\n\n",
    "    Esc - exit \\n\n",
    "    ''')\n",
    "\n",
    "    cv2.imshow('image',im)\n",
    "    while True:\n",
    "        k = cv2.waitKey(0)&0xFF\n",
    "        if k == ord('a'):\n",
    "            curve = hist_curve(im)\n",
    "            cv2.imshow('histogram',curve)\n",
    "            cv2.imshow('image',im)\n",
    "            print('a')\n",
    "        elif k == ord('b'):\n",
    "            print('b')\n",
    "            lines = hist_lines(im)\n",
    "            cv2.imshow('histogram',lines)\n",
    "            cv2.imshow('image',gray)\n",
    "        elif k == ord('c'):\n",
    "            print('c')\n",
    "            equ = cv2.equalizeHist(gray)\n",
    "            lines = hist_lines(equ)\n",
    "            cv2.imshow('histogram',lines)\n",
    "            cv2.imshow('image',equ)\n",
    "        elif k == ord('d'):\n",
    "            print('d')\n",
    "            curve = hist_curve(gray)\n",
    "            cv2.imshow('histogram',curve)\n",
    "            cv2.imshow('image',gray)\n",
    "        elif k == ord('e'):\n",
    "            print('e')\n",
    "            norm = cv2.normalize(gray, gray, alpha = 0,beta = 255,norm_type = cv2.NORM_MINMAX)\n",
    "            lines = hist_lines(norm)\n",
    "            cv2.imshow('histogram',lines)\n",
    "            cv2.imshow('image',norm)\n",
    "        elif k == 27:\n",
    "            print('ESC')\n",
    "            cv2.destroyAllWindows()\n",
    "            break\n",
    "    cv2.destroyAllWindows()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "On running this script, we get the following output. You just have to click on the 'a','b','c','d' or 'e' key to get the different histograms. \n",
    "\n",
    "a - Histogram for color image in curve mode. \n",
    "\n",
    "<p><img src = \"captures/hista.png\"></p>\n",
    "\n",
    "b - Histogram in bin mode.\n",
    "\n",
    "<p><img src = \"captures/histb.png\"></p>\n",
    "\n",
    "c - Equalized histogram (always in bin mode).\n",
    "\n",
    "<p><img src = \"captures/histc.png\"></p>\n",
    "\n",
    "d - Histogram for color image in curve mode.\n",
    "\n",
    "<p><img src = \"captures/histd.png\"></p>\n",
    "\n",
    "e - Histogram for a normalized image in curve mode.\n",
    "\n",
    "<p><img src = \"captures/histe.png\"></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___\n",
    "## Matching Shapes\n",
    "\n",
    "OpenCV comes with a function cv2.matchShapes() which enables us to compare two shapes, or\n",
    "two contours and returns a metric showing the similarity. The lower the result, the better match it is. It is calculated\n",
    "based on the hu-moment values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.466799997373\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "img1 = cv2.imread('images/star.png',0)\n",
    "#img2 = cv2.imread('images/star.png',0)\n",
    "#img2 = cv2.imread('images/diamond.png',0)\n",
    "#img2 = cv2.imread('images/invertedstar.png',0)\n",
    "#img2 = cv2.imread('images/rect.png',0)\n",
    "#img2 = cv2.imread('images/triangle.png',0)\n",
    "img2 = cv2.imread('images/star2.png',0)\n",
    "\n",
    "ret, thresh = cv2.threshold(img1, 127, 255,0)\n",
    "ret, thresh2 = cv2.threshold(img2, 127, 255,0)\n",
    "_,contours,hierarchy = cv2.findContours(thresh,2,1)\n",
    "cnt1 = contours[0]\n",
    "_,contours,hierarchy = cv2.findContours(thresh2,2,1)\n",
    "cnt2 = contours[0]\n",
    "ret = cv2.matchShapes(cnt1,cnt2,1,0.0)\n",
    "print ret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "I tried matching a few shapes.\n",
    "<img src=\"captures/shapes.png\">\n",
    "\n",
    "Here are the resulting outputs:\n",
    "\n",
    "\n",
    "<table style=\"width:50%\">\n",
    "<tr>\n",
    "<td>Star and Star</td><td>0.0</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>Star and Triangle</td><td>0.0915346946157</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>Star and Diamond</td><td>0.138958575287</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>Star and Inverted Star</td><td>0.0228166713275</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>Star and Rect</td><td>0.327394451118</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "This actually means, among the given shapes; Star and Star i.e itself match the most, then triangle, then diamond and ultimately it differs the most from a rectangle. Note here, while comparing two shapes, the color of these shapes might prove to be a factor while comparing. This means that a filled star would be quite different from a normal star. Check yourself to find out different results.\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Template Matching\n",
    "___\n",
    "_Template matching is a technique in digital image processing for finding small parts of an image which match a template image. It can be used in manufacturing as a part of quality control, a way to navigate a mobile robot, or as a way to detect edges in images._\n",
    "\n",
    "Template Matching is a method for searching and finding the location of a template image in a larger image. OpenCV comes with a function <a href=\"http://docs.opencv.org/3.0.0/df/dfb/group__imgproc__object.html#ga586ebfb0a7fb604b35a23d85391329be\">cv2.matchTemplate()</a> for this purpose. It simply slides the template image over the input image (as in 2D convolution) and compares the template and patch of input image under the template image. Several comparison methods are implemented in OpenCV. It returns a grayscale image, where each pixel denotes how much does the neighbourhood of that pixel match with template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "img_rgb = cv2.imread('letters.jpg')\n",
    "img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)\n",
    "\n",
    "template = cv2.imread('K.jpg',0)\n",
    "#template = cv2.imread('S.jpg',0)\n",
    "w, h = template.shape[::-1]\n",
    "\n",
    "\"\"\"\n",
    "We keep the original RGB image, and create a grayscale version.\n",
    "This is because we do all of the processing in the grayscale version,\n",
    "then use the same coordinates for labels and such on the color image.\n",
    "\n",
    "With the main image, we just have the color version and the grayscale version.\n",
    "We load the template and note the dimensions.\n",
    "\"\"\"\n",
    "\n",
    "res = cv2.matchTemplate(img_gray,template,cv2.TM_CCOEFF_NORMED)\n",
    "threshold = 0.8\n",
    "loc = np.where( res >= threshold)\n",
    "\n",
    "\"\"\"\n",
    "We call res the matchTemplate between the img_gray (our main image),\n",
    "the template, and then the matching method we're going to use.\n",
    "We specify a threshold, here 0.8 for 80%.\n",
    "Then we find locations with a logical statement, where the res is greater than or equal to 80%.\n",
    "\n",
    "Finally, we mark all matches on the original image, using the coordinates we found in the gray image:\n",
    "\"\"\"\n",
    "\n",
    "for pt in zip(*loc[::-1]):\n",
    "    cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0,255,255), 2)\n",
    "\n",
    "cv2.imshow('Detected',img_rgb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To test this example of template matching, I used the following image of alphabets as the testimage and the images of K and S as templates:\n",
    "\n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"images/letters.jpg\">\n",
    "\n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"images/k.jpg\">\n",
    "\n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"images/s.jpg\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The resulting output with detected matches looked like:\n",
    "\n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"captures/ck.png\" /> \n",
    "\n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"captures/cs.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can use Template matching for multiple object detection too.\n",
    "The same code can be applied for the following famous atari game and the results would look as follows:\n",
    "<img src=\"captures/ataritm.png\">\n",
    "\n",
    "Make sure you tweak the threshold according to your own purposes.\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Hough Transforms\n",
    "<a href=\"http://en.wikipedia.org/wiki/Hough_transform\">Hough Transform</a> is a popular technique to detect any shape, if you can represent that shape in mathematical form. It can detect the shape even if it is broken or distorted a little bit.\n",
    "\n",
    "### 1] Detect Lines using Hough Transforms:\n",
    "<a href=\"http://docs.opencv.org/3.0.0/dd/d1a/group__imgproc__feature.html#ga46b4e588934f6c8dfd509cc6e0e4545a\">cv2.HoughLines()</a> is a function in OpenCV to detect lines in an image using hough transforms. The arguments passed in the function are explained: First parameter, Input image should be a binary image, so apply threshold or use canny edge detection before finding applying hough transform. Second and third parameters are ρ and θ accuracies respectively. Fourth argument is the threshold, which means minimum vote it should get for it to be considered as a line. Remember, number of votes depend upon number of points on the line. So it represents the minimum length of line that should be detected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "image1 = cv2.imread('images/chess.png')\n",
    "#image1 = cv2.imread('images/building.jpg')\n",
    "gray=cv2.cvtColor(image1,cv2.COLOR_BGR2GRAY)\n",
    "edges = cv2.Canny(gray, 50, 200)\n",
    "\n",
    "lines= cv2.HoughLines(edges, 1, math.pi/180.0, 185, np.array([]), 0, 0)\n",
    "\n",
    "a,b,c = lines.shape\n",
    "for i in range(a):\n",
    "    rho = lines[i][0][0]\n",
    "    theta = lines[i][0][1]\n",
    "    a = math.cos(theta)\n",
    "    b = math.sin(theta)\n",
    "    x0, y0 = a*rho, b*rho\n",
    "    pt1 = ( int(x0+1000*(-b)), int(y0+1000*(a)) )\n",
    "    pt2 = ( int(x0-1000*(-b)), int(y0-1000*(a)) )\n",
    "    cv2.line(image1, pt1, pt2, (0, 0, 255), 2, cv2.LINE_AA)\n",
    "\n",
    "#cv2.imshow('image1',image1)\n",
    "cv2.imwrite('Results/chessboard1.png',image1)\n",
    "#cv2.imwrite('Results/building1.png',image1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "I used two images to test this code: \n",
    "    \n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"images/chess.png\" width = \"350px\" height = \"350px\" /> \n",
    "\n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"images/building.jpg\" width = \"471px\" height = \"325px\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "The resulting output images were :\n",
    "    \n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"results/chessboard1.png\" width = \"350px\" height = \"350px\" /> \n",
    "\n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"results/building1.png\" width = \"471px\" height = \"325px\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___\n",
    "\n",
    "### 2] Detect Circles using Hough Transforms:\n",
    "<a href=\"http://docs.opencv.org/3.0.0/dd/d1a/group__imgproc__feature.html#ga47849c3be0d0406ad3ca45db65a25d2d\"> cv2.HoughCircles()</a> would b used to detect circular patterns in an image. The following code will explain it's usage. If you want to go deeper into theoretical aspects, please click the links to see the docs, not for only cv2.HoughCircles() but also other links provided for the different mentioned functions before. \n",
    "We would be using the following image of coins which is extensively used by researchers to test and apply Computer Vision algos.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "img = cv2.imread('images/coins.png',0)\n",
    "img = cv2.medianBlur(img,5)\n",
    "cimg = cv2.cvtColor(img,cv2.COLOR_GRAY2BGR)\n",
    "circles = cv2.HoughCircles(img,cv2.HOUGH_GRADIENT,1,20,\n",
    "param1=50,param2=30,minRadius=0,maxRadius=0)\n",
    "\n",
    "circles = np.uint16(np.around(circles))\n",
    "for i in circles[0,:]:\n",
    "    # draw the outer circle\n",
    "    cv2.circle(cimg,(i[0],i[1]),i[2],(0,255,0),2)\n",
    "    # draw the center of the circle\n",
    "    cv2.circle(cimg,(i[0],i[1]),2,(0,0,255),3)\n",
    "    \n",
    "cv2.imshow('detected circles',cimg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Here we used the image of coins which is extensively used by researchers to test and apply Computer Vision algos, as we would see further.\n",
    "After applying houghCircles(), the resulting image had the circles in it detected.\n",
    "\n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"images/coins.png\" /> \n",
    "\n",
    "<img  style=\"float:left;padding-left:40px;padding-right:40px;\" src=\"captures/circles.png\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Image Segmentation\n",
    "___\n",
    "\n",
    "For Image Segmentation, OpenCV uses a marker-based watershed algorithm where we give different labels for our object we know.\n",
    "Label the region which we are sure of being the foreground or object with one color (or intensity), label the region\n",
    "which we are sure of being background or non-object with another color and finally the region which we are not sure\n",
    "of anything, label it with 0. That is our marker. Then apply watershed algorithm. Then our marker will be updated\n",
    "with the labels we gave, and the boundaries of objects will have a value of -1\n",
    "We will use the Distance Transform along with watershed to segment mutually touching objects. We will use the same image of coins as above to understand the procedure of Image Segmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "from matplotlib import pyplot as plt\n",
    "img = cv2.imread('images/coins.png')\n",
    "gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)\n",
    "# To get an estimation of the coins, we performed binarization using Otsu's Method.\n",
    "ret, thresh = cv2.threshold(gray,0,255,cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The resulting image would look like:\n",
    "    <img src=\"images/water_thresh.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To remove the small white noises in the image, we'll use morphological opening whereas to remove any small holes in the object, we can use morphological closing. At the moment we have confirmed that the region near to center of objects are foreground and region much away from the object are background. Now we have to define the boundary region of coins.\n",
    "\n",
    "So we need to extract the area which we are sure they are coins. For that we could find the distance transform and apply a proper threshold. Next we need to find the area which we are sure they are not coins. For that, we dilate the result. Dilation increases object boundary to background. This way, we can make sure whatever region in background in result is really a background, since boundary region is removed. See the resulting image below.\n",
    "<img src=\"images/water_fgbg.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now Watershed algorithm would be used to confirm the remaining regions are still unconfirmed, whether it is coins or background.  These areas are normally around the border i.e. boundaries of coins where foreground and background meet (Or\n",
    "even two different coins meet). It can be obtained from subtracting sure_fg area from sure_bg area."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# noise removal\n",
    "kernel = np.ones((3,3),np.uint8)\n",
    "opening = cv2.morphologyEx(thresh,cv2.MORPH_OPEN,kernel, iterations = 2)\n",
    "# sure background area\n",
    "sure_bg = cv2.dilate(opening,kernel,iterations=3)\n",
    "# Finding sure foreground area\n",
    "dist_transform = cv2.distanceTransform(opening,cv2.DIST_L2,5)\n",
    "ret, sure_fg = cv2.threshold(dist_transform,0.7*dist_transform.max(),255,0)\n",
    "# Finding unknown region\n",
    "sure_fg = np.uint8(sure_fg)\n",
    "unknown = cv2.subtract(sure_bg,sure_fg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "See the image below. In the thresholded image, we get some regions of coins which we are sure of coins and they are detached\n",
    "now. \n",
    "[All of this we are doing in image segmentation is seperating the foreground from the background and then seperating the indivdual segments in the foreground by there border. If all you want to do is foreground segmentation and not separate the mutually touching objects, you need not use distance transform, just erosion is sufficient. Erosion is just another method to\n",
    "extract sure foreground area.)\n",
    "<img src=\"images/water_dt.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As we can see we are confident to distinguish among the coins and the background. Hence we create a marker (i.e an array of same size as that of original image, but with int32 datatype) and label the regions inside it. The regions we know for sure (whether foreground or background) are labelled with any positive integers, but different integers, and the area we don't know for sure are just left as zero. For this we use <a href=\"http://docs.opencv.org/3.0.0/d3/dc0/group__imgproc__shape.html#gac2718a64ade63475425558aa669a943a\">cv2.connectedComponents()</a>. It labels background of the image with 0, then other objects are labelled with integers starting from 1.\n",
    "\n",
    "However if background is marked with 0, watershed will consider it as unknown area. So we want to mark it with different integer. Instead, we will mark unknown region, defined by unknown, with 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Marker labelling\n",
    "ret, markers = cv2.connectedComponents(sure_fg)\n",
    "# Add one to all labels so that sure background is not 0, but 1\n",
    "markers = markers+1\n",
    "# Now, mark the region of unknown with zero\n",
    "markers[unknown==255] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "See the result shown in JET colormap. The dark blue region shows unknown region. Sure coins are colored with\n",
    "different values. Remaining area which are sure background are shown in lighter blue compared to unknown region.\n",
    "<img src=\"images/water_marker.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that our marker is ready, we would be finally applying watershed algorithm. Then marker image will be modified. The boundary\n",
    "region will be marked with -1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "markers = cv2.watershed(img,markers)\n",
    "img[markers == -1] = [255,0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "See the result below. For some coins, the region where they touch are segmented properly and for some, they are not:\n",
    "<img src=\"images/water_result.jpg\">\n",
    "\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Foreground Extraction\n",
    "___\n",
    "Here will see GrabCut algorithm to extract foreground in images. The way this algo works can be put in simple terms as it basically finds the foreground and removes the background.\n",
    "\n",
    "From a user's point of view: Initially user draws a rectangle around the foreground region. Then algorithm segments it iteratively to get the best result. User is required to provide some touchups in case, the segmentation may have marked some foreground region as background and vice versa. User just gives some strokes on the images where some faulty results are there. Then in the next iteration, you get better results.\n",
    "\n",
    "See the image below. First player and football is enclosed in a blue rectangle. Then some final touchups with white strokes (denoting foreground) and black strokes (denoting background) is made. And we get a nice result.\n",
    "<img src=\"images/grabcut_output1.jpg\">\n",
    "\n",
    "\n",
    "\n",
    "Below we will see the code to implement this.\n",
    "\n",
    "We would be using the same Messi image as an example:\n",
    "\n",
    "<img src=\"images/messi.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First we import the modules, load in our image, create a mask and specify the background and foreground model, which is used by the algorithm internally. Then the real important part is the rect we define. Arguments to be passed in rect are: rect = **(start_x, start_y, width, height)**. If you are using your own image, you have to define your own rectangle, and a simple way to do it would be to open the image in _Microsoft Paint_ or other image editors and just hover around to find the top left and then  the respective width and height of the rectangle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "img = cv2.imread('images/messi.png')\n",
    "#Creating the mask\n",
    "mask = np.zeros(img.shape[:2],np.uint8)\n",
    "\n",
    "# Specifying the background and the foreground model.\n",
    "bgdModel = np.zeros((1,65),np.float64)\n",
    "fgdModel = np.zeros((1,65),np.float64)\n",
    "\n",
    "# Drawing a rectangle around the foreground.\n",
    "rect = (50,50,380,450)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Then we would be using cv2.grabCut(), which has the folloeing parameters. First the input image, then the mask, then the rectangle for our main object, the background model, foreground model, the amount of iterations to run, and what mode you are using.\n",
    "\n",
    "From here, the mask is changed so that all 0 and 2 pixels are converted to the background, where the 1 and 3 pixels are now the foreground. Then we multiply with the input image to get our final result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "cv2.grabCut(img,mask,rect,bgdModel,fgdModel,5,cv2.GC_INIT_WITH_RECT)\n",
    "mask2 = np.where((mask==2)|(mask==0),0,1).astype('uint8')\n",
    "img = img*mask2[:,:,np.newaxis]\n",
    "\n",
    "plt.imshow(img)\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The foreground extraction of our image would look like:\n",
    "<img src=\"captures/foreground.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Here is the full code to be implemented:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "img = cv2.imread('messi.png')\n",
    "mask = np.zeros(img.shape[:2],np.uint8)\n",
    "\n",
    "bgdModel = np.zeros((1,65),np.float64)  # background model\n",
    "fgdModel = np.zeros((1,65),np.float64)  # foreground model\n",
    "\n",
    "rect = (50,50,380,450)\n",
    "\n",
    "cv2.grabCut(img,mask,rect,bgdModel,fgdModel,5,cv2.GC_INIT_WITH_RECT)\n",
    "\"\"\"\n",
    "Parameters: First the input image, then the mask,\n",
    "then the rectangle for our main object, the background model, foreground model,\n",
    "the amount of iterations to run, and what mode you are using\n",
    "\"\"\"\n",
    "\n",
    "mask2 = np.where((mask==2)|(mask==0),0,1).astype('uint8')\n",
    "img = img*mask2[:,:,np.newaxis]\n",
    "\"\"\"\n",
    "From here, the mask is changed so that all 0 and 2 pixels are converted to the background,\n",
    "where the 1 and 3 pixels are now the foreground.\n",
    "Then we multiply with the input image, and we get our final result.\n",
    "\"\"\"\n",
    "plt.imshow(img)\n",
    "plt.colorbar()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}