1.2-vgg16-from-scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 從零開始構建VGG網絡來學習Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras使得創建深度學習模型變得快速而簡單, 雖然如此很多時候我們只要複製許多官網的範例就\n",
    "可做出很多令人覺得驚奇的結果。但是當要解決的問題需要進行一些模型的調整與優化或是需要構\n",
    "建出一個新論文的網絡結構的時候, 我們就可能會左支右拙的難以招架。\n",
    "\n",
    "在本教程中，您將通過閱讀VGG的原始論文從零開始使用Keras來構建在ILSVRC-2014 (ImageNet competition)競賽中獲的第一名的VGG (Visual Geometry Group, University of Oxford)網絡結構。\n",
    "\n",
    "\n",
    "那麼，重新構建別人已經構建的東西有什麼意義呢？重點是學習。通過完成這次的練習，您將：\n",
    "* 了解更多關於VGG的架構\n",
    "* 了解有關卷積神經網絡的更多信息\n",
    "* 了解如何在Keras中實施某種網絡結構\n",
    "* 通過閱讀論文並實施其中的某些部分可以了解更多底層的原理與原始構想\n",
    "\n",
    "![vgg16](https://www.cs.toronto.edu/~frossard/post/vgg16/vgg16.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 為什麼從VGG開始？\n",
    "* 它很容易實現\n",
    "* 它在ILSVRC-2014（ImageNet競賽）上取得了優異的成績\n",
    "* 它今天被廣泛使用\n",
    "* 它的論文簡單易懂\n",
    "* Keras己經實現VGG在散佈的版本中，所以你可以用來參考與比較"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 讓我們從論文中挖寶\n",
    "在VGG的論文-[VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION](https://arxiv.org/pdf/1409.1556v6.pdf)的2.3章節中提到了以下幾種VGG的網絡結構:\n",
    "\n",
    "![vgg network structure](https://cdn-images-1.medium.com/max/1600/1*FRd9fDM1TXThW2V8ylL7VQ.png)\n",
    "\n",
    "根據論文的測試給果 D (VGG16)與 E (VGG19) 是效果最好的, 由於這兩種網絡構建的方法與技巧幾乎相同, 因此我們選手構建**D (VGG16)**這個網絡結構類型。 \n",
    "\n",
    "在2.1章節中論述了詳細的卷積層的相關訊息:\n",
    "![vgg 2.1](https://cdn-images-1.medium.com/max/1600/1*DwsWBmGCI7qL9ei7n_SdXA.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "歸納一下論文網絡構建訊息:\n",
    "* 輸入圖像尺寸(`input size`)：224 x 224\n",
    "* 感受過瀘器(`receptive field`)的大小是3 x 3\n",
    "* 卷積步長(`stride`)是1個像素\n",
    "* 填充(`padding`)是1（對於3 x 3的感受過瀘器）\n",
    "* 池化層的大小是2×2且步長(`stride`)為2像素\n",
    "* 有兩個完全連接層，每層4096個神經元\n",
    "* 最後一層是具有1000個神經元的`softmax`分類層（代表1000個ImageNet類別）\n",
    "* 激勵函數是`ReLU`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Platform: Windows-7-6.1.7601-SP1\n",
      "Tensorflow version: 1.4.0\n",
      "Keras version: 2.1.1\n"
     ]
    }
   ],
   "source": [
    "# 這個Jupyter Notebook的環境\n",
    "import platform\n",
    "import tensorflow\n",
    "import keras\n",
    "print(\"Platform: {}\".format(platform.platform()))\n",
    "print(\"Tensorflow version: {}\".format(tensorflow.__version__))\n",
    "print(\"Keras version: {}\".format(keras.__version__))\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import numpy as np\n",
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 創建模型 (Sequential)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      \n",
      "_________________________________________________________________\n",
      "block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     \n",
      "_________________________________________________________________\n",
      "block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         \n",
      "_________________________________________________________________\n",
      "block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     \n",
      "_________________________________________________________________\n",
      "block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    \n",
      "_________________________________________________________________\n",
      "block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         \n",
      "_________________________________________________________________\n",
      "block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    \n",
      "_________________________________________________________________\n",
      "block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    \n",
      "_________________________________________________________________\n",
      "block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    \n",
      "_________________________________________________________________\n",
      "block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         \n",
      "_________________________________________________________________\n",
      "block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   \n",
      "_________________________________________________________________\n",
      "block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         \n",
      "_________________________________________________________________\n",
      "block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 25088)             0         \n",
      "_________________________________________________________________\n",
      "fc1 (Dense)                  (None, 4096)              102764544 \n",
      "_________________________________________________________________\n",
      "fc2 (Dense)                  (None, 4096)              16781312  \n",
      "_________________________________________________________________\n",
      "predictions (Dense)          (None, 1000)              4097000   \n",
      "=================================================================\n",
      "Total params: 138,357,544\n",
      "Trainable params: 138,357,544\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPool2D\n",
    "from keras.utils import plot_model\n",
    "\n",
    "# 定義輸入\n",
    "input_shape = (224, 224, 3) # RGB影像224x224 (height, width, channel)\n",
    "\n",
    "# 使用'序貫模型(Sequential)來定義\n",
    "model = Sequential(name='vgg16-sequential')\n",
    "\n",
    "# 第1個卷積區塊 (block1)\n",
    "model.add(Conv2D(64, (3, 3), padding='same', activation='relu', input_shape=input_shape, name='block1_conv1'))\n",
    "model.add(Conv2D(64, (3, 3), padding='same', activation='relu', name='block1_conv2'))\n",
    "model.add(MaxPool2D((2, 2), strides=(2, 2), name='block1_pool'))\n",
    "\n",
    "# 第2個卷積區塊 (block2)\n",
    "model.add(Conv2D(128, (3, 3), padding='same', activation='relu', name='block2_conv1'))\n",
    "model.add(Conv2D(128, (3, 3), padding='same', activation='relu', name='block2_conv2'))\n",
    "model.add(MaxPool2D((2, 2), strides=(2, 2), name='block2_pool'))\n",
    "\n",
    "# 第3個卷積區塊 (block3)\n",
    "model.add(Conv2D(256, (3, 3), padding='same', activation='relu', name='block3_conv1'))\n",
    "model.add(Conv2D(256, (3, 3), padding='same', activation='relu', name='block3_conv2'))\n",
    "model.add(Conv2D(256, (3, 3), padding='same', activation='relu', name='block3_conv3'))\n",
    "model.add(MaxPool2D((2, 2), strides=(2, 2), name='block3_pool'))\n",
    "\n",
    "# 第4個卷積區塊 (block4)\n",
    "model.add(Conv2D(512, (3, 3), padding='same', activation='relu', name='block4_conv1'))\n",
    "model.add(Conv2D(512, (3, 3), padding='same', activation='relu', name='block4_conv2'))\n",
    "model.add(Conv2D(512, (3, 3), padding='same', activation='relu', name='block4_conv3'))\n",
    "model.add(MaxPool2D((2, 2), strides=(2, 2), name='block4_pool'))\n",
    "\n",
    "# 第5個卷積區塊 (block5)\n",
    "model.add(Conv2D(512, (3, 3), padding='same', activation='relu', name='block5_conv1'))\n",
    "model.add(Conv2D(512, (3, 3), padding='same', activation='relu', name='block5_conv2'))\n",
    "model.add(Conv2D(512, (3, 3), padding='same', activation='relu', name='block5_conv3'))\n",
    "model.add(MaxPool2D((2, 2), strides=(2, 2), name='block5_pool'))\n",
    "\n",
    "# 前饋全連接區塊\n",
    "model.add(Flatten(name='flatten'))\n",
    "model.add(Dense(4096, activation='relu', name='fc1'))\n",
    "model.add(Dense(4096, activation='relu', name='fc2'))\n",
    "model.add(Dense(1000, activation='softmax', name='predictions'))\n",
    "\n",
    "# 打印網絡結構\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 確認模型訓練的參數總數\n",
    "根據論文2.3章節的訊息與我們模型的網絡結構參數比對,我們構建的模型138,357,544參數的確符合論文提及的138百萬的訓練參數。\n",
    "\n",
    "![model-parameters](https://cdn-images-1.medium.com/max/1600/1*cY2yGoETl0jBrtky8PGysg.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 創建模型 (Functaional API)\n",
    "\n",
    "使用Keras的functiona api來定義網絡結構。詳細的說明與參考:\n",
    "* [keras.io](https://keras.io/models/model/)\n",
    "* [如何使用Keras函數式API進行深度學習](https://github.com/erhwenkuo/deep-learning-with-keras-notebooks/blob/master/1.1-keras-functional-api.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "img_input (InputLayer)       (None, 224, 224, 3)       0         \n",
      "_________________________________________________________________\n",
      "block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      \n",
      "_________________________________________________________________\n",
      "block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     \n",
      "_________________________________________________________________\n",
      "block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         \n",
      "_________________________________________________________________\n",
      "block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     \n",
      "_________________________________________________________________\n",
      "block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    \n",
      "_________________________________________________________________\n",
      "block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         \n",
      "_________________________________________________________________\n",
      "block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    \n",
      "_________________________________________________________________\n",
      "block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    \n",
      "_________________________________________________________________\n",
      "block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    \n",
      "_________________________________________________________________\n",
      "block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         \n",
      "_________________________________________________________________\n",
      "block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   \n",
      "_________________________________________________________________\n",
      "block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         \n",
      "_________________________________________________________________\n",
      "block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   \n",
      "_________________________________________________________________\n",
      "block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 25088)             0         \n",
      "_________________________________________________________________\n",
      "fc1 (Dense)                  (None, 4096)              102764544 \n",
      "_________________________________________________________________\n",
      "fc2 (Dense)                  (None, 4096)              16781312  \n",
      "_________________________________________________________________\n",
      "predictions (Dense)          (None, 1000)              4097000   \n",
      "=================================================================\n",
      "Total params: 138,357,544\n",
      "Trainable params: 138,357,544\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Activation, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPool2D\n",
    "\n",
    "# 定義輸入\n",
    "input_shape = (224, 224, 3) # RGB影像224x224 (height, width, channel)\n",
    "\n",
    "# 輸入層\n",
    "img_input = Input(shape=input_shape, name='img_input')\n",
    "\n",
    "# 第1個卷積區塊 (block1)\n",
    "x = Conv2D(64, (3, 3), padding='same', activation='relu', name='block1_conv1')(img_input)\n",
    "x = Conv2D(64, (3, 3), padding='same', activation='relu', name='block1_conv2')(x)\n",
    "x = MaxPool2D((2, 2), strides=(2, 2), name='block1_pool')(x)\n",
    "\n",
    "# 第2個卷積區塊 (block2)\n",
    "x = Conv2D(128, (3, 3), padding='same', activation='relu', name='block2_conv1')(x)\n",
    "x = Conv2D(128, (3, 3), padding='same', activation='relu', name='block2_conv2')(x)\n",
    "x = MaxPool2D((2, 2), strides=(2, 2), name='block2_pool')(x)\n",
    "\n",
    "# 第3個卷積區塊 (block3)\n",
    "x = Conv2D(256, (3, 3), padding='same', activation='relu', name='block3_conv1')(x)\n",
    "x = Conv2D(256, (3, 3), padding='same', activation='relu', name='block3_conv2')(x)\n",
    "x = Conv2D(256, (3, 3), padding='same', activation='relu', name='block3_conv3')(x)\n",
    "x = MaxPool2D((2, 2), strides=(2, 2), name='block3_pool')(x)\n",
    "\n",
    "# 第4個卷積區塊 (block4)\n",
    "x = Conv2D(512, (3, 3), padding='same', activation='relu', name='block4_conv1')(x)\n",
    "x = Conv2D(512, (3, 3), padding='same', activation='relu', name='block4_conv2')(x)\n",
    "x = Conv2D(512, (3, 3), padding='same', activation='relu', name='block4_conv3')(x)\n",
    "x = MaxPool2D((2, 2), strides=(2, 2), name='block4_pool')(x)\n",
    "\n",
    "# 第5個卷積區塊 (block5)\n",
    "x = Conv2D(512, (3, 3), padding='same', activation='relu', name='block5_conv1')(x)\n",
    "x = Conv2D(512, (3, 3), padding='same', activation='relu', name='block5_conv2')(x)\n",
    "x = Conv2D(512, (3, 3), padding='same', activation='relu', name='block5_conv3')(x)\n",
    "x = MaxPool2D((2, 2), strides=(2, 2), name='block5_pool')(x)\n",
    "\n",
    "# 前饋全連接區塊\n",
    "x = Flatten(name='flatten')(x)\n",
    "x = Dense(4096, activation='relu', name='fc1')(x)\n",
    "x = Dense(4096, activation='relu', name='fc2')(x)\n",
    "x = Dense(1000, activation='softmax', name='predictions')(x)\n",
    "\n",
    "# 產生模型\n",
    "model2 = Model(inputs=img_input, outputs=x, name='vgg16-funcapi')\n",
    "\n",
    "# 打印網絡結構\n",
    "model2.summary()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型訓練\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要用ImageNet的資料來訓練VGG16的模型則不是一件容易的事喔。\n",
    "\n",
    "VGG論文指出：\n",
    "\n",
    ">On a system equipped with four NVIDIA Titan Black GPUs, training a single net took 2–3 weeks depending on the architecture.\n",
    "\n",
    "也就是說就算你有四張NVIDIA的Titan網卡用Imagenet的影像集來訓練VGG16模型, 可能也得花個2-3星期。即使買的起這樣的硬體，你也得花蠻多的時間來訓練這個模型。\n",
    "\n",
    "幸運的是Keras不僅己經在它的模組中包括了VGG16與VGG19的模型定義以外, 同時也幫大家預訓練好了VGG16與VGG19的模型權重。\n",
    "\n",
    "詳請請見: [keras.io](https://keras.io/applications/#vgg16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 總結 (Conclusion)\n",
    "\n",
    "在這篇文章中有一些個人學習到的一些有趣的重點:\n",
    "    \n",
    "* 在Keras中要建構一個網絡不難, 但了解這個網絡架構的原理則需要多一點耐心\n",
    "* VGG16構建簡單效能高,真是神奇!\n",
    "* VGG16在卷積層的設計是愈後面feature map的size愈小, 而過瀘器(receptive field/fiter/kernel)則愈多"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "參考: \n",
    "* [Learning Keras by Implementing the VGG Network From Scratch](https://hackernoon.com/learning-keras-by-implementing-vgg16-from-scratch-d036733f2d5/)\n",
    "* [Keras官網](http://keras.io/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}