ITSC2018_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Flow Hands-On Tutorial\n",
    "\n",
    "The best way to get start with reinforcement learning... is to get started with reinforcement learning! In this tutorial, we'll use Flow, our tool for optimization of traffic networks by applying control, to train autonomous vehicles to stbilize a network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 0: Installing Flow\n",
    "\n",
    "We begin this tutorial by installing Flow and all its necessary dependencies, namely sumo and some machine learning and python-based modules.\n",
    "\n",
    "### a. Using Windows? (If not, continue to part b)\n",
    "\n",
    "Not all the software packages we wish to install work natively on Windows. Instead, if you are using Windows 10, we recommend you install a Windows Linux Subsystem (WLS) onto your device. In order to do so:\n",
    "\n",
    "- Go the Windows store and download “Ubuntu 18.04”\n",
    "- Download the Xming X Server for Windows: https://sourceforge.net/projects/xming/\n",
    "- Run the WLS from the start menu by typing “Ubuntu 18.04”\n",
    "    - The first time you open an Ubuntu terminal, type: `echo “export DISPLAY=:0” >> ~/.bashrc && source ~/.bashrc`\n",
    "    - In order for graphic user intergace to work properly, make sure to also run Xming whenever you open a new terminal\n",
    "\n",
    "If you are using an earlier version of Windows, your only other option is to install a virtual machine (e.g. [VirtualBox](https://www.virtualbox.org/wiki/Downloads)), set up an [Ubuntu](https://www.ubuntu.com/download/desktop) virtual environment, and install everything you need onto it. If you are in this situation and need some help setting up a virtual environment, please talk to one of the assistants.\n",
    "\n",
    "### b. Installed Anaconda/Miniconda? (If yes, continue to part c)\n",
    "\n",
    "Conda environments are an ideal way to create, export, list, remove and update environments that have different versions of Python and/or packages installed in them. If you have not installed [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://conda.io/miniconda.html) before, we highly recommend you do so now. Moreover, since Anaconda takes long to install, we recommend installing the latter. In order to do so, run the following commands from your terminal (**Note**: update the second command to include the correct URL for your distribution):\n",
    "\n",
    "    cd ~\n",
    "    wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;\n",
    "    bash miniconda.sh -b -p $HOME/miniconda\n",
    "echo 'export PATH=\"$HOME/miniconda/bin:$PATH\"' >> ~/.bashrc\n",
    "\n",
    "### c. Installation instructions\n",
    "\n",
    "You are now prepared to install Flow and its dependencies, e.g. sumo. In order to do so, follow the setup instructions located at: https://flow.readthedocs.io/en/latest/flow_setup.html. When installing sumo, follow the instruction located in [section e](https://flow.readthedocs.io/en/latest/flow_setup.html#e-easy-install-sumo-optional).\n",
    "\n",
    "### d. Unable to install Flow?\n",
    "\n",
    "If you unable to install Flow, or if the installation instruction is taking too long, we recommend using an EC2 instance provided by us (as you will in problem 2c). Please see that section to understand how to do so."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1: Simulating Traffic in a Ring Road\n",
    "\n",
    "In this problem, we will simulate traffic instabilities on a ring road network. The formation of traffic instabilities (often referred to as traffic congestion, stop-and-go traffic, traffic waves, etc.) in ring roads is a widely studied problem, as it is analytically tractable and has been empirically shown to generate similar instabilities in field studies (see [this video](https://www.youtube.com/watch?v=7wm-pZp_mi0)) as those witnessed in real network settings such as highways (see [this video](https://www.youtube.com/watch?v=6ZC9h8jgSj4)). We will simulate the performance of vehicles in a ring road using the microscopic traffic simulator SUMO (see the figure below).\n",
    "\n",
    "<img src=\"img/ring_road.png\" width=\"400\">\n",
    "\n",
    "### a. Modeling microscopic car-following dynamics\n",
    "\n",
    "We begin by implementing a car-following model in Flow that can recreate the types of traffic instabilities experienced in reality. Several car-following models exist to realistically depict the longitudinal (acceleration) behavior of vehicles in a network. One such model is the Intelligent Driver Model (IDM), in which the acceleration $a_{IDM}$ of a vehicle is defined as:\n",
    "\n",
    "\\begin{equation}\n",
    "a_{IDM}(v, v_l, h) = a \\bigg[ 1 - \\bigg( \\frac{v}{v_0} \\bigg) ^\\delta - \\bigg( \\frac{s^* (v, v_l)}{h} \\bigg)^2 \\bigg]\n",
    "\\end{equation}\n",
    "\n",
    "where $v$ the vehicle's speed, $h$ is its bumper-to-bumper headway, $v_l$ is the speed of the vehicle ahead of it, and $s^*$ is the desired headway of the vehicle, denoted by:\n",
    "\n",
    "\\begin{equation}\n",
    "s^*(v, v_l)  = s_0 + \\max \\bigg( 0, v T + \\frac{v (v - v_l)}{2 \\sqrt{ab}}  \\bigg)\n",
    "\\end{equation}\n",
    "\n",
    "where $s_0$, $v_0$, $T$, $\\delta$, $a$, $b$ are given parameters that may be calibrated to model highway traffic.\n",
    "\n",
    "In order to create an acceleration, or car-following, model in Flow, we will used the `BaseController` class. This class can be inherited and it's `get_accel` method can be modified to recreate the desirable acceleration at every given time step (see the cell below).\n",
    "\n",
    "Using the `BaseController` class in Flow, design a controller called *IDM* that can recreate the behavior of this model in simulation. You can create this controller class by filling in the below script. Use the following values for each of the model parameters:\n",
    "\n",
    "- $s_0$: 2 m\n",
    "- $v_0$: 30 m/s\n",
    "- $T$: 1 s\n",
    "- $\\delta$: 4\n",
    "- $a$: 1 m/s$^2$\n",
    "- $b$: 1.5 m/s$^2$\n",
    "\n",
    "For more information of designing controllers in Flow, we recommend you review this [tutorial](https://github.com/berkeleyflow/flow/blob/master/tutorials/tutorial07_controllers.ipynb). **Note**: You are allowed to import any module you find valuable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.controllers import BaseController\n",
    "\n",
    "\n",
    "class IDM(BaseController):\n",
    "\n",
    "    def get_accel(self, env):\n",
    "        # bumper-to-bumper headway\n",
    "        h = env.vehicles.get_headway(self.veh_id)\n",
    "\n",
    "        # speed of the current vehicle\n",
    "        v = env.vehicles.get_speed(self.veh_id)\n",
    "\n",
    "        # speed of the lead vehicle\n",
    "        v_l = env.vehicles.get_speed(env.vehicles.get_leader(self.veh_id))\n",
    "\n",
    "        ######################################\n",
    "        ###### your implementation here ######\n",
    "        acceleration = \n",
    "        ######################################\n",
    "        ######################################\n",
    "\n",
    "        # return the acceleration of the current vehicle\n",
    "        return acceleration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b. Executing the simulation\n",
    "\n",
    "Next, we will run the simulation in SUMO using the *SumoExperiment* class in Flow. This class allows us to specify the type of scenario we would like to simulate as well as the longitudinal and lateral dynamics of vehicles in the simulation. Referring to the [tutorial in Flow on simulating traffic](https://github.com/berkeleyflow/flow/blob/master/tutorials/tutorial01_sumo.ipynb), fill in the below parameters in order to produce an experiment with a single lane ring road network of length 230 m with a total of 22 vehicles following the IDM model from part a), where the vehicles are initially perturbed from equal spacing by an additive random normal term with standard deviation 1.0 m."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# some objects we will use to define the parameters of the simulation\n",
    "from flow.core.params import EnvParams, SumoParams, NetParams, InitialConfig\n",
    "from flow.core.vehicles import Vehicles\n",
    "\n",
    "# these is the scenario class for the ring road \n",
    "# (this does not need to be modified)\n",
    "from flow.scenarios import LoopScenario\n",
    "\n",
    "# this is the environment we will be using during the simulation \n",
    "# (it does not need to be modified)\n",
    "from flow.envs import TestEnv\n",
    "\n",
    "# the EnvParams object is left empty for the above environment \n",
    "# (this does not need to be modified)\n",
    "env_params = EnvParams()\n",
    "\n",
    "######################################################\n",
    "############ modify everything below here ############\n",
    "######################################################\n",
    "\n",
    "# add 22 vehicles with the \"IDM\" acceleration controller from section a)\n",
    "vehicles = Vehicles()\n",
    "vehicles.add()  ### modify this function call ###\n",
    "\n",
    "# modify the NetParams object to support a ring road of length 230 m\n",
    "net_params = NetParams()  ### modify this class instantiation ###\n",
    "\n",
    "# start all vehicles with perturbation standard deviation of 1.0 m\n",
    "initial_config = InitialConfig()  ### modify this class instantiation ###\n",
    "\n",
    "# run the simulation with a simulation step of 0.1s and activate the GUI for visualization purposes\n",
    "sumo_params = SumoParams()  ### modify this class instantiation ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the above parameters are ready, we can start the simulation using the code snippet below to see how well the network performs when the vehicles are initially perturbed. If your model and network are designed correctly, then after some time the vehicles should begin bunching together and accelerating quickly when they are at the front of the backwards propagating queue. This is known as a \"stop-and-go wave\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.experiment import SumoExperiment\n",
    "import numpy as np\n",
    "\n",
    "scenario = LoopScenario(name=\"ring_road\",\n",
    "                        vehicles=vehicles,\n",
    "                        net_params=net_params,\n",
    "                        initial_config=initial_config)\n",
    "\n",
    "env = TestEnv(env_params, sumo_params, scenario)\n",
    "\n",
    "exp = SumoExperiment(env, scenario)\n",
    "info_dict = exp.run(1, 3000)\n",
    "\n",
    "print(\"------------------\")\n",
    "print(\"Average speed in final time step: {} m/s\".format(info_dict[\"velocities\"][0][-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### c. Additional exercises (optional)\n",
    "\n",
    "In this optional section, we provide you with additional exercises to further familiarize yourself with the workings of flow. While the components of the previous section closely followed the content available within the tutorial, these questions are meant to encourage you to further explore the various parametrizations of a scenario and/or environment.\n",
    "\n",
    "To begin with, in the below cell, modify the relevant components of the cell at the start of section 1.b) to replace the ring road scenario with a figure eight with 14 vehicles (we will actually use this scenario in question 2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import  ### FILL IN ###\n",
    "\n",
    "### specify the new vehicles component\n",
    "vehicles_figureeight = \n",
    "\n",
    "### specify the new net_params component\n",
    "### Note: you should set no_internal_links to False to include intersections\n",
    "net_params_figureeight = \n",
    "\n",
    "### recreate the scenario ###\n",
    "scenario =\n",
    "\n",
    "# everything else is the same as before\n",
    "env = TestEnv(env_params, sumo_params, scenario)\n",
    "exp = SumoExperiment(env, scenario)\n",
    "_ = exp.run(1, 1500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next. modify the cell below to generate random initial positions for vehicles in the ring road."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "initial_config_random = InitialConfig()  ### modify this line ###\n",
    "\n",
    "# add the new parameter\n",
    "scenario = LoopScenario(\n",
    "    name=\"ring_road\",\n",
    "    vehicles=vehicles,\n",
    "    net_params=net_params,\n",
    "    initial_config=initial_config_random,  # use the new random initial_config\n",
    ")\n",
    "\n",
    "# everything else is the same as before\n",
    "env = TestEnv(env_params, sumo_params, scenario)\n",
    "exp = SumoExperiment(env, scenario)\n",
    "_ = exp.run(1, 3000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, add a traffic light at the node of the ring road called \"right\" using the `add` method from the `TrafficLights` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.traffic_lights import TrafficLights\n",
    "\n",
    "# place a traffic light at the central node\n",
    "traffic_lights = TrafficLights()\n",
    "traffic_lights.  ### modify this line ###\n",
    "\n",
    "# add the new parameter\n",
    "scenario = LoopScenario(\n",
    "    name=\"ring_road\",\n",
    "    vehicles=vehicles,\n",
    "    net_params=net_params,\n",
    "    initial_config=initial_config,\n",
    "    traffic_lights=traffic_lights,\n",
    ")\n",
    "\n",
    "# everything else is the same as before\n",
    "env = TestEnv(env_params, sumo_params, scenario)\n",
    "exp = SumoExperiment(env, scenario)\n",
    "_ = exp.run(1, 3000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2: Training RL Experiments with Flow\n",
    "\n",
    "Having walked through the procedure through which traffic can be simulated in Flow, we will now walk through the process through which an MDP representing a certain traffic-motivated task can be generated in Flow, and will then attempt to use reinforcement learning techniques to train autonomous vehicles to mitigate traffic deficiencies. For this question, we will consider the toy problem of coordinating vehicles through an intersection represented as a figure eight (see the figure below).\n",
    "\n",
    "<img src=\"img/figure_eight.png\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a. Creating custom environments\n",
    "\n",
    "We begin by designing an MDP that is representative of our problem. Using the Flow computational framework, this can be done by creating an environment object, similar to `TestEnv` environment we were using during simulation in question 1.\n",
    "\n",
    "In the below cell, design an environment than can be used to coordinate vehicles through a figure eight intersection. In order to do so, perform the following tasks:\n",
    "\n",
    "1. Modify the `observation_space` and `get_state` methods so that your state is the speed and position of every vehicle in the network. When collecting the positions of vehicle, use the `self.get_x_by_id` method.\n",
    "2. Modify the `action_space` and `_apply_rl_actions` methods so that your policy's actions are converted to desired accelerations by the RL vehicle in the environment. The actions as defined in the `action_space` method should *not* be bounded.\n",
    "3. Modify the `compute_reward` function to return the average speed of vehicles in the network.\n",
    "\n",
    "**Hints**:\n",
    "\n",
    "- For a review of creating custom environments in flow, please see the following [tutorial](https://github.com/flow-project/flow/blob/master/tutorials/tutorial06_environments.ipynb).\n",
    "- Individual vehicle state information can be collected from the Vehicles class within an environmnet (called by `self.vehicles`). Refer to [this file](https://github.com/flow-project/flow/blob/master/flow/core/vehicles.py) for what sort of information can be collected. The same could be done for scenario/network information using the variable `self.scenario`, which the associated get methods available [here](https://github.com/flow-project/flow/blob/master/flow/scenarios/base_scenario.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.envs import Env\n",
    "import numpy as np\n",
    "from gym.spaces import Box\n",
    "\n",
    "class MyEnv(Env):  # create a new environment\n",
    "    \"\"\"Environment used to train vehicles to coordinate through\n",
    "    an intersection.\n",
    "\n",
    "    States\n",
    "        The states are the speeds and positions of all vehicles.\n",
    "\n",
    "    Actions\n",
    "        The actions are an acceleration for each automated vehicle.\n",
    "\n",
    "    Rewards\n",
    "        The reward function is the average speed of all vehicles in \n",
    "        the network.\n",
    "\n",
    "    Termination\n",
    "        The rollout is terminated if the time horizon is met.\n",
    "    \"\"\"\n",
    "\n",
    "    @property\n",
    "    def action_space(self):\n",
    "        ##############################################################\n",
    "        # specify dimensions and properties of the action space here #\n",
    "        ##############################################################\n",
    "        raise NotImplementedError\n",
    "\n",
    "    @property\n",
    "    def observation_space(self):\n",
    "        #############################################################\n",
    "        # specify dimensions and properties of the state space here #\n",
    "        #############################################################\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def get_state(self, **kwargs):\n",
    "        ####################################\n",
    "        # specify desired state space here #\n",
    "        ####################################\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def _apply_rl_actions(self, rl_actions):\n",
    "        #####################################\n",
    "        # specify desired action space here #\n",
    "        #####################################\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def compute_reward(self, rl_actions, **kwargs):\n",
    "        ########################################\n",
    "        # specify desired reward function here #\n",
    "        ########################################\n",
    "        raise NotImplementedError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b. Testing the Environment\n",
    "\n",
    "In order to test whether your scenario is working properly, modify and run the below cell. If all is well, it will print a \"Sucess!\" at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.params import EnvParams, SumoParams, InitialConfig\n",
    "from flow.core.vehicles import Vehicles\n",
    "from flow.controllers import RLController\n",
    "\n",
    "env_params = EnvParams()\n",
    "sumo_params = sumo_params(render=False)\n",
    "initial_config = InitialConfig()\n",
    "vehicles = Vehicles()\n",
    "vehicles.add(\"rl\", acceleration_controller=(RLController, {}), 1)\n",
    "\n",
    "#################################\n",
    "# Create the scenario file here #\n",
    "#################################\n",
    "scenario =  ### FILL IN ###\n",
    "\n",
    "env = MyEnv(env_params=env_params, \n",
    "            sumo_params=sumo_params, \n",
    "            scenario=scenario)\n",
    "\n",
    "if np.logical_all(env.get_state() != np.array([0, 0])):\n",
    "    print(\"get_state failed.\")\n",
    "elif env.compute_reward([]) != 0:\n",
    "    print(\"compute_reward failed\")\n",
    "else:\n",
    "    env.step(rl_actions=[0.1])\n",
    "\n",
    "    if env.vehicles.get_speed(env.vehicles.get_speed(env.vehicles.get_rl_ids())) != 0.1:\n",
    "        print()\n",
    "    else:\n",
    "        print(\"Sucess!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### c. Training and Visualizing on RLlib\n",
    "\n",
    "The environment we created in section a) is compatible with OpenAI gym, a popular standardization of MDP tasks in the RL community, and accordingly can be trained a variety of differen off-the-shelf RL libraries. For this, we will utilize the RL library `RLLib`.\n",
    "\n",
    "In order to avoid the installation procedure for RLlib (which are described in the installation instructions), for this workshop we will provide each user an EC2 instance to with RLlib and Flow preinstalled, and a jupyter-notebook titled \"rl_exercise.ipynb\" that contains the code you will need to run to execute your experiment. In order to open this notebook, type in a web browser:\n",
    "\n",
    "        http://[url]:8888/?token=fd0720594f408865c7c0185bc209a531c16480c91d18bad2\n",
    "\n",
    "where an appropriate url will be provided by on the of the tutorial assistants upon request.\n",
    "\n",
    "The tutorial new jupyter notebook will walk you through running a RL experiment in RLlib and then visualizing the performance of the learned control strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}