6_reinforcement_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eec6e995",
   "metadata": {},
   "source": [
    "# Reinforcement learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48e79e3a",
   "metadata": {},
   "source": [
    "In this notebook, we give some example of reinforcement algorithm for a pendulum model. Beware that the implementation of the algorithms are ment to be simple, hence are not super efficient: don't use them as reference if you want to implement some dedicated RL for another problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7e88244",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy.linalg import norm,inv,pinv,svd,eig\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13485711",
   "metadata": {},
   "source": [
    "## Environments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54cfa283",
   "metadata": {},
   "source": [
    "We are going to work with an inverted pendulum with limited torque, that must swing to collect energy before raising up to the unstable equilibrium state. As the algorithms that we are going to explore are either working on discrate action-state spaces, or continuous ones, several versions of this environment are proposed. In general, they all work the same: get in random initial configuration with reset, display in meshcat with render, and run a simulation step with step(control). Examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4eb8b6c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tp6.env_pendulum import EnvPendulum,EnvPendulumDiscrete,EnvPendulumHybrid,EnvPendulumSinCos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d2375ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "env = EnvPendulum(1,viewer='meshcat')\n",
    "env.name = str(env.__class__)\n",
    "env.u0 = np.zeros(env.nu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1777ae03",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.jupyter_cell()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9aa2bf9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.render()\n",
    "for i in range(10):\n",
    "    env.step(env.u0)\n",
    "    env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3793e351",
   "metadata": {},
   "source": [
    "\n",
    "We define here 4 main environments that you can similarly test:\n",
    "\n",
    "- EnvPendulum:    state NX=2 continuous, control NU=1 continuous, Euler integration step with DT=1e-2 and high friction\n",
    "- EnvPendulumDiscrete:  state NX=441 discrete, control NU=11 discrete, Euler step DT=0.5 low friction\n",
    "- EnvPendulumSinCos: state NX=3 with x=[cos,sin,vel], control NU=1 control, Euler step DT=1e-2, high friction\n",
    "- EnvPendulumHybrid:  state NX=3 continuous with x=[cos,sin,vel], control NU=11 discrete, Euler step DT=0.5 low friction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc372334",
   "metadata": {},
   "source": [
    "## Value iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50d1a7af",
   "metadata": {},
   "source": [
    "For the first algorithm, we implement a Value iteration, which is an algorithm working on discrete states and discrete actions. As it is not very efficient, we must coarsly discretize the pendulum. Here is the implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a1c3bb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load tp6/qtable.py\n",
    "'''\n",
    "Example of Q-table learning with a simple discretized 1-pendulum environment.\n",
    "-- concerge in 1k  episods with pendulum(1)\n",
    "-- Converge in 10k episods with cozmo model\n",
    "'''\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import signal\n",
    "import time\n",
    "import numpy as np\n",
    "\n",
    "### --- Random seed\n",
    "RANDOM_SEED = 1188 #int((time.time()%10)*1000)\n",
    "print(\"Seed = %d\" % RANDOM_SEED)\n",
    "np.random.seed(RANDOM_SEED)\n",
    "\n",
    "### --- Environment\n",
    "from tp6.env_pendulum import EnvPendulumDiscrete; Env = lambda : EnvPendulumDiscrete(1,viewer='meshcat')\n",
    "env = Env()\n",
    "\n",
    "### --- Hyper paramaters\n",
    "NEPISODES               = 400           # Number of training episodes\n",
    "NSTEPS                  = 50            # Max episode length\n",
    "LEARNING_RATE           = 0.85          # \n",
    "DECAY_RATE              = 0.99          # Discount factor \n",
    "\n",
    "Q     = np.zeros([env.nx,env.nu])       # Q-table initialized to 0\n",
    "\n",
    "def policy(s):\n",
    "    return np.argmax(Q[s,:])\n",
    "\n",
    "def rendertrial(s0=None,maxiter=100):\n",
    "    '''Roll-out from random state using greedy policy.'''\n",
    "    s = env.reset(s0)\n",
    "    for i in range(maxiter):\n",
    "        a = np.argmax(Q[s,:])\n",
    "        s,r = env.step(a)\n",
    "        env.render()\n",
    "    \n",
    "signal.signal(signal.SIGTSTP, lambda x,y:rendertrial()) # Roll-out when CTRL-Z is pressed\n",
    "\n",
    "h_rwd = []                              # Learning history (for plot).\n",
    "for episode in range(1,NEPISODES):\n",
    "    x    = env.reset()\n",
    "    rsum = 0.0\n",
    "    for steps in range(NSTEPS):\n",
    "        u         = np.argmax(Q[x,:] + np.random.randn(1,env.nu)/episode) # Greedy action with noise\n",
    "        x2,reward = env.step(u)\n",
    "        \n",
    "        # Compute reference Q-value at state x respecting HJB\n",
    "        Qref = reward + DECAY_RATE*np.max(Q[x2,:])\n",
    "\n",
    "        # Update Q-Table to better fit HJB\n",
    "        Q[x,u] += LEARNING_RATE*(Qref-Q[x,u])\n",
    "        x       = x2\n",
    "        rsum   += reward\n",
    "\n",
    "    h_rwd.append(rsum)\n",
    "    if not episode%20:\n",
    "        print('Episode #%d done with average cost %.2f' % (episode,sum(h_rwd[-20:])/20))\n",
    "\n",
    "print(\"Total rate of success: %.3f\" % (sum(h_rwd)/NEPISODES))\n",
    "rendertrial()\n",
    "plt.plot( np.cumsum(h_rwd)/range(1,NEPISODES) )\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "247852cd",
   "metadata": {},
   "source": [
    "After convergence, you can try the obtained policy using the method <rendertrial>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "daef013f",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.jupyter_cell()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da358fae",
   "metadata": {},
   "outputs": [],
   "source": [
    "rendertrial(maxiter=NSTEPS)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7d380be",
   "metadata": {},
   "source": [
    "Let's display the optimal flow. As states are denoted by their indexes, we need to recover the 2d state from the index, with the following method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5619f5a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def x2d(s):\n",
    "    return env.decode_x(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29399491",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tp6.flow import plotFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b04726e",
   "metadata": {},
   "outputs": [],
   "source": [
    "plotFlow(env,policy,x2d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09b74991",
   "metadata": {},
   "source": [
    "## Value iteration with a neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1be454",
   "metadata": {},
   "source": [
    "Next, we marginally modifies the value iteration to store the Q function not as a table, but as a neural network. The main modification will be that the Belman contraction must now be achieve with a gradient descent ... and that is much less efficient. Let's see, on the same environment first.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c96b5bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load tp6/deeptable.py\n",
    "'''\n",
    "Example of Q-table learning with a simple discretized 1-pendulum environment using a linear Q network.\n",
    "'''\n",
    "\n",
    "import numpy as np\n",
    "import random\n",
    "import tensorflow as tf\n",
    "import tensorflow.compat.v1 as tf1\n",
    "import matplotlib.pyplot as plt\n",
    "from tp6.env_pendulum import EnvPendulumDiscrete; Env = lambda : EnvPendulumDiscrete(1,viewer='meshcat')\n",
    "import signal\n",
    "import time\n",
    "tf1.disable_eager_execution()\n",
    "\n",
    "\n",
    "### --- Random seed\n",
    "RANDOM_SEED = int((time.time()%10)*1000)\n",
    "print(\"Seed = %d\" % RANDOM_SEED)\n",
    "np.random.seed(RANDOM_SEED)\n",
    "\n",
    "### --- Hyper paramaters\n",
    "NEPISODES               = 2000          # Number of training episodes\n",
    "NSTEPS                  = 50            # Max episode length\n",
    "LEARNING_RATE           = 0.1           # Step length in optimizer\n",
    "DECAY_RATE              = 0.99          # Discount factor \n",
    "\n",
    "### --- Environment\n",
    "env = Env()\n",
    "NX  = env.nx\n",
    "NU  = env.nu\n",
    "\n",
    "### --- Q-value networks\n",
    "class QValueNetwork:\n",
    "    def __init__(self):\n",
    "        x               = tf1.placeholder(shape=[1,NX],dtype=tf.float32)\n",
    "        W               = tf1.Variable(tf1.random_uniform([NX,NU],0,0.01,seed=100))\n",
    "        qvalue          = tf1.matmul(x,W)\n",
    "        u               = tf1.argmax(qvalue,1)\n",
    "\n",
    "        qref            = tf1.placeholder(shape=[1,NU],dtype=tf.float32)\n",
    "        loss            = tf1.reduce_sum(tf.square(qref - qvalue))\n",
    "        optim           = tf1.train.GradientDescentOptimizer(LEARNING_RATE).minimize(loss)\n",
    "\n",
    "        self.x          = x             # Network input\n",
    "        self.qvalue     = qvalue        # Q-value as a function of x\n",
    "        self.u          = u             # Policy  as a function of x\n",
    "        self.qref       = qref          # Reference Q-value at next step (to be set to l+Q o f)\n",
    "        self.optim      = optim         # Optimizer      \n",
    "\n",
    "### --- Tensor flow initialization\n",
    "#tf.reset_default_graph()\n",
    "qvalue  = QValueNetwork()\n",
    "sess = tf1.InteractiveSession()\n",
    "tf1.global_variables_initializer().run()\n",
    "\n",
    "def onehot(ix,n=NX):\n",
    "    '''Return a vector which is 0 everywhere except index <i> set to 1.'''\n",
    "    return np.array([[ (i==ix) for i in range(n) ],],np.float)\n",
    "   \n",
    "def disturb(u,i):\n",
    "    u += int(np.random.randn()*10/(i/50+10))\n",
    "    return np.clip(u,0,NU-1)\n",
    "\n",
    "def rendertrial(maxiter=100):\n",
    "    x = env.reset()\n",
    "    for i in range(maxiter):\n",
    "        u = sess.run(qvalue.u,feed_dict={ qvalue.x:onehot(x) })\n",
    "        x,r = env.step(u)\n",
    "        env.render()\n",
    "        if r==1: print('Reward!'); break\n",
    "signal.signal(signal.SIGTSTP, lambda x,y:rendertrial()) # Roll-out when CTRL-Z is pressed\n",
    "\n",
    "### --- History of search\n",
    "h_rwd = []                              # Learning history (for plot).\n",
    "\n",
    "### --- Training\n",
    "for episode in range(1,NEPISODES):\n",
    "    x    = env.reset()\n",
    "    rsum = 0.0\n",
    "\n",
    "    for step in range(NSTEPS-1):\n",
    "        u = sess.run(qvalue.u,feed_dict={ qvalue.x: onehot(x) })[0] # Greedy policy ...\n",
    "        u = disturb(u,episode)                                      # ... with noise\n",
    "        x2,reward = env.step(u)\n",
    "\n",
    "        # Compute reference Q-value at state x respecting HJB\n",
    "        Q2        = sess.run(qvalue.qvalue,feed_dict={ qvalue.x: onehot(x2) })\n",
    "        Qref      = sess.run(qvalue.qvalue,feed_dict={ qvalue.x: onehot(x ) })\n",
    "        Qref[0,u] = reward + DECAY_RATE*np.max(Q2)\n",
    "\n",
    "        # Update Q-table to better fit HJB\n",
    "        sess.run(qvalue.optim,feed_dict={ qvalue.x    : onehot(x),\n",
    "                                          qvalue.qref : Qref       })\n",
    "\n",
    "        rsum += reward\n",
    "        x = x2\n",
    "        if reward == 1: break\n",
    "\n",
    "    h_rwd.append(rsum)\n",
    "    if not episode%20: print('Episode #%d done with %d sucess' % (episode,sum(h_rwd[-20:])))\n",
    "\n",
    "print(\"Total rate of success: %.3f\" % (sum(h_rwd)/NEPISODES))\n",
    "rendertrial()\n",
    "plt.plot( np.cumsum(h_rwd)/range(1,NEPISODES) )\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c29aab8",
   "metadata": {},
   "source": [
    "See? Each step is much more costly (partly due to the poor implementation, but a gradient step is certainly more costly than a table update), and much less informative: the algorithm is slower to converge."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66c49425",
   "metadata": {},
   "source": [
    "## Q-learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd41453",
   "metadata": {},
   "source": [
    "The good point is that the algorithm scales well for more complex problem, and efficient Q-learning can be implemented on very large one (if you have enough CPUs). Let's take a look at the main one, the Deep-Q algorithm. The input of the neural network can be basically anything (discrete, continuous, an image, etc), the only limitation is that the control must remain discrete, and not too large. So let's take an hybrid version of the same pendulum, with continuous state space (cos,sin,velocity) and discrete control."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b78dfc52",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-16 23:50:32.052347: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/openrobots/lib/:/opt/openrobots/lib64/:/usr/lib\n",
      "2022-03-16 23:50:32.052387: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Seed = 3473\n",
      "You can open the visualizer by visiting the following URL:\n",
      "http://127.0.0.1:7002/static/\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-16 23:50:34.047183: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n",
      "2022-03-16 23:50:34.047213: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (koyasan): /proc/driver/nvidia/version does not exist\n",
      "2022-03-16 23:50:34.047468: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep#  1: lasted 59 steps, reward=-230\n",
      "Ep#  2: lasted 59 steps, reward=-217\n",
      "Ep#  3: lasted 59 steps, reward=-222\n",
      "Ep#  4: lasted 59 steps, reward=-184\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-16 23:50:36.691237: W tensorflow/core/data/root_dataset.cc:200] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-03-16 23:50:36.707204: W tensorflow/core/data/root_dataset.cc:200] Optimization loop failed: CANCELLED: Operation was cancelled\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep#  5: lasted 59 steps, reward=-203\n",
      "Ep#  6: lasted 59 steps, reward=-231\n",
      "Ep#  7: lasted 59 steps, reward=-224\n",
      "Ep#  8: lasted 59 steps, reward=-48\n",
      "Ep#  9: lasted 59 steps, reward=-219\n",
      "Ep# 10: lasted 59 steps, reward=-216\n",
      "Ep# 11: lasted 59 steps, reward=-232\n",
      "Ep# 12: lasted 59 steps, reward=-180\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Input \u001b[0;32mIn [1]\u001b[0m, in \u001b[0;36m<cell line: 79>\u001b[0;34m()\u001b[0m\n\u001b[1;32m    110\u001b[0m         qvalue\u001b[38;5;241m.\u001b[39mtrainer\u001b[38;5;241m.\u001b[39mtrain_on_batch([x_batch,u_batch],qref_batch)\n\u001b[1;32m    112\u001b[0m         \u001b[38;5;66;03m# Update target networks by homotopy.\u001b[39;00m\n\u001b[0;32m--> 113\u001b[0m         \u001b[43mqvalueTarget\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtargetAssign\u001b[49m\u001b[43m(\u001b[49m\u001b[43mqvalue\u001b[49m\u001b[43m,\u001b[49m\u001b[43mUPDATE_RATE\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    115\u001b[0m \u001b[38;5;66;03m# \\\\\\END_FOR step in range(NSTEPS)\u001b[39;00m\n\u001b[1;32m    116\u001b[0m \n\u001b[1;32m    117\u001b[0m \u001b[38;5;66;03m# Display and logging (not mandatory).\u001b[39;00m\n\u001b[1;32m    118\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mEp#\u001b[39m\u001b[38;5;132;01m{:3d}\u001b[39;00m\u001b[38;5;124m: lasted \u001b[39m\u001b[38;5;132;01m{:d}\u001b[39;00m\u001b[38;5;124m steps, reward=\u001b[39m\u001b[38;5;132;01m{:3.0f}\u001b[39;00m\u001b[38;5;124m'\u001b[39m \u001b[38;5;241m.\u001b[39mformat(episode, step,rsum))\n",
      "File \u001b[0;32m~/src/tutos/supaero2022/tp6/qnetwork.py:113\u001b[0m, in \u001b[0;36mQNetwork.targetAssign\u001b[0;34m(self, ref, rate)\u001b[0m\n\u001b[1;32m    111\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m(rate\u001b[38;5;241m<\u001b[39m\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m rate\u001b[38;5;241m>\u001b[39m\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m0\u001b[39m)\n\u001b[1;32m    112\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m v,vref \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mzip\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtrainer\u001b[38;5;241m.\u001b[39mtrainable_variables,ref\u001b[38;5;241m.\u001b[39mtrainer\u001b[38;5;241m.\u001b[39mtrainable_variables):\n\u001b[0;32m--> 113\u001b[0m     v\u001b[38;5;241m.\u001b[39massign(\u001b[43m(\u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[38;5;241;43m-\u001b[39;49m\u001b[43mrate\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mv\u001b[49m\u001b[38;5;241m+\u001b[39mrate\u001b[38;5;241m*\u001b[39mvref)\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/tensorflow/python/ops/variables.py:1075\u001b[0m, in \u001b[0;36mVariable._OverloadOperator.<locals>._run_op\u001b[0;34m(a, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1073\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_run_op\u001b[39m(a, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[1;32m   1074\u001b[0m   \u001b[38;5;66;03m# pylint: disable=protected-access\u001b[39;00m\n\u001b[0;32m-> 1075\u001b[0m   \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mtensor_oper\u001b[49m\u001b[43m(\u001b[49m\u001b[43ma\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvalue\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py:140\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    138\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21merror_handler\u001b[39m(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[1;32m    139\u001b[0m   \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 140\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[43mis_traceback_filtering_enabled\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m:\n\u001b[1;32m    141\u001b[0m       \u001b[38;5;28;01mreturn\u001b[39;00m fn(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m    142\u001b[0m   \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mNameError\u001b[39;00m:\n\u001b[1;32m    143\u001b[0m     \u001b[38;5;66;03m# In some very rare cases,\u001b[39;00m\n\u001b[1;32m    144\u001b[0m     \u001b[38;5;66;03m# `is_traceback_filtering_enabled` (from the outer scope) may not be\u001b[39;00m\n\u001b[1;32m    145\u001b[0m     \u001b[38;5;66;03m# accessible from inside this function\u001b[39;00m\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py:47\u001b[0m, in \u001b[0;36mis_traceback_filtering_enabled\u001b[0;34m()\u001b[0m\n\u001b[1;32m     32\u001b[0m \u001b[38;5;129m@tf_export\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mdebugging.is_traceback_filtering_enabled\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m     33\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mis_traceback_filtering_enabled\u001b[39m():\n\u001b[1;32m     34\u001b[0m   \u001b[38;5;124;03m\"\"\"Check whether traceback filtering is currently enabled.\u001b[39;00m\n\u001b[1;32m     35\u001b[0m \n\u001b[1;32m     36\u001b[0m \u001b[38;5;124;03m  See also `tf.debugging.enable_traceback_filtering()` and\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     45\u001b[0m \u001b[38;5;124;03m    was called).\u001b[39;00m\n\u001b[1;32m     46\u001b[0m \u001b[38;5;124;03m  \"\"\"\u001b[39;00m\n\u001b[0;32m---> 47\u001b[0m   value \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mgetattr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m_ENABLE_TRACEBACK_FILTERING\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mvalue\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[1;32m     48\u001b[0m   \u001b[38;5;28;01mreturn\u001b[39;00m value\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# %load tp6/qlearn.py\n",
    "'''\n",
    "Train a Q-value following a classical Q-learning algorithm (enforcing the\n",
    "satisfaction of HJB method), using a noisy greedy exploration strategy.\n",
    "\n",
    "The result of a training for a continuous pendulum (after 200 iterations) \n",
    "are stored in qvalue.h5.\n",
    "\n",
    "Reference:\n",
    "Mnih, Volodymyr, et al. \"Human-level control through deep reinforcement learning.\" \n",
    "Nature 518.7540 (2015): 529.\n",
    "'''\n",
    "\n",
    "from tp6.env_pendulum import EnvPendulumHybrid; Env = lambda : EnvPendulumHybrid(1,viewer='meshcat')\n",
    "from tp6.qnetwork import QNetwork\n",
    "from collections import deque\n",
    "import time\n",
    "import signal\n",
    "import matplotlib.pyplot as plt\n",
    "import random\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "### --- Random seed\n",
    "RANDOM_SEED = int((time.time()%10)*1000)\n",
    "print(\"Seed = %d\" %  RANDOM_SEED)\n",
    "np .random.seed     (RANDOM_SEED)\n",
    "random.seed         (RANDOM_SEED)\n",
    "\n",
    "### --- Environment\n",
    "env                 = Env()\n",
    "\n",
    "### --- Hyper paramaters\n",
    "NEPISODES               = 1000          # Max training steps\n",
    "NSTEPS                  = 60            # Max episode length\n",
    "QVALUE_LEARNING_RATE    = 0.001         # Base learning rate for the Q-value Network\n",
    "DECAY_RATE              = 0.99          # Discount factor \n",
    "UPDATE_RATE             = 0.01          # Homotopy rate to update the networks\n",
    "REPLAY_SIZE             = 10000         # Size of replay buffer\n",
    "BATCH_SIZE              = 64            # Number of points to be fed in stochastic gradient\n",
    "NH1 = NH2               = 32            # Hidden layer size\n",
    "\n",
    "### --- Replay memory\n",
    "class ReplayItem:\n",
    "    def __init__(self,x,u,r,d,x2):\n",
    "        self.x          = x\n",
    "        self.u          = u\n",
    "        self.reward     = r\n",
    "        self.done       = d\n",
    "        self.x2         = x2\n",
    "replayDeque = deque()\n",
    "\n",
    "### --- Tensor flow initialization\n",
    "qvalue          = QNetwork(nx=env.nx,nu=env.nu,learning_rate=QVALUE_LEARNING_RATE)\n",
    "qvalueTarget    = QNetwork(name='target',nx=env.nx,nu=env.nu)\n",
    "# Uncomment to load networks\n",
    "#qvalue.load()\n",
    "#qvalueTarget.load()\n",
    "\n",
    "def rendertrial(maxiter=NSTEPS,verbose=True):\n",
    "    x = env.reset()\n",
    "    traj = [x.copy()]\n",
    "    rsum = 0.\n",
    "    for i in range(maxiter):\n",
    "        u = qvalue.policy(x)[0]\n",
    "        x, reward = env.step(u)\n",
    "        env.render()\n",
    "        time.sleep(1e-2)\n",
    "        rsum += reward\n",
    "        traj.append(x.copy())\n",
    "    if verbose: print('Lasted ',i,' timestep -- total reward:',rsum)\n",
    "    return np.array(traj)\n",
    "signal.signal(signal.SIGTSTP, lambda x,y:rendertrial()) # Roll-out when CTRL-Z is pressed\n",
    "\n",
    "### History of search\n",
    "h_rwd = []\n",
    "\n",
    "### --- Training\n",
    "for episode in range(1,NEPISODES):\n",
    "    x    = env.reset()\n",
    "    rsum = 0.0\n",
    "\n",
    "    for step in range(NSTEPS):\n",
    "        u       = qvalue.policy(x,                                     # Greedy policy ...\n",
    "                                noise=1. / (1. + episode + step))      # ... with noise\n",
    "        x2,r    = env.step(u)\n",
    "        done    = False # Some environment may return information when task completed\n",
    "\n",
    "        replayDeque.append(ReplayItem(x,u,r,done,x2))                # Feed replay memory ...\n",
    "        if len(replayDeque)>REPLAY_SIZE: replayDeque.popleft()       # ... with FIFO forgetting.\n",
    "\n",
    "        rsum   += r\n",
    "        x       = x2\n",
    "        if done: break\n",
    "        \n",
    "        # Start optimizing networks when memory size > batch size.\n",
    "        if len(replayDeque) > BATCH_SIZE:     \n",
    "            batch = random.sample(replayDeque,BATCH_SIZE)            # Random batch from replay memory.\n",
    "            x_batch    = np.vstack([ b.x      for b in batch ])\n",
    "            u_batch    = np.vstack([ b.u      for b in batch ])\n",
    "            r_batch    = np.array([ [b.reward] for b in batch ])\n",
    "            d_batch    = np.array([ [b.done]   for b in batch ])\n",
    "            x2_batch   = np.vstack([ b.x2     for b in batch ])\n",
    "            \n",
    "            # Compute Q(x,u) from target network\n",
    "            v_batch    = qvalueTarget.value(x2_batch)\n",
    "            qref_batch = r_batch + (d_batch==False)*(DECAY_RATE*v_batch)\n",
    "\n",
    "            # Update qvalue to solve HJB constraint: q = r + q'\n",
    "            qvalue.trainer.train_on_batch([x_batch,u_batch],qref_batch)\n",
    "            \n",
    "            # Update target networks by homotopy.\n",
    "            qvalueTarget.targetAssign(qvalue,UPDATE_RATE)\n",
    "      \n",
    "    # \\\\\\END_FOR step in range(NSTEPS)\n",
    "\n",
    "    # Display and logging (not mandatory).\n",
    "    print('Ep#{:3d}: lasted {:d} steps, reward={:3.0f}' .format(episode, step,rsum))\n",
    "    h_rwd.append(rsum)\n",
    "    if not (episode+1) % 200:     rendertrial(30)\n",
    "\n",
    "# \\\\\\END_FOR episode in range(NEPISODES)\n",
    "\n",
    "print(\"Average reward during trials: %.3f\" % (sum(h_rwd)/NEPISODES))\n",
    "rendertrial()\n",
    "plt.plot( np.cumsum(h_rwd)/range(1,NEPISODES) )\n",
    "plt.show()\n",
    "\n",
    "# Uncomment to save networks\n",
    "#qvalue.save()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a1ddcb4",
   "metadata": {},
   "source": [
    "## Actor critic\n",
    "When the control space is discrete, the optimal policy is directly obtained by maximizing the Q value by an exhaustive search. This does not work for continuous space. In that case, a policy network must be trained in parallel, to greedily optimize the Q function. A famous algorithm for that, which is a near direct extension of Deep-Q, is the Deep Deterministic Policy Gradient (DDPG). \n",
    "\n",
    "Like Deep-Q, it optimizes the Q function to contract the Belman residual. To efficiently do that, it also uses minibatches to avoid the local collapses due to sample dependancy. In addition, it uses a smoothing of the gradient direction, using a so-called \"target network\", that can be understood as an ad-hoc trust region to avoid violent gradient steps. Finally, it greedily optimizes a policy network: one of the main trick of the paper is to compute the gradient direction of the policy network, which uses the jacobian of the value network. But in the following implementation, this is automatically computed by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4af7bf53",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load tp6/ddpg.py\n",
    "'''\n",
    "Deep actor-critic network, \n",
    "From \"Continuous control with deep reinforcement learning\", by Lillicrap et al, arXiv:1509.02971\n",
    "'''\n",
    "\n",
    "from env_pendulum import EnvPendulumSinCos; Env = lambda : EnvPendulumSinCos(1,viewer='meshcat')\n",
    "import gym\n",
    "import tensorflow as tf\n",
    "import tensorflow.keras as tfk\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import random\n",
    "from collections import deque\n",
    "import signal\n",
    "\n",
    "#######################################################################################################33\n",
    "#######################################################################################################33\n",
    "#######################################################################################################33\n",
    "### --- Random seed\n",
    "RANDOM_SEED = 0 # int((time.time()%10)*1000)\n",
    "print(\"Seed = %d\" %  RANDOM_SEED)\n",
    "np .random.seed     (RANDOM_SEED)\n",
    "random.seed         (RANDOM_SEED)\n",
    "tf.random.set_seed  (RANDOM_SEED)\n",
    "\n",
    "### --- Hyper paramaters\n",
    "NEPISODES               = 1000           # Max training steps\n",
    "NSTEPS                  = 200           # Max episode length\n",
    "QVALUE_LEARNING_RATE    = 0.001         # Base learning rate for the Q-value Network\n",
    "POLICY_LEARNING_RATE    = 0.0001        # Base learning rate for the policy network\n",
    "DECAY_RATE              = 0.99          # Discount factor \n",
    "UPDATE_RATE             = 0.01          # Homotopy rate to update the networks\n",
    "REPLAY_SIZE             = 10000         # Size of replay buffer\n",
    "BATCH_SIZE              = 64            # Number of points to be fed in stochastic gradient\n",
    "NH1 = NH2               = 250           # Hidden layer size\n",
    "EXPLORATION_NOISE       = 0.2\n",
    "\n",
    "### --- Environment\n",
    "# problem = \"Pendulum-v1\"\n",
    "# env = gym.make(problem)\n",
    "# NX = env.observation_space.shape[0]\n",
    "# NU = env.action_space.shape[0]\n",
    "# UMAX = env.action_space.high[0]\n",
    "# env.reset(seed=RANDOM_SEED)\n",
    "# assert( env.action_space.low[0]==-UMAX)\n",
    "\n",
    "env                 = Env()             # Continuous pendulum\n",
    "NX                  = env.nx            # ... training converges with q,qdot with 2x more neurones.\n",
    "NU                  = env.nu            # Control is dim-1: joint torque\n",
    "UMAX                = env.umax[0]       # Torque range\n",
    "\n",
    "\n",
    "#######################################################################################################33\n",
    "### NETWORKS ##########################################################################################33\n",
    "#######################################################################################################33\n",
    "\n",
    "class QValueNetwork:\n",
    "    '''\n",
    "    Neural representaion of the Quality function:\n",
    "    Q:  x,y -> Q(x,u) \\in R\n",
    "    '''\n",
    "    def __init__(self,nx,nu,nhiden1=32,nhiden2=256,learning_rate=None):\n",
    "\n",
    "        state_input = tfk.layers.Input(shape=(nx))\n",
    "        state_out = tfk.layers.Dense(nhiden1, activation=\"relu\")(state_input)\n",
    "        state_out = tfk.layers.Dense(nhiden1, activation=\"relu\")(state_out)\n",
    "\n",
    "        action_input = tfk.layers.Input(shape=(nu))\n",
    "        action_out = tfk.layers.Dense(nhiden1, activation=\"relu\")(action_input)\n",
    "\n",
    "        concat = tfk.layers.Concatenate()([state_out, action_out])\n",
    "\n",
    "        out = tfk.layers.Dense(nhiden2, activation=\"relu\")(concat)\n",
    "        out = tfk.layers.Dense(nhiden2, activation=\"relu\")(out)\n",
    "        value_output = tfk.layers.Dense(1)(out)\n",
    "\n",
    "        self.model = tfk.Model([state_input, action_input], value_output)\n",
    "\n",
    "    @tf.function\n",
    "    def targetAssign(self,target,tau=UPDATE_RATE):\n",
    "        for (tar,cur) in zip(target.model.variables,self.model.variables):\n",
    "            tar.assign(cur * tau + tar * (1 - tau))\n",
    " \n",
    "\n",
    "class PolicyNetwork:\n",
    "    '''\n",
    "    Neural representation of the policy function:\n",
    "    Pi: x -> u=Pi(x) \\in R^nu\n",
    "    '''\n",
    "    def __init__(self,nx,nu,umax,nhiden=32,learning_rate=None):\n",
    "        random_init = tf.random_uniform_initializer(minval=-0.005, maxval=0.005)\n",
    "        \n",
    "        state_input = tfk.layers.Input(shape=(nx,))\n",
    "        out = tfk.layers.Dense(nhiden, activation=\"relu\")(state_input)\n",
    "        out = tfk.layers.Dense(nhiden, activation=\"relu\")(out)\n",
    "        policy_output = tfk.layers.Dense(1, activation=\"tanh\",\n",
    "                                         kernel_initializer=random_init)(out)*umax\n",
    "        self.model = tfk.Model(state_input, policy_output)\n",
    "\n",
    "    @tf.function\n",
    "    def targetAssign(self,target,tau=UPDATE_RATE):\n",
    "        for (tar,cur) in zip(target.model.variables,self.model.variables):\n",
    "            tar.assign(cur * tau + tar * (1 - tau))\n",
    "\n",
    "    def numpyPolicy(self,x,noise=None):\n",
    "        '''Eval the policy with numpy input-output (nx,)->(nu,).'''\n",
    "        x_tf = tf.expand_dims(tf.convert_to_tensor(x), 0)\n",
    "        u = np.squeeze(self.model(x_tf).numpy(),0)\n",
    "        if noise is not None:\n",
    "            u = np.clip( u+noise, -UMAX,UMAX)\n",
    "        return u\n",
    "\n",
    "    def __call__(self, x,**kwargs):\n",
    "        return self.numpyPolicy(x,**kwargs)\n",
    "\n",
    "            \n",
    "        \n",
    "#######################################################################################################33\n",
    "\n",
    "class OUNoise:\n",
    "    '''\n",
    "    Ornstein–Uhlenbeck processes are markov random walks with the nice property to eventually\n",
    "    converge to its mean.\n",
    "    We use it for adding some random search at the begining of the exploration.\n",
    "    '''\n",
    "    def __init__(self, mean, std_deviation, theta=0.15, dt=1e-2, y_initial=None,dtype=np.float32):\n",
    "        self.theta = theta\n",
    "        self.mean = mean.astype(dtype)\n",
    "        self.std_dev = std_deviation.astype(dtype)\n",
    "        self.dt = dt\n",
    "        self.dtype=dtype\n",
    "        self.reset(y_initial)\n",
    "\n",
    "    def __call__(self):\n",
    "        # Formula taken from https://www.wikipedia.org/wiki/Ornstein-Uhlenbeck_process.\n",
    "        noise = np.random.normal(size=self.mean.shape).astype(self.dtype)\n",
    "        self.y += \\\n",
    "            self.theta * (self.mean - self.y) * self.dt \\\n",
    "            + self.std_dev * np.sqrt(self.dt) * noise\n",
    "        return self.y.copy()\n",
    "\n",
    "    def reset(self,y_initial = None):\n",
    "        self.y = y_initial.astype(self.dtype) if y_initial is not None else np.zeros_like(self.mean)\n",
    "\n",
    "### --- Replay memory\n",
    "class ReplayItem:\n",
    "    '''\n",
    "    Storage for the minibatch\n",
    "    '''\n",
    "    def __init__(self,x,u,r,d,x2):\n",
    "        self.x          = x\n",
    "        self.u          = u\n",
    "        self.reward     = r\n",
    "        self.done       = d\n",
    "        self.x2         = x2\n",
    "\n",
    "\n",
    "#######################################################################################################33\n",
    "quality = QValueNetwork(NX,NU,NH1,NH2)\n",
    "qualityTarget = QValueNetwork(NX,NU,NH1,NH2)\n",
    "quality.targetAssign(qualityTarget,1)\n",
    "\n",
    "policy = PolicyNetwork(NX,NU,umax=UMAX,nhiden=NH2)\n",
    "policyTarget = PolicyNetwork(NX,NU,umax=UMAX,nhiden=NH2)\n",
    "policy.targetAssign(policyTarget,1)\n",
    "\n",
    "replayDeque = deque()\n",
    "\n",
    "ou_noise = OUNoise(mean=np.zeros(1), std_deviation=float(EXPLORATION_NOISE) * np.ones(1))\n",
    "ou_noise.reset( np.array([ UMAX/2 ]) )\n",
    "\n",
    "#######################################################################################################33\n",
    "### MAIN ACTOR-CRITIC BLOCK\n",
    "#######################################################################################################33\n",
    "\n",
    "critic_optimizer = tfk.optimizers.Adam(QVALUE_LEARNING_RATE)\n",
    "actor_optimizer = tfk.optimizers.Adam(POLICY_LEARNING_RATE)\n",
    "\n",
    "@tf.function\n",
    "def learn(state_batch, action_batch, reward_batch, next_state_batch):\n",
    "    '''\n",
    "    <learn> is isolated in a tf.function to make it more efficient.\n",
    "    @tf.function forces tensorflow to optimize the inner computation graph defined in this function.\n",
    "    '''\n",
    "\n",
    "    # Automatic differentiation of the critic loss, using tf.GradientTape\n",
    "    # The critic loss is the classical Q-learning loss:\n",
    "    #         loss = || Q(x,u) -  (reward + Q(xnext,Pi(xnexT)) ) ||**2\n",
    "    with tf.GradientTape() as tape:\n",
    "        target_actions = policyTarget.model(next_state_batch, training=True)\n",
    "        y = reward_batch + DECAY_RATE * qualityTarget.model(\n",
    "            [next_state_batch, target_actions], training=True\n",
    "        )\n",
    "        critic_value = quality.model([state_batch, action_batch], training=True)\n",
    "        critic_loss = tf.math.reduce_mean(tf.math.square(y - critic_value))\n",
    "        \n",
    "    critic_grad = tape.gradient(critic_loss, quality.model.trainable_variables)\n",
    "    critic_optimizer.apply_gradients(\n",
    "        zip(critic_grad, quality.model.trainable_variables)\n",
    "    )\n",
    "\n",
    "    # Automatic differentiation of the actor loss, using tf.GradientTape\n",
    "    # The actor loss implements a greedy optimization on the quality function\n",
    "    #           loss(u) = Q(x,u)\n",
    "    with tf.GradientTape() as tape:\n",
    "        actions = policy.model(state_batch, training=True)\n",
    "        critic_value = quality.model([state_batch, actions], training=True)\n",
    "        actor_loss = -tf.math.reduce_mean(critic_value)\n",
    "\n",
    "    actor_grad = tape.gradient(actor_loss, policy.model.trainable_variables)\n",
    "    actor_optimizer.apply_gradients(\n",
    "        zip(actor_grad, policy.model.trainable_variables)\n",
    "    )\n",
    "  \n",
    "\n",
    "#######################################################################################################33\n",
    "#######################################################################################################33\n",
    "#######################################################################################################33\n",
    "\n",
    "def rendertrial(maxiter=NSTEPS,verbose=True):\n",
    "    '''\n",
    "    Display a roll-out from random start and optimal feedback.\n",
    "    Press ^Z to get a roll-out at training time.\n",
    "    '''\n",
    "    x = env.reset()\n",
    "    rsum = 0.\n",
    "    for i in range(maxiter):\n",
    "        u = policy(x)\n",
    "        x, reward = env.step(u)[:2]\n",
    "        env.render()\n",
    "        rsum += reward\n",
    "    if verbose: print('Lasted ',i,' timestep -- total reward:',rsum)\n",
    "signal.signal(signal.SIGTSTP, lambda x,y:rendertrial()) # Roll-out when CTRL-Z is pressed\n",
    "env.full.sleepAtDisplay=5e-3\n",
    "\n",
    "# Logs\n",
    "h_rewards = []\n",
    "h_steps   = []\n",
    "\n",
    "# Takes about 4 min to train\n",
    "for episode in range(NEPISODES):\n",
    "\n",
    "    prev_state = env.reset()\n",
    "\n",
    "    for step in range(NSTEPS):\n",
    "    # Uncomment this to see the Actor in action\n",
    "        # But not in a python notebook.\n",
    "        #env.render()\n",
    "\n",
    "        action = policy(prev_state, noise=ou_noise())\n",
    "        state, reward = env.step(action)[:2]\n",
    "        done=False\n",
    "        \n",
    "        replayDeque.append(ReplayItem(prev_state, action, reward, done, state))\n",
    "        \n",
    "        prev_state = state\n",
    "\n",
    "        if len(replayDeque) <= BATCH_SIZE:  continue\n",
    "\n",
    "\n",
    "        ####################################################################\n",
    "        # Sample a minibatch\n",
    "        \n",
    "        batch = random.sample(replayDeque,BATCH_SIZE)            # Random batch from replay memory.\n",
    "        state_batch    = tf.convert_to_tensor([ b.x      for b in batch ])\n",
    "        action_batch    = tf.convert_to_tensor([ b.u      for b in batch ])\n",
    "        reward_batch    = tf.convert_to_tensor([ [ b.reward ] for b in batch ],dtype=np.float32)\n",
    "        done_batch    = tf.convert_to_tensor([ b.done   for b in batch ])\n",
    "        next_state_batch   = tf.convert_to_tensor([ b.x2     for b in batch ])\n",
    "\n",
    "        ####################################################################\n",
    "        # One gradient step for the minibatch\n",
    "\n",
    "        # Critic and actor gradients\n",
    "        learn(state_batch, action_batch, reward_batch, next_state_batch)\n",
    "        # Step smoothing using target networks\n",
    "        policy.targetAssign(policyTarget)\n",
    "        quality.targetAssign(qualityTarget)\n",
    "\n",
    "        if done: break   # stop at episode end.\n",
    "\n",
    "    # Some prints and logs\n",
    "    episodic_reward = sum([ replayDeque[-i-1].reward for i in range(step+1) ])\n",
    "    h_rewards.append( episodic_reward )\n",
    "    h_steps.append(step+1)\n",
    "    \n",
    "    print(f'Ep#{episode:3d}: lasted {step+1:d} steps, reward={episodic_reward:3.1f} ')\n",
    "\n",
    "    \n",
    "    # avg_reward = np.mean(h_rewards[-40:])\n",
    "    # if episode==5 and RANDOM_SEED==0:\n",
    "    #     assert(  abs(avg_reward + 1423.0528188196286) < 1e-3 )\n",
    "    # if episode==0 and RANDOM_SEED==0:\n",
    "    #     assert(  abs(avg_reward + 1712.386325099637) < 1e-3 )\n",
    "        \n",
    "# Plotting graph\n",
    "# Episodes versus Avg. Rewards\n",
    "plt.plot(h_rewards)\n",
    "plt.xlabel(\"Episode\")\n",
    "plt.ylabel(\"Epsiodic Reward\")\n",
    "plt.show()\n",
    "\n",
    "#######################################################################################################33\n",
    "#######################################################################################################33\n",
    "#######################################################################################################33\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58667872",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}