{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backprop Examples\n",
    "\n",
    "We will look at three different ways of computing gradients, all of which rely on the same basic principle (backprop). \n",
    "\n",
    "1. The first method is to code a function and its derivatives manually using the rules of backprop.\n",
    "2. The second method is to use automatic differentiation using a tool called [autograd](https://github.com/HIPS/autograd).\n",
    "3. The third method is to use [Tensorflow](https://www.tensorflow.org/), a powerful machine learning toolkit from Google, which also includes methods to automatically differentiate an expression defined through Tensforflow primitives. \n",
    "\n",
    "### More about ``autograd``\n",
    "\n",
    "[autograd](https://github.com/HIPS/autograd) is a Python package for **algorithmic differentiation**. It allows you to automatically compute the derivative of functions written in (nearly) native code. This makes it really easy to compute derivatives. Under the hood, it is also using reverse mode autodiff (backprop).\n",
    "\n",
    "### Mote about Tensorflow\n",
    "\n",
    "[Tensorflow](https://www.tensorflow.org/) is a powerful machine learning package from Google.\n",
    "\n",
    "You can define functions as \"computation graphs\" using tensor flow operations, and it can automatically perform backprop (aka reverse mode automatic differentiation) to compute the gradient for you.\n",
    "\n",
    "**Note:** You will need to install TensorFlow if you want to run these examples. It is not hard to install, but takes a bit more work to be able to call from a Jupyter notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backprop Example 1\n",
    "\n",
    "We will see three different ways to compute the gradient of\n",
    "\n",
    "$$f(x,y,z) = (2x + y)z$$\n",
    "\n",
    "### Manual backprop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.0\n",
      "[6. 3. 4.]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Backprop example\n",
    "\n",
    "# Compute f(x,y,z) = (2*x+y)*z\n",
    "x = 1.\n",
    "y = 2.\n",
    "z = 3.\n",
    "\n",
    "# Forward pass\n",
    "q = 2.*x + y   # Node 1\n",
    "f = q*z        # Node 2\n",
    "\n",
    "# Backward pass\n",
    "f_bar = 1\n",
    "q_bar = z * f_bar  # Node 2 input\n",
    "z_bar = q * f_bar  # Node 2 input\n",
    "x_bar = 2 * q_bar  # Node 1 input\n",
    "y_bar = 1 * q_bar  # Node 1 input\n",
    "\n",
    "grad = np.array([x_bar, y_bar, z_bar])\n",
    "\n",
    "print(f)\n",
    "print(grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Autograd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.0\n",
      "[array(6.), array(3.), array(4.)]\n"
     ]
    }
   ],
   "source": [
    "import autograd.numpy as np  # Thinly wrapped version of numpy\n",
    "from autograd import grad\n",
    "\n",
    "def f(args):\n",
    "    x,y,z = args\n",
    "    return (2*x + y)*z\n",
    "\n",
    "f_grad = grad(f)  # magic: returns a function that computes the gradient of f\n",
    "\n",
    "x = 1.\n",
    "y = 2.\n",
    "z = 3.\n",
    "\n",
    "print(f([x, y, z]))\n",
    "print(f_grad([x, y, z]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backprop Example 2\n",
    "\n",
    "Here is a slighly more complex example:\n",
    "\n",
    "$$f(x) = 10*\\exp(\\sin(x)) + \\cos^2(x)$$\n",
    "\n",
    "### Manual backprop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23.1780070835713 11.92692295225547\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Backprop example\n",
    "# f(x) = 10*np.exp(np.sin(x)) + np.cos(x)**2\n",
    "\n",
    "# Forward pass\n",
    "x = 1000\n",
    "a = np.sin(x)   # Node 1\n",
    "b = np.cos(x)   # Node 1\n",
    "c = b**2        # Node 3\n",
    "d = np.exp(a)   # Node 4\n",
    "f = 10*d + c    # Node 5 (final output)\n",
    "\n",
    "# Backward pass\n",
    "f_bar = 1\n",
    "d_bar = 10 * f_bar            # Node 5 input\n",
    "c_bar = 1  * f_bar            # Node 5 input\n",
    "a_bar = np.exp(a) * d_bar     # Node 4 input\n",
    "b_bar = 2*b * c_bar           # Node 3 input\n",
    "x_bar =  np.cos(x) * a_bar - np.sin(x) * b_bar  # Node 2 and 1 input\n",
    "\n",
    "print (f, x_bar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Autograd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "670.4691647536183\n",
      "[array(-278.18475186), array(670.29598656)]\n"
     ]
    }
   ],
   "source": [
    "import autograd.numpy as np  # Thinly wrapped version of numpy\n",
    "from autograd import grad\n",
    "\n",
    "def f(args):\n",
    "    x,y = args\n",
    "    return y**3*10*np.exp(np.sin(x)) + np.cos(x)**2\n",
    "\n",
    "f_grad = grad(f)\n",
    "\n",
    "x = 2.\n",
    "y = 3.\n",
    "\n",
    "print(f([x, y]))\n",
    "print(f_grad([x, y]))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}