Google Developers

$\definecolor{input}{RGB}{66, 133, 244} \definecolor{output}{RGB}{219, 68, 55} \definecolor{dinput}{RGB}{244, 180, 0} \definecolor{doutput}{RGB}{15, 157, 88} \definecolor{dweight}{RGB}{102, 0, 255}$

Backpropagation algorithm

The backpropagation algorithm is essential for training large neural networks quickly. This article explains how the algorithm works.

Please scroll down...

Simple neural network

On the right, you see a neural network with one input, one output node and two hidden layers of two nodes each.

Nodes in neighboring layers are connected with weights

w_{i j}

$w_{ij}$ , which are the network parameters.

Activation function

Each node has a total input

x

$\color{input}x$ , an activation function

f (x)

$f(\color{input}x\color{black})$ and an output

y = f (x)

$\color{output}y\color{black}=f(\color{input}x\color{black})$ .

f (x)

$f(\color{input}x\color{black})$ has to be a non-linear function, otherwise the neural network will only be able to learn linear models.

A commonly used activation function is the Sigmoid function:

f (x) = \frac{1}{1 + e^{- x}}

$f(\color{input}x\color{black}) = \frac{1}{1+e^{-\color{input}x}}$ .

Error function

The goal is to learn the weights of the network automatically from data such that the predicted output

y_{o u t p u t}

$\color{output}y_{output}$ is close to the target

y_{t a r g e t}

$\color{output}y_{target}$ for all inputs

x_{i n p u t}

$\color{input}x_{input}$ .

To measure how far we are from the goal, we use an error function

E

$E$ . A commonly used error functon is

E (y_{o u t p u t}, y_{t a r g e t}) = \frac{1}{2} (y_{o u t p u t} - y_{t a r g e t})^{2}

$E(\color{output}y_{output}\color{black},\color{output}y_{target}\color{black}) = \frac{1}{2}(\color{output}y_{output}\color{black} - \color{output}y_{target}\color{black})^2$ .

Forward propagation

We begin by taking an input example

(x_{i n p u t}, y_{t a r g e t})

$(\color{input}x_{input}\color{black},\color{output}y_{target}\color{black})$ and updating the input layer of the network.

For consistency, we consider the input to be like any other node but without an activation function so its output is equal to its input, i.e.

y_{1} = x_{i n p u t}

$\color{output}y_1 \color{black} = \color{input} x_{input}$ .

Forward propagation

Now, we update the first hidden layer. We take the output

y

$\color{output}y$ of the nodes in the previous layer and use the weights to compute the input

x

$\color{input}x$ of the nodes in the next layer.

x_{j} =

$\color{input} x_j \color{black} =$

\sum_{i \in i n (j)} w_{i j} y_{i} + b_{j}

$\sum_{i\in in(j)} w_{ij}\color{output} y_i\color{black} +b_j$

Forward propagation

Then we update the output of the nodes in the first hidden layer. For this we use the activation function,

f (x)

$f(x)$ .

y = f (x)

$\color{output} y \color{black} = f(\color{input} x \color{black})$

Forward propagation

Using these 2 formulas we propagate for the rest of the network and get the final output of the network.

y = f (x)

$\color{output} y \color{black} = f(\color{input} x \color{black})$

x_{j} =

$\color{input} x_j \color{black} =$

\sum_{i \in i n (j)} w_{i j} y_{i} + b_{j}

$\sum_{i\in in(j)} w_{ij}\color{output} y_i \color{black} + b_j$

Error derivative

The backpropagation algorithm decides how much to update each weight of the network after comparing the predicted output with the desired output for a particular example. For this, we need to compute how the error changes with respect to each weight

\frac{d E}{d w_{i j}}

$\color{dweight}\frac{dE}{dw_{ij}}$ .
Once we have the error derivatives, we can update the weights using a simple update rule:

w_{i j} = w_{i j} - α \frac{d E}{d w_{i j}}

$w_{ij} = w_{ij} - \alpha \color{dweight}\frac{dE}{dw_{ij}}$

where

α

$\alpha$ is a positive constant, referred to as the learning rate, which we need to fine-tune empirically.

[Note] The update rule is very simple: if the error goes down when the weight increases (

\frac{d E}{d w_{i j}} < 0

$\color{dweight}\frac{dE}{dw_{ij}}\color{black} < 0$ ), then increase the weight, otherwise if the error goes up when the weight increases (

\frac{d E}{d w_{i j}} > 0

$\color{dweight}\frac{dE}{dw_{ij}} \color{black} > 0$ ), then decrease the weight.

Additional derivatives

To help compute

\frac{d E}{d w_{i j}}

$\color{dweight}\frac{dE}{dw_{ij}}$ , we additionally store for each node two more derivatives: how the error changes with:

the total input of the node $\color{dinput}\frac{dE}{dx}$ and
the output of the node $\color{doutput}\frac{dE}{dy}$ .

Back propagation

Let's begin backpropagating the error derivatives. Since we have the predicted output of this particular input example, we can compute how the error changes with that output. Given our error function

E = \frac{1}{2} (y_{o u t p u t} - y_{t a r g e t})^{2}

$E = \frac{1}{2}(\color{output}y_{output}\color{black} - \color{output}y_{target}\color{black})^2$ we have:

\frac{\partial E}{\partial y_{o u t p u t}} = y_{o u t p u t} - y_{t a r g e t}

$\color{doutput} \frac{\partial E}{\partial y_{output}} \color{black} = \color{output} y_{output} \color{black} - \color{output} y_{target}$

Back propagation

Now that we have

\frac{d E}{d y}

$\color{doutput} \frac{dE}{dy}$ we can get

\frac{d E}{d x}

$\color{dinput}\frac{dE}{dx}$ using the chain rule.

\frac{\partial E}{\partial x} = \frac{d y}{d x} \frac{\partial E}{\partial y} = \frac{d}{d x} f (x) \frac{\partial E}{\partial y}

$\color{dinput} \frac{\partial E}{\partial x} \color{black} = \frac{dy}{dx}\color{doutput}\frac{\partial E}{\partial y} \color{black} = \frac{d}{dx}f(\color{input}x\color{black})\color{doutput}\frac{\partial E}{\partial y}$

where

\frac{d}{d x} f (x) = f (x) (1 - f (x))

$\frac{d}{dx}f(\color{input}x\color{black}) = f(\color{input}x\color{black})(1 - f(\color{input}x\color{black}))$ when

f (x)

$f(\color{input}x\color{black})$ is the Sigmoid activation function.

Back propagation

As soon as we have the error derivative with respect to the total input of a node, we can get the error derivative with respect to the weights coming into that node.

\frac{\partial E}{\partial w_{i j}} = \frac{\partial x_{j}}{\partial w_{i j}} \frac{\partial E}{\partial x_{j}} = y_{i} \frac{\partial E}{\partial x_{j}}

$\color{dweight} \frac{\partial E}{\partial w_{ij}} \color{black} = \frac{\partial x_j}{\partial w_{ij}} \color{dinput}\frac{\partial E}{\partial x_j} \color{black} = \color{output}y_i \color{dinput} \frac{\partial E}{\partial x_j}$

Back propagation

And using the chain rule, we can also get

\frac{d E}{d y}

$\frac{dE}{dy}$ from the previous layer. We have made a full circle.

\frac{\partial E}{\partial y_{i}} = \sum_{j \in o u t (i)} \frac{\partial x_{j}}{\partial y_{i}} \frac{\partial E}{\partial x_{j}} = \sum_{j \in o u t (i)} w_{i j} \frac{\partial E}{\partial x_{j}}

$\color{doutput} \frac{\partial E}{\partial y_i} \color{black} = \sum_{j\in out(i)} \frac{\partial x_j}{\partial y_i} \color{dinput} \frac{\partial E}{\partial x_j} \color{black} = \sum_{j\in out(i)} w_{ij} \color{dinput} \frac{\partial E}{\partial x_j}$

Back propagation

All that is left to do is repeat the previous three formulas until we have computed all the error derivatives.

Backpropagation algorithm

Simple neural network

Activation function

Error function

Forward propagation

Forward propagation

Forward propagation

Forward propagation

Error derivative

Additional derivatives

Back propagation

Back propagation

Back propagation

Back propagation

Back propagation

The end.