What is Backpropagation

Backpropagation (or BP) is an abbreviation for Error Backward Propagation. BP algorithm is mainly composed of forward propagation and backpropagation processes:

• Forward propagation: Input information to the input layer of the neural network, it passes through one or more hidden layers and then outputs from the output layer.
• Backpropagation: Compare the output value with the actual value and pass the error from the output layer to the input layer via the hidden layers; in this process, adjust the weights of neurons using gradient descent technique.

The repetitive adjustments of weights is the training process of the neural network.

Construct Neural Network

Neural Network Structure

Neural network is normally constructed with an input layer, one or more hidden layers, and an output layer. We propose the simple neural network below as an example, sigmoid activation function is applied in the output layer:

Activation Function

Activation functions empowers network to conduct non-linear modeling. Without activation functions, the network can only express linear mappings. There are a variety of activation functions, the formula and graph of the sigmoid function used here are as below:

Initial Weights

Initial weights are randomly generated when the algorithm begins, we assume the initial weights are:

Training Samples

Suppose we have 3 groups of samples as below, the superscript represents the order of the sample:

• Inputs: x⁽¹⁾ = (2,3,1)，x⁽²⁾ = (1,0,2)，x⁽³⁾ = (3,1,1)
• Outputs: t⁽¹⁾ = 0.64，t⁽²⁾ = 0.52，t⁽³⁾ = 0.36

The goal of training is to make the output of the model (y) as close as possible to the actual output (t) when the input (x) is given.

Forward Propagation

Input Layer → Hidden Layer

Neurons h₁ and h₂ are calculated by:

Hidden Layer → Output Layer

Output value y is calculated by:

Calculation

Below is the calculation of the 3 samples:

The actual output values are also listed in the table. Notice that the outputs of the 3 samples are greatly different from the expected values.

Loss Function

Loss function is used to calculate the error between the output of the model and the expected output. Loss function is also known as objective function or cost function. A commonly used loss function is Mean-Square Error (MSE):

where m is the number of samples. The error of this forward propagation is:

E = [(0.64-0.907)² + (0.52-0.698)² + (0.36-0.796)²] / (2*3) = 0.234

Loss function measures the accuracy of the model, the smaller the loss function value, the higher the model accuracy, and the purpose of model training is to reduce the loss function value as much as possible. Think of the input and output as constants, and the loss function as a function with weights as variables. A good way to find the weights that minimize the value of loss function is gradient descent.

Backpropagation

Batch gradient descent (BGD) will be adopted to update weights, i.e., all samples will be involved in the calculation. The learning rate is set to η = 0.5.

If readers are not familiar with gradient descent, please read - Gradient Descent。

Output Layer → Hidden Layer

There are two weights w₁ and w₂ between output layer and hidden layer, we will adjust both respectively.

When adjusting w₁, we need to see how much influence w₁ has on the error E, so that to calculate partial derivative for w₁ by using the chain rule：

Then calculate each gradient respectively:

Calculate with values:

∂E/∂y = [(0.907-0.64) + (0.698-0.52) + (0.796-0.36)] / 3 = 0.294
∂y/∂s = [0.907*(1-0.907) + 0.698*(1-0.698) + 0.796*(1-0.796)] / 3 = 0.152
∂s/∂w₁ = (2.4 + 0.75 + 1.35) / 3 = 1.5

The final result is: ∂E/∂w₁ = 0.294*0.152*1.5 = 0.067

As all 3 samples are to participate in the calculation, when calculating ∂y/∂s and ∂s/∂w₁, we would need to obtain the sum and the average of them.

New w₁ is w₁ := w₁ - η ⋅ ∂E/∂w₁ = 0.8 - 0.5*0.067 = 0.766

The method to adjust w₂ is similar to w₁, we give the result directly here, w₂ is adjusted from 0.2 to 0.167.

Hidden Layer → Input Layer

There are 6 weights v₁₁, v₁₂, v₂₁, v₂₂, v₃₁, and v₃₂ between hidden layer and input layer, we will adjust each of them respectively.

When adjusting v₁₁, we need to calculate how much influence v₁₁ has on the error E, so that to calculate the partial derivative for v₁₁:

We already obtained the first two gradients when adjusting w₁ and w₂, only need to calculate the latter two:

Calculate with values:

∂E/∂y = 0.294
∂y/∂s = 0.152
∂s/∂h₁ = 0.8
∂h₁/∂v₁₁ = (2 + 1 + 3) / 3 = 2

The final result is: ∂E/∂v₁₁ = 0.294*0.152*0.8*2 = 0.072

New v₁₁ is v₁₁ := v₁₁ - η ⋅ ∂E/∂v₁₁ = 0.15 - 0.5*0.072 = 0.114

Adjustment for the other 5 weights is similar to v₁₁, the results are:

• v₁₂ is adjusted from 0.2 to 0.191
• v₂₁ is adjusted from 0.6 to 0.576
• v₂₂ is adjusted from 0.3 to 0.294
• v₃₁ is adjusted from 0.3 to 0.282
• v₃₂ is adjusted from 0.5 to 0.496

Model Training

Apply the adjusted weights to model and use the 3 same samples to conduct forward propagation again, this time the error E = 0.192, which is obviously improved if compares with the first forward propagation with error E = 0.192.

BP algorithm repeats the forward and back-propagations to train the model iteratively until the preset training number or time is met, or the error descends to a set threshold.