Change Password

Please enter the password
Please enter the password Length between [8, 64] ASCII characters Not identical to your email address At least 3 character types from uppercase, lowercase, numbers, and single-byte character symbols
Please enter the password
Submit

Change Nickname

Current Nickname:
Submit
Search
v4.0
    v4.0

    Backpropagation

    What is Backpropagation

    Backpropagation (or BP) is an abbreviation for Error Backward Propagation. BP algorithm is mainly composed of forward propagation and backpropagation processes:

    • Forward propagation: Input information to the input layer of the neural network, it passes through one or more hidden layers and then outputs from the output layer.
    • Backpropagation: Compare the output value with the actual value and pass the error from the output layer to the input layer via the hidden layers; in this process, adjust the weights of neurons using gradient descent technique.

    The repetitive adjustments of weights is the training process of the neural network.

    Construct Neural Network

    Neural Network Structure

    Neural network is normally constructed with an input layer, one or more hidden layers, and an output layer. We propose the simple neural network below as an example, sigmoid activation function is applied in the output layer:

    Activation Function

    Activation functions empowers network to conduct non-linear modeling. Without activation functions, the network can only express linear mappings. There are a variety of activation functions, the formula and graph of the sigmoid function used here are as below:

    Initial Weights

    Initial weights are randomly generated when the algorithm begins, we assume the initial weights are:

    Training Samples

    Suppose we have 3 groups of samples as below, the superscript represents the order of the sample:

    • Inputs: x(1) = (2,3,1),x(2) = (1,0,2),x(3) = (3,1,1)
    • Outputs: t(1) = 0.64,t(2) = 0.52,t(3) = 0.36

    The goal of training is to make the output of the model (y) as close as possible to the actual output (t) when the input (x) is given.

    Forward Propagation

    Input Layer → Hidden Layer

    Neurons h1 and h2 are calculated by:

    Hidden Layer → Output Layer

    Output value y is calculated by:

    Calculation

    Below is the calculation of the 3 samples:

    x
    h1 h2 s y t
    x(1) = (2,3,1) 2.4 1.8 2.28 0.907 0.64
    x(2) = (1,0,2) 0.75 1.2 0.84 0.698 0.52
    x(3) = (3,1,1) 1.35 1.4 1.36 0.796 0.36

    The actual output values are also listed in the table. Notice that the outputs of the 3 samples are greatly different from the expected values.

    Loss Function

    Loss function is used to calculate the error between the output of the model and the expected output. Loss function is also known as objective function or cost function. A commonly used loss function is Mean-Square Error (MSE):

    where m is the number of samples. The error of this forward propagation is:

    E = [(0.64-0.907)2 + (0.52-0.698)2 + (0.36-0.796)2] / (2*3) = 0.234

    Loss function measures the accuracy of the model, the smaller the loss function value, the higher the model accuracy, and the purpose of model training is to reduce the loss function value as much as possible. Think of the input and output as constants, and the loss function as a function with weights as variables. A good way to find the weights that minimize the value of loss function is gradient descent.

    Backpropagation

    Batch gradient descent (BGD) will be adopted to update weights, i.e., all samples will be involved in the calculation. The learning rate is set to η = 0.5.

    If readers are not familiar with gradient descent, please read - Gradient Descent

    Output Layer → Hidden Layer

    There are two weights w1 and w2 between output layer and hidden layer, we will adjust both respectively.

    When adjusting w1, we need to see how much influence w1 has on the error E, so that to calculate partial derivative for w1 by using the chain rule:

    Then calculate each gradient respectively:

    Calculate with values:

    ∂E/∂y = [(0.907-0.64) + (0.698-0.52) + (0.796-0.36)] / 3 = 0.294
    ∂y/∂s = [0.907*(1-0.907) + 0.698*(1-0.698) + 0.796*(1-0.796)] / 3 = 0.152
    ∂s/∂w1 = (2.4 + 0.75 + 1.35) / 3 = 1.5

    The final result is: ∂E/∂w1 = 0.294*0.152*1.5 = 0.067

    As all 3 samples are to participate in the calculation, when calculating ∂y/∂s and ∂s/∂w1, we would need to obtain the sum and the average of them.

    New w1 is w1 := w1 - η ⋅ ∂E/∂w1 = 0.8 - 0.5*0.067 = 0.766

    The method to adjust w2 is similar to w1, we give the result directly here, w2 is adjusted from 0.2 to 0.167.

    Hidden Layer → Input Layer

    There are 6 weights v11, v12, v21, v22, v31, and v32 between hidden layer and input layer, we will adjust each of them respectively.

    When adjusting v11, we need to calculate how much influence v11 has on the error E, so that to calculate the partial derivative for v11:

    We already obtained the first two gradients when adjusting w1 and w2, only need to calculate the latter two:

    Calculate with values:

    ∂E/∂y = 0.294
    ∂y/∂s = 0.152
    ∂s/∂h1 = 0.8
    ∂h1/∂v11 = (2 + 1 + 3) / 3 = 2

    The final result is: ∂E/∂v11 = 0.294*0.152*0.8*2 = 0.072

    New v11 is v11 := v11 - η ⋅ ∂E/∂v11 = 0.15 - 0.5*0.072 = 0.114

    Adjustment for the other 5 weights is similar to v11, the results are:

    • v12 is adjusted from 0.2 to 0.191
    • v21 is adjusted from 0.6 to 0.576
    • v22 is adjusted from 0.3 to 0.294
    • v31 is adjusted from 0.3 to 0.282
    • v32 is adjusted from 0.5 to 0.496

    Model Training

    Apply the adjusted weights to model and use the 3 same samples to conduct forward propagation again, this time the error E = 0.192, which is obviously improved if compares with the first forward propagation with error E = 0.192.

    BP algorithm repeats the forward and back-propagations to train the model iteratively until the preset training number or time is met, or the error descends to a set threshold.

    Please complete the following information to download this book
    *
    公司名称不能为空
    *
    公司邮箱必须填写
    *
    你的名字必须填写
    *
    你的电话必须填写
    *
    你的电话必须填写