Backpropagation, chain rule

• Backpropagation

Whereas Logistic Regression is made of a single layer of weights, Neural Networks are made of many layers with non-linear activation.

We need to have a mechanism to update the weights of the neurons using the gradient of the loss. Such a mechanism is called Backpropagation. It propagates the gradient from the last layer to the first one.

 

• chain rule

It allows you to decompose the calculation of the derivative of a composite function. It is at the core of the backpropagation algorithm.

 

chain rule 1

 

chain rule 2

chain rule 3

 

---

Let's say we are going to Backpropagation for Sigmoid function,

Flow of Sigmoid

 

Then, we will get this.

Sigmoid Example

 

We will get the Backpropagation result as below.

Backpropagation Example