Linear Regression-requires_grad, zero_grad, backward, step

Step

1. Initialize weight and bias : requires_grad=True (if False, This will prevent updating of model weights during fine-tuning.)

2. Set the optimizer and learning rate

3. Set hypothesis

4. Get cost

5. Initialize optimizer : zero_grad()

6. backward : differenciate cost function -> get gradient -> update w, b when backpropagation

7. step : apply learning rate to w, b -> update w, b

 

optimizer(differenciation) : decide direction

learning rate : decide stride

W = torch.zeros(1, requires_grad=True) 
b = torch.zeros(1, requires_grad=True)

optimizer=optim.SGD([w1, w2, w3, b], lr=1e-5)
nb_epochs=1000

for epoch in range(nb_epochs+1):
    h=x1*w1+x2*w2+x3+w3+b
    cost=torch.mean((h-y)**2)
    # initialize gradient to 0
    optimizer.zero_grad()
    
    # calculate w, b on the propagation cost function
    cost.backward()
    
    # update of W and b
    optimizer.step()
    
    if epoch %100==0:
        print('Epoch {:4d}/{} w1: {:.3f} w2: {:.3f} w3: {:.3f} b: {:.3f} Cost: {:.6f}'.format(
            epoch, nb_epochs, w1.item(), w2.item(), w3.item(), b.item(), cost.item()
        ))
        
>>>

Epoch    0/1000 w1: 0.545 w2: 0.462 w3: 0.008 b: 0.008 Cost: 1.261497

...

Epoch 1000/1000 w1: 0.576 w2: 0.431 w3: 0.010 b: 0.010 Cost: 1.061194

tensor.item() : Returns the value of this tensor as a standard Python number. This only works for tensors with one element.