ReNom Basic Calculation using Auto Differentiation

This is an introduction of ReNom basic calculation using auto differentiation.

Auto differentiation for loss function is used in following .grad part.

loss.grad()
Just as an example, we generate data following a sin function, add some noise, and then plot the data into training and test datasets.
Then we define a 2-layer neural network with a naive representation.

Requirements

For this tutorial, you will need the following modules:

In [1]:
# -*- coding: utf-8 -*-
import matplotlib.pyplot as plt
import numpy as np
import renom as rm

Data preparation

As stated above, we’ll now generate data from sin function. The population of our dataset population_distribution is defined as below.
The training data and test data are generated with regard to population_distribution .
In [2]:
population_distribution = lambda x:np.sin(x) + np.random.randn(*x.shape)*0.05

train_x = np.random.rand(1024, 1)*np.pi*2
train_y = population_distribution(train_x)

test_x = np.random.rand(128, 1)*np.pi*2
test_y = population_distribution(test_x)

This graph shows the distribution of the generated dataset. The training set is in blue, the test set in orange.

In [3]:
plt.clf()
plt.grid()
plt.scatter(train_x, train_y, label = "train")
plt.scatter(test_x, test_y, label="test")
plt.title("Population of dataset")
plt.ylabel("y")
plt.xlabel("x")
plt.legend()
plt.show()
../../../_images/notebooks_basic_autodifferentiation_notebook_6_0.png

Neural network definition

We define a 2-layer neural network, with 2 weight parameters and 2 bias parameters. These parameters will be updated with respect to their gradients, so we need to instantiate them as Variable objects. We show the image of the neural network as follows.

In [4]:
INPUT_SIZE = 1
OUTPUT_SIZE = 1
HIDDEN_SIZE = 5

w1 = rm.Variable(np.random.randn(INPUT_SIZE, HIDDEN_SIZE)*0.01)
b1 = rm.Variable(np.zeros((1, HIDDEN_SIZE)))
w2 = rm.Variable(np.random.randn(HIDDEN_SIZE, OUTPUT_SIZE)*0.01)
b2 = rm.Variable(np.zeros((1, OUTPUT_SIZE)))

optimiser = rm.Sgd(0.01)

def nn_forward(x):
    z = rm.dot(rm.tanh(rm.dot(x, w1) + b1), w2) + b2
    return z

def nn(x, y):
    z = nn_forward(x)
    loss = rm.sum(((z - y)**2)/2)
    return loss

Training loop

The training loop is described below.
.grad calculate the gradient of the ReNom Variable Object and update(optimizer) updates the parameter(weight and bias) based on optimizer method.
In [5]:
N = len(train_x)
batch_size = 32
train_curve = []
for i in range(1, 101):
    perm = np.random.permutation(N)
    total_loss = 0
    for j in range(N//batch_size):
        index = perm[j*batch_size:(j+1)*batch_size]
        train_batch_x = train_x[index]
        train_batch_y = train_y[index]
        loss = nn(train_batch_x, train_batch_y)
        loss.grad().update(optimiser)
        total_loss += loss.as_ndarray()
    train_curve.append(total_loss/(j+1))
    if i%10 == 0:
        print("epoch %02d train_loss:%f"%(i, train_curve[-1]))

plt.clf()
plt.grid()
plt.plot(train_curve)
plt.title("Training curve")
plt.ylabel("train error")
plt.xlabel("epoch")
plt.show()
epoch 10 train_loss:1.506160
epoch 20 train_loss:0.620538
epoch 30 train_loss:0.554743
epoch 40 train_loss:0.605716
epoch 50 train_loss:0.503734
epoch 60 train_loss:0.393485
epoch 70 train_loss:0.247103
epoch 80 train_loss:0.219726
epoch 90 train_loss:0.221648
epoch 100 train_loss:0.181218
../../../_images/notebooks_basic_autodifferentiation_notebook_10_1.png

Prediction

Finally, we test our model against the test dataset. Normally, ReNom returns a Node object, and contiues to expand the computational graph. For this to work, the as_ndarray method should be called.

In [6]:
predicted = nn_forward(test_x).as_ndarray()

Now, if we visually inspect the result, we can confirm that the model successfully approximates the test population.

In [7]:
plt.clf()
plt.grid()
plt.scatter(test_x, test_y, label = "true")
plt.scatter(test_x, predicted, label="predicted")
plt.title("Prediction result")
plt.ylabel("y")
plt.xlabel("x")
plt.legend()
plt.show()
../../../_images/notebooks_basic_autodifferentiation_notebook_14_0.png