Tutorial 2 Cifar10 Image classifier

An introduction of Convolutional neural network and how to use GPU.

In this tutorial, we’ll apply a convolutional neural network (CNN) to another standardized dataset, the “CIFAR” data. This dataset is a collection of 60,000 images over 10 classes.

Required libraries

In [1]:
from __future__ import division, print_function
import os
import sys
import pickle

import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelBinarizer
from sklearn.metrics import confusion_matrix, classification_report

import renom as rm
from renom.optimizer import Sgd, Adam
from renom.cuda.cuda import set_cuda_active

GPU-enabled Computing

If you wish to use a GPU, you’ll need to call the set_cuda_active() with the single argument True . This will generally allow training to run much faster than relying on the CPU. You’ll need an NVIDIA GPU installed on your machine.

In [2]:
set_cuda_active(True)

Load data

Here we just unpickle the CIFAR image data, collected from the CIFAR website ( https://www.cs.toronto.edu/~kriz/cifar.html ). As in Tutorial 1, we scale the data to a range of 0 to 1 and binarize the labels.

In [3]:
dir = "../dataset/cifar-10-batches-py/"
paths = ["data_batch_1", "data_batch_2", "data_batch_3",
         "data_batch_4", "data_batch_5"]

def unpickle(f):
    fo = open(f, 'rb')
    if sys.version_info.major == 2:
        # Python 2.7
        d = pickle.load(fo)
    elif sys.version_info.major == 3:
        # Python 3.4
        d = pickle.load(fo, encoding="latin-1")
    fo.close()
    return d

# Load train data.
data = map(unpickle, [os.path.join(dir, p) for p in paths])
train_x = np.vstack([d["data"] for d in data])
train_y = np.vstack([d["labels"] for d in data])

# Load test data.
data = unpickle(os.path.join(dir, "test_batch"))
test_x = np.array(data["data"])
test_y = np.array(data["labels"])

# Rehsape and rescale image.
train_x = train_x.reshape(-1, 3, 32, 32)
train_y = train_y.reshape(-1, 1)
test_x = test_x.reshape(-1, 3, 32, 32)
test_y = test_y.reshape(-1, 1)

train_x = train_x / 255.
test_x = test_x / 255.

# Binalize
labels_train = LabelBinarizer().fit_transform(train_y)
labels_test = LabelBinarizer().fit_transform(test_y)

# Change types.
train_x = train_x.astype(np.float32)
test_x = test_x.astype(np.float32)
labels_train = labels_train.astype(np.float32)
labels_test = labels_test.astype(np.float32)

N = len(train_x)

Neural network definition

Setup the CNN- essentially similar to Tutorial 1, except that here we are using several hidden layers. Also, we try to avoid over-fitting, by using the’‘dropout’’ technique.

In [4]:
class Cifar10(rm.Model):

    def __init__(self):
        super(Cifar10, self).__init__()
        self._l1 = rm.Conv2d(channel=32)
        self._l2 = rm.Conv2d(channel=32)
        self._l3 = rm.Conv2d(channel=64)
        self._l4 = rm.Conv2d(channel=64)
        self._l5 = rm.Dense(512)
        self._l6 = rm.Dense(10)
        self._sd = rm.SpatialDropout(dropout_ratio=0.25)
        self._pool = rm.MaxPool2d(filter=2, stride=2)

    def forward(self, x, inference = False):
        t1 = rm.relu(self._l1(x))
        t2 = self._sd(self._pool(rm.relu(self._l2(t1))), inference)
        t3 = rm.relu(self._l3(t2))
        t4 = self._sd(self._pool(rm.relu(self._l4(t3))), inference)
        t5 = rm.flatten(t4)
        t6 = rm.dropout(rm.relu(self._l5(t5)), inference = inference)
        t7 = self._l6(t5)
        return t7

Definition of a nural network with sequential model

In [5]:
sequential = rm.Sequential([
        rm.Conv2d(channel=32),
        rm.Relu(),
        rm.Conv2d(channel=32),
        rm.Relu(),
        rm.MaxPool2d(filter=2, stride=2),
        rm.Dropout(dropout_ratio=0.25),
        rm.Conv2d(channel=64),
        rm.Relu(),
        rm.Conv2d(channel=64),
        rm.Relu(),
        rm.MaxPool2d(filter=2, stride=2),
        rm.Dropout(dropout_ratio=0.25),
        rm.Flatten(),
        rm.Dense(512),
        rm.Relu(),
        rm.Dropout(dropout_ratio=0.5),
        rm.Dense(10),
    ])

Instantiation

In [6]:
# Choose neural network.
# network = Cifar10()
network = sequential
optimizer = Adam()

Training loop

In the training loop, we recommend running a validation step for each minibatch. This will allow us to check the learning process, and prevent overfitting. This also allows you to diagnose training problems by comparing the validation and training learning curves.

In [7]:
# Hyper parameters
batch = 128
epoch = 20

learning_curve = []
test_learning_curve = []

for i in xrange(epoch):
    perm = np.random.permutation(N)
    loss = 0
    for j in xrange(0, N // batch):
        train_batch = train_x[perm[j * batch:(j + 1) * batch]]
        responce_batch = labels_train[perm[j * batch:(j + 1) * batch]]

        # Loss function
        network.set_models(inference=False)
        with network.train():
            l = rm.softmax_cross_entropy(network(train_batch), responce_batch)

        # Back propagation
        grad = l.grad()

        # Update
        grad.update(optimizer)
        loss += l.as_ndarray()

    train_loss = loss / (N // batch)

    # Validation
    test_loss = 0
    M = len(test_x)
    network.set_models(inference=True)
    for j in range(M//batch):
        test_batch = test_x[j * batch:(j + 1) * batch]
        test_label_batch = labels_test[j * batch:(j + 1) * batch]
        prediction = network(test_batch)
        test_loss += rm.softmax_cross_entropy(prediction, test_label_batch).as_ndarray()
    test_loss /= (j+1)

    test_learning_curve.append(test_loss)
    learning_curve.append(train_loss)
    print("epoch %03d train_loss:%f test_loss:%f"%(i, train_loss, test_loss))
epoch 000 train_loss:1.672494 test_loss:1.328143
epoch 001 train_loss:1.284703 test_loss:1.149910
epoch 002 train_loss:1.118638 test_loss:1.029675
epoch 003 train_loss:1.006356 test_loss:0.936864
epoch 004 train_loss:0.922219 test_loss:0.844404
epoch 005 train_loss:0.860780 test_loss:0.814329
epoch 006 train_loss:0.808263 test_loss:0.800741
epoch 007 train_loss:0.764056 test_loss:0.776209
epoch 008 train_loss:0.731968 test_loss:0.760535
epoch 009 train_loss:0.702022 test_loss:0.724888
epoch 010 train_loss:0.669088 test_loss:0.701457
epoch 011 train_loss:0.650730 test_loss:0.716129
epoch 012 train_loss:0.621038 test_loss:0.698542
epoch 013 train_loss:0.605118 test_loss:0.693534
epoch 014 train_loss:0.586976 test_loss:0.682453
epoch 015 train_loss:0.576019 test_loss:0.676529
epoch 016 train_loss:0.546329 test_loss:0.713515
epoch 017 train_loss:0.536031 test_loss:0.691634
epoch 018 train_loss:0.534909 test_loss:0.708497
epoch 019 train_loss:0.517823 test_loss:0.686890

Model evaluation

Finally, we evaluate the models labeling performance using the same metrics as in Tutorial 1.

In [8]:
network.set_models(inference=True)
predictions = np.argmax(network(test_x).as_ndarray(), axis=1)

# Confusion matrix and classification report.
print(confusion_matrix(test_y, predictions))
print(classification_report(test_y, predictions))

# Learning curve.
plt.plot(learning_curve, linewidth=3, label="train")
plt.plot(test_learning_curve, linewidth=3, label="test")
plt.title("Learning curve")
plt.ylabel("error")
plt.xlabel("epoch")
plt.legend()
plt.grid()
plt.show()
[[865  14  30  10  11   2   5  12  35  16]
 [ 19 892   2   7   3   4   6   4  14  49]
 [ 70   3 639  44  96  56  52  29   7   4]
 [ 32  11  75 520  55 188  65  36   9   9]
 [ 15   3  48  53 750  23  34  65   7   2]
 [  7   4  39  96  50 728  20  49   3   4]
 [  8   3  32  38  29  20 849  12   5   4]
 [ 14   1  21  26  41  56   4 834   0   3]
 [ 84  30  13  16   3   4   6   3 819  22]
 [ 38  73   9  10   3   4   4  17  20 822]]
             precision    recall  f1-score   support

          0       0.75      0.86      0.80      1000
          1       0.86      0.89      0.88      1000
          2       0.70      0.64      0.67      1000
          3       0.63      0.52      0.57      1000
          4       0.72      0.75      0.73      1000
          5       0.67      0.73      0.70      1000
          6       0.81      0.85      0.83      1000
          7       0.79      0.83      0.81      1000
          8       0.89      0.82      0.85      1000
          9       0.88      0.82      0.85      1000

avg / total       0.77      0.77      0.77     10000

../../_images/notebooks_tutorial2-cifar10_notebook_16_1.png
In [ ]: