renom.layers.function

class renom.layers.function.batch_normalize. BatchNormalize ( input_size=None , momentum=0.99 , mode='activation' , epsilon=1e-05 , initializer=<renom.utility.initializer.GlorotNormal object> )

Batch normalization function [1] . This layer accelerates learning speed with reducing internal covariate shift and allow us to set high learning rate.

When the forward propagation, if the argument inference is set to False this layer calculates moving average of mean and variance. Other wise the inference is set to True, this layer uses the moving average which calculated in the above mode.

If the argument mode is set to ‘activation’, this layer normalizes prior-layer features per unit. Otherwise the argument mode is set to ‘feature’, this layer normalizes prior-layer features per channel.

The ‘feature’ mode is only effective for 4D tensor input.

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:
  • input_size ( int ) – Input unit size.
  • momentum ( float ) – Momentum coefficient for the moving average.
  • mode ( str ) – ‘activation’ or ‘feature’.
  • epsilon ( float ) – Small number added to avoid division by zero.
  • initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(3, 2)
>>> x.shape
(3, 2)
>>> layer = rm.BatchNormalize(momentum=0.99)
>>> layer(x, inference=False)
batch_normalize([[-0.05047419,  0.00471613],
                 [-0.00887055, -0.01459344],
                 [ 0.05934474,  0.00987731]], dtype=float32)
[1] Sergey Ioffe, Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift(2015)
class renom.layers.function.conv2d. Conv2d ( channel=32 , filter=3 , padding=0 , stride=1 , input_size=None , initializer=<renom.utility.initializer.GlorotNormal object> )

2d convolution layer.

This class creates a convolution filter to be convolved with the input tensor. The instance of this class only accepts and outputs 4d tensors.

At instantiation, in the case of int input, filter, padding, and stride, the shape will be symmetric.

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:
  • channel ( int ) – The dimensionality of the output.
  • filter ( tuple,int ) – Filter size of the convolution kernel.
  • padding ( tuple,int ) – Size of the zero-padding around the image.
  • stride ( tuple,int ) – Stride-size of the convolution.
  • input_size ( tuple ) – Input unit size. This must be a tuple like (Channel, Height, Width).
  • initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> n, c, h, w = (10, 3, 32, 32)
>>> x = np.random.rand(n, c, h, w)
>>> x.shape
(10, 3, 32, 32)
>>> layer = rm.Conv2d(channel=32)
>>> z = layer(x)
>>> z.shape
(10, 32, 30, 30)

Note

Tensor data format is NCHW .

class renom.layers.function.deconv2d. Deconv2d ( channel=1 , filter=3 , padding=0 , stride=1 , input_size=None , initializer=<renom.utility.initializer.GlorotNormal object> )

2d convolution layer.

This class creates a convolution filter to be convolved with the input tensor. The instance of this class only accepts and outputs 4d tensors.

At instantiation, in the case of int input, filter, padding, and stride, the shape will be symmetric.

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:
  • channel ( int ) – The dimensionality of the output.
  • filter ( tuple,int ) – Filter size to witch used as convolution kernel.
  • padding ( tuple,int ) – Pad around image by 0 according to this size.
  • stride ( tuple,int ) – Specifying the strides of the convolution.
  • input_size ( tuple ) – Input unit size. This must be a tuple like (Channel, Height, Width).
  • initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> n, c, h, w = (10, 3, 32, 32)
>>> x = np.random.rand(n, c, h, w)
>>> x.shape
(10, 3, 32, 32)
>>> layer = rm.Deconv2d(channel=32)
>>> z = layer(x)
>>> z.shape
(10, 32, 34, 34)
class renom.layers.function.dense. Dense ( output_size , input_size=None , initializer=<renom.utility.initializer.GlorotNormal object> )

Fully connected layer as described bellow.

f(x)= w \cdot x + b

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:
  • output_size ( int ) – Output unit size.
  • input_size ( int ) – Input unit size.
  • initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(3, 2)
>>> x.shape
(3, 2)
>>> layer = rm.Dense(3)
>>> z = layer(x)
>>> z.shape
(3, 3)
class renom.layers.function.dropout. Dropout ( dropout_ratio=0.5 )

Applies Dropout [2] to the input.

Dropout function randomly selects a fraction (specified by dropout_ratio) of the data sets them to zero. Remaining data will be rescaled by 1/(1 - dropout_ratio) .

Parameters: dropout_ratio ( float ) – Dropout ratio.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> x = rm.Variable(np.random.rand(3, 2))
>>> x
Variable([[ 0.49312586,  0.95414829],
          [ 0.7346437 ,  0.9014256 ],
          [ 0.09413767,  0.29910043]], dtype=float32)
>>> layer = rm.Dropout(0.2)
>>> z = layer(x)
>>> z
dropout([[ 0.        ,  1.19268537],
         [ 0.91830462,  1.12678194],
         [ 0.        ,  0.37387553]], dtype=float32)
>>> z = rm.dropout(x, 0.75)
>>> z
dropout([[ 0.65750116,  0.        ],
         [ 0.        ,  1.20190084],
         [ 0.12551689,  0.39880058]], dtype=float32)
[2] Hinton, Geoffrey E.; Srivastava, Nitish; Krizhevsky, Alex; Sutskever, Ilya; Salakhutdinov, Ruslan R. (2012). Improving neural networks by preventing co-adaptation of feature detectors
class renom.layers.function.dropout. SpatialDropout ( dropout_ratio=0.5 )

Applies spatial dropout to the input. This function drops feature maps randomly.

Parameters: dropout_ratio ( float ) – Dropout ratio.
Raises: AssertionError – An assertion error will be raised if the input tensor dimension is not 4.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = rm.Variable(np.random.rand(3, 3, 1, 1))
Variable([[[[ 0.55784005]],
           [[ 0.99528867]],
           [[ 0.77544725]]],
          [[[ 0.65406305]],
           [[ 0.27961349]],
           [[ 0.43104461]]],
          [[[ 0.66176379]],
           [[ 0.70499772]],
           [[ 0.87102354]]]], dtype=float32)
>>> z = rm.spatial_dropout(x)
>>> z
spatial_dropout([[[[ 1.1156801 ]],
                  [[ 0.        ]],
                  [[ 1.5508945 ]]],
                 [[[ 1.30812609]],
                  [[ 0.55922699]],
                  [[ 0.        ]]],
                 [[[ 0.        ]],
                  [[ 1.40999544]],
                  [[ 1.74204707]]]], dtype=float32)

Note

SpatialDropout only accepts 4d tensor data.

class renom.layers.function.embedding. Embedding ( output_size , input_size , initializer=<renom.utility.initializer.GlorotNormal object> )

Embedding layer. This layer is the special case of dense layer. The case is that the input value is onehot encoded. Since the onehot encoded input is very sparse, the matrix product performed in the dense layer is redundant.

The difference between dense layer and embedding layer is bellow.

[Dense layer]
data -> onehot encoding -> onehot data -> dense layer -> embedded data
[Embedding layer]
data -> embedding layer -> embedded data
Parameters:
  • output_size ( int ) – Output unit size.
  • input_size ( int ) – Input unit size. This is same as number of embedding characters.
  • initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> N = 4
>>> a = np.arange(N).reshape(N, 1)
>>> layer = rm.Embedding(output_size=1, input_size=8)
>>> out = layer(a)

Note

  1. This layer only accepts matrix which shape is (N, 1) and has integer value. * N is batch size.
  2. Both output_size and input_size must be specified.
class renom.layers.function.flatten. Flatten

This function flattens an input tensor. It does not affect the batch size.

Example

>>> x = np.random.rand(3, 3, 32, 32)
>>> x = rm.Variable(x)
>>> z = rm.flatten(x)
>>> x.shape
(3, 3, 32, 32)
>>> z.shape
(3, 3072)
class renom.layers.function.lrn. Lrn ( n=5 , k=2 , a=0.0001 , b=0.75 )

Local response normalization function [3] .

y_{c_{out},j,i}= \frac{x_{c_{in},i,j}}{(k + a{\sum_{c=max(0, i-n/2)}^{min(N-1, i+n/2)} (x_{c,i,j})})^b}

x_{c,i,j} represents the c th conv filter’s output at the position of (i, j) in the feature map. y_{c_{out},j,i} represents the output of local response normalization. N is the number of the feature maps. n is the adjacent feature map number. The default argument values are recommended.

Parameters:
  • n ( int ) – Number of images used to be normalized.
  • k ( int ) – Constant.
  • a ( float ) – Scale factor.
  • b ( float ) – Exponential factor.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(3, 3, 32, 32)
>>> layer = rm.Lrn()
>>> z=layer(x)
>>> z.shape
(3, 3, 32, 32)
[3] Alex Krizhevsky Krizhevsky, , Ilya IlyaSutskever Sutskever, Geoff. ImageNet Classification with Deep Convolutional Neural Networks
class renom.layers.function.lstm. Lstm ( output_size , input_size=None , initializer=<renom.utility.initializer.GlorotNormal object> )

Long short time memory[4]_ . Lstm object has 8 weights and 4 biases parameters to learn.

Weights applied to the input of the input gate, forget gate and output gate. W_{ij}, Wgi_{ij}, Wgf_{ij}, Wgo_{ij}

Weights applied to the recuurent input of the input gate, forget gate and output gate. R_{ij}, Rgi_{ij}, Rgf_{ij}, Rgo_{ij}

\begin{split}u^t_{i} &= \sum_{j = 0}^{J-1} W_{ij}x^t_{j} + \sum_{k = 0}^{K-1} R_{ik}y^{t-1}_{k} + b_i \\ gi^t_{i} &= \sum_{j = 0}^{J-1} Wgi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgi_{ik}y^{t-1}_{k} + bi_i \\ gf^t_{i} &= \sum_{j = 0}^{J-1} Wgfi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgf_{ik}y^{t-1}_{k} + bi_f \\ go^t_{i} &= \sum_{j = 0}^{J-1} Wgo_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgo_{ik}y^{t-1}_{k} + bi_o \\ s^t_i &= sigmoid(gi^t_{i})tanh(u^t_{i}) + s^{t-1}_isigmoid(gf^t_{i}) \\ y^t_{i} &= go^t_{i}tanh(s^t_{i})\end{split}

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:
  • output_size ( int ) – Output unit size.
  • input_size ( int ) – Input unit size.
  • initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> n, d, t = (2, 3, 4)
>>> x = rm.Variable(np.random.rand(n, d))
>>> layer = rm.Lstm(2)
>>> z = 0
>>> for i in range(t):
...     z += rm.sum(layer(x))
...
>>> grad = z.grad()    # Backpropagation.
>>> grad.get(x)    # Gradient of x.
Add([[-0.01853334, -0.0585249 ,  0.01290053],
     [-0.0205425 , -0.05837972,  0.00467286]], dtype=float32)
>>> layer.truncate()
[4] Learning Precise Timing with LSTM Recurrent Networks
truncate ( )

Truncates temporal connection.

class renom.layers.function.parameterized. Model

Abstract class of neural network model.

train ( )

Context manager to control whether a computational graph will be created or not.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z
...
>>> x = rm.Variable(np.random.rand(3, 3))
>>> y = np.random.rand(3, 2)
>>> model = Model()
>>> z = model(x)
>>>
>>> with model.train():
...     loss = rm.mean_squared_error(z, y)
...
>>> dx1 = loss.grad().get(x)
>>> print("Gradient1 of x is \n{}".format(dx1))
Gradient1 of x is
array([[ 0.85432934,  0.51205811],
       [ 0.20379112,  0.62481132],
       [ 0.49004569,  0.35310219]])
>>>
>>> loss = rm.mean_squared_error(z, y)
>>> dx2 = loss.grad().get(x)
>>> print("Gradient2 of x is \n{}".format(dx2))
Gradient2 of x is
None
values ( )

Generates nested tuple of underlying models and params of models.

Each model generates tuple of two dictionary. The first dictionary contains child models, keyed by attribute name. The second dictionary contains parameters of the model, keyed by attribute name.

Example

(
    # child models of self
    {
        'layer1': (
            {},     # child model of self.layer1
            {       # params of layer1
                'w': [1,2],   # self.layer1.params.w
                'b': [3,4],   # self.layer1.params.b
            }
        ),
        'layer2': (
            {},     # child model of self.layer2
            {       # params of layer2
                'w': [1,2],   # self.layer2.params.w
                'b': [3,4],   # self.layer2.params.b
            }
    },
    # params of self
    {}
)
join_grads ( grads , others )

Merge gradients of other models. Others is a list of tuple of (model, grads) to be merged. Models listed in the others should have same structure with self.

save ( filename )

Save model attributes. For save attributes, please register attributes to the dictionary which is named as ‘SERIALIZED’.

Following example shows how to do it.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class MyModel(rm.Model):
...     SERIALIZED = ('_moving_avg', ) # Put any attributes for saving.
...     def __init__(self):
...         super(MyModel, self).__init__()
...         self._l1 = rm.Dense(2)
...         self._l2 = rm.Dense(1)
...         self._moving_avg = 0
...     def forward(self, x):
...         h = self._l1(x)
...         h = rm.relu(h)
...         h = self._l2(h)
...         self._moving_avg = np.float32(self._moving_avg*0.5 + rm.sum(h)*0.5)
...         return h
...
>>> model = MyModel()
>>> model(np.random.rand(12, 4))
>>> print(model._moving_avg)
1.95637
>>> model.save("test.h5") # Save
>>> model = MyModel() # Reset model object.
>>> model.load("test.h5") # Load
>>> print(model._moving_avg)
1.95637
Parameters: filename ( str ) – File name to save model.
load ( filename )

Load saved weights to model.

Parameters: filename ( str ) – File name of saved model.

Example

>>> model = rm.Dense(2)
>>> model.load("model.hd5")
class renom.layers.function.parameterized. Sequential ( layers , loss_function=None )

Sequential model.

Parameters: layers ( list ) – A list of layer objects.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> x = np.random.rand(32, 50)
>>> sequential = rm.Sequential([
...         rm.Dense(100),
...         rm.Relu(),
...         rm.Dense(10),
...     ])
...
>>> z = sequential(x)
>>> z.shape
(32, 10)
class renom.layers.function.pool2d. MaxPool2d ( filter=3 , padding=0 , stride=1 )

Max pooling function. In the case of int input, filter, padding, and stride, the shape will be symmetric.

Parameters:
  • filter ( tuple,int ) – Filter size of the convolution kernel.
  • padding ( tuple,int ) – Size of the zero-padding around the image.
  • stride ( tuple,int ) – Stride-size of the convolution.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> x = np.random.rand(3, 3, 32, 32)
>>> layer = rm.MaxPool2d(filter=3)
>>> z = layer(x)
>>> z.shape
(3, 3, 30, 30)
>>> z = rm.max_pool2d(x, filter=(3, 3), stride=(1, 1), padding=(0,0))
>>> z.shape
(3, 3, 30, 30)
class renom.layers.function.pool2d. AveragePool2d ( filter=3 , padding=0 , stride=1 )

Average pooling function. In the case of int input, filter, padding, and stride, the shape will be symmetric.

Parameters:
  • filter ( tuple,int ) – Filter size of the convolution kernel.
  • padding ( tuple,int ) – Size of the zero-padding around the image.
  • stride ( tuple,int ) – Stride-size of the convolution.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> x = np.random.rand(3, 3, 32, 32)
>>> layer = rm.AveragePool2d(filter=3)
>>> z = layer(x)
>>> z.shape
(3, 3, 30, 30)
>>> z = rm.average_pool2d(x, filter=(3, 3), stride=(1, 1), padding=(0,0))
>>> z.shape
(3, 3, 30, 30)