renom.layers.function package

class renom.layers.function.batch_normalize. BatchNormalize ( input_size=None , momentum=0.99 , mode='activation' , epsilon=1e-05 , initializer=<renom.utility.initializer.GlorotNormal object> )

Bases: renom.layers.function.parameterized.Parametrized

Batch normalization function [1] . This layer accelerates learning speed with reducing internal covariate shift and allow us to set high learning rate.

When the forward propagation, if the argument inference is set to False this layer calculates moving average of mean and variance. Other wise the inference is set to True, this layer uses the moving average for forward propagation.

If the argument mode is set to ‘feature’, normalize prior-layer features per patch.

Parameters:
  • input_size ( int ) – Input unit size.
  • momentum ( float ) – Momentum coefficient for the moving average.
  • mode ( str ) – ‘activation’ or ‘feature’
  • epsilon ( float ) – Small number added to avoid division by zero

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(3, 2)
>>> x.shape
(3, 2)
>>> layer = rm.BatchNormalize(momentum=0.99)
>>> layer(x, inference=False)
batch_normalize([[-0.05047419,  0.00471613],
                 [-0.00887055, -0.01459344],
                 [ 0.05934474,  0.00987731]], dtype=float32)
[1] Sergey Ioffe, Christian Szegedy.(2015).Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
class renom.layers.function.conv2d. Conv2d ( channel=32 , filter=3 , padding=0 , stride=1 , input_size=None , initializer=<renom.utility.initializer.GlorotNormal object> )

Bases: renom.layers.function.parameterized.Parametrized

2d convolution function.

This class creates a convolution filter to be convolved with the input tensor. The instance of this class only accepts and outputs 4d tensors.

At instantiation, in the case of int input, filter, padding, and stride, the shape will be symmetric.

Parameters:
  • channel ( int ) – The dimensionality of the output.
  • filter ( tuple,int ) – Filter size of the convolution kernel.
  • padding ( tuple,int ) – Size of the zero-padding around the image.
  • stride ( tuple,int ) – Stride-size of the convolution.
  • input_size ( tuple ) – Input unit size. This must be a tuple like (Channel, Height, Width).

Example

>>> import numpy as np
>>> import renom as rm
>>> n, c, h, w = (10, 3, 32, 32)
>>> x = np.random.rand(n, c, h, w)
>>> x.shape
(10, 3, 32, 32)
>>> layer = rm.Conv2d(channel=32)
>>> z = layer(x)
>>> z.shape
(10, 32, 30, 30)

Note

Tensor data format is NCHW .

class renom.layers.function.deconv2d. Deconv2d ( channel=1 , filter=3 , padding=0 , stride=1 , input_size=None , initializer=<renom.utility.initializer.GlorotNormal object> )

Bases: renom.layers.function.parameterized.Parametrized

2d convolution function.

This class creates a convolution filter to be convolved with the input tensor. The instance of this class only accepts and outputs 4d tensors.

At instantiation, in the case of int input, filter, padding, and stride, the shape will be symmetric.

Parameters:
  • channel ( int ) – The dimensionality of the output.
  • filter ( tuple,int ) – Filter size to witch used as convolution kernel.
  • padding ( tuple,int ) – Pad around image by 0 according to this size.
  • stride ( tuple,int ) – Specifying the strides of the convolution.
  • input_size ( tuple ) – Input unit size. This must be a tuple like (Channel, Height, Width).

Example

>>> import numpy as np
>>> import renom as rm
>>> n, c, h, w = (10, 3, 32, 32)
>>> x = np.random.rand(n, c, h, w)
>>> x.shape
(10, 3, 32, 32)
>>> layer = rm.Deconv2d(channel=32)
>>> z = layer(x)
>>> z.shape
(10, 32, 34, 34)
class renom.layers.function.dense. Dense ( output_size , input_size=None , initializer=<renom.utility.initializer.GlorotNormal object> )

Bases: renom.layers.function.parameterized.Parametrized

Fully connected layer as described bellow.

f(x)= w \cdot x + b
Parameters:
  • output_size ( int ) – Output unit size.
  • input_size ( int ) – Input unit size.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(3, 2)
>>> x.shape
(3, 2)
>>> layer = rm.Dense(3)
>>> z = layer(x)
>>> z.shape
(3, 3)
class renom.layers.function.dropout. Dropout ( dropout_ratio=0.5 )

Bases: renom.layers.function.parameterized.Model

Applies Dropout [2] to the input.

Dropout function randomly selects a fraction (specified by dropout_ratio) of the data sets them to zero. Remaining data will be rescaled by 1/(1 - dropout_ratio) .

Parameters: dropout_ratio ( float ) – Dropout ratio.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> x = rm.Variable(np.random.rand(3, 2))
>>> x
Variable([[ 0.49312586,  0.95414829],
          [ 0.7346437 ,  0.9014256 ],
          [ 0.09413767,  0.29910043]], dtype=float32)
>>> layer = rm.Dropout(0.2)
>>> z = layer(x)
>>> z
dropout([[ 0.        ,  1.19268537],
         [ 0.91830462,  1.12678194],
         [ 0.        ,  0.37387553]], dtype=float32)
>>> z = rm.dropout(x, 0.75)
>>> z
dropout([[ 0.65750116,  0.        ],
         [ 0.        ,  1.20190084],
         [ 0.12551689,  0.39880058]], dtype=float32)
[2] Hinton, Geoffrey E.; Srivastava, Nitish; Krizhevsky, Alex; Sutskever, Ilya; Salakhutdinov, Ruslan R. (2012). Improving neural networks by preventing co-adaptation of feature detectors
class renom.layers.function.dropout. SpatialDropout ( dropout_ratio=0.5 )

Bases: renom.layers.function.dropout.Dropout

Applies spatial dropout to the input. This function drops feature maps randomly.

Parameters: dropout_ratio ( float ) – Dropout ratio.
Raises: AssertionError – An assertion error will be raised if the input tensor dimension is not 4.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = rm.Variable(np.random.rand(3, 3, 1, 1))
Variable([[[[ 0.55784005]],
           [[ 0.99528867]],
           [[ 0.77544725]]],
          [[[ 0.65406305]],
           [[ 0.27961349]],
           [[ 0.43104461]]],
          [[[ 0.66176379]],
           [[ 0.70499772]],
           [[ 0.87102354]]]], dtype=float32)
>>> z = rm.spatial_dropout(x)
>>> z
spatial_dropout([[[[ 1.1156801 ]],
                  [[ 0.        ]],
                  [[ 1.5508945 ]]],
                 [[[ 1.30812609]],
                  [[ 0.55922699]],
                  [[ 0.        ]]],
                 [[[ 0.        ]],
                  [[ 1.40999544]],
                  [[ 1.74204707]]]], dtype=float32)

Note

SpatialDropout only accepts 4d tensor data.

class renom.layers.function.flatten. Flatten

Bases: object

This function flattens an input tensor. It does not affect the batch size.

Example

>>> x = np.random.rand(3, 3, 32, 32)
>>> x = rm.Variable(x)
>>> z = rm.flatten(x)
>>> x.shape
(3, 3, 32, 32)
>>> z.shape
(3, 3072)
class renom.layers.function.lrn. Lrn ( n=5 , k=2 , a=0.0001 , b=0.75 )

Bases: object

Local response normalization function [3] .

y_{c_{out},j,i}= x_{c_{in},i,j}/(k + a{\sum_{c=max(0, i-n/2)}^{min(N-1, i+n/2)} (x_{c,i,j})})^b

x_{c,i,j} represents the c th conv. filter’s output at the position of (i, j) in the feature map. y_{c_{out},j,i} represents the output of local response normalization. N is the number of the feature maps. n is the adjacent feature map number. The default argument values are recommended.

Parameters:
  • n ( int ) – Number of images used to be normalized.
  • k ( int ) – Constant.
  • a ( float ) – Scale factor.
  • b ( float ) – Exponential factor.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(3, 3, 32, 32)
>>> layer = rm.Lrn()
>>> z=layer(x)
>>> z.shape
(3, 3, 32, 32)
[3] Alex Krizhevsky Krizhevsky, , Ilya IlyaSutskever Sutskever, Geoff. ImageNet Classification with Deep Convolutional Neural Networks
class renom.layers.function.lstm. Lstm ( output_size , initializer=<renom.utility.initializer.GlorotNormal object> )

Bases: renom.layers.function.parameterized.Parametrized

Long short time memory[4]_ . Lstm object has 12 weights and 4 biases parameters to learn.

Weights applied to the input of the input gate, forget gate and output gate. W_{ij}, Wgi_{ij}, Wgf_{ij}, Wgo_{ij}

Weights applied to the recuurent input of the input gate, forget gate and output gate. R_{ij}, Rgi_{ij}, Rgf_{ij}, Rgo_{ij}

\begin{split}u^t_{i} &= \sum_{j = 0}^{J-1} W_{ij}x^t_{j} + \sum_{k = 0}^{K-1} R_{ik}y^{t-1}_{k} + b_i \\ gi^t_{i} &= \sum_{j = 0}^{J-1} Wgi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgi_{ik}y^{t-1}_{k} + bi_i \\ gf^t_{i} &= \sum_{j = 0}^{J-1} Wgfi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgf_{ik}y^{t-1}_{k} + bi_f \\ go^t_{i} &= \sum_{j = 0}^{J-1} Wgo_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgo_{ik}y^{t-1}_{k} + bi_o \\ s^t_i &= sigmoid(gi^t_{i})tanh(u^t_{i}) + s^{t-1}_isigmoid(gf^t_{i}) \\ y^t_{i} &= go^t_{i}tanh(s^t_{i})\end{split}
Parameters: output_size ( int ) –

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> n, d, t = (2, 3, 4)
>>> x = rm.Variable(np.random.rand(n, d))
>>> layer = rm.Lstm(2)
>>> z = 0
>>> for i in range(t):
...     z += rm.sum(layer(x))
...
>>> grad = z.grad()    # Backpropagation.
>>> grad.get(x)    # Gradient of x.
Add([[-0.01853334, -0.0585249 ,  0.01290053],
     [-0.0205425 , -0.05837972,  0.00467286]], dtype=float32)
>>> layer.truncate()
[4] Learning Precise Timing with LSTM Recurrent Networks
truncate ( )

Truncates temporal connection.

class renom.layers.function.parameterized. Model

Bases: object

Abstract class of neural network model.

train ( )

Context manager to control whether a computational graph will be created or not.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z
...
>>> x = rm.Variable(np.random.rand(3, 3))
>>> y = np.random.rand(3, 2)
>>> model = Model()
>>> z = model(x)
>>>
>>> with model.train():
...     loss = rm.mean_squared_error(z, y)
...
>>> dx1 = loss.grad().get(x)
>>> print("Gradient1 of x is \n{}".format(dx1))
Gradient1 of x is
array([[ 0.85432934,  0.51205811],
       [ 0.20379112,  0.62481132],
       [ 0.49004569,  0.35310219]])
>>>
>>> loss = rm.mean_squared_error(z, y)
>>> dx2 = loss.grad().get(x)
>>> print("Gradient2 of x is \n{}".format(dx2))
Gradient2 of x is
None
values ( )

Generates nested tuple of underlying models and params of models.

Each model generates tuple of two dictionary. The first dictionary contains child models, keyed by attribute name. The second dictionary contains parameters of the model, keyed by attribute name.

Example

(
    # child models of self
    {
        'layer1': (
            {},     # child model of self.layer1
            {       # params of layer1
                'w': [1,2],   # self.layer1.params.w
                'b': [3,4],   # self.layer1.params.b
            }
        ),
        'layer2': (
            {},     # child model of self.layer2
            {       # params of layer2
                'w': [1,2],   # self.layer2.params.w
                'b': [3,4],   # self.layer2.params.b
            }
    },
    # params of self
    {}
)
join_grads ( grads , others )

Merge gradients of other models. Others is a list of tuple of (model, grads) to be merged. Models listed in the others should have same structure with self.

save ( filename )

Save model weights.

Parameters: filename ( str ) – File name to save model.

Example

>>> model = rm.Dense(2)
>>> model.save("model.hd5")
load ( filename )

Load saved weights to model.

Parameters: filename ( str ) – File name of saved model.

Example

>>> model = rm.Dense(2)
>>> model.load("model.hd5")
class renom.layers.function.parameterized. Sequential ( layers , loss_function=None )

Bases: renom.layers.function.parameterized.Model

Sequential model.

Parameters: layers ( list ) – A list of layer objects.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> x = np.random.rand(32, 50)
>>> sequential = rm.Sequential([
...         rm.Dense(100),
...         rm.Relu(),
...         rm.Dense(10),
...     ])
...
>>> z = sequential(x)
>>> z.shape
(32, 10)
class renom.layers.function.peephple_lstm. PeepholeLstm ( output_size , input_size=None , initializer=<renom.utility.initializer.GlorotNormal object> )

Bases: renom.layers.function.parameterized.Parametrized

Long short time memory with peephole [4]_ . Lstm object has 12 weights and 4 biases parameters to learn.

Weights applied to the input of the input gate, forget gate and output gate. W_{ij}, Wgi_{ij}, Wgf_{ij}, Wgo_{ij}

Weights applied to the recuurent input of the input gate, forget gate and output gate. R_{ij}, Rgi_{ij}, Rgf_{ij}, Rgo_{ij}

Weights applied to the state input of the input gate, forget gate and output gate. P_{ij}, Pgi_{ij}, Pgf_{ij}, Pgo_{ij}

\begin{split}u^t_{i} &= \sum_{j = 0}^{J-1} W_{ij}x^t_{j} + \sum_{k = 0}^{K-1} R_{ik}y^{t-1}_{k} + P_{i}s^{t-1}_{i} + b_i \\ gi^t_{i} &= \sum_{j = 0}^{J-1} Wgi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgi_{ik}y^{t-1}_{k} + Pgi_{i}s^{t-1}_{i} + bi_i \\ gf^t_{i} &= \sum_{j = 0}^{J-1} Wgfi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgf_{ik}y^{t-1}_{k} + Pgf_{i}s^{t-1}_{i} + bi_f \\ go^t_{i} &= \sum_{j = 0}^{J-1} Wgo_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgo_{ik}y^{t-1}_{k} + Pgo_{i}s^{t}_{i} + bi_o \\ s^t_i &= sigmoid(gi^t_{i})tanh(u^t_{i}) + s^{t-1}_isigmoid(gf^t_{i}) \\ y^t_{i} &= go^t_{i}tanh(s^t_{i})\end{split}
Parameters:
  • output_size ( int ) – Output unit size.
  • input_size ( int ) – Input unit size.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> n, d, t = (2, 3, 4)
>>> x = rm.Variable(np.random.rand(n, d))
>>> layer = rm.Lstm(2)
>>> z = 0
>>> for i in range(t):
...     z += rm.sum(layer(x))
...
>>> grad = z.grad()    # Backpropagation.
>>> grad.get(x)    # Gradient of x.
Add([[-0.01853334, -0.0585249 ,  0.01290053],
     [-0.0205425 , -0.05837972,  0.00467286]], dtype=float32)
>>> layer.truncate()
[4] Felix A. Gers, Nicol N. Schraudolph, J ̈urgen Schmidhuber. Learning Precise Timing with LSTM Recurrent Networks
truncate ( )

Truncates temporal connection.

class renom.layers.function.pool2d. MaxPool2d ( filter=3 , padding=0 , stride=1 )

Bases: renom.layers.function.pool2d.PoolBase

Max pooling function. In the case of int input, filter, padding, and stride, the shape will be symmetric.

Parameters:
  • filter ( tuple,int ) – Filter size of the convolution kernel.
  • padding ( tuple,int ) – Size of the zero-padding around the image.
  • stride ( tuple,int ) – Stride-size of the convolution.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> x = np.random.rand(3, 3, 32, 32)
>>> layer = rm.MaxPool2d(filter=3)
>>> z = layer(x)
>>> z.shape
(3, 3, 30, 30)
>>> z = rm.max_pool2d(x, filter=(3, 3), stride=(1, 1), padding=(0,0))
>>> z.shape
(3, 3, 30, 30)
class renom.layers.function.pool2d. AveragePool2d ( filter=3 , padding=0 , stride=1 )

Bases: renom.layers.function.pool2d.PoolBase

Average pooling function. In the case of int input, filter, padding, and stride, the shape will be symmetric.

Parameters:
  • filter ( tuple,int ) – Filter size of the convolution kernel.
  • padding ( tuple,int ) – Size of the zero-padding around the image.
  • stride ( tuple,int ) – Stride-size of the convolution.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> x = np.random.rand(3, 3, 32, 32)
>>> layer = rm.AveragePool2d(filter=3)
>>> z = layer(x)
>>> z.shape
(3, 3, 30, 30)
>>> z = rm.average_pool2d(x, filter=(3, 3), stride=(1, 1), padding=(0,0))
>>> z.shape
(3, 3, 30, 30)