renom.utility.distributor package

class renom.utility.distributor.distributor. Distributor ( x=None , y=None , path=None , data_table=None )

Bases: object

Distributor class This is the base class of a data distributor.

Parameters:
  • x ( ndarray ) – Input data.
  • y ( ndarray ) – Target data.
  • path ( string ) – Path to data.
>>> import numpy as np
>>> from renom.utility.distributor.distributor import NdarrayDistributor
>>> x = np.random.randn(100, 100)
>>> y = np.random.randn(100, 1)
>>> distributor = NdarrayDistributor(x, y)
>>> batch_x, batch_y = distributor.batch(10).next()
>>> batch_x.shape
(10, 100)
>>> batch_y.shape
(10, 1)
batch ( batch_size , shuffle=True )

This function returns minibatch .

Parameters:
  • batch_size ( int ) – Size of batch.
  • shuffle ( bool ) – If True is passed, data will be selected randomly.
split ( ratio=0.8 , shuffle=True )

データを分割し、新たなDistributorインスタンスを作成する。

Parameters:
  • ratio ( float ) – 分割比
  • shuffle ( boolean ) – 真のとき、分割時に並び順をシャッフルする。
class renom.utility.distributor.distributor. NdarrayDistributor ( x , y , **kwargs )

Bases: renom.utility.distributor.distributor.Distributor

NumpyArrayを扱うDistributorクラス

Parameters:
  • x ( ndarray ) – 入力データ
  • y ( ndarray ) – 教師データ
class renom.utility.distributor.distributor. TimeSeriesDistributor ( x , y , **kwargs )

Bases: renom.utility.distributor.distributor.NdarrayDistributor

時系列データを扱うDistributorクラス。 時系列データは以下のフォーマットで与えられる必要がある。

N: データ数 T: 時系列長 D: データ次元 (N, T, D)

Parameters:
  • x ( ndarray ) – 入力データ
  • y ( ndarray ) – 教師データ
class renom.utility.distributor.imageloader. ImageLoader ( batches , color='RGB' )

Bases: object

ImageLoader is a generator that yields images in batches. By inputting list of image path, ImageLoader load images and yields according to number of batch size.

Parameters:
  • batches ( list ) – List of image list of a batch
  • color ( str ) – Color Space of Input Image
Example:
>>> batches = [[('/data/file1.jpg', '/data/file2.jpg')], [('/data/file3.jpg', '/data/file4.jpg')] ]
>>> loader = ImageLoader(batches)
>>> for i, (x) in enumerate(dist.batch(2)):
...    print 'Batch', i
wait_images ( )

yeild images from batches which is list of image.

class renom.utility.distributor.threadingdistributor. ImageDistributor ( image_path_list , y_list=None , class_list=None , imsize=(32 , 32) , color='RGB' , augmentation=None )

Bases: object

Base class for image distribution. Use subclasses ImageClassificationDistributor, ImageDetectionDistributor, ImageSegmentationDistributor depending on the image task. Or sublass it for original image tasks.

Parameters:
  • image_path_list ( list ) – list of image path
  • y_list ( list ) – list of labels (bbox and class) for every image (2 dimensional array)
  • class_list ( list ) – list of classes name for this dataset
  • shuffle ( bool ) – If True, apply datasets shuffle per epoch
  • imsize ( tuple ) – resize input image for converting batch ndarray
  • color ( str ) – color of Input Image. [“RGB”, “GRAY”]
  • augmentation ( fucntion ) – augmentater for Input Image
class renom.utility.distributor.threadingdistributor. ImageDetectionDistributor ( image_path_list , y_list=None , class_list=None , imsize=(360 , 360) , color='RGB' , augmentation=None )

Bases: renom.utility.distributor.threadingdistributor.ImageDistributor

Distributor class for tasks of image detection. Labels are expected to be Bounding boxes and Classes. ex:) np.array([[center x, center y, x_top_left, height, 0, 0, 0, 1, 0]])

Parameters:
  • image_path_list ( list ) – list of image path
  • y_list ( list ) – list of labels (bbox and class) for every image
  • class_list ( list ) – list of classes name for this dataset
  • shuffle ( bool ) – If True, apply datasets shuffle per epoch
  • imsize ( tuple ) – resize input image for converting batch ndarray
  • color ( str ) – color of Input Image. [“RGB”, “GRAY”]
  • augmentation ( fucntion ) – augmentater for Input Image
Example:
>>> from renom.utility.load.imageloader.threadingdistributor import ImageDetectionDistributor
>>> from renom.utility.image.data_augmentation import *
>>> datagenerator = DataAugmentation([
...     Flip(1),
...     Rotate(20),
...     Crop(size=(300, 300)),
...     Resize(size=(500, 500)),
...     Shift((20, 50)),
...     Color_jitter(v = (0.5, 2.0)),
...     Zoom(zoom_rate=(1.2, 2))
...     # Rescale(option='zero'),
... ], random = True)
>>> dist = ImageDetectionDistributor(x_list, y_list=y_list,
                                    class_list=class_list,callback=datagenerator,
                                    shuffle=True, imsize=(360, 360), color='RGB')
>>> for i, (x, y) in enumerate(dist.batch(32)):
...     print 'Batch', i
batch ( batch_size , shuffle )

generator of batch images.

Parameters: batch_size ( int ) – size of a batch.
Returns: Images(4 dimension) of input data for Network. If including labels, return with transformed labels
Return type: ndarray
class renom.utility.distributor.threadingdistributor. ImageClassificationDistributor ( image_path_list , y_list=None , class_list=None , imsize=(360 , 360) , color='RGB' , augmentation=None )

Bases: renom.utility.distributor.threadingdistributor.ImageDistributor

Distributor class for tasks of image classification.

Parameters:
  • image_path_list ( list ) – list of image path
  • y_list ( list ) – list of labels (bbox and class) for every image
  • class_list ( list ) – list of classes name for this dataset
  • shuffle ( bool ) – If True, apply datasets shuffle per epoch
  • imsize ( tuple ) – resize input image for converting batch ndarray
  • color ( str ) – color of Input Image. [“RGB”, “GRAY”]
  • augmentation ( fucntion ) – augmentater for Input Image
Example:
>>> from renom.utility.load.imageloader.threadingdistributor import ImageClassificationDistributor
>>> from renom.utility.image.data_augmentation import *
>>> datagenerator = DataAugmentation([
...     Flip(1),
...     Rotate(20),
...     Crop(size=(300, 300)),
...     Resize(size=(500, 500)),
...     Shift((20, 50)),
...     Color_jitter(v = (0.5, 2.0)),
...     Zoom(zoom_rate=(1.2, 2))
...     # Rescale(option='zero'),
... ], random = True)
>>> dist = ImageClassificationDistributor(x_list, y_list=y_list,
                                            class_list=class_list, callback=datagenerator,
                                            shuffle=True, imsize=(360, 360), color='RGB')
>>> for i, (x, y) in enumerate(dist.batch(32)):
...     print 'Batch', i
batch ( batch_size , shuffle )
Parameters: batch_size ( int ) – size of a batch.
Returns: Images(4 dimension) of input data for Network. If including labels, return with original labels
Return type: ndarray
renom.utility.distributor.utilities. get_appropriate_directory ( directory )

Use to check ‘directory’ is a directory and ends with ‘/’

Parameters: directory ( str ) – a directory path
Return str: a directory path which ends with ‘/’
Example:
>>> from utilities import get_appropriate_directory
>>> get_appropriate_directory('hogehoge')
'hogehoge/'
>>> get_appropriate_directory('NOT_EXISTING_DIRECTORY')
'input existing directory'
renom.utility.distributor.utilities. convert_class_to_onehot ( class_list )

Use to create a one-hot vectors list from class_list

Parameters: class_list ( list ) – sorted list of all the class names
Return list: list of one-hot vectors the index of which corresponds to the index of class_list
Example:
>>> from utilities import convert_class_to_onehot
>>> class_list = ['apple', 'pen', 'pinapple']
>>> convert_class_to_onehot(class_list)
[[0, 0, 1], [0, 1, 0], [1, 0, 0]]
renom.utility.distributor.utilities. is_image ( filepath )

Use to check if the extention of ‘filepath’ is that of image files

Parameters: filepath ( str ) – a path which tells a certain image file
Return boolean:
Example:
>>> from utilities import is_image
>>> img_filepath, noimg_filepath = './hogehoge.jpg', './foobar.txt'
>>> is_image(img_filepath)
True
>>> is_image(noimg_filepath)
False
renom.utility.distributor.utilities. get_class_from_onehot ( onehot_vector , class_list )

Use to get the corresponding name from a one-hot vector

Parameters:
  • onehot_vector ( list (of int) ) – one-hot vector
  • class_list ( list (of str) ) – a sorted list of all the class names
Return str:

name of the class

Example:
>>> from utilities import get_class_from_onehot
>>> onehot_vectors = [[0, 1, 0], [1, 0, 0]]
>>> class_list = ['apple', 'pen', 'pinapple']
>>> transformed = [get_class_from_onehot(onehot, class_list) for onehot in onehot_vectors]
>>> transformed
['pen', 'pinapple']
renom.utility.distributor.utilities. convert_minmax_to_wh ( bndbox )

Use to transform [x_min, y_min, x_max, y_max] to [X(x_center), Y(y_center), W(width), H(height)].

Parameters: bndbox ( list (of int) ) – bounding box which is expressed as [x_min, y_min, x_max, y_max]
Return trans_list (of double):
Example:
>>> from utililies import convert_minmax_to_wh
>>> bndbox = [10, 20, 100, 80]
>>> transformed = convert_minmax_to_wh(bndbox)
>>> transformed
[55.0, 50.0, 90.0, 60.0]
renom.utility.distributor.utilities. convert_wh_to_minmax ( bndbox )

Use to transform [X(x_center), Y(y_center), W(width), H(height)] to [x_min, y_min, x_max, y_max].

Parameters: bndbox ( list (of double) ) – bounding box which is expressed as [X(x_center), Y(y_center), W(width), H(height)]
Return list (of int):
Example:
>>> bndbox = [30.5, 60.0, 50.0, 80.0]
>>> transformed = convert_wh_to_minmax(bndbox)
>>> transformed
[5, 20, 55, 100]
renom.utility.distributor.utilities. convert_coco_to_minmax ( bndbox )

Use to transform [xmin, ymin, W, H] to [xmin, ymin, xmax, ymax]

Parameters: bndbox ( list (of double) ) – bounding box which is expressed as [x_min, y_min, W(width), H(height)]
Return list (of int):
Example:
>>> from utilities import convert_coco_to_minmax
>>> bndbox = [10.0, 20.0, 40.0, 60.0]
>>> transformed = convert_coco_to_minmax(bndbox)
>>> transformed
[10, 20, 30, 50]
renom.utility.distributor.utilities. get_num_images ( Y_list )

Use to get number of Y_list.

Parameters: Y_list ( list ) – result from load_for_detection.load_for_detection
Result int: the number of the images in Y_list
Example:
>>> from utilities import get_num_images
>>> Y_list = [{'data': [{'bndbox': [1, 111, 58, 169], 'name': 'tvmonitor'},
                        {'bndbox': [371, 160, 476, 404], 'name': 'person'},
                        {'bndbox': [227, 99, 365, 404], 'name': 'person'},
                        {'bndbox': [146, 207, 263, 366], 'name': 'chair'}],
               'filepath': 'VOC2012/2007_001717.jpg'},
              {'data': [{'bndbox': [153, 22, 355, 342], 'name': 'sheep'}],
               'filepath': 'VOC2012/2007_001872.jpg'}]
>>> get_num_images(Y_list)
2
renom.utility.distributor.utilities. get_max_num_objects ( Y_list )

Use to get the maximum number of the objects in a image.

Parameters: Y_list ( list: ) – result from load_for_detection.load_for_detection
Return int: number of the maximum number of the objects in one image
Example:
>>> from utilities import get_max_num_objects
>>> Y_list = [{'data': [{'bndbox': [1, 111, 58, 169], 'name': 'tvmonitor'},
                    {'bndbox': [371, 160, 476, 404], 'name': 'person'},
                    {'bndbox': [227, 99, 365, 404], 'name': 'person'},
                    {'bndbox': [146, 207, 263, 366], 'name': 'chair'}],
               'filepath': 'VOC2012/2007_001717.jpg'},
              {'data': [{'bndbox': [153, 22, 355, 342], 'name': 'sheep'}],
               'filepath': 'VOC2012/2007_001872.jpg'}]
>>> get_max_num_objects(Y_list)
4
renom.utility.distributor.utilities. get_class_list ( Y_list )

Use to get a list of the names sorted by alphabet that appear in Y_list. This is a legacy code.

Parameters: Y_list ( list ) – result from load_for detection.load_for_detection
Return list: sorted list of the class names that appear in Y_list
Example:
>>> from utilities import get_class_list
>>> Y_list = [{'data': [{'bndbox': [1, 111, 58, 169], 'name': 'tvmonitor'},
                        {'bndbox': [371, 160, 476, 404], 'name': 'person'},
                        {'bndbox': [227, 99, 365, 404], 'name': 'person'},
                        {'bndbox': [146, 207, 263, 366], 'name': 'chair'}],
               'filepath': 'VOC2012/2007_001717.jpg'},
              {'data': [{'bndbox': [153, 22, 355, 342], 'name': 'sheep'}],
               'filepath': 'VOC2012/2007_001872.jpg'}]
>>> class_list = get_class_list(Y_list)
>>> class_list
['chair', 'person', 'sheep', 'tvmonitor']
renom.utility.distributor.utilities. convert_name_to_onehot ( Y_list )

Use to convert all the name in Y_list (result from load_for_detection) to one hot data. names are sorted by alphabet. This is a legacy code.

Parameters: Y_list ( list ) – result from load_for_detection.load_for_detection
Return dictionary:
the keys are names and the values are the corresponding one-hot vectors
Example:
>>> from utilities import convert_name_to_onehot
>>> Y_list = [{'data': [{'bndbox': [1, 111, 58, 169], 'name': 'tvmonitor'},
                        {'bndbox': [371, 160, 476, 404], 'name': 'person'},
                        {'bndbox': [227, 99, 365, 404], 'name': 'person'},
                        {'bndbox': [146, 207, 263, 366], 'name': 'chair'}],
               'filepath': 'VOC2012/2007_001717.jpg'},
              {'data': [{'bndbox': [153, 22, 355, 342], 'name': 'sheep'}],
               'filepath': 'VOC2012/2007_001872.jpg'}]
>>> result = convert_name_to_onehot(Y_list)
>>> result
{'chair': [0, 0, 0, 1], 'person', [0, 0, 1, 0], 'sheep': [0, 1, 0, 0], 'tvmonitor': [1, 0, 0, 0]}
renom.utility.distributor.utilities. make_ndarray ( Y_list , class_length )

Use to make ndarray from the result of load_for_detection

Parameters:
  • Y_list ( list ) – result from load_for_detection.load_for_detection
  • class_length ( int ) – length of the class_list
Return tuple(np.ndarray, int):
element1: ndarray([[X Y W H 0 0 0 1 0 X Y W H 0 0 0 0 1 ...], ...]])
element2: int
Example:
>>> import numpy as np
>>> from utilities import make_ndarray, get_max_num_length
>>> Y_list = [[{'bndbox': convert_minmax_to_wh([1, 111, 58, 169]), 'name': [1, 0, 0, 0]},
               {'bndbox': trans_minmax_to_WH([371, 160, 476, 404]), 'name': [0, 0, 1, 0]},
               {'bndbox': trans_minmax_to_WH([227, 99, 365, 404]), 'name': [0, 0, 1, 0]},
               {'bndbox': trans_minmax_to_WH([146, 207, 263, 366]), 'name': [0, 0, 0, 1]}],
              [{'bndbox': trans_minmax_to_WH([153, 22, 355, 342]), 'name': [0, 1, 0, 0}]]
>>> class_length = get_max_num_length(Y_list)
>>> result = make_ndarray(Y_list, class_length)
>>> result
array([[  29.5,  140. ,   57. ,   58. ,    1. ,    0. ,    0. ,    0. ,
         423.5,  282. ,  105. ,  244. ,    0. ,    0. ,    1. ,    0. ,
         296. ,  251.5,  138. ,  305. ,    0. ,    0. ,    1. ,    0. ,
         204.5,  286.5,  117. ,  159. ,    0. ,    0. ,    0. ,    1. ],
       [ 254. ,  182. ,  202. ,  320. ,    0. ,    1. ,    0. ,    0. ,
           0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,
           0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,
           0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ]])
renom.utility.distributor.utilities. generate_colors_from_name_list ( class_list )

Use to generate a dictionary whose key and value are a name and color value

Parameters: class_list ( list ) – sorted list of all the class names
Return dict: {name1: rgb1, name2: rgb2, ...}
Example:
>>> from utilities import generate_colors_from_name_list
>>> class_list = ['apple', 'pen', 'pinapple']
>>> rgb_dict = generate_colors_from_name_list(class_list)
>>> rgb_dict
{'apple': (105, 130, 29), 'pen': (246, 195, 65), 'pinapple': (197, 195, 92)}
(values are randomly chosen)
renom.utility.distributor.utilities. imshow_with_bndboxes ( image , names , bndboxes , save_name=None )

Use to show an image with the corresponding bounding boxes and object names

Parameters:
  • image ( np.ndarray ) – numpy array of an image
  • names ( list ) – list of the class names
  • bndboxes ( list ) – list of the bounding boxes (= [ [xmin, ymin, xmax, ymax], ... ]
  • save_name ( str or None ) – filepath to which the created image is to be saved
Return None:
Example:
>>> import numpy as np
>>> from PIL import Image
>>> from utilities import imshow_with_bndboxes
>>> from load_for_detection import load_from_xml
>>> xmlpath = '../../test/test_dataset_test_load_for_detection_data/2007_000027.xml'
>>> data = load_from_xml(xmlpath)
>>> image = np.array(Image.open(data['filepath']))
>>> names = [dct['name'] for dct in data['data']]
>>> bndboxes = [dct['bndbox'] for dct in data['data']]
>>> save_name = '2007_000027_with_bndboxes.jpg'
>>> imshow_with_bndboxes(image, names, bndboxes, save_name)
renom.utility.distributor.utilities. imshow_batch ( images , shape=None , output_path='./output.png' )

Use to show multiple images

Parameters:
  • images ( list or np.ndarray ) – list (list(image1, image2, ...)) or np.ndarray (np.ndarray[numofimage][height][width][colors])
  • shape ( list ) – 2-dimentional list which represents [the number of vertical images, the number of horizontal images]
Return None:
Example:
>>> import numpy as np
>>> from PIL import Image
>>> from utilities import imshow_batch
>>> image_path = ['../../doc/img_autodoc/image0001.jpg',
                  '../../doc/img_autodoc/image0002.jpg',
                  ...
                  '../../doc/img_autodoc/image0016.jpg']
>>> images = [np.array(Image.open(path)) for path in image_path]
>>> shape = [4, 4]
>>> output_path = 'concatenated.jpg'
>>> imshow_batch(images, shape, output_path)
renom.utility.distributor.utilities. read_bndbox_from_ndarray ( ndarray , num_class )

Use to read bounding boxes from a given ndarray.

Parameters:
  • ndarray ( ndarray ) – result from make_ndarray
  • num_class ( int ) – the number of all the classes
Return tuple(list, list):
element1: bounding boxes ( = [boxes_im1, boxes_im2, ...] )
element2: onehot_vectors ( = [onehot_im1, onehot_im2, ...])
Example:
>>> import numpy as np
>>> from utilities import read_bndbox_from_ndarray
>>> ndarray = np.array([[  29.5,  140. ,   57. ,   58. ,    1. ,    0. ,    0. ,    0. ,
                          423.5,  282. ,  105. ,  244. ,    0. ,    0. ,    1. ,    0. ,
                          296. ,  251.5,  138. ,  305. ,    0. ,    0. ,    1. ,    0. ,
                          204.5,  286.5,  117. ,  159. ,    0. ,    0. ,    0. ,    1. ],
                        [ 254. ,  182. ,  202. ,  320. ,    0. ,    1. ,    0. ,    0. ,
                            0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,
                            0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,
                            0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ,    0. ]])
>>> num_class = 4
>>> bounding_boxes, onehot_vectors = read_bndbox_from_ndarray(ndarray, num_class)
>>> bounding_boxes
[[[  29.5,  140. ,   57. ,   58. ],
  [ 423.5,  282. ,  105. ,  244. ],
  [ 296. ,  251.5,  138. ,  305. ],
  [ 204.5,  286.5,  117. ,  159. ]],
 [[ 254. ,  182. ,  202. ,  320. ]]]
 >>> onehot_vectors
 >>> [[[1, 0, 0, 0],
       [0, 0, 1, 0],
       [0, 0, 1, 0],
       [0, 0, 0, 1]],
      [[0, 1, 0, 0]]]
renom.utility.distributor.utilities. build_yolo_labels ( y , total_w , total_h , cells , classes )

Use to transform a list of objects per image into a image*cells*cells*(5+classes) matrix. Each cell in image can only be labeled for 1 object.

“5” represents: objectness (0 or 1) and X Y W H

Parameters:
  • y ( np.ndarray ) – np.ndarray ([batch][width][height])
  • total_w ( int ) – length of rows of y
  • total_h ( int ) – length of columns of y
  • cells ( int ) – grid size
  • classes ( int ) – length of class_list
Return np.ndarray:

[batch][cells][cells][1 + 4 + classes]

Example:
>>> Input: 2 objects in first image, 5 classes
>>> y[0] = X Y W H 0 1 0 0 0 X Y W H 0 0 0 1 0
    |---1st object----||---2nd object---|
>>> Output: 7 * 7 cells * (1 + 4 + 5) per image
>>> truth[0,0,0] = 1 X Y W H 0 1 0 0 0
    (cell 0,0 has first object)
>>> truth[0,0,1] = 0 0 0 0 0 0 0 0 0 0
    (cell 0,1 has no object)