Data

Data Reader

class deoxys.data.data_reader.DataReader(*args, **kwargs)[source]

The base class of the Data Reader. Any newly created DataReader will inherit from this class.

original_test
test_generator

Data Generator for the test dataset

Returns:An DataGenerator instance that generates the test dataset
Return type:deoxys.data.DataGenerator
train_generator

Data Generator for the training dataset

Returns:An DataGenerator instance that generates the train dataset
Return type:deoxys.data.DataGenerator
val_generator

Data Generator for the validation dataset

Returns:An DataGenerator instance that generates the validation dataset
Return type:deoxys.data.DataGenerator
class deoxys.data.data_reader.DataReaders[source]

A singleton that contains all the registered customized DataReaders

data_readers
register(key, dr)[source]
unregister(key)[source]
class deoxys.data.data_reader.H5MultiReader(filename, batch_size=32, preprocessors=None, x_name='x', y_name='y', batch_cache=10, train_folds=None, test_folds=None, val_folds=None, fold_prefix='fold', shuffle=False, augmentations=None, other_input_names=None, other_preprocessors=None, other_augmentations=None)[source]

DataReader that use data from an hdf5 file.

Initialize a HDF5 Data Reader, which reads data from a HDF5 file. This file should be split into groups. Each group contain datasets, each of which is a column in the data.

Example:

The dataset X contain 1000 samples, with 4 columns: x, y, z, t. Where x is the main input, y and z are supporting information (index, descriptions) and t is the target for prediction. We want to test 30% of this dataset, and have a cross validation of 100 samples.

Then, the hdf5 containing dataset X should have 10 groups, each group contains 100 samples. We can name these groups ‘fold_1’, ‘fold_2’, ‘fold_3’, … , ‘fold_9’, ‘fold_10’. Each group will then have 4 datasets: x, y, z and t, each of which has 100 items.

Since x is the main input, then x_name=’x’, and t is the target for prediction, then y_name=’t’. We named the groups in the form of fold_n, then fold_prefix=’fold’.

Let’s assume the data is stratified, we want to test on the last 30% of the data, so test_folds=[8, 9, 10]. 100 samples is used for cross-validation. Thus, one option for train_folds and val_folds is train_folds=[1,2,3,4,5,6] and val_folds=[7]. Or in another experiment, you can set train_folds=[2,3,4,5,6,7] and val_folds=[1].

If the hdf5 didn’t has any formular for group name, then you can set fold_prefix=None then put the full group name directly to train_folds, val_folds and test_folds.

Parameters:
  • filename (str) – The hdf5 file name that contains the data.
  • batch_size (int, optional) – Number of sample to feeds in the neural network in each step, by default 32
  • preprocessors (list of deoxys.data.Preprocessor, optional) – List of preprocessors to apply on the data, by default None
  • x_name (str, optional) – Dataset name to be use as input, by default ‘x’
  • y_name (str, optional) – Dataset name to be use as target, by default ‘y’
  • batch_cache (int, optional) – Number of batches to be cached when reading the file, by default 10
  • train_folds (list of int, or list of str, optional) – List of folds to be use as train data, by default None
  • test_folds (list of int, or list of str, optional) – List of folds to be use as test data, by default None
  • val_folds (list of int, or list of str, optional) – List of folds to be use as validation data, by default None
  • fold_prefix (str, optional) – The prefix of the group name in the HDF5 file, by default ‘fold’
  • shuffle (bool, optional) – shuffle data while training, by default False
  • augmentations (list of deoxys.data.Preprocessor, optional) – apply augmentation when generating traing data, by default None
original_test

Return a dictionary of all data in the test set

original_val

Return a dictionary of all data in the val set

test_generator

returns: A DataGenerator for generating batches of data for testing :rtype: deoxys.data.DataGenerator

train_generator

returns: A DataGenerator for generating batches of data for training :rtype: deoxys.data.DataGenerator

val_generator

returns: A DataGenerator for generating batches of data for validation :rtype: deoxys.data.DataGenerator

class deoxys.data.data_reader.H5PatchReader(filename, batch_size=32, preprocessors=None, x_name='x', y_name='y', batch_cache=10, train_folds=None, test_folds=None, val_folds=None, fold_prefix='fold', patch_size=128, overlap=0.5, shuffle=False, augmentations=False, preprocess_first=True, drop_fraction=0.1, check_drop_channel=None, bounding_box=False)[source]
original_test

Return a dictionary of all data in the test set

original_val

Return a dictionary of all data in the val set

test_generator

returns: A DataGenerator for generating batches of data for testing :rtype: deoxys.data.DataGenerator

train_generator

returns: A DataGenerator for generating batches of data for training :rtype: deoxys.data.DataGenerator

val_generator

returns: A DataGenerator for generating batches of data for validation :rtype: deoxys.data.DataGenerator

class deoxys.data.data_reader.H5Reader(filename, batch_size=32, preprocessors=None, x_name='x', y_name='y', batch_cache=10, train_folds=None, test_folds=None, val_folds=None, fold_prefix='fold', shuffle=False, augmentations=None)[source]

DataReader that use data from an hdf5 file.

Initialize a HDF5 Data Reader, which reads data from a HDF5 file. This file should be split into groups. Each group contain datasets, each of which is a column in the data.

Example:

The dataset X contain 1000 samples, with 4 columns: x, y, z, t. Where x is the main input, y and z are supporting information (index, descriptions) and t is the target for prediction. We want to test 30% of this dataset, and have a cross validation of 100 samples.

Then, the hdf5 containing dataset X should have 10 groups, each group contains 100 samples. We can name these groups ‘fold_1’, ‘fold_2’, ‘fold_3’, … , ‘fold_9’, ‘fold_10’. Each group will then have 4 datasets: x, y, z and t, each of which has 100 items.

Since x is the main input, then x_name=’x’, and t is the target for prediction, then y_name=’t’. We named the groups in the form of fold_n, then fold_prefix=’fold’.

Let’s assume the data is stratified, we want to test on the last 30% of the data, so test_folds=[8, 9, 10]. 100 samples is used for cross-validation. Thus, one option for train_folds and val_folds is train_folds=[1,2,3,4,5,6] and val_folds=[7]. Or in another experiment, you can set train_folds=[2,3,4,5,6,7] and val_folds=[1].

If the hdf5 didn’t has any formular for group name, then you can set fold_prefix=None then put the full group name directly to train_folds, val_folds and test_folds.

Parameters:
  • filename (str) – The hdf5 file name that contains the data.
  • batch_size (int, optional) – Number of sample to feeds in the neural network in each step, by default 32
  • preprocessors (list of deoxys.data.Preprocessor, optional) – List of preprocessors to apply on the data, by default None
  • x_name (str, optional) – Dataset name to be use as input, by default ‘x’
  • y_name (str, optional) – Dataset name to be use as target, by default ‘y’
  • batch_cache (int, optional) – Number of batches to be cached when reading the file, by default 10
  • train_folds (list of int, or list of str, optional) – List of folds to be use as train data, by default None
  • test_folds (list of int, or list of str, optional) – List of folds to be use as test data, by default None
  • val_folds (list of int, or list of str, optional) – List of folds to be use as validation data, by default None
  • fold_prefix (str, optional) – The prefix of the group name in the HDF5 file, by default ‘fold’
  • shuffle (bool, optional) – shuffle data while training, by default False
  • augmentations (list of deoxys.data.Preprocessor, optional) – apply augmentation when generating traing data, by default None
original_test

Return a dictionary of all data in the test set

original_val

Return a dictionary of all data in the val set

test_generator

returns: A DataGenerator for generating batches of data for testing :rtype: deoxys.data.DataGenerator

train_generator

returns: A DataGenerator for generating batches of data for training :rtype: deoxys.data.DataGenerator

val_generator

returns: A DataGenerator for generating batches of data for validation :rtype: deoxys.data.DataGenerator

class deoxys.data.data_reader.HDF5Reader(filename, batch_size=32, preprocessors=None, x_name='x', y_name='y', batch_cache=10, train_folds=None, test_folds=None, val_folds=None, fold_prefix='fold')[source]

DataReader that use data from an hdf5 file.

Initialize a HDF5 Data Reader, which reads data from a HDF5 file. This file should be split into groups. Each group contain datasets, each of which is a column in the data.

Example:

The dataset X contain 1000 samples, with 4 columns: x, y, z, t. Where x is the main input, y and z are supporting information (index, descriptions) and t is the target for prediction. We want to test 30% of this dataset, and have a cross validation of 100 samples.

Then, the hdf5 containing dataset X should have 10 groups, each group contains 100 samples. We can name these groups ‘fold_1’, ‘fold_2’, ‘fold_3’, … , ‘fold_9’, ‘fold_10’. Each group will then have 4 datasets: x, y, z and t, each of which has 100 items.

Since x is the main input, then x_name=’x’, and t is the target for prediction, then y_name=’t’. We named the groups in the form of fold_n, then fold_prefix=’fold’.

Let’s assume the data is stratified, we want to test on the last 30% of the data, so test_folds=[8, 9, 10]. 100 samples is used for cross-validation. Thus, one option for train_folds and val_folds is train_folds=[1,2,3,4,5,6] and val_folds=[7]. Or in another experiment, you can set train_folds=[2,3,4,5,6,7] and val_folds=[1].

If the hdf5 didn’t has any formular for group name, then you can set fold_prefix=None then put the full group name directly to train_folds, val_folds and test_folds.

Parameters:
  • filename (str) – The hdf5 file name that contains the data.
  • batch_size (int, optional) – Number of sample to feeds in the neural network in each step, by default 32
  • preprocessors (list of deoxys.data.Preprocessor, optional) – List of preprocessors to apply on the data, by default None
  • x_name (str, optional) – Dataset name to be use as input, by default ‘x’
  • y_name (str, optional) – Dataset name to be use as target, by default ‘y’
  • batch_cache (int, optional) – Number of batches to be cached when reading the file, by default 10
  • train_folds (list of int, or list of str, optional) – List of folds to be use as train data, by default None
  • test_folds (list of int, or list of str, optional) – List of folds to be use as test data, by default None
  • val_folds (list of int, or list of str, optional) – List of folds to be use as validation data, by default None
  • fold_prefix (str, optional) – The prefix of the group name in the HDF5 file, by default ‘fold’
original_test

Return a dictionary of all data in the test set

original_val

Return a dictionary of all data in the val set

test_generator

returns: A DataGenerator for generating batches of data for testing :rtype: deoxys.data.DataGenerator

train_generator

returns: A DataGenerator for generating batches of data for training :rtype: deoxys.data.DataGenerator

val_generator

returns: A DataGenerator for generating batches of data for validation :rtype: deoxys.data.DataGenerator

deoxys.data.data_reader.datareader_from_config(config)[source]
deoxys.data.data_reader.register_datareader(key, dr)[source]

Register the customized data reader. If the key name is already registered, it will raise a KeyError exception.

Parameters:
  • key (str) – The unique key-name of the data reader
  • dr (deoxys.data.DataReader) – The customized data reader class
deoxys.data.data_reader.unregister_datareader(key)[source]

Remove the registered data reader with the key-name

Parameters:key (str) – The key-name of the data reader to be removed

Data Generator

class deoxys.data.data_generator.DataGenerator[source]
description

Description of the size and number of input items in the data

Returns:List of information
[{
    'shape': (128, 128, 2),
    'total': 100
},{
    'shape': (256, 256, 2),
    'total': 100
}]
Return type:list of dictionary
generate()[source]
total_batch

Total number of batches to iterate all data

Returns:Total number of batches to iterate all data
Return type:int
class deoxys.data.data_generator.H5DataGenerator(h5file, batch_size=32, batch_cache=10, preprocessors=None, x_name='x', y_name='y', folds=None, shuffle=False, augmentations=None)[source]
description

Description of the size and number of input items in the data

Returns:List of information
[{
    'shape': (128, 128, 2),
    'total': 100
},{
    'shape': (256, 256, 2),
    'total': 100
}]
Return type:list of dictionary
generate()[source]

Create a generator that generate a batch of data

Yields:tuple of 2 arrays – batch of (input, target)
next_fold()[source]
next_seg()[source]
total_batch

Total number of batches to iterate all data. It will be used as the number of steps per epochs when training or validating data in a model.

Returns:Total number of batches to iterate all data
Return type:int
class deoxys.data.data_generator.H5MultiDataGenerator(h5file, batch_size=32, batch_cache=10, preprocessors=None, x_name='x', y_name='y', folds=None, shuffle=False, augmentations=None, other_input_names=None, other_preprocessors=None, other_augmentations=None)[source]
description

Description of the size and number of input items in the data

Returns:List of information
[{
    'shape': (128, 128, 2),
    'total': 100
},{
    'shape': (256, 256, 2),
    'total': 100
}]
Return type:list of dictionary
generate()[source]

Create a generator that generate a batch of data

Yields:tuple of 2 arrays – batch of (input, target)
next_fold()[source]
next_seg()[source]
total_batch

Total number of batches to iterate all data. It will be used as the number of steps per epochs when training or validating data in a model.

Returns:Total number of batches to iterate all data
Return type:int
class deoxys.data.data_generator.H5PatchGenerator(h5_filename, batch_size=32, batch_cache=10, preprocessors=None, x_name='x', y_name='y', folds=None, patch_size=128, overlap=0.5, shuffle=False, augmentations=False, preprocess_first=True, drop_fraction=0, check_drop_channel=None, bounding_box=False)[source]
description

Description of the size and number of input items in the data

Returns:List of information
[{
    'shape': (128, 128, 2),
    'total': 100
},{
    'shape': (256, 256, 2),
    'total': 100
}]
Return type:list of dictionary
generate()[source]

Create a generator that generate a batch of data

Yields:tuple of 2 arrays – batch of (input, target)
next_fold()[source]
next_seg()[source]
total_batch

Total number of batches to iterate all data. It will be used as the number of steps per epochs when training or validating data in a model.

Returns:Total number of batches to iterate all data
Return type:int
class deoxys.data.data_generator.HDF5DataGenerator(h5file, batch_size=32, batch_cache=10, preprocessors=None, x_name='x', y_name='y', folds=None)[source]
description

Description of the size and number of input items in the data

Returns:List of information
[{
    'shape': (128, 128, 2),
    'total': 100
},{
    'shape': (256, 256, 2),
    'total': 100
}]
Return type:list of dictionary
generate()[source]

Create a generator that generate a batch of data

Yields:tuple of 2 arrays – batch of (input, target)
next_fold()[source]
next_seg()[source]
total_batch

Total number of batches to iterate all data. It will be used as the number of steps per epochs when training or validating data in a model.

Returns:Total number of batches to iterate all data
Return type:int

Data Preprocessor

class deoxys.data.preprocessor.BasePreprocessor(**kwargs)[source]
transform(x, y)[source]
class deoxys.data.preprocessor.ChannelRemoval(channel=1)[source]

Remove one or more channels from the images

Parameters:channel (int, list, tuple, optional) – the index of the channel to be removed, by default 1
transform(images, targets)[source]
class deoxys.data.preprocessor.ChannelSelector(channel=0)[source]

Select / filter one or more channels from the images

Parameters:channel (int, list, tuple, optional) – the index of the channel to be selected, by default 0
transform(images, targets)[source]
class deoxys.data.preprocessor.ClassificationImageAugmentation2D(rotation_range=0, rotation_chance=0.2, zoom_range=1, zoom_chance=0.2, shift_range=None, shift_chance=0.1, flip_axis=None, brightness_range=1, brightness_channel=None, brightness_chance=0.1, contrast_range=1, contrast_channel=None, contrast_chance=0.1, noise_variance=0, noise_channel=None, noise_chance=0.1, blur_range=0, blur_channel=None, blur_chance=0.1, fill_mode='constant', cval=0)[source]

Apply transformation in 2d input images for augmentation in classification. Check ClassificationImageAugmentation3D for augmentation on 3d images

Parameters:
  • rotation_range (int, optional) – range of the angle rotation, in degree, by default 0 (no rotation)
  • rotation_chance (float, optional) – probability to apply rotation transformation to an image, by default 0.2
  • zoom_range (float, list, tuple optional) – the range of zooming, zooming in when the number is less than 1, and zoom out when the number if larger than 1. If a float, then it is the range between that number and 1, by default 1 (no zooming)
  • zoom_chance (float, optional) – probability to apply zoom transformation to an image, by default 0.2
  • shift_range (tuple or list, optional) – the range of translation in each axis, by default None (no shifts)
  • shift_chance (float, optional) – probability to apply translation transformation to an image, by default 0.1
  • flip_axis (int, tuple, list, optional) – flip by one or more axis (in the single image) with a probability of 0.5, by default None (no flipping). flip_axis=0 means the image will be flipped vertically, while flip_axis=1 means the image will be flipped horizontally.
  • brightness_range (float, tuple, list, optional) –

    range of the brightness portion, based on the max intensity value of each channel. For example, when the max intensity value of one channel is 1.0, and the brightness is chaned by 1.2, then every pixel in that channel will increase the intensity value by 0.2.

    \[0.2 = 1.0 \cdot (1.2 - 1)\]

    By default 1 (no changes in brightness)

  • brightness_channel (int, tuple, list, optional) – the channel(s) to apply changes in brightness, by default None (apply to all channels)
  • brightness_chance (float, optional) – probability to apply brightness change transform to an image, by default 0.1
  • contrast_range (float, tuple, list, optional) – range of the contrast portion, (the histogram range is scaled up or down). By default 1 (no changes in contrast)
  • contrast_channel (int, tuple, list, optional) – the channel(s) to apply changes in contrast, by default None (apply to all channels)
  • contrast_chance (float, optional) – probability to apply contrast change transform to an image, by default 0.1
  • noise_variance (float, tuple, list, optional) – range of the noise variance when adding Gaussian noise to the image, by default 0 (no adding noise)
  • noise_channel (int, tuple, list, optional) – the channel(s) to apply Gaussian noise, by default None (apply to all channels)
  • noise_chance (float, optional) – probability to apply gaussian noise to an image, by default 0.1
  • blur_range (int, tuple, list, optional) – range of the blur sigma when applying the Gaussian filter to the image, by default 0 (no blur)
  • blur_channel (int, tuple, list, optional) – the channel(s) to apply Gaussian blur, by default None (apply to all channels)
  • blur_chance (float, optional) – probability to apply gaussian blur to an image, by default 0.1
  • fill_mode (str, optional) – the fill mode in affine transformation (rotation, zooming, shifting / translation), one of {‘reflect’, ‘constant’, ‘nearest’, ‘mirror’, ‘wrap’}, by default ‘constant’
  • cval (int, optional) – When rotation, or zooming, or shifting is applied to the image, cval is the value to fill past edges of input if fill_mode is ‘constant’. By default 0
transform(images, targets)[source]

Apply augmentation to a batch of images

Parameters:
  • images (np.array) – the image batch
  • targets (np.array, optional) – the target batch, by default None
Returns:

the transformed images batch (and target)
Return type:

np.array
class deoxys.data.preprocessor.ClassificationImageAugmentation3D(rotation_range=0, rotation_axis=0, rotation_chance=0.2, zoom_range=1, zoom_chance=0.2, shift_range=None, shift_chance=0.1, flip_axis=None, brightness_range=1, brightness_channel=None, brightness_chance=0.1, contrast_range=1, contrast_channel=None, contrast_chance=0.1, noise_variance=0, noise_channel=None, noise_chance=0.1, blur_range=0, blur_channel=None, blur_chance=0.1, fill_mode='constant', cval=0)[source]

Apply transformation in 3d input images for augmentation in classification.

Parameters:
  • rotation_range (int, optional) – range of the angle rotation, in degree, by default 0 (no rotation)
  • rotation_chance (float, optional) – probability to apply rotation transformation to an image, by default 0.2
  • zoom_range (float, list, tuple optional) – the range of zooming, zooming in when the number is less than 1, and zoom out when the number if larger than 1. If a float, then it is the range between that number and 1, by default 1 (no zooming)
  • zoom_chance (float, optional) – probability to apply zoom transformation to an image, by default 0.2
  • shift_range (tuple or list, optional) – the range of translation in each axis, by default None (no shifts)
  • shift_chance (float, optional) – probability to apply translation transformation to an image, by default 0.1
  • flip_axis (int, tuple, list, optional) – flip by one or more axis (in the single image) with a probability of 0.5, by default None (no flipping). flip_axis=0 means the image will be flipped vertically, while flip_axis=1 means the image will be flipped horizontally.
  • brightness_range (float, tuple, list, optional) –

    range of the brightness portion, based on the max intensity value of each channel. For example, when the max intensity value of one channel is 1.0, and the brightness is chaned by 1.2, then every pixel in that channel will increase the intensity value by 0.2.

    \[0.2 = 1.0 \cdot (1.2 - 1)\]

    By default 1 (no changes in brightness)

  • brightness_channel (int, tuple, list, optional) – the channel(s) to apply changes in brightness, by default None (apply to all channels)
  • brightness_chance (float, optional) – probability to apply brightness change transform to an image, by default 0.1
  • contrast_range (float, tuple, list, optional) – range of the contrast portion, (the histogram range is scaled up or down). By default 1 (no changes in contrast)
  • contrast_channel (int, tuple, list, optional) – the channel(s) to apply changes in contrast, by default None (apply to all channels)
  • contrast_chance (float, optional) – probability to apply contrast change transform to an image, by default 0.1
  • noise_variance (float, tuple, list, optional) – range of the noise variance when adding Gaussian noise to the image, by default 0 (no adding noise)
  • noise_channel (int, tuple, list, optional) – the channel(s) to apply Gaussian noise, by default None (apply to all channels)
  • noise_chance (float, optional) – probability to apply gaussian noise to an image, by default 0.1
  • blur_range (int, tuple, list, optional) – range of the blur sigma when applying the Gaussian filter to the image, by default 0 (no blur)
  • blur_channel (int, tuple, list, optional) – the channel(s) to apply Gaussian blur, by default None (apply to all channels)
  • blur_chance (float, optional) – probability to apply gaussian blur to an image, by default 0.1
  • fill_mode (str, optional) – the fill mode in affine transformation (rotation, zooming, shifting / translation), one of {‘reflect’, ‘constant’, ‘nearest’, ‘mirror’, ‘wrap’}, by default ‘constant’
  • cval (int, optional) – When rotation, or zooming, or shifting is applied to the image, cval is the value to fill past edges of input if fill_mode is ‘constant’. By default 0
transform(images, targets)[source]

Apply augmentation to a batch of images

Parameters:
  • images (np.array) – the image batch
  • targets (np.array, optional) – the target batch, by default None
Returns:

the transformed images batch (and target)
Return type:

np.array
class deoxys.data.preprocessor.DummyPreprocessor[source]
transform(x, y)[source]
class deoxys.data.preprocessor.HounsfieldWindowingPreprocessor(window_center, window_width, channel, hounsfield_offset=1024)[source]

Set the range of the images

Parameters:
  • window_center (int, float) – the center of the range
  • window_width (int, float) – the range
  • channel (int) – the index of the channel to apply windowing
  • hounsfield_offset (int, float) – the Hounsfield offset
class deoxys.data.preprocessor.ImageAffineTransformPreprocessor(rotation_degree=0, rotation_axis=2, zoom_factor=1, shift=None, flip_axis=None, fill_mode='constant', cval=0)[source]
transform(images, targets)[source]
class deoxys.data.preprocessor.ImageAugmentation2D(rotation_range=0, rotation_chance=0.2, zoom_range=1, zoom_chance=0.2, shift_range=None, shift_chance=0.1, flip_axis=None, brightness_range=1, brightness_channel=None, brightness_chance=0.1, contrast_range=1, contrast_channel=None, contrast_chance=0.1, noise_variance=0, noise_channel=None, noise_chance=0.1, blur_range=0, blur_channel=None, blur_chance=0.1, fill_mode='constant', cval=0)[source]

Apply transformation in 2d image (and mask label) for augmentation. Check ImageAugmentation3D for augmentation on 3d images

Parameters:
  • rotation_range (int, optional) – range of the angle rotation, in degree, by default 0 (no rotation)
  • rotation_chance (float, optional) – probability to apply rotation transformation to an image, by default 0.2
  • zoom_range (float, list, tuple optional) – the range of zooming, zooming in when the number is less than 1, and zoom out when the number if larger than 1. If a float, then it is the range between that number and 1, by default 1 (no zooming)
  • zoom_chance (float, optional) – probability to apply zoom transformation to an image, by default 0.2
  • shift_range (tuple or list, optional) – the range of translation in each axis, by default None (no shifts)
  • shift_chance (float, optional) – probability to apply translation transformation to an image, by default 0.1
  • flip_axis (int, tuple, list, optional) – flip by one or more axis (in the single image) with a probability of 0.5, by default None (no flipping). flip_axis=0 means the image will be flipped vertically, while flip_axis=1 means the image will be flipped horizontally.
  • brightness_range (float, tuple, list, optional) –

    range of the brightness portion, based on the max intensity value of each channel. For example, when the max intensity value of one channel is 1.0, and the brightness is chaned by 1.2, then every pixel in that channel will increase the intensity value by 0.2.

    \[0.2 = 1.0 \cdot (1.2 - 1)\]

    By default 1 (no changes in brightness)

  • brightness_channel (int, tuple, list, optional) – the channel(s) to apply changes in brightness, by default None (apply to all channels)
  • brightness_chance (float, optional) – probability to apply brightness change transform to an image, by default 0.1
  • contrast_range (float, tuple, list, optional) – range of the contrast portion, (the histogram range is scaled up or down). By default 1 (no changes in contrast)
  • contrast_channel (int, tuple, list, optional) – the channel(s) to apply changes in contrast, by default None (apply to all channels)
  • contrast_chance (float, optional) – probability to apply contrast change transform to an image, by default 0.1
  • noise_variance (float, tuple, list, optional) – range of the noise variance when adding Gaussian noise to the image, by default 0 (no adding noise)
  • noise_channel (int, tuple, list, optional) – the channel(s) to apply Gaussian noise, by default None (apply to all channels)
  • noise_chance (float, optional) – probability to apply gaussian noise to an image, by default 0.1
  • blur_range (int, tuple, list, optional) – range of the blur sigma when applying the Gaussian filter to the image, by default 0 (no blur)
  • blur_channel (int, tuple, list, optional) – the channel(s) to apply Gaussian blur, by default None (apply to all channels)
  • blur_chance (float, optional) – probability to apply gaussian blur to an image, by default 0.1
  • fill_mode (str, optional) – the fill mode in affine transformation (rotation, zooming, shifting / translation), one of {‘reflect’, ‘constant’, ‘nearest’, ‘mirror’, ‘wrap’}, by default ‘constant’
  • cval (int, optional) – When rotation, or zooming, or shifting is applied to the image, cval is the value to fill past edges of input if fill_mode is ‘constant’. By default 0
transform(images, targets)[source]

Apply augmentation to a batch of images

Parameters:
  • images (np.array) – the image batch
  • targets (np.array, optional) – the target batch, by default None
Returns:

the transformed images batch (and target)
Return type:

np.array
class deoxys.data.preprocessor.ImageAugmentation3D(rotation_range=0, rotation_axis=0, rotation_chance=0.2, zoom_range=1, zoom_chance=0.2, shift_range=None, shift_chance=0.1, flip_axis=None, brightness_range=1, brightness_channel=None, brightness_chance=0.1, contrast_range=1, contrast_channel=None, contrast_chance=0.1, noise_variance=0, noise_channel=None, noise_chance=0.1, blur_range=0, blur_channel=None, blur_chance=0.1, fill_mode='constant', cval=0)[source]

Apply transformation in 3d image (and mask label) for augmentation

Parameters:
  • rotation_range (int, optional) – range of the angle rotation, in degree, by default 0 (no rotation)
  • rotation_axis (int, optional) – the axis of one image to apply rotation, by default 0
  • rotation_chance (float, optional) – probability to apply rotation transformation to an image, by default 0.2
  • zoom_range (float, list, tuple optional) – the range of zooming, zooming in when the number is less than 1, and zoom out when the number if larger than 1. If a float, then it is the range between that number and 1, by default 1 (no zooming)
  • zoom_chance (float, optional) – probability to apply zoom transformation to an image, by default 0.2
  • shift_range (tuple or list, optional) – the range of translation in each axis, by default None (no shifts)
  • shift_chance (float, optional) – probability to apply translation transformation to an image, by default 0.1
  • flip_axis (int, tuple, list, optional) – flip by one or more axis (in the single image), by default None (no flipping)
  • brightness_range (float, tuple, list, optional) –

    range of the brightness portion, based on the max intensity value of each channel. For example, when the max intensity value of one channel is 1.0, and the brightness is chaned by 1.2, then every pixel in that channel will increase the intensity value by 0.2.

    \[0.2 = 1.0 \cdot (1.2 - 1)\]

    By default 1 (no changes in brightness)

  • brightness_channel (int, tuple, list, optional) – the channel(s) to apply changes in brightness, by default None (apply to all channels)
  • brightness_chance (float, optional) – probability to apply brightness change transform to an image, by default 0.1
  • contrast_range (float, tuple, list, optional) – range of the contrast portion, (the histogram range is scaled up or down). By default 1 (no changes in contrast)
  • contrast_channel (int, tuple, list, optional) – the channel(s) to apply changes in contrast, by default None (apply to all channels)
  • contrast_chance (float, optional) – probability to apply contrast change transform to an image, by default 0.1
  • noise_variance (float, tuple, list, optional) – range of the noise variance when adding Gaussian noise to the image, by default 0 (no adding noise)
  • noise_channel (int, tuple, list, optional) – the channel(s) to apply Gaussian noise, by default None (apply to all channels)
  • noise_chance (float, optional) – probability to apply gaussian noise to an image, by default 0.1
  • blur_range (int, tuple, list, optional) – range of the blur sigma when applying the Gaussian filter to the image, by default 0 (no blur)
  • blur_channel (int, tuple, list, optional) – the channel(s) to apply Gaussian blur, by default None (apply to all channels)
  • blur_chance (float, optional) – probability to apply gaussian blur to an image, by default 0.1
  • fill_mode (str, optional) – the fill mode in affine transformation (rotation, zooming, shifting / translation), one of {‘reflect’, ‘constant’, ‘nearest’, ‘mirror’, ‘wrap’}, by default ‘constant’
  • cval (int, optional) – When rotation, or zooming, or shifting is applied to the image, cval is the value to fill past edges of input if fill_mode is ‘constant’. By default 0
class deoxys.data.preprocessor.ImageNormalizerPreprocessor(vmin=None, vmax=None)[source]

Normalize all channels to the range of the close interval [0, 1]

Parameters:
  • vmin (int, float, list, tuple, optional.) – If an int or a float, it will be the lower limits in all channels, else it should be a list of lower values associated with all axes. By default None (choosing the minimum value of each channel in the image batch)
  • vmax (int, float, list, tuple, optional) – If an int or a float, it will be the upper limits in all channels, else it should be a list of upper values associated with all axes. By default None (choosing the maximum value of each channel in the image batch)
transform(images, targets)[source]
class deoxys.data.preprocessor.KerasImagePreprocessorX(shuffle=False, featurewise_center=False, samplewise_center=False, featurewise_std_normalization=False, samplewise_std_normalization=False, zca_whitening=False, zca_epsilon=1e-06, rotation_range=0, width_shift_range=0.0, height_shift_range=0.0, brightness_range=None, shear_range=0.0, zoom_range=0.0, channel_shift_range=0.0, fill_mode='nearest', cval=0.0, horizontal_flip=False, vertical_flip=False, rescale=None, scale_down=None, preprocessing_function=None, data_format='channels_last', interpolation_order=1, dtype='float32')[source]

Apply keras image augmentation to the input images

transform(x, y)[source]
class deoxys.data.preprocessor.KerasImagePreprocessorY(shuffle=True, featurewise_center=False, samplewise_center=False, featurewise_std_normalization=False, samplewise_std_normalization=False, zca_whitening=False, zca_epsilon=1e-06, rotation_range=0, width_shift_range=0.0, height_shift_range=0.0, brightness_range=None, shear_range=0.0, zoom_range=0.0, channel_shift_range=0.0, fill_mode='nearest', cval=0.0, horizontal_flip=False, vertical_flip=False, rescale=None, scale_down=None, preprocessing_function=None, data_format='channels_last', dtype='float32')[source]
transform(x, y)[source]
class deoxys.data.preprocessor.Preprocessors[source]

A singleton that contains all the registered customized preprocessors

preprocessors
register(key, preprocessor)[source]
unregister(key)[source]
class deoxys.data.preprocessor.SingleChannelPreprocessor(**kwargs)[source]

Make single channel images have one axis for the channel

transform(x, y)[source]
class deoxys.data.preprocessor.UnetPaddingPreprocessor(depth=4, mode='auto')[source]

Pad the images so that their sizes would not change after going through a standard Unet model.

Parameters:
  • depth (int, optional) – number of maxpooling layer in the Unet, by default 4
  • mode (str, optional) – way to fill the values in the paddings, currently only support ‘auto’ (pad with 0)
transform(images, targets)[source]
class deoxys.data.preprocessor.WindowingPreprocessor(window_center, window_width, channel)[source]

Set the range of the images

Parameters:
  • window_center (int, float) – the center of the range
  • window_width (int, float) – the range
  • channel (int) – the index of the channel to apply windowing
perform_windowing(image)[source]
transform(images, targets)[source]
deoxys.data.preprocessor.preprocessor_from_config(config)[source]
deoxys.data.preprocessor.register_preprocessor(key, preprocessor)[source]

Register the customized preprocessor. If the key name is already registered, it will raise a KeyError exception

Parameters:
  • key (str) – The unique key-name of the preprocessor
  • preprocessor (deoxys.data.BasePreprocessor) – The customized preprocessor class
deoxys.data.preprocessor.unregister_preprocessor(key)[source]

Remove the registered preprocessor with the key-name

Parameters:key (str) – The key-name of the preprocessor to be removed